Encoding data structure
A brief overview of what I’ve learned while attempting to efficiently encode a data structure using TypeScript and Deno, inspired by existing practices in the Cardano blockchain ecosystem.
My data: [95, 1, 43, 80, 155, 66, 69, 69]
- simple integers representing image IDs for character layering in my game.
I need both binary and string formats for easy endpoint calling.
Attempt 1 - AI-Driven Binary Encoding with Optimal Bit Usage
Spoiler alert: Not the most efficient method.
const encoded = encode_data_with_length([95, 1, 43, 80, 155, 66, 69, 69]); // 68 bytes
// encoded = 10000101111100000001001010110101000010011011010000100100010101000101 (First 4 bits determine the number of bits per elements, in this case it selected 8 bits)
const decoded = decode_data_with_length(encoded); // 25 bytes - So skipping the encoding is way more efficient, but not very usable with an endpoint
To note, in this exact case, using the following code produces the same output:
Buffer.from(
JSON.stringify({
visual: new Uint8Array([95, 1, 43, 80, 155, 66, 69, 69]),
}),
).byteLength
However, I needed a custom solution as not all data has an 8-bit structure.
Attempt 2 - CBOR (Concise Binary Object Representation)
import { Buffer } from "node:buffer";
import { decode, encode } from "npm:cbor-x";
console.log(
encode([95, 1, 43, 80, 155, 66, 69, 69]).toString("hex"), // Prints a usable structure: 88185f01182b1850189b184218451845
encode([95, 1, 43, 80, 155, 66, 69, 69]).byteLength, // 16 bytes !!!!!
decode(Buffer.from("88185f01182b1850189b184218451845", "hex")), // simply use a buffer, load the hex encoded and voila !
);
CBOR is the most straightforward and efficient method for encoding/decoding my dataset, similar to how it’s used in Cardano transactions.