Webux Lab

By Studio Webux

Experimenting with data encoding

TG
Tommy Gingras Studio Webux 2024-10-11

Encoding data structure

A brief overview of what I’ve learned while attempting to efficiently encode a data structure using TypeScript and Deno, inspired by existing practices in the Cardano blockchain ecosystem.

My data: [95, 1, 43, 80, 155, 66, 69, 69] - simple integers representing image IDs for character layering in my game. I need both binary and string formats for easy endpoint calling.


Attempt 1 - AI-Driven Binary Encoding with Optimal Bit Usage

Spoiler alert: Not the most efficient method.

const encoded = encode_data_with_length([95, 1, 43, 80, 155, 66, 69, 69]); // 68 bytes
// encoded = 10000101111100000001001010110101000010011011010000100100010101000101 (First 4 bits determine the number of bits per elements, in this case it selected 8 bits)
const decoded = decode_data_with_length(encoded); // 25 bytes - So skipping the encoding is way more efficient, but not very usable with an endpoint

To note, in this exact case, using the following code produces the same output:

Buffer.from(
  JSON.stringify({
    visual: new Uint8Array([95, 1, 43, 80, 155, 66, 69, 69]),
  }),
).byteLength

However, I needed a custom solution as not all data has an 8-bit structure.

Attempt 2 - CBOR (Concise Binary Object Representation)

import { Buffer } from "node:buffer";
import { decode, encode } from "npm:cbor-x";

console.log(
  encode([95, 1, 43, 80, 155, 66, 69, 69]).toString("hex"), // Prints a usable structure: 88185f01182b1850189b184218451845
  encode([95, 1, 43, 80, 155, 66, 69, 69]).byteLength, // 16 bytes !!!!!
  decode(Buffer.from("88185f01182b1850189b184218451845", "hex")), // simply use a buffer, load the hex encoded and voila !
);

CBOR is the most straightforward and efficient method for encoding/decoding my dataset, similar to how it’s used in Cardano transactions.


Search