javascript binary data

overview

JavaScript has a convoluted mishmosh of classes used to store and manipulate binary (byte based) data. Additionally, Node.JS added a nice extension that was built on top of the traditional binary data classes for use in Node.JS apps, but it's not included in standard JavaScript for browsers. This text aims to sort out the different types of binary based data structures available in modern javascript in 2023, and proposes some best practices.

only static length blocks of binary data, not dynamic length

All binary data manipulation discussed here is done by using classes that manipulate static length blocks of memory. That is, you either allocate memory having a fixed length in bytes, or are given the memory block by some third party code. You can not increase or decrease the size of that memory. If you really needed to change the size, you would have to allocate a new block of memory, and then manually copy the old block to the new block, then destroy the old block. Note: there is a new resize() function in the ArrayBuffer spec that hasn't been implemented everywhere yet. resize() technically adds a dynamic size change ability to the memory but it doesn't feel dynamic the way a string or an array does. See the short discussion at the end of this text on dynamic storage of binary data.

the ArrayBuffer and its friends

All static binary data in JavaScript is stored in an ArrayBuffer which is then read from and written to via the DataView, Uint8Array and TextEncoder/Decoder classes. This diagram shows how JavaScript's built-in binary data classes fit together. Notice that there are multiple ways an ArrayBuffer can be read from and written to, and it will be the developer's job to determine the right one to use for each case.

classDiagram
  ArrayBuffer <-- TypedArray
  ArrayBuffer <-- DataView
  class ArrayBuffer {
    byteLength: number
  }
  TypedArray <|-- Uint8Array
  TypedArray <|-- Uint32Array
  class TypedArray {
    buffer: ArrayBuffer
    byteLength: number
  }
  class DataView {
    buffer: ArrayBuffer
    byteLength: number
    getUint8(byteOffset)
    setUint8(byteOffset, value)
    getInt32(byteOffset, isLittleEndian=false)
    setInt32(byteOffset, value, isLittleEndian=false)
  }
  class Uint8Array {
    index[]: number
  }
  class Uint32Array {
    index[]: number
  }

classDiagram
  class TextEncoder {
    encoding:string = 'utf-8'
    encode(s:string):Uint8Array
    encodeInto(source:string, dest:Uint8Array)
  }
  class TextDecoder {
    encoding:string = 'utf-8'
    decode(buffer:Uint8Array|DataView|ArrayBuffer):string
  }
  class Blob {
    arrayBuffer()
    stream()
  }

node.js also has its own way (the Buffer class)

The classes mentioned above would be all you would need to manipulate static length binary data in Node.JS, however, node includes an additional method for handling binary data in the form of a Buffer class. Buffer inherits from Uint8Array and therefore combines all the functionality of Uint8Array, plus it adds the functionality of DataView and TextEncoder/Decoder into one convenient class. While node can still use the TypedArray, DataView & TextEncoder classes, the Buffer class is very convenient to use.

classDiagram
  ArrayBuffer <-- TypedArray
  ArrayBuffer <-- DataView
  class ArrayBuffer {
    byteLength: number
  }
  TypedArray <|-- Uint8Array
  TypedArray <|-- Uint32Array
  class TypedArray {
    buffer: ArrayBuffer
    byteLength: number
  }
  class DataView {
    buffer: ArrayBuffer
    byteLength: number
    getUint8(byteOffset)
    setUint8(byteOffset, value)
    getInt32(byteOffset, isLittleEndian=false)
    setInt32(byteOffset, value, isLittleEndian=false)
  }
  class Uint8Array {
    index[]: number
  }
  class Uint32Array {
    index[]: number
  }
  Uint8Array <|-- Buffer
  class Buffer {
    readInt8(byteOffset)
    writeInt8(byteOffset, value)
    readUInt32BE(byteOffset)
    writeUInt32BE(byteOffset, value)
    toString(encoding='utf8', start=0, end=lastByte)
    write(string, offset=0, length=all, encoding='utf8')
 }
%%% class Buffer {class:'green'}

classDiagram
  class TextEncoder {
    encoding:string = 'utf-8'
    encode(s:string):Uint8Array
    encodeInto(source:string, dest:Uint8Array)
  }
  class TextDecoder {
    encoding:string = 'utf-8'
    decode(buffer:Uint8Array|DataView|ArrayBuffer):string
  }
  class Blob {
    arrayBuffer()
    stream()
  }

dynamic length binary data

binary data in strings?

Technically, JavaScript can store binary data in strings dynamically. Unlike some other languages where strings are terminated with null characters, JavaScript simply treats character 0x00 as any other. It's therefore quite tempting to add binary data into a string because then you'll have an easy-to-use dynamic data structure for binary data! However, I do not recommend doing this. While it may seem to work and be ever so convenient, there are numerous tricky caveats involved: JavaScript will automatically interpret data in strings as being "string data" and will gladly transform and adjust the bytes as it sees fit to handle proper Unicode character encoding. Additionally, JavaScript actually stores strings as 2 bytes per character in memory. So, each byte of binary data you put into a string will be stored as 2 bytes, which will double your memory usage---but the main reason for avoiding this is to stay away from the strange character encoding behavior that your binary data will be inadvertently subjected to. An excellent example of this is clearly described here: Binary data in the browser: Untangling an encoding mess with JavaScript Typed Arrays

array of number: a dynamic length binary storage option

The closest thing JavaScript has to dynamic length storage of binary data would be arrays of numbers, where the numbers are limited to values from 0 to 255. ex: let byteArray = [0,10,255] The static length binary storage classes can also convert their contents to and from a number array like so:

examples of converting from a number array containing bytes to Uint8Array, and Buffer

// an array of bytes ...
let byteArray = [0,10,255];
byteArray.push(99); // <-- adding a byte dynamically

// ... to Uint8Array
let uInt8Array = Uint8Array.from(byteArray);
console.log(uInt8Array); // Uint8Array(4) [0, 10, 255, 99]

// ... to Buffer (in Node.JS)
let buffer = Buffer.from(byteArray);
console.log(buffer); // <Buffer 00 0a ff 63> aka [0, 10, 255, 99]

examples of converting from the above uInt8Array, and buffer back to number arrays containing bytes.

// ... from uInt8Array
let byteArray2 = [...uInt8Array.values()];
console.log(byteArray2); // [0, 10, 255, 99]

// ... from buffer (in Node.JS)
let byteArray2 = [...buffer.values()];
console.log(buffer); // [0, 10, 255, 99]

conclusion

If you are writing a library to be used on both the browser and in Node, the best way to deal with binary data is to have all of your functions take and return ArrayBuffer, that way, the developer using your functions can access the data however they see fit. (via the Uint8Array , DataView, Blob, and Buffer in Node.) After that, it's up to you how you want to access the data inside your functions, keeping in mind that the Buffer object is only built into Node, but not the browser. Lastly, if you want to store binary data in a dynamic length data structure, arrays of numbers are the only viable option, as storing binary data in strings is problematic.