The .sgn format

Signa's sign format: the positions of the body, head and hands, frame by frame, one segment per word. About a tenth of the size of a .pose file.

Layout

A file is one or more segments, one per word, one after the other. Every number is little-endian.

Bytes	Content
4	`SGN1`
4	u32: the length of the rest of this segment
2 + 2	u16 width, u16 height of the original frame, in pixels
2	u16 frames per second × 100
1 + 1	u8 body points (B), u8 points per hand (H)
2 + n	u16 length, then the word in UTF-8
4	u32 frame count (F)
F × P × 5	For every frame, P = B + 1 + 2H points (below)

Each point is an i16 dx and i16 dy in 1/16 of a pixel, the change since the same point in the previous frame (starting from 0), and a u8 confidence from 0 to 255.

Points

Order	Points
Body (B = 8)	Left and right shoulder, left and right elbow, left and right wrist, left and right hip
Head (1)	The middle of the face
Left hand (H = 21)	The wrist, then four points per finger from the thumb to the little finger, each from knuckle to tip
Right hand (H = 21)	The same, for the right hand

A point with a low confidence (below about 0.2) was not seen in that frame: hide it, or hold it where it last was.

Decoding

JavaScript

// Decodes every complete segment in `bytes`; returns the words and how many
// bytes they used, so a stream reader can keep the incomplete rest.
function decodeSgn(bytes) {
  const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
  const words = [];
  let off = 0;
  while (bytes.length - off >= 8) {
    const n = view.getUint32(off + 4, true);
    if (bytes.length - off < 8 + n) break; // not all here yet
    let o = off + 8;
    const fps = view.getUint16(o + 4, true) / 100;
    const body = bytes[o + 6], hand = bytes[o + 7];
    const textLen = view.getUint16(o + 8, true);
    const text = new TextDecoder().decode(bytes.subarray(o + 10, o + 10 + textLen));
    const count = view.getUint32(o + 10 + textLen, true);
    o += 14 + textLen;
    const points = body + 1 + 2 * hand;
    const x = new Int32Array(points), y = new Int32Array(points);
    const frames = [];
    for (let f = 0; f < count; f++) {
      const frame = [];
      for (let i = 0; i < points; i++) {
        x[i] += view.getInt16(o, true);
        y[i] += view.getInt16(o + 2, true);
        frame.push({ x: x[i] / 16, y: y[i] / 16, seen: bytes[o + 4] / 255 });
        o += 5;
      }
      frames.push({
        body: frame.slice(0, body),
        head: frame[body],
        left: frame.slice(body + 1, body + 1 + hand),
        right: frame.slice(body + 1 + hand),
      });
    }
    words.push({ text, fps, frames });
    off += 8 + n;
  }
  return { words, used: off };
}

Playing it smoothly

Play the frames at the segment's frame rate (we play them a little faster, ×1.25, which reads more naturally). Between two words, ease from the last frame of one to the first frame of the next over about five frames, and blend between frames to match the screen's refresh rate.