Video API

Frame stepping (forEachFrame, estimateFps) is in the main package. Encoding lives in a separate entry, image-to-pixel/video, with this project's own MP4 and WebM muxers (MIT, no dependencies).

import { createPixelator, forEachFrame, encodeGIF } from 'image-to-pixel';
import { createVideoEncoder, videoSupport } from 'image-to-pixel/video';

const px = await createPixelator();
const settings = { size: { width: 160 }, dither: { method: 'bluenoise' } };

let encoder;
await forEachFrame(video, {
  fps: 24, start: 0, end: 4,
  async onFrame({ index, total }) {
    const frame = await px.process(video, settings);
    encoder ??= await createVideoEncoder({ format: 'mp4', fps: 24, width: frame.width, height: frame.height, scale: 4 });
    await encoder.add(frame);
  },
});
const blob = await encoder.finish();     // video/mp4

With the source's audio#

import { createVideoEncoder, decodeAudio } from 'image-to-pixel/video';

const audio = await decodeAudio(file, { start: 0, end: 4 });   // null if the file has no audio track
encoder ??= await createVideoEncoder({ format: 'mp4', fps: 24, width: frame.width, height: frame.height, scale: 4, audio });
// … add frames as above
const blob = await encoder.finish();     // video and sound
console.log(encoder.audioCodec);         // 'aac', 'opus', or null if this browser can't encode audio

forEachFrame(video, options)#

Seeks to the middle of each frame, waits until it is decoded, and calls onFrame. The next seek waits for onFrame to finish, so no frame is skipped or repeated however slow the processing is.

estimateFps(video, fallback = 30) plays the video muted for a moment and measures presented frames with requestVideoFrameCallback, snapping to common rates.

createVideoEncoder(options)#

Returns { add(frame), finish(), audioCodec }. add takes any { width, height, data }, such as a result. finish resolves with a Blob. audioCodec is 'aac' or 'opus' when the soundtrack is included, and null when there is none or the browser can't encode audio (the video is still made).

MP4 gets AAC when the browser can encode it and Opus otherwise; WebM always gets Opus.

decodeAudio(source, options)#

Decodes the audio track of a File, Blob, ArrayBuffer, URL or <video> element with Web Audio. Resolves with { sampleRate, numberOfChannels, channels, duration }: planar Float32Arrays, 48 kHz by default, at most two channels (surround is mixed down). Resolves with null when there is no audio track, the browser can't decode it, or there is no Web Audio (Node).

encodeAudio(pcm, { codec | config, bitrate, maxDuration, onChunk }) encodes it with AudioEncoder. findAudioConfig(codec) returns a configuration this browser accepts (platform AAC encoders are picky about sample rate and bitrate, so a few common ones are tried) or null, and audioCodecSupported(codec) is its yes/no form.

encodeVideo(frames, options) does the same for an array. videoSupport() reports { mp4, webm, webcodecs, aac, opus } for the current browser.

GIF#

GIF needs no add-on: encodeGIF(frames, { scale, loop }) from the main package takes { width, height, data, delay } frames (delay in ms) and returns the bytes. When all frames fit in 256 colours, which pixel art always does, they share one colour table and the output is lossless.

Muxers#

The muxers are exported too, for encoders of your own.

import { createMp4Muxer, createWebmMuxer } from 'image-to-pixel/video';

const mux = createMp4Muxer({ width, height });
const encoder = new VideoEncoder({ output: (chunk, meta) => mux.addChunk(chunk, meta), error: console.error });
encoder.configure({ codec: 'avc1.42001f', width, height, avc: { format: 'avc' } });
// … encode frames, await encoder.flush()
const bytes = mux.finish();   // Uint8Array, a complete .mp4

addChunk(chunk, meta) takes VideoEncoder output directly. For MP4, configure the encoder with avc: { format: 'avc' } so it emits the decoder configuration the file needs. If that configuration is malformed (Chrome on Windows has produced one with doubled NAL headers), the muxer rebuilds it from the SPS and PPS in the first keyframe. Lower-level addSample() / addFrame() take raw encoded frames with timestamps in microseconds.

For sound, create the muxer with audio: { codec, sampleRate, numberOfChannels } and pass AudioEncoder output to addAudioChunk(chunk, meta), or raw frames to addAudioSample() (MP4) / addAudioFrame() (WebM). For AAC in MP4, configure the encoder with aac: { format: 'aac' } so it emits the AudioSpecificConfig the file needs.