5.2 KiB

Raw Blame History

03 — browser webcam capture

How to get JPEG frames out of getUserMedia() and onto the WebSocket the backend already serves. This part has zero unknowns; it's just plumbing. Documenting the exact code so future-you doesn't have to re-research it.

End-to-end flow

                 ┌────────────────────────────────────┐
 user webcam  ─► │ MediaStream (getUserMedia)         │
                 │  ▼                                  │
                 │ <video autoplay muted>             │
                 │  ▼ (every N ms)                    │
                 │ <canvas> drawImage(video)          │
                 │  ▼                                  │
                 │ canvas.toBlob('image/jpeg', 0.6)   │
                 │  ▼                                  │
                 │ ArrayBuffer  ────► ws.send(binary) │
                 └────────────────────────────────────┘
                                  │
                                  ▼
                          backend Esp32Bridge
                                  │
                                  ▼
                     queued JPEG (ring of 1–2 frames)
                                  │
                                  ▼
                    firmware esp_camera_fb_get()

Browser-side reference (essence; goes into a Velxio React hook)

// useWebcamFrames.ts — simplified, the production hook lives in
// frontend/src/hooks/useWebcamFrames.ts (not yet written)

const stream = await navigator.mediaDevices.getUserMedia({
  video: { width: 320, height: 240, frameRate: { ideal: 10, max: 15 } },
  audio: false,
});

const video = document.createElement('video');
video.srcObject = stream;
video.muted = true;
await video.play();

const canvas = new OffscreenCanvas(320, 240);
const ctx = canvas.getContext('2d')!;

setInterval(async () => {
  ctx.drawImage(video, 0, 0, 320, 240);
  const blob = await canvas.convertToBlob({ type: 'image/jpeg', quality: 0.6 });
  const buf = await blob.arrayBuffer();
  ws.send(buf);                        // binary frame
}, 1000 / 10);                          // 10 fps target

OffscreenCanvas.convertToBlob is supported in Chrome 69+, Firefox 105+, Safari 16.4+ — fine for our audience (the same browsers that run our WebGL/Web Components stack).

Sizing

A 320×240 JPEG at quality 0.6 averages 8–14 KB. At 10 fps that's ~120 KB/s up — trivial over a local WebSocket, and well within the current 16 MB max_size already configured in esp_qemu_manager for binary frames.

VGA (640×480) doubles the bandwidth and the latency; start at QVGA (320×240) and only enlarge if a sketch needs it.

Backend framing

We don't introduce a new WebSocket channel. We re-use the existing Esp32Bridge WebSocket and add one new message type:

// Browser → backend (binary or {type, b64}):
{ "type": "camera_frame",
  "data": { "fmt": "jpeg", "w": 320, "h": 240, "b64": "<…>" } }

The backend keeps a small ring buffer (size 2: current + next) per client. When the firmware-side shim asks for a frame, it gets the "current"; the next browser-pushed frame replaces "next" and atomically becomes "current".

Existing analogue: the DHT22 / HC-SR04 sensor-attach traffic on the same WS — see backend/app/services/esp32_lib_bridge.py.

Permissions caveat

getUserMedia requires:

HTTPS (or localhost) — production already has TLS via the nginx layer in deploy/nginx.prod.conf, dev is localhost.
A user gesture — a "Start camera" button. Not auto-prompt on page load.
Per-origin permission grant — Chrome/Firefox remember it, Safari prompts every session.

A small UI toggle in the canvas header is enough. Don't auto-start; the user has to opt in.

Failure modes worth catching

Symptom	Likely cause
`NotAllowedError`	User denied permission
`NotFoundError`	No camera on device
`NotReadableError`	Camera busy in another app
`OverconstrainedError`	`width:320` not supported by sensor
Black frames	Permission granted but no `<video>.play()`
Frame drops on backend	Bandwidth or backend ring overflow

The hook should expose a status: 'idle' | 'requesting' | 'streaming' | 'error' so the UI can show a clear state next to the board.

5.2 KiB Raw Blame History Unescape Escape