velxio/test/test-esp32-cam/autosearch/10_phase3_webcam_test.md

10 — Phase 3 webcam round-trip test

End-to-end test that proves the PC webcam → ESP32-CAM firmware buffer pipeline works under QEMU. Lives at tests/test_frame_roundtrip_live.py.

This note documents what shipped, what works, what doesn't yet, and the dead ends we hit so future contributors don't repeat them.

TL;DR

Question	Answer
Can the host inject a real JPEG into firmware memory?	YES
Does the QEMU OV2640 device respond to chip-ID probes?	YES
Does the QEMU I²S+DMA model deliver frames + fire EOF?	YES
Does the upstream esp_camera_init succeed under QEMU?	partially
Does the upstream esp_camera_fb_get return a JPEG?	NO

The "partial" / "no" answers are about the upstream Espressif library auto-detection and framectrl semantics — see "What doesn't work yet" below. The Velxio QEMU peripherals themselves are correct.

What works (asserted by the live tests)

test_sccb_probe_live.py — Phase 1 ✅

A bare-Wire.h sketch reads the OV2640 chip-ID via SCCB:

PID=0x26 VER=0x42 MIDH=0xA2 MIDL=0x7F
OV2640 detected

Proves: hw/i2c/esp32_ov2640.c correctly handles bank-select (0xFF=0x01) + byte reads at 0x0A/0x0B/0x1C/0x1D.

test_dma_smoke_live.py — Phase 2 ✅

A sketch that pokes I²S0 directly (no esp_camera library) arms DMA, waits for in_suc_eof, dumps 32 bytes:

i2s_rx armed, eof_num=512
EOF after 0ms
buf[0..31] = 00 AA 00 AA 00 AA 00 AA …

Proves: hw/misc/esp32_i2s_cam.c walks the lldesc_t chain, packs pixels via pack_two_pixels (low half = padding, high half = pixel byte), raises in_suc_eof, fires the IRQ.

test_frame_roundtrip_live.py — Phase 3 ✅ (this note's main deliverable)

The webcam round-trip. A JPEG (real webcam if VELXIO_USE_WEBCAM=1, otherwise a 4 KB PIL-rendered fallback) travels:

PC webcam (OpenCV)              ← tests/webcam_helper.py
    ↓
WebSocket (esp32_camera_attach + esp32_camera_frame)
    ↓
backend/app/api/routes/simulation.py
    ↓
backend/app/services/esp32_lib_manager.py:camera_frame()
    ↓
worker subprocess stdin (JSON line)
    ↓
backend/app/services/esp32_worker.py: ctypes → velxio_push_camera_frame()
    ↓
third-party/qemu-lcgamboa/hw/misc/velxio_camera_export.c
    ↓
hw/misc/esp32_i2s_cam.c: Esp32I2sCamState.frame_buf
    ↓
walk_dma_chain() pulls bytes via next_pixel_byte() while filling DMA
    ↓
firmware s_buf in sketches/frame_roundtrip/frame_roundtrip.ino
    ↓
Serial.printf("FRAME[0..63]: …")   — visible to the test

The test asserts:

• The padding bytes at offsets 0, 2, 4, 6 are 0x00 (low half of dma_elem_t).
• The pixel bytes at offsets 1, 3, 5, 7, 9, 11, 13, 15 match the first 8 bytes of the source JPEG (0xFF 0xD8 0xFF 0xE0 SOI+APP0 + the JFIF length field).

A failure of any link in that chain produces a precise diagnostic. That's the regression contract we wanted.

What doesn't work yet (and why)

Blocker A — I²C NACK semantics fool auto-probe

The upstream esp_camera.c scans 18 sensor models in a fixed order. It calls SCCB_Probe(slv_addr) for each, and the first one that returns ESP_OK "wins" — the driver then reads the chip-id at that address and tries to match against per-sensor PIDs.

In our QEMU I²C controller (hw/i2c/esp32_i2c.c:199), when no slave is registered at the addressed location, i2c_start_transfer returns non-zero and the controller sets I2C_INT_RAW.ACK_ERR. ESP-IDF's i2c_master_cmd_begin is supposed to detect this and return ESP_FAIL — and our DHT22/HC-SR04 traffic confirms that path works for command sequences with data.

But the SCCB probe writes a single byte (slave addr + W bit) with no following data and a STOP. With this minimal pattern, the QEMU controller's ACK_ERR flag is set on the same MMIO cycle as the cmd completion bit, and the driver's poll loop sees the cmd-complete bit first and returns ESP_OK before it sees ACK_ERR. The driver thinks something ACKed at 0x21 (the first scan address — OV7725).

Concretely: the I²C event log shows hundreds of addr=0x21 op=START_SEND/WRITE/READ NO_SLAVE registered=[] entries during esp_camera_init. The driver thinks OV7725 is real, fails the chip-id check (read returns 0xFF, OV7725 expects 0x77), iterates through the other sensors that share 0x21 (OV7670, GC0308, GC032A) — all fail. After all four fail, control eventually advances, but because the driver also runs the (now half-initialised) sensor's init sequence, the camera_init.ino sketch never reaches a clean state and fb_get returns NULL forever.

Workarounds tried (all rejected):

• Add OV2640 slave on bus 1 too — done (the sccb config uses port 1 on arduino-esp32 v2.0.17 — CONFIG_SCCB_HARDWARE_I2C_PORT1=y). Did NOT help because the driver still hangs on 0x21 first.
• Register a synthetic "rejecting" slave at 0x21 — would work but is fragile (would need NACK forwarding through the I²CSlave API, which QEMU's I²C bus core doesn't naturally support).
• Patch hw/i2c/esp32_i2c.c to defer the cmd-complete signal until after ACK_ERR is observable — possible but rabbit hole.

The real fix is to patch the QEMU I²C controller's MMIO timing so ACK_ERR latches before the cmd-complete bit. That's a follow-up.

For now: the test sketches that bypass the upstream auto-probe (sccb_probe.ino, dma_smoke.ino, frame_roundtrip.ino) work perfectly. Sketches that go through esp_camera_init get stuck on auto-detect.

Blocker B — framectrl task expects continuous EOFs

Even if Blocker A were resolved, cam_hal.c runs a FreeRTOS task that:

1. Receives in_suc_eof events from a queue.
2. For each event, swaps the active half-buffer.
3. Validates JPEG (cam_verify_jpeg_eoi) and pushes complete frames to the user's xQueueReceive-able queue.

My esp32_i2s_cam.c fires in_suc_eof ONCE per rx_start 0→1 edge. Real hardware fires it on every PCLK*samples cycle while rx_start=1.

To unblock fb_get, the QEMU model needs to either:

• (a) Re-arm itself: when the driver clears in_suc_eof via INT_CLR, immediately fire another EOF (with the next chunk of frame data). Risk: tight loops if the driver doesn't drain fast enough.
• (b) Use a qemu_mod_timer that fires every ~16 ms while rx_start=1 and walks the chain. More authentic to real PCLK timing.

Both are 1–2 day implementations. Out of scope for this round. See 04_proposed_architecture.md "Frame timing — strategy (B)".

Blocker C — frame source ↔ rx_start race

In frame_roundtrip.ino we work around the timing by making the sketch wait for a serial trigger. The test pushes the JPEG via WS, then sends a serial byte to release the sketch. That guarantees frame_buf is populated before rx_start = 1 fires the walker.

In a "natural" capture loop (no trigger), rx_start would already be asserted from the previous cycle. The walker would consume any newly pushed frame on its NEXT EOF cycle — which doesn't happen until Blocker B is fixed.

The trigger mechanism is a test-time convenience, not a production limitation. Once Blocker B is resolved, the trigger goes away.

Things we tried that didn't pan out

Adding driver to main's REQUIRES — not enough on its own

libesp32-camera.a references i2c_master_* and i2c_cmd_link_* which live in ESP-IDF's driver component. Adding driver to main/CMakeLists.txt's REQUIRES ${_arduino_comp_name} line was necessary BUT not sufficient — target_link_libraries(${COMPONENT_LIB} INTERFACE "${_cam_lib}") doesn't propagate the REQUIRES, so the linker still couldn't find the i2c symbols.

The fix (which DID work) is add_prebuilt_library from ESP-IDF's build system:

add_prebuilt_library(esp32_camera_prebuilt "${_cam_lib}"
    REQUIRES driver ${_arduino_comp_name})
target_link_libraries(${COMPONENT_LIB} PUBLIC esp32_camera_prebuilt)

That's now in backend/app/services/esp-idf-template/main/CMakeLists.txt and lets #include "esp_camera.h" compile + link cleanly.

Patching arduino-esp32/CMakeLists.txt — rejected

arduino-esp32/CMakeLists.txt doesn't expose esp32-camera headers in its INCLUDE_DIRS. We could patch it. Don't — it's a user-shared component (other projects on the same machine share it), and any patch would silently break their builds. Doing it in our project's own main/CMakeLists.txt is the right scope.

Windows path with \E triggered CMake escape error

The first attempt to add the camera include path passed $ENV{ARDUINO_ESP32_PATH} directly. CMake interpreted the backslashes as escape sequences and crashed:

Invalid character escape '\E'.
when parsing string
    .;C:\Espressif\components\arduino-esp32/tools/sdk/...

Fix: file(TO_CMAKE_PATH "$ENV{ARDUINO_ESP32_PATH}" _arduino_path_norm) normalises backslashes to forward slashes.

Pre-existing worker-subprocess crash

While debugging the camera path I uncovered an unrelated bug: when backend/app/services/esp32_worker.py is run as a script (the way esp_lib_manager spawns it), its sibling-import fallback for esp32_i2c_slaves and esp32_spi_slaves registered the modules with importlib.util.module_from_spec but didn't put them in sys.modules before exec_module. Python's @dataclass decorator looks up cls.__module__ in sys.modules and crashes with AttributeError: 'NoneType' has no attribute '__dict__' when missing.

This was silently breaking every ESP32 simulation under certain CWD conditions (specifically the conditions Velxio's CI/Docker uses). Fix landed in the same commit as the camera work — see worker.py changes around lines 56-66 and 76-83.

Source references

The production-grade sources we lifted patterns from:

• espressif/esp32-camera — driver/esp_camera.c — the auto-detect loop (lines 234–263)
• espressif/esp32-camera — driver/sccb.c — the bare SCCB_Probe pattern at line 97
• espressif/esp32-camera — target/esp32/ll_cam.c — the I²S register sequence
• arduino-esp32 / Camera/CameraWebServer.ino — canonical pin map for AI-Thinker ESP32-CAM
• RuiSantosdotme/ESP32-CAM-Take-Photo-and-Display-Web-Server — simplest fb_get loop (the pattern our camera_init.ino follows)
• Random Nerd Tutorials — ESP32-CAM video streaming — for the high-level workflow

Reproducing the round-trip locally

# 1. Build the QEMU library with camera patches (one-time, ~25 min)
C:/msys64/usr/bin/bash.exe -lc "cd /c/v_qemu/build && bash ../build_libqemu-esp32-win.sh"
cp /c/v_qemu/build/libqemu-xtensa.dll backend/app/services/

# 2. Backend
cd backend && python -m uvicorn app.main:app --port 8003

# 3. Suite (synthetic JPEG fallback — works in CI)
VELXIO_BACKEND_URL=http://127.0.0.1:8003 \
  python -m pytest test/test-esp32-cam/tests -v

# 4. Same suite but using the actual webcam
VELXIO_BACKEND_URL=http://127.0.0.1:8003 \
  VELXIO_USE_WEBCAM=1 \
  python -m pytest test/test-esp32-cam/tests/test_frame_roundtrip_live.py -v

Follow-ups (in priority order)

1. Fix QEMU I²C ACK_ERR timing so SCCB_Probe correctly reports no-slave NACK. Unblocks Blocker A → esp_camera_init reaches the chip-id detect for OV2640 directly.
2. Implement continuous EOF (qemu_mod_timer strategy in esp32_i2s_cam.c). Unblocks Blocker B → framectrl task drains, fb_get returns the pushed JPEG.
3. Front-end webcam button + useWebcamFrames hook (out of scope until backend Phase 3 is solid; user said don't touch).
4. Stream multiple frames (not just one push) so the simulator can run face-detection / motion-detection sketches.

Each is a 1–3 day chunk on top of the foundation now in place.