Three coordinated fixes that together close the "ESP-IDF compile takes
5-7 min every time" gap and prevent the failure mode where a user clicking
compile multiple times spawns six ninja processes that peel each other
apart on a modest VPS.
What was wrong
- /compile/start generated a fresh uuid4 every call, so 6 clicks = 6
independent builds racing each other. Saw load average 30 on the prod
VPS during a real BMP280 attempt today.
- No concurrency limit anywhere; asyncio.create_task() fired without
gating.
- ccache was wired in last week (PR #149) but reported 18,350 cacheable
calls and **0 hits** because the build dir was a fresh
tempfile.TemporaryDirectory(prefix='espidf_') per compile. The random
/tmp/espidf_<random>/ path baked into -I and -fmacro-prefix-map flags
→ different command line every compile → ccache hash miss every time.
What this PR does
1. Job deduplication (`backend/app/api/routes/compile.py`)
- New `_job_key(files, board_fqbn)` returns SHA-256 of normalised file
names + contents + board. Order-independent.
- New `JOB_BY_KEY: dict[str, str]` indexes hash → job_id.
- `compile_start` checks JOB_BY_KEY before spawning a new task; if a
job for this exact content is already pending or running, returns
the existing job_id (logs `[compile] dedup hit — reusing job <id>`).
- `_purge_expired_jobs` evicts both COMPILE_JOBS and JOB_BY_KEY,
keeping the index consistent. Edge case where two jobs share a key
(old finished, new running) is handled — only evict the key entry
if it still points at the purged job.
2. Concurrency control (`backend/app/api/routes/compile.py`)
- `_COMPILE_SEMAPHORE = asyncio.Semaphore(2)` global cap on
simultaneous compiles.
- `_target_lock(board_fqbn)` returns a per-target asyncio.Lock so
concurrent compiles to the SAME board (sharing the persistent build
dir) serialise. Different boards still run in parallel up to the
semaphore cap.
- `_compile_job` acquires sema → per-target lock → flips state to
`running` → calls `_run_compile`. Pending state now accurately
reflects "queued waiting for resources".
3. Persistent build dir (`backend/app/services/espidf_compiler.py`)
- New `_prepare_persistent_project_dir(idf_target)` materialises
`/var/lib/velxio-build/<target>/project/` from the template on
first use; on subsequent compiles it wipes only `main/` and
`user_libs/` (the per-compile parts) and leaves `build/` alone so
ninja's incremental cache + ccache .o files survive.
- Toolchain version sentinel (`.idf_version`) wipes the whole target
dir if the ESP-IDF or arduino-esp32 version changes — cached
objects from the old toolchain are no longer ABI-compatible.
- `compile()` is now a thin dispatcher: persistent path or fallback
to the legacy `tempfile.TemporaryDirectory()` flow. The actual
build logic was extracted into `_compile_in_dir()` so both paths
share one implementation, no duplication.
- Escape hatch: `VELXIO_PERSISTENT_BUILD_DIR=0` env var falls back
to the tempfile path without rebuilding the image. Critical for
production safety.
4. ccache normalisation (`Dockerfile.standalone`)
- + `ENV CCACHE_BASEDIR=/var/lib/velxio-build` makes ccache canonicalise
absolute paths under that prefix when computing the cache key.
Robustens hits against any future subdir rearrangement.
5. Docker compose (`docker-compose.yml`)
- + named volume `velxio-build:/var/lib/velxio-build` so the persistent
build dir survives `docker compose up -d --build`.
- + env `VELXIO_PERSISTENT_BUILD_DIR=1` (default ON; users disable
without rebuilding).
Expected impact
- Cold first compile per container per target: unchanged (~5-7 min).
- Same sketch re-compiled: ~2-5 s (everything cached).
- Different sketch, same target: ~5-30 s (only user code + new lib steps
rebuild; ESP-IDF base hits cache).
- Different sketch with new libraries: ~30-90 s (new lib component
compiles; rest hits cache).
- Concurrent clicks on same example: 1 build, others poll the same
job_id. No more six-ninja meltdown.
Tests
- `test/backend/unit/test_compile_dedup.py` covers `_job_key` stability +
variance and `_purge_expired_jobs` consistency (including the
"two jobs share a key" edge case).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the loop on the four ESP32 fixes that landed earlier in this
series. Each previously-broken or noisy example now has a regression
test that compiles the sketch through the production ESP-IDF compiler
and runs it in QEMU via esp32_worker.py — the same code path the
WebSocket /ws/{client_id} endpoint drives in production. Testing the
worker directly skips the WS transport but exercises the same compile
→ flash → boot → serial cascade.
Coverage:
- TestEsp32SerialCleanliness — DHT22, Servo+Pot, Joystick, Dual ADC
Asserts the user's Serial.print substring shows up AND no
`I (xxx) gpio:|wifi:|phy:` info-level ESP-IDF logs leak through.
This validates the sdkconfig CONFIG_LOG_DEFAULT_LEVEL_WARN change
from commit b373c97.
- TestEsp32CompileSuccess — BLE Advertise, LEDC RGB
BLE Advertise validates the sdkconfig switch to Bluedroid (was
NimBLE-only, which broke arduino-esp32's BLEDevice.h).
LEDC RGB validates velxio_compat.h's ledcAttach() shim from
commit f6f6f43; the sketch uses the arduino-esp32 3.x one-shot API
on a 2.0.17 toolchain.
- TestEsp32WiFiSketches — WiFi Connect, WiFi WebServer
Regression coverage to make sure the sdkconfig changes didn't break
WiFi association. Connect must reach an "IP Address:" line; Server
must report "Server started".
- frontend/src/__tests__/component-metadata-bmp280.test.ts
Sanity check that the BMP280 entry from commit 1f3f2e0 survives
metadata regeneration.
All ESP-IDF/QEMU tests use unittest.skipUnless on
_toolchain_available() so they no-op cleanly on dev boxes without
libqemu-xtensa, and only do real work in the Docker CI image.
Sketches are inlined verbatim from the public Velxio examples.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1. Backend test (test_arduino_cli_attinycore.py): the entrypoint script
was renamed deploy/ → docker/ in commit b736aea but this test still
pointed at the old path. Update the read_text() call + docstring.
2. Frontend CI (frontend-tests.yml): the cache key
`frontend-${{ hashFiles('frontend/package-lock.json') }}` was tied to
a file that has since been gitignored (commit eb9a3ec). hashFiles()
on a missing file returns the same empty hash forever, so every CI
run was restoring the same stale node_modules — including the
symlinks to `file:../third-party/wokwi-elements` that existed before
the npm migration in commit 531c337. On revalidation, npm tried to
run wokwi-elements' `prepare` script (`husky install && npm run
build`), which failed with "husky: not found".
Drop the cache step entirely; lock files aren't committed so cache
keys can't be made meaningful without overcomplication. Adds ~30s
per CI run, but actually correct. Also pass --no-audit --no-fund
to npm install for cleaner logs.
- Added tcp_nat.py to handle TCP NAT, implementing the three-way handshake, data flow, and connection state management.
- Introduced udp_nat.py for UDP NAT, managing chip-initiated datagrams and maintaining flow state.
- Created integration tests in test_picow_net_bridge.py to validate the functionality of the TCP and UDP NAT implementations, including ARP, DHCP, ICMP, and DNS interactions.
- Implemented handshake tests to validate initial bus state and register responses.
- Created end-to-end tests for Pico W LED blinking using MicroPython firmware.
- Added SDPCM framing tests to ensure proper encoding and decoding of control frames.
- Developed IOCTL tests to verify command responses and state changes in the emulator.
- Established a full lifecycle test for WiFi operations, including scanning, connecting, and packet handling.
- Introduced TypeScript configuration for test files to ensure compatibility and strict type checking.
Follows the same pattern as test_esp32_spice_analog.mjs and
test_esp32_spice_ntc_bridge.mjs: compile an ESP32 sketch on-the-fly via
/api/compile/, boot QEMU through the backend WebSocket, and sweep lux
levels while verifying analogRead() returns what ngspice solved.
The netlist uses the exact photodiode cards emitted by
frontend/.../componentToSpice.ts (D_<id> + I_<id>_ph + DPHOTO model), so
any drift in the frontend SPICE mapper surfaces here.
Validated against the live Docker container — lux=0/1000/2500 produce
raw=4095/2854/992, matching the expected 4095/2854/993 within ±1 LSB
and monotonically decreasing with brightness as expected.
The package-lock.json churn is a pre-existing drift: eecircuit-engine
was in package.json but missing from the lock — npm install re-added it.
Companion workflow change (registering the test in backend-e2e-tests.yml)
lives in a separate commit that requires a PAT with workflow scope to push.
Three new end-to-end tests that combine ESP32 QEMU emulation (via backend
WebSocket) with ngspice-WASM analog circuit solving:
1. test_esp32_spice_analog.mjs — voltage divider sweep
- Compiles a sketch that reads analogRead(34)
- Solves two voltage dividers with ngspice (R1/R2=10k/10k then 10k/30k)
- Injects solved V(mid) into ESP32's ADC via esp32_adc_set
- Verifies Serial output matches within +-50 counts (12-bit ADC)
- Confirms circuit change is detected (different ADC values)
2. test_esp32_spice_ntc_bridge.mjs — Wheatstone bridge temperature sweep
- NTC thermistor in a bridge (0C / 25C / 50C)
- ngspice solves the bridge for each temperature
- ESP32 reads both legs (ADC34+ADC35), computes R_ntc and T via beta model
- Verifies temperature within +-5C tolerance across sweep
3. test_esp32_spice_smoke.mjs — ngspice-only smoke test (no backend needed)
Also adds eecircuit-engine to test/backend/e2e/package.json.
Prerequisites: backend on localhost:8001 with esp32 core installed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Implemented a comprehensive test script (test_micropython_pico.mjs) that performs the following:
- Part 1: Checks backend compilation of a simple Arduino sketch for the rp2040:rp2040:rpipico board.
- Part 2: Downloads MicroPython v1.20.0 UF2 firmware and loads it into a rp2040js simulator.
- Part 3: Simulates the Pico, verifies REPL output, and checks execution of injected Python code.
- Includes detailed logging and error handling for each step of the process.
refactor: Update wiring examples for E32 OLED integration; correct pin mappings for VCC, GND, DATA, and CLK
test: Improve unit tests for ESPIDFCompiler; add scenarios for library resolution and CMake patching
chore: Mark subproject commits as dirty for wokwi-libs
- Implement tests for BMP280, DS1307, DS3231, I2CWriteSink, and MPU6050 slaves in test_i2c_slaves.py.
- Create test suite for Velxio MCP server tools in test_mcp_tools.py, covering Wokwi utilities and circuit management functions.
- Add tests for parsing WiFi and BLE serial output in test_wifi_status_parser.py, ensuring correct status events are captured.
davidmonterocrespo24