A sketch whose only LVGL include was <lvgl.h> died on a wall of errors from
Adafruit LvGL Glue, a library it never included. LVGL's sources carry
`#include "lv_conf.h"` behind `#if !LV_CONF_SKIP`; the include scan is
textual and does not evaluate that guard, so the header was queued even for a
build defining LV_CONF_SKIP=1, resolved against the shared library dir, and
found the copy the glue library vendors for LVGL v7. That merged the whole
library, whose sources then failed against the LVGL v8 actually in use
('lv_fs_drv_t' has no member named 'user_data').
Same failure shape as the Hash.h -> stemihexapod case already documented
next to it, so the fix is the same: a project-supplied config header never
resolves against user libraries. A project shipping its own lv_conf.h is
unaffected (sketch files are found in the build's own source dir), and a
project that really wants the glue library still reaches it by including
its own header.
The serial monitor turns any 192.168.4.x it sees into an "Open IoT Gateway"
link, because that is the subnet QEMU's slirp leases to a connected board. The
ESP32's SoftAP default address is the same 192.168.4.1, so a captive-portal
sketch got the same link. It cannot work: an access point never receives a DHCP
lease, so it never reaches 'got_ip', so it never registers the bridge the proxy
looks up. The click landed on a 404 whose text then told the user to make sure
the sketch connected to WiFi, which is precisely what that sketch had chosen not
to do. The user is doing everything right and the product is lying twice.
Gate the link on the status every family already reports: QEMU through
wifi_status_parser, the in-browser JS engines, and the Pico W's cyw43
peripheral all emit 'got_ip'. Without it the address stays plain text.
Measured on a XIAO ESP32S3 Sense, one sketch per mode:
softAP 192.168.4.1 gateway links 1 -> 0
station 10.13.37.42 gateway links 1 -> 1 (unchanged)
The 404 body is rewritten to name softAP as the likely cause and point at a
network that exists in the simulator, instead of accusing the sketch. Covered
by a test that fails if the accusatory sentence comes back.
The QEMU libraries left the public `qemu-prebuilt` release: they are
GPL-2.0 derivatives of QEMU, and distributing one obliges us to hand the
recipient the corresponding source, so binary and source now live in one
place -- velxio.dev's gated endpoint, where the qemu-source-<commit>
archive sits beside them.
Six documents still pointed at the release. Two of them shipped runnable
commands that now 404: the deploy-to-container curl pair in the WiFi
notes, and the e2e README's claim about where CI gets its libraries
(that workflow had in fact moved to the gated endpoint some time ago --
only the README was stale).
Also corrects two things that were already wrong before today:
QEMU_RELEASE_URL is documented with the release as its default, but
Dockerfile.standalone defaults it to empty and reaches the binaries via
VELXIO_LICENSE_KEY; and the asset inventory omitted the macOS, Windows
and esp32s3_rev0_rom entries.
The symbol server assumed one library layout: the content-addressed cache,
where an entry dir name IS its content hash. A host without that cache --
Velxio Desktop's sidecar, and any self-host -- has its libraries where
arduino-cli put them, under plain folder names with the version inside
library.properties, and got a 404 for every request. The editor there falls
back to the engine's embedded catalog, so `client.` on a library the user
installed himself resolves to nothing.
Both layouts are now sources, tried best-first. The only real difference is
mutability: `lib upgrade` rewrites a sketchbook folder in place and keeps its
name, so a scan cannot be keyed by that name. Sketchbook entries are keyed by
a cheap stat-only stamp (file count, total size, newest mtime) in the same
`<norm_name>@<norm_version>-<12 hex>` shape, which makes the disk cache exact
rather than merely long-lived. Cache entries keep their existing keys, so no
deployed scan is invalidated.
Where the sketchbook is: an explicit override, then the sketchbook env vars,
then arduino-cli itself (`config get directories.user`, probed once and
remembered) -- guessing ~/Arduino is wrong on the platforms whose default is
~/Documents/Arduino. A sketchbook that resolves to the cache is dropped: the
pro overlay points arduino-cli at a symlink to the cache root, and scanning
it twice under two shapes buys nothing.
Scans for a sketchbook are written to `<sketchbook>/.velxio-symbols/`, beside
`libraries/` and never inside it, so nothing Velxio writes shows up in
`arduino-cli lib list`.
The type index now keys on a fingerprint of the source's entry keys instead
of the entry count, which is exact for both layouts (an in-place upgrade
changes no count). Deployed indexes rebuild once, ~1s from the cached scans.
A manifest-scoped compile only sees the libraries the project declares, so
it dies on the first undeclared transitive dependency — a header the user
had no reason to name. The scan-all retry exists to rescue exactly that,
and when it succeeds nothing is lost. When it fails as well, we returned
the SCOPED failure, which describes a library only our own scoping made
missing.
Today a hand-written SSD1306 sketch was reported as
Adafruit_GFX.h:12:10: fatal error: Adafruit_I2CDevice.h: No such file
while the retry had already resolved BusIO and reached the real bug, an
`#include <Adafruit_SSD1306>` with no .h. The user went looking for a
missing library that was installed all along.
The retry runs in the superset environment, so its error is the sketch's
own in both compilers. Prefer it whenever it carries text; the scoped
result still stands in when it does not.
Probed against staging the same way as the previous commit: lvgl.h and
esp_lvgl_port.h both died at "No such file or directory", so a visitor
could not paste a single screen of esp-dev-kits UI code. Declared only
when the source includes the header, so no other build pays for it.
A header that lives in a registry component and is not on
_MANAGED_COMPONENT_INCLUDES dies at "No such file or directory": the
project template ships no idf_component.yml, so nothing fetches it. That
list covered five display/camera components, which left the pieces the
official Espressif board demos are actually made of unavailable —
verified by compiling a one-include probe for each against staging:
led_strip.h FAIL (the C3-LCDkit knob_panel's WS2812 driver)
iot_button.h FAIL (its BOOT/knob-press handling)
iot_knob.h FAIL (its EC11 quadrature decoder)
esp_lcd_touch_gt911 (the P4 devkit's panel) and esp_codec_dev (every
audio demo on the VoCat and the P4) are added on the same grounds.
Anyone pasting code from esp-dev-kits or esp-bsp hit this on the first
line. Detection is unchanged — the dependency is declared only when the
source actually includes the header.
Tests: ManagedComponentDetectionTests in test_espidf_pure_mode.py, incl.
negative cases (a user's own my_led_strip.h must not match).
A sketch that includes BTstackLib on a CYW43 board fails with arduino-pico's
static_assert: "This library needs Bluetooth enabled. Use the 'Tools->IP/
Bluetooth Stack' menu in the IDE to enable it." There is no such menu here, so
it reads as a setting we forgot to expose. What is actually true is simpler:
the CYW43439's Bluetooth controller is not emulated — only its WiFi side is —
so there is nothing to switch on.
Appends that, leaving the compiler's own diagnostic intact above it because the
developer still needs the real text. Deliberately a separate helper from
humanize_cli_error, which REPLACES arduino-cli's own failures; this one only
adds context under a compiler message.
Worth knowing for whoever picks up the BT work: this is the honest-failure
milestone. Making a BT sketch compile at all means flipping this board's FQBN
to ipbtstack=ipv4btcble, which swaps liblwip.a for liblwip-bt.a and costs every
sketch on the board ~71 KB flash and ~18 KB RAM — and leaves it reaching
HCI_STATE_WORKING while advertising to nobody. The mechanism is mapped in
project/pimoroni-pico-plus-2w/STATUS.md.
MicroPython over the QEMU worker stopped loading: every project failed with
"the board stopped answering while writing _vlx_main.py", the board sitting
idle at a healthy `>>>`.
The uploader shipped in 664c3581 waits for the board's end-of-execution marker
before sending the next step — that wait is the whole point, it replaced a
fixed delay that pushed 64 bytes every 150 ms. The marker never arrived. Raw
REPL answers each statement with `OK` <EOT> traceback <EOT> `>` and not one
newline in it, and the UART buffer flushed only on newline, carriage return, a
period or 256 bytes. The answer sat in the buffer; the uploader timed out
waiting for something the backend was holding.
Nothing had needed that answer before: the old uploader pasted the whole
program and never listened. EOT is a message boundary in this protocol, so the
buffer treats it as one.
Found by forcing a MicroPython project down the QEMU path with ?esp32sim=qemu —
the residual risk 664c3581 flagged and could not exercise, because velxio.dev
routes every supported ESP32 kind to the in-browser engines. Self-hosters from
this repo have no engines and always take this path, so it broke for exactly
the people the #219 fix was written for.
`WiFi.begin()` panicked the guest: `Guru Meditation Error: Core 0 panic'ed
(LoadStorePIFAddrError)`, EXCVADDR 0x60033c00. Reported against a self-hosted
dev setup, and it did not reproduce on velxio.dev — which routes every
supported ESP32 kind to the in-browser engines. The engines are not in this
repo, so anyone self-hosting from it always runs QEMU, which is where the crash
lives. It was verified away on a path the reporter cannot take.
0x60033c00 is the WiFi MAC at its APB alias, and the fork only instantiates
that device when a NIC is attached (`qemu_find_nic_info(TYPE_ESP32_WIFI)` in
hw/xtensa/esp32.c). The worker attached one only when `wifi_enabled` was true,
so a regex over the user's source decided whether the machine had a radio at
all. Reproduced against the real worker: same sketch, same firmware, only the
flag differs — true connects on 192.168.4.x, false panics exactly as reported.
Two things were wrong, and the second is why the first was fatal.
The scan was too narrow. Arduino mode read only the entry sketch while pure-IDF
mode read every file, so `#include <WiFi.h>` in a project header was judged
WiFi-less. Verified against the live backend: main sketch -> has_wifi=true, the
same include one file away -> false. Both branches read the whole project now.
The frontend also trusted a false over its own all-files scan; the two signals
are OR'd, because a stale or narrower false is the direction that hurts.
But no scan should have been load-bearing. A real ESP32 has a radio whether or
not the sketch uses it, so the emulated one does too: the NIC is attached
unconditionally on machines that model a MAC. A misread sketch now fails to
connect instead of faulting. Only those machines — the S3 machine models no
radio (hw/xtensa/esp32s3.c never looks for the NIC), so handing it one would
leave an unconsumed netdev. `wifi_enabled` still gates the host forward, which
exposes the guest's server and does belong to the sketch.
Measured before making it unconditional, on a sketch that never touches WiFi:
boot time, guest-clock-vs-real-time and container CPU unchanged over 4 paired
runs. The AP's beacon timer runs on QEMU_CLOCK_REALTIME — the reason -icount is
off for Xtensa, b0fe5b5d — but idle beacons cost nothing measurable. One host,
Linux x86-64, one sim at a time; small effects would not show.
With the radio always present, the MicroPython `network` stub loses its reason
to exist on this path: it was there because `WLAN(STA_IF)` hangs forever with no
NIC behind it. Keeping the old condition would put the stub in front of a
working device whenever the scan misread the sketch — #262, from the other
direction. It now depends only on whether the backend models a radio.
And when this class of fault does happen, the dump says "memory" and nothing
else; a week went into #260 for that reason. LoadStorePIFAddrError is always an
unmapped peripheral, so the worker now says so in the serial monitor, naming
the WiFi MAC when the address is in its page.
test_esp32_wifi_args re-implemented the arg logic in its own `_simulate_args`,
so it would have passed unchanged through this entire bug — the same trap
1800f572 called out for the detector. The decision is now `wifi_nic_arg()` and
the test calls it.
Verified end to end against the QEMU worker: the multi-file WiFi project
compiles with has_wifi=true and connects, and a WiFi-less sketch still boots and
runs with the radio attached.
`test_unknown_boards_share_the_default_pair` required every unrecognised
FQBN to collapse onto the default (esp32, esp32) key. 82f25a03 stopped
that on purpose: `_idf_target` defaults ANY unknown board to 'esp32', so
an AVR / RP2040 / STM32 compile inherited the ESP32 identity and queued
behind unrelated ESP-IDF builds it shares no build dir with. The test
was never updated, so master has been red since.
Replaced with the contract the code actually implements, split so a
future change cannot quietly take the useful half away:
- non-ESP32 FQBNs get distinct identities and distinct locks (an AVR
build must not wait on an ESP-IDF one);
- a non-ESP32 board still serialises with ITSELF — dropping the shared
key must not drop the per-board lock;
- an unknown `esp32:` FQBN still shares the default pair, because that
one really does land in the shared build dir, and there
over-serialising is the safe direction.
The merged component copied every library into ONE interleaved tree and
then exported EVERY directory that held a file. A library's private
internals therefore sat on the global -I and hijacked #includes issued by
other libraries, by the arduino-esp32 core and by ESP-IDF itself. Each
incident was patched by adding a name to _SHADOW_STD_HEADERS — chasing
symptoms, since the denylist grows with every library that ships a
generically-named internal header (three landed yesterday alone:
FastLED's platforms/stub/Arduino.h, its fl/stl/asio/http/http_parser.h,
and earlier LovyanGFX's lgfx/internal/limits.h).
Adopt arduino-cli's model instead — the resolver every other Velxio board
family already delegates to. It shares one global -I list too, but adds
exactly ONE directory per library: <lib>/src for the recursive layout,
<lib> for the flat legacy one, never a subdirectory; utility/ stays
private to its owner. arduino-esp32 does the same as an IDF component.
- Each library is copied into its own subtree of the component, so two
libraries shipping the same relative path no longer collapse into one
silently-picked copy.
- INCLUDE_DIRS is one root per library, in resolution order. Internals
stay copied and reachable through file-relative includes.
- Legacy-layout utility/ dirs go in PRIV_INCLUDE_DIRS.
- Root-level header collisions (the only shadowing still possible) are
logged the way the Arduino IDE reports them, instead of being silent.
- The transitive scan now walks only the library just copied.
One component is deliberate: a component per library would swap this
failure mode for REQUIRES-order shadowing (IDF exports component includes
as plain -I, never -isystem) plus IDF component-name collisions, and buys
no caching ccache does not already provide.
Verified against the real FastLED+U8g2 project WITH the two denylist
entries removed: green, byte-identical firmware size. The 3-library
SSD1306+GFX+BusIO corpus is green too. VELXIO_PER_LIB_ROOTS=0 restores
the legacy layout; the mode is folded into the build-variant hash so the
two can never share a configured build dir.
A MicroPython library in Velxio IS a .py file in the project, copied onto
the board's filesystem before main.py runs. That mechanism worked, but
getting the file meant hunting GitHub by hand — the gap issue #214 is
about. Now the Library Manager fills it, in both halves:
Backend: /api/micropython-libs proxies the official micropython-lib
package index (the same one `mip` uses on real hardware). /search filters
the index; /fetch resolves a package plus its declared deps into .py
files. The proxy is what makes this safe to expose: only the official
index is reachable (the name must exist in index.json, files come from its
content-addressed store), source only (.py, never .mpy bytecode), path-
safe, and capped at 24 files / 256 KB per install. In-process TTL cache
over the index.
Frontend: the same Library Manager modal, mode-aware. On a MicroPython
board the search hits the index and "Add to project" writes the package's
files into the board's workspace (upsert — re-adding is the upgrade path),
visible in the explorer and travelling with the project. Before typing it
shows featured picks (ssd1306, umqtt.*, urequests) and the modules already
frozen into the firmware (neopixel, dht, onewire, ds18x20, machine,
network) labeled "built into the firmware", so searching for one answers
"just import it" instead of an empty list.
The file-materialization prelude now creates package directories: umqtt
ships as umqtt/simple.py, and MicroPython's open() does not mkdir, so the
prelude died with ENOENT on any packaged module.
Also ships a gallery example ("ESP32 + MicroPython: use an external
library") that documents the file-is-the-library mechanism with the
ssd1306 driver on the wired OLED.
arduino-cli reports errors as plain text or, with --format json, as a JSON
object on stderr. We forwarded that blob verbatim, so on a fresh install
with no library index yet (and no way through the firewall/proxy to
downloads.arduino.cc) the in-app agent told the user "the library server
returns a JSON error" ten times in a row.
humanize_cli_error() maps the known shapes — index download failed,
library zip download failed, library not found — to a single sentence
that says what to check, and keeps the raw text under raw_error. Applied
to search_libraries (rc != 0 and non-JSON stdout) and install_library.
Tests use the real arduino-cli 1.0.4 output captured offline.
`_prepare_persistent_project_dir` keys the persistent build dir on
(idf_target, variant), but the lock was keyed on the FQBN. Several FQBNs
resolve to one variant — arduino-esp32's boards.txt gives both `esp32` and
`esp32cam` build.variant=esp32 — so those two were free to build in the SAME
directory at the same time.
Measured against staging, compiling both concurrently:
esp32 285504 bytes sha f65ca909… contains its own marker
esp32cam 285504 bytes sha f65ca909… contains the OTHER sketch's marker
Byte-identical firmware, and the esp32cam caller got the esp32 sketch's
binary. In the product that is two gallery tabs open at once: the first tab
prints the second example's serial log and does nothing itself. It made
esp32-pwm-led-rgb look broken (dark LED, every SPICE node 0 V) when the
example is fine — it was running someone else's program.
Key the lock on the same identity as the directory. Boards with genuinely
separate build dirs still compile in parallel up to the semaphore cap; an
unresolvable FQBN falls back to the default pair, which is the safe
direction (serialising more only costs throughput).
Also in this commit: battery-voltage-monitor ships the 9 V battery its own
description promises. The divider's top end was wired to nothing, so A0 read
0 V and the sketch printed "Battery: 0.0 V / WARNING: Battery low!" forever.
The dead-branch pruner marked any assumed-live branch as 'taken', so the
#else of an UNKNOWN condition was skipped. GxEPD2 guards its GFX base
class with '#if ENABLE_GxEPD2_GFX (default 0) ... #else #include
<Adafruit_GFX.h>' — the pruned #else silently dropped the Adafruit_GFX
dependency and every GxEPD2 e-paper example failed with 'fatal error:
Adafruit_GFX.h: No such file'. Found by the nightly gallery compile smoke.
'taken' is now set only by PROVABLY-true conditions (all identifiers in
the known-true set), keeping the 2026-08-03 ESPAsyncWebServer behaviour:
provably-dead branches still prune, provably-true branches still kill
their #else.
Three-part fix for the library-contamination class of ESP32 compile
failures (a user's project kept pulling unrelated libraries because 79%
of projects compile with an empty manifest, forcing scan-all resolution
over ~1000 shared libraries):
1. Name-affinity ranking in _find_library_for_header: the provider whose
name matches the header stem wins (exact normalized match > esp32
arch declaration > contains-stem, alphabetical only as tie-break).
Previously first-alphabetical won, so any stray library shipping a
generic header (Servo.h has 6 providers) could hijack the build.
2. Manifest auto-migration: scan-all compiles now report the libraries
the build really used (manifest_incomplete +
manifest_suggested_libraries on the compile result); the editor folds
single-candidate suggestions into the board's declared manifest after
a green build, so the NEXT compile runs scoped. One green build = one
migrated project; empty-manifest scan-all behaviour is untouched.
3. Install-declares: the Wokwi import library modal records successful
installs in the active board's manifest, and .vlx export/import now
round-trips board.libraries.
Tests: test_espidf_library_ranking.py (9) + library-manifest.test.ts (19).
Reported 2026-08-03: every ESP32 build in a user's project failed with
errors from libraries he never installed, emitted from user_libs_all --
a folder his file list does not even show, in projects he had recreated
from scratch.
Root cause: _detect_external_includes was a bare regex with no
preprocessor state. ESPAsyncWebServer guards <Hash.h> behind
'#if defined(ESP8266) || defined(TARGET_RP2040)', so the compiler never
sees it on ESP32 -- but the scanner reported it anyway. The resolver
then searched the ~1000 shared-cache libraries, matched stemi-hexapod's
src/Hash.h (a hexapod-robot library), merged it WHOLE, and compiled all
of its sources; each failed on its own unrelated includes.
The scanner now tracks #if/#ifdef/#ifndef/#elif/#else/#endif nesting and
strips comments. A branch is skipped ONLY when every identifier in its
condition is a platform macro known to be undefined on ESP32; unknown
macros keep the branch LIVE, because hiding a genuinely needed header
would be a much worse failure than the old over-reporting. Verified
against the real ESPAsyncWebServer shape plus nesting/elif/else/comment
cases (9 new tests, 15 green in the espidf suites).
Complements the platform-header guard from the previous commit, which
stays as defence in depth.
Reported by a user whose every ESP32 build failed with errors from
libraries he never installed (SoftwareSerial.h, ArduinoBearSSL.h,
Adafruit_GFX.h, AsyncElegantOTA.h) coming out of user_libs_all, in a
project whose file list doesn't even show that folder.
Traced end to end: ESPAsyncWebServer includes <Hash.h> (an ESP8266-core
header, guarded for that platform, so arduino-esp32's tree has no such
file). The global scan then resolved it to stemi-hexapod's src/Hash.h --
a hexapod-robot library -- merged the WHOLE library into user_libs_all,
and its Serial.cpp / Hash.cpp / Server.cpp / BluetoothLowEnergy.cpp /
ExpansionDriver.cpp were compiled, each failing on its own unrelated
includes. Log line: 'Merging "stemihexapod" into user_libs_all for
<Hash.h>'.
Hash.h and friends now join the never-resolve-against-user-libs set.
They cannot be discovered by the core-tree scan (the core genuinely
lacks them) yet their names are generic enough that some installed
library owns a file by that name.
Installing ArduinoJson made the project stop compiling, with an error that
never mentioned ArduinoJson:
CMake Error at user_libs/user_libs_all/CMakeLists.txt:15
(target_compile_definitions):
target_compile_definitions may only set INTERFACE properties on
INTERFACE targets
-- Configuring incomplete, errors occurred!
ArduinoJson ships headers only. With no sources, idf_component_register()
builds the merged user-libs component as an INTERFACE library, and PRIVATE
scope on an INTERFACE target is a hard CMake error - so the configure step
died before a single file was compiled. The user saw only a failing build
and reported it as "ArduinoJson can't be found even after it has been
installed" (#124).
Those definitions map alloca/memcpy_P/memcmp_P for library C sources that
use them undeclared; with no sources there is nothing to define, so they are
emitted only when the component actually compiles something. A library that
carries its own .cpp is unchanged - which is why LiquidCrystal_I2C (#257)
worked and hid this.
Direct sketch includes of arch-excluded libs merge; transitive pulls
stay skipped; 'all' counts as wildcard. Replaces the old blanket-skip
assertion that correctly blocked the deploy gate.
Arduino trata igual #include "Lib.h" que #include <Lib.h> para librerias, y
los ejemplos de los propios fabricantes usan la forma con comillas: los de
M5Stack para el Cardputer abren con #include "M5Cardputer.h". Al escanear solo
la forma con angulos, ese sketch no llegaba siquiera al resolutor de librerias y
moria con 'fatal error: M5Cardputer.h: No such file', mientras el mismo sketch
con angulos compilaba sin problema. Cualquiera que pegue un ejemplo oficial se
daba de bruces con esto.
Se pasan ademas los nombres de los ficheros del propio sketch, para que un
#include "Common.h" local siga siendo una cabecera del proyecto y no se
confunda con una libreria.
test/backend/unit/test_espidf_compiler.py cubre ambas formas, el espaciado, las
cabeceras internas de IDF y la cabecera propia del proyecto.
Las dos ramas habian divergido: master llevaba el modo lenguaje ESP-IDF puro
(#139) y v3.2 la ruta de compilacion IDF v5.5 para toda la familia ESP32 mas
los arreglos de venv/toolchain. Ambas tocaban espidf_compiler.py.
Los dos lados son ejes ORTOGONALES y se conservan enteros:
- use_idf5 / arduino_mode (v3.2): que arbol IDF usa el build (5.5 vs 4.4) y
si cabe Arduino-como-componente.
- pure_idf (master): el modo LENGUAJE que elige el usuario; sus ficheros son
las fuentes del componente main con su propio app_main().
Resolucion:
- _build_env acepta los tres. Un build IDF puro fuerza arduino_mode a falso:
la plantilla CMake mete el componente arduino-esp32 en cuanto existe
ARDUINO_ESP32_PATH, asi que dejarlo puesto compilaba el core de Arduino en
un build que no tiene sketch. Lo cazaron los tests de master.
- VELXIO_PURE_SKETCH solo con pure_idf, nunca con arduino_mode a falso a
secas: un target sin core arduino-esp32 sigue entregando un SKETCH al
traductor legacy y no debe tomar la rama del glob puro.
- La identidad del build-dir suma los dos tokens (|idf:N|ard:N y |lang:pure):
ningun par de esas combinaciones puede compartir un build/ configurado.
- La cadena de escritura de fuentes queda pure_idf -> arduino_mode -> legacy.
- sdkconfig: render de v3.2 (con target/use_idf5) mas el filtrado de simbolos
CONFIG_ARDUINO* de master cuando el build es puro.
test/backend/unit/test_espidf_compiler.py: los 7 tests que ya estaban rotos en
v3.2 (AttributeError: idf5_path, fixture sin actualizar desde que se anadio la
seleccion de IDF) vuelven a pasar.
Verificado: backend 293 pasan / 0 fallan (v3.2 traia 7 rotos); frontend 2268
pasan / 0 fallan en los dos shards.
Adds a third entry to the board language selector next to Arduino C++
and MicroPython: ESP-IDF. In this mode the user writes a plain ESP-IDF
project — app_main() entry point, FreeRTOS + driver APIs — and the
backend compiles it through the same ESP-IDF toolchain it already uses
for ESP32 Arduino sketches, just without the arduino-esp32 component.
Backend:
- CompileRequest.language ('espidf') threaded through the sync + async
compile paths and folded into the dedup job key (language='arduino'
and omitted hash identically so old clients keep dedupping).
- espidf_compiler: pure_idf flag. User files are written into main/
as-is (no Arduino.h wrap, no velxio_compat.h, Arduino library
resolution skipped), ARDUINO_ESP32_PATH is dropped from the build env
and VELXIO_PURE_SKETCH raised so the template CMake compiles the
user's own sources via a glob branch. Pure builds get their own
persistent build-dir variant through the eff_hash fold.
- QEMU WiFi compat for IDF-style code: esp_wifi.h/esp_wifi_init
detection sets has_wifi, and literal #define SSID/PASS plus
wifi_config_t designated initializers are normalized to the QEMU AP.
- CONFIG_ARDUINO_* lines are stripped from sdkconfig.defaults in pure
mode (the symbols don't exist without the arduino component).
Frontend:
- LanguageMode gains 'espidf'; BOARD_SUPPORTS_ESPIDF covers the ESP32
family (Xtensa, S3, C3). Toolbar shows the option only for those.
- Switching modes seeds a main.c blink skeleton (app_main + gpio
driver), mirroring the MicroPython main.py flow.
- compileCode sends language='espidf'; run/stop paths are unchanged
(the QEMU worker consumes the same merged flash image).
- New gallery example: esp32-idf-blink (LED + resistor on GPIO 2).
Tests: unit coverage for the build-env switch, IDF wifi normalization,
job-key variance, file-group seeding and the new example; verified
end-to-end in a container from the prod image (pure build produces a
bootable flash image; Arduino-mode build unchanged, same variant hash).
test/backend/unit/test_picow_inbound_gateway.py and
test/backend/integration/test_picow_net_bridge.py imported
app.services.picow_net, which moved to the private overlay in the open-core
split. They now live under pro/backend/tests/ and run against the overlay.
Removing them here unbreaks the OSS pytest collection (and the deploy gate).
ESP32 web-server examples are reachable from the browser via
/api/gateway/<client_id>/ (QEMU slirp hostfwd). The Pico W server lives
in the browser-side lwIP, so there was no inbound path: visiting the
chip's IP did nothing.
Add the mirror of tcp_nat.py: tcp_inbound.TcpInbound originates a TCP
connection INTO the chip over the WebSocket bridge (SYN -> SYN+ACK ->
ACK -> request -> response -> FIN), so the backend can fetch a page the
sketch serves on 10.13.37.42:80 and hand it back to the browser.
- bridge.py routes chip TCP segments addressed to a gateway-opened
connection to TcpInbound (before the chip-initiated NAT, which would
RST them); exposes http_into_chip() + ensure_chip_mac() (primes the
chip's gateway ARP).
- iot_gateway.py: same /api/gateway/<client_id>/ route now falls through
to the Pico W bridge when there's no ESP32 instance, builds a raw
HTTP/1.1 request, and parses the chip's response. Same plan gate, same
URL shape — the browser sees no difference between ESP32 and Pico W.
Validated end to end (real RP2040 emulator serving an HTTP page ->
gateway returns it) plus 6 unit tests for the TCP state machine,
response parsing and ARP priming.
P2.0 first cut took the first-alphabetical lib providing a header and then
checked manifest membership. When several installed libs ship the same header
(e.g. DHT118266, DHT_sensor_library, servodht11 all have DHT.h), the stray
first-match got rejected and the header was dropped even though the declared
lib provides it.
_find_manifest_library_for_header: when a manifest is supplied, pick the first
DECLARED library that provides the header. This both selects the right lib and
excludes undeclared ones. No manifest = legacy first-match.
Test strengthened with a stray same-header lib that sorts first.
When a compile supplies a 'libraries' manifest, _resolve_library_components
merges a USER-installed library only if it's declared in that set. A sketch
therefore never picks up an unrelated library from the shared dir (another
user's install, or a same-named clash) — the manifest is the resolution scope.
- _resolve_library_components(allowed_libraries): gate user-lib merges on
manifest membership; match by folder name OR library.properties name=,
normalised (display name vs on-disk folder differ by separators/case).
Core/bundled libs are never gated. None = legacy scan-all (unchanged).
- Threaded through compile() -> _compile_in_dir.
- compile.py: CompileRequest.libraries; folded into the async dedup _job_key
so a different manifest doesn't dedup to a job built with another.
Opt-in: omitting 'libraries' preserves current behaviour exactly.
Regression: test/backend/unit/test_espidf_core_first.py::TestManifestScope
A user library that ships a core-named header (e.g. WiFiEspAT/src/WiFi.h)
could shadow the arduino-esp32 core during ESP-IDF library resolution.
WiFiEspAT shadowing WiFi.h pulled EspAtDrv.cpp into the build, whose
const char OK[]/STATUS[] collide with ESP-IDF's enum STATUS in
rom/ets_sys.h, breaking every ESP32 sketch that #include <WiFi.h>.
_resolve_library_components now:
- skips a header entirely when the arduino-esp32 core provides it
(computed set from cores/ + libraries/, cached), so a user lib can
never shadow WiFi.h/Wire.h/SPI.h/WebServer.h/...
- skips a resolved user lib whose library.properties architectures=
excludes esp32/*.
Regression: test/backend/unit/test_espidf_core_first.py
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
test_busy_wait_100us and test_busy_wait_1us measure busy_wait_us()
elapsed time against absolute thresholds (500µs / 100µs). Under
contended CI/deploy-gate machines these can blow through the budget
even when the busy-wait implementation is correct, blocking deploys
that have nothing to do with DHT22 timing.
Same pattern already applied to test_response_timing_analysis in
this file — skipped via @unittest.skipIf(os.environ['CI']=='true').
User report: on the solar-tracker project (5218f9e3) only one servo
moved and the log showed `ch=0 duty=X% gpio=12` (wrong — servoPan was
attached to GPIO 13) and `ch=1 ... gpio=-1` (servoTilt's channel
never resolved).
Root cause traced through the GPIO Matrix dump: the firmware does
exactly what the Arduino-ESP32 Servo library says — `ledcAttachPin(
13, 0)` writes signal 71 (LEDC_HS_SIG_OUT0) into `gpio_out_sel[13]`,
and `ledcAttachPin(12, 1)` writes signal 72 (LEDC_HS_SIG_OUT1) into
`gpio_out_sel[12]`. Per the ESP32 Technical Reference Manual section
4.11, Table 4-3:
71 .. 78 → LEDC HS channels 0..7
79 .. 86 → LEDC LS channels 0..7
The legacy worker code at esp32_worker.py:426 used the off-by-one
range `72 <= signal <= 87` with `ledc_ch = signal - 72`. The mistake
masked itself for single-servo projects because the 0x5000 duty
callback's channel index was internally consistent with the bogus
math, so the duty STILL reached the correctly-routed pin (just
labelled wrong). The new SignalRouter unit tests caught the
discrepancy the moment two servos drove distinct channels: signal
71 (HS_CH0, gpio 13) was REJECTED by the off-by-one filter and
signal 72 (HS_CH1, gpio 12) was misclassified as channel 0.
When I ported the legacy range into `esp32_signals.SIG_LEDC_HS_CH0_OUT_IDX`
the bug came along for the ride. Fix both modules:
* `backend/app/services/esp32_signals.py`: HS 71-78, LS 79-86.
* `frontend/src/simulation/esp32-signals.ts`: mirror.
* tests updated; 20 backend + 23 frontend pass.
After deploy the user's two servos will resolve to their declared
pins:
ch=0 duty=X% gpio=13 (servoPan, was wrongly emitting gpio=12)
ch=1 duty=X% gpio=12 (servoTilt, was wrongly emitting gpio=-1)
This is also why the multi-servo blink "patch" in commit 77bf897
appeared to help: with both pins ALIASED to the same channel via
the off-by-one, the broadcast fallback was the only thing producing
ANY movement on the second servo at all.
Replaces the per-peripheral ad-hoc `_ledc_gpio_map` cache with a
proper signal-routing abstraction that mirrors the ESP32 SoC's
IO_MUX + GPIO Matrix exactly. Same idea as real silicon: signal
sources (LEDC channels, RMT, MCPWM, ...) → 40-entry routing table
→ GPIO pins.
Motivation (from user bug report in
velxio.dev/project/5218f9e3-136d-43b3-bba1-6cebde21e1a4): two
ESP32 servos on a solar-tracker visibly oscillated between two
positions instead of moving smoothly when the user changed LDR
sliders. Commit 77bf897 patched it (per-channel gpio memo +
broadcast guard) but the user requested a proper hardware-fidel
architecture, not patches.
Backend:
* `app/services/signal_router.py` — SignalRouter class. Forward
index (gpio → signal_id) + reverse index (signal_id → set of
gpios). `replace_snapshot()` returns the diff for the polling-
fallback path; future C plugin hook becomes a push without
touching this code.
* `app/services/esp32_signals.py` — Signal id constants from
ESP32 TRM (LEDC HS 72-79, LS 80-87) + `ledc_signal_for_channel()`
helper.
* `app/services/esp32_worker.py` — `_ledc_gpio_map` is gone;
`_refresh_ledc_gpio_map` replaced by `_refresh_signal_routing`
which emits `gpio_routing {gpio, signal_id}` events on diff.
The 0x5000 LEDC callback and the LEDC poll thread now emit
`ledc_duty {channel, duty_pct}` (canonical, no gpio) alongside
the legacy `ledc_update {channel, duty, gpio}` for back-compat
during rollout.
Frontend:
* `simulation/SignalRouter.ts` — 1-to-1 TS mirror of the Python
class. Same forward + reverse index; same `pinsForSignal` /
`updateRouting` / `clearRouting` API.
* `simulation/esp32-signals.ts` — Signal id constants, mirror
of the Python module.
* `simulation/Esp32Bridge.ts` — new `onLedcDuty`, `onGpioRouting`,
`onGpioRoutingClear` callbacks; handlers for the new event types.
* `store/useSimulatorStore.ts` — `makeLedcDutyHandler` looks up
pins via `router.pinsForSignal(ledcSignalForChannel(channel))`
and dispatches per pin. `makeGpioRoutingHandler` /
`makeGpioRoutingClearHandler` keep the mirror in sync. Per-board
`signalRouterMap` parallels `pinManagerMap` in lifecycle.
`makeLedcUpdateHandler` (and its memo workaround from 77bf897)
stays wired for back-compat during rollout; removed in a
follow-up commit once prod is verified stable on the new path.
Tests:
* `test/backend/unit/test_signal_router.py` (20 tests) covers
update/clear semantics, idempotency, multi-pin routing,
snapshot diff, channel↔signal-id helpers, and the multi-servo
regression scenario.
* `frontend/src/__tests__/SignalRouter.test.ts` (17 tests) is the
mirror — same scenarios on the TS side.
* `frontend/src/__tests__/esp32-multi-servo-gpio-matrix.test.ts`
(6 tests) drives the end-to-end SignalRouter handler pipeline,
asserts that two servos on GPIO 13/12 via LEDC channels 0/1
move independently (no mirroring), that re-routing carries
cleanly, and — critically — that `PinManager.broadcastPwm` is
never called.
Totals: +700 LOC, 1876 frontend tests pass (was 1853), 278 backend
unit tests pass (was 259).
Docs: ESP32_EMULATION.md §9.2 rewritten with the new architecture
diagram + a runbook for adding future peripherals through the
SignalRouter.
The C plugin hook in qemu-lcgamboa that would push gpio_out_sel
writes synchronously (eliminating the polling race window entirely)
is the next step — kept as a follow-up because the polling-fallback
path here already resolves the routing before each duty event
fires, so the bug is fixed end-to-end. The plugin work removes the
race condition fundamentally.
test_response_timing_analysis measures the actual µs duration of the
DHT22 preamble LOW pulse and asserts it stays under 1000µs (real
hardware target ~80µs, busy-wait tolerance ~500µs). On a deploy box
under load (concurrent docker build + zstd compression + container
runtime) GIL contention inflates the observed timing far past the
threshold — the deploy gate just hit 3242µs and aborted.
Mirror the same @skipIf(CI=='true') gate the sibling
test_response_data_matches_payload already has (line 390-393).
Locally / when debugging the DHT22 path the test still runs in full.
deploy.sh's vitest + pytest output was polluted with three benign
but loud warnings that buried real signal:
1. AVRSimulator.start() unconditionally read `window.__spiceDebug`.
In node-side vitest runs `window` is undefined → ReferenceError
→ console.warn('[spice] debug dump failed', e). Logged once per
AVR test. Guarded with `typeof window !== 'undefined'`; in
production the browser path is unchanged.
2. pinPositionCalculator.calculatePinPosition() warned every time
document.getElementById returned null. In node-side tests there
is no real DOM and every wire-related test triggers the warning
for every component. Skip the console.warn when
import.meta.env.MODE === 'test' (vitest sets MODE=test); the
function still returns null and production retains the
actionable warning for unmounted components.
3. test_esp32_wifi_args.py::test_start_instance_accepts_wifi_params
mocked asyncio.create_task with no side_effect, so the coroutine
from self._boot(...) leaked and triggered a "coroutine never
awaited" RuntimeWarning. Mock now closes the coroutine.
After fixes:
frontend tests: 0 spice/pinPositionCalculator stderr lines
backend tests: 259 passed, 15 skipped, 1 warning (starlette
third-party python_multipart deprecation —
not ours, fixed when starlette updates).
User report: clicked Pi 3 board → nothing visible happens. Three
defects, all on the same path:
1. The kernel cmdline carried over from the original pre-OSS-split
code: `quiet init=/bin/sh`. Result: kernel boot messages
suppressed, then dropped straight to bare /bin/sh with no PS1 so
the user sees an empty serial. Removed both. The kernel cmdline
is now just `console=ttyAMA0 root=/dev/mmcblk0p2 rootwait rw
dwc_otg.lpm_enable=0`, which lets systemd start a real
serial-getty@ttyAMA0.service.
2. Pi OS Trixie armhf since Bookworm ships without a default user
(no more pi/raspberry). With cmdline #1 fixed, the user would
land at a login prompt and be stuck. Fix: pre-bake a systemd
drop-in at /etc/systemd/system/serial-getty@ttyAMA0.service.d/
autologin.conf that uses `agetty --autologin root` so the serial
console drops to a root shell on first prompt. The browser
canvas IS the authentication boundary; the SD image is mounted
RO via a qcow2 overlay so per-session edits don't persist.
Edit happens in velxio-prod/scripts/configure-pi3-autologin.sh
(to follow in a separate commit).
3. Architectural: the original cache-hit probe was size-only.
Today's SD image rebake produced a file with identical byte count
but different SHA256 — the cache served stale content for every
request even after a manifest bump. Fix: write a sidecar
`<file>.sha256` after every successful materialise and trust it
on subsequent probes. Manifest SHA bumps invalidate the cache
regardless of size. Two regression tests guard this:
- test_provider_sidecar_invalidates_on_sha_mismatch
- test_provider_missing_sidecar_treats_file_as_invalid
Manifest bumped to version "2026-04-21+autologin" for the SD image
(kernel + DTB unchanged, still 2026-04-21).
Pi 3 simulation had been broken since at least April 2026 (51
fail-events / 24h per docs/PI3_EMULATION_BROKEN.md). Two distinct
defects compounded:
1. qemu_manager.py hard-coded paths for kernel8.img, a device-tree
blob (under a DOS 8.3 short name!), and a 5.4 GiB Raspberry Pi OS
SD image — none of which shipped in the repo or were pulled at
image build.
2. qemu-system-arm + qemu-utils were missing from the Docker image
entirely, so even with the boot files in place QEMU couldn't
launch. Add both to Dockerfile.standalone (~200 MB).
The architecture fix is a new `app.services.boot_images` module:
* Manifest-driven (boot_images/manifest.json, versioned in repo,
declares SHA256 + size for each file, supports an optional
`compressed.{encoding,sha256,size_bytes}` block for assets shipped
as .zst).
* `BootImageProvider` materialises files lazily, atomically (temp +
rename), verifies SHA256 pre- AND post-decompression, caches under
/var/cache/velxio/boot-images, serialises concurrent get() calls
per image set via asyncio.Lock.
* `AssetDownloader` Protocol with two impls:
- `LicenseGatedDownloader` — same flow ESP32 / RISC-V QEMU libs
use (VELXIO_BINARY_BASE_URL + VELXIO_LICENSE_KEY).
- `LocalDirectoryDownloader` — for tests + in-prod use where the
licence-module storage is already on the same filesystem (saves
the loopback HTTP roundtrip on a 1.4 GiB blob).
* `build_downloader_from_env()` picks one — local-dir wins if both
sets of env vars are present, so the prod box short-circuits to
direct disk reads automatically.
* Lifespan hook in qemu_manager.py pre-warms the cache on container
boot so first-time user requests don't pay the 30-60 s download
+ decompress latency.
Adding a future board kind (Pi 4 / Pi 5) is now: upload assets via
upload-binary.sh, append an entry to manifest.json, register a
lifespan pre-warm in the new board's service module. Zero edits to
provider.py / downloader.py.
Manifest entries for raspberry-pi-3:
kernel8.img 9 695 883 bytes (uncompressed)
bcm2710-rpi-3-b.dtb 34 687 bytes (uncompressed)
raspios-trixie-armhf.img 5 729 419 264 bytes raw
/ 1 488 002 803 bytes .zst on wire (zstd -19)
source: 2026-04-21 build from raspberrypi.com
Tests: 21 new unit tests covering manifest parsing, integrity
helpers, both downloaders, and the provider's idempotent /
concurrent / integrity / decompression / warmup paths. In-process
FakeDownloader keeps the suite under 1 s and httpx-free.
Docs: new docs/BOOT_IMAGES.md describes the architecture, on-disk
layout, named-volume operation, and the procedure for adding a new
image set.
Three coordinated fixes that together close the "ESP-IDF compile takes
5-7 min every time" gap and prevent the failure mode where a user clicking
compile multiple times spawns six ninja processes that peel each other
apart on a modest VPS.
What was wrong
- /compile/start generated a fresh uuid4 every call, so 6 clicks = 6
independent builds racing each other. Saw load average 30 on the prod
VPS during a real BMP280 attempt today.
- No concurrency limit anywhere; asyncio.create_task() fired without
gating.
- ccache was wired in last week (PR #149) but reported 18,350 cacheable
calls and **0 hits** because the build dir was a fresh
tempfile.TemporaryDirectory(prefix='espidf_') per compile. The random
/tmp/espidf_<random>/ path baked into -I and -fmacro-prefix-map flags
→ different command line every compile → ccache hash miss every time.
What this PR does
1. Job deduplication (`backend/app/api/routes/compile.py`)
- New `_job_key(files, board_fqbn)` returns SHA-256 of normalised file
names + contents + board. Order-independent.
- New `JOB_BY_KEY: dict[str, str]` indexes hash → job_id.
- `compile_start` checks JOB_BY_KEY before spawning a new task; if a
job for this exact content is already pending or running, returns
the existing job_id (logs `[compile] dedup hit — reusing job <id>`).
- `_purge_expired_jobs` evicts both COMPILE_JOBS and JOB_BY_KEY,
keeping the index consistent. Edge case where two jobs share a key
(old finished, new running) is handled — only evict the key entry
if it still points at the purged job.
2. Concurrency control (`backend/app/api/routes/compile.py`)
- `_COMPILE_SEMAPHORE = asyncio.Semaphore(2)` global cap on
simultaneous compiles.
- `_target_lock(board_fqbn)` returns a per-target asyncio.Lock so
concurrent compiles to the SAME board (sharing the persistent build
dir) serialise. Different boards still run in parallel up to the
semaphore cap.
- `_compile_job` acquires sema → per-target lock → flips state to
`running` → calls `_run_compile`. Pending state now accurately
reflects "queued waiting for resources".
3. Persistent build dir (`backend/app/services/espidf_compiler.py`)
- New `_prepare_persistent_project_dir(idf_target)` materialises
`/var/lib/velxio-build/<target>/project/` from the template on
first use; on subsequent compiles it wipes only `main/` and
`user_libs/` (the per-compile parts) and leaves `build/` alone so
ninja's incremental cache + ccache .o files survive.
- Toolchain version sentinel (`.idf_version`) wipes the whole target
dir if the ESP-IDF or arduino-esp32 version changes — cached
objects from the old toolchain are no longer ABI-compatible.
- `compile()` is now a thin dispatcher: persistent path or fallback
to the legacy `tempfile.TemporaryDirectory()` flow. The actual
build logic was extracted into `_compile_in_dir()` so both paths
share one implementation, no duplication.
- Escape hatch: `VELXIO_PERSISTENT_BUILD_DIR=0` env var falls back
to the tempfile path without rebuilding the image. Critical for
production safety.
4. ccache normalisation (`Dockerfile.standalone`)
- + `ENV CCACHE_BASEDIR=/var/lib/velxio-build` makes ccache canonicalise
absolute paths under that prefix when computing the cache key.
Robustens hits against any future subdir rearrangement.
5. Docker compose (`docker-compose.yml`)
- + named volume `velxio-build:/var/lib/velxio-build` so the persistent
build dir survives `docker compose up -d --build`.
- + env `VELXIO_PERSISTENT_BUILD_DIR=1` (default ON; users disable
without rebuilding).
Expected impact
- Cold first compile per container per target: unchanged (~5-7 min).
- Same sketch re-compiled: ~2-5 s (everything cached).
- Different sketch, same target: ~5-30 s (only user code + new lib steps
rebuild; ESP-IDF base hits cache).
- Different sketch with new libraries: ~30-90 s (new lib component
compiles; rest hits cache).
- Concurrent clicks on same example: 1 build, others poll the same
job_id. No more six-ninja meltdown.
Tests
- `test/backend/unit/test_compile_dedup.py` covers `_job_key` stability +
variance and `_purge_expired_jobs` consistency (including the
"two jobs share a key" edge case).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the loop on the four ESP32 fixes that landed earlier in this
series. Each previously-broken or noisy example now has a regression
test that compiles the sketch through the production ESP-IDF compiler
and runs it in QEMU via esp32_worker.py — the same code path the
WebSocket /ws/{client_id} endpoint drives in production. Testing the
worker directly skips the WS transport but exercises the same compile
→ flash → boot → serial cascade.
Coverage:
- TestEsp32SerialCleanliness — DHT22, Servo+Pot, Joystick, Dual ADC
Asserts the user's Serial.print substring shows up AND no
`I (xxx) gpio:|wifi:|phy:` info-level ESP-IDF logs leak through.
This validates the sdkconfig CONFIG_LOG_DEFAULT_LEVEL_WARN change
from commit b373c97.
- TestEsp32CompileSuccess — BLE Advertise, LEDC RGB
BLE Advertise validates the sdkconfig switch to Bluedroid (was
NimBLE-only, which broke arduino-esp32's BLEDevice.h).
LEDC RGB validates velxio_compat.h's ledcAttach() shim from
commit f6f6f43; the sketch uses the arduino-esp32 3.x one-shot API
on a 2.0.17 toolchain.
- TestEsp32WiFiSketches — WiFi Connect, WiFi WebServer
Regression coverage to make sure the sdkconfig changes didn't break
WiFi association. Connect must reach an "IP Address:" line; Server
must report "Server started".
- frontend/src/__tests__/component-metadata-bmp280.test.ts
Sanity check that the BMP280 entry from commit 1f3f2e0 survives
metadata regeneration.
All ESP-IDF/QEMU tests use unittest.skipUnless on
_toolchain_available() so they no-op cleanly on dev boxes without
libqemu-xtensa, and only do real work in the Docker CI image.
Sketches are inlined verbatim from the public Velxio examples.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1. Backend test (test_arduino_cli_attinycore.py): the entrypoint script
was renamed deploy/ → docker/ in commit b736aea but this test still
pointed at the old path. Update the read_text() call + docstring.
2. Frontend CI (frontend-tests.yml): the cache key
`frontend-${{ hashFiles('frontend/package-lock.json') }}` was tied to
a file that has since been gitignored (commit eb9a3ec). hashFiles()
on a missing file returns the same empty hash forever, so every CI
run was restoring the same stale node_modules — including the
symlinks to `file:../third-party/wokwi-elements` that existed before
the npm migration in commit 531c337. On revalidation, npm tried to
run wokwi-elements' `prepare` script (`husky install && npm run
build`), which failed with "husky: not found".
Drop the cache step entirely; lock files aren't committed so cache
keys can't be made meaningful without overcomplication. Adds ~30s
per CI run, but actually correct. Also pass --no-audit --no-fund
to npm install for cleaner logs.
- Added tcp_nat.py to handle TCP NAT, implementing the three-way handshake, data flow, and connection state management.
- Introduced udp_nat.py for UDP NAT, managing chip-initiated datagrams and maintaining flow state.
- Created integration tests in test_picow_net_bridge.py to validate the functionality of the TCP and UDP NAT implementations, including ARP, DHCP, ICMP, and DNS interactions.
- Implemented handshake tests to validate initial bus state and register responses.
- Created end-to-end tests for Pico W LED blinking using MicroPython firmware.
- Added SDPCM framing tests to ensure proper encoding and decoding of control frames.
- Developed IOCTL tests to verify command responses and state changes in the emulator.
- Established a full lifecycle test for WiFi operations, including scanning, connecting, and packet handling.
- Introduced TypeScript configuration for test files to ensure compatibility and strict type checking.
Follows the same pattern as test_esp32_spice_analog.mjs and
test_esp32_spice_ntc_bridge.mjs: compile an ESP32 sketch on-the-fly via
/api/compile/, boot QEMU through the backend WebSocket, and sweep lux
levels while verifying analogRead() returns what ngspice solved.
The netlist uses the exact photodiode cards emitted by
frontend/.../componentToSpice.ts (D_<id> + I_<id>_ph + DPHOTO model), so
any drift in the frontend SPICE mapper surfaces here.
Validated against the live Docker container — lux=0/1000/2500 produce
raw=4095/2854/992, matching the expected 4095/2854/993 within ±1 LSB
and monotonically decreasing with brightness as expected.
The package-lock.json churn is a pre-existing drift: eecircuit-engine
was in package.json but missing from the lock — npm install re-added it.
Companion workflow change (registering the test in backend-e2e-tests.yml)
lives in a separate commit that requires a PAT with workflow scope to push.
Three new end-to-end tests that combine ESP32 QEMU emulation (via backend
WebSocket) with ngspice-WASM analog circuit solving:
1. test_esp32_spice_analog.mjs — voltage divider sweep
- Compiles a sketch that reads analogRead(34)
- Solves two voltage dividers with ngspice (R1/R2=10k/10k then 10k/30k)
- Injects solved V(mid) into ESP32's ADC via esp32_adc_set
- Verifies Serial output matches within +-50 counts (12-bit ADC)
- Confirms circuit change is detected (different ADC values)
2. test_esp32_spice_ntc_bridge.mjs — Wheatstone bridge temperature sweep
- NTC thermistor in a bridge (0C / 25C / 50C)
- ngspice solves the bridge for each temperature
- ESP32 reads both legs (ADC34+ADC35), computes R_ntc and T via beta model
- Verifies temperature within +-5C tolerance across sweep
3. test_esp32_spice_smoke.mjs — ngspice-only smoke test (no backend needed)
Also adds eecircuit-engine to test/backend/e2e/package.json.
Prerequisites: backend on localhost:8001 with esp32 core installed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Implemented a comprehensive test script (test_micropython_pico.mjs) that performs the following:
- Part 1: Checks backend compilation of a simple Arduino sketch for the rp2040:rp2040:rpipico board.
- Part 2: Downloads MicroPython v1.20.0 UF2 firmware and loads it into a rp2040js simulator.
- Part 3: Simulates the Pico, verifies REPL output, and checks execution of injected Python code.
- Includes detailed logging and error handling for each step of the process.
David Montero