The symbol server assumed one library layout: the content-addressed cache,
where an entry dir name IS its content hash. A host without that cache --
Velxio Desktop's sidecar, and any self-host -- has its libraries where
arduino-cli put them, under plain folder names with the version inside
library.properties, and got a 404 for every request. The editor there falls
back to the engine's embedded catalog, so `client.` on a library the user
installed himself resolves to nothing.
Both layouts are now sources, tried best-first. The only real difference is
mutability: `lib upgrade` rewrites a sketchbook folder in place and keeps its
name, so a scan cannot be keyed by that name. Sketchbook entries are keyed by
a cheap stat-only stamp (file count, total size, newest mtime) in the same
`<norm_name>@<norm_version>-<12 hex>` shape, which makes the disk cache exact
rather than merely long-lived. Cache entries keep their existing keys, so no
deployed scan is invalidated.
Where the sketchbook is: an explicit override, then the sketchbook env vars,
then arduino-cli itself (`config get directories.user`, probed once and
remembered) -- guessing ~/Arduino is wrong on the platforms whose default is
~/Documents/Arduino. A sketchbook that resolves to the cache is dropped: the
pro overlay points arduino-cli at a symlink to the cache root, and scanning
it twice under two shapes buys nothing.
Scans for a sketchbook are written to `<sketchbook>/.velxio-symbols/`, beside
`libraries/` and never inside it, so nothing Velxio writes shows up in
`arduino-cli lib list`.
The type index now keys on a fingerprint of the source's entry keys instead
of the entry count, which is exact for both layouts (an in-place upgrade
changes no count). Deployed indexes rebuild once, ~1s from the cached scans.
A manifest-scoped compile only sees the libraries the project declares, so
it dies on the first undeclared transitive dependency — a header the user
had no reason to name. The scan-all retry exists to rescue exactly that,
and when it succeeds nothing is lost. When it fails as well, we returned
the SCOPED failure, which describes a library only our own scoping made
missing.
Today a hand-written SSD1306 sketch was reported as
Adafruit_GFX.h:12:10: fatal error: Adafruit_I2CDevice.h: No such file
while the retry had already resolved BusIO and reached the real bug, an
`#include <Adafruit_SSD1306>` with no .h. The user went looking for a
missing library that was installed all along.
The retry runs in the superset environment, so its error is the sketch's
own in both compilers. Prefer it whenever it carries text; the scoped
result still stands in when it does not.
Probed against staging the same way as the previous commit: lvgl.h and
esp_lvgl_port.h both died at "No such file or directory", so a visitor
could not paste a single screen of esp-dev-kits UI code. Declared only
when the source includes the header, so no other build pays for it.
A header that lives in a registry component and is not on
_MANAGED_COMPONENT_INCLUDES dies at "No such file or directory": the
project template ships no idf_component.yml, so nothing fetches it. That
list covered five display/camera components, which left the pieces the
official Espressif board demos are actually made of unavailable —
verified by compiling a one-include probe for each against staging:
led_strip.h FAIL (the C3-LCDkit knob_panel's WS2812 driver)
iot_button.h FAIL (its BOOT/knob-press handling)
iot_knob.h FAIL (its EC11 quadrature decoder)
esp_lcd_touch_gt911 (the P4 devkit's panel) and esp_codec_dev (every
audio demo on the VoCat and the P4) are added on the same grounds.
Anyone pasting code from esp-dev-kits or esp-bsp hit this on the first
line. Detection is unchanged — the dependency is declared only when the
source actually includes the header.
Tests: ManagedComponentDetectionTests in test_espidf_pure_mode.py, incl.
negative cases (a user's own my_led_strip.h must not match).
A sketch that includes BTstackLib on a CYW43 board fails with arduino-pico's
static_assert: "This library needs Bluetooth enabled. Use the 'Tools->IP/
Bluetooth Stack' menu in the IDE to enable it." There is no such menu here, so
it reads as a setting we forgot to expose. What is actually true is simpler:
the CYW43439's Bluetooth controller is not emulated — only its WiFi side is —
so there is nothing to switch on.
Appends that, leaving the compiler's own diagnostic intact above it because the
developer still needs the real text. Deliberately a separate helper from
humanize_cli_error, which REPLACES arduino-cli's own failures; this one only
adds context under a compiler message.
Worth knowing for whoever picks up the BT work: this is the honest-failure
milestone. Making a BT sketch compile at all means flipping this board's FQBN
to ipbtstack=ipv4btcble, which swaps liblwip.a for liblwip-bt.a and costs every
sketch on the board ~71 KB flash and ~18 KB RAM — and leaves it reaching
HCI_STATE_WORKING while advertising to nobody. The mechanism is mapped in
project/pimoroni-pico-plus-2w/STATUS.md.
MicroPython over the QEMU worker stopped loading: every project failed with
"the board stopped answering while writing _vlx_main.py", the board sitting
idle at a healthy `>>>`.
The uploader shipped in 664c3581 waits for the board's end-of-execution marker
before sending the next step — that wait is the whole point, it replaced a
fixed delay that pushed 64 bytes every 150 ms. The marker never arrived. Raw
REPL answers each statement with `OK` <EOT> traceback <EOT> `>` and not one
newline in it, and the UART buffer flushed only on newline, carriage return, a
period or 256 bytes. The answer sat in the buffer; the uploader timed out
waiting for something the backend was holding.
Nothing had needed that answer before: the old uploader pasted the whole
program and never listened. EOT is a message boundary in this protocol, so the
buffer treats it as one.
Found by forcing a MicroPython project down the QEMU path with ?esp32sim=qemu —
the residual risk 664c3581 flagged and could not exercise, because velxio.dev
routes every supported ESP32 kind to the in-browser engines. Self-hosters from
this repo have no engines and always take this path, so it broke for exactly
the people the #219 fix was written for.
`WiFi.begin()` panicked the guest: `Guru Meditation Error: Core 0 panic'ed
(LoadStorePIFAddrError)`, EXCVADDR 0x60033c00. Reported against a self-hosted
dev setup, and it did not reproduce on velxio.dev — which routes every
supported ESP32 kind to the in-browser engines. The engines are not in this
repo, so anyone self-hosting from it always runs QEMU, which is where the crash
lives. It was verified away on a path the reporter cannot take.
0x60033c00 is the WiFi MAC at its APB alias, and the fork only instantiates
that device when a NIC is attached (`qemu_find_nic_info(TYPE_ESP32_WIFI)` in
hw/xtensa/esp32.c). The worker attached one only when `wifi_enabled` was true,
so a regex over the user's source decided whether the machine had a radio at
all. Reproduced against the real worker: same sketch, same firmware, only the
flag differs — true connects on 192.168.4.x, false panics exactly as reported.
Two things were wrong, and the second is why the first was fatal.
The scan was too narrow. Arduino mode read only the entry sketch while pure-IDF
mode read every file, so `#include <WiFi.h>` in a project header was judged
WiFi-less. Verified against the live backend: main sketch -> has_wifi=true, the
same include one file away -> false. Both branches read the whole project now.
The frontend also trusted a false over its own all-files scan; the two signals
are OR'd, because a stale or narrower false is the direction that hurts.
But no scan should have been load-bearing. A real ESP32 has a radio whether or
not the sketch uses it, so the emulated one does too: the NIC is attached
unconditionally on machines that model a MAC. A misread sketch now fails to
connect instead of faulting. Only those machines — the S3 machine models no
radio (hw/xtensa/esp32s3.c never looks for the NIC), so handing it one would
leave an unconsumed netdev. `wifi_enabled` still gates the host forward, which
exposes the guest's server and does belong to the sketch.
Measured before making it unconditional, on a sketch that never touches WiFi:
boot time, guest-clock-vs-real-time and container CPU unchanged over 4 paired
runs. The AP's beacon timer runs on QEMU_CLOCK_REALTIME — the reason -icount is
off for Xtensa, b0fe5b5d — but idle beacons cost nothing measurable. One host,
Linux x86-64, one sim at a time; small effects would not show.
With the radio always present, the MicroPython `network` stub loses its reason
to exist on this path: it was there because `WLAN(STA_IF)` hangs forever with no
NIC behind it. Keeping the old condition would put the stub in front of a
working device whenever the scan misread the sketch — #262, from the other
direction. It now depends only on whether the backend models a radio.
And when this class of fault does happen, the dump says "memory" and nothing
else; a week went into #260 for that reason. LoadStorePIFAddrError is always an
unmapped peripheral, so the worker now says so in the serial monitor, naming
the WiFi MAC when the address is in its page.
test_esp32_wifi_args re-implemented the arg logic in its own `_simulate_args`,
so it would have passed unchanged through this entire bug — the same trap
1800f572 called out for the detector. The decision is now `wifi_nic_arg()` and
the test calls it.
Verified end to end against the QEMU worker: the multi-file WiFi project
compiles with has_wifi=true and connects, and a WiFi-less sketch still boots and
runs with the radio attached.
`test_unknown_boards_share_the_default_pair` required every unrecognised
FQBN to collapse onto the default (esp32, esp32) key. 82f25a03 stopped
that on purpose: `_idf_target` defaults ANY unknown board to 'esp32', so
an AVR / RP2040 / STM32 compile inherited the ESP32 identity and queued
behind unrelated ESP-IDF builds it shares no build dir with. The test
was never updated, so master has been red since.
Replaced with the contract the code actually implements, split so a
future change cannot quietly take the useful half away:
- non-ESP32 FQBNs get distinct identities and distinct locks (an AVR
build must not wait on an ESP-IDF one);
- a non-ESP32 board still serialises with ITSELF — dropping the shared
key must not drop the per-board lock;
- an unknown `esp32:` FQBN still shares the default pair, because that
one really does land in the shared build dir, and there
over-serialising is the safe direction.
The merged component copied every library into ONE interleaved tree and
then exported EVERY directory that held a file. A library's private
internals therefore sat on the global -I and hijacked #includes issued by
other libraries, by the arduino-esp32 core and by ESP-IDF itself. Each
incident was patched by adding a name to _SHADOW_STD_HEADERS — chasing
symptoms, since the denylist grows with every library that ships a
generically-named internal header (three landed yesterday alone:
FastLED's platforms/stub/Arduino.h, its fl/stl/asio/http/http_parser.h,
and earlier LovyanGFX's lgfx/internal/limits.h).
Adopt arduino-cli's model instead — the resolver every other Velxio board
family already delegates to. It shares one global -I list too, but adds
exactly ONE directory per library: <lib>/src for the recursive layout,
<lib> for the flat legacy one, never a subdirectory; utility/ stays
private to its owner. arduino-esp32 does the same as an IDF component.
- Each library is copied into its own subtree of the component, so two
libraries shipping the same relative path no longer collapse into one
silently-picked copy.
- INCLUDE_DIRS is one root per library, in resolution order. Internals
stay copied and reachable through file-relative includes.
- Legacy-layout utility/ dirs go in PRIV_INCLUDE_DIRS.
- Root-level header collisions (the only shadowing still possible) are
logged the way the Arduino IDE reports them, instead of being silent.
- The transitive scan now walks only the library just copied.
One component is deliberate: a component per library would swap this
failure mode for REQUIRES-order shadowing (IDF exports component includes
as plain -I, never -isystem) plus IDF component-name collisions, and buys
no caching ccache does not already provide.
Verified against the real FastLED+U8g2 project WITH the two denylist
entries removed: green, byte-identical firmware size. The 3-library
SSD1306+GFX+BusIO corpus is green too. VELXIO_PER_LIB_ROOTS=0 restores
the legacy layout; the mode is folded into the build-variant hash so the
two can never share a configured build dir.
A MicroPython library in Velxio IS a .py file in the project, copied onto
the board's filesystem before main.py runs. That mechanism worked, but
getting the file meant hunting GitHub by hand — the gap issue #214 is
about. Now the Library Manager fills it, in both halves:
Backend: /api/micropython-libs proxies the official micropython-lib
package index (the same one `mip` uses on real hardware). /search filters
the index; /fetch resolves a package plus its declared deps into .py
files. The proxy is what makes this safe to expose: only the official
index is reachable (the name must exist in index.json, files come from its
content-addressed store), source only (.py, never .mpy bytecode), path-
safe, and capped at 24 files / 256 KB per install. In-process TTL cache
over the index.
Frontend: the same Library Manager modal, mode-aware. On a MicroPython
board the search hits the index and "Add to project" writes the package's
files into the board's workspace (upsert — re-adding is the upgrade path),
visible in the explorer and travelling with the project. Before typing it
shows featured picks (ssd1306, umqtt.*, urequests) and the modules already
frozen into the firmware (neopixel, dht, onewire, ds18x20, machine,
network) labeled "built into the firmware", so searching for one answers
"just import it" instead of an empty list.
The file-materialization prelude now creates package directories: umqtt
ships as umqtt/simple.py, and MicroPython's open() does not mkdir, so the
prelude died with ENOENT on any packaged module.
Also ships a gallery example ("ESP32 + MicroPython: use an external
library") that documents the file-is-the-library mechanism with the
ssd1306 driver on the wired OLED.
arduino-cli reports errors as plain text or, with --format json, as a JSON
object on stderr. We forwarded that blob verbatim, so on a fresh install
with no library index yet (and no way through the firewall/proxy to
downloads.arduino.cc) the in-app agent told the user "the library server
returns a JSON error" ten times in a row.
humanize_cli_error() maps the known shapes — index download failed,
library zip download failed, library not found — to a single sentence
that says what to check, and keeps the raw text under raw_error. Applied
to search_libraries (rc != 0 and non-JSON stdout) and install_library.
Tests use the real arduino-cli 1.0.4 output captured offline.
`_prepare_persistent_project_dir` keys the persistent build dir on
(idf_target, variant), but the lock was keyed on the FQBN. Several FQBNs
resolve to one variant — arduino-esp32's boards.txt gives both `esp32` and
`esp32cam` build.variant=esp32 — so those two were free to build in the SAME
directory at the same time.
Measured against staging, compiling both concurrently:
esp32 285504 bytes sha f65ca909… contains its own marker
esp32cam 285504 bytes sha f65ca909… contains the OTHER sketch's marker
Byte-identical firmware, and the esp32cam caller got the esp32 sketch's
binary. In the product that is two gallery tabs open at once: the first tab
prints the second example's serial log and does nothing itself. It made
esp32-pwm-led-rgb look broken (dark LED, every SPICE node 0 V) when the
example is fine — it was running someone else's program.
Key the lock on the same identity as the directory. Boards with genuinely
separate build dirs still compile in parallel up to the semaphore cap; an
unresolvable FQBN falls back to the default pair, which is the safe
direction (serialising more only costs throughput).
Also in this commit: battery-voltage-monitor ships the 9 V battery its own
description promises. The divider's top end was wired to nothing, so A0 read
0 V and the sketch printed "Battery: 0.0 V / WARNING: Battery low!" forever.
The dead-branch pruner marked any assumed-live branch as 'taken', so the
#else of an UNKNOWN condition was skipped. GxEPD2 guards its GFX base
class with '#if ENABLE_GxEPD2_GFX (default 0) ... #else #include
<Adafruit_GFX.h>' — the pruned #else silently dropped the Adafruit_GFX
dependency and every GxEPD2 e-paper example failed with 'fatal error:
Adafruit_GFX.h: No such file'. Found by the nightly gallery compile smoke.
'taken' is now set only by PROVABLY-true conditions (all identifiers in
the known-true set), keeping the 2026-08-03 ESPAsyncWebServer behaviour:
provably-dead branches still prune, provably-true branches still kill
their #else.
Three-part fix for the library-contamination class of ESP32 compile
failures (a user's project kept pulling unrelated libraries because 79%
of projects compile with an empty manifest, forcing scan-all resolution
over ~1000 shared libraries):
1. Name-affinity ranking in _find_library_for_header: the provider whose
name matches the header stem wins (exact normalized match > esp32
arch declaration > contains-stem, alphabetical only as tie-break).
Previously first-alphabetical won, so any stray library shipping a
generic header (Servo.h has 6 providers) could hijack the build.
2. Manifest auto-migration: scan-all compiles now report the libraries
the build really used (manifest_incomplete +
manifest_suggested_libraries on the compile result); the editor folds
single-candidate suggestions into the board's declared manifest after
a green build, so the NEXT compile runs scoped. One green build = one
migrated project; empty-manifest scan-all behaviour is untouched.
3. Install-declares: the Wokwi import library modal records successful
installs in the active board's manifest, and .vlx export/import now
round-trips board.libraries.
Tests: test_espidf_library_ranking.py (9) + library-manifest.test.ts (19).
Reported 2026-08-03: every ESP32 build in a user's project failed with
errors from libraries he never installed, emitted from user_libs_all --
a folder his file list does not even show, in projects he had recreated
from scratch.
Root cause: _detect_external_includes was a bare regex with no
preprocessor state. ESPAsyncWebServer guards <Hash.h> behind
'#if defined(ESP8266) || defined(TARGET_RP2040)', so the compiler never
sees it on ESP32 -- but the scanner reported it anyway. The resolver
then searched the ~1000 shared-cache libraries, matched stemi-hexapod's
src/Hash.h (a hexapod-robot library), merged it WHOLE, and compiled all
of its sources; each failed on its own unrelated includes.
The scanner now tracks #if/#ifdef/#ifndef/#elif/#else/#endif nesting and
strips comments. A branch is skipped ONLY when every identifier in its
condition is a platform macro known to be undefined on ESP32; unknown
macros keep the branch LIVE, because hiding a genuinely needed header
would be a much worse failure than the old over-reporting. Verified
against the real ESPAsyncWebServer shape plus nesting/elif/else/comment
cases (9 new tests, 15 green in the espidf suites).
Complements the platform-header guard from the previous commit, which
stays as defence in depth.
Reported by a user whose every ESP32 build failed with errors from
libraries he never installed (SoftwareSerial.h, ArduinoBearSSL.h,
Adafruit_GFX.h, AsyncElegantOTA.h) coming out of user_libs_all, in a
project whose file list doesn't even show that folder.
Traced end to end: ESPAsyncWebServer includes <Hash.h> (an ESP8266-core
header, guarded for that platform, so arduino-esp32's tree has no such
file). The global scan then resolved it to stemi-hexapod's src/Hash.h --
a hexapod-robot library -- merged the WHOLE library into user_libs_all,
and its Serial.cpp / Hash.cpp / Server.cpp / BluetoothLowEnergy.cpp /
ExpansionDriver.cpp were compiled, each failing on its own unrelated
includes. Log line: 'Merging "stemihexapod" into user_libs_all for
<Hash.h>'.
Hash.h and friends now join the never-resolve-against-user-libs set.
They cannot be discovered by the core-tree scan (the core genuinely
lacks them) yet their names are generic enough that some installed
library owns a file by that name.
Installing ArduinoJson made the project stop compiling, with an error that
never mentioned ArduinoJson:
CMake Error at user_libs/user_libs_all/CMakeLists.txt:15
(target_compile_definitions):
target_compile_definitions may only set INTERFACE properties on
INTERFACE targets
-- Configuring incomplete, errors occurred!
ArduinoJson ships headers only. With no sources, idf_component_register()
builds the merged user-libs component as an INTERFACE library, and PRIVATE
scope on an INTERFACE target is a hard CMake error - so the configure step
died before a single file was compiled. The user saw only a failing build
and reported it as "ArduinoJson can't be found even after it has been
installed" (#124).
Those definitions map alloca/memcpy_P/memcmp_P for library C sources that
use them undeclared; with no sources there is nothing to define, so they are
emitted only when the component actually compiles something. A library that
carries its own .cpp is unchanged - which is why LiquidCrystal_I2C (#257)
worked and hid this.
Direct sketch includes of arch-excluded libs merge; transitive pulls
stay skipped; 'all' counts as wildcard. Replaces the old blanket-skip
assertion that correctly blocked the deploy gate.
Arduino trata igual #include "Lib.h" que #include <Lib.h> para librerias, y
los ejemplos de los propios fabricantes usan la forma con comillas: los de
M5Stack para el Cardputer abren con #include "M5Cardputer.h". Al escanear solo
la forma con angulos, ese sketch no llegaba siquiera al resolutor de librerias y
moria con 'fatal error: M5Cardputer.h: No such file', mientras el mismo sketch
con angulos compilaba sin problema. Cualquiera que pegue un ejemplo oficial se
daba de bruces con esto.
Se pasan ademas los nombres de los ficheros del propio sketch, para que un
#include "Common.h" local siga siendo una cabecera del proyecto y no se
confunda con una libreria.
test/backend/unit/test_espidf_compiler.py cubre ambas formas, el espaciado, las
cabeceras internas de IDF y la cabecera propia del proyecto.
Las dos ramas habian divergido: master llevaba el modo lenguaje ESP-IDF puro
(#139) y v3.2 la ruta de compilacion IDF v5.5 para toda la familia ESP32 mas
los arreglos de venv/toolchain. Ambas tocaban espidf_compiler.py.
Los dos lados son ejes ORTOGONALES y se conservan enteros:
- use_idf5 / arduino_mode (v3.2): que arbol IDF usa el build (5.5 vs 4.4) y
si cabe Arduino-como-componente.
- pure_idf (master): el modo LENGUAJE que elige el usuario; sus ficheros son
las fuentes del componente main con su propio app_main().
Resolucion:
- _build_env acepta los tres. Un build IDF puro fuerza arduino_mode a falso:
la plantilla CMake mete el componente arduino-esp32 en cuanto existe
ARDUINO_ESP32_PATH, asi que dejarlo puesto compilaba el core de Arduino en
un build que no tiene sketch. Lo cazaron los tests de master.
- VELXIO_PURE_SKETCH solo con pure_idf, nunca con arduino_mode a falso a
secas: un target sin core arduino-esp32 sigue entregando un SKETCH al
traductor legacy y no debe tomar la rama del glob puro.
- La identidad del build-dir suma los dos tokens (|idf:N|ard:N y |lang:pure):
ningun par de esas combinaciones puede compartir un build/ configurado.
- La cadena de escritura de fuentes queda pure_idf -> arduino_mode -> legacy.
- sdkconfig: render de v3.2 (con target/use_idf5) mas el filtrado de simbolos
CONFIG_ARDUINO* de master cuando el build es puro.
test/backend/unit/test_espidf_compiler.py: los 7 tests que ya estaban rotos en
v3.2 (AttributeError: idf5_path, fixture sin actualizar desde que se anadio la
seleccion de IDF) vuelven a pasar.
Verificado: backend 293 pasan / 0 fallan (v3.2 traia 7 rotos); frontend 2268
pasan / 0 fallan en los dos shards.
Adds a third entry to the board language selector next to Arduino C++
and MicroPython: ESP-IDF. In this mode the user writes a plain ESP-IDF
project — app_main() entry point, FreeRTOS + driver APIs — and the
backend compiles it through the same ESP-IDF toolchain it already uses
for ESP32 Arduino sketches, just without the arduino-esp32 component.
Backend:
- CompileRequest.language ('espidf') threaded through the sync + async
compile paths and folded into the dedup job key (language='arduino'
and omitted hash identically so old clients keep dedupping).
- espidf_compiler: pure_idf flag. User files are written into main/
as-is (no Arduino.h wrap, no velxio_compat.h, Arduino library
resolution skipped), ARDUINO_ESP32_PATH is dropped from the build env
and VELXIO_PURE_SKETCH raised so the template CMake compiles the
user's own sources via a glob branch. Pure builds get their own
persistent build-dir variant through the eff_hash fold.
- QEMU WiFi compat for IDF-style code: esp_wifi.h/esp_wifi_init
detection sets has_wifi, and literal #define SSID/PASS plus
wifi_config_t designated initializers are normalized to the QEMU AP.
- CONFIG_ARDUINO_* lines are stripped from sdkconfig.defaults in pure
mode (the symbols don't exist without the arduino component).
Frontend:
- LanguageMode gains 'espidf'; BOARD_SUPPORTS_ESPIDF covers the ESP32
family (Xtensa, S3, C3). Toolbar shows the option only for those.
- Switching modes seeds a main.c blink skeleton (app_main + gpio
driver), mirroring the MicroPython main.py flow.
- compileCode sends language='espidf'; run/stop paths are unchanged
(the QEMU worker consumes the same merged flash image).
- New gallery example: esp32-idf-blink (LED + resistor on GPIO 2).
Tests: unit coverage for the build-env switch, IDF wifi normalization,
job-key variance, file-group seeding and the new example; verified
end-to-end in a container from the prod image (pure build produces a
bootable flash image; Arduino-mode build unchanged, same variant hash).
The test/ tree carried a lot of one-off research, repro, and exploration
material (autosearch notes, test_100_days, test_Raspberry_Pi_Pico_W, the Intel/Z80
+ custom-chip + ePaper + micropython repro harnesses, etc.) that no GitHub Actions
workflow nor deploy.sh runs. It bloats a fresh clone for no community benefit.
Keep only the dirs CI actually exercises:
- test/backend/ (backend-unit-tests.yml, backend-e2e-tests.yml, deploy.sh)
- test/esp32_cam/ (backend-e2e-tests.yml)
- test/test_circuit/ (test-circuit.yml)
No workflow, script, or Dockerfile referenced any removed path.
test/backend/unit/test_picow_inbound_gateway.py and
test/backend/integration/test_picow_net_bridge.py imported
app.services.picow_net, which moved to the private overlay in the open-core
split. They now live under pro/backend/tests/ and run against the overlay.
Removing them here unbreaks the OSS pytest collection (and the deploy gate).
ESP32 web-server examples are reachable from the browser via
/api/gateway/<client_id>/ (QEMU slirp hostfwd). The Pico W server lives
in the browser-side lwIP, so there was no inbound path: visiting the
chip's IP did nothing.
Add the mirror of tcp_nat.py: tcp_inbound.TcpInbound originates a TCP
connection INTO the chip over the WebSocket bridge (SYN -> SYN+ACK ->
ACK -> request -> response -> FIN), so the backend can fetch a page the
sketch serves on 10.13.37.42:80 and hand it back to the browser.
- bridge.py routes chip TCP segments addressed to a gateway-opened
connection to TcpInbound (before the chip-initiated NAT, which would
RST them); exposes http_into_chip() + ensure_chip_mac() (primes the
chip's gateway ARP).
- iot_gateway.py: same /api/gateway/<client_id>/ route now falls through
to the Pico W bridge when there's no ESP32 instance, builds a raw
HTTP/1.1 request, and parses the chip's response. Same plan gate, same
URL shape — the browser sees no difference between ESP32 and Pico W.
Validated end to end (real RP2040 emulator serving an HTTP page ->
gateway returns it) plus 6 unit tests for the TCP state machine,
response parsing and ARP priming.
P2.0 first cut took the first-alphabetical lib providing a header and then
checked manifest membership. When several installed libs ship the same header
(e.g. DHT118266, DHT_sensor_library, servodht11 all have DHT.h), the stray
first-match got rejected and the header was dropped even though the declared
lib provides it.
_find_manifest_library_for_header: when a manifest is supplied, pick the first
DECLARED library that provides the header. This both selects the right lib and
excludes undeclared ones. No manifest = legacy first-match.
Test strengthened with a stray same-header lib that sorts first.
When a compile supplies a 'libraries' manifest, _resolve_library_components
merges a USER-installed library only if it's declared in that set. A sketch
therefore never picks up an unrelated library from the shared dir (another
user's install, or a same-named clash) — the manifest is the resolution scope.
- _resolve_library_components(allowed_libraries): gate user-lib merges on
manifest membership; match by folder name OR library.properties name=,
normalised (display name vs on-disk folder differ by separators/case).
Core/bundled libs are never gated. None = legacy scan-all (unchanged).
- Threaded through compile() -> _compile_in_dir.
- compile.py: CompileRequest.libraries; folded into the async dedup _job_key
so a different manifest doesn't dedup to a job built with another.
Opt-in: omitting 'libraries' preserves current behaviour exactly.
Regression: test/backend/unit/test_espidf_core_first.py::TestManifestScope
The gallery example loaded but the Z80 never visibly ran: the screen stayed
frozen on garbage. Two multi-chip async-load races, neither caught by the
existing headless tests (which drive RESET manually and attach the display
before boot):
1. RESET edge-vs-level race. The Z80 only left reset on the RISING edge of
RESET (a pin watch). In the browser the 7 chips instantiate asynchronously,
so the small power-on-reset chip releases RESET before the larger Z80 has
registered its watch -> the edge is lost and the CPU stays in reset forever.
Fix: on_clock samples the RESET level (hardware-accurate; RESET is
level-sensitive) so a missed edge self-corrects. An undriven RESET reads low,
so the CPU safely stays in reset until something drives it high.
Repro/guard: chipbus-galaksija-reset-race (race ordering must still boot).
2. Display-snoop load-order race. galaksija-display was a passive write-snoop;
the ROM paints the screen ONCE at boot then idles, so a display that comes up
late misses every write and shows stale content forever. A snoop cannot
recover writes it never saw. Fix: fold the screen into the RAM chip
(galaksija-ram-display) and render from the ACTUAL video RAM (0x2800-0x2BFF,
internal 0x0800 with A0-A12 wiring) on a ~30 fps timer - correct regardless
of load order, exactly how the real machine scans video RAM.
Repro/guard: chipbus-galaksija-display-snoop-race (late snoop shows nothing)
+ chipbus-galaksija-ram-display (renders even when first paint is post-boot).
The example now has 6 chips (RAM+display merged, gdisp dropped), 76 wires.
Verified live in the browser: boots to "@'READY", shows the ">" prompt, and
pressing A echoes ">A_" through keyboard -> Z80 -> video RAM -> display. The
full chipbus suite is 45/45.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a memory-mapped keyboard so you can type into the Galaksija. Based on
the libretro Galaksija core's scheme (not guessed): reading 0x2000+offset
returns 0xFE when the key at that matrix offset is held, 0xFF otherwise;
the keyMap gives the offset per key ('A'=1 ... Enter=48, Space=31, etc.).
- galaksija-keyboard.c: drives reads of 0x2000-0x203F from a keys[] table and
exports set_key(offset, down) for the host to push key events. Never drives
outside the keyboard range.
- galaksija-ram.c: ram-64k variant that yields reads of 0x2000-0x203F to the
keyboard (writes still go to RAM), so the two never fight for the bus.
- ChipRuntime: ChipInstance.hasKeyboard + setKey() expose the chip's set_key.
- CustomChipPart: bridges browser keydown/keyup (by KeyboardEvent.code, via
GALAKSIJA_KEY_OFFSET) into the chip, ignoring keystrokes while the code
editor or an input is focused so typing code is never hijacked.
- The gallery example gains the keyboard chip (now 7 chips, 99 wires) and uses
galaksija-ram.
Test chipbus-galaksija-keyboard: pressing 'A' (offset 1) makes the BASIC
monitor echo "A" after its ">" prompt and advances the cursor. 41 chipbus
tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ships the full Galaksija (1983 Z80 home computer) as a runnable Retro
gallery example, plus the pieces needed to run a multi-chip bus live in the
browser.
Gallery example (examples-retro-intel.ts, id 'galaksija-z80-computer'):
Z80 + galaksija-rom (public-domain ROM A+B) + ram-64k + inverter (A13
decode) + galaksija-display + a power-on reset chip, wired chip-to-chip
over the bus (76 wires), no board. Click Resume and it boots the real ROM
to the "READY" prompt on the green display. Chip wasm is embedded
(wasmBase64) so it runs without a backend compile.
- ChipRuntime.tickTimers gains a wall-clock budget (CustomChipPart passes
6 ms): a faithful-but-slow event-driven bus can't run a real-time CPU in
one animation frame, so without a cap a Z80 fetching over the settle
kernel froze the tab. With the budget the sim advances slower than real
time (boots over a few seconds) and the UI stays responsive; fast
single-chip examples finish under budget and are unaffected.
- galaksija-display: blits its framebuffer on a ~30 fps timer instead of on
every character write, so a clear-screen burst doesn't flood the canvas.
- reset-gen: power-on reset (pulses RESET high, ties WAIT/BUSREQ/INT/NMI
high) so the machine boots on Resume without a manual reset.
- chipbus flag now defaults ON (override with ?chipbus=off): chip-to-chip
buses are a core capability; single-chip and board nets never take this
path, so the only thing enabled is multi-chip buses, previously broken.
Verified live in the browser: the example boots and renders "@'READY" with
the ">_" prompt, responsive. Full suite 2084 pass (5 pre-existing,
unrelated env failures).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Galaksija stores ASCII codes in its 0x2800 video RAM (verified by snooping
the boot: it writes "@'READY" + ">_" prompt). The original CHRGEN ROM uses
a hardware-specific addressing that does not map char-code*8 to a glyph, so
rendering through it produced garbled output. Render the ASCII codes with
the public-domain IBM/VGA 8x8 font (font8x8 by Daniel Hepper / Marcel
Sondaar) instead -- legible green-on-black phosphor text. The boot screen
now reads "@'READY" with the ">_" input prompt, exactly like a real
Galaksija. Tests updated to check the bright-green channel.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
galaksija-display.c: a 32x16 text video chip that renders the Galaksija
video RAM. It is a passive bus snoop -- watches WR + address + data, and on
a write into the 0x2800 video region stores the character and renders that
cell into a 256x128 framebuffer using the public-domain CHRGEN font (code*8,
bit 0 = lit). It never drives the bus. The host blits the framebuffer to the
chip canvas (vx_framebuffer_init / vx_buffer_write).
Two tests:
- chipbus-galaksija-display: snoop+render smoke test (a write of 'R' to
0x2802 lights its cell; unwritten cells stay blank).
- chipbus-galaksija-computer: the COMPLETE machine over the chip-to-chip bus
(Z80 + galaksija-rom + ram-64k + inverter decode + galaksija-display) boots
the public-domain ROM and renders the monitor's "READY" prompt on screen.
40 chipbus tests across 10 files pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The public-domain Galaksija ROM (Voja Antonic; ROM A monitor + integer
BASIC, ROM B float BASIC, 8 KB) runs on a standalone Z80 + external ROM +
RAM + an inverter for address decode, all chip-to-chip over the shared bus,
no board:
ROM 0x0000-0x1FFF rom.CE = A13
RAM 0x2000-0x3FFF ram.CE = NOT A13 (the inverter chip)
RD -> both OE ; WR -> RAM WE
Pin-level boot proof (mirrors test_intel/test_z80/galaksija.test.js): watch
M1, read the address bus on each opcode fetch, and confirm the Z80 leaves
the reset vector (DI; SUB A; JP 0x03DA), reaches the init routine at 0x03DA,
and runs 1000+ fetches across 50+ distinct ROM addresses -- the real
firmware executing end-to-end through the settle-kernel bus. The on-screen
"READY" prompt is the next milestone (needs the video display chip
rendering the 0x2800 video RAM).
galaksija-rom.c embeds the public-domain ROM A+B image. 38 chipbus tests
across 8 files pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The architectural heart of the retro computer, proven on real chips. A Z80,
a 32K ROM, a 64K RAM and an inverter (address-decode glue) are wired
chip-to-chip over a shared address + data bus, no board:
ROM at 0x0000-0x7FFF rom.CE = A15
RAM at 0x8000-0xFFFF ram.CE = NOT A15 (the inverter chip)
RD -> both OE ; WR -> RAM WE
The ROM program writes 0x5A to RAM at 0x8000, clears A, reads it back, and
HALTs only if the byte survived. HALT going low proves the full core works:
the Z80 runs from ROM, the inverter decodes A15 to select RAM (the settle
kernel drives the combinational glue across hops), and the RAM latches a
write and returns it on a read over the shared tri-state bus, all within
synchronous bus cycles. Adds z80-ram-rom.c (boot image) + ram-64k/inverter
fixtures. 37 chipbus tests across 7 files pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
End-to-end validation of Phases 0-2 on an actual CPU. The real Z80
(examples/intel/z80.c) and a 32K EPROM (z80-boot-rom.c, a rom-32k variant
holding JP 0x0006 / HALT) are wired chip-to-chip over a shared address +
data bus with no board. RD drives the ROM's OE; CE is left enabled.
Booting exercises all three phases at once: the Z80 drives the address ->
the ROM reacts on the shared net key (Phase 0); asserts RD -> the ROM
tri-state-drives the data bus while the Z80 released it (Phase 1); and reads
the data bus in the SAME tickTimers step, getting the settled byte
(Phase 2 settle-before-read). The Z80 fetches C3,06,00, jumps to 0x0006,
fetches 76, and HALTs -> drives HALT low, which the test observes.
z80.wasm is compiled from the committed examples/intel/z80.c; the boot ROM
source + chip.json live in test_custom_chips/sdk/examples. All 36 chipbus
tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
End-to-end proof of the chip-to-chip net-key fix through the real
ChipRuntime + PinManager (not a unit stub). Two chips compiled from C
with wasi-sdk:
- bus-driver.c: drives 0xA5 onto D0..D7 at setup.
- bus-reader.c: polls D0..D7 on a 1ms timer, mirrors onto OUT0..OUT7.
Wired chip-to-chip with no board; with the chipbus flag both chips' Dn
pins resolve to one shared net key, so the reader reproduces 0xA5.
- sdk/examples/bus-{driver,reader}.{c,chip.json}: the proof chips.
- __tests__/fixtures/chipbus/*.wasm: committed fixtures (regenerate with
the test_intel/scripts/compile-chip.sh flags).
- __tests__/chipbus-twochip-integration.test.ts: loads the fixtures via a
relative path; skipIf they are absent.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A user library that ships a core-named header (e.g. WiFiEspAT/src/WiFi.h)
could shadow the arduino-esp32 core during ESP-IDF library resolution.
WiFiEspAT shadowing WiFi.h pulled EspAtDrv.cpp into the build, whose
const char OK[]/STATUS[] collide with ESP-IDF's enum STATUS in
rom/ets_sys.h, breaking every ESP32 sketch that #include <WiFi.h>.
_resolve_library_components now:
- skips a header entirely when the arduino-esp32 core provides it
(computed set from cores/ + libraries/, cached), so a user lib can
never shadow WiFi.h/Wire.h/SPI.h/WebServer.h/...
- skips a resolved user lib whose library.properties architectures=
excludes esp32/*.
Regression: test/backend/unit/test_espidf_core_first.py
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The 2.9" tri-colour ESP32 alert badge rendered the red ALERT pill as white:
the red plane (0x26) was received but landed out of bounds and was dropped.
GxEPD2_3C writes the 0x24 (black) plane then the 0x26 (red) plane WITHOUT
re-seeking the RAM address counter between them — it relies on the SSD168x
counter wrapping back to the window start after the last byte of the window.
Our decoder advanced Y past the window end instead of wrapping, so every
0x26 byte hit y >= rows and was discarded (red_ram stayed all-init).
Mirror the hardware: when the X cursor wraps at the end of a row, advance Y
with a wrap at the active window boundary (yrange), honouring the data-entry
Y direction. Applied identically to the worker slave, the browser decoder,
and the Python golden reference so the three stay in lockstep. No regression
on the mono panels (their counter is re-seeked per plane, so the wrap is a
no-op for them); verified the tri-colour pill now renders red and the 2.9"
weather / 2.13" clock / 1.54" hello panels are unchanged.
ePaper panels rendered rotated/misaligned on AVR and RP2040 (e.g. the 2.13"
Pico clock came out sideways and clipped). The ESP32 worker decoder was just
taught to compose in the controller's native RAM geometry and rotate to the
display orientation, but the browser-side SSD168xDecoder (used by AVR/RP2040)
still composed at display dims with no rotation, so the two diverged.
- SSD168xDecoder.ts: port the worker's native-window compose + rotation.
* Size RAM to the longer side both ways so a rotated native layout
(128x296 behind a 296x128 panel) isn't truncated.
* Compose in the active RAM window, then rotate via the inverse of
Adafruit_GFX setRotation(1). Detect orientation by BYTE width so a
non-multiple-of-8 native width (the 2.13" panel is 122 px) is handled.
* Track the UNION of windows per frame: paged drivers (GxEPD2 page height
< panel) set one partial window per page, so compose must use the full
native area, not just the last page's strip. Fixes the all-white render
on paged panels (1.54" Uno, 4.2" Pico, 7.5" ESP32).
* Add an isBwr option: B/W panels treat 0x26 as a 2nd mono plane (white
only if both planes white), tri-colour panels keep red-wins.
* Default the active window to display geometry; the firmware overrides it.
- EPaperPart.ts: pass isBwr = cfg.palette === 'bwr' to the decoder.
- esp32_spi_slaves.py / esp32_worker.py: mirror the byte-aware rotation +
window-union in the worker, and derive is_bwr from panel_kind on the
runtime sensor_attach path too (fixes the tri-colour ESP32 alert badge).
- test_epaper/ssd168x_decoder.py: re-port the golden reference to match
(keeps the 3-way TS/Python/worker identity invariant). Tests updated to
construct tri-colour cases with is_bwr/palette='bwr'.
- examples-displays-epaper.ts: the Pico VCC wire referenced '3V3(OUT)',
which the velxio-pi-pico-w element doesn't expose (it has '3V3'), so the
wire snapped to the board corner. Use '3V3'.
Drives the simulation backend WebSocket directly, bypassing the
useSimulatorStore WiFi-stub prelude, and injects a minimal MP
program that exercises network.WLAN(STA_IF).active(True) +
.connect() + .isconnected() loop.
Used by Phase 7 of the velxio MP WiFi emulation project to capture
the register trace from a DEBUG=1 build of libqemu-xtensa.so. The
test exercises every API call the smart-ui-eyes example needs from
the network module.
Run:
node --experimental-websocket test/test_micropython_wifi_trace/test.mjs \
--backend=http://localhost:3080 --timeout=60
docker logs velxio-app 2>&1 | grep -E '\[(wifi|phya|ana )\]' | tail -200
Two Node.js tests that hit the simulation backend WebSocket directly
and inject minimal MicroPython programs via raw-paste REPL:
- test_micropython_i2c_minimal: smallest possible repro. I2C(0)+scan+
single-byte writeto. Used to prove the bug is NOT in the basic I2C
layer — this test passes both before and after the fix.
- test_micropython_i2c_ssd1306_repro: walks the full SSD1306 init
sequence (25-cmd init loop + 6x addr writes + writevto 8B + writevto
1024B). Used to prove the bug is NOT in the cmd sequence or the
writevto path — this test also passes both before and after.
These two tests refuted the original "missing TRANS_DONE IRQ"
hypothesis and pointed the investigation toward the file-load vs
raw-REPL difference, which led to identifying the per-byte _emit
bottleneck in esp32_worker.py.
Run with:
node --experimental-websocket test/test_micropython_i2c_minimal/test.mjs \
--backend=http://localhost:3080 --timeout=120
Requires the velxio container (or local backend with QEMU libs) on
the given backend URL.
Two small fixes after running the test inside the prod container for
the first time:
- The prod image lays out the backend at /app/app/, not /app/backend/app/
(the Dockerfile.standalone COPYs only the inner package). Use /app
as the sys.path root so `from app.pro.services import ...` resolves.
- The CHIP=0x60 and BLOCK= prints race against the socket drain. The
test was treating "saw CHIP= but BLOCK= not in buffer yet" as a
hard failure and exiting before the second I2C read finished.
Gate the success path on both markers present and keep polling
otherwise.
Verified end-to-end in the prod container:
[proto] >>> ['I2C', '1', '76', 'RR', 'd0', '1']
[proto] <<< I2C_DATA 1 76 60
[proto] >>> ['I2C', '1', '76', 'RR', 'f7', '8']
[proto] <<< I2C_DATA 1 76 530280155e607b50
[test] OK — guest read chip ID = 0x60
[test] OK — block read BLOCK=530280155e607b50
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the public extension points the velxio-prod overlay uses to bind
real canvas-side I2C/SPI/UART models (BME280, future MCP23017, etc.)
to a running Pi guest's protocol shims:
- qemu_manager: set_pi_slave_handler(fn) / get_pi_slave_handler() for
pi_attach_slave + pi_detach_slave WebSocket messages. OSS image
leaves the hook unset so the messages are silently dropped.
- simulation route: parses the two new WS message types and forwards
them to the registered handler when present.
- RaspberryPi3Bridge: attachSlave(spec) / detachSlave(spec) frontend
side of the protocol.
- piSlaveScanner: at simulation start walks components + wires,
identifies I2C/SPI/UART peers wired to Pi protocol pins (40-pin
header physical-pin numbering), and emits one attach per
bus/address pair (deduped across SDA+SCL wires).
- RaspberryPiWorkspace: invokes the scanner once the bridge is open,
with retries to ride out the WS-still-connecting race.
- integration test: pi3_bme280_attach.py boots the Pi, pre-attaches a
BME280 via the slave handler, runs a host-side proto loop, runs
guest python smbus2.read_byte_data(0x76, 0xD0) and asserts the
console reads back CHIP=0x60.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Phase 2 E2E test was sending the Python GPIO command via
'python3 -c "..."' but bash quote-nesting silently corrupted the
script — the python process started, printed nothing, exited 0, and
the test asserted 'GPIO_SETUP 17 out' was missing in proto bytes
(it never got sent because the python script never ran).
Switch the test to base64-encode the script + pipe through base64 -d
into a file, then execute. Verified end-to-end now:
[test] proto received 36 bytes:
GPIO_SETUP 17 out pud_off
GPIO 17 1
[test] ✓ shim → proto pipeline works
Also bump the rootfs manifest entry to the final Phase 2 build
(d6d4a274 raw / debd1c33 zst, version 2026.05+phase2-shims-final).
Earlier auto-discovery in _transport.py was hanging at import time
on some glob/sysfs interaction. Now hardcoded /dev/vport1p1 which
is the empirical path under -M virt + virtio-blk-pci on slot 0.
raspi3b pl011 RX is broken in QEMU 10 + kernel 6.12 — see
project/pi-emulation/decisions.md for the full debugging trail.
This commit lands Phase 1 of the rebuild: switch the QEMU machine
to virt + cortex-a53, boot the velxio kernel/initramfs/rootfs over
virtio-blk-pci, and expose the user shell on /dev/hvc0 via
virtio-serial-pci + virtconsole.
End-to-end smoke verified: boot → agetty autologin → bash prompt →
echo round-trip returns the typed token. Tested inside the prod
container with QEMU 10.0.8 and our cloud-derived kernel 6.12.88.
What changed:
backend/app/services/qemu_manager.py
PI3_IMAGE_SET -> raspberry-pi-3-virt
PI3_KERNEL_NAME / PI3_INITRAMFS_NAME / PI3_ROOTFS_NAME new
QEMU cmd rewritten end-to-end:
-M virt -cpu cortex-a53 -smp 4 -m 1G
-kernel <velxio-kernel-arm64> -initrd <velxio-initramfs-arm64.cpio.gz>
-drive ... -device virtio-blk-pci (NOT virtio-blk-device — mmio
variant left /dev/vda unregistered)
-nic none -display none -monitor none -serial none
-chardev socket... -device virtio-serial-pci -device virtconsole
(user console -> /dev/hvc0)
-chardev socket... -device virtserialport,name=velxio-protocol
(Phase 2 channel -> /dev/vport0p2)
No -dtb (virt generates its own), no -append init=... (kernel runs
our initramfs which then switch_root to rootfs and exec's its
/sbin/init — Alpine OpenRC).
backend/app/services/boot_images/manifest.json
New image set raspberry-pi-3-virt with three assets uploaded via
the existing license-endpoint pipeline. Old raspberry-pi-3 entry
flagged deprecated:true and kept for one release for rollback.
test/pi3_console_boot/test_pi3_console_boot.py
Updated QEMU argv to match qemu_manager exactly. Markers now look
for the Velxio Pi Simulator MOTD + 'login on hvc0' (autologin
proof). Round-trip echo still required to pass.
test_busy_wait_100us and test_busy_wait_1us measure busy_wait_us()
elapsed time against absolute thresholds (500µs / 100µs). Under
contended CI/deploy-gate machines these can blow through the budget
even when the busy-wait implementation is correct, blocking deploys
that have nothing to do with DHT22 timing.
Same pattern already applied to test_response_timing_analysis in
this file — skipped via @unittest.skipIf(os.environ['CI']=='true').
David Montero
dependabot[bot]