Hardening pass on 57325aa, which fixed the reported HC-SR04 divider but did it
by returning the first answer any branch produced. Five topologies people
really draw break under that rule, and each one now has a test that fails on
57325aa and passes here:
- a ground tie point almost always carries something else (a button's
pull-down). Reaching a rail has to END the walk, or a grounded pin resolves
to whatever GPIO that pull-down hangs off. Same through a breadboard rail;
- two custom chips on one wire stopped sharing a chipbus net key, because a
nested frame answered with its own neighbour's per-endpoint key before the
root could mint the net-canonical one;
- a chip one hop away ended the search before a real GPIO on the same node
was ever looked at;
- a node first reached deep, where the hop budget cut it short, was never
re-walked when a shorter route arrived.
So the walk now models what a net IS. It collects the whole NODE (every pin
joined without crossing a component: the wires, and a breadboard strip's
holes), then judges the set: a driven pad wins outright; a supply pad means the
node IS the rail and the walk stops there; only then does it cross a passive to
the next node, where a rail is the weaker answer that lets a level-shifted
sensor resolve. Wires, components and breadboard groups are indexed per store
array, and a driven pad short-circuits collection — the common case is now
0.08 ms/pin on a 300-wire canvas, better than before this work started, with
the shared-rail worst case at ~2 ms.
Second half: a part that models its own line keeps it, on every board.
connectDigitalInputsToMcu thresholds the solved node into every input pin, and
its `sourcedNets` gate only ever protected part-driven lines by accident — it
holds while no component sits on the net, and stops the moment the user adds
the pull-up or the divider real hardware needs. Then the connector pins the
line an HC-SR04 or a DHT22 is answering on. simulation/partPinOwnership records
what each part drives (implicitly, through the simulator it is handed) and the
connector skips those pins; the ESP32 bridge answers the same question for
sensors the backend or an engine emulates, through a new optional `ownsPin`
seam. Parts SPICE can describe never claim anything, so a mis-wired button
still reads stuck — that property is why spiceDrivenInputs exists.
Also: setSensors merges per FIELD, so the coarser registration path can no
longer drop the finer one's echo_pin; the ESP32 bridge's guard is narrowed to
the single-wire sensor types, since an ePaper's DC pin or a keypad row must
stay drivable; and the Mega and Pico W pin maps get the supply-pad guard the
other boards have, where '5V' parsed as D5 and '3V3' as D3.
Verified in a browser against a local build of this tree, driving the real app:
HC-SR04 and DHT22, direct wiring and level-shifted, on Uno (avr8js), Pico
(rp2040js), ESP32 (esp32js and QEMU) and ESP32-S3 (esp32s3js) — see
scripts/sensor-wiring-matrix.mjs in velxio-prod.
An HC-SR04 on an ESP32-S3 answered "No echo" forever the moment its ECHO
went through the 1k/2k2 divider a real 5 V sensor needs, and worked again
when the divider came out (reported on Discord, project esp32-s3-distance).
Nothing about the divider is wrong: three separate resolvers could not see
past it.
1. The wire walk treated a net as a chain. Arriving at the divider's tap it
hopped straight across the lower resistor, hit GND, and answered "tied to
a rail" (-1) — never looking at the OTHER wire on that same tap, where
GPIO41 was. A rail was also allowed to end the search, so a plain pull-up
drawn before the signal wire hid the GPIO behind it the same way. The walk
now explores the whole node (same depth, one shared visited set) and
treats a supply pad as the weakest answer: remembered, returned only when
no driven pin was found anywhere.
2. The walk moved to simulation/PinTrace so the store can ask the same
question. It pre-registered backend sensors with its own single-wire
lookup — one wire, far end must be a board — so ECHO (landing on a
resistor) resolved to nothing. That entry REPLACES the part's, keyed by
pin, so the echo pin was silently dropped and the QEMU worker fell back to
TRIG+1: it pulsed GPIO22, which nobody was reading. Both the data pin and
the extra pins now resolve through the shared walk, so a sensor behind a
breadboard strip or a divider registers like any other.
3. Esp32Bridge.sendPinEvent now refuses a pad a registered sensor owns, the
guard the in-browser engines already have. connectDigitalInputsToMcu
thresholds the solved node, and `sourcedNets` does not hold it back once
real components sit on the net — the divider makes ECHO exactly that, and
it solves at ~0 V because nothing models the sensor's output.
Test covers the reported circuit end to end: the walk, and the echo_pin that
reaches the start_esp32 payload.
A Visuino-built `Generated.ino.merged.bin` was reported as failing to run in
Velxio. It runs fine. The screenshot shows the ESP32 bootloader completing and
the sketch executing; what it also shows is
[E][STA.cpp:357] connect(): STA connect failed! 0x300a: ESP_ERR_WIFI_SSID
[W][WiFiGeneric.cpp:900] setTxPower(): Neither AP or STA has been started
which is WiFi association, not firmware loading — and reads as "the emulator is
broken" to someone who has no reason to know the difference.
The emulated radio broadcasts four networks and nothing else: Velxio-GUEST,
PICSimLabWifi, Espressif, MasseyWifi, all open. Sketches compiled here never
have to know, because espidf_compiler rewrites their SSID literal on the way to
the emulator. Firmware that arrives already built skips that step by
construction — there is no compile to rewrite anything — so it hunts for the
network it was built for and never finds it.
Confirmed on the real emulator rather than reasoned about: the same sketch
built with SSID "MyHomeNet" reports FAILED, and with "Velxio-GUEST" connects on
192.168.4.x. Same board, same path, only the string differs.
So the upload path now reads the image and, when it carries the Arduino WiFi
stack but names none of the reachable networks, prints one line saying which
networks exist and why a compiled-elsewhere binary cannot be fixed up
automatically. When the image already names one it says nothing: a note nobody
needs is noise, and noise is how people learn to ignore the console. The
markers are deliberately narrow for the same reason — `esp_wifi` alone appears
in images that never touch the radio.
The SSID list moves to EMULATED_WIFI_SSIDS in types/board.ts. It was written by
hand inside the MicroPython prelude string, and a second hand-written copy is
precisely what #268 was made of.
Not closed by this: the reporter's exact error is ESP_ERR_WIFI_SSID from
esp_wifi_connect, meaning the SSID handed to the stack was invalid rather than
merely absent from the air — forcing an empty SSID here produces a different
message from a different line (STA.cpp:387, "SSID too long or missing!"). That
may be his Visuino configuration or his core version; it needs his binary to
tell, which he offered to send.
Export a project to .zip, start a new project, import the zip: the circuit
comes back and the board does not. Two reporters; the second noticed it "only
works with wokwi components", which is the shape of it — the passives are real
Wokwi part types and survive, the board was not one.
The bug was in the EXPORT, not the import. wokwiZip knew four boards and wrote
`wokwi-arduino-uno`/`uno` for anything else, while the connections kept the
board's real canvas id. The attached repro says so in six lines: parts declare
an `uno`, connections name `esp32`.
Of those four, only two round-tripped. 'arduino-mega' was missing from the
export's board-id test, so a Mega's wires named `arduino-mega` while its part
was `mega`. 'raspberry-pi-pico' was tested against 'nano-rp2040', an id nothing
has been issued since boards became instances, and the import mapped it back to
the same dead string. Nobody noticed because the module had no tests.
"Velxio has four boards" was written down five times: two tables in the export,
one in the import, a hand-copied union in the import dispatcher, and BoardType
in the store — which setBoardType already ignored, casting to BoardKind to do
its job. All five are gone, plus BOARD_FQBN and BOARD_LABELS, two more from the
same era that nothing imported.
A board may claim a Wokwi part type only when a Wokwi element draws it. That is
not a preference: connections carry PIN NAMES, and the names a wire carries are
whatever the element on the canvas calls its pins. Those four render through
wokwi-elements, so an Uno wire says `GND.2`, `A4`, `5V` and the file opens over
there. Every other board renders through a velxio-* element with our names —
ours calls a pin `34` where Wokwi's esp32-devkit-v1 calls it `D34` — so
claiming a Wokwi board for one would produce a file that loads in Wokwi and
attaches every wire to a pin that does not exist. They travel as
`board-velxio-<kind>`, which also round-trips the overlay's kinds, runtime
strings this module cannot enumerate.
Nothing is coerced any more. An unrecognised diagram returns a null board and
says so instead of answering 'arduino-uno'; a second board in the diagram is
reported rather than dropped; wires naming a part the diagram lacks are called
out, which is what the reporter's own file now does when you import it.
Two more halves surfaced only by driving the real editor:
The board was still missing after import, because setBoardType re-kinds the
ACTIVE board and a new project has none — nothing to re-kind, so the circuit
arrived without its chip exactly as reported. The importers now add a board
when there is none, activate it (addBoard promotes but does not sync the flat
legacy fields), and retarget the wires to the id the board actually got.
And the export read `boardType`, the flat mirror, which only tracks the active
board through setActiveBoardId/setBoardType and reports every Raspberry Pi as
an 'arduino-uno' on purpose. It now reads the board itself, so the kind and the
canvas id come off one object and cannot disagree.
Adversarial review of the fix caught six more, every one reproduced.
The worst was not a loss but a corruption, and it was mine: a canvas with two
boards wrote BOTH boards' wires against the exported board's part id, because
for every non-Arduino family that id IS the first-of-kind canvas id. The second
board's components came back attached to the first board's pins — a different
circuit, from a file that was internally consistent, so nothing complained. My
first attempt at a dangling-wire warning could not see it either: it allowed
the board KIND as a defined id, which is precisely the id a second board
writes. The check now reads the raw diagram, the export leaves other boards'
wires out rather than mis-addressing them, and it says what did not travel.
The rest: isKnownBoardKind used `in`, so 'toString' and '__proto__' passed a
guard whose input comes from files people send each other; the importer's
extension list admitted only the Arduino four, so a MicroPython project came
back with an empty editor; the exporter never wrote libraries.txt though the
importer has always read it and offered to install what it finds; and the
unknown-board message claimed the circuit landed on the current board without
retargeting its wires there.
41 tests, over every board kind, which is the assertion this module never had:
it was written when Velxio had four boards and nothing failed when it grew to
thirty. Verified in a real browser through the reporter's flow — build the
circuit, File > Export, new project, import — for arduino-uno (unchanged),
arduino-mega and raspberry-pi-pico (both previously broken) and esp32/esp32-s3.
Before and after are identical in all five.
Still lossy, and out of scope here: a .zip carries one board, so a multi-board
project must be saved as .vlx to survive whole. Wire waypoints, one shade of
green, and board-level state (boardOptions, SPIFFS, languageMode) do not travel
either.
ADC_PIN_MAP and ADC_PIN_TO_GPIO are keyed by the BoardKind union. An overlay
board's kind is a runtime string that can never join that union, so
connectAnalogInputsToMcu hit `if (!adcPins) continue;` at all three sites for
every pro board: a divider, an LDR, an op-amp output wired to one solved
correctly in SPICE and then had nowhere to go. Parts that inject their own value
(a potentiometer, through partUtils) were unaffected, which is exactly what made
this easy to miss — analog looked like it worked.
A pro board now declares its pads through ProBoardDef.adcPins, and pad->GPIO
resolution consults its own pinToNumber first. Both fall back to the existing
tables untouched, so no OSS board changes behaviour.
The pad NAME matters, not the GPIO: on the Pimoroni Pico Plus 2 W the analog
pads are silked 26/27/28 and tied to GP40/41/42 through 1k, so the two disagree
by design.
A sketch that includes BTstackLib on a CYW43 board fails with arduino-pico's
static_assert: "This library needs Bluetooth enabled. Use the 'Tools->IP/
Bluetooth Stack' menu in the IDE to enable it." There is no such menu here, so
it reads as a setting we forgot to expose. What is actually true is simpler:
the CYW43439's Bluetooth controller is not emulated — only its WiFi side is —
so there is nothing to switch on.
Appends that, leaving the compiler's own diagnostic intact above it because the
developer still needs the real text. Deliberately a separate helper from
humanize_cli_error, which REPLACES arduino-cli's own failures; this one only
adds context under a compiler message.
Worth knowing for whoever picks up the BT work: this is the honest-failure
milestone. Making a BT sketch compile at all means flipping this board's FQBN
to ipbtstack=ipv4btcble, which swaps liblwip.a for liblwip-bt.a and costs every
sketch on the board ~71 KB flash and ~18 KB RAM — and leaves it reaching
HCI_STATE_WORKING while advertising to nobody. The mechanism is mapped in
project/pimoroni-pico-plus-2w/STATUS.md.
MicroPython over the QEMU worker stopped loading: every project failed with
"the board stopped answering while writing _vlx_main.py", the board sitting
idle at a healthy `>>>`.
The uploader shipped in 664c3581 waits for the board's end-of-execution marker
before sending the next step — that wait is the whole point, it replaced a
fixed delay that pushed 64 bytes every 150 ms. The marker never arrived. Raw
REPL answers each statement with `OK` <EOT> traceback <EOT> `>` and not one
newline in it, and the UART buffer flushed only on newline, carriage return, a
period or 256 bytes. The answer sat in the buffer; the uploader timed out
waiting for something the backend was holding.
Nothing had needed that answer before: the old uploader pasted the whole
program and never listened. EOT is a message boundary in this protocol, so the
buffer treats it as one.
Found by forcing a MicroPython project down the QEMU path with ?esp32sim=qemu —
the residual risk 664c3581 flagged and could not exercise, because velxio.dev
routes every supported ESP32 kind to the in-browser engines. Self-hosters from
this repo have no engines and always take this path, so it broke for exactly
the people the #219 fix was written for.
`WiFi.begin()` panicked the guest: `Guru Meditation Error: Core 0 panic'ed
(LoadStorePIFAddrError)`, EXCVADDR 0x60033c00. Reported against a self-hosted
dev setup, and it did not reproduce on velxio.dev — which routes every
supported ESP32 kind to the in-browser engines. The engines are not in this
repo, so anyone self-hosting from it always runs QEMU, which is where the crash
lives. It was verified away on a path the reporter cannot take.
0x60033c00 is the WiFi MAC at its APB alias, and the fork only instantiates
that device when a NIC is attached (`qemu_find_nic_info(TYPE_ESP32_WIFI)` in
hw/xtensa/esp32.c). The worker attached one only when `wifi_enabled` was true,
so a regex over the user's source decided whether the machine had a radio at
all. Reproduced against the real worker: same sketch, same firmware, only the
flag differs — true connects on 192.168.4.x, false panics exactly as reported.
Two things were wrong, and the second is why the first was fatal.
The scan was too narrow. Arduino mode read only the entry sketch while pure-IDF
mode read every file, so `#include <WiFi.h>` in a project header was judged
WiFi-less. Verified against the live backend: main sketch -> has_wifi=true, the
same include one file away -> false. Both branches read the whole project now.
The frontend also trusted a false over its own all-files scan; the two signals
are OR'd, because a stale or narrower false is the direction that hurts.
But no scan should have been load-bearing. A real ESP32 has a radio whether or
not the sketch uses it, so the emulated one does too: the NIC is attached
unconditionally on machines that model a MAC. A misread sketch now fails to
connect instead of faulting. Only those machines — the S3 machine models no
radio (hw/xtensa/esp32s3.c never looks for the NIC), so handing it one would
leave an unconsumed netdev. `wifi_enabled` still gates the host forward, which
exposes the guest's server and does belong to the sketch.
Measured before making it unconditional, on a sketch that never touches WiFi:
boot time, guest-clock-vs-real-time and container CPU unchanged over 4 paired
runs. The AP's beacon timer runs on QEMU_CLOCK_REALTIME — the reason -icount is
off for Xtensa, b0fe5b5d — but idle beacons cost nothing measurable. One host,
Linux x86-64, one sim at a time; small effects would not show.
With the radio always present, the MicroPython `network` stub loses its reason
to exist on this path: it was there because `WLAN(STA_IF)` hangs forever with no
NIC behind it. Keeping the old condition would put the stub in front of a
working device whenever the scan misread the sketch — #262, from the other
direction. It now depends only on whether the backend models a radio.
And when this class of fault does happen, the dump says "memory" and nothing
else; a week went into #260 for that reason. LoadStorePIFAddrError is always an
unmapped peripheral, so the worker now says so in the serial monitor, naming
the WiFi MAC when the address is in its page.
test_esp32_wifi_args re-implemented the arg logic in its own `_simulate_args`,
so it would have passed unchanged through this entire bug — the same trap
1800f572 called out for the detector. The decision is now `wifi_nic_arg()` and
the test calls it.
Verified end to end against the QEMU worker: the multi-file WiFi project
compiles with has_wifi=true and connects, and a WiFi-less sketch still boots and
runs with the radio attached.
The CYW43439 emulator is stateful — bus handshake, backplane window, SDPCM
sequence and credit, event mask, link state — and attachPioPeripheral is
idempotent, called once at board-add. So the chip's state SURVIVED every later
Run while the guest driver started over from scratch. The second Run had a
fresh driver talking to a chip mid-conversation: the handshake "passed" against
stale registers, IOCTL replies came back zeroed (WiFi.macAddress() read all
00s), and the first call that actually waited on the chip — WiFi.begin(), or
the second IOCTL of any kind — blocked forever.
A rebuilt MCU means a rebooted board, so the chip is power-cycled with it: the
peripheral is replaced wholesale (it owns its network bridge and its whole
register model, so there is no partial-reset path to get wrong) right before
the FIFO hooks are re-installed on the new PIO.
Also: overlay board simulators now get onSimulationStart. The Pimoroni Pico
Plus 2 W and Badger 2350 carry the same CYW43439 die on an RP2350 simulator,
which is not an RP2040Simulator, so they never had their network bridge opened
— WiFi that reached an IP and could then reach nothing.
Verified on the RP2350 side with a harness that boots a real arduino-pico
binary twice in one simulator (pro overlay); before the fix run 2 hung with a
zeroed MAC, after it both runs associate.
A MicroPython library in Velxio is a .py file copied onto the board before
main.py runs. It got there inlined into the program as ONE Python string
literal:
with open("mqtt_as.py",'w') as _f:
_f.write("...36 KB of escaped source on one line...")
so compiling the program asked for one CONTIGUOUS allocation the size of the
file. Issue #219 is that bill coming due on a PSRAM-less ESP32: 36407 bytes of
mqtt_as.py, `MemoryError: memory allocation failed, allocating 36410 bytes`.
Never the firmware version the thread suspected.
The v1.28 bump (bd1613a9) made that single allocation fit, so the reported
message is gone — but the mechanism was not, and what it costs today is worse
than the error was. Measured on the reporter's own project, on velxio.dev and
on staging: 216 s pinned in the raw REPL before the first line ran (the paste
went out at 64 bytes per 150 ms, ~426 B/s), and then `OSError: WiFi Out of
Memory`, because 36 KB of heap was still held by the compiled literal when
esp_wifi came up. The project could not work.
Nothing is inlined now. Each file is written by a sequence of small executions
(open / write(b'...') x N / close) — the shape mpremote's `fs cp` uses against
real hardware — so peak guest RAM is a few dozen bytes per step whatever the
library weighs. The program is written as a file too and started with execfile,
whose lexer streams off flash: a big sketch compiles in the same memory a small
one does, and it still runs in __main__ exactly as the pasted code did.
Flow control is the other half. One burst is in flight at a time, sized to fit
the 128-byte RX FIFO both backends model, and what releases the next is the
board's own end-of-execution marker — proof it consumed the last one, instead
of a delay chosen by hoping. A step can only overflow one burst on a
pathologically long file path, and those few bursts fall back to timed pacing
rather than corrupt the stream. The hundreds of protocol exchanges stay out of
the serial monitor; a progress line goes in their place, and a board-side
failure mid-upload is reported instead of starting a program whose library
never arrived.
One implementation, shared by the QEMU bridge and the in-browser engines. The
hand-ported second copy is why this bug lived in both paths, and the pro
overlay now re-exports this module instead of keeping its own.
Also fixes a latent corruption: sanitizeForRepl's inline-comment rule used to
run over that one-line literal, so a ` #` inside a library file made it strip
non-ASCII from everything after it. Library bytes are never tokenized now.
Verified on staging with the reporter's project (main.py + mqtt_as.py): upload
complete in 36.5 s, then "Got reliable connection / Connecting to broker /
publish 0". The WiFi OOM is gone with the heap it was competing for.
Not exercised live: the OSS QEMU MicroPython path — velxio.dev routes every
supported ESP32 kind to the in-browser engines, so staging cannot reach it. It
runs the same session and is covered by the unit tests, and its 800/200 ms
handshake timings are preserved rather than unified with the engines' 400/150.
The ESP32 and STM32 shims only offered `feedUart(uart, data: string)` and
`serialWrite(text)`, both of which run the string through TextEncoder. That
is UTF-8, so every byte >= 0x80 leaves as TWO bytes. Fine for the text
protocols the seam was written for (NMEA out of the GPS module, AT chatter);
silently fatal for anything framed.
A part answering a binary protocol has no way to spot it. A ten-byte reply of
AA 55 04 00 FF FD 01 FD 55 AA
reaches the guest as fifteen bytes,
C2 AA 55 04 00 C3 BF C3 BD 01 C3 BD 55 C2 AA
and no parser recovers from that: the sync word is gone, the length field
disagrees with the frame, the checksum covers bytes that were never sent. The
symptom is a device that answers and a sketch that times out anyway, with
nothing in the console — which is what makes it expensive to find. Protocols
whose replies happen to be ASCII pass straight through, so the seam looks
fine right up until a sync word is 0xAA.
Both shims now also expose `sendSerialBytes(bytes: number[], uart = 0)`,
forwarding to the bridge method of the same name without encoding. feedUart
keeps its text contract and its callers; its doc comment now says so.
The test pins the difference rather than the implementation: same frame down
both paths, asserting the text one mangles it and the raw one does not, plus
a round trip of all 256 byte values.
boardPinToNumber had two failure shapes across the ESP32 family, both silent.
A pad the mapper cannot name returns null, and a null pin means the wire
connects to nothing: the part sits on the canvas, the sketch drives a dead
pin, and the only trace is one console.warn from WirePin.
- DevKit V1 RX0/TX0. Unlike RX2/TX2, which have numeric twins at the same
coordinate ('16'/'17', be24243e), these have none, so they were the only
route to GPIO3 and GPIO1 and a wire to either did nothing.
- S3 DevKitC 40/41/42/45/46/47/48, rejected by a GPIO ceiling of 39 shared
with the classic ESP32. The ceiling is now per family.
- Nano ESP32 RX0/TX1/B0/B1. Its map named them D0/D1, pads the element does
not draw, leaving GPIO44 and GPIO43 unreachable.
- Reset pads (RST) on the S3, C3 and Nano, now -1 like the classic board's
EN rather than null.
The second shape is worse, because the pad does something wrong instead of
nothing: the numeric fallback is parseInt(pinName, 10), and parseInt('3V', 10)
is 3. The Wemos Lolin32 Lite's supply pad resolved to GPIO3, and the XIAO S3,
XIAO C3, Nano ESP32 and Aitewin C3 SuperMini resolved 3V3 to GPIO3 and 5V to
GPIO5 — a wire to the rail drove a real pin. Those branches had no power
guard at all; the shared POWER_PAD_RE replaces the ad-hoc startsWith checks.
Same reason the Wemos SPICE group now lists the '3V' pad it actually draws:
it declared '3V3', a name that board does not expose.
Every expected value is cross-checked against the `target` field of the
upstream wokwi-boards board.json, which is what surfaced this: the pad data
in PR #165 (naweiss) generates that field, and diffing it against our tables
turned up the whole class. Six of the ten ESP32-family boards now match
upstream exactly.
Left deliberately: the UART pad numbers (S3 TX/RX are GPIO43/44 and C3
TX/RX are GPIO21/20, not the classic 1/3) and the DevKitC V4 flash block
(D0-D3/CMD/CLK are GPIO6-11). Unlike the above, those pads resolve today, so
changing them moves an existing wire to a different pin.
The test walks every pad of every ESP32-family board and fails on either
shape. Pad lists are copied from Esp32Element.ts, as in
pin-inspector-layout.test.ts, because vitest runs in the node environment.
Co-authored-by: naweiss <naweiss@g.jct.ac.il>
An overlay board was invisible to the build-options model: its kind is not in
the OSS family sets, so isEsp32Family() said no, the Board Options menu never
appeared, and every compile went out with board_options: null - the
conservative family default of CONFIG_SPIRAM=n.
That is wrong for a module that always ships PSRAM. The ESP32-S3-EYE carries
8 MB of octal PSRAM and esp_camera allocates its frame buffer with
MALLOC_CAP_SPIRAM, so its camera examples compiled cleanly and then died at run
time with 'cam_dma_config(509): frame buffer malloc failed' - a hardware fact
surfacing as a mysterious driver error.
Boards now declare defaultBoardOptions, sent when the project saved none of its
own. ONLY the declared keys are sent: the backend backfills the rest from its
own defaults, which are deliberately not the modal's (huge_app vs min_spiffs),
so a board that gains one declared knob does not silently get repartitioned.
Boards that declare nothing keep sending null and build bit-for-bit as before.
Also route the family predicates through the registry, so overlay ESP32 boards
finally get the Tools menu they were missing.
arduino-esp32 3.x keeps most of its requires PRIVATE, so a sketch that
includes esp_lcd_panel_io.h dies with 'No such file or directory' -
while the SAME header resolves fine as soon as any user library is
present, because the merged user-libs component requires esp_lcd. That
made the failure look random.
Detect the IDF-tree components a sketch includes and add them to main's
REQUIRES, the same shape as the registry managed-component table next to
it. Guarded on the component existing in the IDF tree, so older IDF
generations are unaffected.
Found writing the ESP32-S3-EYE camera example, and it matters beyond
that board: esp_lcd, esp_cache, esp_ldo_regulator and esp_psram are
exactly what an S3/P4 sketch reaches for once it drives a panel.
David's kit is the ESP32-S3-EYE, not the ESP-EYE the project had been
built against - different product, and only this one has a screen. Card
art follows the real board: camera and function buttons on the main
board, the orange SUB silk band, then the 240x240 panel.
The 4-pin module's picker icon still drew the screen as a flat dark
rect, so the catalogue entry looked like a different part from the one
you get on the canvas. Same glass treatment as the element: bonded
panel with its bevel, inset active area, flex tail and sheen.
The generated components-metadata.json is synced by hand because the
generator skips without a third-party/wokwi-elements clone; the value
is copied verbatim from component-overrides.json, which stays the
source of truth, so a later regeneration reproduces it exactly.
Both SSD1306 modules are real parts and both belong in the catalogue -
the mismatch was in the docs, not the parts:
- the 8-pin Adafruit breakout was served a datasheet listing a 4-pin
I2C pinout (VCC/GND/SCL/SDA) that is not its header at all; it now
documents its own 9-way pinout with the I2C and SPI role of each pin;
- the 4-pin I2C module had NO datasheet, because docs resolve by
component id and ssd1306-i2c-4pin.md did not exist - so the part
showed nothing. It has one now.
Each points at the other so it is obvious which one you want.
The screen was a bare <rect fill='#1A1A1A'>, so the part read as a
cut-out in the board rather than one integrated piece - which is what
the module actually is: the glass panel is bonded ON TOP of the PCB and
overhangs the 128x64 active area on every side.
Now drawn as a real panel: glass body with a lit stand-off bevel, the
active area inset within it (the canvas still lands exactly on it), the
flex tail running down to the driver, and a faint specular sheen so it
reads as a screen even while dark. Body 108 -> 118 px for the overhang.
ssd1306-render tests 9/9.
`test_unknown_boards_share_the_default_pair` required every unrecognised
FQBN to collapse onto the default (esp32, esp32) key. 82f25a03 stopped
that on purpose: `_idf_target` defaults ANY unknown board to 'esp32', so
an AVR / RP2040 / STM32 compile inherited the ESP32 identity and queued
behind unrelated ESP-IDF builds it shares no build dir with. The test
was never updated, so master has been red since.
Replaced with the contract the code actually implements, split so a
future change cannot quietly take the useful half away:
- non-ESP32 FQBNs get distinct identities and distinct locks (an AVR
build must not wait on an ESP-IDF one);
- a non-ESP32 board still serialises with ITSELF — dropping the shared
key must not drop the per-board lock;
- an unknown `esp32:` FQBN still shares the default pair, because that
one really does land in the shared build dir, and there
over-serialising is the safe direction.
Measured on staging: 46 per-variant build trees held 25 GB - each one a
full copy of the same ESP-IDF component objects - while ccache, the
cache that serves EVERY target and variant, was capped at 8 GB and
permanently full: 99.97% of its limit, 153 evictions, 47% lifetime hit
rate. Objects were being thrown out before they could be reused, so
builds recompiled work they had already done.
Variant cap 12 -> 6 and ccache 8G -> 20G. The asymmetry is deliberate:
an evicted variant dir rebuilds quickly FROM ccache, whereas a ccache
miss recompiles from source. Pruning the cold variants freed 8 GB on
the staging box (88% -> 84% disk) before the larger cache can grow into
it.
Follow-ups from the compile-latency audit, all measured on staging:
- The per-target build lock keyed every board through _idf_target, which
defaults unknown FQBNs to 'esp32' - so AVR, RP2040 and STM32 compiles
all took the 'esp32::esp32' key and queued behind unrelated ESP32
builds. The lock now only applies to boards that actually route to the
ESP-IDF lane (the same startswith('esp32:') predicate _run_compile
uses); everything else keys on its own FQBN.
- ESP-IDF re-validated its python environment on every cmake configure
(~1s of ~9s). The image pins that env at build time, so pass
PYTHON_DEPS_CHECKED=1 - the escape hatch idf.py uses for itself.
- _is_core_installed shelled out to 'arduino-cli core list' (~1.5s) on
every non-AVR compile from inside an async handler, blocking the whole
event loop rather than just that request. Memoized, and invalidated
where a core is installed.
New hand-written code, unique source, idle box, steady state:
ESP32-P4 ESP-IDF 28.8s -> 11.1s
ESP32-S3 ESP-IDF 39.3s -> 11.5s
ESP32-C3 Arduino 56.2s -> 21.0s
(the ESP-IDF numbers also carry the ldgen memoizer from the previous
commit; the queue waits the audit saw - 41s and 43s - were the audit's
own five concurrent builds and are what the lock-key fix targets)
Profiled a one-file ESP-IDF compile on staging: 36.7 s total, of which
ldgen.py alone was 15.2 s - consistently, on every single build. It
re-parses all 48 component linker.lf fragments with pyparsing each time
(3.8 M _parseNoCache calls; parse_fragment_file 15.6 s under cProfile),
even though its output depends only on the sdkconfig and those
fragments, never on the user's source.
pyparsing's packrat cache was tried first and made it WORSE (15.2 s ->
27.4 s), so the fix is memoizing the whole run: a wrapper hashes the
argv plus the CONTENT of every file it references and reuses the
generated linker script when they are unchanged. Verified on staging:
15.2 s -> 0.08 s on a hit, byte-identical output, and correctly missing
when the sdkconfig or any linker.lf changes (restoring the sdkconfig
hits again - it is content-addressed, not mtime-based). Any surprise
falls through to the untouched upstream script, so a build can never
break because of the cache.
New hand-written code on the ESP32-P4 lane: 36.7 s -> 11.5 s.
Running a gallery example took 51 s before a single pixel appeared.
Measured split: 28 s compile, 23 s emulator boot. Both halves had
avoidable work.
Compile: the dedup only collapsed builds still IN FLIGHT, so an
identical rebuild cost the same as a cold one (28 s cold, 27 s warm).
Successful results are now stored under the build volume keyed by the
same content hash, LRU-pruned, and served as an already-finished job so
the client's poll loop is unchanged. Gallery examples are byte-identical
for every visitor, so one build serves everyone: 32.8 s -> 0.6 s.
The key also folds in a fingerprint of the two modules that TURN a
request into build flags, so changing sdkconfig generation invalidates
every stale binary automatically instead of serving images built with
the old flags. Failed builds are never cached.
Boot: CONFIG_SPIRAM_MEMTEST defaults on and walks all 32 MB before
app_main - tens of millions of guest accesses to test RAM that is
modelled, not physical. Off for both the P4 hex and the classic
quad/octal chunks: 35M -> 20M guest instructions to LCD-READY.
End to end on staging, ESP32-P4 MIPI-DSI example: 51 s -> 11 s.
The VoCat card was a rounded rectangle - the product is a round puck
with two cat-ear bumps, a mic in each and the green LED between them
(official product photo). The P4 card was maroon; the Function-EV PCB
is black.
The merged component copied every library into ONE interleaved tree and
then exported EVERY directory that held a file. A library's private
internals therefore sat on the global -I and hijacked #includes issued by
other libraries, by the arduino-esp32 core and by ESP-IDF itself. Each
incident was patched by adding a name to _SHADOW_STD_HEADERS — chasing
symptoms, since the denylist grows with every library that ships a
generically-named internal header (three landed yesterday alone:
FastLED's platforms/stub/Arduino.h, its fl/stl/asio/http/http_parser.h,
and earlier LovyanGFX's lgfx/internal/limits.h).
Adopt arduino-cli's model instead — the resolver every other Velxio board
family already delegates to. It shares one global -I list too, but adds
exactly ONE directory per library: <lib>/src for the recursive layout,
<lib> for the flat legacy one, never a subdirectory; utility/ stays
private to its owner. arduino-esp32 does the same as an IDF component.
- Each library is copied into its own subtree of the component, so two
libraries shipping the same relative path no longer collapse into one
silently-picked copy.
- INCLUDE_DIRS is one root per library, in resolution order. Internals
stay copied and reachable through file-relative includes.
- Legacy-layout utility/ dirs go in PRIV_INCLUDE_DIRS.
- Root-level header collisions (the only shadowing still possible) are
logged the way the Arduino IDE reports them, instead of being silent.
- The transitive scan now walks only the library just copied.
One component is deliberate: a component per library would swap this
failure mode for REQUIRES-order shadowing (IDF exports component includes
as plain -I, never -isystem) plus IDF component-name collisions, and buys
no caching ccache does not already provide.
Verified against the real FastLED+U8g2 project WITH the two denylist
entries removed: green, byte-identical firmware size. The 3-library
SSD1306+GFX+BusIO corpus is green too. VELXIO_PER_LIB_ROOTS=0 restores
the legacy layout; the mode is folded into the build-variant hash so the
two can never share a configured build dir.
Validated live against the ESP32 alarm-keypad project. Three fixes over
the initial implementation:
- Active-row model: the write-pin callback only fires on OUTPUT-latch
changes (after the first pass re-driving the row LOW is a no-change
write), and dir alone misses the first pass. A matrix scan drives one
row at a time with the latch LOW, so 'the row that most recently went
active (dir=OUTPUT, or latch write LOW)' IS the scanned row.
- Idempotent install: the canvas effect re-attaches on every boards[]
mutation (each serial line); reinstalling wiped the pressed-key set
mid-scan. Same wiring now keeps the existing state dict.
- Gap-free listeners: the keypad's button-press listeners were attached
on a 300ms setTimeout per effect run, opening a listener-less window
on every re-run in which a press was lost entirely. Attach
synchronously and drop the sendSensorDetach cleanup (detach+reattach
on unrelated re-renders is exactly the wipe the idempotent install
guards against).
A 4x4 membrane keypad scan is us-timing-critical: firmware drives one
row LOW and reads the columns back within the same loop pass, far
faster than the browser<->QEMU websocket round trip, so the browser-side
matrix simulation could never register a key on ESP32 boards.
Worker: new 'matrix-keypad' sensor type (rows/cols GPIO lists, pressed
key set). _on_dir_change/_on_pin_change recompute the column levels
synchronously on the QEMU thread — the pinMode(OUTPUT) transition is
the trigger on steady-state passes, since the row's output latch stays
LOW and the digitalWrite fires no change callback. Async gpio_in writes
to keypad-owned columns are ignored so the browser's generic matrix
logic can't clobber them.
Frontend: SimulatorCanvas attaches the keypad as a matrix-keypad sensor
when every R/C pin resolves to an ESP32-family board GPIO, and streams
the pressed-key set from the element's button-press/release events.
_resolve_library_components does minutes of synchronous filesystem work
for a big library (FastLED: BFS over every header + copying thousands of
files) directly inside async compile(), starving the event loop: the
/api/compile/start response never flushed and clients timed out at 30s
while the build itself was healthy. Run it via asyncio.to_thread like
the cmake and ninja steps already do.
IDF's esp_http_server.h includes <http_parser.h> for the HTTP_GET/...
method enums. FastLED ships fl/stl/asio/http/http_parser.h and the merge
put that directory on the global -I, so the IDF include resolved to
FastLED's file and every httpd_method_t use failed with 'HTTP_GET was
not declared'. FastLED only references its copy path-qualified, so the
directory can safely stay off -I.
The arduino-mode branch used only the FIRST .ino in the request and
silently dropped every other tab, so any multi-tab sketch (the standard
Wokwi/Arduino-IDE project layout) died with 'not declared in this scope'
on globals defined in the main file. Concatenate the main sketch
(literal sketch.ino, else the first .ino) followed by the remaining
tabs alphabetically — the arduino-cli order — with #line markers so
diagnostics keep pointing at the real tab/line. Also REQUIRE
http_parser: exposing esp_http_server.h activates FastLED's fl net
server, which uses the HTTP_GET/HTTP_POST enums http_parser provides.
esp_cache.h (esp_mm) was the next one-header-at-a-time failure after
esp_http_server. Scan FastLED 3.10's esp platform tree once and add the
full candidate set (esp_mm, esp_hw_support, heap, esp_system, esp_rom,
log, app_update, bt, esp_psram, esp_pm) — every entry remains guarded on
the component existing in the IDF tree, so older IDF generations are
unaffected.
Two more merge gaps surfaced by FastLED 3.10 + U8g2 in one sketch:
- src/fl/test/ was fully dropped because 'test' matched the generic
excluded_dirs check at ANY depth, but FastLED's unity build includes
"fl/test/_build.cpp.hpp". Repo-level dirs named test/examples/docs
stay fully excluded; the same names nested inside a src-layout tree are
now header-only (fragments copied, sources still never compiled).
- FastLED's fl/net remote code includes <esp_http_server.h>, which the
Arduino IDE provides via the core's global include path. Add
esp_http_server to the merged component's guarded REQUIRES list.
FastLED's unity build (src/fl/build/extras+.cpp) includes
"extras/_build.cpp.hpp", but extras/ was in the excluded_dirs set so the
fragment was never copied and the merged-component build died with
'No such file or directory'. Treat extras/ as header-only instead: copy
.h/.hpp/.inl fragments so unity includes resolve, but keep compiling
nothing from it — extras/ traditionally holds repo-only content.
FastLED ships a host-test stub src/platforms/stub/Arduino.h. When another
merged library (e.g. U8g2) does #include <Arduino.h>, the stub resolved
first and broke the build: 'yield' was not declared in U8g2lib.cpp and
fl::string polluted String overload resolution inside the core WString.h.
Reuse the _SHADOW_STD_HEADERS mechanism so the stub's directory stays off
the global include path; FastLED reaches it via a file-relative include.
arduino-cli FQBNs may carry a menu-option suffix
(vendor:arch:board_id:Menu1=val1,...). The espidf lane previously derived
cdc_on_boot exclusively from the board id's boards.txt default, so a
generic-devkit board could never be built with Serial on the USB
Serial/JTAG CDC port.
- _fqbn_board_id_and_options(): split board id and menu options, keeping
the old last-segment behaviour for plain 3-part FQBNs.
- _arduino_board_flags(): CDCOnBoot=cdc overlays cdc_on_boot=1 on the
boards.txt base flags (mirroring <id>.menu.CDCOnBoot.cdc.build.
cdc_on_boot=1); the per-board cache stores only base flags, overrides
are applied on a copy.
- _arduino_variant(): board id extraction goes through the same helper so
a menu suffix no longer corrupts the variant lookup.
First user: the ESP32-C3-LCDkit board def pins
esp32:esp32:esp32c3:CDCOnBoot=cdc — the real kit has no UART bridge, its
only console is the USB Serial/JTAG port, and the esp32c3js engine now
models the always-attached host that HWCDC's connection gate requires
(esp32c3js v1.8.0).
A micropython example that ships only 'code:' landed its Python in
sketch.ino (the mode switch was right, the name was not - seen on the
SensAirShuttle ws2812 example). main.py / main.c / sketch.ino by mode.
The interface requires both fields but the Docker build never
type-checks the pro overlay's example data (vite build only), so a
missing field shipped and loadExample's unguarded .filter/.map threw -
the page surfaced as a sticky 404 on a registered example (four
SensAirShuttle examples on staging). Load an armless example as having
none instead; authoring still declares the fields required.
The Function-EV module always carries 32 MB AP-hex PSRAM and esp_lcd
allocates DSI framebuffers from it unconditionally - without it the
board's flagship peripheral cannot run. Explicit psram=disabled still
wins; the UI only sends the field after the user opens the
board-options modal. Validated: an optionless espidf compile now embeds
esp_psram.
The P4 gate ('until the engines validate it') can flip: esp32p4js
v0.3.3 models the AP hex PSRAM + MSPI1 command controller. The classic
SPIRAM chunk emitted S3-era symbols (SPEED_80M, MODE_QUAD/OCT) that do
not exist in the P4's Kconfig.spiram - it gets its own chunk: MODE_HEX
+ SPEED_200M (tree default, rev0-safe; 250M needs rev >= 3). Validated
on staging: an esp_lcd_ek79007 espidf build with psram=enabled boots in
the engine to streaming 1024x600 DSI frames.
The ESP32-C5 sensing devkit lands in the hosted editor (BME690 + BMM350
Bosch shuttles, 1.9" touch LCD, WiFi 6). Same contract as the other
Espressif cards: id matches the hosted BoardKind (esp-sensairshuttle,
in LAUNCH_ITEMS) so the ad vanishes where the real board registers and
stays suppressed during the embargo.
Two more Espressif boards land in the hosted editor; the OSS picker
advertises them like the ESP-EYE/LCDkit/C6 cards. Ids match the hosted
BoardKinds (esp-vocat, esp32-p4) so the ads vanish automatically where
the real boards register, and both ids are in LAUNCH_ITEMS so the
hosted build suppresses the ads during the embargo.
Also removes two accidental double commas that left holes in the ad
arrays.
Generalizes the esp_camera.h lesson into a table (esp_lcd_st77916,
esp_lcd_touch_cst816s, esp_lcd_gc9a01, esp_lcd_ek79007, esp32-camera)
and applies it in the arduino branch, the no-core pure-IDF fallback AND
the explicit language=espidf branch (issue #139) - the last one is where
the VoCat display example lives and previously died at the first
include. Validated on staging: the ST77916 colour-bars espidf example
now fetches espressif/esp_lcd_st77916 from the registry and compiles.
nginx 504s a cold ESP-IDF build long before it finishes; Starlette then
cancels the handler and the cancellation killed the build subprocess
MID-WRITE, leaving truncated .obj files in the persistent per-target
build cache - every later build of that target failed with
'ranlib: file truncated' until the cache was wiped (bitten twice on
staging the day the P4 lane landed). asyncio.shield lets the build run
to completion with a consistent cache; only the response is lost, and
the async /start + /status flow remains the right path for long builds.
_idf_target mapped only c3/c6/s3 and defaulted everything else to
'esp32', so a P4 fqbn compiled with the XTENSA toolchain and
-DARDUINO_ESP32P4_DEV. Adds p4/c5 predicates, IDF5-target membership,
0x2000 bootloader offsets, the riscv toolchain requirement, PSRAM held
off until the engines validate it, P4 CPU-freq drops (its kconfig
choices do not overlap the classic set), a rev0 sdkconfig pin for the P4
(the in-browser engine runs the only published mask ROM and v3 firmware
reads a flash descriptor rev0 never wrote), and an honest refusal for C5
arduino builds (no core exists). Validated on staging: an esp32p4
arduino sketch now produces a merged image with the riscv32 toolchain
that the esp32p4js engine accepts.
The overlay's ESP32-P4 (and the coming C5) run ONLY on their in-browser
engines: neither chip has a QEMU machine anywhere. toQemuBoardType now
throws a clear error for those families instead of booting the wrong
machine and wedging on the first register.
Overlay boards with a camera now declare builtInCamera on their def (bool,
or { maxFrameBytes } for drivers with tight frame buffers) and the canvas
CameraToggle keys off it - the hardcoded esp32-cam / xiao-esp32s3-sense
kinds keep working unchanged.
The knob kit joins the overlay board program (Espressif partner batch);
OSS builds show the shop-window card until the pro overlay registers the
real kind.
David Montero