The symbol server assumed one library layout: the content-addressed cache,
where an entry dir name IS its content hash. A host without that cache --
Velxio Desktop's sidecar, and any self-host -- has its libraries where
arduino-cli put them, under plain folder names with the version inside
library.properties, and got a 404 for every request. The editor there falls
back to the engine's embedded catalog, so `client.` on a library the user
installed himself resolves to nothing.
Both layouts are now sources, tried best-first. The only real difference is
mutability: `lib upgrade` rewrites a sketchbook folder in place and keeps its
name, so a scan cannot be keyed by that name. Sketchbook entries are keyed by
a cheap stat-only stamp (file count, total size, newest mtime) in the same
`<norm_name>@<norm_version>-<12 hex>` shape, which makes the disk cache exact
rather than merely long-lived. Cache entries keep their existing keys, so no
deployed scan is invalidated.
Where the sketchbook is: an explicit override, then the sketchbook env vars,
then arduino-cli itself (`config get directories.user`, probed once and
remembered) -- guessing ~/Arduino is wrong on the platforms whose default is
~/Documents/Arduino. A sketchbook that resolves to the cache is dropped: the
pro overlay points arduino-cli at a symlink to the cache root, and scanning
it twice under two shapes buys nothing.
Scans for a sketchbook are written to `<sketchbook>/.velxio-symbols/`, beside
`libraries/` and never inside it, so nothing Velxio writes shows up in
`arduino-cli lib list`.
The type index now keys on a fingerprint of the source's entry keys instead
of the entry count, which is exact for both layouts (an in-place upgrade
changes no count). Deployed indexes rebuild once, ~1s from the cached scans.
A manifest-scoped compile only sees the libraries the project declares, so
it dies on the first undeclared transitive dependency — a header the user
had no reason to name. The scan-all retry exists to rescue exactly that,
and when it succeeds nothing is lost. When it fails as well, we returned
the SCOPED failure, which describes a library only our own scoping made
missing.
Today a hand-written SSD1306 sketch was reported as
Adafruit_GFX.h:12:10: fatal error: Adafruit_I2CDevice.h: No such file
while the retry had already resolved BusIO and reached the real bug, an
`#include <Adafruit_SSD1306>` with no .h. The user went looking for a
missing library that was installed all along.
The retry runs in the superset environment, so its error is the sketch's
own in both compilers. Prefer it whenever it carries text; the scoped
result still stands in when it does not.
Probed against staging the same way as the previous commit: lvgl.h and
esp_lvgl_port.h both died at "No such file or directory", so a visitor
could not paste a single screen of esp-dev-kits UI code. Declared only
when the source includes the header, so no other build pays for it.
A header that lives in a registry component and is not on
_MANAGED_COMPONENT_INCLUDES dies at "No such file or directory": the
project template ships no idf_component.yml, so nothing fetches it. That
list covered five display/camera components, which left the pieces the
official Espressif board demos are actually made of unavailable —
verified by compiling a one-include probe for each against staging:
led_strip.h FAIL (the C3-LCDkit knob_panel's WS2812 driver)
iot_button.h FAIL (its BOOT/knob-press handling)
iot_knob.h FAIL (its EC11 quadrature decoder)
esp_lcd_touch_gt911 (the P4 devkit's panel) and esp_codec_dev (every
audio demo on the VoCat and the P4) are added on the same grounds.
Anyone pasting code from esp-dev-kits or esp-bsp hit this on the first
line. Detection is unchanged — the dependency is declared only when the
source actually includes the header.
Tests: ManagedComponentDetectionTests in test_espidf_pure_mode.py, incl.
negative cases (a user's own my_led_strip.h must not match).
A sketch that includes BTstackLib on a CYW43 board fails with arduino-pico's
static_assert: "This library needs Bluetooth enabled. Use the 'Tools->IP/
Bluetooth Stack' menu in the IDE to enable it." There is no such menu here, so
it reads as a setting we forgot to expose. What is actually true is simpler:
the CYW43439's Bluetooth controller is not emulated — only its WiFi side is —
so there is nothing to switch on.
Appends that, leaving the compiler's own diagnostic intact above it because the
developer still needs the real text. Deliberately a separate helper from
humanize_cli_error, which REPLACES arduino-cli's own failures; this one only
adds context under a compiler message.
Worth knowing for whoever picks up the BT work: this is the honest-failure
milestone. Making a BT sketch compile at all means flipping this board's FQBN
to ipbtstack=ipv4btcble, which swaps liblwip.a for liblwip-bt.a and costs every
sketch on the board ~71 KB flash and ~18 KB RAM — and leaves it reaching
HCI_STATE_WORKING while advertising to nobody. The mechanism is mapped in
project/pimoroni-pico-plus-2w/STATUS.md.
MicroPython over the QEMU worker stopped loading: every project failed with
"the board stopped answering while writing _vlx_main.py", the board sitting
idle at a healthy `>>>`.
The uploader shipped in 664c3581 waits for the board's end-of-execution marker
before sending the next step — that wait is the whole point, it replaced a
fixed delay that pushed 64 bytes every 150 ms. The marker never arrived. Raw
REPL answers each statement with `OK` <EOT> traceback <EOT> `>` and not one
newline in it, and the UART buffer flushed only on newline, carriage return, a
period or 256 bytes. The answer sat in the buffer; the uploader timed out
waiting for something the backend was holding.
Nothing had needed that answer before: the old uploader pasted the whole
program and never listened. EOT is a message boundary in this protocol, so the
buffer treats it as one.
Found by forcing a MicroPython project down the QEMU path with ?esp32sim=qemu —
the residual risk 664c3581 flagged and could not exercise, because velxio.dev
routes every supported ESP32 kind to the in-browser engines. Self-hosters from
this repo have no engines and always take this path, so it broke for exactly
the people the #219 fix was written for.
`WiFi.begin()` panicked the guest: `Guru Meditation Error: Core 0 panic'ed
(LoadStorePIFAddrError)`, EXCVADDR 0x60033c00. Reported against a self-hosted
dev setup, and it did not reproduce on velxio.dev — which routes every
supported ESP32 kind to the in-browser engines. The engines are not in this
repo, so anyone self-hosting from it always runs QEMU, which is where the crash
lives. It was verified away on a path the reporter cannot take.
0x60033c00 is the WiFi MAC at its APB alias, and the fork only instantiates
that device when a NIC is attached (`qemu_find_nic_info(TYPE_ESP32_WIFI)` in
hw/xtensa/esp32.c). The worker attached one only when `wifi_enabled` was true,
so a regex over the user's source decided whether the machine had a radio at
all. Reproduced against the real worker: same sketch, same firmware, only the
flag differs — true connects on 192.168.4.x, false panics exactly as reported.
Two things were wrong, and the second is why the first was fatal.
The scan was too narrow. Arduino mode read only the entry sketch while pure-IDF
mode read every file, so `#include <WiFi.h>` in a project header was judged
WiFi-less. Verified against the live backend: main sketch -> has_wifi=true, the
same include one file away -> false. Both branches read the whole project now.
The frontend also trusted a false over its own all-files scan; the two signals
are OR'd, because a stale or narrower false is the direction that hurts.
But no scan should have been load-bearing. A real ESP32 has a radio whether or
not the sketch uses it, so the emulated one does too: the NIC is attached
unconditionally on machines that model a MAC. A misread sketch now fails to
connect instead of faulting. Only those machines — the S3 machine models no
radio (hw/xtensa/esp32s3.c never looks for the NIC), so handing it one would
leave an unconsumed netdev. `wifi_enabled` still gates the host forward, which
exposes the guest's server and does belong to the sketch.
Measured before making it unconditional, on a sketch that never touches WiFi:
boot time, guest-clock-vs-real-time and container CPU unchanged over 4 paired
runs. The AP's beacon timer runs on QEMU_CLOCK_REALTIME — the reason -icount is
off for Xtensa, b0fe5b5d — but idle beacons cost nothing measurable. One host,
Linux x86-64, one sim at a time; small effects would not show.
With the radio always present, the MicroPython `network` stub loses its reason
to exist on this path: it was there because `WLAN(STA_IF)` hangs forever with no
NIC behind it. Keeping the old condition would put the stub in front of a
working device whenever the scan misread the sketch — #262, from the other
direction. It now depends only on whether the backend models a radio.
And when this class of fault does happen, the dump says "memory" and nothing
else; a week went into #260 for that reason. LoadStorePIFAddrError is always an
unmapped peripheral, so the worker now says so in the serial monitor, naming
the WiFi MAC when the address is in its page.
test_esp32_wifi_args re-implemented the arg logic in its own `_simulate_args`,
so it would have passed unchanged through this entire bug — the same trap
1800f572 called out for the detector. The decision is now `wifi_nic_arg()` and
the test calls it.
Verified end to end against the QEMU worker: the multi-file WiFi project
compiles with has_wifi=true and connects, and a WiFi-less sketch still boots and
runs with the radio attached.
arduino-esp32 3.x keeps most of its requires PRIVATE, so a sketch that
includes esp_lcd_panel_io.h dies with 'No such file or directory' -
while the SAME header resolves fine as soon as any user library is
present, because the merged user-libs component requires esp_lcd. That
made the failure look random.
Detect the IDF-tree components a sketch includes and add them to main's
REQUIRES, the same shape as the registry managed-component table next to
it. Guarded on the component existing in the IDF tree, so older IDF
generations are unaffected.
Found writing the ESP32-S3-EYE camera example, and it matters beyond
that board: esp_lcd, esp_cache, esp_ldo_regulator and esp_psram are
exactly what an S3/P4 sketch reaches for once it drives a panel.
Measured on staging: 46 per-variant build trees held 25 GB - each one a
full copy of the same ESP-IDF component objects - while ccache, the
cache that serves EVERY target and variant, was capped at 8 GB and
permanently full: 99.97% of its limit, 153 evictions, 47% lifetime hit
rate. Objects were being thrown out before they could be reused, so
builds recompiled work they had already done.
Variant cap 12 -> 6 and ccache 8G -> 20G. The asymmetry is deliberate:
an evicted variant dir rebuilds quickly FROM ccache, whereas a ccache
miss recompiles from source. Pruning the cold variants freed 8 GB on
the staging box (88% -> 84% disk) before the larger cache can grow into
it.
Follow-ups from the compile-latency audit, all measured on staging:
- The per-target build lock keyed every board through _idf_target, which
defaults unknown FQBNs to 'esp32' - so AVR, RP2040 and STM32 compiles
all took the 'esp32::esp32' key and queued behind unrelated ESP32
builds. The lock now only applies to boards that actually route to the
ESP-IDF lane (the same startswith('esp32:') predicate _run_compile
uses); everything else keys on its own FQBN.
- ESP-IDF re-validated its python environment on every cmake configure
(~1s of ~9s). The image pins that env at build time, so pass
PYTHON_DEPS_CHECKED=1 - the escape hatch idf.py uses for itself.
- _is_core_installed shelled out to 'arduino-cli core list' (~1.5s) on
every non-AVR compile from inside an async handler, blocking the whole
event loop rather than just that request. Memoized, and invalidated
where a core is installed.
New hand-written code, unique source, idle box, steady state:
ESP32-P4 ESP-IDF 28.8s -> 11.1s
ESP32-S3 ESP-IDF 39.3s -> 11.5s
ESP32-C3 Arduino 56.2s -> 21.0s
(the ESP-IDF numbers also carry the ldgen memoizer from the previous
commit; the queue waits the audit saw - 41s and 43s - were the audit's
own five concurrent builds and are what the lock-key fix targets)
Running a gallery example took 51 s before a single pixel appeared.
Measured split: 28 s compile, 23 s emulator boot. Both halves had
avoidable work.
Compile: the dedup only collapsed builds still IN FLIGHT, so an
identical rebuild cost the same as a cold one (28 s cold, 27 s warm).
Successful results are now stored under the build volume keyed by the
same content hash, LRU-pruned, and served as an already-finished job so
the client's poll loop is unchanged. Gallery examples are byte-identical
for every visitor, so one build serves everyone: 32.8 s -> 0.6 s.
The key also folds in a fingerprint of the two modules that TURN a
request into build flags, so changing sdkconfig generation invalidates
every stale binary automatically instead of serving images built with
the old flags. Failed builds are never cached.
Boot: CONFIG_SPIRAM_MEMTEST defaults on and walks all 32 MB before
app_main - tens of millions of guest accesses to test RAM that is
modelled, not physical. Off for both the P4 hex and the classic
quad/octal chunks: 35M -> 20M guest instructions to LCD-READY.
End to end on staging, ESP32-P4 MIPI-DSI example: 51 s -> 11 s.
The merged component copied every library into ONE interleaved tree and
then exported EVERY directory that held a file. A library's private
internals therefore sat on the global -I and hijacked #includes issued by
other libraries, by the arduino-esp32 core and by ESP-IDF itself. Each
incident was patched by adding a name to _SHADOW_STD_HEADERS — chasing
symptoms, since the denylist grows with every library that ships a
generically-named internal header (three landed yesterday alone:
FastLED's platforms/stub/Arduino.h, its fl/stl/asio/http/http_parser.h,
and earlier LovyanGFX's lgfx/internal/limits.h).
Adopt arduino-cli's model instead — the resolver every other Velxio board
family already delegates to. It shares one global -I list too, but adds
exactly ONE directory per library: <lib>/src for the recursive layout,
<lib> for the flat legacy one, never a subdirectory; utility/ stays
private to its owner. arduino-esp32 does the same as an IDF component.
- Each library is copied into its own subtree of the component, so two
libraries shipping the same relative path no longer collapse into one
silently-picked copy.
- INCLUDE_DIRS is one root per library, in resolution order. Internals
stay copied and reachable through file-relative includes.
- Legacy-layout utility/ dirs go in PRIV_INCLUDE_DIRS.
- Root-level header collisions (the only shadowing still possible) are
logged the way the Arduino IDE reports them, instead of being silent.
- The transitive scan now walks only the library just copied.
One component is deliberate: a component per library would swap this
failure mode for REQUIRES-order shadowing (IDF exports component includes
as plain -I, never -isystem) plus IDF component-name collisions, and buys
no caching ccache does not already provide.
Verified against the real FastLED+U8g2 project WITH the two denylist
entries removed: green, byte-identical firmware size. The 3-library
SSD1306+GFX+BusIO corpus is green too. VELXIO_PER_LIB_ROOTS=0 restores
the legacy layout; the mode is folded into the build-variant hash so the
two can never share a configured build dir.
Validated live against the ESP32 alarm-keypad project. Three fixes over
the initial implementation:
- Active-row model: the write-pin callback only fires on OUTPUT-latch
changes (after the first pass re-driving the row LOW is a no-change
write), and dir alone misses the first pass. A matrix scan drives one
row at a time with the latch LOW, so 'the row that most recently went
active (dir=OUTPUT, or latch write LOW)' IS the scanned row.
- Idempotent install: the canvas effect re-attaches on every boards[]
mutation (each serial line); reinstalling wiped the pressed-key set
mid-scan. Same wiring now keeps the existing state dict.
- Gap-free listeners: the keypad's button-press listeners were attached
on a 300ms setTimeout per effect run, opening a listener-less window
on every re-run in which a press was lost entirely. Attach
synchronously and drop the sendSensorDetach cleanup (detach+reattach
on unrelated re-renders is exactly the wipe the idempotent install
guards against).
A 4x4 membrane keypad scan is us-timing-critical: firmware drives one
row LOW and reads the columns back within the same loop pass, far
faster than the browser<->QEMU websocket round trip, so the browser-side
matrix simulation could never register a key on ESP32 boards.
Worker: new 'matrix-keypad' sensor type (rows/cols GPIO lists, pressed
key set). _on_dir_change/_on_pin_change recompute the column levels
synchronously on the QEMU thread — the pinMode(OUTPUT) transition is
the trigger on steady-state passes, since the row's output latch stays
LOW and the digitalWrite fires no change callback. Async gpio_in writes
to keypad-owned columns are ignored so the browser's generic matrix
logic can't clobber them.
Frontend: SimulatorCanvas attaches the keypad as a matrix-keypad sensor
when every R/C pin resolves to an ESP32-family board GPIO, and streams
the pressed-key set from the element's button-press/release events.
_resolve_library_components does minutes of synchronous filesystem work
for a big library (FastLED: BFS over every header + copying thousands of
files) directly inside async compile(), starving the event loop: the
/api/compile/start response never flushed and clients timed out at 30s
while the build itself was healthy. Run it via asyncio.to_thread like
the cmake and ninja steps already do.
IDF's esp_http_server.h includes <http_parser.h> for the HTTP_GET/...
method enums. FastLED ships fl/stl/asio/http/http_parser.h and the merge
put that directory on the global -I, so the IDF include resolved to
FastLED's file and every httpd_method_t use failed with 'HTTP_GET was
not declared'. FastLED only references its copy path-qualified, so the
directory can safely stay off -I.
The arduino-mode branch used only the FIRST .ino in the request and
silently dropped every other tab, so any multi-tab sketch (the standard
Wokwi/Arduino-IDE project layout) died with 'not declared in this scope'
on globals defined in the main file. Concatenate the main sketch
(literal sketch.ino, else the first .ino) followed by the remaining
tabs alphabetically — the arduino-cli order — with #line markers so
diagnostics keep pointing at the real tab/line. Also REQUIRE
http_parser: exposing esp_http_server.h activates FastLED's fl net
server, which uses the HTTP_GET/HTTP_POST enums http_parser provides.
esp_cache.h (esp_mm) was the next one-header-at-a-time failure after
esp_http_server. Scan FastLED 3.10's esp platform tree once and add the
full candidate set (esp_mm, esp_hw_support, heap, esp_system, esp_rom,
log, app_update, bt, esp_psram, esp_pm) — every entry remains guarded on
the component existing in the IDF tree, so older IDF generations are
unaffected.
Two more merge gaps surfaced by FastLED 3.10 + U8g2 in one sketch:
- src/fl/test/ was fully dropped because 'test' matched the generic
excluded_dirs check at ANY depth, but FastLED's unity build includes
"fl/test/_build.cpp.hpp". Repo-level dirs named test/examples/docs
stay fully excluded; the same names nested inside a src-layout tree are
now header-only (fragments copied, sources still never compiled).
- FastLED's fl/net remote code includes <esp_http_server.h>, which the
Arduino IDE provides via the core's global include path. Add
esp_http_server to the merged component's guarded REQUIRES list.
FastLED's unity build (src/fl/build/extras+.cpp) includes
"extras/_build.cpp.hpp", but extras/ was in the excluded_dirs set so the
fragment was never copied and the merged-component build died with
'No such file or directory'. Treat extras/ as header-only instead: copy
.h/.hpp/.inl fragments so unity includes resolve, but keep compiling
nothing from it — extras/ traditionally holds repo-only content.
FastLED ships a host-test stub src/platforms/stub/Arduino.h. When another
merged library (e.g. U8g2) does #include <Arduino.h>, the stub resolved
first and broke the build: 'yield' was not declared in U8g2lib.cpp and
fl::string polluted String overload resolution inside the core WString.h.
Reuse the _SHADOW_STD_HEADERS mechanism so the stub's directory stays off
the global include path; FastLED reaches it via a file-relative include.
arduino-cli FQBNs may carry a menu-option suffix
(vendor:arch:board_id:Menu1=val1,...). The espidf lane previously derived
cdc_on_boot exclusively from the board id's boards.txt default, so a
generic-devkit board could never be built with Serial on the USB
Serial/JTAG CDC port.
- _fqbn_board_id_and_options(): split board id and menu options, keeping
the old last-segment behaviour for plain 3-part FQBNs.
- _arduino_board_flags(): CDCOnBoot=cdc overlays cdc_on_boot=1 on the
boards.txt base flags (mirroring <id>.menu.CDCOnBoot.cdc.build.
cdc_on_boot=1); the per-board cache stores only base flags, overrides
are applied on a copy.
- _arduino_variant(): board id extraction goes through the same helper so
a menu suffix no longer corrupts the variant lookup.
First user: the ESP32-C3-LCDkit board def pins
esp32:esp32:esp32c3:CDCOnBoot=cdc — the real kit has no UART bridge, its
only console is the USB Serial/JTAG port, and the esp32c3js engine now
models the always-attached host that HWCDC's connection gate requires
(esp32c3js v1.8.0).
The Function-EV module always carries 32 MB AP-hex PSRAM and esp_lcd
allocates DSI framebuffers from it unconditionally - without it the
board's flagship peripheral cannot run. Explicit psram=disabled still
wins; the UI only sends the field after the user opens the
board-options modal. Validated: an optionless espidf compile now embeds
esp_psram.
The P4 gate ('until the engines validate it') can flip: esp32p4js
v0.3.3 models the AP hex PSRAM + MSPI1 command controller. The classic
SPIRAM chunk emitted S3-era symbols (SPEED_80M, MODE_QUAD/OCT) that do
not exist in the P4's Kconfig.spiram - it gets its own chunk: MODE_HEX
+ SPEED_200M (tree default, rev0-safe; 250M needs rev >= 3). Validated
on staging: an esp_lcd_ek79007 espidf build with psram=enabled boots in
the engine to streaming 1024x600 DSI frames.
Generalizes the esp_camera.h lesson into a table (esp_lcd_st77916,
esp_lcd_touch_cst816s, esp_lcd_gc9a01, esp_lcd_ek79007, esp32-camera)
and applies it in the arduino branch, the no-core pure-IDF fallback AND
the explicit language=espidf branch (issue #139) - the last one is where
the VoCat display example lives and previously died at the first
include. Validated on staging: the ST77916 colour-bars espidf example
now fetches espressif/esp_lcd_st77916 from the registry and compiles.
nginx 504s a cold ESP-IDF build long before it finishes; Starlette then
cancels the handler and the cancellation killed the build subprocess
MID-WRITE, leaving truncated .obj files in the persistent per-target
build cache - every later build of that target failed with
'ranlib: file truncated' until the cache was wiped (bitten twice on
staging the day the P4 lane landed). asyncio.shield lets the build run
to completion with a consistent cache; only the response is lost, and
the async /start + /status flow remains the right path for long builds.
_idf_target mapped only c3/c6/s3 and defaulted everything else to
'esp32', so a P4 fqbn compiled with the XTENSA toolchain and
-DARDUINO_ESP32P4_DEV. Adds p4/c5 predicates, IDF5-target membership,
0x2000 bootloader offsets, the riscv toolchain requirement, PSRAM held
off until the engines validate it, P4 CPU-freq drops (its kconfig
choices do not overlap the classic set), a rev0 sdkconfig pin for the P4
(the in-browser engine runs the only published mask ROM and v3 firmware
reads a flash descriptor rev0 never wrote), and an honest refusal for C5
arduino builds (no core exists). Validated on staging: an esp32p4
arduino sketch now produces a merged image with the riscv32 toolchain
that the esp32p4js engine accepts.
One Semaphore(2) gated every compile, so while two ESP32 builds ran (cmake +
ninja, minutes on a cold cache) an Arduino Uno blink sat 'compiling' for as
long as they took. Two lanes now: heavy (ESP-IDF, still 2 — the VPS load
constraint is real) and light (arduino-cli boards, 3), plus a console line
when a job is queued so the wait is visible instead of looking like a hung
build. The per-target lock that keeps concurrent builds out of one ESP-IDF
build dir is unchanged.
A MicroPython library in Velxio IS a .py file in the project, copied onto
the board's filesystem before main.py runs. That mechanism worked, but
getting the file meant hunting GitHub by hand — the gap issue #214 is
about. Now the Library Manager fills it, in both halves:
Backend: /api/micropython-libs proxies the official micropython-lib
package index (the same one `mip` uses on real hardware). /search filters
the index; /fetch resolves a package plus its declared deps into .py
files. The proxy is what makes this safe to expose: only the official
index is reachable (the name must exist in index.json, files come from its
content-addressed store), source only (.py, never .mpy bytecode), path-
safe, and capped at 24 files / 256 KB per install. In-process TTL cache
over the index.
Frontend: the same Library Manager modal, mode-aware. On a MicroPython
board the search hits the index and "Add to project" writes the package's
files into the board's workspace (upsert — re-adding is the upgrade path),
visible in the explorer and travelling with the project. Before typing it
shows featured picks (ssd1306, umqtt.*, urequests) and the modules already
frozen into the firmware (neopixel, dht, onewire, ds18x20, machine,
network) labeled "built into the firmware", so searching for one answers
"just import it" instead of an empty list.
The file-materialization prelude now creates package directories: umqtt
ships as umqtt/simple.py, and MicroPython's open() does not mkdir, so the
prelude died with ENOENT on any packaged module.
Also ships a gallery example ("ESP32 + MicroPython: use an external
library") that documents the file-is-the-library mechanism with the
ssd1306 driver on the wired OLED.
arduino-cli reports errors as plain text or, with --format json, as a JSON
object on stderr. We forwarded that blob verbatim, so on a fresh install
with no library index yet (and no way through the firewall/proxy to
downloads.arduino.cc) the in-app agent told the user "the library server
returns a JSON error" ten times in a row.
humanize_cli_error() maps the known shapes — index download failed,
library zip download failed, library not found — to a single sentence
that says what to check, and keeps the raw text under raw_error. Applied
to search_libraries (rc != 0 and non-JSON stdout) and install_library.
Tests use the real arduino-cli 1.0.4 output captured offline.
ESP-IDF compiles with -ggdb, and for debug builds ccache folds the cwd into
every cache key by default — CCACHE_BASEDIR does not exempt it (verified with
CCACHE_DEBUG on 4.11: 'Hashing CWD /var/lib/velxio-build/esp32s3/v_<hash>/
project/build'). Each build variant lives in its own dir, so every variant had
a private key space: eight esp32 variants of the SAME board each recompiled
all ~2100 IDF objects from scratch, the cache filled with dead per-directory
entries, and a cold Cardputer/S3 build (2263 steps) overran the 600s ninja
timeout on a busy box. CCACHE_NOHASHDIR=1 makes variants of the same
board/sdkconfig share objects; a cold variant becomes mostly cache hits. Only
DW_AT_comp_dir in .debug_info of a reused object can differ, which never
reaches the flashed .bin.
The timeout paths were silent server-side; they now log the build dir and how
many ninja steps completed, and the user-facing message says the partial
build is kept and a retry resumes.
Measured against the user corpus, the by-resolved-type long tail was
losing what the old catch-all happened to include: for `Adafruit_SH1106G
display(...); display.setTextSize(` the class itself declares one method
(begin) and everything people call lives four levels up (Adafruit_SH110X ->
Adafruit_GrayOLED -> Adafruit_GFX -> Print). Same story for MCUFRIEND_kbv,
Adafruit_NeoMatrix and every GFX-derived display driver.
Three changes:
- The scanner records `bases` on class symbols (access/virtual keywords
dropped, `ns::Base` -> Base, template args cut). 2,529 classes across the
production cache now carry their bases.
- /type/{name} walks the chain breadth-first through the type index (bases
routinely live in a different library), re-owns inherited members to the
requested type so the editor sees one flat class, tags them with
`inheritedFrom`, and lists the chain in `bases` -- including bases NOT in
the cache (Print, Stream, or an embedded one like Adafruit_GFX) so the
host can supply those itself. Depth capped at 8. Adafruit_SH1106G goes
from 1 member to 64.
- `void a(int), b(), c(char);` declares three methods; the extractor took
only the first. TFT_eSPI declares setTextColor/setTextSize and
setCursor/getCursorX/... this way. Split on top-level commas after a
complete `name(args)` group. TFT_eSPI 118 -> 173 members.
Rescan of the 1,125-entry production cache: +274 member symbols, 15
libraries gain, none lose.
The completion-safe long-tail path. Feeding whole library symbol sets into
the editor's candidate pool was measured to make ranking worse (U8g2 alone
injects ~250 module-level class names; ArduinoJson dozens of ARDUINOJSON_*
macros). What does help is answering `client.` once tree-sitter has resolved
`client` to a declared type the embedded catalog does not know -- and for
that the engine needs "the members of type T", not "the symbols of library
L". This endpoint returns exactly that: symbols whose owner is the requested
class, from whichever cached library declares it, in the same payload shape
as /symbols/ so the engine's RemoteSymbolCache stores it unchanged.
Resolution: a class-name -> cache-entry index built once over the library
cache (each per-entry scan is already disk-cached), persisted under
.symbols/_types.json (3,818 classes / 199 KB over the 1,125-entry
production cache; ~1 s to rebuild from cached scans, ~60 s cold), refreshed
after a few hours or when the cache grows. Many classes are declared by more
than one library -- the real one plus forks and bundles vendoring a copy
(PubSubClient inside "ESP8266 Microgear", NTPClient inside "ESP8266 Weather
Station", AccelStepper inside "Arduino Learning Board") -- so the pick is:
the library named after the class, else the one declaring the most members
of it (vendored copies are trimmed), else alphabetical.
Generic, not pro-specific: any host with a library cache can use it.
_scan_text applied `public:` / `private:` / `protected:` only inside
handle_statement, which the `{` branch skipped whenever the statement opened
a nested enum, struct or class. So `public:\n typedef enum { ... }` -- the
first thing in AccelStepper's class body, and a common idiom -- took the enum
branch, the label was never seen, the class stayed in its default private
access, and every method after the enum was dropped. AccelStepper shipped 4
of its public methods; DallasTemperature, SoftWire, MFRC522_I2C, RTCZero
shipped none.
Labels are now consumed before the branch. Rescanned all 1,125 entries of
the production library cache: 82 libraries gain members (+1,532 total,
DallasTemperature 0 -> 57), and the 20 that "lose" one or two were leaking
private/protected members past the old check (PMS::loop is under private:,
hd44780::waitReady under protected:) -- precision, not regression.
`_prepare_persistent_project_dir` keys the persistent build dir on
(idf_target, variant), but the lock was keyed on the FQBN. Several FQBNs
resolve to one variant — arduino-esp32's boards.txt gives both `esp32` and
`esp32cam` build.variant=esp32 — so those two were free to build in the SAME
directory at the same time.
Measured against staging, compiling both concurrently:
esp32 285504 bytes sha f65ca909… contains its own marker
esp32cam 285504 bytes sha f65ca909… contains the OTHER sketch's marker
Byte-identical firmware, and the esp32cam caller got the esp32 sketch's
binary. In the product that is two gallery tabs open at once: the first tab
prints the second example's serial log and does nothing itself. It made
esp32-pwm-led-rgb look broken (dark LED, every SPICE node 0 V) when the
example is fine — it was running someone else's program.
Key the lock on the same identity as the directory. Boards with genuinely
separate build dirs still compile in parallel up to the semaphore cap; an
unresolvable FQBN falls back to the default pair, which is the safe
direction (serialising more only costs throughput).
Also in this commit: battery-voltage-monitor ships the 9 V battery its own
description promises. The divider's top end was wired to nothing, so A0 read
0 V and the sketch printed "Battery: 0.0 V / WARNING: Battery low!" forever.
The custom sketch.ino.cpp path never ran arduino-cli's ctags prototype
pass, so the universal Arduino idiom (helpers defined after loop()) died
with "'foo' was not declared in this scope" on every ESP32 build. A
conservative generator (skips templates, default args, methods,
function-pointer params) injects prototypes before the first definition
with a #line directive so error line numbers stay put. 8 unit tests.
IDF 5.5 renamed ESP_TASK_WDT to the EN/INIT pair; the v4.4 template's =n
became EN=n, compiling the whole implementation out — every sketch using
esp_task_wdt_add/reset (Firebase's client does internally) failed at
link. The v5 fixup now emits EN=y INIT=n: the API links, the watchdog
never auto-starts under emulation. v4.4 semantics untouched.
GET /api/intellisense/symbols/{Name@Version} scans the content-addressed
library cache entry's headers (dependency-free C++ scanner) into an editor
symbol table, disk-cached per entry and served with long cache headers.
Plain OSS hosts without a lib cache 404 and the editor degrades gracefully.
C3/S3/C6 default to CONFIG_BT_BLE_50_FEATURES_SUPPORTED with 4.2 OFF, so
arduino-esp32's Bluedroid BLE library (esp_ble_gap_config_adv_data_raw,
esp_ble_gap_start_advertising) died at link with undefined references —
esp32c3-ble-advertise never compiled. Enabling 42+50 together matches
arduino-esp32's own per-target sdkconfigs; on classic ESP32 these flags
are already the Kconfig defaults. Found by the nightly gallery smoke.
The dead-branch pruner marked any assumed-live branch as 'taken', so the
#else of an UNKNOWN condition was skipped. GxEPD2 guards its GFX base
class with '#if ENABLE_GxEPD2_GFX (default 0) ... #else #include
<Adafruit_GFX.h>' — the pruned #else silently dropped the Adafruit_GFX
dependency and every GxEPD2 e-paper example failed with 'fatal error:
Adafruit_GFX.h: No such file'. Found by the nightly gallery compile smoke.
'taken' is now set only by PROVABLY-true conditions (all identifiers in
the known-true set), keeping the 2026-08-03 ESPAsyncWebServer behaviour:
provably-dead branches still prune, provably-true branches still kill
their #else.
The ESP32 examples now default to the in-browser JS engines, whose lwIP
serves the sketch's HTTP server from the simulation tab itself. Opening
the IoT gateway in a new tab backgrounds that tab, the browser throttles
its timers, and the in-chip server can't answer before the proxy's
timeout — the exact failure the Pico W iframe path already solved.
- WifiStatus grows an inBrowser flag (set by the overlay's JS bridges);
the serial-monitor link and the WiFi badge now open the in-app iframe
for any board that carries it, not just pi-pico-w.
- /api/gateway polls ~1s before answering 404: the net bridge dials on
'got_ip', which races the sketch printing its URL, so a fast click
used to land before the bridge registered.
The main component had them; the merged user-libraries component did not, so
a warning INSIDE a vendor library was still fatal. M5Stack's MahonyAHRS.cpp
(fast inverse square root, a float read through a long*) fails under
-Werror=uninitialized, which made installing the vendor's own library kill
every build using it. Library code the user cannot edit must not stop their
build; the diagnostics still print as warnings.
The library's fast inverse square root reads a float through a long*, GCC
flags y as used uninitialized, and -Werror turns a vendor library the user
cannot edit into a failed build. Demoted to a warning like its siblings.
Two of M5Stack's own published Cardputer examples fail to build here and
compile fine in the Arduino IDE: SSHClient passes an int to a %lu, and
mic_wav_record has a local GCC thinks may be used uninitialized. ESP-IDF
promotes both categories to errors, which is right for IDF itself and wrong
for a sketch a user pasted from the vendor's repository — the emulator looks
broken for code that is not.
format and maybe-uninitialized join the existing relaxation list, which
already carries comment, parentheses, sign-compare and the rest for exactly
this reason. They stay warnings, so the diagnostic is still printed; -Wall is
untouched.
useProjectStore.currentExampleId — stamped by loadExample, dropped the
moment a real project (or a blank workspace) takes over. The compile
request and the run event carry it as example_id, analytics only, so
'which examples get compiled and run most' becomes answerable.
Several boards share one FQBN (the Pimoroni RP2350 trio all compile as
rpipico2), so FQBN-keyed metrics can't tell them apart. The compile
request and the run event now carry board_kind — analytics only, rides
into the metric hook's extra.
suppressOnlineOnlyAds(): overlay seam so the hosted build can silence
'available in the online editor' ad cards for items it manages — an
embargoed item must vanish, not advertise the site the user is on.
Pure OSS builds never call it; ads behave exactly as before.
Declared inside the has_arduino branch, so every successful pure
ESP-IDF (language=espidf) build crashed with UnboundLocalError when the
success return read it. Regression from today's manifest auto-migration
change; the Arduino paths were live-verified but the espidf mode was
not.
a2nr
David Montero