A header that lives in a registry component and is not on
_MANAGED_COMPONENT_INCLUDES dies at "No such file or directory": the
project template ships no idf_component.yml, so nothing fetches it. That
list covered five display/camera components, which left the pieces the
official Espressif board demos are actually made of unavailable —
verified by compiling a one-include probe for each against staging:
led_strip.h FAIL (the C3-LCDkit knob_panel's WS2812 driver)
iot_button.h FAIL (its BOOT/knob-press handling)
iot_knob.h FAIL (its EC11 quadrature decoder)
esp_lcd_touch_gt911 (the P4 devkit's panel) and esp_codec_dev (every
audio demo on the VoCat and the P4) are added on the same grounds.
Anyone pasting code from esp-dev-kits or esp-bsp hit this on the first
line. Detection is unchanged — the dependency is declared only when the
source actually includes the header.
Tests: ManagedComponentDetectionTests in test_espidf_pure_mode.py, incl.
negative cases (a user's own my_led_strip.h must not match).
`WiFi.begin()` panicked the guest: `Guru Meditation Error: Core 0 panic'ed
(LoadStorePIFAddrError)`, EXCVADDR 0x60033c00. Reported against a self-hosted
dev setup, and it did not reproduce on velxio.dev — which routes every
supported ESP32 kind to the in-browser engines. The engines are not in this
repo, so anyone self-hosting from it always runs QEMU, which is where the crash
lives. It was verified away on a path the reporter cannot take.
0x60033c00 is the WiFi MAC at its APB alias, and the fork only instantiates
that device when a NIC is attached (`qemu_find_nic_info(TYPE_ESP32_WIFI)` in
hw/xtensa/esp32.c). The worker attached one only when `wifi_enabled` was true,
so a regex over the user's source decided whether the machine had a radio at
all. Reproduced against the real worker: same sketch, same firmware, only the
flag differs — true connects on 192.168.4.x, false panics exactly as reported.
Two things were wrong, and the second is why the first was fatal.
The scan was too narrow. Arduino mode read only the entry sketch while pure-IDF
mode read every file, so `#include <WiFi.h>` in a project header was judged
WiFi-less. Verified against the live backend: main sketch -> has_wifi=true, the
same include one file away -> false. Both branches read the whole project now.
The frontend also trusted a false over its own all-files scan; the two signals
are OR'd, because a stale or narrower false is the direction that hurts.
But no scan should have been load-bearing. A real ESP32 has a radio whether or
not the sketch uses it, so the emulated one does too: the NIC is attached
unconditionally on machines that model a MAC. A misread sketch now fails to
connect instead of faulting. Only those machines — the S3 machine models no
radio (hw/xtensa/esp32s3.c never looks for the NIC), so handing it one would
leave an unconsumed netdev. `wifi_enabled` still gates the host forward, which
exposes the guest's server and does belong to the sketch.
Measured before making it unconditional, on a sketch that never touches WiFi:
boot time, guest-clock-vs-real-time and container CPU unchanged over 4 paired
runs. The AP's beacon timer runs on QEMU_CLOCK_REALTIME — the reason -icount is
off for Xtensa, b0fe5b5d — but idle beacons cost nothing measurable. One host,
Linux x86-64, one sim at a time; small effects would not show.
With the radio always present, the MicroPython `network` stub loses its reason
to exist on this path: it was there because `WLAN(STA_IF)` hangs forever with no
NIC behind it. Keeping the old condition would put the stub in front of a
working device whenever the scan misread the sketch — #262, from the other
direction. It now depends only on whether the backend models a radio.
And when this class of fault does happen, the dump says "memory" and nothing
else; a week went into #260 for that reason. LoadStorePIFAddrError is always an
unmapped peripheral, so the worker now says so in the serial monitor, naming
the WiFi MAC when the address is in its page.
test_esp32_wifi_args re-implemented the arg logic in its own `_simulate_args`,
so it would have passed unchanged through this entire bug — the same trap
1800f572 called out for the detector. The decision is now `wifi_nic_arg()` and
the test calls it.
Verified end to end against the QEMU worker: the multi-file WiFi project
compiles with has_wifi=true and connects, and a WiFi-less sketch still boots and
runs with the radio attached.
arduino-esp32 3.x keeps most of its requires PRIVATE, so a sketch that
includes esp_lcd_panel_io.h dies with 'No such file or directory' -
while the SAME header resolves fine as soon as any user library is
present, because the merged user-libs component requires esp_lcd. That
made the failure look random.
Detect the IDF-tree components a sketch includes and add them to main's
REQUIRES, the same shape as the registry managed-component table next to
it. Guarded on the component existing in the IDF tree, so older IDF
generations are unaffected.
Found writing the ESP32-S3-EYE camera example, and it matters beyond
that board: esp_lcd, esp_cache, esp_ldo_regulator and esp_psram are
exactly what an S3/P4 sketch reaches for once it drives a panel.
Measured on staging: 46 per-variant build trees held 25 GB - each one a
full copy of the same ESP-IDF component objects - while ccache, the
cache that serves EVERY target and variant, was capped at 8 GB and
permanently full: 99.97% of its limit, 153 evictions, 47% lifetime hit
rate. Objects were being thrown out before they could be reused, so
builds recompiled work they had already done.
Variant cap 12 -> 6 and ccache 8G -> 20G. The asymmetry is deliberate:
an evicted variant dir rebuilds quickly FROM ccache, whereas a ccache
miss recompiles from source. Pruning the cold variants freed 8 GB on
the staging box (88% -> 84% disk) before the larger cache can grow into
it.
Follow-ups from the compile-latency audit, all measured on staging:
- The per-target build lock keyed every board through _idf_target, which
defaults unknown FQBNs to 'esp32' - so AVR, RP2040 and STM32 compiles
all took the 'esp32::esp32' key and queued behind unrelated ESP32
builds. The lock now only applies to boards that actually route to the
ESP-IDF lane (the same startswith('esp32:') predicate _run_compile
uses); everything else keys on its own FQBN.
- ESP-IDF re-validated its python environment on every cmake configure
(~1s of ~9s). The image pins that env at build time, so pass
PYTHON_DEPS_CHECKED=1 - the escape hatch idf.py uses for itself.
- _is_core_installed shelled out to 'arduino-cli core list' (~1.5s) on
every non-AVR compile from inside an async handler, blocking the whole
event loop rather than just that request. Memoized, and invalidated
where a core is installed.
New hand-written code, unique source, idle box, steady state:
ESP32-P4 ESP-IDF 28.8s -> 11.1s
ESP32-S3 ESP-IDF 39.3s -> 11.5s
ESP32-C3 Arduino 56.2s -> 21.0s
(the ESP-IDF numbers also carry the ldgen memoizer from the previous
commit; the queue waits the audit saw - 41s and 43s - were the audit's
own five concurrent builds and are what the lock-key fix targets)
Running a gallery example took 51 s before a single pixel appeared.
Measured split: 28 s compile, 23 s emulator boot. Both halves had
avoidable work.
Compile: the dedup only collapsed builds still IN FLIGHT, so an
identical rebuild cost the same as a cold one (28 s cold, 27 s warm).
Successful results are now stored under the build volume keyed by the
same content hash, LRU-pruned, and served as an already-finished job so
the client's poll loop is unchanged. Gallery examples are byte-identical
for every visitor, so one build serves everyone: 32.8 s -> 0.6 s.
The key also folds in a fingerprint of the two modules that TURN a
request into build flags, so changing sdkconfig generation invalidates
every stale binary automatically instead of serving images built with
the old flags. Failed builds are never cached.
Boot: CONFIG_SPIRAM_MEMTEST defaults on and walks all 32 MB before
app_main - tens of millions of guest accesses to test RAM that is
modelled, not physical. Off for both the P4 hex and the classic
quad/octal chunks: 35M -> 20M guest instructions to LCD-READY.
End to end on staging, ESP32-P4 MIPI-DSI example: 51 s -> 11 s.
The merged component copied every library into ONE interleaved tree and
then exported EVERY directory that held a file. A library's private
internals therefore sat on the global -I and hijacked #includes issued by
other libraries, by the arduino-esp32 core and by ESP-IDF itself. Each
incident was patched by adding a name to _SHADOW_STD_HEADERS — chasing
symptoms, since the denylist grows with every library that ships a
generically-named internal header (three landed yesterday alone:
FastLED's platforms/stub/Arduino.h, its fl/stl/asio/http/http_parser.h,
and earlier LovyanGFX's lgfx/internal/limits.h).
Adopt arduino-cli's model instead — the resolver every other Velxio board
family already delegates to. It shares one global -I list too, but adds
exactly ONE directory per library: <lib>/src for the recursive layout,
<lib> for the flat legacy one, never a subdirectory; utility/ stays
private to its owner. arduino-esp32 does the same as an IDF component.
- Each library is copied into its own subtree of the component, so two
libraries shipping the same relative path no longer collapse into one
silently-picked copy.
- INCLUDE_DIRS is one root per library, in resolution order. Internals
stay copied and reachable through file-relative includes.
- Legacy-layout utility/ dirs go in PRIV_INCLUDE_DIRS.
- Root-level header collisions (the only shadowing still possible) are
logged the way the Arduino IDE reports them, instead of being silent.
- The transitive scan now walks only the library just copied.
One component is deliberate: a component per library would swap this
failure mode for REQUIRES-order shadowing (IDF exports component includes
as plain -I, never -isystem) plus IDF component-name collisions, and buys
no caching ccache does not already provide.
Verified against the real FastLED+U8g2 project WITH the two denylist
entries removed: green, byte-identical firmware size. The 3-library
SSD1306+GFX+BusIO corpus is green too. VELXIO_PER_LIB_ROOTS=0 restores
the legacy layout; the mode is folded into the build-variant hash so the
two can never share a configured build dir.
_resolve_library_components does minutes of synchronous filesystem work
for a big library (FastLED: BFS over every header + copying thousands of
files) directly inside async compile(), starving the event loop: the
/api/compile/start response never flushed and clients timed out at 30s
while the build itself was healthy. Run it via asyncio.to_thread like
the cmake and ninja steps already do.
IDF's esp_http_server.h includes <http_parser.h> for the HTTP_GET/...
method enums. FastLED ships fl/stl/asio/http/http_parser.h and the merge
put that directory on the global -I, so the IDF include resolved to
FastLED's file and every httpd_method_t use failed with 'HTTP_GET was
not declared'. FastLED only references its copy path-qualified, so the
directory can safely stay off -I.
The arduino-mode branch used only the FIRST .ino in the request and
silently dropped every other tab, so any multi-tab sketch (the standard
Wokwi/Arduino-IDE project layout) died with 'not declared in this scope'
on globals defined in the main file. Concatenate the main sketch
(literal sketch.ino, else the first .ino) followed by the remaining
tabs alphabetically — the arduino-cli order — with #line markers so
diagnostics keep pointing at the real tab/line. Also REQUIRE
http_parser: exposing esp_http_server.h activates FastLED's fl net
server, which uses the HTTP_GET/HTTP_POST enums http_parser provides.
esp_cache.h (esp_mm) was the next one-header-at-a-time failure after
esp_http_server. Scan FastLED 3.10's esp platform tree once and add the
full candidate set (esp_mm, esp_hw_support, heap, esp_system, esp_rom,
log, app_update, bt, esp_psram, esp_pm) — every entry remains guarded on
the component existing in the IDF tree, so older IDF generations are
unaffected.
Two more merge gaps surfaced by FastLED 3.10 + U8g2 in one sketch:
- src/fl/test/ was fully dropped because 'test' matched the generic
excluded_dirs check at ANY depth, but FastLED's unity build includes
"fl/test/_build.cpp.hpp". Repo-level dirs named test/examples/docs
stay fully excluded; the same names nested inside a src-layout tree are
now header-only (fragments copied, sources still never compiled).
- FastLED's fl/net remote code includes <esp_http_server.h>, which the
Arduino IDE provides via the core's global include path. Add
esp_http_server to the merged component's guarded REQUIRES list.
FastLED's unity build (src/fl/build/extras+.cpp) includes
"extras/_build.cpp.hpp", but extras/ was in the excluded_dirs set so the
fragment was never copied and the merged-component build died with
'No such file or directory'. Treat extras/ as header-only instead: copy
.h/.hpp/.inl fragments so unity includes resolve, but keep compiling
nothing from it — extras/ traditionally holds repo-only content.
FastLED ships a host-test stub src/platforms/stub/Arduino.h. When another
merged library (e.g. U8g2) does #include <Arduino.h>, the stub resolved
first and broke the build: 'yield' was not declared in U8g2lib.cpp and
fl::string polluted String overload resolution inside the core WString.h.
Reuse the _SHADOW_STD_HEADERS mechanism so the stub's directory stays off
the global include path; FastLED reaches it via a file-relative include.
arduino-cli FQBNs may carry a menu-option suffix
(vendor:arch:board_id:Menu1=val1,...). The espidf lane previously derived
cdc_on_boot exclusively from the board id's boards.txt default, so a
generic-devkit board could never be built with Serial on the USB
Serial/JTAG CDC port.
- _fqbn_board_id_and_options(): split board id and menu options, keeping
the old last-segment behaviour for plain 3-part FQBNs.
- _arduino_board_flags(): CDCOnBoot=cdc overlays cdc_on_boot=1 on the
boards.txt base flags (mirroring <id>.menu.CDCOnBoot.cdc.build.
cdc_on_boot=1); the per-board cache stores only base flags, overrides
are applied on a copy.
- _arduino_variant(): board id extraction goes through the same helper so
a menu suffix no longer corrupts the variant lookup.
First user: the ESP32-C3-LCDkit board def pins
esp32:esp32:esp32c3:CDCOnBoot=cdc — the real kit has no UART bridge, its
only console is the USB Serial/JTAG port, and the esp32c3js engine now
models the always-attached host that HWCDC's connection gate requires
(esp32c3js v1.8.0).
The Function-EV module always carries 32 MB AP-hex PSRAM and esp_lcd
allocates DSI framebuffers from it unconditionally - without it the
board's flagship peripheral cannot run. Explicit psram=disabled still
wins; the UI only sends the field after the user opens the
board-options modal. Validated: an optionless espidf compile now embeds
esp_psram.
The P4 gate ('until the engines validate it') can flip: esp32p4js
v0.3.3 models the AP hex PSRAM + MSPI1 command controller. The classic
SPIRAM chunk emitted S3-era symbols (SPEED_80M, MODE_QUAD/OCT) that do
not exist in the P4's Kconfig.spiram - it gets its own chunk: MODE_HEX
+ SPEED_200M (tree default, rev0-safe; 250M needs rev >= 3). Validated
on staging: an esp_lcd_ek79007 espidf build with psram=enabled boots in
the engine to streaming 1024x600 DSI frames.
Generalizes the esp_camera.h lesson into a table (esp_lcd_st77916,
esp_lcd_touch_cst816s, esp_lcd_gc9a01, esp_lcd_ek79007, esp32-camera)
and applies it in the arduino branch, the no-core pure-IDF fallback AND
the explicit language=espidf branch (issue #139) - the last one is where
the VoCat display example lives and previously died at the first
include. Validated on staging: the ST77916 colour-bars espidf example
now fetches espressif/esp_lcd_st77916 from the registry and compiles.
_idf_target mapped only c3/c6/s3 and defaulted everything else to
'esp32', so a P4 fqbn compiled with the XTENSA toolchain and
-DARDUINO_ESP32P4_DEV. Adds p4/c5 predicates, IDF5-target membership,
0x2000 bootloader offsets, the riscv toolchain requirement, PSRAM held
off until the engines validate it, P4 CPU-freq drops (its kconfig
choices do not overlap the classic set), a rev0 sdkconfig pin for the P4
(the in-browser engine runs the only published mask ROM and v3 firmware
reads a flash descriptor rev0 never wrote), and an honest refusal for C5
arduino builds (no core exists). Validated on staging: an esp32p4
arduino sketch now produces a merged image with the riscv32 toolchain
that the esp32p4js engine accepts.
ESP-IDF compiles with -ggdb, and for debug builds ccache folds the cwd into
every cache key by default — CCACHE_BASEDIR does not exempt it (verified with
CCACHE_DEBUG on 4.11: 'Hashing CWD /var/lib/velxio-build/esp32s3/v_<hash>/
project/build'). Each build variant lives in its own dir, so every variant had
a private key space: eight esp32 variants of the SAME board each recompiled
all ~2100 IDF objects from scratch, the cache filled with dead per-directory
entries, and a cold Cardputer/S3 build (2263 steps) overran the 600s ninja
timeout on a busy box. CCACHE_NOHASHDIR=1 makes variants of the same
board/sdkconfig share objects; a cold variant becomes mostly cache hits. Only
DW_AT_comp_dir in .debug_info of a reused object can differ, which never
reaches the flashed .bin.
The timeout paths were silent server-side; they now log the build dir and how
many ninja steps completed, and the user-facing message says the partial
build is kept and a retry resumes.
The custom sketch.ino.cpp path never ran arduino-cli's ctags prototype
pass, so the universal Arduino idiom (helpers defined after loop()) died
with "'foo' was not declared in this scope" on every ESP32 build. A
conservative generator (skips templates, default args, methods,
function-pointer params) injects prototypes before the first definition
with a #line directive so error line numbers stay put. 8 unit tests.
IDF 5.5 renamed ESP_TASK_WDT to the EN/INIT pair; the v4.4 template's =n
became EN=n, compiling the whole implementation out — every sketch using
esp_task_wdt_add/reset (Firebase's client does internally) failed at
link. The v5 fixup now emits EN=y INIT=n: the API links, the watchdog
never auto-starts under emulation. v4.4 semantics untouched.
The dead-branch pruner marked any assumed-live branch as 'taken', so the
#else of an UNKNOWN condition was skipped. GxEPD2 guards its GFX base
class with '#if ENABLE_GxEPD2_GFX (default 0) ... #else #include
<Adafruit_GFX.h>' — the pruned #else silently dropped the Adafruit_GFX
dependency and every GxEPD2 e-paper example failed with 'fatal error:
Adafruit_GFX.h: No such file'. Found by the nightly gallery compile smoke.
'taken' is now set only by PROVABLY-true conditions (all identifiers in
the known-true set), keeping the 2026-08-03 ESPAsyncWebServer behaviour:
provably-dead branches still prune, provably-true branches still kill
their #else.
The main component had them; the merged user-libraries component did not, so
a warning INSIDE a vendor library was still fatal. M5Stack's MahonyAHRS.cpp
(fast inverse square root, a float read through a long*) fails under
-Werror=uninitialized, which made installing the vendor's own library kill
every build using it. Library code the user cannot edit must not stop their
build; the diagnostics still print as warnings.
Declared inside the has_arduino branch, so every successful pure
ESP-IDF (language=espidf) build crashed with UnboundLocalError when the
success return read it. Regression from today's manifest auto-migration
change; the Arduino paths were live-verified but the espidf mode was
not.
Three-part fix for the library-contamination class of ESP32 compile
failures (a user's project kept pulling unrelated libraries because 79%
of projects compile with an empty manifest, forcing scan-all resolution
over ~1000 shared libraries):
1. Name-affinity ranking in _find_library_for_header: the provider whose
name matches the header stem wins (exact normalized match > esp32
arch declaration > contains-stem, alphabetical only as tie-break).
Previously first-alphabetical won, so any stray library shipping a
generic header (Servo.h has 6 providers) could hijack the build.
2. Manifest auto-migration: scan-all compiles now report the libraries
the build really used (manifest_incomplete +
manifest_suggested_libraries on the compile result); the editor folds
single-candidate suggestions into the board's declared manifest after
a green build, so the NEXT compile runs scoped. One green build = one
migrated project; empty-manifest scan-all behaviour is untouched.
3. Install-declares: the Wokwi import library modal records successful
installs in the active board's manifest, and .vlx export/import now
round-trips board.libraries.
Tests: test_espidf_library_ranking.py (9) + library-manifest.test.ts (19).
Reported 2026-08-03: every ESP32 build in a user's project failed with
errors from libraries he never installed, emitted from user_libs_all --
a folder his file list does not even show, in projects he had recreated
from scratch.
Root cause: _detect_external_includes was a bare regex with no
preprocessor state. ESPAsyncWebServer guards <Hash.h> behind
'#if defined(ESP8266) || defined(TARGET_RP2040)', so the compiler never
sees it on ESP32 -- but the scanner reported it anyway. The resolver
then searched the ~1000 shared-cache libraries, matched stemi-hexapod's
src/Hash.h (a hexapod-robot library), merged it WHOLE, and compiled all
of its sources; each failed on its own unrelated includes.
The scanner now tracks #if/#ifdef/#ifndef/#elif/#else/#endif nesting and
strips comments. A branch is skipped ONLY when every identifier in its
condition is a platform macro known to be undefined on ESP32; unknown
macros keep the branch LIVE, because hiding a genuinely needed header
would be a much worse failure than the old over-reporting. Verified
against the real ESPAsyncWebServer shape plus nesting/elif/else/comment
cases (9 new tests, 15 green in the espidf suites).
Complements the platform-header guard from the previous commit, which
stays as defence in depth.
Reported by a user whose every ESP32 build failed with errors from
libraries he never installed (SoftwareSerial.h, ArduinoBearSSL.h,
Adafruit_GFX.h, AsyncElegantOTA.h) coming out of user_libs_all, in a
project whose file list doesn't even show that folder.
Traced end to end: ESPAsyncWebServer includes <Hash.h> (an ESP8266-core
header, guarded for that platform, so arduino-esp32's tree has no such
file). The global scan then resolved it to stemi-hexapod's src/Hash.h --
a hexapod-robot library -- merged the WHOLE library into user_libs_all,
and its Serial.cpp / Hash.cpp / Server.cpp / BluetoothLowEnergy.cpp /
ExpansionDriver.cpp were compiled, each failing on its own unrelated
includes. Log line: 'Merging "stemihexapod" into user_libs_all for
<Hash.h>'.
Hash.h and friends now join the never-resolve-against-user-libs set.
They cannot be discovered by the core-tree scan (the core genuinely
lacks them) yet their names are generic enough that some installed
library owns a file by that name.
Installing ArduinoJson made the project stop compiling, with an error that
never mentioned ArduinoJson:
CMake Error at user_libs/user_libs_all/CMakeLists.txt:15
(target_compile_definitions):
target_compile_definitions may only set INTERFACE properties on
INTERFACE targets
-- Configuring incomplete, errors occurred!
ArduinoJson ships headers only. With no sources, idf_component_register()
builds the merged user-libs component as an INTERFACE library, and PRIVATE
scope on an INTERFACE target is a hard CMake error - so the configure step
died before a single file was compiled. The user saw only a failing build
and reported it as "ArduinoJson can't be found even after it has been
installed" (#124).
Those definitions map alloca/memcpy_P/memcmp_P for library C sources that
use them undeclared; with no sources there is nothing to define, so they are
emitted only when the component actually compiles something. A library that
carries its own .cpp is unchanged - which is why LiquidCrystal_I2C (#257)
worked and hid this.
Two-tier rule replacing the blanket skip:
- 'all' now counts as wildcard ('*'): LiquidCrystal_I2C 2.0.0 declares
architectures=all and was skipped on every ESP32 build, breaking the
most popular I2C LCD library with a bare 'No such file or directory'.
- Headers the SKETCH includes directly merge even when the library
declares a foreign-only architecture (many avr-declared libs are pure
Wire/SPI code that compiles fine; a truthful compile error beats a
missing-header one). Transitive pulls keep the HARD guard, which is
the path that dragged SAMD-only Adafruit_ZeroDMA into ESP32 builds.
Un crash entre ambos (el NameError de board_fqbn) dejaba defaults==rendered
con un sdkconfig stale, y el re-seed de kconfig no disparaba nunca mas en ese
dir persistente — medido: CONSOLE_SECONDARY quedo en usb_serial_jtag pese a
los defaults nuevos. El bloque de velxio_board.cmake pasa a despues.
El bloque de velxio_board.cmake usaba board_fqbn, que vivia en compile() y no
en la firma de _compile_in_dir: parametro nuevo con default None, propagado en
los dos call sites, y guard para el camino sin fqbn (tests directos).
Tres piezas:
- _arduino_board_flags lee <id>.build.board y <id>.build.cdc_on_boot de
boards.txt (cache, mismo patron que _arduino_variant).
- Cada compile escribe velxio_board.cmake (incluido OPTIONAL por el
CMakeLists ANTES de project(), asi los defines llegan a TODOS los
componentes — que Serial sea el CDC se decide dentro del core Arduino):
ARDUINO_<BOARD> siempre que boards.txt lo nombre (los sketches de vendor
hacen #ifdef ARDUINO_XIAO_ESP32S3), y ARDUINO_USB_CDC_ON_BOOT=1 +
ARDUINO_USB_MODE=1 cuando cdc_on_boot=1. USB_MODE va FORZADO a 1 (HWCDC
por USB-Serial-JTAG, la opcion de menu USBMode=hwcdc): el motor modela esa
consola, no la pila OTG/TinyUSB. Ir por archivo y no por env hace el flag
dependencia de configure: CMake reconfigura al cambiar.
- CONFIG_ESP_CONSOLE_SECONDARY_NONE en el sdkconfig: sin ella el ROM/IDF
espejan cada byte de log al USB y el monitor unico del emulador (que
mezcla ambos streams) mostraria el log entero dos veces en builds CDC.
_normalize_options rellena arduinoVariant con _arduino_variant(board_fqbn),
que ya leia boards.txt; la plantilla ya consumia la clave. Sin esto todas las
placas S3 compilaban como esp32s3 generico: sin nombres Dx del XIAO y con
Serial en UART0, cuyo RX por defecto es GPIO44 = D7 del XIAO — un
pinMode(D7) tardio desinstala el driver y silencia los prints del loop
(medido en el banco con el ejemplo del Round Display).
Los dos ejemplos de camara (esp32cam-lcd-preview y esp32cam-webcam-demo) morian
con "esp_camera.h: No such file or directory".
No es que el ejemplo estuviera mal: es una regresion de la migracion. El arbol
viejo /opt/arduino-esp32 (2.x) traia un SDK precompilado con esp32-camera dentro,
en tools/sdk/<target>/include/esp32-camera/, para esp32, c3, s2 y s3. El
componente 3.x que usamos ahora NO lo trae, y su idf_component.yml tampoco
depende de el, asi que el header simplemente dejo de existir. Por eso funcionaba
antes y dejo de funcionar sin que nadie tocara el ejemplo.
Mezclar los headers de la 2.x en este build no es opcion — vienen con su SDK
precompilado y aqui compilamos contra IDF 5. Lo correcto en el mundo 3.x es
pedirle el componente al gestor: la plantilla no traia idf_component.yml, asi que
todo lo gestionado llegaba de rebote por el manifiesto de arduino-esp32, y lo que
el no arrastra hay que pedirlo explicitamente.
_detect_camera_usage() ve el include y _add_managed_components() escribe
main/idf_component.yml con espressif/esp32-camera. Verificado en el contenedor
sobre el mismo directorio que fallaba: el gestor descarga 2.1.7, el configure
termina con esp32-camera en la lista de componentes, y sketch.ino.cpp compila.
Arduino trata igual #include "Lib.h" que #include <Lib.h> para librerias, y
los ejemplos de los propios fabricantes usan la forma con comillas: los de
M5Stack para el Cardputer abren con #include "M5Cardputer.h". Al escanear solo
la forma con angulos, ese sketch no llegaba siquiera al resolutor de librerias y
moria con 'fatal error: M5Cardputer.h: No such file', mientras el mismo sketch
con angulos compilaba sin problema. Cualquiera que pegue un ejemplo oficial se
daba de bruces con esto.
Se pasan ademas los nombres de los ficheros del propio sketch, para que un
#include "Common.h" local siga siendo una cabecera del proyecto y no se
confunda con una libreria.
test/backend/unit/test_espidf_compiler.py cubre ambas formas, el espaciado, las
cabeceras internas de IDF y la cabecera propia del proyecto.
Las dos ramas habian divergido: master llevaba el modo lenguaje ESP-IDF puro
(#139) y v3.2 la ruta de compilacion IDF v5.5 para toda la familia ESP32 mas
los arreglos de venv/toolchain. Ambas tocaban espidf_compiler.py.
Los dos lados son ejes ORTOGONALES y se conservan enteros:
- use_idf5 / arduino_mode (v3.2): que arbol IDF usa el build (5.5 vs 4.4) y
si cabe Arduino-como-componente.
- pure_idf (master): el modo LENGUAJE que elige el usuario; sus ficheros son
las fuentes del componente main con su propio app_main().
Resolucion:
- _build_env acepta los tres. Un build IDF puro fuerza arduino_mode a falso:
la plantilla CMake mete el componente arduino-esp32 en cuanto existe
ARDUINO_ESP32_PATH, asi que dejarlo puesto compilaba el core de Arduino en
un build que no tiene sketch. Lo cazaron los tests de master.
- VELXIO_PURE_SKETCH solo con pure_idf, nunca con arduino_mode a falso a
secas: un target sin core arduino-esp32 sigue entregando un SKETCH al
traductor legacy y no debe tomar la rama del glob puro.
- La identidad del build-dir suma los dos tokens (|idf:N|ard:N y |lang:pure):
ningun par de esas combinaciones puede compartir un build/ configurado.
- La cadena de escritura de fuentes queda pure_idf -> arduino_mode -> legacy.
- sdkconfig: render de v3.2 (con target/use_idf5) mas el filtrado de simbolos
CONFIG_ARDUINO* de master cuando el build es puro.
test/backend/unit/test_espidf_compiler.py: los 7 tests que ya estaban rotos en
v3.2 (AttributeError: idf5_path, fixture sin actualizar desde que se anadio la
seleccion de IDF) vuelven a pasar.
Verificado: backend 293 pasan / 0 fallan (v3.2 traia 7 rotos); frontend 2268
pasan / 0 fallan en los dos shards.
Adds a third entry to the board language selector next to Arduino C++
and MicroPython: ESP-IDF. In this mode the user writes a plain ESP-IDF
project — app_main() entry point, FreeRTOS + driver APIs — and the
backend compiles it through the same ESP-IDF toolchain it already uses
for ESP32 Arduino sketches, just without the arduino-esp32 component.
Backend:
- CompileRequest.language ('espidf') threaded through the sync + async
compile paths and folded into the dedup job key (language='arduino'
and omitted hash identically so old clients keep dedupping).
- espidf_compiler: pure_idf flag. User files are written into main/
as-is (no Arduino.h wrap, no velxio_compat.h, Arduino library
resolution skipped), ARDUINO_ESP32_PATH is dropped from the build env
and VELXIO_PURE_SKETCH raised so the template CMake compiles the
user's own sources via a glob branch. Pure builds get their own
persistent build-dir variant through the eff_hash fold.
- QEMU WiFi compat for IDF-style code: esp_wifi.h/esp_wifi_init
detection sets has_wifi, and literal #define SSID/PASS plus
wifi_config_t designated initializers are normalized to the QEMU AP.
- CONFIG_ARDUINO_* lines are stripped from sdkconfig.defaults in pure
mode (the symbols don't exist without the arduino component).
Frontend:
- LanguageMode gains 'espidf'; BOARD_SUPPORTS_ESPIDF covers the ESP32
family (Xtensa, S3, C3). Toolbar shows the option only for those.
- Switching modes seeds a main.c blink skeleton (app_main + gpio
driver), mirroring the MicroPython main.py flow.
- compileCode sends language='espidf'; run/stop paths are unchanged
(the QEMU worker consumes the same merged flash image).
- New gallery example: esp32-idf-blink (LED + resistor on GPIO 2).
Tests: unit coverage for the build-env switch, IDF wifi normalization,
job-key variance, file-group seeding and the new example; verified
end-to-end in a container from the prod image (pure build produces a
bootable flash image; Arduino-mode build unchanged, same variant hash).
Ports the v5 compiler layer: /opt/esp-idf-v5 + arduino-esp32 3.x
discovery (IDF5_PATH / ARDUINO_ESP32_5_PATH overridable), per-compile IDF
selection routing the family to the v5 tree, esp32c6 target support, and
the docker-runtime hardening (version-matched python venv pinned on PATH,
toolchain ordering by the tree's tools.json, user-libs component REQUIRES
for the IDF components Arduino libraries include directly). The LED_BUILTIN
fallback in the project template becomes a whitelist so non-esp32 variants
(C6 first) never hit the redefinition trap. Includes the option tests.
The newest-env-wins heuristic pinned a 5.x env for a 4.4 tree — each
generation's dependency check only accepts its own requirement set. The
version is read from tools/cmake/version.cmake of the active IDF tree,
falling back to the newest env when unreadable.
Two failure modes seen on a fresh docker deploy where the final image's
system python differs from the one the IDF venv was created with:
- IDF's cmake derives the venv dir name from the SYSTEM python version
and fails with 'python doesn't exist' before configure; and when cmake
is invoked directly (not via idf.py) its PYTHON property falls back to
the bare 'python' from PATH. The Linux branch of _build_env now pins
IDF_PYTHON_ENV_PATH to the newest env under the tools root and puts its
bin first on PATH (mirroring what the Windows branch already did).
- When several toolchain generations share one tools root, glob order
decided which won the PATH race. Toolchain dirs are now ordered by the
versions the IDF tree declares in tools/tools.json.
_idf_target had no S3 case, so every S3 FQBN (esp32s3 / XIAO_ESP32S3 /
nano_nora) compiled as plain esp32 (LX6) and could not boot the S3 QEMU
machine. Add _is_esp32s3 (covers all three FQBNs; nano_nora has no 's3'
token) + return 'esp32s3'; add the xtensa-esp32s3-elf toolchain to the
build PATH; place the S3 second-stage bootloader at flash offset 0x0
(like C3) instead of 0x1000 via 'is_c3 or idf_target==esp32s3' at the merge.
Requires the xtensa-esp32s3-elf toolchain in the image
(install.sh esp32,esp32c3,esp32s3) and a libqemu-xtensa with an
esp32s3(-picsimlab) machine.
arduino-esp32's WiFiClientSecure/ssl_client.cpp wraps its ENTIRE body
(start_ssl_client, ssl_init, send_ssl_data, ...) in
#if !defined(MBEDTLS_KEY_EXCHANGE_SOME_PSK_ENABLED) ... #else <body> #endif
ESP-IDF's mbedtls defaults the PSK key-exchange modes OFF, so the object
compiled empty and any sketch using WiFiClientSecure — including
HTTPClient.begin(url), which links the secure client even for http:// —
failed to link with "undefined reference to start_ssl_client". Commenting
out begin() let the optimizer drop the unused client, which is why it
"compiled when commented".
- sdkconfig.defaults.in: enable the PSK key-exchange ciphersuites
(CONFIG_MBEDTLS_PSK_MODES + the four KEY_EXCHANGE_*PSK), matching
arduino-esp32's own sdkconfig.
- espidf_compiler: ESP-IDF only seeds sdkconfig from sdkconfig.defaults when
sdkconfig is ABSENT. Persistent build dirs live in the build volume and
keep a stale sdkconfig across image rebuilds, so the new CONFIG_* would
never reach kconfig. Drop the generated sdkconfig when the rendered
defaults change so it re-seeds on configure.
- test: assert the rendered sdkconfig enables PSK.
Verified end-to-end: the reported WiFi+HTTPClient sketch now compiles to a
1.1 MB binary (was a hard link error before).
The last paths that read the shared global /root/Arduino/libraries: a compile
with NO manifest (libraries=null — any from-scratch/anon sketch; the manifest is
never auto-derived from #includes) and the incomplete-manifest scan-all retry
(which re-enters compile unscoped). Both fell through to the global dir,
bypassing the cache entirely (a cached lib still failed when global was gone).
Now, when no scope is materialized, point the library search at the cache: the
cache root is itself a valid Arduino libraries dir (each <name@ver-sha> child is
a library), exposed via env (pro overlay sets them):
- arduino-cli: ARDUINO_DIRECTORIES_USER = VELXIO_FALLBACK_SKETCHBOOK (whose
libraries/ -> cache root) when scope_dir is None.
- ESP-IDF: _find_arduino_libraries_dir() prefers VELXIO_FALLBACK_LIBRARIES_DIR.
Unset (OSS self-host) -> legacy global, unchanged.
Also strip a trailing @version from manifest names (_bare_lib_names): norm_name
fused 'ArduinoJson@6.21.5' -> 'arduinojson6215' (cache miss -> global scan-all);
the per-board boards_json manifest is the path that still carries @version.
So a scoped compile can resolve the project OWNER's per-user custom libraries
(not the requester's) — a shared/embed/anon compile of someone else's project
still finds that owner's uploaded libs.
- core/hooks.py: new get_project_owner hook; materialize_library_scope gains an
opaque owner_id param (no-op default unchanged).
- espidf_compiler.compile/_attempt: thread owner_id to the materializer.
- compile.py: resolve owner via get_project_owner(project_id), pass to compile.
Additive: the OSS image (no overlay) ignores owner_id; index libs still resolve
from the cache. Foundation for per-user custom-lib storage (P2.2a write side).
A scoped ESP32 compile can now resolve libraries from a per-compile directory
provided by an overlay (the manifest's libs symlinked from a content-addressed
cache, with a legacy-dir fallback) instead of the single shared global volume.
- core/hooks.py: register_materialize_library_scope / materialize_library_scope
(no-op default -> None, so the OSS image keeps its single scan-all dir).
- espidf_compiler: _attempt(allowed) calls the hook, folds the returned content
token into the build-variant eff_hash (a content change gets a clean build
dir), passes libraries_dir to _compile_in_dir (arduino_libs = libraries_dir or
_find_arduino_libraries_dir()), and removes the throwaway dir after. The
graceful scan-all fallback (allowed=None) keeps using the default dir, so the
worst case of any materializer failure is fall-back-to-legacy (no break).
Per-variant build dirs fixed the cross-compile staleness, but the FIRST build of
a cold variant can still occasionally hit ESP-IDF nested-build flakiness (cmake /
bootloader / managed-components / sdkconfig). These are infrastructure failures,
never user code, and clear on a retry once the variant dir is warmer. Retry the
attempt once when the failure matches infrastructure markers (not a user-sketch
or missing-library error). Cheap via ccache + ninja incremental.
Replaces the wipe-on-change approach (which left ESP-IDF's nested bootloader /
managed-components build in a broken state under rapid reconfigure -
intermittent 'managed_components_list.temp.cmake: No such file'). Each distinct
configuration (board options x resolved library set, via the variant key the
caller already computes) now gets its OWN persistent project dir with its OWN
build/, never wiped or reconfigured for a different config:
- same config -> same dir -> warm ninja incremental + ccache (fast iterate);
- different config -> different dir -> isolated, consistent, no staleness,
no nested-build breakage;
- the scoped vs scan-all fallback attempts land in different dirs, so the
double-compile no longer corrupts a shared build/.
Variant dirs are LRU-bounded (_MAX_BUILD_VARIANTS per target); the global ccache
warms a fresh/evicted variant in seconds. Cleans up the old single-project
layout on first run. Fixes the cross-compile staleness for legacy AND manifest
compiles, and the fallback regression.
David Montero Crespo