A manifest-scoped compile only sees the libraries the project declares, so
it dies on the first undeclared transitive dependency — a header the user
had no reason to name. The scan-all retry exists to rescue exactly that,
and when it succeeds nothing is lost. When it fails as well, we returned
the SCOPED failure, which describes a library only our own scoping made
missing.
Today a hand-written SSD1306 sketch was reported as
Adafruit_GFX.h:12:10: fatal error: Adafruit_I2CDevice.h: No such file
while the retry had already resolved BusIO and reached the real bug, an
`#include <Adafruit_SSD1306>` with no .h. The user went looking for a
missing library that was installed all along.
The retry runs in the superset environment, so its error is the sketch's
own in both compilers. Prefer it whenever it carries text; the scoped
result still stands in when it does not.
A sketch that includes BTstackLib on a CYW43 board fails with arduino-pico's
static_assert: "This library needs Bluetooth enabled. Use the 'Tools->IP/
Bluetooth Stack' menu in the IDE to enable it." There is no such menu here, so
it reads as a setting we forgot to expose. What is actually true is simpler:
the CYW43439's Bluetooth controller is not emulated — only its WiFi side is —
so there is nothing to switch on.
Appends that, leaving the compiler's own diagnostic intact above it because the
developer still needs the real text. Deliberately a separate helper from
humanize_cli_error, which REPLACES arduino-cli's own failures; this one only
adds context under a compiler message.
Worth knowing for whoever picks up the BT work: this is the honest-failure
milestone. Making a BT sketch compile at all means flipping this board's FQBN
to ipbtstack=ipv4btcble, which swaps liblwip.a for liblwip-bt.a and costs every
sketch on the board ~71 KB flash and ~18 KB RAM — and leaves it reaching
HCI_STATE_WORKING while advertising to nobody. The mechanism is mapped in
project/pimoroni-pico-plus-2w/STATUS.md.
Follow-ups from the compile-latency audit, all measured on staging:
- The per-target build lock keyed every board through _idf_target, which
defaults unknown FQBNs to 'esp32' - so AVR, RP2040 and STM32 compiles
all took the 'esp32::esp32' key and queued behind unrelated ESP32
builds. The lock now only applies to boards that actually route to the
ESP-IDF lane (the same startswith('esp32:') predicate _run_compile
uses); everything else keys on its own FQBN.
- ESP-IDF re-validated its python environment on every cmake configure
(~1s of ~9s). The image pins that env at build time, so pass
PYTHON_DEPS_CHECKED=1 - the escape hatch idf.py uses for itself.
- _is_core_installed shelled out to 'arduino-cli core list' (~1.5s) on
every non-AVR compile from inside an async handler, blocking the whole
event loop rather than just that request. Memoized, and invalidated
where a core is installed.
New hand-written code, unique source, idle box, steady state:
ESP32-P4 ESP-IDF 28.8s -> 11.1s
ESP32-S3 ESP-IDF 39.3s -> 11.5s
ESP32-C3 Arduino 56.2s -> 21.0s
(the ESP-IDF numbers also carry the ldgen memoizer from the previous
commit; the queue waits the audit saw - 41s and 43s - were the audit's
own five concurrent builds and are what the lock-key fix targets)
arduino-cli reports errors as plain text or, with --format json, as a JSON
object on stderr. We forwarded that blob verbatim, so on a fresh install
with no library index yet (and no way through the firewall/proxy to
downloads.arduino.cc) the in-app agent told the user "the library server
returns a JSON error" ten times in a row.
humanize_cli_error() maps the known shapes — index download failed,
library zip download failed, library not found — to a single sentence
that says what to check, and keeps the raw text under raw_error. Applied
to search_libraries (rc != 0 and non-JSON stdout) and install_library.
Tests use the real arduino-cli 1.0.4 output captured offline.
File names may now carry '/' paths; the explorer renders each board group
as a collapsible folder tree (folders first, files in group order, CSS-var
indentation because .fe-file-item pins padding with !important). New-folder
buttons on the board header and per-folder hover actions (new file here,
new subfolder, delete with confirm when files are inside; a folder holding
every file of the group cannot be deleted - groups keep >= 1 file).
Model: names are the source of truth; only EMPTY folders are tracked
(folderGroups per group) so they survive until a file lands in them, then
materialise into the name prefix. Round-trips through .vlx (optional
folderGroups field, laxly validated), the auth stash draft, and
loadProjectState. Examples gain boards[].files so a single board can ship
a full multi-file tree (first entry = main sketch).
Backend: the arduino-cli sketch writer now creates subdirectories for
'/'-pathed names and REJECTS names that resolve outside the sketch dir
('..' traversal) - previously it crashed on the former and wrote outside
the tmp dir on the latter.
list_installed_libraries() now honors VELXIO_FALLBACK_SKETCHBOOK (pro overlay) so
GET /api/libraries/list enumerates the content-addressed cache instead of the
shared global dir — the Installed tab survives the global volume's retirement
(audit finding #3). Unset (OSS self-host) -> default sketchbook, unchanged.
The last paths that read the shared global /root/Arduino/libraries: a compile
with NO manifest (libraries=null — any from-scratch/anon sketch; the manifest is
never auto-derived from #includes) and the incomplete-manifest scan-all retry
(which re-enters compile unscoped). Both fell through to the global dir,
bypassing the cache entirely (a cached lib still failed when global was gone).
Now, when no scope is materialized, point the library search at the cache: the
cache root is itself a valid Arduino libraries dir (each <name@ver-sha> child is
a library), exposed via env (pro overlay sets them):
- arduino-cli: ARDUINO_DIRECTORIES_USER = VELXIO_FALLBACK_SKETCHBOOK (whose
libraries/ -> cache root) when scope_dir is None.
- ESP-IDF: _find_arduino_libraries_dir() prefers VELXIO_FALLBACK_LIBRARIES_DIR.
Unset (OSS self-host) -> legacy global, unchanged.
Also strip a trailing @version from manifest names (_bare_lib_names): norm_name
fused 'ArduinoJson@6.21.5' -> 'arduinojson6215' (cache miss -> global scan-all);
the per-board boards_json manifest is the path that still carries @version.
Empirically, arduino-cli's --libraries ADDS to the search path — the global
sketchbook is STILL scanned, so the prior commit did NOT isolate reads from the
shared global volume. Point ARDUINO_DIRECTORIES_USER at the scratch sketchbook
instead (its <scratch>/libraries becomes the ONLY user-library dir); cores +
board-manager URLs live in the DATA dir and are untouched.
In-container functest (AVR uno): a cache-only lib compiles via the scope (never
in global); a global-only lib NOT in the manifest is INVISIBLE to the scoped
compile and only recovers via the scan-all retry (manifest_incomplete=True) —
proving the global volume is no longer scanned for a manifest-scoped build.
arduino-cli compiles now read libraries from a per-compile --libraries dir of
symlinks materialized by the pro overlay (owner store -> cache -> legacy), via
the existing materialize_library_scope hook, instead of scanning the shared
mutable global volume. --libraries only overrides the USER library search path,
so cores + board-manager URLs (RP2040 earlephilhower, ATTinyCore) are untouched.
Mirrors the ESP-IDF P2.1e graceful fallback: an incomplete manifest (a needed or
transitive lib not declared) makes the scoped compile miss a header; we retry
ONCE scan-all (no --libraries) and surface manifest_incomplete, so a partial
manifest degrades to legacy behavior instead of hard-failing.
Dedup correctness: the manifest + owner are resolved ONCE (shared
_resolve_compile_scope) and folded into the /compile/start dedup key AND threaded
into the build, so two owners with identical sketch+board but different custom
libs never coalesce to one another's job, and the key never diverges from the
bytes the build uses. Owner folded only when a manifest applies (index-only
compiles keep cross-owner dedup).
STM32 emulation (open-core, runs via libqemu-arm in the backend worker):
- backend: stm32_lib_manager + stm32_worker (GPIO, USART, I2C/SPI device models
reusing the ESP32 slaves, live sensor updates), arduino_cli STM32 branch,
start_stm32 simulation route.
- frontend: Stm32Bridge + Stm32BluePill(/BlackPill) web components (Wokwi SVGs),
board kinds, Interconnect/boardPinMapping/boardProtocols wiring, example
projects (blink, serial, I2C BMP280/MPU6050/DS1307/SSD1306/weather, 7-seg,
RGB, button, switch, stepper, cross-board interconnect).
- Raspberry Pi 4/5 board elements + thumbnails.
Pro board gating (generic OSS->Pro seam; entitlement logic lives in the overlay):
- lib/proBoardGate.ts: isProBoardKind (STM32 + every QEMU Raspberry Pi),
installBoardGateImpl/boardGateDecision, triggerProUpgradePrompt.
- PRO badge on those boards in the component picker; gate at the picker add +
the run backstop (startBoard).
- backend/app/services/board_access.py: server-side enforcement seam for the
simulation WebSocket; STM32/Pi unavailable -> Pro-framed message.
- desktop: generic QemuDownloadPrompt + Stm32QemuPrompt (download-behind-license,
mirrors the ESP32 prompt).
- .gitignore: never ship libqemu-* binaries in the public image.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ATTinyCore >=1.5.0 declares ATTinyCore:micronucleus@2.5-azd1b as a tool
dependency, hosted at https://azduino.com/bin/micronucleus/. That host
has been unreachable (connection refused) for extended periods, causing
every ATtiny85 compile to fail at the core-install step with:
Download failed: performing HEAD request: ... dial tcp ...: connection refused
Failed to install required core: ATTinyCore:avr
micronucleus is only used for USB upload — never for compilation — but
arduino-cli refuses to install a core whose tool deps cannot fetch.
Pin to 1.4.1, the last release whose micronucleus binary is hosted on
github.com (digistump release, reachable). The FQBN clock options we
ship (clock=16pll on attinyx5, etc.) are unchanged across 1.4.x.
- backend/app/services/arduino_cli.py: new CORE_INSTALL_VERSIONS map
consulted by ensure_core_for_board so the runtime auto-install
passes "ATTinyCore:avr@1.4.1" instead of unversioned latest.
- backend/Dockerfile and docker/entrypoint.sh: same pin so a fresh
image bakes 1.4.1 in and never hits the runtime fallback path.
Existing regression tests in test/backend/unit/test_arduino_cli_attinycore.py
still pass (they assert presence, not version).
Backend:
- api/routes/compile.py accepts board-specific compile options
and dedups in-flight identical requests
- services/espidf_compiler.py expanded ESP-IDF wrapper with the new
options surface (sdkconfig.defaults.in
template added)
- services/arduino_cli.py honour the new options envelope
- services/esp32_lib_bridge.py thread board options through to QEMU
Tests:
- tests/test_compile_request_dedup.py end-to-end dedup behaviour
- tests/test_espidf_options.py covers the new options parsing
Frontend:
- services/compilation.ts client-side mirror — sends the new
options field on every compile request
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Backend:
- Add version field to InstallLibraryRequest
- Add fallback and requested_version to InstallResponse
- Add DELETE /api/libraries/uninstall endpoint
- Enhance install_library() for versioned installs (LibName@version)
- Add semver validation and fallback logic
- Add uninstall_library() method
- Fix _parse_version() to reject non-numeric version parts
Frontend:
- Update installLibrary() with optional version parameter
- Add uninstallLibrary() and resolveLibraryVersion() helpers
- Add version selector dropdown in Library Manager
- Add UNINSTALL button for installed libraries
- Show fallback messages when requested version unavailable
- Add parseLibSpec() and version badges in InstallLibrariesModal
- Implemented handshake tests to validate initial bus state and register responses.
- Created end-to-end tests for Pico W LED blinking using MicroPython firmware.
- Added SDPCM framing tests to ensure proper encoding and decoding of control frames.
- Developed IOCTL tests to verify command responses and state changes in the emulator.
- Established a full lifecycle test for WiFi operations, including scanning, connecting, and packet handling.
- Introduced TypeScript configuration for test files to ensure compatibility and strict type checking.
- Updated components-metadata.json with new generated timestamp.
- Refactored Esp32C3Simulator.ts to remove unnecessary debug variables and logging, and added support for additional ROM functions.
- Modified useSimulatorStore.ts to clarify bridge usage for ESP32 boards.
- Updated submodules for QEMU and other libraries to indicate dirty state.
- Added test_esp32c3_emulation.py for end-to-end testing of ESP32-C3 emulation, including compilation, flash image merging, and GPIO event checking.
- Added detailed logging for GPIO changes, system events, and errors in simulation websocket.
- Improved ESP32 firmware handling by merging individual binaries into a single 4MB flash image.
- Updated ESP32 bridge to handle serial output and GPIO changes with appropriate logging.
- Introduced integration test for ESP32 emulation, covering compilation, WebSocket connection, and event handling.
- Enhanced examples to include ESP32 projects and updated the examples gallery to reflect new board types.
- Refactored simulator store to manage ESP32 bridge and simulator instances more effectively.
- Updated requirements to include esptool for ESP32 firmware management.
- Added ESP32 emulation plan and architecture documentation.
- Created `esp_qemu_manager.py` for managing ESP32 QEMU instances.
- Modified backend API routes to support ESP32 firmware loading and GPIO handling.
- Introduced `Esp32Bridge.ts` for frontend communication with ESP32 instances.
- Refactored simulator store to support multiple boards, including Raspberry Pi and Arduino.
- Created `RaspberryPi3Bridge.ts` for WebSocket communication between frontend and backend for Raspberry Pi.
- Updated QEMU manager to handle multiple serial ports for Raspberry Pi GPIO communication.
- Enhanced SimulatorCanvas to render multiple boards and manage wire routing between them.
- Implemented board picker modal for selecting and adding boards to the canvas.
- Updated editor to support multiple file groups per board.
- Added migration logic for loading old project formats into the new multi-board structure.
- Ensured backward compatibility with existing components and functionality.
- Add LoginPage and RegisterPage for user authentication.
- Create UserProfilePage to display user projects.
- Implement ProjectPage for viewing and editing individual projects.
- Introduce authService for handling user login, registration, and session management.
- Add projectService for managing project data retrieval and manipulation.
- Enhance EditorPage with file management capabilities and save prompts.
- Introduce Zustand stores for managing authentication, editor state, and project state.
- Add reserved usernames utility to prevent certain usernames during registration.
- Update compilation service to handle multiple files for Arduino sketches.
- Simplified serial data handling in `useSimulatorStore` for both AVR and RP2040 simulators.
- Introduced `boardPinMapping.ts` to map wokwi-element pin names to simulator GPIO/pin numbers for Arduino Uno and Nano RP2040.
- Added `compilationLogger.ts` to parse compile results into structured log entries for better console output.
- Added SimulatorCanvas component for rendering the simulator interface.
- Integrated Wokwi components (Arduino, LED, Resistor, Pushbutton, Potentiometer) into the simulator.
- Created PinManager to handle pin state changes and notifications.
- Developed AVRSimulator class for emulating Arduino Uno functionality.
- Implemented hex file loading and compilation service.
- Added CSS styles for the simulator interface.
- Established Zustand stores for managing editor and simulator states.
- Created utility functions for parsing Intel HEX format.
- Set up Vite configuration for the frontend project.
- Added batch scripts for starting backend and frontend servers, and updating Wokwi libraries.
David Montero
ZhadowValker