The symbol server assumed one library layout: the content-addressed cache,
where an entry dir name IS its content hash. A host without that cache --
Velxio Desktop's sidecar, and any self-host -- has its libraries where
arduino-cli put them, under plain folder names with the version inside
library.properties, and got a 404 for every request. The editor there falls
back to the engine's embedded catalog, so `client.` on a library the user
installed himself resolves to nothing.
Both layouts are now sources, tried best-first. The only real difference is
mutability: `lib upgrade` rewrites a sketchbook folder in place and keeps its
name, so a scan cannot be keyed by that name. Sketchbook entries are keyed by
a cheap stat-only stamp (file count, total size, newest mtime) in the same
`<norm_name>@<norm_version>-<12 hex>` shape, which makes the disk cache exact
rather than merely long-lived. Cache entries keep their existing keys, so no
deployed scan is invalidated.
Where the sketchbook is: an explicit override, then the sketchbook env vars,
then arduino-cli itself (`config get directories.user`, probed once and
remembered) -- guessing ~/Arduino is wrong on the platforms whose default is
~/Documents/Arduino. A sketchbook that resolves to the cache is dropped: the
pro overlay points arduino-cli at a symlink to the cache root, and scanning
it twice under two shapes buys nothing.
Scans for a sketchbook are written to `<sketchbook>/.velxio-symbols/`, beside
`libraries/` and never inside it, so nothing Velxio writes shows up in
`arduino-cli lib list`.
The type index now keys on a fingerprint of the source's entry keys instead
of the entry count, which is exact for both layouts (an in-place upgrade
changes no count). Deployed indexes rebuild once, ~1s from the cached scans.
Follow-ups from the compile-latency audit, all measured on staging:
- The per-target build lock keyed every board through _idf_target, which
defaults unknown FQBNs to 'esp32' - so AVR, RP2040 and STM32 compiles
all took the 'esp32::esp32' key and queued behind unrelated ESP32
builds. The lock now only applies to boards that actually route to the
ESP-IDF lane (the same startswith('esp32:') predicate _run_compile
uses); everything else keys on its own FQBN.
- ESP-IDF re-validated its python environment on every cmake configure
(~1s of ~9s). The image pins that env at build time, so pass
PYTHON_DEPS_CHECKED=1 - the escape hatch idf.py uses for itself.
- _is_core_installed shelled out to 'arduino-cli core list' (~1.5s) on
every non-AVR compile from inside an async handler, blocking the whole
event loop rather than just that request. Memoized, and invalidated
where a core is installed.
New hand-written code, unique source, idle box, steady state:
ESP32-P4 ESP-IDF 28.8s -> 11.1s
ESP32-S3 ESP-IDF 39.3s -> 11.5s
ESP32-C3 Arduino 56.2s -> 21.0s
(the ESP-IDF numbers also carry the ldgen memoizer from the previous
commit; the queue waits the audit saw - 41s and 43s - were the audit's
own five concurrent builds and are what the lock-key fix targets)
Running a gallery example took 51 s before a single pixel appeared.
Measured split: 28 s compile, 23 s emulator boot. Both halves had
avoidable work.
Compile: the dedup only collapsed builds still IN FLIGHT, so an
identical rebuild cost the same as a cold one (28 s cold, 27 s warm).
Successful results are now stored under the build volume keyed by the
same content hash, LRU-pruned, and served as an already-finished job so
the client's poll loop is unchanged. Gallery examples are byte-identical
for every visitor, so one build serves everyone: 32.8 s -> 0.6 s.
The key also folds in a fingerprint of the two modules that TURN a
request into build flags, so changing sdkconfig generation invalidates
every stale binary automatically instead of serving images built with
the old flags. Failed builds are never cached.
Boot: CONFIG_SPIRAM_MEMTEST defaults on and walks all 32 MB before
app_main - tens of millions of guest accesses to test RAM that is
modelled, not physical. Off for both the P4 hex and the classic
quad/octal chunks: 35M -> 20M guest instructions to LCD-READY.
End to end on staging, ESP32-P4 MIPI-DSI example: 51 s -> 11 s.
nginx 504s a cold ESP-IDF build long before it finishes; Starlette then
cancels the handler and the cancellation killed the build subprocess
MID-WRITE, leaving truncated .obj files in the persistent per-target
build cache - every later build of that target failed with
'ranlib: file truncated' until the cache was wiped (bitten twice on
staging the day the P4 lane landed). asyncio.shield lets the build run
to completion with a consistent cache; only the response is lost, and
the async /start + /status flow remains the right path for long builds.
One Semaphore(2) gated every compile, so while two ESP32 builds ran (cmake +
ninja, minutes on a cold cache) an Arduino Uno blink sat 'compiling' for as
long as they took. Two lanes now: heavy (ESP-IDF, still 2 — the VPS load
constraint is real) and light (arduino-cli boards, 3), plus a console line
when a job is queued so the wait is visible instead of looking like a hung
build. The per-target lock that keeps concurrent builds out of one ESP-IDF
build dir is unchanged.
A MicroPython library in Velxio IS a .py file in the project, copied onto
the board's filesystem before main.py runs. That mechanism worked, but
getting the file meant hunting GitHub by hand — the gap issue #214 is
about. Now the Library Manager fills it, in both halves:
Backend: /api/micropython-libs proxies the official micropython-lib
package index (the same one `mip` uses on real hardware). /search filters
the index; /fetch resolves a package plus its declared deps into .py
files. The proxy is what makes this safe to expose: only the official
index is reachable (the name must exist in index.json, files come from its
content-addressed store), source only (.py, never .mpy bytecode), path-
safe, and capped at 24 files / 256 KB per install. In-process TTL cache
over the index.
Frontend: the same Library Manager modal, mode-aware. On a MicroPython
board the search hits the index and "Add to project" writes the package's
files into the board's workspace (upsert — re-adding is the upgrade path),
visible in the explorer and travelling with the project. Before typing it
shows featured picks (ssd1306, umqtt.*, urequests) and the modules already
frozen into the firmware (neopixel, dht, onewire, ds18x20, machine,
network) labeled "built into the firmware", so searching for one answers
"just import it" instead of an empty list.
The file-materialization prelude now creates package directories: umqtt
ships as umqtt/simple.py, and MicroPython's open() does not mkdir, so the
prelude died with ENOENT on any packaged module.
Also ships a gallery example ("ESP32 + MicroPython: use an external
library") that documents the file-is-the-library mechanism with the
ssd1306 driver on the wired OLED.
Measured against the user corpus, the by-resolved-type long tail was
losing what the old catch-all happened to include: for `Adafruit_SH1106G
display(...); display.setTextSize(` the class itself declares one method
(begin) and everything people call lives four levels up (Adafruit_SH110X ->
Adafruit_GrayOLED -> Adafruit_GFX -> Print). Same story for MCUFRIEND_kbv,
Adafruit_NeoMatrix and every GFX-derived display driver.
Three changes:
- The scanner records `bases` on class symbols (access/virtual keywords
dropped, `ns::Base` -> Base, template args cut). 2,529 classes across the
production cache now carry their bases.
- /type/{name} walks the chain breadth-first through the type index (bases
routinely live in a different library), re-owns inherited members to the
requested type so the editor sees one flat class, tags them with
`inheritedFrom`, and lists the chain in `bases` -- including bases NOT in
the cache (Print, Stream, or an embedded one like Adafruit_GFX) so the
host can supply those itself. Depth capped at 8. Adafruit_SH1106G goes
from 1 member to 64.
- `void a(int), b(), c(char);` declares three methods; the extractor took
only the first. TFT_eSPI declares setTextColor/setTextSize and
setCursor/getCursorX/... this way. Split on top-level commas after a
complete `name(args)` group. TFT_eSPI 118 -> 173 members.
Rescan of the 1,125-entry production cache: +274 member symbols, 15
libraries gain, none lose.
The completion-safe long-tail path. Feeding whole library symbol sets into
the editor's candidate pool was measured to make ranking worse (U8g2 alone
injects ~250 module-level class names; ArduinoJson dozens of ARDUINOJSON_*
macros). What does help is answering `client.` once tree-sitter has resolved
`client` to a declared type the embedded catalog does not know -- and for
that the engine needs "the members of type T", not "the symbols of library
L". This endpoint returns exactly that: symbols whose owner is the requested
class, from whichever cached library declares it, in the same payload shape
as /symbols/ so the engine's RemoteSymbolCache stores it unchanged.
Resolution: a class-name -> cache-entry index built once over the library
cache (each per-entry scan is already disk-cached), persisted under
.symbols/_types.json (3,818 classes / 199 KB over the 1,125-entry
production cache; ~1 s to rebuild from cached scans, ~60 s cold), refreshed
after a few hours or when the cache grows. Many classes are declared by more
than one library -- the real one plus forks and bundles vendoring a copy
(PubSubClient inside "ESP8266 Microgear", NTPClient inside "ESP8266 Weather
Station", AccelStepper inside "Arduino Learning Board") -- so the pick is:
the library named after the class, else the one declaring the most members
of it (vendored copies are trimmed), else alphabetical.
Generic, not pro-specific: any host with a library cache can use it.
`_prepare_persistent_project_dir` keys the persistent build dir on
(idf_target, variant), but the lock was keyed on the FQBN. Several FQBNs
resolve to one variant — arduino-esp32's boards.txt gives both `esp32` and
`esp32cam` build.variant=esp32 — so those two were free to build in the SAME
directory at the same time.
Measured against staging, compiling both concurrently:
esp32 285504 bytes sha f65ca909… contains its own marker
esp32cam 285504 bytes sha f65ca909… contains the OTHER sketch's marker
Byte-identical firmware, and the esp32cam caller got the esp32 sketch's
binary. In the product that is two gallery tabs open at once: the first tab
prints the second example's serial log and does nothing itself. It made
esp32-pwm-led-rgb look broken (dark LED, every SPICE node 0 V) when the
example is fine — it was running someone else's program.
Key the lock on the same identity as the directory. Boards with genuinely
separate build dirs still compile in parallel up to the semaphore cap; an
unresolvable FQBN falls back to the default pair, which is the safe
direction (serialising more only costs throughput).
Also in this commit: battery-voltage-monitor ships the 9 V battery its own
description promises. The divider's top end was wired to nothing, so A0 read
0 V and the sketch printed "Battery: 0.0 V / WARNING: Battery low!" forever.
GET /api/intellisense/symbols/{Name@Version} scans the content-addressed
library cache entry's headers (dependency-free C++ scanner) into an editor
symbol table, disk-cached per entry and served with long cache headers.
Plain OSS hosts without a lib cache 404 and the editor degrades gracefully.
The ESP32 examples now default to the in-browser JS engines, whose lwIP
serves the sketch's HTTP server from the simulation tab itself. Opening
the IoT gateway in a new tab backgrounds that tab, the browser throttles
its timers, and the in-chip server can't answer before the proxy's
timeout — the exact failure the Pico W iframe path already solved.
- WifiStatus grows an inBrowser flag (set by the overlay's JS bridges);
the serial-monitor link and the WiFi badge now open the in-app iframe
for any board that carries it, not just pi-pico-w.
- /api/gateway polls ~1s before answering 404: the net bridge dials on
'got_ip', which races the sketch printing its URL, so a fast click
used to land before the bridge registered.
useProjectStore.currentExampleId — stamped by loadExample, dropped the
moment a real project (or a blank workspace) takes over. The compile
request and the run event carry it as example_id, analytics only, so
'which examples get compiled and run most' becomes answerable.
Several boards share one FQBN (the Pimoroni RP2350 trio all compile as
rpipico2), so FQBN-keyed metrics can't tell them apart. The compile
request and the run event now carry board_kind — analytics only, rides
into the metric hook's extra.
suppressOnlineOnlyAds(): overlay seam so the hosted build can silence
'available in the online editor' ad cards for items it manages — an
embargoed item must vanish, not advertise the site the user is on.
Pure OSS builds never call it; ads behave exactly as before.
NewsMarkdown (shared renderer): YouTube links render as a thumbnail +
play badge that swaps to the youtube-nocookie iframe on click (nothing
loads from the player until the user opts in); images render inline;
all links open in a new tab. Product decision supersedes the alt-text
image stripping: posts embed screenshots/demos, and the README now
documents that embedded media loads in the viewer's browser.
Security review follow-ups (all LOW):
- proxy: negative cache on upstream failure (no outbound fetch per
request while velxio.dev is unreachable) and a 2MB body cap via
streaming read (httpx timeouts are per read op, not per body)
- NewsModal: OSS builds render markdown images as alt text so viewers'
browsers never contact remote hosts; pro cloud builds show them
Editor shows scheduled product-news posts as a markdown modal: max one
per day, oldest first, each post only ever once (marked seen at
delivery). OSS default source proxies velxio.dev's public feed through
the local backend (cached ~6h, anonymous, VELXIO_NEWS=off to disable)
with seen-state in localStorage; a pro overlay can register a
server-backed per-user source via lib/newsSource.ts.
- Add EmbedBridge.ts PostMessage bridge for Elemes iframe integration
- Add DeployProgress.tsx component for compile/transfer/flash progress UI
- Add DeployProgress.css styling for overlay progress bar
- Modify SerialMonitor.tsx to support BLE/Hardware toggle by Elemes
- Modify vite.config.ts to use VITE_BASE_PATH=/velxio/
- Modify deploy/nginx.conf to remove root redirect
- Update useSimulatorStore.ts with deployState, hwSerialOutput
- Update App.tsx for /editor route
- Add backend endpoints for Elemes auth, projects, dependencies
This commit allows Velxio to run embedded in Elemes lesson pages via iframe
with PostMessage communication for code loading, circuit state, and serial
monitoring.
Lets metrics overlays distinguish agent-triggered compiles from manual
ones (the AI assistant's compiles were previously invisible in project
counters). Loose optional field — omitted means manual.
Cada guest es un proceso QEMU con su propia RAM (1-2 GB segun placa) y
sus hilos de vCPU, asi que el limite lo pone la maquina, no el codigo.
No habia ningun tope: el usuario N simplemente empujaba la caja a swap y
la sesion de TODOS se volvia lenta, que es peor que decirle al usuario N
que espere un minuto.
VELXIO_PI_MAX_INSTANCES (6) guests simultaneos en la maquina
VELXIO_PI_MAX_PER_OWNER (2) guests por persona
El "owner" es el hash de la cookie de sesion: sirve solo para contar, no
se guarda ni se lee de vuelta, y cae al host del cliente cuando no hay
cookie (sidecar de escritorio, tests). Sin identidad solo aplica el tope
global.
El rechazo llega como un mensaje que dice que hacer ("prueba en un
minuto" / "para una de tus sesiones"), no como un fallo mudo, y se
comprueba ANTES de lanzar el proceso.
La placa QEMU-Linux tiene DOS flujos serie y hasta ahora el cableado
usaba el equivocado: el enrutado entregaba los bytes del vecino a la
consola (el shell) y sacaba al cable la cháchara del arranque. El
header, que es lo que el usuario cablea, no existia.
Ahora el canal de protocolo lleva dos ops nuevas:
UARTTX <b64> el guest transmitio por el header -> al canvas
UARTRX el guest pregunta que le llego -> UART_RXQ <b64>
El backend guarda una cola por instancia (acotada a 64 KB, que un script
que no lee nunca no la haga crecer) y el websocket acepta `pi_uart_rx`
con los bytes que el vecino manda. En el frontend el bridge gana
onUartTx / sendUartBytes y el Interconnect engancha ESE flujo en vez de
la consola para las placas Pi.
Con esto el mismo script -- import serial, escribir, dormir, leer --
funciona en los dos motores.
A profile's extra_drive is the same file for everyone; an overlay may
need a disk built for THIS session (what the project declared). New
seam set_pi_extra_drive_resolver(fn) receives the client id, the board
and the start_pi payload and returns raw images, mounted read-only after
the profile's own. The WS route forwards msg_data and the bridge gained
startPayload so a client can declare it. Generic: no package manager,
no OS knowledge in the OSS tree.
- profile key extra_drive: optional read-only second virtio-blk so an
overlay can ship guest-side shim libraries (/dev/vdb)
- SENS <name> protocol op: canvas-fed named values (built-in sensors /
buttons) served from PiInstance.sensor_state, pushed by the frontend
via the new pi_sensor_state WS message
- DISP <b64> protocol op: guest display commands forwarded to the
frontend as 'display' events (built-in screens)
- RaspberryPi3Bridge: onDisplay / onGpioPwm callbacks + setSensorState
- SimulatorCanvas hands piFamily boards their Pi bridge in
attachBuiltins (was ESP32-only)
Adds a third entry to the board language selector next to Arduino C++
and MicroPython: ESP-IDF. In this mode the user writes a plain ESP-IDF
project — app_main() entry point, FreeRTOS + driver APIs — and the
backend compiles it through the same ESP-IDF toolchain it already uses
for ESP32 Arduino sketches, just without the arduino-esp32 component.
Backend:
- CompileRequest.language ('espidf') threaded through the sync + async
compile paths and folded into the dedup job key (language='arduino'
and omitted hash identically so old clients keep dedupping).
- espidf_compiler: pure_idf flag. User files are written into main/
as-is (no Arduino.h wrap, no velxio_compat.h, Arduino library
resolution skipped), ARDUINO_ESP32_PATH is dropped from the build env
and VELXIO_PURE_SKETCH raised so the template CMake compiles the
user's own sources via a glob branch. Pure builds get their own
persistent build-dir variant through the eff_hash fold.
- QEMU WiFi compat for IDF-style code: esp_wifi.h/esp_wifi_init
detection sets has_wifi, and literal #define SSID/PASS plus
wifi_config_t designated initializers are normalized to the QEMU AP.
- CONFIG_ARDUINO_* lines are stripped from sdkconfig.defaults in pure
mode (the symbols don't exist without the arduino component).
Frontend:
- LanguageMode gains 'espidf'; BOARD_SUPPORTS_ESPIDF covers the ESP32
family (Xtensa, S3, C3). Toolbar shows the option only for those.
- Switching modes seeds a main.c blink skeleton (app_main + gpio
driver), mirroring the MicroPython main.py flow.
- compileCode sends language='espidf'; run/stop paths are unchanged
(the QEMU worker consumes the same merged flash image).
- New gallery example: esp32-idf-blink (LED + resistor on GPIO 2).
Tests: unit coverage for the build-env switch, IDF wifi normalization,
job-key variance, file-group seeding and the new example; verified
end-to-end in a container from the prod image (pure build produces a
bootable flash image; Arduino-mode build unchanged, same variant hash).
Move the CYW43439 (Pico W) WiFi emulation out of the open-source tree so it
can ship as a paid feature in a private overlay. OSS keeps a plain Pico W
(no WiFi); the overlay registers the cyw43 protocol + backend network stack
at runtime via generic seams.
Frontend:
- Add simulation/PioPeripheral.ts: a generic "PIO bus peripheral" seam
(feedWord / inDiscardableWriteData / resetFraming / hostWakeLevel /
onHostWake / onSimulationStart). No factory is installed in OSS, so
createPioPeripheral() returns null and a Pico W simulates as a plain Pico.
- RP2040Simulator: keep the fragile PIO-FIFO plumbing (it must re-run after
loadMicroPython swaps the chip) but drive it through PioPeripheral instead
of an inlined cyw43 import (attachCyw43 -> attachPioPeripheral, etc.).
- useSimulatorStore: generic attach/detach + setBoardWifiStatus; drop the
cyw43 bridge map.
- MicroPythonLoader: add registerFirmwareVariant() so an overlay can add the
RPI_PICO_W build; remove the OSS pico-w config + bundled .uf2.
- Delete simulation/cyw43/ (moved to the overlay).
Backend:
- core/hooks.py: add generic register_ws_sim_handler / dispatch_ws_sim_message
and register_gateway_proxy / dispatch_gateway_proxy seams.
- simulation.py: route start_picow / stop_picow / picow_packet_out through the
ws_sim_handler hook (the overlay handles + gates them).
- iot_gateway.py: resolve the Pico W gateway through the gateway_proxy hook.
- Delete services/picow_net/ + picow_net_bridge.py (moved to the overlay).
Tests: move the cyw43/picow suites to the overlay; update RP2040Simulator
mock stubs to attachPioPeripheral.
Two real-world failures hit the Pico W gateway that the headless e2e
(node fetch, tiny request, fast timing) didn't surface:
- A browser sends KILOBYTES of headers (cookies, User-Agent, sec-*).
Forwarded verbatim, the request overran the chip's small recv()
(e.g. recv(1024)); lwIP then RST the connection on close-with-unread-
data, crashing blocking-socket sketches with ECONNRESET. Now we forward
a MINIMAL request (method, path, Host, Connection: close, and
Content-Type/Length for bodies) — nothing a tiny server can choke on.
- After a gateway request completed we dropped the connection immediately,
so a late chip segment (retransmitted FIN / trailing ACK) no longer
matched and fell through to the chip-initiated NAT, which RST it. Add a
short TIME_WAIT: keep the connection briefly and re-ACK late segments so
the chip closes cleanly, never RSTing it.
ESP32 web-server examples are reachable from the browser via
/api/gateway/<client_id>/ (QEMU slirp hostfwd). The Pico W server lives
in the browser-side lwIP, so there was no inbound path: visiting the
chip's IP did nothing.
Add the mirror of tcp_nat.py: tcp_inbound.TcpInbound originates a TCP
connection INTO the chip over the WebSocket bridge (SYN -> SYN+ACK ->
ACK -> request -> response -> FIN), so the backend can fetch a page the
sketch serves on 10.13.37.42:80 and hand it back to the browser.
- bridge.py routes chip TCP segments addressed to a gateway-opened
connection to TcpInbound (before the chip-initiated NAT, which would
RST them); exposes http_into_chip() + ensure_chip_mac() (primes the
chip's gateway ARP).
- iot_gateway.py: same /api/gateway/<client_id>/ route now falls through
to the Pico W bridge when there's no ESP32 instance, builds a raw
HTTP/1.1 request, and parses the chip's response. Same plan gate, same
URL shape — the browser sees no difference between ESP32 and Pico W.
Validated end to end (real RP2040 emulator serving an HTTP page ->
gateway returns it) plus 6 unit tests for the TCP state machine,
response parsing and ARP priming.
Add a working microSD card part backed by a FAT16 image, following the
Wokwi storage model: the project's own workspace files are auto-copied
onto the card (free), and an optional "SD Card" panel uploads extra
files (gated as a paid feature by the velxio.dev overlay; OSS default
allows it).
Frontend (in-browser AVR / RP2040):
- ProtocolParts.ts: rewrite the microsd-card part from a handshake stub
into a real SD-over-SPI device (reply-first Ncr timing, SDSC byte
addressing, single/multi-block read+write, CSD/CID, full CMD set).
- utils/fatImage.ts: dependency-free FAT16 super-floppy builder (8.3 + LFN).
- utils/sdCardFiles.ts: assemble the card image from workspace files plus
uploaded files; base64 helpers.
- components/simulator/SdCardPanel.tsx + ComponentPropertyDialog: upload UI.
- DynamicComponent + useSimulatorStore: build and inject the image on run.
- lib/proSdCardGate.ts: overlay-installable gate for the upload action.
- data/examples-storage-microsd.ts: Arduino Uno + ESP32 gallery examples.
Backend (ESP32 via QEMU):
- services/esp32_sd_slave.py: synchronous SD-over-SPI slave (Python port of
the browser part) with a sparse backing store, idle-state R1 tracking and
real CRC16 on data blocks when the host enables CRC (CMD59) -- both
required by ESP-IDF's sdspi driver.
- esp32_worker.py: route SPI bytes to the slave (returns MISO synchronously)
and feed write-only bulk transfers.
- esp32_lib_manager.py + routes/simulation.py: forward the FAT image
(sd_card.image_b64) from the start config into the worker.
Tested:
- frontend: protocol-parts, fat-image, sd-card-gate and microsd-real-firmware
(real Arduino SD.h on avr8js) -- 86 passing.
- backend: test_esp32_sd_slave (10) covering the ESP-IDF init sequence and
CRC16; validated end to end by running a real SD.h sketch in libqemu-xtensa
(mount, directory listing, read and write-readback).
The last paths that read the shared global /root/Arduino/libraries: a compile
with NO manifest (libraries=null — any from-scratch/anon sketch; the manifest is
never auto-derived from #includes) and the incomplete-manifest scan-all retry
(which re-enters compile unscoped). Both fell through to the global dir,
bypassing the cache entirely (a cached lib still failed when global was gone).
Now, when no scope is materialized, point the library search at the cache: the
cache root is itself a valid Arduino libraries dir (each <name@ver-sha> child is
a library), exposed via env (pro overlay sets them):
- arduino-cli: ARDUINO_DIRECTORIES_USER = VELXIO_FALLBACK_SKETCHBOOK (whose
libraries/ -> cache root) when scope_dir is None.
- ESP-IDF: _find_arduino_libraries_dir() prefers VELXIO_FALLBACK_LIBRARIES_DIR.
Unset (OSS self-host) -> legacy global, unchanged.
Also strip a trailing @version from manifest names (_bare_lib_names): norm_name
fused 'ArduinoJson@6.21.5' -> 'arduinojson6215' (cache miss -> global scan-all);
the per-board boards_json manifest is the path that still carries @version.
The compile scope resolved a project OWNER's per-user custom libraries for ANY
project_id a requester supplied, with no visibility check — so a requester who
knew a victim's PRIVATE project_id + custom-lib name could compile a binary
against the victim's private uploaded library. Replace the ungated
get_project_owner hook with resolve_compile_owner(project_id, requester_id),
which returns the owner ONLY when the requester IS the owner OR the project is
shareable (public/unlisted); otherwise None -> the caller falls back to the
requester's OWN store. Fails closed on any error.
Also gate the server-side manifest fallback by the same rule (symmetry): a
private project's declared library NAMES are no longer read into a non-owner's
compile scope. The owner's own compile and shared/embed compiles of public/
unlisted projects keep their backend-authoritative scope unchanged.
Empirically, arduino-cli's --libraries ADDS to the search path — the global
sketchbook is STILL scanned, so the prior commit did NOT isolate reads from the
shared global volume. Point ARDUINO_DIRECTORIES_USER at the scratch sketchbook
instead (its <scratch>/libraries becomes the ONLY user-library dir); cores +
board-manager URLs live in the DATA dir and are untouched.
In-container functest (AVR uno): a cache-only lib compiles via the scope (never
in global); a global-only lib NOT in the manifest is INVISIBLE to the scoped
compile and only recovers via the scan-all retry (manifest_incomplete=True) —
proving the global volume is no longer scanned for a manifest-scoped build.
arduino-cli compiles now read libraries from a per-compile --libraries dir of
symlinks materialized by the pro overlay (owner store -> cache -> legacy), via
the existing materialize_library_scope hook, instead of scanning the shared
mutable global volume. --libraries only overrides the USER library search path,
so cores + board-manager URLs (RP2040 earlephilhower, ATTinyCore) are untouched.
Mirrors the ESP-IDF P2.1e graceful fallback: an incomplete manifest (a needed or
transitive lib not declared) makes the scoped compile miss a header; we retry
ONCE scan-all (no --libraries) and surface manifest_incomplete, so a partial
manifest degrades to legacy behavior instead of hard-failing.
Dedup correctness: the manifest + owner are resolved ONCE (shared
_resolve_compile_scope) and folded into the /compile/start dedup key AND threaded
into the build, so two owners with identical sketch+board but different custom
libs never coalesce to one another's job, and the key never diverges from the
bytes the build uses. Owner folded only when a manifest applies (index-only
compiles keep cross-owner dedup).
New OSS warm_library hook; /api/libraries/install now calls it (with the
requester id for the anon policy) instead of mutating the shared global
libraries volume — so that volume stops growing and can be retired (P2.5).
Falls back to the legacy arduino-cli global install when no overlay is loaded
(OSS self-host parity).
- compile.py: owner_id = project owner ELSE the requester (so an unsaved
compile resolves the libs the user just uploaded, which are their own);
threaded requester_id into _run_compile from both call sites.
- LibraryManagerModal: on a custom .zip upload, auto-add the lib to the active
board's velxio.json + show the Project tab, so the compile resolves it via the
owner per-user path (the upload now lands in the per-user store, not the
shared dir, so it must be declared to be found).
So a scoped compile can resolve the project OWNER's per-user custom libraries
(not the requester's) — a shared/embed/anon compile of someone else's project
still finds that owner's uploaded libs.
- core/hooks.py: new get_project_owner hook; materialize_library_scope gains an
opaque owner_id param (no-op default unchanged).
- espidf_compiler.compile/_attempt: thread owner_id to the materializer.
- compile.py: resolve owner via get_project_owner(project_id), pass to compile.
Additive: the OSS image (no overlay) ignores owner_id; index libs still resolve
from the cache. Foundation for per-user custom-lib storage (P2.2a write side).
Library manifests are now PER-BOARD (each board carries its own velxio.json),
so two boards in one project can use different (even conflicting) libraries
without clashing — the multi-board extension of the no-clash guarantee.
- board.libraries on BoardInstance + serialisableBoard: rides in boards_json,
so it round-trips, dirty-checks, autosaves and restores natively. This also
removes the load-restore hacks (useLibraryManifestStore + applyProjectManifest
deleted): the manifest is plain board state.
- loadProjectState now restores per-board boardOptions/spiffsFiles/libraries
(it previously dropped them).
- EditorToolbar single + compile-all send the COMPILING board's libraries.
- Backend compile.py prefers the client's per-board request.libraries; the
project-level libraries_json (now the union of all boards) is the fallback.
- buildLoadPayload migrates pre-per-board projects: seed each board with the
project union so they keep compiling scoped.
- Library Manager 'In project' tab edits the ACTIVE board's velxio.json (shows
the board name) and the add field is now an autocomplete (installed libs +
index search) so users pick from a list instead of typing names.
Deletes useLibraryManifestStore.ts + applyProjectManifest.ts.
Adds the get_project_libraries hook: the compile route reads a saved project's
declared library manifest (by project_id) and uses it as the ESP-IDF resolution
scope, preferring it over the client-sent manifest. So a saved project always
compiles against only its own declared libraries — never another user's, or
another project's, stray install in the shared dir — authoritatively from the
server, independent of frontend wiring. Client-sent manifest still used for
unsaved examples; None/empty → legacy scan-all. Overlay fills the hook in
register_pro; OSS default is no-op (None).
A manifest-scoped compile that fails because a header isn't in the manifest
(an undeclared/transitive dependency) now retries once with scan-all, so a
project with an incomplete manifest still compiles instead of regressing.
The response reports manifest_incomplete=true and
manifest_suggested_libraries={header: [candidate lib names]} so the manifest
can be auto-completed (P2.4) or the user prompted to add the missing library.
This de-risks turning on manifest sending (P2.3): an incomplete example/project
manifest can never break a build that worked before.
- compile(): _attempt(allowed) helper; retry scan-all on missing-lib failure.
- _missing_library_headers / _suggest_libraries_for_headers helpers.
- CompileResponse.manifest_incomplete + manifest_suggested_libraries.
When a compile supplies a 'libraries' manifest, _resolve_library_components
merges a USER-installed library only if it's declared in that set. A sketch
therefore never picks up an unrelated library from the shared dir (another
user's install, or a same-named clash) — the manifest is the resolution scope.
- _resolve_library_components(allowed_libraries): gate user-lib merges on
manifest membership; match by folder name OR library.properties name=,
normalised (display name vs on-disk folder differ by separators/case).
Core/bundled libs are never gated. None = legacy scan-all (unchanged).
- Threaded through compile() -> _compile_in_dir.
- compile.py: CompileRequest.libraries; folded into the async dedup _job_key
so a different manifest doesn't dedup to a job built with another.
Opt-in: omitting 'libraries' preserves current behaviour exactly.
Regression: test/backend/unit/test_espidf_core_first.py::TestManifestScope
STM32 emulation (open-core, runs via libqemu-arm in the backend worker):
- backend: stm32_lib_manager + stm32_worker (GPIO, USART, I2C/SPI device models
reusing the ESP32 slaves, live sensor updates), arduino_cli STM32 branch,
start_stm32 simulation route.
- frontend: Stm32Bridge + Stm32BluePill(/BlackPill) web components (Wokwi SVGs),
board kinds, Interconnect/boardPinMapping/boardProtocols wiring, example
projects (blink, serial, I2C BMP280/MPU6050/DS1307/SSD1306/weather, 7-seg,
RGB, button, switch, stepper, cross-board interconnect).
- Raspberry Pi 4/5 board elements + thumbnails.
Pro board gating (generic OSS->Pro seam; entitlement logic lives in the overlay):
- lib/proBoardGate.ts: isProBoardKind (STM32 + every QEMU Raspberry Pi),
installBoardGateImpl/boardGateDecision, triggerProUpgradePrompt.
- PRO badge on those boards in the component picker; gate at the picker add +
the run backstop (startBoard).
- backend/app/services/board_access.py: server-side enforcement seam for the
simulation WebSocket; STM32/Pi unavailable -> Pro-framed message.
- desktop: generic QemuDownloadPrompt + Stm32QemuPrompt (download-behind-license,
mirrors the ESP32 prompt).
- .gitignore: never ship libqemu-* binaries in the public image.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a generic gating hook so a private overlay can restrict the IoT
gateway proxy to paid plans without the OSS image carrying any plan
logic. register_iot_gateway_gate() installs an async callback that
returns None to allow or a detail dict to block; the OSS default (no
overlay) allows everyone, and a failing gate fails OPEN so the gateway
can never be taken down by a buggy overlay.
gateway_proxy() calls the gate first. When blocked it content-
negotiates the 402: browsers (Accept: text/html — the frontend opens
the gateway via window.open) get a small styled upgrade page with a
link to /pricing; programmatic fetch/XHR callers get the JSON detail.
No behaviour change for the open-source image — the gate is a no-op
there.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Brings hardware flashing into Velxio Desktop. Per-board "Flash to
real board" entry in the canvas context menu opens a modal that
enumerates USB serial ports, lets the user pick one, then
streams arduino-cli upload output live until the board is flashed.
Backend (Phase D1) — backend/app/api/routes/flash.py (new):
POST /api/flash/upload (multipart: board_id, port, fqbn,
program_format, program)
→ SSE stream of {phase, line?, progress?} events
→ final {phase:'done', success, elapsed_ms, error?}
- Wraps `arduino-cli upload -p <port> -i <file> --fqbn <fqbn> -v`
so AVR (avrdude), ESP32 (esptool), RP2040 (picotool), SAMD
(bossac) all share one code path — arduino-cli internally
dispatches by FQBN.
- Per-port asyncio.Lock prevents two simultaneous flashes from
fighting over the same /dev/ttyACM0.
- Allow-list of FQBN prefixes (arduino:avr, ATTinyCore:avr,
rp2040:rp2040, esp32:esp32, arduino:samd) so a typo can't
cause a confusing arduino-cli error.
- Format allow-list (hex / bin / uf2 / elf) drives the temp
file extension - arduino-cli uses the extension to route to
the right uploader.
- 8MB hard cap on the uploaded program (real sketches are
well under that; protects against a runaway frontend).
- X-Accel-Buffering: no header so nginx doesn't hold the SSE
chunks until the flash completes.
Frontend (Phase D3):
- frontend/src/services/flashService.ts (new):
async generator streamFlash() yields parsed SSE events.
Handles the base64-vs-text gotcha (compile returns hex_content
as text but binary_content as base64; for binary formats we
atob() into a Uint8Array before posting so the form upload
sends actual bytes, not the base64 ASCII).
- frontend/src/components/simulator/FlashModal.tsx (new):
Three-state UI: picking (port dropdown), flashing (progress
bar + live log), success/error (verdict + retry).
Empty-ports state shows a Linux dialout-group hint.
- SimulatorCanvas.tsx: board context menu gains "Flash to real
board" entry, gated on isTauri() + presence of compiledProgram.
Hidden in web (WebSerial is a separate sprint).
- tauriBridge.ts: SerialPortInfo type + listSerialPorts() helper
that invokes the Rust shell command added in Phase D2.
The sidecar already has arduino-cli on PATH (per
`pro/desktop/sidecar/main.py::_expose_bundled_arduino_cli`), so
no installer changes are needed — flash works the moment the
0.4.x desktop bundle ships with these commits.
Plan + remaining phase tracked in project/hardware-flashing/.
D2 (Rust serial enum) committed separately as a Tauri-shell-only
concern; D4 (manual smoke matrix with real boards) requires
physical hardware so it stays a follow-up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a third format to /api/compile-rom: `c` (C source compiled by SDCC
to Z80 bytes). Same chip-program flow as 8080/Z80 asm — write C in a
project file, click Compile, click Run.
Backend:
- backend/app/services/c_compile.py — async SDCC wrapper. Locates the
sdcc binary on PATH (or via SDCC env var, or common Windows install
paths) and shells out with target=mz80 + --code-loc 0x100 --data-loc
0x8000. Parses the resulting Intel HEX into raw ROM bytes. Pure 8080
is rejected with a clear error (SDCC has no 8080 backend; Z80 ROMs
also run on the i8080-cpu chip if you avoid Z80-only ops).
- rom_compile.py: compile_rom is now async; the new c branch delegates
to c_compile. compile_rom_endpoint awaits it.
Frontend:
- romCompileService: RomFormat gains 'c'; formatForFile maps .c/.cpp to
'c'. isChipProgramFile intentionally still excludes .c — disambiguation
happens at the EditorToolbar level.
- EditorToolbar: the chip-program path also fires when a custom-chip
has programFile === activeFile.name (regardless of extension). That
lets .c files route to /api/compile-rom (SDCC) when bound to a CPU
chip, while .c files NOT bound to any chip continue to route to
arduino-cli as before.
Docker:
- Dockerfile.standalone adds `sdcc` to the apt-get install list, so the
prod image ships with SDCC out of the box.
Example:
- /examples/z80-led-chaser-c — z80-cpu chip + chaser.c (a Larson
scanner written in C with __at() MMIO definitions). Compiles cleanly
with SDCC's --code-loc 0x100 default crt0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a new way to use the retro CPU chips: write your program in a
project file (.s / .asm / .hex / .bin), click Compile, click Run, and
the same chip emulates whatever you wrote. Same chip + different ROMs =
mini PC, calculator, LED demo, Kill-the-Bit game, etc.
SDK:
- velxio-chip.h gets two new host imports:
uint32_t vx_rom_size(void);
void vx_rom_read(uint32_t off, uint8_t* dst, uint32_t len);
CPU-emulator chips call these in chip_setup to pull their program out
of the host's romBytes property.
Frontend runtime:
- ChipRuntime accepts opts.romBytes (Uint8Array) and exposes the new
imports, copying bytes into chip memory on vx_rom_read.
- CustomChipPart pulls component.properties.romBytes (base64) and passes
it through.
- Component registry declares three new custom-chip properties:
romBytes (base64), programFile (matching project filename), and
programTarget (cpu name).
New programmable bundled chip:
- frontend/src/components/customChips/examples/intel/i8080-cpu.{c,chip.json}
Same clean-room 8080 emulator as i8080-repl/i8080-counter, but ROM is
loaded externally via vx_rom_*. Has 8 LEDs, 8 buttons, UART, 16 KB RAM,
32 KB of external ROM.
Backend:
- New /api/compile-rom endpoint and rom_compile service that turns
chip-program source into ROM bytes. 8080 ASM is assembled by the
in-tree two-pass assembler (moved to backend/app/services/asm8080.py).
Intel HEX records are parsed; raw .bin is passed through. Future targets
(z80, 8086, 4004) are scaffolded but not wired yet.
EditorToolbar:
- Compile button detects when the active file is .s/.asm/.hex/.bin and
routes to compile-rom instead of arduino-cli. The compiled bytes are
injected into every custom-chip on the canvas whose programFile property
matches the active filename (or is empty).
Example:
- /examples/i8080-killbits loads Dean McDaniel's 1975 Kill-the-Bit on
the programmable i8080-cpu chip. killbits.s is shipped as a project
file alongside sketch.ino; the user clicks Compile then Run and the
LED walks across 8 outputs, buttons kill it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Backend:
- api/routes/compile.py accepts board-specific compile options
and dedups in-flight identical requests
- services/espidf_compiler.py expanded ESP-IDF wrapper with the new
options surface (sdkconfig.defaults.in
template added)
- services/arduino_cli.py honour the new options envelope
- services/esp32_lib_bridge.py thread board options through to QEMU
Tests:
- tests/test_compile_request_dedup.py end-to-end dedup behaviour
- tests/test_espidf_options.py covers the new options parsing
Frontend:
- services/compilation.ts client-side mirror — sends the new
options field on every compile request
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the public extension points the velxio-prod overlay uses to bind
real canvas-side I2C/SPI/UART models (BME280, future MCP23017, etc.)
to a running Pi guest's protocol shims:
- qemu_manager: set_pi_slave_handler(fn) / get_pi_slave_handler() for
pi_attach_slave + pi_detach_slave WebSocket messages. OSS image
leaves the hook unset so the messages are silently dropped.
- simulation route: parses the two new WS message types and forwards
them to the registered handler when present.
- RaspberryPi3Bridge: attachSlave(spec) / detachSlave(spec) frontend
side of the protocol.
- piSlaveScanner: at simulation start walks components + wires,
identifies I2C/SPI/UART peers wired to Pi protocol pins (40-pin
header physical-pin numbering), and emits one attach per
bus/address pair (deduped across SDA+SCL wires).
- RaspberryPiWorkspace: invokes the scanner once the bridge is open,
with retries to ride out the WS-still-connecting race.
- integration test: pi3_bme280_attach.py boots the Pi, pre-attaches a
BME280 via the slave handler, runs a host-side proto loop, runs
guest python smbus2.read_byte_data(0x76, 0xD0) and asserts the
console reads back CHIP=0x60.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 2 of the OSS / pro split. The hook seams introduced in Phase 1
let stateless routes (compile, libraries, simulation, iot_gateway)
run without the auth/DB stack importable. Now we actually delete the
stack:
app/api/routes/auth.py
app/api/routes/projects.py
app/api/routes/admin.py
app/api/routes/metrics.py
app/models/{user,project,usage_event,password_reset_token}.py
app/schemas/{auth,admin,project}.py
app/core/{dependencies,security}.py
app/database/session.py
app/services/{metrics,odoo_mail,project_files}.py
app/utils/{geo,slug,boards}.py
Private deployments (velxio.dev) get the same modules back via the
velxio-prod overlay: pro/backend/app/api/routes/auth.py etc. are
COPYed onto /app/... at container build time, and register_pro()
includes their routers + registers the lifespan/metrics/auth hooks.
main.py shrank back to the stateless router includes + a single
`run_lifespan_startup()` call. The Phase-1 try-import block that wired
record_compile / get_current_user_id from upstream is gone — those
adapters live in pro now.
Verification:
OSS only: 20 routes (compile, libraries, simulation, gateway).
OSS + pro: 94 routes — identical to pre-refactor velxio.dev.
Net change: -2400 lines from OSS, all of which moved to velxio-prod's
overlay. Self-hosted OSS users lose accounts + project persistence;
the Phase 4 .vlx export/import gives them a portable replacement.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
First phase of the OSS / pro split. Goal: open the seams so the auth/DB/admin
stack can move into the private overlay (Phase 2-3) without the routes that
stay in OSS (compile, libraries, simulation, iot_gateway) having to know.
Backend
-------
* New app/core/hooks.py — registry for record_compile, get_current_user_id,
and lifespan startup tasks. Each hook is a no-op by default; overlays
call register_* in register_pro(app) to plug in a real implementation.
* compile.py now imports only from app.core.hooks. Drops the direct deps on
app.core.dependencies, app.database.session, app.models.user, and
app.services.metrics. Route signatures use `Depends(get_current_user_id)`
instead of `Depends(get_current_user)`; the metric helper passes user_id
through rather than a User instance.
* compile_chip.py drops the unused _current_user Depends entirely.
* main.py wraps the auth/DB stack import in try/except. When it succeeds
(today's behavior on velxio.dev), an adapter bridges record_compile and
get_current_user_id to the existing app.services.metrics + dependencies,
and the create_all + ALTER TABLE migration block runs via a registered
lifespan_startup hook. When it fails (the post-Phase-2 OSS image), main
logs "running stateless" and skips registering anything — the routes
still load and behave as no-ops for metrics + always-anonymous for auth.
Frontend
--------
* useAutoSaveProject becomes a skeleton: one useState + one useEffect that
delegates to an installed AutoSaveImpl. installAutoSaveImpl() replaces
the impl without changing hook count, so React's rules-of-hooks stay
satisfied even after the impl moves out of OSS.
* New hooks/autoSaveImpl.ts holds the original logic (debouncing, dirty
detection, owner eligibility, fetch keepalive on unload), refactored to
emit() instead of useState. It self-registers at module load; main.tsx
imports it for the side effect.
* AppHeader wraps the entire user-vs-login UI in a data-velxio-slot
="header-auth" boundary. Today the OSS UI still renders inside the slot
— the overlay can portal-inject additional items now, and in Phase 3
the slot becomes the sole owner of header auth UX.
Behavior is identical on velxio.dev (pro overlay imports everything
successfully, every adapter wires up). The change is purely structural:
deleting the auth/DB modules tomorrow no longer crashes OSS at import.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the register-then-immediately-forgot race for good. Two parallel
calls to /velxio/api/send-welcome and /velxio/api/send-password-reset
were hitting Odoo's REPEATABLE-READ snapshot timing: both workers took
their snapshot BEFORE either had committed the partner row, so each
ran its own INSERT and the second blew up on the velxio_user_id
unique constraint — costing one of the two emails.
Add a new sync_partner() helper that POSTs to a new Odoo route
/velxio/api/upsert-partner (lives in velxio_subscription) which only
upserts the partner — no mail, no subscription, fast. Register and
forgot-password await this BEFORE firing the async welcome / reset
tasks. Each user's flow is sequential at the Velxio HTTP layer, so
the snapshot race disappears.
sync_partner() reuses the existing _post() error-swallowing pattern.
If Odoo is down, sync_partner returns None and the await is a no-op
— the user still registers / gets the generic 200, the welcome /
reset endpoints retain their defensive upsert as a fallback path.
a2nr
David Montero