Commit Graph

266 Commits

Author SHA1 Message Date
David Montero Crespo 0368d24aab fix(espidf): broaden merged-component REQUIRES to the IDF components FastLED reaches
esp_cache.h (esp_mm) was the next one-header-at-a-time failure after
esp_http_server. Scan FastLED 3.10's esp platform tree once and add the
full candidate set (esp_mm, esp_hw_support, heap, esp_system, esp_rom,
log, app_update, bt, esp_psram, esp_pm) — every entry remains guarded on
the component existing in the IDF tree, so older IDF generations are
unaffected.
2026-08-19 20:03:26 +02:00
David Montero Crespo 147694f2d7 fix(espidf): demote nested junk-named dirs to header-only; require esp_http_server
Two more merge gaps surfaced by FastLED 3.10 + U8g2 in one sketch:

- src/fl/test/ was fully dropped because 'test' matched the generic
  excluded_dirs check at ANY depth, but FastLED's unity build includes
  "fl/test/_build.cpp.hpp". Repo-level dirs named test/examples/docs
  stay fully excluded; the same names nested inside a src-layout tree are
  now header-only (fragments copied, sources still never compiled).

- FastLED's fl/net remote code includes <esp_http_server.h>, which the
  Arduino IDE provides via the core's global include path. Add
  esp_http_server to the merged component's guarded REQUIRES list.
2026-08-19 19:56:45 +02:00
David Montero Crespo b6e84c5d71 fix(espidf): copy extras/ header fragments into merged libs (never their sources)
FastLED's unity build (src/fl/build/extras+.cpp) includes
"extras/_build.cpp.hpp", but extras/ was in the excluded_dirs set so the
fragment was never copied and the merged-component build died with
'No such file or directory'. Treat extras/ as header-only instead: copy
.h/.hpp/.inl fragments so unity includes resolve, but keep compiling
nothing from it — extras/ traditionally holds repo-only content.
2026-08-19 19:51:10 +02:00
David Montero Crespo 39b2ccf515 fix(espidf): keep dirs shipping an Arduino.h shadow header off the merged -I path
FastLED ships a host-test stub src/platforms/stub/Arduino.h. When another
merged library (e.g. U8g2) does #include <Arduino.h>, the stub resolved
first and broke the build: 'yield' was not declared in U8g2lib.cpp and
fl::string polluted String overload resolution inside the core WString.h.
Reuse the _SHADOW_STD_HEADERS mechanism so the stub's directory stays off
the global include path; FastLED reaches it via a file-relative include.
2026-08-19 19:41:23 +02:00
David Montero Crespo 3558d9ba39 feat(espidf): honor CDCOnBoot menu option in 4-part FQBNs
arduino-cli FQBNs may carry a menu-option suffix
(vendor:arch:board_id:Menu1=val1,...). The espidf lane previously derived
cdc_on_boot exclusively from the board id's boards.txt default, so a
generic-devkit board could never be built with Serial on the USB
Serial/JTAG CDC port.

- _fqbn_board_id_and_options(): split board id and menu options, keeping
  the old last-segment behaviour for plain 3-part FQBNs.
- _arduino_board_flags(): CDCOnBoot=cdc overlays cdc_on_boot=1 on the
  boards.txt base flags (mirroring <id>.menu.CDCOnBoot.cdc.build.
  cdc_on_boot=1); the per-board cache stores only base flags, overrides
  are applied on a copy.
- _arduino_variant(): board id extraction goes through the same helper so
  a menu suffix no longer corrupts the variant lookup.

First user: the ESP32-C3-LCDkit board def pins
esp32:esp32:esp32c3:CDCOnBoot=cdc — the real kit has no UART bridge, its
only console is the USB Serial/JTAG port, and the esp32c3js engine now
models the always-attached host that HWCDC's connection gate requires
(esp32c3js v1.8.0).
2026-08-19 16:39:59 +02:00
David Montero Crespo bf81ac4ae5 feat(compile): esp32p4 defaults to PSRAM enabled
The Function-EV module always carries 32 MB AP-hex PSRAM and esp_lcd
allocates DSI framebuffers from it unconditionally - without it the
board's flagship peripheral cannot run. Explicit psram=disabled still
wins; the UI only sends the field after the user opens the
board-options modal. Validated: an optionless espidf compile now embeds
esp_psram.
2026-08-19 10:47:37 +02:00
David Montero Crespo eb8d86e1aa feat(compile): enable PSRAM for esp32p4 with the P4's hex-mode symbols
The P4 gate ('until the engines validate it') can flip: esp32p4js
v0.3.3 models the AP hex PSRAM + MSPI1 command controller. The classic
SPIRAM chunk emitted S3-era symbols (SPEED_80M, MODE_QUAD/OCT) that do
not exist in the P4's Kconfig.spiram - it gets its own chunk: MODE_HEX
+ SPEED_200M (tree default, rev0-safe; 250M needs rev >= 3). Validated
on staging: an esp_lcd_ek79007 espidf build with psram=enabled boots in
the engine to streaming 1024x600 DSI frames.
2026-08-19 10:44:06 +02:00
David Montero Crespo f606da2cbe feat(compile): resolve registry components from includes in every mode
Generalizes the esp_camera.h lesson into a table (esp_lcd_st77916,
esp_lcd_touch_cst816s, esp_lcd_gc9a01, esp_lcd_ek79007, esp32-camera)
and applies it in the arduino branch, the no-core pure-IDF fallback AND
the explicit language=espidf branch (issue #139) - the last one is where
the VoCat display example lives and previously died at the first
include. Validated on staging: the ST77916 colour-bars espidf example
now fetches espressif/esp_lcd_st77916 from the registry and compiles.
2026-08-19 07:51:10 +02:00
David Montero Crespo 2656376cd5 fix(compile): shield the sync endpoint from client disconnects
nginx 504s a cold ESP-IDF build long before it finishes; Starlette then
cancels the handler and the cancellation killed the build subprocess
MID-WRITE, leaving truncated .obj files in the persistent per-target
build cache - every later build of that target failed with
'ranlib: file truncated' until the cache was wiped (bitten twice on
staging the day the P4 lane landed). asyncio.shield lets the build run
to completion with a consistent cache; only the response is lost, and
the async /start + /status flow remains the right path for long builds.
2026-08-19 07:29:26 +02:00
David Montero Crespo 010a112bc8 feat(compile): ESP32-P4 and ESP32-C5 targets in the espidf lane
_idf_target mapped only c3/c6/s3 and defaulted everything else to
'esp32', so a P4 fqbn compiled with the XTENSA toolchain and
-DARDUINO_ESP32P4_DEV. Adds p4/c5 predicates, IDF5-target membership,
0x2000 bootloader offsets, the riscv toolchain requirement, PSRAM held
off until the engines validate it, P4 CPU-freq drops (its kconfig
choices do not overlap the classic set), a rev0 sdkconfig pin for the P4
(the in-browser engine runs the only published mask ROM and v3 firmware
reads a flash descriptor rev0 never wrote), and an honest refusal for C5
arduino builds (no core exists). Validated on staging: an esp32p4
arduino sketch now produces a merged image with the riscv32 toolchain
that the esp32p4js engine accepts.
2026-08-19 06:35:33 +02:00
David Montero 33ee270ae1 compile: arduino-cli builds no longer queue behind ESP-IDF builds
One Semaphore(2) gated every compile, so while two ESP32 builds ran (cmake +
ninja, minutes on a cold cache) an Arduino Uno blink sat 'compiling' for as
long as they took. Two lanes now: heavy (ESP-IDF, still 2 — the VPS load
constraint is real) and light (arduino-cli boards, 3), plus a console line
when a job is queued so the wait is visible instead of looking like a hung
build. The per-target lock that keeps concurrent builds out of one ESP-IDF
build dir is unchanged.
2026-08-18 22:08:54 +02:00
David Montero Crespo d7a4b5e4eb feat(micropython): install libraries from the micropython-lib index (#214)
A MicroPython library in Velxio IS a .py file in the project, copied onto
the board's filesystem before main.py runs. That mechanism worked, but
getting the file meant hunting GitHub by hand — the gap issue #214 is
about. Now the Library Manager fills it, in both halves:

Backend: /api/micropython-libs proxies the official micropython-lib
package index (the same one `mip` uses on real hardware). /search filters
the index; /fetch resolves a package plus its declared deps into .py
files. The proxy is what makes this safe to expose: only the official
index is reachable (the name must exist in index.json, files come from its
content-addressed store), source only (.py, never .mpy bytecode), path-
safe, and capped at 24 files / 256 KB per install. In-process TTL cache
over the index.

Frontend: the same Library Manager modal, mode-aware. On a MicroPython
board the search hits the index and "Add to project" writes the package's
files into the board's workspace (upsert — re-adding is the upgrade path),
visible in the explorer and travelling with the project. Before typing it
shows featured picks (ssd1306, umqtt.*, urequests) and the modules already
frozen into the firmware (neopixel, dht, onewire, ds18x20, machine,
network) labeled "built into the firmware", so searching for one answers
"just import it" instead of an empty list.

The file-materialization prelude now creates package directories: umqtt
ships as umqtt/simple.py, and MicroPython's open() does not mkdir, so the
prelude died with ENOENT on any packaged module.

Also ships a gallery example ("ESP32 + MicroPython: use an external
library") that documents the file-is-the-library mechanism with the
ssd1306 driver on the wired OLED.
2026-08-18 21:50:10 +02:00
David Montero 8abed79ad9 fix(arduino-cli): library search/install failures become one actionable sentence
arduino-cli reports errors as plain text or, with --format json, as a JSON
object on stderr. We forwarded that blob verbatim, so on a fresh install
with no library index yet (and no way through the firewall/proxy to
downloads.arduino.cc) the in-app agent told the user "the library server
returns a JSON error" ten times in a row.

humanize_cli_error() maps the known shapes — index download failed,
library zip download failed, library not found — to a single sentence
that says what to check, and keeps the raw text under raw_error. Applied
to search_libraries (rc != 0 and non-JSON stdout) and install_library.
Tests use the real arduino-cli 1.0.4 output captured offline.
2026-08-18 00:28:07 +02:00
David Montero 7d63209d1e fix(espidf): ccache hash_dir=false so build variants share objects; log ninja/cmake timeouts
ESP-IDF compiles with -ggdb, and for debug builds ccache folds the cwd into
every cache key by default — CCACHE_BASEDIR does not exempt it (verified with
CCACHE_DEBUG on 4.11: 'Hashing CWD /var/lib/velxio-build/esp32s3/v_<hash>/
project/build'). Each build variant lives in its own dir, so every variant had
a private key space: eight esp32 variants of the SAME board each recompiled
all ~2100 IDF objects from scratch, the cache filled with dead per-directory
entries, and a cold Cardputer/S3 build (2263 steps) overran the 600s ninja
timeout on a busy box. CCACHE_NOHASHDIR=1 makes variants of the same
board/sdkconfig share objects; a cold variant becomes mostly cache hits. Only
DW_AT_comp_dir in .debug_info of a reused object can differ, which never
reaches the flashed .bin.

The timeout paths were silent server-side; they now log the build dir and how
many ninja steps completed, and the user-facing message says the partial
build is kept and a retry resumes.
2026-08-17 19:15:28 +02:00
David Montero 9118529d10 feat(intellisense): /type/ follows inheritance; extractor records bases and splits comma declarators
Measured against the user corpus, the by-resolved-type long tail was
losing what the old catch-all happened to include: for `Adafruit_SH1106G
display(...); display.setTextSize(` the class itself declares one method
(begin) and everything people call lives four levels up (Adafruit_SH110X ->
Adafruit_GrayOLED -> Adafruit_GFX -> Print). Same story for MCUFRIEND_kbv,
Adafruit_NeoMatrix and every GFX-derived display driver.

Three changes:

- The scanner records `bases` on class symbols (access/virtual keywords
  dropped, `ns::Base` -> Base, template args cut). 2,529 classes across the
  production cache now carry their bases.

- /type/{name} walks the chain breadth-first through the type index (bases
  routinely live in a different library), re-owns inherited members to the
  requested type so the editor sees one flat class, tags them with
  `inheritedFrom`, and lists the chain in `bases` -- including bases NOT in
  the cache (Print, Stream, or an embedded one like Adafruit_GFX) so the
  host can supply those itself. Depth capped at 8. Adafruit_SH1106G goes
  from 1 member to 64.

- `void a(int), b(), c(char);` declares three methods; the extractor took
  only the first. TFT_eSPI declares setTextColor/setTextSize and
  setCursor/getCursorX/... this way. Split on top-level commas after a
  complete `name(args)` group. TFT_eSPI 118 -> 173 members.

Rescan of the 1,125-entry production cache: +274 member symbols, 15
libraries gain, none lose.
2026-08-17 19:08:57 +02:00
David Montero 1a112324ab feat(intellisense): GET /api/intellisense/type/{name} — members of one class
The completion-safe long-tail path. Feeding whole library symbol sets into
the editor's candidate pool was measured to make ranking worse (U8g2 alone
injects ~250 module-level class names; ArduinoJson dozens of ARDUINOJSON_*
macros). What does help is answering `client.` once tree-sitter has resolved
`client` to a declared type the embedded catalog does not know -- and for
that the engine needs "the members of type T", not "the symbols of library
L". This endpoint returns exactly that: symbols whose owner is the requested
class, from whichever cached library declares it, in the same payload shape
as /symbols/ so the engine's RemoteSymbolCache stores it unchanged.

Resolution: a class-name -> cache-entry index built once over the library
cache (each per-entry scan is already disk-cached), persisted under
.symbols/_types.json (3,818 classes / 199 KB over the 1,125-entry
production cache; ~1 s to rebuild from cached scans, ~60 s cold), refreshed
after a few hours or when the cache grows. Many classes are declared by more
than one library -- the real one plus forks and bundles vendoring a copy
(PubSubClient inside "ESP8266 Microgear", NTPClient inside "ESP8266 Weather
Station", AccelStepper inside "Arduino Learning Board") -- so the pick is:
the library named after the class, else the one declaring the most members
of it (vendored copies are trimmed), else alphabetical.

Generic, not pro-specific: any host with a library cache can use it.
2026-08-17 18:51:08 +02:00
David Montero 15126cd27a fix(intellisense): access labels apply to statements that open a nested brace
_scan_text applied `public:` / `private:` / `protected:` only inside
handle_statement, which the `{` branch skipped whenever the statement opened
a nested enum, struct or class. So `public:\n typedef enum { ... }` -- the
first thing in AccelStepper's class body, and a common idiom -- took the enum
branch, the label was never seen, the class stayed in its default private
access, and every method after the enum was dropped. AccelStepper shipped 4
of its public methods; DallasTemperature, SoftWire, MFRC522_I2C, RTCZero
shipped none.

Labels are now consumed before the branch. Rescanned all 1,125 entries of
the production library cache: 82 libraries gain members (+1,532 total,
DallasTemperature 0 -> 57), and the 20 that "lose" one or two were leaking
private/protected members past the old check (PMS::loop is under private:,
hd44780::waitReady under protected:) -- precision, not regression.
2026-08-17 18:47:46 +02:00
David Montero Crespo 5545089389 fix(compile): lock the build DIRECTORY, not the FQBN — concurrent compiles swapped binaries
`_prepare_persistent_project_dir` keys the persistent build dir on
(idf_target, variant), but the lock was keyed on the FQBN. Several FQBNs
resolve to one variant — arduino-esp32's boards.txt gives both `esp32` and
`esp32cam` build.variant=esp32 — so those two were free to build in the SAME
directory at the same time.

Measured against staging, compiling both concurrently:

  esp32     285504 bytes  sha f65ca909…  contains its own marker
  esp32cam  285504 bytes  sha f65ca909…  contains the OTHER sketch's marker

Byte-identical firmware, and the esp32cam caller got the esp32 sketch's
binary. In the product that is two gallery tabs open at once: the first tab
prints the second example's serial log and does nothing itself. It made
esp32-pwm-led-rgb look broken (dark LED, every SPICE node 0 V) when the
example is fine — it was running someone else's program.

Key the lock on the same identity as the directory. Boards with genuinely
separate build dirs still compile in parallel up to the semaphore cap; an
unresolvable FQBN falls back to the default pair, which is the safe
direction (serialising more only costs throughput).

Also in this commit: battery-voltage-monitor ships the 9 V battery its own
description promises. The divider's top end was wired to nothing, so A0 read
0 V and the sketch printed "Battery: 0.0 V / WARNING: Battery low!" forever.
2026-08-17 09:12:26 +02:00
David Montero b58ecd1c37 fix(espidf): generate .ino forward declarations + link the task-WDT API
The custom sketch.ino.cpp path never ran arduino-cli's ctags prototype
pass, so the universal Arduino idiom (helpers defined after loop()) died
with "'foo' was not declared in this scope" on every ESP32 build. A
conservative generator (skips templates, default args, methods,
function-pointer params) injects prototypes before the first definition
with a #line directive so error line numbers stay put. 8 unit tests.

IDF 5.5 renamed ESP_TASK_WDT to the EN/INIT pair; the v4.4 template's =n
became EN=n, compiling the whole implementation out — every sketch using
esp_task_wdt_add/reset (Firebase's client does internally) failed at
link. The v5 fixup now emits EN=y INIT=n: the API links, the watchdog
never auto-starts under emulation. v4.4 semantics untouched.
2026-08-15 08:29:39 +02:00
David Montero Crespo 63a91a08ef feat(backend): intellisense symbol server for the library long tail
GET /api/intellisense/symbols/{Name@Version} scans the content-addressed
library cache entry's headers (dependency-free C++ scanner) into an editor
symbol table, disk-cached per entry and served with long cache headers.
Plain OSS hosts without a lib cache 404 and the editor degrades gracefully.
2026-08-15 06:46:55 +02:00
David Montero 0f279c901b fix(espidf): enable BLE 4.2 legacy APIs on BLE5-only SoCs
C3/S3/C6 default to CONFIG_BT_BLE_50_FEATURES_SUPPORTED with 4.2 OFF, so
arduino-esp32's Bluedroid BLE library (esp_ble_gap_config_adv_data_raw,
esp_ble_gap_start_advertising) died at link with undefined references —
esp32c3-ble-advertise never compiled. Enabling 42+50 together matches
arduino-esp32's own per-target sdkconfigs; on classic ESP32 these flags
are already the Kconfig defaults. Found by the nightly gallery smoke.
2026-08-15 04:19:21 +02:00
David Montero 0d2e73ee00 fix(espidf): unknown #if must not prune its #else in the include scanner
The dead-branch pruner marked any assumed-live branch as 'taken', so the
#else of an UNKNOWN condition was skipped. GxEPD2 guards its GFX base
class with '#if ENABLE_GxEPD2_GFX (default 0) ... #else #include
<Adafruit_GFX.h>' — the pruned #else silently dropped the Adafruit_GFX
dependency and every GxEPD2 e-paper example failed with 'fatal error:
Adafruit_GFX.h: No such file'. Found by the nightly gallery compile smoke.

'taken' is now set only by PROVABLY-true conditions (all identifiers in
the known-true set), keeping the 2026-08-03 ESPAsyncWebServer behaviour:
provably-dead branches still prune, provably-true branches still kill
their #else.
2026-08-15 04:14:00 +02:00
David Montero Crespo 8391cc7fa8 fix(gateway): keep in-browser WiFi boards reachable through the IoT gateway
The ESP32 examples now default to the in-browser JS engines, whose lwIP
serves the sketch's HTTP server from the simulation tab itself. Opening
the IoT gateway in a new tab backgrounds that tab, the browser throttles
its timers, and the in-chip server can't answer before the proxy's
timeout — the exact failure the Pico W iframe path already solved.

- WifiStatus grows an inBrowser flag (set by the overlay's JS bridges);
  the serial-monitor link and the WiFi badge now open the in-app iframe
  for any board that carries it, not just pi-pico-w.
- /api/gateway polls ~1s before answering 404: the net bridge dials on
  'got_ip', which races the sketch printing its URL, so a fast click
  used to land before the bridge registered.
2026-08-14 19:45:39 +02:00
David Montero efabac0a05 fix(compile): strip @version/@wokwi pin suffixes in manifest name matching 2026-08-14 18:20:27 +02:00
David Montero Crespo ae70595239 fix(compile): user_libs_all gets the -Werror relaxations too
The main component had them; the merged user-libraries component did not, so
a warning INSIDE a vendor library was still fatal. M5Stack's MahonyAHRS.cpp
(fast inverse square root, a float read through a long*) fails under
-Werror=uninitialized, which made installing the vendor's own library kill
every build using it. Library code the user cannot edit must not stop their
build; the diagnostics still print as warnings.
2026-08-12 14:38:49 +02:00
David Montero Crespo 81ca6314f4 fix(compile): -Wno-error=uninitialized — M5Stack's own MahonyAHRS trips it
The library's fast inverse square root reads a float through a long*, GCC
flags y as used uninitialized, and -Werror turns a vendor library the user
cannot edit into a failed build. Demoted to a warning like its siblings.
2026-08-12 14:31:53 +02:00
David Montero Crespo 30a61c7807 fix(compile): stop -Werror rejecting vendor example code
Two of M5Stack's own published Cardputer examples fail to build here and
compile fine in the Arduino IDE: SSHClient passes an int to a %lu, and
mic_wav_record has a local GCC thinks may be used uninitialized. ESP-IDF
promotes both categories to errors, which is right for IDF itself and wrong
for a sketch a user pasted from the vendor's repository — the emulator looks
broken for code that is not.

format and maybe-uninitialized join the existing relaxation list, which
already carries comment, parentheses, sign-compare and the rest for exactly
this reason. They stay warnings, so the diagnostic is still printed; -Wall is
untouched.
2026-08-12 06:30:09 +02:00
David Montero Crespo 044b619d18 fix(ci): cap mcp below 2.0 — the backend suite has not run since 2026-07-29
mcp 2.0.0 shipped on 2026-07-28 and deleted mcp.server.fastmcp, the module
app/mcp/server.py imports. requirements.txt asked for `mcp>=1.0.0` with no
ceiling, so the very next CI run resolved the new major and died importing
test_mcp_tools.py.

That failure is a COLLECTION error, not a test failure: pytest aborts with
exit 2 before running anything, so all 295 unit tests have been silently
unrun on every push since. The workflow was red, but red for a reason
nobody's code caused.

The deploy venv escaped only by accident — it was built while 1.x was still
current, and deploy.sh rebuilds it when requirements.txt changes, so the
next rebuild would have walked into the same wall and taken the deploy gate
down with it.

Verified in a clean venv built from this file: resolves to mcp 1.29.0,
322 passed / 18 skipped, test_mcp_tools.py's 35 tests among them.

Raising the cap means porting to the 2.x server API (mcp.server.mcpserver).
2026-08-10 17:14:39 +02:00
David Montero Crespo 0b875d4440 Merge remote-tracking branch 'origin/master' into v3.2-worktree
# Conflicts:
#	backend/app/services/espidf_compiler.py
#	frontend/src/components/editor/EditorToolbar.tsx
#	frontend/src/utils/vlxFile.ts
2026-08-10 17:00:24 +02:00
David Montero Crespo c0d3543573 feat(metrics): compiles and runs remember which gallery example they came from
useProjectStore.currentExampleId — stamped by loadExample, dropped the
moment a real project (or a blank workspace) takes over. The compile
request and the run event carry it as example_id, analytics only, so
'which examples get compiled and run most' becomes answerable.
2026-08-09 22:34:26 +02:00
David Montero Crespo 8b2715affd feat(metrics): compiles and runs carry the editor BoardKind; ads learn to stay quiet
Several boards share one FQBN (the Pimoroni RP2350 trio all compile as
rpipico2), so FQBN-keyed metrics can't tell them apart. The compile
request and the run event now carry board_kind — analytics only, rides
into the metric hook's extra.

suppressOnlineOnlyAds(): overlay seam so the hosted build can silence
'available in the online editor' ad cards for items it manages — an
embargoed item must vanish, not advertise the site the user is on.
Pure OSS builds never call it; ads behave exactly as before.
2026-08-09 20:48:11 +02:00
David Montero 588c356e7e fix(espidf): hoist merged_libs_report to function scope
Declared inside the has_arduino branch, so every successful pure
ESP-IDF (language=espidf) build crashed with UnboundLocalError when the
success return read it. Regression from today's manifest auto-migration
change; the Arduino paths were live-verified but the espidf mode was
not.
2026-08-05 20:13:17 +02:00
David Montero Crespo 64deb204ae feat(news): rich media in posts - inline images/GIFs + YouTube click-to-play previews
NewsMarkdown (shared renderer): YouTube links render as a thumbnail +
play badge that swaps to the youtube-nocookie iframe on click (nothing
loads from the player until the user opts in); images render inline;
all links open in a new tab. Product decision supersedes the alt-text
image stripping: posts embed screenshots/demos, and the README now
documents that embedded media loads in the viewer's browser.
2026-08-05 16:25:08 +02:00
David Montero Crespo 6205e32741 fix(news): harden the feed proxy + honor the self-host privacy promise
Security review follow-ups (all LOW):
- proxy: negative cache on upstream failure (no outbound fetch per
  request while velxio.dev is unreachable) and a 2MB body cap via
  streaming read (httpx timeouts are per read op, not per body)
- NewsModal: OSS builds render markdown images as alt text so viewers'
  browsers never contact remote hosts; pro cloud builds show them
2026-08-05 15:59:03 +02:00
David Montero Crespo 801bb94eab feat(news): What's New modal - product news feed, once-per-user queue
Editor shows scheduled product-news posts as a markdown modal: max one
per day, oldest first, each post only ever once (marked seen at
delivery). OSS default source proxies velxio.dev's public feed through
the local backend (cached ~6h, anonymous, VELXIO_NEWS=off to disable)
with seen-state in localStorage; a pro overlay can register a
server-backed per-user source via lib/newsSource.ts.
2026-08-05 15:38:14 +02:00
David Montero Crespo c9fe6f199d feat(editor): folders - create/delete folder trees in the file explorer
File names may now carry '/' paths; the explorer renders each board group
as a collapsible folder tree (folders first, files in group order, CSS-var
indentation because .fe-file-item pins padding with !important). New-folder
buttons on the board header and per-folder hover actions (new file here,
new subfolder, delete with confirm when files are inside; a folder holding
every file of the group cannot be deleted - groups keep >= 1 file).

Model: names are the source of truth; only EMPTY folders are tracked
(folderGroups per group) so they survive until a file lands in them, then
materialise into the name prefix. Round-trips through .vlx (optional
folderGroups field, laxly validated), the auth stash draft, and
loadProjectState. Examples gain boards[].files so a single board can ship
a full multi-file tree (first entry = main sketch).

Backend: the arduino-cli sketch writer now creates subdirectories for
'/'-pathed names and REJECTS names that resolve outside the sketch dir
('..' traversal) - previously it crashed on the former and wrote outside
the tmp dir on the latter.
2026-08-05 06:48:38 +02:00
David Montero cd67362312 feat(libraries): manifest auto-migration + name-affinity ranking + install-declares
Three-part fix for the library-contamination class of ESP32 compile
failures (a user's project kept pulling unrelated libraries because 79%
of projects compile with an empty manifest, forcing scan-all resolution
over ~1000 shared libraries):

1. Name-affinity ranking in _find_library_for_header: the provider whose
   name matches the header stem wins (exact normalized match > esp32
   arch declaration > contains-stem, alphabetical only as tie-break).
   Previously first-alphabetical won, so any stray library shipping a
   generic header (Servo.h has 6 providers) could hijack the build.

2. Manifest auto-migration: scan-all compiles now report the libraries
   the build really used (manifest_incomplete +
   manifest_suggested_libraries on the compile result); the editor folds
   single-candidate suggestions into the board's declared manifest after
   a green build, so the NEXT compile runs scoped. One green build = one
   migrated project; empty-manifest scan-all behaviour is untouched.

3. Install-declares: the Wokwi import library modal records successful
   installs in the active board's manifest, and .vlx export/import now
   round-trips board.libraries.

Tests: test_espidf_library_ranking.py (9) + library-manifest.test.ts (19).
2026-08-04 06:03:32 +02:00
David Montero 0c65de16ad espidf: preprocessor-aware include scanning (root cause of foreign libraries in user_libs_all)
Reported 2026-08-03: every ESP32 build in a user's project failed with
errors from libraries he never installed, emitted from user_libs_all --
a folder his file list does not even show, in projects he had recreated
from scratch.

Root cause: _detect_external_includes was a bare regex with no
preprocessor state. ESPAsyncWebServer guards <Hash.h> behind
'#if defined(ESP8266) || defined(TARGET_RP2040)', so the compiler never
sees it on ESP32 -- but the scanner reported it anyway. The resolver
then searched the ~1000 shared-cache libraries, matched stemi-hexapod's
src/Hash.h (a hexapod-robot library), merged it WHOLE, and compiled all
of its sources; each failed on its own unrelated includes.

The scanner now tracks #if/#ifdef/#ifndef/#elif/#else/#endif nesting and
strips comments. A branch is skipped ONLY when every identifier in its
condition is a platform macro known to be undefined on ESP32; unknown
macros keep the branch LIVE, because hiding a genuinely needed header
would be a much worse failure than the old over-reporting. Verified
against the real ESPAsyncWebServer shape plus nesting/elif/else/comment
cases (9 new tests, 15 green in the espidf suites).

Complements the platform-header guard from the previous commit, which
stays as defence in depth.
2026-08-04 05:45:24 +02:00
David Montero 7b5d30791c espidf: never resolve generic platform headers against user libraries
Reported by a user whose every ESP32 build failed with errors from
libraries he never installed (SoftwareSerial.h, ArduinoBearSSL.h,
Adafruit_GFX.h, AsyncElegantOTA.h) coming out of user_libs_all, in a
project whose file list doesn't even show that folder.

Traced end to end: ESPAsyncWebServer includes <Hash.h> (an ESP8266-core
header, guarded for that platform, so arduino-esp32's tree has no such
file). The global scan then resolved it to stemi-hexapod's src/Hash.h --
a hexapod-robot library -- merged the WHOLE library into user_libs_all,
and its Serial.cpp / Hash.cpp / Server.cpp / BluetoothLowEnergy.cpp /
ExpansionDriver.cpp were compiled, each failing on its own unrelated
includes. Log line: 'Merging "stemihexapod" into user_libs_all for
<Hash.h>'.

Hash.h and friends now join the never-resolve-against-user-libs set.
They cannot be discovered by the core-tree scan (the core genuinely
lacks them) yet their names are generic enough that some installed
library owns a file by that name.
2026-08-04 05:23:42 +02:00
David Montero Crespo 0d9a26d951 fix(espidf): a header-only library broke every ESP32 build
Installing ArduinoJson made the project stop compiling, with an error that
never mentioned ArduinoJson:

    CMake Error at user_libs/user_libs_all/CMakeLists.txt:15
      (target_compile_definitions):
      target_compile_definitions may only set INTERFACE properties on
      INTERFACE targets
    -- Configuring incomplete, errors occurred!

ArduinoJson ships headers only. With no sources, idf_component_register()
builds the merged user-libs component as an INTERFACE library, and PRIVATE
scope on an INTERFACE target is a hard CMake error - so the configure step
died before a single file was compiled. The user saw only a failing build
and reported it as "ArduinoJson can't be found even after it has been
installed" (#124).

Those definitions map alloca/memcpy_P/memcmp_P for library C sources that
use them undeclared; with no sources there is nothing to define, so they are
emitted only when the component actually compiles something. A library that
carries its own .cpp is unchanged - which is why LiquidCrystal_I2C (#257)
worked and hid this.
2026-08-03 07:26:01 +02:00
David Montero Crespo cd2dc2f54b Merge remote-tracking branch 'origin/master' into v3.2-worktree 2026-08-01 04:23:55 +02:00
David Montero d4b0b3d822 espidf: fix architecture guard skipping usable libraries (issue #257)
Two-tier rule replacing the blanket skip:
- 'all' now counts as wildcard ('*'): LiquidCrystal_I2C 2.0.0 declares
  architectures=all and was skipped on every ESP32 build, breaking the
  most popular I2C LCD library with a bare 'No such file or directory'.
- Headers the SKETCH includes directly merge even when the library
  declares a foreign-only architecture (many avr-declared libs are pure
  Wire/SPI code that compiles fine; a truthful compile error beats a
  missing-header one). Transitive pulls keep the HARD guard, which is
  the path that dragged SAMD-only Adafruit_ZeroDMA into ESP32 builds.
2026-07-31 22:04:49 +02:00
David Montero 730ab02bca compile: thread initiated_by through CompileRequest and metrics extras
Lets metrics overlays distinguish agent-triggered compiles from manual
ones (the AI assistant's compiles were previously invisible in project
counters). Loose optional field — omitted means manual.
2026-07-31 20:22:29 +02:00
David Montero Crespo b7a191eaab feat(pi): techo de capacidad para los guests QEMU (multiusuario)
Cada guest es un proceso QEMU con su propia RAM (1-2 GB segun placa) y
sus hilos de vCPU, asi que el limite lo pone la maquina, no el codigo.
No habia ningun tope: el usuario N simplemente empujaba la caja a swap y
la sesion de TODOS se volvia lenta, que es peor que decirle al usuario N
que espere un minuto.

  VELXIO_PI_MAX_INSTANCES   (6)  guests simultaneos en la maquina
  VELXIO_PI_MAX_PER_OWNER   (2)  guests por persona

El "owner" es el hash de la cookie de sesion: sirve solo para contar, no
se guarda ni se lee de vuelta, y cae al host del cliente cuando no hay
cookie (sidecar de escritorio, tests). Sin identidad solo aplica el tope
global.

El rechazo llega como un mensaje que dice que hacer ("prueba en un
minuto" / "para una de tus sesiones"), no como un fallo mudo, y se
comprueba ANTES de lanzar el proceso.
2026-07-29 21:15:04 +02:00
David Montero Crespo 93388c675a fix(pi): perifericos completos en la familia Pi — entradas, buses y pines
Auditoria de "la Pi tiene todo lo de la placa real" con cuatro huecos
encontrados y cerrados:

1) GPIO de entrada en modo Linux: GPIO_IN respondia VAL 0 fijo (stub de
   la fase 2), asi que GPIO.input() leia 0 eternamente aunque el canvas
   empujara el nivel. El backend guarda ahora el ultimo nivel por pin
   (set_pin_state lo escribe) y GPIO_IN contesta de ahi. Los flancos
   (SET) siguen llegando al guest como antes.

2) UART del header hacia otra placa: el shim del rootfs ya hablaba
   `UART <port> TX <hex>` / RX_REQ, pero sin modelo de esclavo el
   backend tragaba los bytes. Ahora TX sin esclavo se emite al canvas
   (uart_tx) y RX_REQ sin esclavo drena la cola que llena pi_uart_rx —
   el mismo protocolo de siempre, sin ops nuevas.

3) El escaner de esclavos I2C/SPI/UART estaba doblemente muerto:
   clasificaba por numero fisico de pin ('3','5','19'...) cuando el
   elemento expone GPIOxx, y su unico llamador era RaspberryPiWorkspace,
   que el terminal unificado reemplazo. Acepta ambos nombres y corre en
   onBooted del store.

4) boardPinToNumber solo mapeaba los pines de la 3/4/5; la Zero, 1B+ y
   2B (mismo header de 40 pines, mismo elemento) se quedaban sin mapa.
2026-07-29 17:05:39 +02:00
David Montero Crespo 508d2e141e feat(uart): el puerto serie del header tambien en modo Linux
La placa QEMU-Linux tiene DOS flujos serie y hasta ahora el cableado
usaba el equivocado: el enrutado entregaba los bytes del vecino a la
consola (el shell) y sacaba al cable la cháchara del arranque. El
header, que es lo que el usuario cablea, no existia.

Ahora el canal de protocolo lleva dos ops nuevas:

  UARTTX <b64>   el guest transmitio por el header -> al canvas
  UARTRX         el guest pregunta que le llego -> UART_RXQ <b64>

El backend guarda una cola por instancia (acotada a 64 KB, que un script
que no lee nunca no la haga crecer) y el websocket acepta `pi_uart_rx`
con los bytes que el vecino manda. En el frontend el bridge gana
onUartTx / sendUartBytes y el Interconnect engancha ESE flujo en vez de
la consola para las placas Pi.

Con esto el mismo script -- import serial, escribir, dormir, leer --
funciona en los dos motores.
2026-07-29 16:53:45 +02:00
David Montero Crespo 84e32ef1f2 feat(qemu): optional restricted guest egress via a single guestfwd tunnel
Guests stay on '-nic none' unless a profile opts in, and opting in does
NOT mean internet: the NIC is user,restrict=on (no route out, no route
to the host LAN) with one guestfwd to whatever command the overlay
configures — a filtering proxy in practice. Keeps 'user code never gets
a raw socket outside' true by construction.
2026-07-29 04:39:03 +02:00
David Montero Crespo 8d4f298f68 feat(qemu): session ceiling for guest instances
A forgotten tab pinned a QEMU process and its RAM for as long as the
browser stayed open. Instances now shut themselves down after
MAX_SESSION_SECONDS (2 h default, env-tunable) and say so on the serial
line instead of vanishing.
2026-07-29 04:33:12 +02:00
David Montero Crespo 78f0181435 feat(qemu): per-session extra drives + start_pi payload passthrough
A profile's extra_drive is the same file for everyone; an overlay may
need a disk built for THIS session (what the project declared). New
seam set_pi_extra_drive_resolver(fn) receives the client id, the board
and the start_pi payload and returns raw images, mounted read-only after
the profile's own. The WS route forwards msg_data and the bridge gained
startPayload so a client can declare it. Generic: no package manager,
no OS knowledge in the OSS tree.
2026-07-29 04:31:21 +02:00
David Montero Crespo 1cdcb5a967 feat(pi-family): built-in peripheral plumbing for overlay QEMU-Linux boards
- profile key extra_drive: optional read-only second virtio-blk so an
  overlay can ship guest-side shim libraries (/dev/vdb)
- SENS <name> protocol op: canvas-fed named values (built-in sensors /
  buttons) served from PiInstance.sensor_state, pushed by the frontend
  via the new pi_sensor_state WS message
- DISP <b64> protocol op: guest display commands forwarded to the
  frontend as 'display' events (built-in screens)
- RaspberryPi3Bridge: onDisplay / onGpioPwm callbacks + setSensorState
- SimulatorCanvas hands piFamily boards their Pi bridge in
  attachBuiltins (was ESP32-only)
2026-07-28 15:06:25 +02:00