velxio/frontend/src/services/compilation.ts

205 lines
6.9 KiB
TypeScript
Raw Normal View History

import axios from 'axios';
import { getApiBase } from '../lib/apiBase';
import type { ESP32BoardOptions, SpiffsFile } from '../types/boardOptions';
export interface SketchFile {
name: string;
content: string;
}
/**
* Per-board build options forwarded to the backend. Only meaningful for
* ESP32 targets — `board_options` is structurally translated into sdkconfig
* knobs and a generated partitions.csv by the ESP-IDF compiler. `spiffs_files`
* (if non-empty) are baked into a SPIFFS partition image via `mkspiffs`.
*/
export interface CompileExtras {
boardOptions?: ESP32BoardOptions;
spiffsFiles?: SpiffsFile[];
// P2.3 — declared library manifest (the loaded example/project's libraries).
// Sent as the ESP-IDF resolution SCOPE; null/omitted = legacy scan-all.
// Ignored by the backend for non-ESP32 (arduino-cli) boards.
libraries?: string[] | null;
feat(esp32): pure ESP-IDF language mode for the ESP32 family (#139) Adds a third entry to the board language selector next to Arduino C++ and MicroPython: ESP-IDF. In this mode the user writes a plain ESP-IDF project — app_main() entry point, FreeRTOS + driver APIs — and the backend compiles it through the same ESP-IDF toolchain it already uses for ESP32 Arduino sketches, just without the arduino-esp32 component. Backend: - CompileRequest.language ('espidf') threaded through the sync + async compile paths and folded into the dedup job key (language='arduino' and omitted hash identically so old clients keep dedupping). - espidf_compiler: pure_idf flag. User files are written into main/ as-is (no Arduino.h wrap, no velxio_compat.h, Arduino library resolution skipped), ARDUINO_ESP32_PATH is dropped from the build env and VELXIO_PURE_SKETCH raised so the template CMake compiles the user's own sources via a glob branch. Pure builds get their own persistent build-dir variant through the eff_hash fold. - QEMU WiFi compat for IDF-style code: esp_wifi.h/esp_wifi_init detection sets has_wifi, and literal #define SSID/PASS plus wifi_config_t designated initializers are normalized to the QEMU AP. - CONFIG_ARDUINO_* lines are stripped from sdkconfig.defaults in pure mode (the symbols don't exist without the arduino component). Frontend: - LanguageMode gains 'espidf'; BOARD_SUPPORTS_ESPIDF covers the ESP32 family (Xtensa, S3, C3). Toolbar shows the option only for those. - Switching modes seeds a main.c blink skeleton (app_main + gpio driver), mirroring the MicroPython main.py flow. - compileCode sends language='espidf'; run/stop paths are unchanged (the QEMU worker consumes the same merged flash image). - New gallery example: esp32-idf-blink (LED + resistor on GPIO 2). Tests: unit coverage for the build-env switch, IDF wifi normalization, job-key variance, file-group seeding and the new example; verified end-to-end in a container from the prod image (pure build produces a bootable flash image; Arduino-mode build unchanged, same variant hash).
2026-07-24 11:37:01 +07:00
// Pure ESP-IDF mode (issue #139): 'espidf' compiles the files as a pure
// ESP-IDF project (user app_main, no arduino-esp32 component). Omitted /
// undefined = classic Arduino sketch compile. ESP32 boards only.
language?: 'espidf';
// Who triggered the compile — 'agent' when the AI assistant's tool did.
// Threads through to backend metrics; omitted = manual user action.
initiatedBy?: 'agent';
}
export interface CompileResult {
success: boolean;
hex_content?: string;
binary_content?: string; // base64-encoded .bin for RP2040
binary_type?: 'bin' | 'uf2';
has_wifi?: boolean; // True when sketch uses WiFi (ESP32 only)
stdout: string;
stderr: string;
error?: string;
core_install_log?: string;
}
feat(compile): async compile + status polling — no more 524 timeouts The synchronous /api/compile endpoint forced one long-lived HTTP request to span the entire build. Cloudflare's 100s edge timeout cuts that off mid-flight for any cold ESP-IDF compile (BMP280 takes 5-7 min on first run). The user-visible symptom was HTTP 524 well before the backend even noticed. Backend (compile.py) - New `POST /api/compile/start` returns `{job_id}` immediately and spawns the actual compile as an asyncio.create_task background. - New `GET /api/compile/status/{job_id}` returns the current job state (`pending` | `running` | `done` | `error`). Each poll completes in milliseconds, far under any edge timeout. - Existing `POST /api/compile/` kept verbatim for backward compatibility (AVR/RP2040 builds finish in seconds and don't trip 524). - Build logic extracted into `_run_compile()` so both paths share one implementation; no duplicated ESP-IDF / arduino-cli branching. - Async path opens its own short-lived DB session via AsyncSessionLocal for metric recording — the request-scoped session is dead by the time the background task finishes. - COMPILE_JOBS dict purges entries 30 minutes after completion so a busy server doesn't grow unboundedly. Frontend (compilation.ts) - compileCode() now: POST /compile/start → poll /compile/status every 2s until state ∈ {done, error}, with a 15-minute client-side cap. - 30s axios timeout per individual call (not per build) so transient network blips during a long compile auto-retry instead of failing. - 404 on /status throws (job expired / server restarted); other poll errors warn and retry. Surfaces structured error responses verbatim so the editor's compile-error panel keeps working unchanged. Limitation: COMPILE_JOBS lives in-process; if velxio ever scales to multiple FastAPI workers this needs to move to Redis or sqlite. Single- instance is fine today. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 10:22:09 +07:00
interface CompileStartResponse {
job_id: string;
}
interface CompileStatusResponse {
state: 'pending' | 'running' | 'done' | 'error';
started_at: number;
finished_at: number | null;
feat(compile): stream live ESP-IDF cmake + ninja output to the console A user reported on Discord: "the Velxio Console doesn't update anything, it just waits until the very end and displays everything in one go". True for the async compile path — /compile/status only carried `state` and the final `result`, so the editor's CompilationConsole stayed empty during the 5-7 minute cold ESP-IDF builds and dumped 1500 lines at once when the build finished. This wires live build output through the whole stack. Backend (espidf_compiler.py) - New _run_with_streaming() helper. When a progress_callback is provided it spawns the subprocess via Popen + stdout/stderr drain threads and invokes the callback line-by-line. When None it falls back to the existing subprocess.run(capture_output=True) one-shot path so the unit-test code that doesn't care about live output is unaffected. - compile() and _compile_in_dir() take an optional ProgressCallback. - _run_cmake / _run_ninja closures now go through _run_with_streaming with that callback. cmake configure (~2-5 s) + ninja (~5-300+ s) both stream now; the ninja output is the one users actually want to watch. Backend (compile.py) - _compile_job seeds COMPILE_JOBS[id]['stdout_buffer'] = '' and defines on_progress_line(line) which appends to it. Buffer capped at 256 KB (tail kept) so a runaway build can't OOM the FastAPI process. - The buffer is preserved on both the success and the error path so late polls still see the log even after state transitions to done/error. - /compile/status now returns the buffer as a `stdout` field. CompileStatusResponse gains the field with default '' so old clients that don't read it still work. Frontend (compilation.ts) - compileCode() takes a 4th argument: optional CompileProgress callback fired every poll while state ∈ {pending, running}. Carries the cumulative stdout (caller computes deltas) plus elapsed seconds. - Surfaces the new `stdout` field of /compile/status and forwards it to the callback. Errors thrown from the callback are swallowed — a faulty UI hook must never break the polling loop. Frontend (EditorToolbar.tsx) - Both compileCode() call sites (Run and Compile-All) now pass an onProgress callback. It tracks `lastStreamedLen` per-compile, splits each new delta on newlines, and appends them as `info`-typed CompilationLog entries via setCompileLogs. The Compile-All flow prefixes each line with the board label so multi-board builds stay readable. - After the build settles, the existing parseCompileResult call still runs and appends the structured analysis on top of the live stream — that's where FAILED-block detection + the `error`-typed entries that drive the auto-switch-to-errors filter live. Net effect on the user complaint: cold ESP-IDF builds now show the ninja [N/1483] progress lines streaming into the console as they happen, instead of staring at an empty panel for 5-7 minutes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 04:36:58 +07:00
stdout: string;
feat(compile): async compile + status polling — no more 524 timeouts The synchronous /api/compile endpoint forced one long-lived HTTP request to span the entire build. Cloudflare's 100s edge timeout cuts that off mid-flight for any cold ESP-IDF compile (BMP280 takes 5-7 min on first run). The user-visible symptom was HTTP 524 well before the backend even noticed. Backend (compile.py) - New `POST /api/compile/start` returns `{job_id}` immediately and spawns the actual compile as an asyncio.create_task background. - New `GET /api/compile/status/{job_id}` returns the current job state (`pending` | `running` | `done` | `error`). Each poll completes in milliseconds, far under any edge timeout. - Existing `POST /api/compile/` kept verbatim for backward compatibility (AVR/RP2040 builds finish in seconds and don't trip 524). - Build logic extracted into `_run_compile()` so both paths share one implementation; no duplicated ESP-IDF / arduino-cli branching. - Async path opens its own short-lived DB session via AsyncSessionLocal for metric recording — the request-scoped session is dead by the time the background task finishes. - COMPILE_JOBS dict purges entries 30 minutes after completion so a busy server doesn't grow unboundedly. Frontend (compilation.ts) - compileCode() now: POST /compile/start → poll /compile/status every 2s until state ∈ {done, error}, with a 15-minute client-side cap. - 30s axios timeout per individual call (not per build) so transient network blips during a long compile auto-retry instead of failing. - 404 on /status throws (job expired / server restarted); other poll errors warn and retry. Surfaces structured error responses verbatim so the editor's compile-error panel keeps working unchanged. Limitation: COMPILE_JOBS lives in-process; if velxio ever scales to multiple FastAPI workers this needs to move to Redis or sqlite. Single- instance is fine today. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 10:22:09 +07:00
result: CompileResult | null;
error: string | null;
}
feat(compile): stream live ESP-IDF cmake + ninja output to the console A user reported on Discord: "the Velxio Console doesn't update anything, it just waits until the very end and displays everything in one go". True for the async compile path — /compile/status only carried `state` and the final `result`, so the editor's CompilationConsole stayed empty during the 5-7 minute cold ESP-IDF builds and dumped 1500 lines at once when the build finished. This wires live build output through the whole stack. Backend (espidf_compiler.py) - New _run_with_streaming() helper. When a progress_callback is provided it spawns the subprocess via Popen + stdout/stderr drain threads and invokes the callback line-by-line. When None it falls back to the existing subprocess.run(capture_output=True) one-shot path so the unit-test code that doesn't care about live output is unaffected. - compile() and _compile_in_dir() take an optional ProgressCallback. - _run_cmake / _run_ninja closures now go through _run_with_streaming with that callback. cmake configure (~2-5 s) + ninja (~5-300+ s) both stream now; the ninja output is the one users actually want to watch. Backend (compile.py) - _compile_job seeds COMPILE_JOBS[id]['stdout_buffer'] = '' and defines on_progress_line(line) which appends to it. Buffer capped at 256 KB (tail kept) so a runaway build can't OOM the FastAPI process. - The buffer is preserved on both the success and the error path so late polls still see the log even after state transitions to done/error. - /compile/status now returns the buffer as a `stdout` field. CompileStatusResponse gains the field with default '' so old clients that don't read it still work. Frontend (compilation.ts) - compileCode() takes a 4th argument: optional CompileProgress callback fired every poll while state ∈ {pending, running}. Carries the cumulative stdout (caller computes deltas) plus elapsed seconds. - Surfaces the new `stdout` field of /compile/status and forwards it to the callback. Errors thrown from the callback are swallowed — a faulty UI hook must never break the polling loop. Frontend (EditorToolbar.tsx) - Both compileCode() call sites (Run and Compile-All) now pass an onProgress callback. It tracks `lastStreamedLen` per-compile, splits each new delta on newlines, and appends them as `info`-typed CompilationLog entries via setCompileLogs. The Compile-All flow prefixes each line with the board label so multi-board builds stay readable. - After the build settles, the existing parseCompileResult call still runs and appends the structured analysis on top of the live stream — that's where FAILED-block detection + the `error`-typed entries that drive the auto-switch-to-errors filter live. Net effect on the user complaint: cold ESP-IDF builds now show the ninja [N/1483] progress lines streaming into the console as they happen, instead of staring at an empty panel for 5-7 minutes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 04:36:58 +07:00
/**
* Live progress callback — called on every poll while state ∈ {pending,
* running}. `stdout` is the full live cmake + ninja output captured so far
* (cap of ~256 KB on the server side, tail kept). Caller can compute a
* delta against the previous call if it wants to append-only render.
*/
export type CompileProgress = (info: {
state: 'pending' | 'running';
stdout: string;
elapsedSeconds: number;
}) => void;
feat(compile): async compile + status polling — no more 524 timeouts The synchronous /api/compile endpoint forced one long-lived HTTP request to span the entire build. Cloudflare's 100s edge timeout cuts that off mid-flight for any cold ESP-IDF compile (BMP280 takes 5-7 min on first run). The user-visible symptom was HTTP 524 well before the backend even noticed. Backend (compile.py) - New `POST /api/compile/start` returns `{job_id}` immediately and spawns the actual compile as an asyncio.create_task background. - New `GET /api/compile/status/{job_id}` returns the current job state (`pending` | `running` | `done` | `error`). Each poll completes in milliseconds, far under any edge timeout. - Existing `POST /api/compile/` kept verbatim for backward compatibility (AVR/RP2040 builds finish in seconds and don't trip 524). - Build logic extracted into `_run_compile()` so both paths share one implementation; no duplicated ESP-IDF / arduino-cli branching. - Async path opens its own short-lived DB session via AsyncSessionLocal for metric recording — the request-scoped session is dead by the time the background task finishes. - COMPILE_JOBS dict purges entries 30 minutes after completion so a busy server doesn't grow unboundedly. Frontend (compilation.ts) - compileCode() now: POST /compile/start → poll /compile/status every 2s until state ∈ {done, error}, with a 15-minute client-side cap. - 30s axios timeout per individual call (not per build) so transient network blips during a long compile auto-retry instead of failing. - 404 on /status throws (job expired / server restarted); other poll errors warn and retry. Surfaces structured error responses verbatim so the editor's compile-error panel keeps working unchanged. Limitation: COMPILE_JOBS lives in-process; if velxio ever scales to multiple FastAPI workers this needs to move to Redis or sqlite. Single- instance is fine today. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 10:22:09 +07:00
const POLL_INTERVAL_MS = 2000;
const MAX_POLL_DURATION_MS = 15 * 60 * 1000; // 15 minutes — covers cold ESP-IDF builds
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
/**
* Compile a sketch via the async job pipeline.
*
* POST /compile/start → { job_id }
feat(compile): stream live ESP-IDF cmake + ninja output to the console A user reported on Discord: "the Velxio Console doesn't update anything, it just waits until the very end and displays everything in one go". True for the async compile path — /compile/status only carried `state` and the final `result`, so the editor's CompilationConsole stayed empty during the 5-7 minute cold ESP-IDF builds and dumped 1500 lines at once when the build finished. This wires live build output through the whole stack. Backend (espidf_compiler.py) - New _run_with_streaming() helper. When a progress_callback is provided it spawns the subprocess via Popen + stdout/stderr drain threads and invokes the callback line-by-line. When None it falls back to the existing subprocess.run(capture_output=True) one-shot path so the unit-test code that doesn't care about live output is unaffected. - compile() and _compile_in_dir() take an optional ProgressCallback. - _run_cmake / _run_ninja closures now go through _run_with_streaming with that callback. cmake configure (~2-5 s) + ninja (~5-300+ s) both stream now; the ninja output is the one users actually want to watch. Backend (compile.py) - _compile_job seeds COMPILE_JOBS[id]['stdout_buffer'] = '' and defines on_progress_line(line) which appends to it. Buffer capped at 256 KB (tail kept) so a runaway build can't OOM the FastAPI process. - The buffer is preserved on both the success and the error path so late polls still see the log even after state transitions to done/error. - /compile/status now returns the buffer as a `stdout` field. CompileStatusResponse gains the field with default '' so old clients that don't read it still work. Frontend (compilation.ts) - compileCode() takes a 4th argument: optional CompileProgress callback fired every poll while state ∈ {pending, running}. Carries the cumulative stdout (caller computes deltas) plus elapsed seconds. - Surfaces the new `stdout` field of /compile/status and forwards it to the callback. Errors thrown from the callback are swallowed — a faulty UI hook must never break the polling loop. Frontend (EditorToolbar.tsx) - Both compileCode() call sites (Run and Compile-All) now pass an onProgress callback. It tracks `lastStreamedLen` per-compile, splits each new delta on newlines, and appends them as `info`-typed CompilationLog entries via setCompileLogs. The Compile-All flow prefixes each line with the board label so multi-board builds stay readable. - After the build settles, the existing parseCompileResult call still runs and appends the structured analysis on top of the live stream — that's where FAILED-block detection + the `error`-typed entries that drive the auto-switch-to-errors filter live. Net effect on the user complaint: cold ESP-IDF builds now show the ninja [N/1483] progress lines streaming into the console as they happen, instead of staring at an empty panel for 5-7 minutes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 04:36:58 +07:00
* GET /compile/status/<job_id> (×N) → { state, stdout, result?, error? }
feat(compile): async compile + status polling — no more 524 timeouts The synchronous /api/compile endpoint forced one long-lived HTTP request to span the entire build. Cloudflare's 100s edge timeout cuts that off mid-flight for any cold ESP-IDF compile (BMP280 takes 5-7 min on first run). The user-visible symptom was HTTP 524 well before the backend even noticed. Backend (compile.py) - New `POST /api/compile/start` returns `{job_id}` immediately and spawns the actual compile as an asyncio.create_task background. - New `GET /api/compile/status/{job_id}` returns the current job state (`pending` | `running` | `done` | `error`). Each poll completes in milliseconds, far under any edge timeout. - Existing `POST /api/compile/` kept verbatim for backward compatibility (AVR/RP2040 builds finish in seconds and don't trip 524). - Build logic extracted into `_run_compile()` so both paths share one implementation; no duplicated ESP-IDF / arduino-cli branching. - Async path opens its own short-lived DB session via AsyncSessionLocal for metric recording — the request-scoped session is dead by the time the background task finishes. - COMPILE_JOBS dict purges entries 30 minutes after completion so a busy server doesn't grow unboundedly. Frontend (compilation.ts) - compileCode() now: POST /compile/start → poll /compile/status every 2s until state ∈ {done, error}, with a 15-minute client-side cap. - 30s axios timeout per individual call (not per build) so transient network blips during a long compile auto-retry instead of failing. - 404 on /status throws (job expired / server restarted); other poll errors warn and retry. Surfaces structured error responses verbatim so the editor's compile-error panel keeps working unchanged. Limitation: COMPILE_JOBS lives in-process; if velxio ever scales to multiple FastAPI workers this needs to move to Redis or sqlite. Single- instance is fine today. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 10:22:09 +07:00
*
* Each individual request returns in milliseconds, so Cloudflare's 100s edge
* timeout never kicks in — even when the underlying ESP-IDF cold build runs
* for 5-7 minutes. Falls back to throwing an Error after MAX_POLL_DURATION_MS.
feat(compile): stream live ESP-IDF cmake + ninja output to the console A user reported on Discord: "the Velxio Console doesn't update anything, it just waits until the very end and displays everything in one go". True for the async compile path — /compile/status only carried `state` and the final `result`, so the editor's CompilationConsole stayed empty during the 5-7 minute cold ESP-IDF builds and dumped 1500 lines at once when the build finished. This wires live build output through the whole stack. Backend (espidf_compiler.py) - New _run_with_streaming() helper. When a progress_callback is provided it spawns the subprocess via Popen + stdout/stderr drain threads and invokes the callback line-by-line. When None it falls back to the existing subprocess.run(capture_output=True) one-shot path so the unit-test code that doesn't care about live output is unaffected. - compile() and _compile_in_dir() take an optional ProgressCallback. - _run_cmake / _run_ninja closures now go through _run_with_streaming with that callback. cmake configure (~2-5 s) + ninja (~5-300+ s) both stream now; the ninja output is the one users actually want to watch. Backend (compile.py) - _compile_job seeds COMPILE_JOBS[id]['stdout_buffer'] = '' and defines on_progress_line(line) which appends to it. Buffer capped at 256 KB (tail kept) so a runaway build can't OOM the FastAPI process. - The buffer is preserved on both the success and the error path so late polls still see the log even after state transitions to done/error. - /compile/status now returns the buffer as a `stdout` field. CompileStatusResponse gains the field with default '' so old clients that don't read it still work. Frontend (compilation.ts) - compileCode() takes a 4th argument: optional CompileProgress callback fired every poll while state ∈ {pending, running}. Carries the cumulative stdout (caller computes deltas) plus elapsed seconds. - Surfaces the new `stdout` field of /compile/status and forwards it to the callback. Errors thrown from the callback are swallowed — a faulty UI hook must never break the polling loop. Frontend (EditorToolbar.tsx) - Both compileCode() call sites (Run and Compile-All) now pass an onProgress callback. It tracks `lastStreamedLen` per-compile, splits each new delta on newlines, and appends them as `info`-typed CompilationLog entries via setCompileLogs. The Compile-All flow prefixes each line with the board label so multi-board builds stay readable. - After the build settles, the existing parseCompileResult call still runs and appends the structured analysis on top of the live stream — that's where FAILED-block detection + the `error`-typed entries that drive the auto-switch-to-errors filter live. Net effect on the user complaint: cold ESP-IDF builds now show the ninja [N/1483] progress lines streaming into the console as they happen, instead of staring at an empty panel for 5-7 minutes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 04:36:58 +07:00
*
* `onProgress` (optional): called every poll with the live cmake + ninja
* output so the editor can stream the compilation console instead of
* waiting for everything at the end.
feat(compile): async compile + status polling — no more 524 timeouts The synchronous /api/compile endpoint forced one long-lived HTTP request to span the entire build. Cloudflare's 100s edge timeout cuts that off mid-flight for any cold ESP-IDF compile (BMP280 takes 5-7 min on first run). The user-visible symptom was HTTP 524 well before the backend even noticed. Backend (compile.py) - New `POST /api/compile/start` returns `{job_id}` immediately and spawns the actual compile as an asyncio.create_task background. - New `GET /api/compile/status/{job_id}` returns the current job state (`pending` | `running` | `done` | `error`). Each poll completes in milliseconds, far under any edge timeout. - Existing `POST /api/compile/` kept verbatim for backward compatibility (AVR/RP2040 builds finish in seconds and don't trip 524). - Build logic extracted into `_run_compile()` so both paths share one implementation; no duplicated ESP-IDF / arduino-cli branching. - Async path opens its own short-lived DB session via AsyncSessionLocal for metric recording — the request-scoped session is dead by the time the background task finishes. - COMPILE_JOBS dict purges entries 30 minutes after completion so a busy server doesn't grow unboundedly. Frontend (compilation.ts) - compileCode() now: POST /compile/start → poll /compile/status every 2s until state ∈ {done, error}, with a 15-minute client-side cap. - 30s axios timeout per individual call (not per build) so transient network blips during a long compile auto-retry instead of failing. - 404 on /status throws (job expired / server restarted); other poll errors warn and retry. Surfaces structured error responses verbatim so the editor's compile-error panel keeps working unchanged. Limitation: COMPILE_JOBS lives in-process; if velxio ever scales to multiple FastAPI workers this needs to move to Redis or sqlite. Single- instance is fine today. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 10:22:09 +07:00
*/
export async function compileCode(
files: SketchFile[],
board: string = 'arduino:avr:uno',
2026-04-26 05:46:52 +07:00
projectId?: string | null,
feat(compile): stream live ESP-IDF cmake + ninja output to the console A user reported on Discord: "the Velxio Console doesn't update anything, it just waits until the very end and displays everything in one go". True for the async compile path — /compile/status only carried `state` and the final `result`, so the editor's CompilationConsole stayed empty during the 5-7 minute cold ESP-IDF builds and dumped 1500 lines at once when the build finished. This wires live build output through the whole stack. Backend (espidf_compiler.py) - New _run_with_streaming() helper. When a progress_callback is provided it spawns the subprocess via Popen + stdout/stderr drain threads and invokes the callback line-by-line. When None it falls back to the existing subprocess.run(capture_output=True) one-shot path so the unit-test code that doesn't care about live output is unaffected. - compile() and _compile_in_dir() take an optional ProgressCallback. - _run_cmake / _run_ninja closures now go through _run_with_streaming with that callback. cmake configure (~2-5 s) + ninja (~5-300+ s) both stream now; the ninja output is the one users actually want to watch. Backend (compile.py) - _compile_job seeds COMPILE_JOBS[id]['stdout_buffer'] = '' and defines on_progress_line(line) which appends to it. Buffer capped at 256 KB (tail kept) so a runaway build can't OOM the FastAPI process. - The buffer is preserved on both the success and the error path so late polls still see the log even after state transitions to done/error. - /compile/status now returns the buffer as a `stdout` field. CompileStatusResponse gains the field with default '' so old clients that don't read it still work. Frontend (compilation.ts) - compileCode() takes a 4th argument: optional CompileProgress callback fired every poll while state ∈ {pending, running}. Carries the cumulative stdout (caller computes deltas) plus elapsed seconds. - Surfaces the new `stdout` field of /compile/status and forwards it to the callback. Errors thrown from the callback are swallowed — a faulty UI hook must never break the polling loop. Frontend (EditorToolbar.tsx) - Both compileCode() call sites (Run and Compile-All) now pass an onProgress callback. It tracks `lastStreamedLen` per-compile, splits each new delta on newlines, and appends them as `info`-typed CompilationLog entries via setCompileLogs. The Compile-All flow prefixes each line with the board label so multi-board builds stay readable. - After the build settles, the existing parseCompileResult call still runs and appends the structured analysis on top of the live stream — that's where FAILED-block detection + the `error`-typed entries that drive the auto-switch-to-errors filter live. Net effect on the user complaint: cold ESP-IDF builds now show the ninja [N/1483] progress lines streaming into the console as they happen, instead of staring at an empty panel for 5-7 minutes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 04:36:58 +07:00
onProgress?: CompileProgress,
extras?: CompileExtras,
): Promise<CompileResult> {
console.log('Sending compilation request to:', `${getApiBase()}/compile/start`);
feat(compile): async compile + status polling — no more 524 timeouts The synchronous /api/compile endpoint forced one long-lived HTTP request to span the entire build. Cloudflare's 100s edge timeout cuts that off mid-flight for any cold ESP-IDF compile (BMP280 takes 5-7 min on first run). The user-visible symptom was HTTP 524 well before the backend even noticed. Backend (compile.py) - New `POST /api/compile/start` returns `{job_id}` immediately and spawns the actual compile as an asyncio.create_task background. - New `GET /api/compile/status/{job_id}` returns the current job state (`pending` | `running` | `done` | `error`). Each poll completes in milliseconds, far under any edge timeout. - Existing `POST /api/compile/` kept verbatim for backward compatibility (AVR/RP2040 builds finish in seconds and don't trip 524). - Build logic extracted into `_run_compile()` so both paths share one implementation; no duplicated ESP-IDF / arduino-cli branching. - Async path opens its own short-lived DB session via AsyncSessionLocal for metric recording — the request-scoped session is dead by the time the background task finishes. - COMPILE_JOBS dict purges entries 30 minutes after completion so a busy server doesn't grow unboundedly. Frontend (compilation.ts) - compileCode() now: POST /compile/start → poll /compile/status every 2s until state ∈ {done, error}, with a 15-minute client-side cap. - 30s axios timeout per individual call (not per build) so transient network blips during a long compile auto-retry instead of failing. - 404 on /status throws (job expired / server restarted); other poll errors warn and retry. Surfaces structured error responses verbatim so the editor's compile-error panel keeps working unchanged. Limitation: COMPILE_JOBS lives in-process; if velxio ever scales to multiple FastAPI workers this needs to move to Redis or sqlite. Single- instance is fine today. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 10:22:09 +07:00
console.log('Board:', board);
console.log(
'Files:',
files.map((f) => f.name),
);
// Translate camelCase frontend keys to snake_case backend keys. Backend
// only inspects these fields for esp32:* FQBNs — other boards pass them
// through unread.
const board_options = extras?.boardOptions ? { ...extras.boardOptions } : null;
const spiffs_files = extras?.spiffsFiles?.length
? extras.spiffsFiles.map((f) => ({ name: f.name, content_b64: f.contentB64 }))
: null;
// P2.3 — library manifest (resolution scope). null = legacy scan-all.
const libraries = extras?.libraries && extras.libraries.length ? extras.libraries : null;
feat(compile): async compile + status polling — no more 524 timeouts The synchronous /api/compile endpoint forced one long-lived HTTP request to span the entire build. Cloudflare's 100s edge timeout cuts that off mid-flight for any cold ESP-IDF compile (BMP280 takes 5-7 min on first run). The user-visible symptom was HTTP 524 well before the backend even noticed. Backend (compile.py) - New `POST /api/compile/start` returns `{job_id}` immediately and spawns the actual compile as an asyncio.create_task background. - New `GET /api/compile/status/{job_id}` returns the current job state (`pending` | `running` | `done` | `error`). Each poll completes in milliseconds, far under any edge timeout. - Existing `POST /api/compile/` kept verbatim for backward compatibility (AVR/RP2040 builds finish in seconds and don't trip 524). - Build logic extracted into `_run_compile()` so both paths share one implementation; no duplicated ESP-IDF / arduino-cli branching. - Async path opens its own short-lived DB session via AsyncSessionLocal for metric recording — the request-scoped session is dead by the time the background task finishes. - COMPILE_JOBS dict purges entries 30 minutes after completion so a busy server doesn't grow unboundedly. Frontend (compilation.ts) - compileCode() now: POST /compile/start → poll /compile/status every 2s until state ∈ {done, error}, with a 15-minute client-side cap. - 30s axios timeout per individual call (not per build) so transient network blips during a long compile auto-retry instead of failing. - 404 on /status throws (job expired / server restarted); other poll errors warn and retry. Surfaces structured error responses verbatim so the editor's compile-error panel keeps working unchanged. Limitation: COMPILE_JOBS lives in-process; if velxio ever scales to multiple FastAPI workers this needs to move to Redis or sqlite. Single- instance is fine today. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 10:22:09 +07:00
let jobId: string;
try {
const startResp = await axios.post<CompileStartResponse>(
`${getApiBase()}/compile/start`,
{
files,
board_fqbn: board,
project_id: projectId ?? null,
board_options,
spiffs_files,
libraries,
feat(esp32): pure ESP-IDF language mode for the ESP32 family (#139) Adds a third entry to the board language selector next to Arduino C++ and MicroPython: ESP-IDF. In this mode the user writes a plain ESP-IDF project — app_main() entry point, FreeRTOS + driver APIs — and the backend compiles it through the same ESP-IDF toolchain it already uses for ESP32 Arduino sketches, just without the arduino-esp32 component. Backend: - CompileRequest.language ('espidf') threaded through the sync + async compile paths and folded into the dedup job key (language='arduino' and omitted hash identically so old clients keep dedupping). - espidf_compiler: pure_idf flag. User files are written into main/ as-is (no Arduino.h wrap, no velxio_compat.h, Arduino library resolution skipped), ARDUINO_ESP32_PATH is dropped from the build env and VELXIO_PURE_SKETCH raised so the template CMake compiles the user's own sources via a glob branch. Pure builds get their own persistent build-dir variant through the eff_hash fold. - QEMU WiFi compat for IDF-style code: esp_wifi.h/esp_wifi_init detection sets has_wifi, and literal #define SSID/PASS plus wifi_config_t designated initializers are normalized to the QEMU AP. - CONFIG_ARDUINO_* lines are stripped from sdkconfig.defaults in pure mode (the symbols don't exist without the arduino component). Frontend: - LanguageMode gains 'espidf'; BOARD_SUPPORTS_ESPIDF covers the ESP32 family (Xtensa, S3, C3). Toolbar shows the option only for those. - Switching modes seeds a main.c blink skeleton (app_main + gpio driver), mirroring the MicroPython main.py flow. - compileCode sends language='espidf'; run/stop paths are unchanged (the QEMU worker consumes the same merged flash image). - New gallery example: esp32-idf-blink (LED + resistor on GPIO 2). Tests: unit coverage for the build-env switch, IDF wifi normalization, job-key variance, file-group seeding and the new example; verified end-to-end in a container from the prod image (pure build produces a bootable flash image; Arduino-mode build unchanged, same variant hash).
2026-07-24 11:37:01 +07:00
language: extras?.language ?? null,
initiated_by: extras?.initiatedBy ?? null,
},
feat(compile): async compile + status polling — no more 524 timeouts The synchronous /api/compile endpoint forced one long-lived HTTP request to span the entire build. Cloudflare's 100s edge timeout cuts that off mid-flight for any cold ESP-IDF compile (BMP280 takes 5-7 min on first run). The user-visible symptom was HTTP 524 well before the backend even noticed. Backend (compile.py) - New `POST /api/compile/start` returns `{job_id}` immediately and spawns the actual compile as an asyncio.create_task background. - New `GET /api/compile/status/{job_id}` returns the current job state (`pending` | `running` | `done` | `error`). Each poll completes in milliseconds, far under any edge timeout. - Existing `POST /api/compile/` kept verbatim for backward compatibility (AVR/RP2040 builds finish in seconds and don't trip 524). - Build logic extracted into `_run_compile()` so both paths share one implementation; no duplicated ESP-IDF / arduino-cli branching. - Async path opens its own short-lived DB session via AsyncSessionLocal for metric recording — the request-scoped session is dead by the time the background task finishes. - COMPILE_JOBS dict purges entries 30 minutes after completion so a busy server doesn't grow unboundedly. Frontend (compilation.ts) - compileCode() now: POST /compile/start → poll /compile/status every 2s until state ∈ {done, error}, with a 15-minute client-side cap. - 30s axios timeout per individual call (not per build) so transient network blips during a long compile auto-retry instead of failing. - 404 on /status throws (job expired / server restarted); other poll errors warn and retry. Surfaces structured error responses verbatim so the editor's compile-error panel keeps working unchanged. Limitation: COMPILE_JOBS lives in-process; if velxio ever scales to multiple FastAPI workers this needs to move to Redis or sqlite. Single- instance is fine today. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 10:22:09 +07:00
{ withCredentials: true, timeout: 30000 },
);
feat(compile): async compile + status polling — no more 524 timeouts The synchronous /api/compile endpoint forced one long-lived HTTP request to span the entire build. Cloudflare's 100s edge timeout cuts that off mid-flight for any cold ESP-IDF compile (BMP280 takes 5-7 min on first run). The user-visible symptom was HTTP 524 well before the backend even noticed. Backend (compile.py) - New `POST /api/compile/start` returns `{job_id}` immediately and spawns the actual compile as an asyncio.create_task background. - New `GET /api/compile/status/{job_id}` returns the current job state (`pending` | `running` | `done` | `error`). Each poll completes in milliseconds, far under any edge timeout. - Existing `POST /api/compile/` kept verbatim for backward compatibility (AVR/RP2040 builds finish in seconds and don't trip 524). - Build logic extracted into `_run_compile()` so both paths share one implementation; no duplicated ESP-IDF / arduino-cli branching. - Async path opens its own short-lived DB session via AsyncSessionLocal for metric recording — the request-scoped session is dead by the time the background task finishes. - COMPILE_JOBS dict purges entries 30 minutes after completion so a busy server doesn't grow unboundedly. Frontend (compilation.ts) - compileCode() now: POST /compile/start → poll /compile/status every 2s until state ∈ {done, error}, with a 15-minute client-side cap. - 30s axios timeout per individual call (not per build) so transient network blips during a long compile auto-retry instead of failing. - 404 on /status throws (job expired / server restarted); other poll errors warn and retry. Surfaces structured error responses verbatim so the editor's compile-error panel keeps working unchanged. Limitation: COMPILE_JOBS lives in-process; if velxio ever scales to multiple FastAPI workers this needs to move to Redis or sqlite. Single- instance is fine today. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 10:22:09 +07:00
jobId = startResp.data.job_id;
console.log('[compile] queued job', jobId);
} catch (error) {
console.error('Compilation request failed:', error);
feat(compile): async compile + status polling — no more 524 timeouts The synchronous /api/compile endpoint forced one long-lived HTTP request to span the entire build. Cloudflare's 100s edge timeout cuts that off mid-flight for any cold ESP-IDF compile (BMP280 takes 5-7 min on first run). The user-visible symptom was HTTP 524 well before the backend even noticed. Backend (compile.py) - New `POST /api/compile/start` returns `{job_id}` immediately and spawns the actual compile as an asyncio.create_task background. - New `GET /api/compile/status/{job_id}` returns the current job state (`pending` | `running` | `done` | `error`). Each poll completes in milliseconds, far under any edge timeout. - Existing `POST /api/compile/` kept verbatim for backward compatibility (AVR/RP2040 builds finish in seconds and don't trip 524). - Build logic extracted into `_run_compile()` so both paths share one implementation; no duplicated ESP-IDF / arduino-cli branching. - Async path opens its own short-lived DB session via AsyncSessionLocal for metric recording — the request-scoped session is dead by the time the background task finishes. - COMPILE_JOBS dict purges entries 30 minutes after completion so a busy server doesn't grow unboundedly. Frontend (compilation.ts) - compileCode() now: POST /compile/start → poll /compile/status every 2s until state ∈ {done, error}, with a 15-minute client-side cap. - 30s axios timeout per individual call (not per build) so transient network blips during a long compile auto-retry instead of failing. - 404 on /status throws (job expired / server restarted); other poll errors warn and retry. Surfaces structured error responses verbatim so the editor's compile-error panel keeps working unchanged. Limitation: COMPILE_JOBS lives in-process; if velxio ever scales to multiple FastAPI workers this needs to move to Redis or sqlite. Single- instance is fine today. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 10:22:09 +07:00
if (axios.isAxiosError(error) && error.response) {
// Server returned a structured error (422, 500, etc.) — surface as a
// failed CompileResult so the editor can show stderr/error.
return error.response.data as CompileResult;
}
throw error instanceof Error
? error
: new Error('No response from server. Is the backend running?');
}
const startedAt = Date.now();
// Initial small delay so we don't hit /status before the background task
// has even moved past 'pending'.
await sleep(500);
while (true) {
if (Date.now() - startedAt > MAX_POLL_DURATION_MS) {
throw new Error(
`Compile timed out client-side after ${Math.round(MAX_POLL_DURATION_MS / 1000)}s`,
);
}
feat(compile): async compile + status polling — no more 524 timeouts The synchronous /api/compile endpoint forced one long-lived HTTP request to span the entire build. Cloudflare's 100s edge timeout cuts that off mid-flight for any cold ESP-IDF compile (BMP280 takes 5-7 min on first run). The user-visible symptom was HTTP 524 well before the backend even noticed. Backend (compile.py) - New `POST /api/compile/start` returns `{job_id}` immediately and spawns the actual compile as an asyncio.create_task background. - New `GET /api/compile/status/{job_id}` returns the current job state (`pending` | `running` | `done` | `error`). Each poll completes in milliseconds, far under any edge timeout. - Existing `POST /api/compile/` kept verbatim for backward compatibility (AVR/RP2040 builds finish in seconds and don't trip 524). - Build logic extracted into `_run_compile()` so both paths share one implementation; no duplicated ESP-IDF / arduino-cli branching. - Async path opens its own short-lived DB session via AsyncSessionLocal for metric recording — the request-scoped session is dead by the time the background task finishes. - COMPILE_JOBS dict purges entries 30 minutes after completion so a busy server doesn't grow unboundedly. Frontend (compilation.ts) - compileCode() now: POST /compile/start → poll /compile/status every 2s until state ∈ {done, error}, with a 15-minute client-side cap. - 30s axios timeout per individual call (not per build) so transient network blips during a long compile auto-retry instead of failing. - 404 on /status throws (job expired / server restarted); other poll errors warn and retry. Surfaces structured error responses verbatim so the editor's compile-error panel keeps working unchanged. Limitation: COMPILE_JOBS lives in-process; if velxio ever scales to multiple FastAPI workers this needs to move to Redis or sqlite. Single- instance is fine today. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 10:22:09 +07:00
let status: CompileStatusResponse;
try {
const resp = await axios.get<CompileStatusResponse>(
`${getApiBase()}/compile/status/${jobId}`,
feat(compile): async compile + status polling — no more 524 timeouts The synchronous /api/compile endpoint forced one long-lived HTTP request to span the entire build. Cloudflare's 100s edge timeout cuts that off mid-flight for any cold ESP-IDF compile (BMP280 takes 5-7 min on first run). The user-visible symptom was HTTP 524 well before the backend even noticed. Backend (compile.py) - New `POST /api/compile/start` returns `{job_id}` immediately and spawns the actual compile as an asyncio.create_task background. - New `GET /api/compile/status/{job_id}` returns the current job state (`pending` | `running` | `done` | `error`). Each poll completes in milliseconds, far under any edge timeout. - Existing `POST /api/compile/` kept verbatim for backward compatibility (AVR/RP2040 builds finish in seconds and don't trip 524). - Build logic extracted into `_run_compile()` so both paths share one implementation; no duplicated ESP-IDF / arduino-cli branching. - Async path opens its own short-lived DB session via AsyncSessionLocal for metric recording — the request-scoped session is dead by the time the background task finishes. - COMPILE_JOBS dict purges entries 30 minutes after completion so a busy server doesn't grow unboundedly. Frontend (compilation.ts) - compileCode() now: POST /compile/start → poll /compile/status every 2s until state ∈ {done, error}, with a 15-minute client-side cap. - 30s axios timeout per individual call (not per build) so transient network blips during a long compile auto-retry instead of failing. - 404 on /status throws (job expired / server restarted); other poll errors warn and retry. Surfaces structured error responses verbatim so the editor's compile-error panel keeps working unchanged. Limitation: COMPILE_JOBS lives in-process; if velxio ever scales to multiple FastAPI workers this needs to move to Redis or sqlite. Single- instance is fine today. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 10:22:09 +07:00
{ withCredentials: true, timeout: 30000 },
);
status = resp.data;
} catch (error) {
// Transient poll error — log, wait, retry. Only abort on 404 (job
// expired or never existed).
if (axios.isAxiosError(error) && error.response?.status === 404) {
throw new Error(`Compile job ${jobId} not found (server may have restarted)`);
}
feat(compile): async compile + status polling — no more 524 timeouts The synchronous /api/compile endpoint forced one long-lived HTTP request to span the entire build. Cloudflare's 100s edge timeout cuts that off mid-flight for any cold ESP-IDF compile (BMP280 takes 5-7 min on first run). The user-visible symptom was HTTP 524 well before the backend even noticed. Backend (compile.py) - New `POST /api/compile/start` returns `{job_id}` immediately and spawns the actual compile as an asyncio.create_task background. - New `GET /api/compile/status/{job_id}` returns the current job state (`pending` | `running` | `done` | `error`). Each poll completes in milliseconds, far under any edge timeout. - Existing `POST /api/compile/` kept verbatim for backward compatibility (AVR/RP2040 builds finish in seconds and don't trip 524). - Build logic extracted into `_run_compile()` so both paths share one implementation; no duplicated ESP-IDF / arduino-cli branching. - Async path opens its own short-lived DB session via AsyncSessionLocal for metric recording — the request-scoped session is dead by the time the background task finishes. - COMPILE_JOBS dict purges entries 30 minutes after completion so a busy server doesn't grow unboundedly. Frontend (compilation.ts) - compileCode() now: POST /compile/start → poll /compile/status every 2s until state ∈ {done, error}, with a 15-minute client-side cap. - 30s axios timeout per individual call (not per build) so transient network blips during a long compile auto-retry instead of failing. - 404 on /status throws (job expired / server restarted); other poll errors warn and retry. Surfaces structured error responses verbatim so the editor's compile-error panel keeps working unchanged. Limitation: COMPILE_JOBS lives in-process; if velxio ever scales to multiple FastAPI workers this needs to move to Redis or sqlite. Single- instance is fine today. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 10:22:09 +07:00
console.warn('[compile] status poll error, retrying:', error);
await sleep(POLL_INTERVAL_MS);
continue;
}
if (status.state === 'done' && status.result) {
const elapsed = Math.round((Date.now() - startedAt) / 1000);
console.log(`[compile] job ${jobId} done in ${elapsed}s`);
return status.result;
}
if (status.state === 'error') {
console.error(`[compile] job ${jobId} errored:`, status.error);
return {
success: false,
feat(compile): stream live ESP-IDF cmake + ninja output to the console A user reported on Discord: "the Velxio Console doesn't update anything, it just waits until the very end and displays everything in one go". True for the async compile path — /compile/status only carried `state` and the final `result`, so the editor's CompilationConsole stayed empty during the 5-7 minute cold ESP-IDF builds and dumped 1500 lines at once when the build finished. This wires live build output through the whole stack. Backend (espidf_compiler.py) - New _run_with_streaming() helper. When a progress_callback is provided it spawns the subprocess via Popen + stdout/stderr drain threads and invokes the callback line-by-line. When None it falls back to the existing subprocess.run(capture_output=True) one-shot path so the unit-test code that doesn't care about live output is unaffected. - compile() and _compile_in_dir() take an optional ProgressCallback. - _run_cmake / _run_ninja closures now go through _run_with_streaming with that callback. cmake configure (~2-5 s) + ninja (~5-300+ s) both stream now; the ninja output is the one users actually want to watch. Backend (compile.py) - _compile_job seeds COMPILE_JOBS[id]['stdout_buffer'] = '' and defines on_progress_line(line) which appends to it. Buffer capped at 256 KB (tail kept) so a runaway build can't OOM the FastAPI process. - The buffer is preserved on both the success and the error path so late polls still see the log even after state transitions to done/error. - /compile/status now returns the buffer as a `stdout` field. CompileStatusResponse gains the field with default '' so old clients that don't read it still work. Frontend (compilation.ts) - compileCode() takes a 4th argument: optional CompileProgress callback fired every poll while state ∈ {pending, running}. Carries the cumulative stdout (caller computes deltas) plus elapsed seconds. - Surfaces the new `stdout` field of /compile/status and forwards it to the callback. Errors thrown from the callback are swallowed — a faulty UI hook must never break the polling loop. Frontend (EditorToolbar.tsx) - Both compileCode() call sites (Run and Compile-All) now pass an onProgress callback. It tracks `lastStreamedLen` per-compile, splits each new delta on newlines, and appends them as `info`-typed CompilationLog entries via setCompileLogs. The Compile-All flow prefixes each line with the board label so multi-board builds stay readable. - After the build settles, the existing parseCompileResult call still runs and appends the structured analysis on top of the live stream — that's where FAILED-block detection + the `error`-typed entries that drive the auto-switch-to-errors filter live. Net effect on the user complaint: cold ESP-IDF builds now show the ninja [N/1483] progress lines streaming into the console as they happen, instead of staring at an empty panel for 5-7 minutes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 04:36:58 +07:00
stdout: status.stdout || '',
feat(compile): async compile + status polling — no more 524 timeouts The synchronous /api/compile endpoint forced one long-lived HTTP request to span the entire build. Cloudflare's 100s edge timeout cuts that off mid-flight for any cold ESP-IDF compile (BMP280 takes 5-7 min on first run). The user-visible symptom was HTTP 524 well before the backend even noticed. Backend (compile.py) - New `POST /api/compile/start` returns `{job_id}` immediately and spawns the actual compile as an asyncio.create_task background. - New `GET /api/compile/status/{job_id}` returns the current job state (`pending` | `running` | `done` | `error`). Each poll completes in milliseconds, far under any edge timeout. - Existing `POST /api/compile/` kept verbatim for backward compatibility (AVR/RP2040 builds finish in seconds and don't trip 524). - Build logic extracted into `_run_compile()` so both paths share one implementation; no duplicated ESP-IDF / arduino-cli branching. - Async path opens its own short-lived DB session via AsyncSessionLocal for metric recording — the request-scoped session is dead by the time the background task finishes. - COMPILE_JOBS dict purges entries 30 minutes after completion so a busy server doesn't grow unboundedly. Frontend (compilation.ts) - compileCode() now: POST /compile/start → poll /compile/status every 2s until state ∈ {done, error}, with a 15-minute client-side cap. - 30s axios timeout per individual call (not per build) so transient network blips during a long compile auto-retry instead of failing. - 404 on /status throws (job expired / server restarted); other poll errors warn and retry. Surfaces structured error responses verbatim so the editor's compile-error panel keeps working unchanged. Limitation: COMPILE_JOBS lives in-process; if velxio ever scales to multiple FastAPI workers this needs to move to Redis or sqlite. Single- instance is fine today. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 10:22:09 +07:00
stderr: '',
error: status.error || 'Compile failed',
};
}
feat(compile): stream live ESP-IDF cmake + ninja output to the console A user reported on Discord: "the Velxio Console doesn't update anything, it just waits until the very end and displays everything in one go". True for the async compile path — /compile/status only carried `state` and the final `result`, so the editor's CompilationConsole stayed empty during the 5-7 minute cold ESP-IDF builds and dumped 1500 lines at once when the build finished. This wires live build output through the whole stack. Backend (espidf_compiler.py) - New _run_with_streaming() helper. When a progress_callback is provided it spawns the subprocess via Popen + stdout/stderr drain threads and invokes the callback line-by-line. When None it falls back to the existing subprocess.run(capture_output=True) one-shot path so the unit-test code that doesn't care about live output is unaffected. - compile() and _compile_in_dir() take an optional ProgressCallback. - _run_cmake / _run_ninja closures now go through _run_with_streaming with that callback. cmake configure (~2-5 s) + ninja (~5-300+ s) both stream now; the ninja output is the one users actually want to watch. Backend (compile.py) - _compile_job seeds COMPILE_JOBS[id]['stdout_buffer'] = '' and defines on_progress_line(line) which appends to it. Buffer capped at 256 KB (tail kept) so a runaway build can't OOM the FastAPI process. - The buffer is preserved on both the success and the error path so late polls still see the log even after state transitions to done/error. - /compile/status now returns the buffer as a `stdout` field. CompileStatusResponse gains the field with default '' so old clients that don't read it still work. Frontend (compilation.ts) - compileCode() takes a 4th argument: optional CompileProgress callback fired every poll while state ∈ {pending, running}. Carries the cumulative stdout (caller computes deltas) plus elapsed seconds. - Surfaces the new `stdout` field of /compile/status and forwards it to the callback. Errors thrown from the callback are swallowed — a faulty UI hook must never break the polling loop. Frontend (EditorToolbar.tsx) - Both compileCode() call sites (Run and Compile-All) now pass an onProgress callback. It tracks `lastStreamedLen` per-compile, splits each new delta on newlines, and appends them as `info`-typed CompilationLog entries via setCompileLogs. The Compile-All flow prefixes each line with the board label so multi-board builds stay readable. - After the build settles, the existing parseCompileResult call still runs and appends the structured analysis on top of the live stream — that's where FAILED-block detection + the `error`-typed entries that drive the auto-switch-to-errors filter live. Net effect on the user complaint: cold ESP-IDF builds now show the ninja [N/1483] progress lines streaming into the console as they happen, instead of staring at an empty panel for 5-7 minutes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 04:36:58 +07:00
// state ∈ {pending, running} — surface live build output if requested
if (onProgress) {
try {
onProgress({
state: status.state,
stdout: status.stdout || '',
elapsedSeconds: Math.round((Date.now() - startedAt) / 1000),
});
} catch (err) {
// A faulty UI hook must never break the polling loop.
console.warn('[compile] onProgress threw:', err);
}
}
feat(compile): async compile + status polling — no more 524 timeouts The synchronous /api/compile endpoint forced one long-lived HTTP request to span the entire build. Cloudflare's 100s edge timeout cuts that off mid-flight for any cold ESP-IDF compile (BMP280 takes 5-7 min on first run). The user-visible symptom was HTTP 524 well before the backend even noticed. Backend (compile.py) - New `POST /api/compile/start` returns `{job_id}` immediately and spawns the actual compile as an asyncio.create_task background. - New `GET /api/compile/status/{job_id}` returns the current job state (`pending` | `running` | `done` | `error`). Each poll completes in milliseconds, far under any edge timeout. - Existing `POST /api/compile/` kept verbatim for backward compatibility (AVR/RP2040 builds finish in seconds and don't trip 524). - Build logic extracted into `_run_compile()` so both paths share one implementation; no duplicated ESP-IDF / arduino-cli branching. - Async path opens its own short-lived DB session via AsyncSessionLocal for metric recording — the request-scoped session is dead by the time the background task finishes. - COMPILE_JOBS dict purges entries 30 minutes after completion so a busy server doesn't grow unboundedly. Frontend (compilation.ts) - compileCode() now: POST /compile/start → poll /compile/status every 2s until state ∈ {done, error}, with a 15-minute client-side cap. - 30s axios timeout per individual call (not per build) so transient network blips during a long compile auto-retry instead of failing. - 404 on /status throws (job expired / server restarted); other poll errors warn and retry. Surfaces structured error responses verbatim so the editor's compile-error panel keeps working unchanged. Limitation: COMPILE_JOBS lives in-process; if velxio ever scales to multiple FastAPI workers this needs to move to Redis or sqlite. Single- instance is fine today. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 10:22:09 +07:00
await sleep(POLL_INTERVAL_MS);
}
}