velxio/backend/app/main.py

142 lines
5.4 KiB
Python
Raw Normal View History

import logging
import sys
import asyncio
from contextlib import asynccontextmanager
logging.basicConfig(level=logging.INFO, format='%(levelname)s %(name)s: %(message)s')
# On Windows, asyncio defaults to SelectorEventLoop which does NOT support
# create_subprocess_exec (raises NotImplementedError). Force ProactorEventLoop.
if sys.platform == 'win32':
asyncio.set_event_loop_policy(asyncio.WindowsProactorEventLoopPolicy())
refactor(oss-split): remove auth/DB/admin stack from OSS Phase 2 of the OSS / pro split. The hook seams introduced in Phase 1 let stateless routes (compile, libraries, simulation, iot_gateway) run without the auth/DB stack importable. Now we actually delete the stack: app/api/routes/auth.py app/api/routes/projects.py app/api/routes/admin.py app/api/routes/metrics.py app/models/{user,project,usage_event,password_reset_token}.py app/schemas/{auth,admin,project}.py app/core/{dependencies,security}.py app/database/session.py app/services/{metrics,odoo_mail,project_files}.py app/utils/{geo,slug,boards}.py Private deployments (velxio.dev) get the same modules back via the velxio-prod overlay: pro/backend/app/api/routes/auth.py etc. are COPYed onto /app/... at container build time, and register_pro() includes their routers + registers the lifespan/metrics/auth hooks. main.py shrank back to the stateless router includes + a single `run_lifespan_startup()` call. The Phase-1 try-import block that wired record_compile / get_current_user_id from upstream is gone — those adapters live in pro now. Verification: OSS only: 20 routes (compile, libraries, simulation, gateway). OSS + pro: 94 routes — identical to pre-refactor velxio.dev. Net change: -2400 lines from OSS, all of which moved to velxio-prod's overlay. Self-hosted OSS users lose accounts + project persistence; the Phase 4 .vlx export/import gives them a portable replacement. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 23:36:31 +07:00
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
feat(flash): write compiled sketches to real USB boards (phases D1+D3) Brings hardware flashing into Velxio Desktop. Per-board "Flash to real board" entry in the canvas context menu opens a modal that enumerates USB serial ports, lets the user pick one, then streams arduino-cli upload output live until the board is flashed. Backend (Phase D1) — backend/app/api/routes/flash.py (new): POST /api/flash/upload (multipart: board_id, port, fqbn, program_format, program) → SSE stream of {phase, line?, progress?} events → final {phase:'done', success, elapsed_ms, error?} - Wraps `arduino-cli upload -p <port> -i <file> --fqbn <fqbn> -v` so AVR (avrdude), ESP32 (esptool), RP2040 (picotool), SAMD (bossac) all share one code path — arduino-cli internally dispatches by FQBN. - Per-port asyncio.Lock prevents two simultaneous flashes from fighting over the same /dev/ttyACM0. - Allow-list of FQBN prefixes (arduino:avr, ATTinyCore:avr, rp2040:rp2040, esp32:esp32, arduino:samd) so a typo can't cause a confusing arduino-cli error. - Format allow-list (hex / bin / uf2 / elf) drives the temp file extension - arduino-cli uses the extension to route to the right uploader. - 8MB hard cap on the uploaded program (real sketches are well under that; protects against a runaway frontend). - X-Accel-Buffering: no header so nginx doesn't hold the SSE chunks until the flash completes. Frontend (Phase D3): - frontend/src/services/flashService.ts (new): async generator streamFlash() yields parsed SSE events. Handles the base64-vs-text gotcha (compile returns hex_content as text but binary_content as base64; for binary formats we atob() into a Uint8Array before posting so the form upload sends actual bytes, not the base64 ASCII). - frontend/src/components/simulator/FlashModal.tsx (new): Three-state UI: picking (port dropdown), flashing (progress bar + live log), success/error (verdict + retry). Empty-ports state shows a Linux dialout-group hint. - SimulatorCanvas.tsx: board context menu gains "Flash to real board" entry, gated on isTauri() + presence of compiledProgram. Hidden in web (WebSerial is a separate sprint). - tauriBridge.ts: SerialPortInfo type + listSerialPorts() helper that invokes the Rust shell command added in Phase D2. The sidecar already has arduino-cli on PATH (per `pro/desktop/sidecar/main.py::_expose_bundled_arduino_cli`), so no installer changes are needed — flash works the moment the 0.4.x desktop bundle ships with these commits. Plan + remaining phase tracked in project/hardware-flashing/. D2 (Rust serial enum) committed separately as a Tauri-shell-only concern; D4 (manual smoke matrix with real boards) requires physical hardware so it stays a follow-up. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-27 10:20:20 +07:00
from app.api.routes import compile, compile_chip, compile_rom, flash, libraries
from app.core.config import settings
refactor(oss-split): remove auth/DB/admin stack from OSS Phase 2 of the OSS / pro split. The hook seams introduced in Phase 1 let stateless routes (compile, libraries, simulation, iot_gateway) run without the auth/DB stack importable. Now we actually delete the stack: app/api/routes/auth.py app/api/routes/projects.py app/api/routes/admin.py app/api/routes/metrics.py app/models/{user,project,usage_event,password_reset_token}.py app/schemas/{auth,admin,project}.py app/core/{dependencies,security}.py app/database/session.py app/services/{metrics,odoo_mail,project_files}.py app/utils/{geo,slug,boards}.py Private deployments (velxio.dev) get the same modules back via the velxio-prod overlay: pro/backend/app/api/routes/auth.py etc. are COPYed onto /app/... at container build time, and register_pro() includes their routers + registers the lifespan/metrics/auth hooks. main.py shrank back to the stateless router includes + a single `run_lifespan_startup()` call. The Phase-1 try-import block that wired record_compile / get_current_user_id from upstream is gone — those adapters live in pro now. Verification: OSS only: 20 routes (compile, libraries, simulation, gateway). OSS + pro: 94 routes — identical to pre-refactor velxio.dev. Net change: -2400 lines from OSS, all of which moved to velxio-prod's overlay. Self-hosted OSS users lose accounts + project persistence; the Phase 4 .vlx export/import gives them a portable replacement. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 23:36:31 +07:00
from app.core.hooks import run_lifespan_startup
refactor(oss-split): introduce extension hooks for auth, DB, metrics, auto-save First phase of the OSS / pro split. Goal: open the seams so the auth/DB/admin stack can move into the private overlay (Phase 2-3) without the routes that stay in OSS (compile, libraries, simulation, iot_gateway) having to know. Backend ------- * New app/core/hooks.py — registry for record_compile, get_current_user_id, and lifespan startup tasks. Each hook is a no-op by default; overlays call register_* in register_pro(app) to plug in a real implementation. * compile.py now imports only from app.core.hooks. Drops the direct deps on app.core.dependencies, app.database.session, app.models.user, and app.services.metrics. Route signatures use `Depends(get_current_user_id)` instead of `Depends(get_current_user)`; the metric helper passes user_id through rather than a User instance. * compile_chip.py drops the unused _current_user Depends entirely. * main.py wraps the auth/DB stack import in try/except. When it succeeds (today's behavior on velxio.dev), an adapter bridges record_compile and get_current_user_id to the existing app.services.metrics + dependencies, and the create_all + ALTER TABLE migration block runs via a registered lifespan_startup hook. When it fails (the post-Phase-2 OSS image), main logs "running stateless" and skips registering anything — the routes still load and behave as no-ops for metrics + always-anonymous for auth. Frontend -------- * useAutoSaveProject becomes a skeleton: one useState + one useEffect that delegates to an installed AutoSaveImpl. installAutoSaveImpl() replaces the impl without changing hook count, so React's rules-of-hooks stay satisfied even after the impl moves out of OSS. * New hooks/autoSaveImpl.ts holds the original logic (debouncing, dirty detection, owner eligibility, fetch keepalive on unload), refactored to emit() instead of useState. It self-registers at module load; main.tsx imports it for the side effect. * AppHeader wraps the entire user-vs-login UI in a data-velxio-slot ="header-auth" boundary. Today the OSS UI still renders inside the slot — the overlay can portal-inject additional items now, and in Phase 3 the slot becomes the sole owner of header auth UX. Behavior is identical on velxio.dev (pro overlay imports everything successfully, every adapter wires up). The change is purely structural: deleting the auth/DB modules tomorrow no longer crashes OSS at import. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 23:24:51 +07:00
logger = logging.getLogger(__name__)
def _asyncio_exception_handler(loop: asyncio.AbstractEventLoop, context: dict) -> None:
"""Prevent unhandled asyncio task exceptions from killing the uvicorn process.
Normally uvicorn re-raises unhandled task exceptions at the event-loop level,
which can crash the whole process. The main culprit is a race condition in
websockets <12.0 (legacy/protocol.py AssertionError during keepalive ping).
Upgrading websockets>=12.0 is the primary fix; this handler is a safety net.
"""
exc = context.get("exception")
msg = context.get("message", "")
if exc is not None:
logger.error("Unhandled asyncio task exception (swallowed): %s%r", msg, exc)
else:
# No exception object — let default handler deal with it
loop.default_exception_handler(context)
@asynccontextmanager
async def lifespan(_app: FastAPI):
asyncio.get_event_loop().set_exception_handler(_asyncio_exception_handler)
refactor(oss-split): introduce extension hooks for auth, DB, metrics, auto-save First phase of the OSS / pro split. Goal: open the seams so the auth/DB/admin stack can move into the private overlay (Phase 2-3) without the routes that stay in OSS (compile, libraries, simulation, iot_gateway) having to know. Backend ------- * New app/core/hooks.py — registry for record_compile, get_current_user_id, and lifespan startup tasks. Each hook is a no-op by default; overlays call register_* in register_pro(app) to plug in a real implementation. * compile.py now imports only from app.core.hooks. Drops the direct deps on app.core.dependencies, app.database.session, app.models.user, and app.services.metrics. Route signatures use `Depends(get_current_user_id)` instead of `Depends(get_current_user)`; the metric helper passes user_id through rather than a User instance. * compile_chip.py drops the unused _current_user Depends entirely. * main.py wraps the auth/DB stack import in try/except. When it succeeds (today's behavior on velxio.dev), an adapter bridges record_compile and get_current_user_id to the existing app.services.metrics + dependencies, and the create_all + ALTER TABLE migration block runs via a registered lifespan_startup hook. When it fails (the post-Phase-2 OSS image), main logs "running stateless" and skips registering anything — the routes still load and behave as no-ops for metrics + always-anonymous for auth. Frontend -------- * useAutoSaveProject becomes a skeleton: one useState + one useEffect that delegates to an installed AutoSaveImpl. installAutoSaveImpl() replaces the impl without changing hook count, so React's rules-of-hooks stay satisfied even after the impl moves out of OSS. * New hooks/autoSaveImpl.ts holds the original logic (debouncing, dirty detection, owner eligibility, fetch keepalive on unload), refactored to emit() instead of useState. It self-registers at module load; main.tsx imports it for the side effect. * AppHeader wraps the entire user-vs-login UI in a data-velxio-slot ="header-auth" boundary. Today the OSS UI still renders inside the slot — the overlay can portal-inject additional items now, and in Phase 3 the slot becomes the sole owner of header auth UX. Behavior is identical on velxio.dev (pro overlay imports everything successfully, every adapter wires up). The change is purely structural: deleting the auth/DB modules tomorrow no longer crashes OSS at import. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 23:24:51 +07:00
# Each module that needs async startup (DB schema creation, legacy column
# migrations, cache warmers, …) registers a hook with
# register_lifespan_startup() at import time. The OSS auth/DB stack
# registers the create_all + ALTER TABLE migration block above; the
# private overlay's register_pro() can add more. Running zero hooks is
# the expected behavior of a stateless OSS image.
await run_lifespan_startup()
yield
app = FastAPI(
title="Arduino Emulator API",
description="Compilation and project management API",
version="1.0.0",
lifespan=lifespan,
# Moved from /docs to /api/docs so the frontend /docs/* documentation
# routes are served by the React SPA without any nginx conflict.
docs_url="/api/docs",
redoc_url="/api/redoc",
openapi_url="/api/openapi.json",
)
# CORS — local Vite dev, the prod web origin, AND the Velxio Desktop
# Tauri origins. The desktop bundle runs from a non-http scheme so
# every fetch to velxio.dev is cross-origin and the browser blocks
# preflight unless we explicitly allow the Tauri scheme(s).
#
# Tauri origin per OS:
# - macOS / Linux: `tauri://localhost`
# - Windows: `http://tauri.localhost`
# - older Tauri: `https://tauri.localhost`
# All three are listed so the desktop bundle works regardless of
# host OS or Tauri version.
app.add_middleware(
CORSMiddleware,
allow_origins=[
"http://localhost:5173",
"http://localhost:5174",
"http://localhost:5175",
"tauri://localhost",
"http://tauri.localhost",
"https://tauri.localhost",
settings.FRONTEND_URL,
],
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
# Include routers
app.include_router(compile.router, prefix="/api/compile", tags=["compilation"])
app.include_router(compile_chip.router, prefix="/api/compile-chip", tags=["custom-chips"])
feat(chips): programmable retro CPU chips with external ROM Adds a new way to use the retro CPU chips: write your program in a project file (.s / .asm / .hex / .bin), click Compile, click Run, and the same chip emulates whatever you wrote. Same chip + different ROMs = mini PC, calculator, LED demo, Kill-the-Bit game, etc. SDK: - velxio-chip.h gets two new host imports: uint32_t vx_rom_size(void); void vx_rom_read(uint32_t off, uint8_t* dst, uint32_t len); CPU-emulator chips call these in chip_setup to pull their program out of the host's romBytes property. Frontend runtime: - ChipRuntime accepts opts.romBytes (Uint8Array) and exposes the new imports, copying bytes into chip memory on vx_rom_read. - CustomChipPart pulls component.properties.romBytes (base64) and passes it through. - Component registry declares three new custom-chip properties: romBytes (base64), programFile (matching project filename), and programTarget (cpu name). New programmable bundled chip: - frontend/src/components/customChips/examples/intel/i8080-cpu.{c,chip.json} Same clean-room 8080 emulator as i8080-repl/i8080-counter, but ROM is loaded externally via vx_rom_*. Has 8 LEDs, 8 buttons, UART, 16 KB RAM, 32 KB of external ROM. Backend: - New /api/compile-rom endpoint and rom_compile service that turns chip-program source into ROM bytes. 8080 ASM is assembled by the in-tree two-pass assembler (moved to backend/app/services/asm8080.py). Intel HEX records are parsed; raw .bin is passed through. Future targets (z80, 8086, 4004) are scaffolded but not wired yet. EditorToolbar: - Compile button detects when the active file is .s/.asm/.hex/.bin and routes to compile-rom instead of arduino-cli. The compiled bytes are injected into every custom-chip on the canvas whose programFile property matches the active filename (or is empty). Example: - /examples/i8080-killbits loads Dean McDaniel's 1975 Kill-the-Bit on the programmable i8080-cpu chip. killbits.s is shipped as a project file alongside sketch.ino; the user clicks Compile then Run and the LED walks across 8 outputs, buttons kill it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 09:38:18 +07:00
app.include_router(compile_rom.router, prefix="/api/compile-rom", tags=["custom-chips"])
app.include_router(libraries.router, prefix="/api/libraries", tags=["libraries"])
feat(flash): write compiled sketches to real USB boards (phases D1+D3) Brings hardware flashing into Velxio Desktop. Per-board "Flash to real board" entry in the canvas context menu opens a modal that enumerates USB serial ports, lets the user pick one, then streams arduino-cli upload output live until the board is flashed. Backend (Phase D1) — backend/app/api/routes/flash.py (new): POST /api/flash/upload (multipart: board_id, port, fqbn, program_format, program) → SSE stream of {phase, line?, progress?} events → final {phase:'done', success, elapsed_ms, error?} - Wraps `arduino-cli upload -p <port> -i <file> --fqbn <fqbn> -v` so AVR (avrdude), ESP32 (esptool), RP2040 (picotool), SAMD (bossac) all share one code path — arduino-cli internally dispatches by FQBN. - Per-port asyncio.Lock prevents two simultaneous flashes from fighting over the same /dev/ttyACM0. - Allow-list of FQBN prefixes (arduino:avr, ATTinyCore:avr, rp2040:rp2040, esp32:esp32, arduino:samd) so a typo can't cause a confusing arduino-cli error. - Format allow-list (hex / bin / uf2 / elf) drives the temp file extension - arduino-cli uses the extension to route to the right uploader. - 8MB hard cap on the uploaded program (real sketches are well under that; protects against a runaway frontend). - X-Accel-Buffering: no header so nginx doesn't hold the SSE chunks until the flash completes. Frontend (Phase D3): - frontend/src/services/flashService.ts (new): async generator streamFlash() yields parsed SSE events. Handles the base64-vs-text gotcha (compile returns hex_content as text but binary_content as base64; for binary formats we atob() into a Uint8Array before posting so the form upload sends actual bytes, not the base64 ASCII). - frontend/src/components/simulator/FlashModal.tsx (new): Three-state UI: picking (port dropdown), flashing (progress bar + live log), success/error (verdict + retry). Empty-ports state shows a Linux dialout-group hint. - SimulatorCanvas.tsx: board context menu gains "Flash to real board" entry, gated on isTauri() + presence of compiledProgram. Hidden in web (WebSerial is a separate sprint). - tauriBridge.ts: SerialPortInfo type + listSerialPorts() helper that invokes the Rust shell command added in Phase D2. The sidecar already has arduino-cli on PATH (per `pro/desktop/sidecar/main.py::_expose_bundled_arduino_cli`), so no installer changes are needed — flash works the moment the 0.4.x desktop bundle ships with these commits. Plan + remaining phase tracked in project/hardware-flashing/. D2 (Rust serial enum) committed separately as a Tauri-shell-only concern; D4 (manual smoke matrix with real boards) requires physical hardware so it stays a follow-up. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-27 10:20:20 +07:00
# Hardware flash: subprocesses arduino-cli upload to write a compiled
# sketch to a real USB-attached board. Desktop-only in practice (the
# web build has no access to local serial ports without WebSerial),
# but the route lives in OSS so self-hosters with a sidecar reach
# get it too.
app.include_router(flash.router, prefix="/api/flash", tags=["flash"])
refactor(oss-split): introduce extension hooks for auth, DB, metrics, auto-save First phase of the OSS / pro split. Goal: open the seams so the auth/DB/admin stack can move into the private overlay (Phase 2-3) without the routes that stay in OSS (compile, libraries, simulation, iot_gateway) having to know. Backend ------- * New app/core/hooks.py — registry for record_compile, get_current_user_id, and lifespan startup tasks. Each hook is a no-op by default; overlays call register_* in register_pro(app) to plug in a real implementation. * compile.py now imports only from app.core.hooks. Drops the direct deps on app.core.dependencies, app.database.session, app.models.user, and app.services.metrics. Route signatures use `Depends(get_current_user_id)` instead of `Depends(get_current_user)`; the metric helper passes user_id through rather than a User instance. * compile_chip.py drops the unused _current_user Depends entirely. * main.py wraps the auth/DB stack import in try/except. When it succeeds (today's behavior on velxio.dev), an adapter bridges record_compile and get_current_user_id to the existing app.services.metrics + dependencies, and the create_all + ALTER TABLE migration block runs via a registered lifespan_startup hook. When it fails (the post-Phase-2 OSS image), main logs "running stateless" and skips registering anything — the routes still load and behave as no-ops for metrics + always-anonymous for auth. Frontend -------- * useAutoSaveProject becomes a skeleton: one useState + one useEffect that delegates to an installed AutoSaveImpl. installAutoSaveImpl() replaces the impl without changing hook count, so React's rules-of-hooks stay satisfied even after the impl moves out of OSS. * New hooks/autoSaveImpl.ts holds the original logic (debouncing, dirty detection, owner eligibility, fetch keepalive on unload), refactored to emit() instead of useState. It self-registers at module load; main.tsx imports it for the side effect. * AppHeader wraps the entire user-vs-login UI in a data-velxio-slot ="header-auth" boundary. Today the OSS UI still renders inside the slot — the overlay can portal-inject additional items now, and in Phase 3 the slot becomes the sole owner of header auth UX. Behavior is identical on velxio.dev (pro overlay imports everything successfully, every adapter wires up). The change is purely structural: deleting the auth/DB modules tomorrow no longer crashes OSS at import. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 23:24:51 +07:00
refactor(oss-split): remove auth/DB/admin stack from OSS Phase 2 of the OSS / pro split. The hook seams introduced in Phase 1 let stateless routes (compile, libraries, simulation, iot_gateway) run without the auth/DB stack importable. Now we actually delete the stack: app/api/routes/auth.py app/api/routes/projects.py app/api/routes/admin.py app/api/routes/metrics.py app/models/{user,project,usage_event,password_reset_token}.py app/schemas/{auth,admin,project}.py app/core/{dependencies,security}.py app/database/session.py app/services/{metrics,odoo_mail,project_files}.py app/utils/{geo,slug,boards}.py Private deployments (velxio.dev) get the same modules back via the velxio-prod overlay: pro/backend/app/api/routes/auth.py etc. are COPYed onto /app/... at container build time, and register_pro() includes their routers + registers the lifespan/metrics/auth hooks. main.py shrank back to the stateless router includes + a single `run_lifespan_startup()` call. The Phase-1 try-import block that wired record_compile / get_current_user_id from upstream is gone — those adapters live in pro now. Verification: OSS only: 20 routes (compile, libraries, simulation, gateway). OSS + pro: 94 routes — identical to pre-refactor velxio.dev. Net change: -2400 lines from OSS, all of which moved to velxio-prod's overlay. Self-hosted OSS users lose accounts + project persistence; the Phase 4 .vlx export/import gives them a portable replacement. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 23:36:31 +07:00
# Auth / projects / admin / metrics routers used to be wired up here, gated
# on the auth/DB stack being importable. Phase 2 of the OSS split moved
# them out of upstream entirely — they now live under the private overlay's
# pro/backend/app/api/routes/ and are registered by register_pro(app)
# below. The OSS image carries none of them: anonymous, stateless.
# WebSockets
from app.api.routes import simulation
app.include_router(simulation.router, prefix="/api/simulation", tags=["simulation"])
# IoT Gateway — HTTP proxy for ESP32 web servers
from app.api.routes import iot_gateway
app.include_router(iot_gateway.router, prefix="/api/gateway", tags=["iot-gateway"])
# Optional pro extension. The `app.pro` package only exists in private builds
# (overlaid at Docker build time by an external repo) — its absence in the
# open-source image is expected and silently ignored. Anyone with private
# extensions can drop a package at `backend/app/pro/` exposing
# `register_pro(app)` and have it auto-loaded here without further edits.
try:
from app.pro import register_pro # type: ignore[import-not-found]
register_pro(app)
except ImportError:
pass
@app.get("/")
def root():
return {
"message": "Arduino Emulator API",
"version": "1.0.0",
"docs": "/api/docs",
}
@app.get("/health")
def health_check():
return {"status": "healthy"}