velxio/Dockerfile.standalone

201 lines
7.9 KiB
Docker
Raw Normal View History

# ---- Stage 0: QEMU .so + ROM binaries ----
# Downloads arch-specific .so from GitHub Release (e.g. libqemu-xtensa-amd64.so)
# and renames to libqemu-xtensa.so so the backend needs no changes.
# Local prebuilt files (prebuilt/qemu/) are used if present.
FROM ubuntu:22.04 AS qemu-provider
RUN apt-get update && apt-get install -y --no-install-recommends curl ca-certificates \
&& rm -rf /var/lib/apt/lists/*
ARG TARGETARCH
ARG QEMU_RELEASE_URL=https://github.com/davidmonterocrespo24/velxio/releases/download/qemu-prebuilt
# Copy the prebuilt directory (may contain .so+ROM files or just the .gitkeep)
RUN mkdir -p /qemu
COPY prebuilt/qemu/ /qemu/
# Download arch-specific .so and arch-independent ROM files
RUN cd /qemu \
&& for base in libqemu-xtensa libqemu-riscv32; do \
f="${base}.so" ; \
if [ ! -f "$f" ]; then \
echo "Downloading ${base}-${TARGETARCH}.so → $f ..." ; \
curl -fSL -o "$f" "${QEMU_RELEASE_URL}/${base}-${TARGETARCH}.so" ; \
else \
echo "Using local $f ($(stat -c%s "$f") bytes)" ; \
fi ; \
done \
&& for f in esp32-v3-rom.bin esp32-v3-rom-app.bin esp32c3-rom.bin; do \
if [ ! -f "$f" ]; then \
echo "Downloading $f ..." ; \
curl -fSL -o "$f" "${QEMU_RELEASE_URL}/$f" ; \
else \
echo "Using local $f ($(stat -c%s "$f") bytes)" ; \
fi ; \
done \
&& ls -lh /qemu/
# ---- Stage 0.5: ESP-IDF toolchain for ESP32 compilation ----
FROM ubuntu:22.04 AS espidf-builder
RUN apt-get update && apt-get install -y --no-install-recommends \
git wget flex bison gperf python3 python3-pip python3-venv \
cmake ninja-build ccache libffi-dev libssl-dev \
libusb-1.0-0 ca-certificates \
&& rm -rf /var/lib/apt/lists/*
# Install ESP-IDF 4.4.7 (matches Arduino ESP32 core 2.0.17 / lcgamboa QEMU ROM)
RUN git clone -b v4.4.7 --recursive --depth=1 --shallow-submodules \
https://github.com/espressif/esp-idf.git /opt/esp-idf
WORKDIR /opt/esp-idf
# Install toolchains for esp32 (Xtensa) and esp32c3 (RISC-V) only
RUN ./install.sh esp32,esp32c3
# Clean up large unnecessary files to reduce image size
RUN rm -rf .git docs examples \
&& find /root/.espressif -name '*.tar.*' -delete 2>/dev/null || true
# Install Arduino-as-component for full Arduino API support in ESP-IDF builds
RUN git clone --branch 2.0.17 --depth=1 --recursive --shallow-submodules \
https://github.com/espressif/arduino-esp32.git /opt/arduino-esp32 \
&& rm -rf /opt/arduino-esp32/.git
refactor: rename wokwi-libs/ → third-party/ The directory grew well beyond Wokwi-only contents: it now hosts lcgamboa's QEMU fork (qemu-lcgamboa), Espressif's esp32-camera, the ngspice WASM build, fritzing-parts, picowi, an alternative QEMU (qemu-esp32), the 100_Days_100_IoT_Projects examples repo, and Wokwi's own avr8js/rp2040js/wokwi-elements/wokwi-features/wokwi-boards. "wokwi-libs" was misleading — half the contents have nothing to do with Wokwi. "third-party/" is the standard convention for vendored external dependencies. Mechanical changes: Path rename: wokwi-libs/ → third-party/ update-wokwi-libs.bat → update-third-party.bat docs/WOKWI_LIBS.md → docs/THIRD_PARTY.md Submodule reconfiguration: .gitmodules — 4 path= and section names updated .git/modules/wokwi-libs/ → .git/modules/third-party/ each submodule's .git file rewired to ../../.git/modules/third-party/<name> Reference updates (~80 files): vite.config.ts aliases, Dockerfile COPY paths, GH Actions workflow steps, build_qemu_*.sh, all docs/* and test/*/autosearch/* entries that mention the path, package-lock.json file: dependencies, .gitignore patterns, sitemap.xml + index.html SEO blurbs, scripts/generate-component-*, .dockerignore, .idea/vcs.xml. Bulk replaced both `wokwi-libs/` (path) and bare `wokwi-libs` (textual mentions in docs/comments). Verified: - npx tsc -b --noEmit produces no new errors related to these paths - vite.config.ts aliases now point at ../third-party/avr8js etc. - All 4 git submodules (avr8js, rp2040js, wokwi-elements, wokwi-features) are linked under third-party/ with their worktrees re-populated and config files referencing the new path - `grep -r wokwi-libs` returns zero hits outside node_modules, .vite, frontend/dist, third-party/ (upstream submodule contents), *.pyc caches, and *.dll.pre-camera rollback binaries Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 10:58:57 +07:00
# ---- Stage 1: Build frontend and third-party ----
FROM node:20 AS frontend-builder
WORKDIR /app
fix(install): unblock self-hosting + drop forced wokwi clones Resolves several install pain points reported by users (#108, #120) and removes the obligatory upstream-clone step that confused contributors and slowed down every Docker build. Install fixes: - nginx: server_name → catch-all default_server, drop Debian's stock site so reverse-proxied users no longer get the "Welcome to nginx" page. - entrypoint: auto-generate SECRET_KEY at first boot, persisted under data/.secret_key. backend/.env is now optional in docker-compose.yml. - backend: add greenlet>=3.0.0 (SQLAlchemy async dep that was missing on some Python builds — caused uvicorn startup failures on WSL). Wokwi libs come from npm: - @wokwi/elements 1.9.2, avr8js 0.21.0, rp2040js 1.3.2 are pinned in frontend/package.json. Vite aliases removed. - Dockerfile.standalone no longer clones avr8js / rp2040js / wokwi-elements / wokwi-boards. Frontend stage is just COPY + npm install + build:docker. - Board SVGs vendored under frontend/public/boards/ (10 deduped against existing files, 2 truly new). third-party/wokwi-* clones become reference- only credits — generate-component-metadata.ts skips gracefully when absent. Production config split out: - docker-compose.prod.yml, deploy/nginx.prod.conf, nginx-host-velxio*.conf, update-third-party.bat removed. Production deployment lives in its own repo: https://github.com/velxio/velxio-prod (host nginx + HTTPS + backups + pinned upstream commit). Verified locally: 1161 frontend tests pass, build:docker completes clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 10:04:11 +07:00
# avr8js, rp2040js and @wokwi/elements are pulled directly from the npm
# registry (see frontend/package.json) — no upstream git clones needed.
# Board SVGs live in frontend/public/boards/, component SVGs in
# frontend/public/component-svgs/, and components-metadata.json is committed.
COPY frontend/ frontend/
COPY scripts/ scripts/
WORKDIR /app/frontend
# Lock files aren't committed in this repo (they're gitignored) — see the
# note in .gitignore. The `rm -f` below is defense-in-depth in case
# someone runs `docker build .` from a tree where a local lock exists.
RUN rm -f package-lock.json \
&& npm install --include=optional \
&& npm run build:docker
# ---- Stage 2: Final Production Image ----
FROM python:3.12-slim
# Install system dependencies, nginx, and QEMU .so runtime libraries
RUN apt-get update && apt-get install -y --no-install-recommends \
curl \
ca-certificates \
nginx \
libglib2.0-0 \
libgcrypt20 \
libslirp0 \
libpixman-1-0 \
libfdt1 \
cmake \
ninja-build \
libusb-1.0-0 \
git \
perf(espidf): drop in ccache for ESP32 compiles (~10× warm speedup) Cold first compile per container is unchanged (cache empty). Subsequent compiles drop from ~5-7 minutes to ~30-60 seconds because every ESP-IDF base object (FreeRTOS, lwIP, esp_wifi, libsodium, soc, hal, …) hits the cache. The user's BMP280 example, which hangs on cold compile, completes near-instantly on the second attempt. Why a transparent cache is safe: ccache hashes the preprocessed source + flags + compiler. A cache hit only happens when the input is byte-for-byte identical to a prior compile. Different sketches with different libraries still get correct cache misses; there is no path where one project's output contaminates another. Changes - Dockerfile.standalone: install ccache, set CCACHE_DIR=/var/cache/ccache, IDF_CCACHE_ENABLE=1, configure 2 GB cap with compression. Compression (level 6) cuts cache disk usage by ~40% with negligible CPU overhead. - docker-compose.yml: named volume `ccache:/var/cache/ccache` so the cache survives `docker compose up -d --build` (without it, every image rebuild discards the cache). - backend/app/services/espidf_compiler.py: pass `-DCCACHE_ENABLE=1` to cmake when IDF_CCACHE_ENABLE is truthy. ESP-IDF's project.cmake (`set_property(GLOBAL PROPERTY RULE_LAUNCH_COMPILE ccache)` on line 374) is what actually wires ccache in; without the cmake -D flag the env var alone has no effect because we don't go through idf.py. Escape hatch: set IDF_CCACHE_ENABLE=0 in compose env to disable without rebuilding the image. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 10:44:29 +07:00
ccache \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/* \
&& pip install --no-cache-dir packaging
# Install arduino-cli into /usr/local/bin directly (avoids touching /bin)
RUN curl -fsSL https://raw.githubusercontent.com/arduino/arduino-cli/master/install.sh \
| BINDIR=/usr/local/bin sh
# Only install arduino-cli binary here. Core installation (arduino:avr,
# rp2040:rp2040) is done at first boot by entrypoint.sh and persisted
# in the mounted /root/.arduino15 volume.
# ESP32 compilation uses ESP-IDF instead of arduino-cli.
WORKDIR /app
# Data directory for persistent SQLite database (mounted as a volume at runtime)
RUN mkdir -p /app/data
# Install Python backend dependencies
COPY backend/requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# Copy backend application code
COPY backend/app/ ./app/
feat: persist multi-board projects + add auto-save The project save/load pipeline only persisted a single `board_type`, so multi-board workspaces silently lost every board except the active one on save, and wires referencing the dropped boards' IDs orphaned to the canvas corner on reload. An audit of the production backup found 74/306 projects (24%) with at least one orphaned wire and 174/301 non-trivial projects whose code was still the default Blink template — strong signal that users save once and never re-save. Backend - Add `boards_json` column on `projects` with idempotent ALTER TABLE in the lifespan migration list. - New `FileGroup` schema + `file_groups` array on ProjectCreate/Update/Response. Legacy `files`/`code` kept for back-compat. - `project_files.py` now uses `{pid}/{groupId}/{filename}` subdirs via `read_groups`/`write_groups`. Legacy flat layouts are auto-promoted on read; legacy single-list `files` only updates the active group, leaving other boards' files intact. - `_persist_files_from_body` honors file_groups → files → code priority. Frontend - `useSimulatorStore.addBoard` accepts an optional `explicitId` so saved board IDs can be restored verbatim (wires reference IDs literally). - New `loadProjectState({boards, fileGroups, components, wires, activeBoardId})` action: tears down current boards, recreates from the payload, restores file groups atomically, recalculates wire positions on the next frame, and refreshes the Interconnect. - `useEditorStore.replaceFileGroups` for atomic multi-group restore. - `SaveProjectModal` and `ProjectByIdPage`/`ProjectPage` now go through `buildSavePayload` / `buildLoadPayload` (handles pre-backfill projects by synthesising a default board from `board_type`). Auto-save (#useAutoSaveProject hook) - 2.5s debounced silent PUT triggered ONLY when an authenticated user has a `currentProject` with a UUID. State hash detects real changes vs. UI-only churn; baseline is reset on project load so the just-loaded state isn't immediately re-saved. - `beforeunload` flush via `fetch keepalive: true` (supports PUT + credentials, survives unload). - Compact status indicator in `AppHeader` (idle/dirty/saving/saved/error). Backfill script (one-off, idempotent) - `backend/scripts/backfill_boards_2026_05.py` populates `boards_json` for legacy projects. Heuristic per project, based on which board IDs the wires reference: Case A — wires only ref 'arduino-uno' but board_type ≠ uno: rename id→board_type and rewrite wire endpoints. Case B — single-board normal: keep verbatim. Case C — multi-board: recreate one board per distinct ref, infer kind by stripping trailing -N suffix. Also moves any flat files into the active board's group subdir. Stdlib-only, runs from host or `docker exec`. Docker - `Dockerfile.standalone` now copies `backend/scripts/` into the image so the backfill is callable via `docker exec velxio-app python /app/scripts/backfill_boards_2026_05.py --apply`. Verified locally on the restored production backup (363 projects): 33 Case A, 316 Case B, 14 Case C, 135 wire endpoints renamed, 0 orphans. Re-running the script after apply skips all 363 (idempotent). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 23:43:33 +07:00
# One-off maintenance scripts (e.g. backfill_boards_2026_05). Pure stdlib —
# run with: docker exec velxio-app python /app/scripts/<script> --apply
COPY backend/scripts/ ./scripts/
fix(install): unblock self-hosting + drop forced wokwi clones Resolves several install pain points reported by users (#108, #120) and removes the obligatory upstream-clone step that confused contributors and slowed down every Docker build. Install fixes: - nginx: server_name → catch-all default_server, drop Debian's stock site so reverse-proxied users no longer get the "Welcome to nginx" page. - entrypoint: auto-generate SECRET_KEY at first boot, persisted under data/.secret_key. backend/.env is now optional in docker-compose.yml. - backend: add greenlet>=3.0.0 (SQLAlchemy async dep that was missing on some Python builds — caused uvicorn startup failures on WSL). Wokwi libs come from npm: - @wokwi/elements 1.9.2, avr8js 0.21.0, rp2040js 1.3.2 are pinned in frontend/package.json. Vite aliases removed. - Dockerfile.standalone no longer clones avr8js / rp2040js / wokwi-elements / wokwi-boards. Frontend stage is just COPY + npm install + build:docker. - Board SVGs vendored under frontend/public/boards/ (10 deduped against existing files, 2 truly new). third-party/wokwi-* clones become reference- only credits — generate-component-metadata.ts skips gracefully when absent. Production config split out: - docker-compose.prod.yml, deploy/nginx.prod.conf, nginx-host-velxio*.conf, update-third-party.bat removed. Production deployment lives in its own repo: https://github.com/velxio/velxio-prod (host nginx + HTTPS + backups + pinned upstream commit). Verified locally: 1161 frontend tests pass, build:docker completes clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 10:04:11 +07:00
# Setup Nginx configuration. Remove Debian's stock site so it doesn't shadow
# ours as the default_server (was Issue #108: users behind reverse proxies got
# the "Welcome to nginx" page because the stock site claimed default_server).
RUN rm -f /etc/nginx/sites-enabled/default
COPY docker/nginx.conf /etc/nginx/conf.d/default.conf
# Copy built frontend assets from builder stage
COPY --from=frontend-builder /app/frontend/dist /usr/share/nginx/html
# Copy and configure entrypoint script (fix Windows CRLF → LF)
COPY docker/entrypoint.sh /app/entrypoint.sh
RUN sed -i 's/\r$//' /app/entrypoint.sh && chmod +x /app/entrypoint.sh
# ── ESP32 emulation: pre-built QEMU .so + ROM binaries ──────────────────────
# Downloaded from GitHub Release (public — no access to qemu-lcgamboa needed)
# libqemu-xtensa.so → ESP32 / ESP32-S3 (Xtensa LX6/LX7)
# libqemu-riscv32.so → ESP32-C3 (RISC-V RV32IMC)
# esp32-v3-rom*.bin → boot/app ROM images required by esp32-picsimlab machine
# esp32c3-rom.bin → ROM image required by esp32c3-picsimlab machine
# NOTE: ROM files must live in the same directory as the .so (worker passes -L
# to QEMU pointing at os.path.dirname(lib_path))
RUN mkdir -p /app/lib
COPY --from=qemu-provider /qemu/ /app/lib/
# Activate ESP32 emulation
# QEMU_ESP32_LIB → Xtensa library (ESP32, ESP32-S3)
# QEMU_RISCV32_LIB → RISC-V library (ESP32-C3 and variants)
ENV QEMU_ESP32_LIB=/app/lib/libqemu-xtensa.so
ENV QEMU_RISCV32_LIB=/app/lib/libqemu-riscv32.so
# ── ESP-IDF toolchain for ESP32 compilation ──────────────────────────────────
# Copied from espidf-builder stage: IDF framework + cross-compiler toolchains
COPY --from=espidf-builder /opt/esp-idf /opt/esp-idf
COPY --from=espidf-builder /root/.espressif /root/.espressif
COPY --from=espidf-builder /opt/arduino-esp32 /opt/arduino-esp32
ENV IDF_PATH=/opt/esp-idf
ENV IDF_TOOLS_PATH=/root/.espressif
ENV ARDUINO_ESP32_PATH=/opt/arduino-esp32
perf(espidf): drop in ccache for ESP32 compiles (~10× warm speedup) Cold first compile per container is unchanged (cache empty). Subsequent compiles drop from ~5-7 minutes to ~30-60 seconds because every ESP-IDF base object (FreeRTOS, lwIP, esp_wifi, libsodium, soc, hal, …) hits the cache. The user's BMP280 example, which hangs on cold compile, completes near-instantly on the second attempt. Why a transparent cache is safe: ccache hashes the preprocessed source + flags + compiler. A cache hit only happens when the input is byte-for-byte identical to a prior compile. Different sketches with different libraries still get correct cache misses; there is no path where one project's output contaminates another. Changes - Dockerfile.standalone: install ccache, set CCACHE_DIR=/var/cache/ccache, IDF_CCACHE_ENABLE=1, configure 2 GB cap with compression. Compression (level 6) cuts cache disk usage by ~40% with negligible CPU overhead. - docker-compose.yml: named volume `ccache:/var/cache/ccache` so the cache survives `docker compose up -d --build` (without it, every image rebuild discards the cache). - backend/app/services/espidf_compiler.py: pass `-DCCACHE_ENABLE=1` to cmake when IDF_CCACHE_ENABLE is truthy. ESP-IDF's project.cmake (`set_property(GLOBAL PROPERTY RULE_LAUNCH_COMPILE ccache)` on line 374) is what actually wires ccache in; without the cmake -D flag the env var alone has no effect because we don't go through idf.py. Escape hatch: set IDF_CCACHE_ENABLE=0 in compose env to disable without rebuilding the image. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 10:44:29 +07:00
# ── ccache for ESP-IDF compiles ──────────────────────────────────────────────
# ESP-IDF's build system honours IDF_CCACHE_ENABLE=1 and routes every C/C++
# compile through ccache. Cold first compile per container is unchanged
# (cache is empty), but the second and subsequent compiles drop from
# ~5-7 minutes to ~30-60 seconds because every ESP-IDF base object
# (FreeRTOS, lwIP, esp_wifi, libsodium, …) hits the cache.
#
# Cache lives at /var/cache/ccache. Mount as a docker volume in
# docker-compose.yml so the cache survives `docker compose up -d --build`.
# Without the volume, the cache rebuilds itself on first compile after each
# image rebuild — still better than no cache.
ENV CCACHE_DIR=/var/cache/ccache
ENV IDF_CCACHE_ENABLE=1
RUN mkdir -p /var/cache/ccache \
&& ccache --max-size=2G \
&& ccache --set-config=compression=true \
&& ccache --set-config=compression_level=6
# Install ESP-IDF Python dependencies using the final image's Python
# The requirements.txt has version constraints required by ESP-IDF 4.4.x
RUN grep -v 'esp-windows-curses' /opt/esp-idf/requirements.txt \
| pip install --no-cache-dir -r /dev/stdin
EXPOSE 80
CMD ["/app/entrypoint.sh"]