velxio/docker-compose.yml

53 lines
1.8 KiB
YAML
Raw Normal View History

# Development Docker Compose — same image as production for full parity.
# Usage: docker compose up --build
# Access: http://localhost:3080
fix(install): unblock self-hosting + drop forced wokwi clones Resolves several install pain points reported by users (#108, #120) and removes the obligatory upstream-clone step that confused contributors and slowed down every Docker build. Install fixes: - nginx: server_name → catch-all default_server, drop Debian's stock site so reverse-proxied users no longer get the "Welcome to nginx" page. - entrypoint: auto-generate SECRET_KEY at first boot, persisted under data/.secret_key. backend/.env is now optional in docker-compose.yml. - backend: add greenlet>=3.0.0 (SQLAlchemy async dep that was missing on some Python builds — caused uvicorn startup failures on WSL). Wokwi libs come from npm: - @wokwi/elements 1.9.2, avr8js 0.21.0, rp2040js 1.3.2 are pinned in frontend/package.json. Vite aliases removed. - Dockerfile.standalone no longer clones avr8js / rp2040js / wokwi-elements / wokwi-boards. Frontend stage is just COPY + npm install + build:docker. - Board SVGs vendored under frontend/public/boards/ (10 deduped against existing files, 2 truly new). third-party/wokwi-* clones become reference- only credits — generate-component-metadata.ts skips gracefully when absent. Production config split out: - docker-compose.prod.yml, deploy/nginx.prod.conf, nginx-host-velxio*.conf, update-third-party.bat removed. Production deployment lives in its own repo: https://github.com/velxio/velxio-prod (host nginx + HTTPS + backups + pinned upstream commit). Verified locally: 1161 frontend tests pass, build:docker completes clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 10:04:11 +07:00
#
# backend/.env is OPTIONAL. If missing, the entrypoint generates a random
# SECRET_KEY so the app boots out-of-the-box. Create backend/.env (copy
# backend/.env.example) only when you need OAuth or want a fixed SECRET_KEY.
services:
velxio:
build:
context: .
dockerfile: Dockerfile.standalone
container_name: velxio-dev
restart: unless-stopped
ports:
- "3080:80"
env_file:
fix(install): unblock self-hosting + drop forced wokwi clones Resolves several install pain points reported by users (#108, #120) and removes the obligatory upstream-clone step that confused contributors and slowed down every Docker build. Install fixes: - nginx: server_name → catch-all default_server, drop Debian's stock site so reverse-proxied users no longer get the "Welcome to nginx" page. - entrypoint: auto-generate SECRET_KEY at first boot, persisted under data/.secret_key. backend/.env is now optional in docker-compose.yml. - backend: add greenlet>=3.0.0 (SQLAlchemy async dep that was missing on some Python builds — caused uvicorn startup failures on WSL). Wokwi libs come from npm: - @wokwi/elements 1.9.2, avr8js 0.21.0, rp2040js 1.3.2 are pinned in frontend/package.json. Vite aliases removed. - Dockerfile.standalone no longer clones avr8js / rp2040js / wokwi-elements / wokwi-boards. Frontend stage is just COPY + npm install + build:docker. - Board SVGs vendored under frontend/public/boards/ (10 deduped against existing files, 2 truly new). third-party/wokwi-* clones become reference- only credits — generate-component-metadata.ts skips gracefully when absent. Production config split out: - docker-compose.prod.yml, deploy/nginx.prod.conf, nginx-host-velxio*.conf, update-third-party.bat removed. Production deployment lives in its own repo: https://github.com/velxio/velxio-prod (host nginx + HTTPS + backups + pinned upstream commit). Verified locally: 1161 frontend tests pass, build:docker completes clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 10:04:11 +07:00
- path: ./backend/.env
required: false
environment:
- DATABASE_URL=sqlite+aiosqlite:////app/data/velxio.db
- DATA_DIR=/app/data
- IDF_PATH=/opt/esp-idf
- IDF_TOOLS_PATH=/root/.espressif
- ARDUINO_ESP32_PATH=/opt/arduino-esp32
perf(espidf): drop in ccache for ESP32 compiles (~10× warm speedup) Cold first compile per container is unchanged (cache empty). Subsequent compiles drop from ~5-7 minutes to ~30-60 seconds because every ESP-IDF base object (FreeRTOS, lwIP, esp_wifi, libsodium, soc, hal, …) hits the cache. The user's BMP280 example, which hangs on cold compile, completes near-instantly on the second attempt. Why a transparent cache is safe: ccache hashes the preprocessed source + flags + compiler. A cache hit only happens when the input is byte-for-byte identical to a prior compile. Different sketches with different libraries still get correct cache misses; there is no path where one project's output contaminates another. Changes - Dockerfile.standalone: install ccache, set CCACHE_DIR=/var/cache/ccache, IDF_CCACHE_ENABLE=1, configure 2 GB cap with compression. Compression (level 6) cuts cache disk usage by ~40% with negligible CPU overhead. - docker-compose.yml: named volume `ccache:/var/cache/ccache` so the cache survives `docker compose up -d --build` (without it, every image rebuild discards the cache). - backend/app/services/espidf_compiler.py: pass `-DCCACHE_ENABLE=1` to cmake when IDF_CCACHE_ENABLE is truthy. ESP-IDF's project.cmake (`set_property(GLOBAL PROPERTY RULE_LAUNCH_COMPILE ccache)` on line 374) is what actually wires ccache in; without the cmake -D flag the env var alone has no effect because we don't go through idf.py. Escape hatch: set IDF_CCACHE_ENABLE=0 in compose env to disable without rebuilding the image. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 10:44:29 +07:00
# ccache for ESP-IDF compiles — the named volume below persists hits
# across container restarts and image rebuilds. Set IDF_CCACHE_ENABLE=0
# to disable without changing the Dockerfile.
- IDF_CCACHE_ENABLE=1
- CCACHE_DIR=/var/cache/ccache
perf(compile): dedup, concurrency limits, and persistent build dir for ESP-IDF Three coordinated fixes that together close the "ESP-IDF compile takes 5-7 min every time" gap and prevent the failure mode where a user clicking compile multiple times spawns six ninja processes that peel each other apart on a modest VPS. What was wrong - /compile/start generated a fresh uuid4 every call, so 6 clicks = 6 independent builds racing each other. Saw load average 30 on the prod VPS during a real BMP280 attempt today. - No concurrency limit anywhere; asyncio.create_task() fired without gating. - ccache was wired in last week (PR #149) but reported 18,350 cacheable calls and **0 hits** because the build dir was a fresh tempfile.TemporaryDirectory(prefix='espidf_') per compile. The random /tmp/espidf_<random>/ path baked into -I and -fmacro-prefix-map flags → different command line every compile → ccache hash miss every time. What this PR does 1. Job deduplication (`backend/app/api/routes/compile.py`) - New `_job_key(files, board_fqbn)` returns SHA-256 of normalised file names + contents + board. Order-independent. - New `JOB_BY_KEY: dict[str, str]` indexes hash → job_id. - `compile_start` checks JOB_BY_KEY before spawning a new task; if a job for this exact content is already pending or running, returns the existing job_id (logs `[compile] dedup hit — reusing job <id>`). - `_purge_expired_jobs` evicts both COMPILE_JOBS and JOB_BY_KEY, keeping the index consistent. Edge case where two jobs share a key (old finished, new running) is handled — only evict the key entry if it still points at the purged job. 2. Concurrency control (`backend/app/api/routes/compile.py`) - `_COMPILE_SEMAPHORE = asyncio.Semaphore(2)` global cap on simultaneous compiles. - `_target_lock(board_fqbn)` returns a per-target asyncio.Lock so concurrent compiles to the SAME board (sharing the persistent build dir) serialise. Different boards still run in parallel up to the semaphore cap. - `_compile_job` acquires sema → per-target lock → flips state to `running` → calls `_run_compile`. Pending state now accurately reflects "queued waiting for resources". 3. Persistent build dir (`backend/app/services/espidf_compiler.py`) - New `_prepare_persistent_project_dir(idf_target)` materialises `/var/lib/velxio-build/<target>/project/` from the template on first use; on subsequent compiles it wipes only `main/` and `user_libs/` (the per-compile parts) and leaves `build/` alone so ninja's incremental cache + ccache .o files survive. - Toolchain version sentinel (`.idf_version`) wipes the whole target dir if the ESP-IDF or arduino-esp32 version changes — cached objects from the old toolchain are no longer ABI-compatible. - `compile()` is now a thin dispatcher: persistent path or fallback to the legacy `tempfile.TemporaryDirectory()` flow. The actual build logic was extracted into `_compile_in_dir()` so both paths share one implementation, no duplication. - Escape hatch: `VELXIO_PERSISTENT_BUILD_DIR=0` env var falls back to the tempfile path without rebuilding the image. Critical for production safety. 4. ccache normalisation (`Dockerfile.standalone`) - + `ENV CCACHE_BASEDIR=/var/lib/velxio-build` makes ccache canonicalise absolute paths under that prefix when computing the cache key. Robustens hits against any future subdir rearrangement. 5. Docker compose (`docker-compose.yml`) - + named volume `velxio-build:/var/lib/velxio-build` so the persistent build dir survives `docker compose up -d --build`. - + env `VELXIO_PERSISTENT_BUILD_DIR=1` (default ON; users disable without rebuilding). Expected impact - Cold first compile per container per target: unchanged (~5-7 min). - Same sketch re-compiled: ~2-5 s (everything cached). - Different sketch, same target: ~5-30 s (only user code + new lib steps rebuild; ESP-IDF base hits cache). - Different sketch with new libraries: ~30-90 s (new lib component compiles; rest hits cache). - Concurrent clicks on same example: 1 build, others poll the same job_id. No more six-ninja meltdown. Tests - `test/backend/unit/test_compile_dedup.py` covers `_job_key` stability + variance and `_purge_expired_jobs` consistency (including the "two jobs share a key" edge case). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 13:33:11 +07:00
# Persistent ESP-IDF build dir (one per board target). Lets ninja's
# incremental cache + ccache hits combine across compiles, taking warm
# builds from ~5-7 min to ~5-30s. Set VELXIO_PERSISTENT_BUILD_DIR=0
# to disable without rebuilding the image (fallback to per-compile
# tempfile.TemporaryDirectory behaviour).
- VELXIO_PERSISTENT_BUILD_DIR=1
volumes:
- ./data:/app/data
- arduino-libs:/root/.arduino15
perf(espidf): drop in ccache for ESP32 compiles (~10× warm speedup) Cold first compile per container is unchanged (cache empty). Subsequent compiles drop from ~5-7 minutes to ~30-60 seconds because every ESP-IDF base object (FreeRTOS, lwIP, esp_wifi, libsodium, soc, hal, …) hits the cache. The user's BMP280 example, which hangs on cold compile, completes near-instantly on the second attempt. Why a transparent cache is safe: ccache hashes the preprocessed source + flags + compiler. A cache hit only happens when the input is byte-for-byte identical to a prior compile. Different sketches with different libraries still get correct cache misses; there is no path where one project's output contaminates another. Changes - Dockerfile.standalone: install ccache, set CCACHE_DIR=/var/cache/ccache, IDF_CCACHE_ENABLE=1, configure 2 GB cap with compression. Compression (level 6) cuts cache disk usage by ~40% with negligible CPU overhead. - docker-compose.yml: named volume `ccache:/var/cache/ccache` so the cache survives `docker compose up -d --build` (without it, every image rebuild discards the cache). - backend/app/services/espidf_compiler.py: pass `-DCCACHE_ENABLE=1` to cmake when IDF_CCACHE_ENABLE is truthy. ESP-IDF's project.cmake (`set_property(GLOBAL PROPERTY RULE_LAUNCH_COMPILE ccache)` on line 374) is what actually wires ccache in; without the cmake -D flag the env var alone has no effect because we don't go through idf.py. Escape hatch: set IDF_CCACHE_ENABLE=0 in compose env to disable without rebuilding the image. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 10:44:29 +07:00
- ccache:/var/cache/ccache
perf(compile): dedup, concurrency limits, and persistent build dir for ESP-IDF Three coordinated fixes that together close the "ESP-IDF compile takes 5-7 min every time" gap and prevent the failure mode where a user clicking compile multiple times spawns six ninja processes that peel each other apart on a modest VPS. What was wrong - /compile/start generated a fresh uuid4 every call, so 6 clicks = 6 independent builds racing each other. Saw load average 30 on the prod VPS during a real BMP280 attempt today. - No concurrency limit anywhere; asyncio.create_task() fired without gating. - ccache was wired in last week (PR #149) but reported 18,350 cacheable calls and **0 hits** because the build dir was a fresh tempfile.TemporaryDirectory(prefix='espidf_') per compile. The random /tmp/espidf_<random>/ path baked into -I and -fmacro-prefix-map flags → different command line every compile → ccache hash miss every time. What this PR does 1. Job deduplication (`backend/app/api/routes/compile.py`) - New `_job_key(files, board_fqbn)` returns SHA-256 of normalised file names + contents + board. Order-independent. - New `JOB_BY_KEY: dict[str, str]` indexes hash → job_id. - `compile_start` checks JOB_BY_KEY before spawning a new task; if a job for this exact content is already pending or running, returns the existing job_id (logs `[compile] dedup hit — reusing job <id>`). - `_purge_expired_jobs` evicts both COMPILE_JOBS and JOB_BY_KEY, keeping the index consistent. Edge case where two jobs share a key (old finished, new running) is handled — only evict the key entry if it still points at the purged job. 2. Concurrency control (`backend/app/api/routes/compile.py`) - `_COMPILE_SEMAPHORE = asyncio.Semaphore(2)` global cap on simultaneous compiles. - `_target_lock(board_fqbn)` returns a per-target asyncio.Lock so concurrent compiles to the SAME board (sharing the persistent build dir) serialise. Different boards still run in parallel up to the semaphore cap. - `_compile_job` acquires sema → per-target lock → flips state to `running` → calls `_run_compile`. Pending state now accurately reflects "queued waiting for resources". 3. Persistent build dir (`backend/app/services/espidf_compiler.py`) - New `_prepare_persistent_project_dir(idf_target)` materialises `/var/lib/velxio-build/<target>/project/` from the template on first use; on subsequent compiles it wipes only `main/` and `user_libs/` (the per-compile parts) and leaves `build/` alone so ninja's incremental cache + ccache .o files survive. - Toolchain version sentinel (`.idf_version`) wipes the whole target dir if the ESP-IDF or arduino-esp32 version changes — cached objects from the old toolchain are no longer ABI-compatible. - `compile()` is now a thin dispatcher: persistent path or fallback to the legacy `tempfile.TemporaryDirectory()` flow. The actual build logic was extracted into `_compile_in_dir()` so both paths share one implementation, no duplication. - Escape hatch: `VELXIO_PERSISTENT_BUILD_DIR=0` env var falls back to the tempfile path without rebuilding the image. Critical for production safety. 4. ccache normalisation (`Dockerfile.standalone`) - + `ENV CCACHE_BASEDIR=/var/lib/velxio-build` makes ccache canonicalise absolute paths under that prefix when computing the cache key. Robustens hits against any future subdir rearrangement. 5. Docker compose (`docker-compose.yml`) - + named volume `velxio-build:/var/lib/velxio-build` so the persistent build dir survives `docker compose up -d --build`. - + env `VELXIO_PERSISTENT_BUILD_DIR=1` (default ON; users disable without rebuilding). Expected impact - Cold first compile per container per target: unchanged (~5-7 min). - Same sketch re-compiled: ~2-5 s (everything cached). - Different sketch, same target: ~5-30 s (only user code + new lib steps rebuild; ESP-IDF base hits cache). - Different sketch with new libraries: ~30-90 s (new lib component compiles; rest hits cache). - Concurrent clicks on same example: 1 build, others poll the same job_id. No more six-ninja meltdown. Tests - `test/backend/unit/test_compile_dedup.py` covers `_job_key` stability + variance and `_purge_expired_jobs` consistency (including the "two jobs share a key" edge case). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 13:33:11 +07:00
- velxio-build:/var/lib/velxio-build
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 90s
volumes:
arduino-libs:
perf(espidf): drop in ccache for ESP32 compiles (~10× warm speedup) Cold first compile per container is unchanged (cache empty). Subsequent compiles drop from ~5-7 minutes to ~30-60 seconds because every ESP-IDF base object (FreeRTOS, lwIP, esp_wifi, libsodium, soc, hal, …) hits the cache. The user's BMP280 example, which hangs on cold compile, completes near-instantly on the second attempt. Why a transparent cache is safe: ccache hashes the preprocessed source + flags + compiler. A cache hit only happens when the input is byte-for-byte identical to a prior compile. Different sketches with different libraries still get correct cache misses; there is no path where one project's output contaminates another. Changes - Dockerfile.standalone: install ccache, set CCACHE_DIR=/var/cache/ccache, IDF_CCACHE_ENABLE=1, configure 2 GB cap with compression. Compression (level 6) cuts cache disk usage by ~40% with negligible CPU overhead. - docker-compose.yml: named volume `ccache:/var/cache/ccache` so the cache survives `docker compose up -d --build` (without it, every image rebuild discards the cache). - backend/app/services/espidf_compiler.py: pass `-DCCACHE_ENABLE=1` to cmake when IDF_CCACHE_ENABLE is truthy. ESP-IDF's project.cmake (`set_property(GLOBAL PROPERTY RULE_LAUNCH_COMPILE ccache)` on line 374) is what actually wires ccache in; without the cmake -D flag the env var alone has no effect because we don't go through idf.py. Escape hatch: set IDF_CCACHE_ENABLE=0 in compose env to disable without rebuilding the image. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 10:44:29 +07:00
ccache:
perf(compile): dedup, concurrency limits, and persistent build dir for ESP-IDF Three coordinated fixes that together close the "ESP-IDF compile takes 5-7 min every time" gap and prevent the failure mode where a user clicking compile multiple times spawns six ninja processes that peel each other apart on a modest VPS. What was wrong - /compile/start generated a fresh uuid4 every call, so 6 clicks = 6 independent builds racing each other. Saw load average 30 on the prod VPS during a real BMP280 attempt today. - No concurrency limit anywhere; asyncio.create_task() fired without gating. - ccache was wired in last week (PR #149) but reported 18,350 cacheable calls and **0 hits** because the build dir was a fresh tempfile.TemporaryDirectory(prefix='espidf_') per compile. The random /tmp/espidf_<random>/ path baked into -I and -fmacro-prefix-map flags → different command line every compile → ccache hash miss every time. What this PR does 1. Job deduplication (`backend/app/api/routes/compile.py`) - New `_job_key(files, board_fqbn)` returns SHA-256 of normalised file names + contents + board. Order-independent. - New `JOB_BY_KEY: dict[str, str]` indexes hash → job_id. - `compile_start` checks JOB_BY_KEY before spawning a new task; if a job for this exact content is already pending or running, returns the existing job_id (logs `[compile] dedup hit — reusing job <id>`). - `_purge_expired_jobs` evicts both COMPILE_JOBS and JOB_BY_KEY, keeping the index consistent. Edge case where two jobs share a key (old finished, new running) is handled — only evict the key entry if it still points at the purged job. 2. Concurrency control (`backend/app/api/routes/compile.py`) - `_COMPILE_SEMAPHORE = asyncio.Semaphore(2)` global cap on simultaneous compiles. - `_target_lock(board_fqbn)` returns a per-target asyncio.Lock so concurrent compiles to the SAME board (sharing the persistent build dir) serialise. Different boards still run in parallel up to the semaphore cap. - `_compile_job` acquires sema → per-target lock → flips state to `running` → calls `_run_compile`. Pending state now accurately reflects "queued waiting for resources". 3. Persistent build dir (`backend/app/services/espidf_compiler.py`) - New `_prepare_persistent_project_dir(idf_target)` materialises `/var/lib/velxio-build/<target>/project/` from the template on first use; on subsequent compiles it wipes only `main/` and `user_libs/` (the per-compile parts) and leaves `build/` alone so ninja's incremental cache + ccache .o files survive. - Toolchain version sentinel (`.idf_version`) wipes the whole target dir if the ESP-IDF or arduino-esp32 version changes — cached objects from the old toolchain are no longer ABI-compatible. - `compile()` is now a thin dispatcher: persistent path or fallback to the legacy `tempfile.TemporaryDirectory()` flow. The actual build logic was extracted into `_compile_in_dir()` so both paths share one implementation, no duplication. - Escape hatch: `VELXIO_PERSISTENT_BUILD_DIR=0` env var falls back to the tempfile path without rebuilding the image. Critical for production safety. 4. ccache normalisation (`Dockerfile.standalone`) - + `ENV CCACHE_BASEDIR=/var/lib/velxio-build` makes ccache canonicalise absolute paths under that prefix when computing the cache key. Robustens hits against any future subdir rearrangement. 5. Docker compose (`docker-compose.yml`) - + named volume `velxio-build:/var/lib/velxio-build` so the persistent build dir survives `docker compose up -d --build`. - + env `VELXIO_PERSISTENT_BUILD_DIR=1` (default ON; users disable without rebuilding). Expected impact - Cold first compile per container per target: unchanged (~5-7 min). - Same sketch re-compiled: ~2-5 s (everything cached). - Different sketch, same target: ~5-30 s (only user code + new lib steps rebuild; ESP-IDF base hits cache). - Different sketch with new libraries: ~30-90 s (new lib component compiles; rest hits cache). - Concurrent clicks on same example: 1 build, others poll the same job_id. No more six-ninja meltdown. Tests - `test/backend/unit/test_compile_dedup.py` covers `_job_key` stability + variance and `_purge_expired_jobs` consistency (including the "two jobs share a key" edge case). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 13:33:11 +07:00
velxio-build: