The Dockerfile uses **4 stages** to minimize final image size while building all dependencies.
### Stage 0: qemu-provider
**Base image:** `ubuntu:22.04`
**Purpose:** Downloads pre-built QEMU shared libraries (`.so`) and ESP32 ROM binary files from a GitHub Release. These are the QEMU emulation libraries built from the `qemu-lcgamboa` fork that enable ESP32 and ESP32-C3 simulation.
-`TARGETARCH` is automatically injected by Docker Buildx. It resolves to `amd64` or `arm64` depending on the target platform.
- The stage first checks for local prebuilt files in `prebuilt/qemu/`. If present, those are used (useful for local development). If not present, the files are downloaded from the GitHub Release.
- **Architecture-specific files** (different binary per CPU architecture):
-`libqemu-xtensa-${TARGETARCH}.so` → saved as `libqemu-xtensa.so`
-`libqemu-riscv32-${TARGETARCH}.so` → saved as `libqemu-riscv32.so`
- **Architecture-independent files** (same binary for all architectures):
-`esp32-v3-rom.bin` — ESP32 boot ROM
-`esp32-v3-rom-app.bin` — ESP32 application ROM
-`esp32c3-rom.bin` — ESP32-C3 boot ROM
The renaming from `libqemu-xtensa-amd64.so` to `libqemu-xtensa.so` means the backend code needs no architecture-aware logic — it always loads `libqemu-xtensa.so` regardless of host architecture.
### Stage 0.5: espidf-builder
**Base image:** `ubuntu:22.04`
**Purpose:** Installs the full ESP-IDF 4.4.7 development framework with cross-compiler toolchains for ESP32 (Xtensa) and ESP32-C3 (RISC-V), plus Arduino-as-component for full Arduino API support.
- **ESP-IDF version 4.4.7** is pinned because it matches the QEMU ROM binaries built from the lcgamboa fork. Newer ESP-IDF versions (5.x) are **not compatible** with the ROM images.
- **Arduino ESP32 core 2.0.17** matches IDF 4.4.x. The 3.x series uses IDF 5.x and is incompatible.
-`install.sh esp32,esp32c3` downloads only the Xtensa and RISC-V toolchains (not ESP32-S2, S3, etc.) to reduce image size.
- ESP-IDF's `install.sh`**auto-detects host architecture** and downloads the correct native toolchain (x86_64 or aarch64). No special handling needed for ARM64.
- The `.git` directory, docs, and examples are removed after installation to reduce size. Downloaded `.tar.*` archives in `.espressif` are also cleaned up.
**Important note on ARM64 builds:** When Docker Buildx builds the arm64 variant on an amd64 CI runner, it uses QEMU user-mode emulation. This makes the ESP-IDF stage **very slow** (~30-60 minutes) but works correctly. The GitHub Actions cache (`cache-from: type=gha`) ensures this only happens once.
The git submodule pointers in this repo for `rp2040js` and `wokwi-elements` are stale — they point to very old commits that predate `package.json`. Cloning fresh from GitHub HEAD ensures we get working, up-to-date versions. This is also why the GitHub Actions workflow does **not** use `submodules: recursive` in the checkout step.
**Build order:**
1.`avr8js` — `npm install && npm run build`
2.`rp2040js` — `npm install && npm run build`
3.`wokwi-elements` — `npm install && npm run build`
4. Frontend — `npm install && npm run build:docker`
**`build:docker` vs `build`:** The `build:docker` script runs `vite build` only (no `tsc -b` type-checking). This is intentional because there are known pre-existing TypeScript errors (wokwi-elements JSX custom element types, `@monaco-editor/react` compatibility with React 19) that don't affect runtime behavior.
### Stage 2: Final Production Image
**Base image:** `python:3.12-slim`
**Purpose:** The final, deployable image that contains everything needed to run Velxio.
| `libqemu-xtensa.so` | Built on x86_64 Ubuntu 20.04 | Built on aarch64 Ubuntu 22.04 |
| `libqemu-riscv32.so` | Built on x86_64 Ubuntu 20.04 | Built on aarch64 Ubuntu 22.04 |
The following files are **architecture-independent** (same binary for both):
| File | Description |
|------|-------------|
| `esp32-v3-rom.bin` | ESP32 boot ROM |
| `esp32-v3-rom-app.bin` | ESP32 application ROM |
| `esp32c3-rom.bin` | ESP32-C3 boot ROM |
### ESP-IDF on ARM64
ESP-IDF's `install.sh` automatically detects the host architecture and downloads native toolchains:
- On amd64: downloads `xtensa-esp32-elf-*-linux-amd64.tar.gz`
- On arm64: downloads `xtensa-esp32-elf-*-linux-arm64.tar.gz`
No special handling is needed in the Dockerfile. However, when Buildx is building the arm64 image on an amd64 runner (which is the case in GitHub Actions), it uses QEMU user-mode emulation to run the arm64 container. This makes:
-`install.sh` very slow (downloading + extracting under emulation)
-`pip install` slow
- The overall arm64 build significantly longer than amd64
The GitHub Actions cache (`cache-from: type=gha, cache-to: type=gha,mode=max`) ensures that once the arm64 layers are built, they are cached and reused on subsequent builds.
This workflow compiles the QEMU shared libraries from the lcgamboa fork (a modified QEMU with ESP32/ESP32-C3 machine emulation) and uploads them as GitHub Release assets to the main Velxio repository.
### Matrix Strategy
The workflow uses a matrix strategy to build natively on two different architectures:
```yaml
strategy:
matrix:
include:
- runner: ubuntu-22.04
arch: amd64
container: ubuntu:20.04
- runner: ubuntu-24.04-arm
arch: arm64
container: ubuntu:22.04
```
**Why different containers?**
- **amd64** uses `ubuntu:20.04` for maximum glibc compatibility (glibc 2.31). The resulting `.so` will work on any Linux with glibc >= 2.31.
- **arm64** uses `ubuntu:22.04` because GitHub's `ubuntu-24.04-arm` runners are relatively new and `ubuntu:20.04` arm64 images have occasional package availability issues. glibc 2.35 is used, which is still compatible with the final Docker image (Debian Bookworm, glibc 2.36).
**Why native ARM64 runners?**
QEMU itself is a large C project. Cross-compiling or building under QEMU user-mode emulation would be extremely slow (hours). Using GitHub's native `ubuntu-24.04-arm` runners gives native ARM64 build speed (~15-20 minutes).
### libiconv Stub Workaround
QEMU's configure script adds `-liconv` to the linker flags. On Linux, iconv is part of glibc — there is no separate `libiconv` package. To satisfy the linker without installing a non-existent library:
This creates an empty static archive `libiconv.a` in the architecture-correct library directory. The linker finds it, sees no symbols (none are needed since glibc provides iconv), and is satisfied.
The `dpkg-architecture` command returns the multiarch triplet (e.g., `x86_64-linux-gnu` or `aarch64-linux-gnu`), ensuring the stub is placed in the correct directory for each architecture.
### Artifact Upload to GitHub Release
The workflow has two jobs:
1.**`build`** (runs on both amd64 and arm64):
- Compiles `libqemu-xtensa.so` and `libqemu-riscv32.so`
- Renames with architecture suffix: `libqemu-xtensa-amd64.so`, `libqemu-xtensa-arm64.so`
- ROM files are only collected from the amd64 job (they're architecture-independent)
- Uploads as GitHub Actions artifacts
2.**`upload-release`** (runs after both build jobs complete):
- Downloads both artifact archives
- Uploads all files to the `qemu-prebuilt` tag on the `davidmonterocrespo24/velxio` repository
- Uses `--clobber` to overwrite existing files if the release already exists
- Requires the `VELXIO_RELEASE_TOKEN` secret (a PAT with `contents:write` on the velxio repo)
**Release structure at `github.com/davidmonterocrespo24/velxio/releases/tag/qemu-prebuilt`:**
This uses GitHub Actions' built-in cache backend for Docker layer caching:
-`type=gha` — GitHub Actions cache (up to 10GB per repo)
-`mode=max` — Cache all layers, not just the final stage
This is critical for ARM64 builds because the ESP-IDF stage is very slow under QEMU emulation. Once cached, subsequent builds skip the heavy stages entirely.
ESP-IDF's `export.sh` adds the cross-compiler toolchains to `$PATH` and sets up the build environment. Without sourcing this script, ESP32 compilation would fail.
**Version pinning:** The fallback installs `esp32:esp32@2.0.17` specifically. This is critical because:
- Version 2.0.17 uses IDF 4.4.x internally, matching the QEMU ROM binaries
- Version 3.x uses IDF 5.x, which is **incompatible** with the QEMU ROM images
- Using the wrong version causes boot failures in emulation
# Start Nginx on port 80 (foreground — keeps container alive)
exec nginx -g "daemon off;"
```
The backend binds to `127.0.0.1:8001` (localhost only — not exposed to the network). Nginx on port 80 is the only externally-accessible service and proxies API requests to the backend.
Using `exec nginx` replaces the shell process with Nginx, making it PID 1. This ensures proper signal handling — when Docker sends SIGTERM (on `docker stop`), Nginx receives it directly and shuts down gracefully.
proxy_read_timeout 300s; # 5 minutes — compilation can be slow
proxy_connect_timeout 75s;
}
```
All `/api/*` requests are proxied to the FastAPI backend. The 300-second read timeout accommodates ESP32 compilation, which can take several minutes (especially on first build when ESP-IDF initializes the build cache).
WebSocket connections are used for real-time ESP32 simulation communication. The 24-hour timeout ensures long-running simulation sessions aren't terminated. This location block **must come before** the generic `/api/` block so Nginx matches it with higher priority.
### SPA Routing
```nginx
location / {
try_files $uri $uri/ /index.html;
}
```
This is the standard single-page application (SPA) routing pattern. For any URL that doesn't match a static file or another location block, Nginx serves `index.html` and lets React Router handle the routing client-side.
Vite generates content-hashed filenames (e.g., `index-a1b2c3.js`), so JS/CSS files can be cached indefinitely — when the content changes, the hash changes and a new URL is used.
### SEO Configuration
```nginx
# Never cache sitemap/robots — crawlers always get the latest
Gzip is enabled for text-based content types. The `gzip_min_length 1024` prevents compressing very small responses where compression overhead would exceed savings.
The bind mount for `./data` allows easy database backup and inspection from the host. The named volume for `arduino-libs` persists board core installations across container restarts.
- **`start_period: 90s`** — Gives the container 90 seconds to start before health checks begin counting failures. This accounts for first-boot arduino-cli core installation which can take over a minute.
- The `/health` endpoint is proxied by Nginx to FastAPI's `/health` endpoint, verifying both services are running.
# Build for both architectures (requires push to registry)
docker buildx build \
--platform linux/amd64,linux/arm64 \
-f Dockerfile.standalone \
-t ghcr.io/user/velxio:latest \
--push .
```
---
## Troubleshooting
### "no matching manifest for linux/arm64/v8"
**Problem:** Running `docker pull` on Apple Silicon Mac fails because the image was built for amd64 only.
**Solution:** The multi-platform build was added in the `docker-publish.yml` workflow with `platforms: linux/amd64,linux/arm64`. Ensure:
1. The `build-libqemu.yml` workflow has run successfully for both architectures (check that `libqemu-xtensa-arm64.so` exists in the `qemu-prebuilt` release)
2. The `docker-publish.yml` workflow has run after the QEMU ARM64 binaries were uploaded
3. The `docker/setup-qemu-action@v3` step is present in the workflow
### CRLF line ending issues in entrypoint.sh
**Problem:** When developing on Windows, `entrypoint.sh` may have CRLF line endings, causing `/bin/bash^M: bad interpreter`.
**Solution:** The Dockerfile includes a fix:
```dockerfile
RUN sed -i 's/\r$//' /app/entrypoint.sh && chmod +x /app/entrypoint.sh
```
This strips carriage returns from the file inside the container. No git configuration changes needed.
### ESP-IDF compilation fails with "command not found"
**Problem:** ESP-IDF cross-compilers (`xtensa-esp32-elf-gcc`) not found when compiling ESP32 sketches.
**Cause:** `export.sh` was not sourced, so the toolchains aren't in `$PATH`.
**Solution:** The entrypoint script sources ESP-IDF on startup:
```bash
. /opt/esp-idf/export.sh
```
If running manually inside the container, source it yourself:
```bash
docker exec -it velxio bash
source /opt/esp-idf/export.sh
```
### QEMU .so fails to load (missing shared libraries)
**Problem:** `ctypes.cdll.LoadLibrary` fails with missing dependencies.
**Cause:** The final image is missing runtime dependencies for the QEMU shared library.
**Solution:** The following packages are installed in Stage 2:
**Problem:** Container takes several minutes to become ready on first start.
**Cause:** The entrypoint script downloads and installs arduino-cli board cores:
-`arduino:avr` — ~150MB
-`rp2040:rp2040` — ~300MB
**Solution:** This is expected on first boot. The `arduino-libs` volume persists these installations, so subsequent starts are fast. Use the health check's `start_period: 90s` to give the container enough time.
### Build cache invalidation
**Problem:** Docker rebuild downloads everything from scratch despite no code changes.
**Cause:** The `COPY` instruction invalidates the cache if any file in the context changes.
**Solution:** The Dockerfile is structured to maximize cache reuse:
1. System packages (rarely change) — cached
2.`requirements.txt` copy + `pip install` — only invalidated when dependencies change
3. Application code copy — invalidated on every push
For the GitHub Actions cache:
```yaml
cache-from: type=gha
cache-to: type=gha,mode=max
```
The `mode=max` caches all intermediate layers, not just the final layer. This is important for the ESP-IDF stage which is very slow to build from scratch.
### Docker Hub token permissions
**Problem:** `docker-publish.yml` fails at the Docker Hub login step.
**Solution:** Create a Docker Hub access token:
1. Go to Docker Hub → Account Settings → Security → New Access Token
2. Set permissions to Read & Write
3. Add as GitHub repository secrets:
-`DOCKERHUB_USERNAME` — Your Docker Hub username
-`DOCKERHUB_TOKEN` — The access token
### QEMU build fails for ARM64
**Problem:** The `build-libqemu.yml` workflow fails on the `ubuntu-24.04-arm` runner.
**Possible causes:**
1.**Runner not available:** GitHub's ARM64 runners (`ubuntu-24.04-arm`) require a GitHub plan that supports them. Check your repository's Actions settings.
2.**Package differences:** The ARM64 container uses `ubuntu:22.04` which may have slightly different package versions. Check the build logs for missing dependencies.
3.**libiconv stub path:** The `dpkg-architecture` command must be available. It's part of `dpkg-dev` which may need to be installed explicitly in the container.
### WebSocket connection drops
**Problem:** ESP32 simulation WebSocket disconnects after a period of inactivity.
**Cause:** Default Nginx proxy timeouts.
**Solution:** The Nginx config sets 24-hour timeouts for WebSocket connections:
```nginx
proxy_read_timeout 86400s;
proxy_send_timeout 86400s;
```
If issues persist, check if there's a load balancer or CDN in front of Nginx that has its own timeout settings.
### Compilation timeout
**Problem:** Arduino compilation requests time out.
**Cause:** The default Nginx `proxy_read_timeout` may be too short for ESP32 compilation, which involves the full ESP-IDF build system.
**Solution:** The Nginx config sets a 5-minute timeout for API requests:
```nginx
proxy_read_timeout 300s;
```
For ESP32 first-time compilation (cold build cache), this may still not be enough. The ESP-IDF build system caches intermediate results, so subsequent compilations are much faster.
