a2nr
/
velxio
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
velxio/backend/app/services/boot_images/manifest.json

38 lines
1.7 KiB
JSON
Raw Normal View History

feat(sim): boot_images module + Pi 3 emulation restored Pi 3 simulation had been broken since at least April 2026 (51 fail-events / 24h per docs/PI3_EMULATION_BROKEN.md). Two distinct defects compounded: 1. qemu_manager.py hard-coded paths for kernel8.img, a device-tree blob (under a DOS 8.3 short name!), and a 5.4 GiB Raspberry Pi OS SD image — none of which shipped in the repo or were pulled at image build. 2. qemu-system-arm + qemu-utils were missing from the Docker image entirely, so even with the boot files in place QEMU couldn't launch. Add both to Dockerfile.standalone (~200 MB). The architecture fix is a new `app.services.boot_images` module: * Manifest-driven (boot_images/manifest.json, versioned in repo, declares SHA256 + size for each file, supports an optional `compressed.{encoding,sha256,size_bytes}` block for assets shipped as .zst). * `BootImageProvider` materialises files lazily, atomically (temp + rename), verifies SHA256 pre- AND post-decompression, caches under /var/cache/velxio/boot-images, serialises concurrent get() calls per image set via asyncio.Lock. * `AssetDownloader` Protocol with two impls: - `LicenseGatedDownloader` — same flow ESP32 / RISC-V QEMU libs use (VELXIO_BINARY_BASE_URL + VELXIO_LICENSE_KEY). - `LocalDirectoryDownloader` — for tests + in-prod use where the licence-module storage is already on the same filesystem (saves the loopback HTTP roundtrip on a 1.4 GiB blob). * `build_downloader_from_env()` picks one — local-dir wins if both sets of env vars are present, so the prod box short-circuits to direct disk reads automatically. * Lifespan hook in qemu_manager.py pre-warms the cache on container boot so first-time user requests don't pay the 30-60 s download + decompress latency. Adding a future board kind (Pi 4 / Pi 5) is now: upload assets via upload-binary.sh, append an entry to manifest.json, register a lifespan pre-warm in the new board's service module. Zero edits to provider.py / downloader.py. Manifest entries for raspberry-pi-3: kernel8.img 9 695 883 bytes (uncompressed) bcm2710-rpi-3-b.dtb 34 687 bytes (uncompressed) raspios-trixie-armhf.img 5 729 419 264 bytes raw / 1 488 002 803 bytes .zst on wire (zstd -19) source: 2026-04-21 build from raspberrypi.com Tests: 21 new unit tests covering manifest parsing, integrity helpers, both downloaders, and the provider's idempotent / concurrent / integrity / decompression / warmup paths. In-process FakeDownloader keeps the suite under 1 s and httpx-free. Docs: new docs/BOOT_IMAGES.md describes the architecture, on-disk layout, named-volume operation, and the procedure for adding a new image set.
2026-05-16 10:41:01 +07:00
{
"_comment": "Source of truth for QEMU boot files. Adding a new board kind means appending an entry under image_sets; the BootImageProvider picks it up via load_manifest() with no Python edits. SHA256s are verified after download; mismatches raise IntegrityError and the cache slot is not populated. See docs/BOOT_IMAGES.md for the architecture.",
"version": 1,
"image_sets": {
"raspberry-pi-3": {
"description": "Raspberry Pi 3 Model B boot files — kernel, device tree, and Raspberry Pi OS Trixie (armhf) SD image. Targets the QEMU 'raspi3b' machine type. The SD image ships compressed (zstd -19) to cut the licence-endpoint download from ~5.3 GiB to ~1.4 GiB; the provider decompresses on the host into the named volume cache.",
"images": [
{
"name": "kernel8.img",
"asset_id": "kernel8-pi3",
fix(pi3): decompressed kernel + explicit earlycon PL011 address Two more defects making Pi 3 boot silently: 1. The kernel8.img that ships in the Pi OS armhf boot partition is a gzip-compressed PE-COFF Image (first 4 bytes 0x1f8b0800). QEMU's `-kernel` does NOT auto-decompress; it tries to execute the gzip header as ARM code and the CPU faults immediately. Result: zero bytes on ttyAMA0, simulator looks dead. Switch the asset_id to a pre-decompressed kernel (24 MiB raw vs 9.7 MiB gzipped) so QEMU gets a valid Image to boot. 2. Even with a real kernel, the original cmdline `console=ttyAMA0` alone wasn't enough — the kernel can't initialise the BCM2837 PL011 UART early enough for `printk` to reach the serial console under QEMU's bare-metal boot (no Pi firmware to set it up beforehand). Adding `earlycon=pl011,mmio32,0x3f201000` makes the kernel program the UART itself in the early boot path. Verified: boot output starts streaming within 100 ms of QEMU launch instead of never. The cmdline also locks the baud rate at 115200 to match the agetty drop-in created by scripts/configure-pi3-autologin.sh.
2026-05-16 11:46:04 +07:00
"sha256": "2f87328ab8d36c266ac0545e59a7eed53841e146a75e0010929d3dafd0209310",
"size_bytes": 24367616,
"version": "2026-03-11+raw"
feat(sim): boot_images module + Pi 3 emulation restored Pi 3 simulation had been broken since at least April 2026 (51 fail-events / 24h per docs/PI3_EMULATION_BROKEN.md). Two distinct defects compounded: 1. qemu_manager.py hard-coded paths for kernel8.img, a device-tree blob (under a DOS 8.3 short name!), and a 5.4 GiB Raspberry Pi OS SD image — none of which shipped in the repo or were pulled at image build. 2. qemu-system-arm + qemu-utils were missing from the Docker image entirely, so even with the boot files in place QEMU couldn't launch. Add both to Dockerfile.standalone (~200 MB). The architecture fix is a new `app.services.boot_images` module: * Manifest-driven (boot_images/manifest.json, versioned in repo, declares SHA256 + size for each file, supports an optional `compressed.{encoding,sha256,size_bytes}` block for assets shipped as .zst). * `BootImageProvider` materialises files lazily, atomically (temp + rename), verifies SHA256 pre- AND post-decompression, caches under /var/cache/velxio/boot-images, serialises concurrent get() calls per image set via asyncio.Lock. * `AssetDownloader` Protocol with two impls: - `LicenseGatedDownloader` — same flow ESP32 / RISC-V QEMU libs use (VELXIO_BINARY_BASE_URL + VELXIO_LICENSE_KEY). - `LocalDirectoryDownloader` — for tests + in-prod use where the licence-module storage is already on the same filesystem (saves the loopback HTTP roundtrip on a 1.4 GiB blob). * `build_downloader_from_env()` picks one — local-dir wins if both sets of env vars are present, so the prod box short-circuits to direct disk reads automatically. * Lifespan hook in qemu_manager.py pre-warms the cache on container boot so first-time user requests don't pay the 30-60 s download + decompress latency. Adding a future board kind (Pi 4 / Pi 5) is now: upload assets via upload-binary.sh, append an entry to manifest.json, register a lifespan pre-warm in the new board's service module. Zero edits to provider.py / downloader.py. Manifest entries for raspberry-pi-3: kernel8.img 9 695 883 bytes (uncompressed) bcm2710-rpi-3-b.dtb 34 687 bytes (uncompressed) raspios-trixie-armhf.img 5 729 419 264 bytes raw / 1 488 002 803 bytes .zst on wire (zstd -19) source: 2026-04-21 build from raspberrypi.com Tests: 21 new unit tests covering manifest parsing, integrity helpers, both downloaders, and the provider's idempotent / concurrent / integrity / decompression / warmup paths. In-process FakeDownloader keeps the suite under 1 s and httpx-free. Docs: new docs/BOOT_IMAGES.md describes the architecture, on-disk layout, named-volume operation, and the procedure for adding a new image set.
2026-05-16 10:41:01 +07:00
},
{
"name": "bcm2710-rpi-3-b.dtb",
"asset_id": "dtb-bcm2710-rpi-3-b",
"sha256": "496fa7723583a0374b5b2e2be997301368d70b411a47bc74a394291c3636ad13",
"size_bytes": 34687,
"version": "2026-04-21"
},
{
"name": "raspios-trixie-armhf.img",
"asset_id": "raspios-trixie-armhf-zst",
fix(pi3): velxio-init reads stdin from /dev/ttyAMA1 (was /dev/console) User-reported bug: Pi 3 simulator showed boot output but keyboard input was ignored — the shell was effectively read-only. Root cause: velxio-init's bash redirect was `</dev/console >/dev/console`. From userspace, /dev/console is write-only — it is the kernel's printk target and accepts writes (so we saw boot output fine) but reads return EOF / block forever. Bash never saw a keystroke and the user couldn't type. Fix: read AND write through /dev/ttyAMA1. The 12 s devtmpfs-wait already in velxio-init guarantees the device node exists by the time the shell-respawn loop runs. setsid -c still gives bash a controlling terminal so PS1, job control, and Ctrl-C all work. Manifest version bumped to 2026-04-21+ttyAMA1; sidecar SHA check invalidates the cached SD on every velxio backend so the fix lands without an operator dance.
2026-05-16 22:01:46 +07:00
"sha256": "1c5d05449be589fcdbc3283364462d40dd5efbc685e2bd5169fd857a311ff5b0",
feat(sim): boot_images module + Pi 3 emulation restored Pi 3 simulation had been broken since at least April 2026 (51 fail-events / 24h per docs/PI3_EMULATION_BROKEN.md). Two distinct defects compounded: 1. qemu_manager.py hard-coded paths for kernel8.img, a device-tree blob (under a DOS 8.3 short name!), and a 5.4 GiB Raspberry Pi OS SD image — none of which shipped in the repo or were pulled at image build. 2. qemu-system-arm + qemu-utils were missing from the Docker image entirely, so even with the boot files in place QEMU couldn't launch. Add both to Dockerfile.standalone (~200 MB). The architecture fix is a new `app.services.boot_images` module: * Manifest-driven (boot_images/manifest.json, versioned in repo, declares SHA256 + size for each file, supports an optional `compressed.{encoding,sha256,size_bytes}` block for assets shipped as .zst). * `BootImageProvider` materialises files lazily, atomically (temp + rename), verifies SHA256 pre- AND post-decompression, caches under /var/cache/velxio/boot-images, serialises concurrent get() calls per image set via asyncio.Lock. * `AssetDownloader` Protocol with two impls: - `LicenseGatedDownloader` — same flow ESP32 / RISC-V QEMU libs use (VELXIO_BINARY_BASE_URL + VELXIO_LICENSE_KEY). - `LocalDirectoryDownloader` — for tests + in-prod use where the licence-module storage is already on the same filesystem (saves the loopback HTTP roundtrip on a 1.4 GiB blob). * `build_downloader_from_env()` picks one — local-dir wins if both sets of env vars are present, so the prod box short-circuits to direct disk reads automatically. * Lifespan hook in qemu_manager.py pre-warms the cache on container boot so first-time user requests don't pay the 30-60 s download + decompress latency. Adding a future board kind (Pi 4 / Pi 5) is now: upload assets via upload-binary.sh, append an entry to manifest.json, register a lifespan pre-warm in the new board's service module. Zero edits to provider.py / downloader.py. Manifest entries for raspberry-pi-3: kernel8.img 9 695 883 bytes (uncompressed) bcm2710-rpi-3-b.dtb 34 687 bytes (uncompressed) raspios-trixie-armhf.img 5 729 419 264 bytes raw / 1 488 002 803 bytes .zst on wire (zstd -19) source: 2026-04-21 build from raspberrypi.com Tests: 21 new unit tests covering manifest parsing, integrity helpers, both downloaders, and the provider's idempotent / concurrent / integrity / decompression / warmup paths. In-process FakeDownloader keeps the suite under 1 s and httpx-free. Docs: new docs/BOOT_IMAGES.md describes the architecture, on-disk layout, named-volume operation, and the procedure for adding a new image set.
2026-05-16 10:41:01 +07:00
"size_bytes": 5729419264,
fix(pi3): velxio-init reads stdin from /dev/ttyAMA1 (was /dev/console) User-reported bug: Pi 3 simulator showed boot output but keyboard input was ignored — the shell was effectively read-only. Root cause: velxio-init's bash redirect was `</dev/console >/dev/console`. From userspace, /dev/console is write-only — it is the kernel's printk target and accepts writes (so we saw boot output fine) but reads return EOF / block forever. Bash never saw a keystroke and the user couldn't type. Fix: read AND write through /dev/ttyAMA1. The 12 s devtmpfs-wait already in velxio-init guarantees the device node exists by the time the shell-respawn loop runs. setsid -c still gives bash a controlling terminal so PS1, job control, and Ctrl-C all work. Manifest version bumped to 2026-04-21+ttyAMA1; sidecar SHA check invalidates the cached SD on every velxio backend so the fix lands without an operator dance.
2026-05-16 22:01:46 +07:00
"version": "2026-04-21+ttyAMA1",
feat(sim): boot_images module + Pi 3 emulation restored Pi 3 simulation had been broken since at least April 2026 (51 fail-events / 24h per docs/PI3_EMULATION_BROKEN.md). Two distinct defects compounded: 1. qemu_manager.py hard-coded paths for kernel8.img, a device-tree blob (under a DOS 8.3 short name!), and a 5.4 GiB Raspberry Pi OS SD image — none of which shipped in the repo or were pulled at image build. 2. qemu-system-arm + qemu-utils were missing from the Docker image entirely, so even with the boot files in place QEMU couldn't launch. Add both to Dockerfile.standalone (~200 MB). The architecture fix is a new `app.services.boot_images` module: * Manifest-driven (boot_images/manifest.json, versioned in repo, declares SHA256 + size for each file, supports an optional `compressed.{encoding,sha256,size_bytes}` block for assets shipped as .zst). * `BootImageProvider` materialises files lazily, atomically (temp + rename), verifies SHA256 pre- AND post-decompression, caches under /var/cache/velxio/boot-images, serialises concurrent get() calls per image set via asyncio.Lock. * `AssetDownloader` Protocol with two impls: - `LicenseGatedDownloader` — same flow ESP32 / RISC-V QEMU libs use (VELXIO_BINARY_BASE_URL + VELXIO_LICENSE_KEY). - `LocalDirectoryDownloader` — for tests + in-prod use where the licence-module storage is already on the same filesystem (saves the loopback HTTP roundtrip on a 1.4 GiB blob). * `build_downloader_from_env()` picks one — local-dir wins if both sets of env vars are present, so the prod box short-circuits to direct disk reads automatically. * Lifespan hook in qemu_manager.py pre-warms the cache on container boot so first-time user requests don't pay the 30-60 s download + decompress latency. Adding a future board kind (Pi 4 / Pi 5) is now: upload assets via upload-binary.sh, append an entry to manifest.json, register a lifespan pre-warm in the new board's service module. Zero edits to provider.py / downloader.py. Manifest entries for raspberry-pi-3: kernel8.img 9 695 883 bytes (uncompressed) bcm2710-rpi-3-b.dtb 34 687 bytes (uncompressed) raspios-trixie-armhf.img 5 729 419 264 bytes raw / 1 488 002 803 bytes .zst on wire (zstd -19) source: 2026-04-21 build from raspberrypi.com Tests: 21 new unit tests covering manifest parsing, integrity helpers, both downloaders, and the provider's idempotent / concurrent / integrity / decompression / warmup paths. In-process FakeDownloader keeps the suite under 1 s and httpx-free. Docs: new docs/BOOT_IMAGES.md describes the architecture, on-disk layout, named-volume operation, and the procedure for adding a new image set.
2026-05-16 10:41:01 +07:00
"compressed": {
"encoding": "zstd",
fix(pi3): velxio-init reads stdin from /dev/ttyAMA1 (was /dev/console) User-reported bug: Pi 3 simulator showed boot output but keyboard input was ignored — the shell was effectively read-only. Root cause: velxio-init's bash redirect was `</dev/console >/dev/console`. From userspace, /dev/console is write-only — it is the kernel's printk target and accepts writes (so we saw boot output fine) but reads return EOF / block forever. Bash never saw a keystroke and the user couldn't type. Fix: read AND write through /dev/ttyAMA1. The 12 s devtmpfs-wait already in velxio-init guarantees the device node exists by the time the shell-respawn loop runs. setsid -c still gives bash a controlling terminal so PS1, job control, and Ctrl-C all work. Manifest version bumped to 2026-04-21+ttyAMA1; sidecar SHA check invalidates the cached SD on every velxio backend so the fix lands without an operator dance.
2026-05-16 22:01:46 +07:00
"sha256": "2f5e62277cba51bc66f4a2d76666eae34e8ef5a1fbba4169b9f2824830afe115",
"size_bytes": 1488004697
feat(sim): boot_images module + Pi 3 emulation restored Pi 3 simulation had been broken since at least April 2026 (51 fail-events / 24h per docs/PI3_EMULATION_BROKEN.md). Two distinct defects compounded: 1. qemu_manager.py hard-coded paths for kernel8.img, a device-tree blob (under a DOS 8.3 short name!), and a 5.4 GiB Raspberry Pi OS SD image — none of which shipped in the repo or were pulled at image build. 2. qemu-system-arm + qemu-utils were missing from the Docker image entirely, so even with the boot files in place QEMU couldn't launch. Add both to Dockerfile.standalone (~200 MB). The architecture fix is a new `app.services.boot_images` module: * Manifest-driven (boot_images/manifest.json, versioned in repo, declares SHA256 + size for each file, supports an optional `compressed.{encoding,sha256,size_bytes}` block for assets shipped as .zst). * `BootImageProvider` materialises files lazily, atomically (temp + rename), verifies SHA256 pre- AND post-decompression, caches under /var/cache/velxio/boot-images, serialises concurrent get() calls per image set via asyncio.Lock. * `AssetDownloader` Protocol with two impls: - `LicenseGatedDownloader` — same flow ESP32 / RISC-V QEMU libs use (VELXIO_BINARY_BASE_URL + VELXIO_LICENSE_KEY). - `LocalDirectoryDownloader` — for tests + in-prod use where the licence-module storage is already on the same filesystem (saves the loopback HTTP roundtrip on a 1.4 GiB blob). * `build_downloader_from_env()` picks one — local-dir wins if both sets of env vars are present, so the prod box short-circuits to direct disk reads automatically. * Lifespan hook in qemu_manager.py pre-warms the cache on container boot so first-time user requests don't pay the 30-60 s download + decompress latency. Adding a future board kind (Pi 4 / Pi 5) is now: upload assets via upload-binary.sh, append an entry to manifest.json, register a lifespan pre-warm in the new board's service module. Zero edits to provider.py / downloader.py. Manifest entries for raspberry-pi-3: kernel8.img 9 695 883 bytes (uncompressed) bcm2710-rpi-3-b.dtb 34 687 bytes (uncompressed) raspios-trixie-armhf.img 5 729 419 264 bytes raw / 1 488 002 803 bytes .zst on wire (zstd -19) source: 2026-04-21 build from raspberrypi.com Tests: 21 new unit tests covering manifest parsing, integrity helpers, both downloaders, and the provider's idempotent / concurrent / integrity / decompression / warmup paths. In-process FakeDownloader keeps the suite under 1 s and httpx-free. Docs: new docs/BOOT_IMAGES.md describes the architecture, on-disk layout, named-volume operation, and the procedure for adding a new image set.
2026-05-16 10:41:01 +07:00
}
}
]
}
}
}