velxio/backend/app/services/boot_images/__init__.py

103 lines
2.7 KiB
Python
Raw Normal View History

feat(sim): boot_images module + Pi 3 emulation restored Pi 3 simulation had been broken since at least April 2026 (51 fail-events / 24h per docs/PI3_EMULATION_BROKEN.md). Two distinct defects compounded: 1. qemu_manager.py hard-coded paths for kernel8.img, a device-tree blob (under a DOS 8.3 short name!), and a 5.4 GiB Raspberry Pi OS SD image — none of which shipped in the repo or were pulled at image build. 2. qemu-system-arm + qemu-utils were missing from the Docker image entirely, so even with the boot files in place QEMU couldn't launch. Add both to Dockerfile.standalone (~200 MB). The architecture fix is a new `app.services.boot_images` module: * Manifest-driven (boot_images/manifest.json, versioned in repo, declares SHA256 + size for each file, supports an optional `compressed.{encoding,sha256,size_bytes}` block for assets shipped as .zst). * `BootImageProvider` materialises files lazily, atomically (temp + rename), verifies SHA256 pre- AND post-decompression, caches under /var/cache/velxio/boot-images, serialises concurrent get() calls per image set via asyncio.Lock. * `AssetDownloader` Protocol with two impls: - `LicenseGatedDownloader` — same flow ESP32 / RISC-V QEMU libs use (VELXIO_BINARY_BASE_URL + VELXIO_LICENSE_KEY). - `LocalDirectoryDownloader` — for tests + in-prod use where the licence-module storage is already on the same filesystem (saves the loopback HTTP roundtrip on a 1.4 GiB blob). * `build_downloader_from_env()` picks one — local-dir wins if both sets of env vars are present, so the prod box short-circuits to direct disk reads automatically. * Lifespan hook in qemu_manager.py pre-warms the cache on container boot so first-time user requests don't pay the 30-60 s download + decompress latency. Adding a future board kind (Pi 4 / Pi 5) is now: upload assets via upload-binary.sh, append an entry to manifest.json, register a lifespan pre-warm in the new board's service module. Zero edits to provider.py / downloader.py. Manifest entries for raspberry-pi-3: kernel8.img 9 695 883 bytes (uncompressed) bcm2710-rpi-3-b.dtb 34 687 bytes (uncompressed) raspios-trixie-armhf.img 5 729 419 264 bytes raw / 1 488 002 803 bytes .zst on wire (zstd -19) source: 2026-04-21 build from raspberrypi.com Tests: 21 new unit tests covering manifest parsing, integrity helpers, both downloaders, and the provider's idempotent / concurrent / integrity / decompression / warmup paths. In-process FakeDownloader keeps the suite under 1 s and httpx-free. Docs: new docs/BOOT_IMAGES.md describes the architecture, on-disk layout, named-volume operation, and the procedure for adding a new image set.
2026-05-16 10:41:01 +07:00
"""Boot-image provider for QEMU-emulated boards (Raspberry Pi 3, etc.).
See ``manifest.json`` next to this module for the declared image sets,
and ``docs/BOOT_IMAGES.md`` for the architecture overview.
Public entry point:
from app.services.boot_images import get_default_provider
provider = get_default_provider()
images = await provider.get("raspberry-pi-3")
# images["kernel8.img"], images["bcm2710-rpi-3-b.dtb"], ...
"""
from __future__ import annotations
import os
from pathlib import Path
from typing import Optional
from .downloader import (
AssetDownloader,
LicenseGatedDownloader,
LocalDirectoryDownloader,
build_downloader_from_env,
)
from .errors import (
BootImageError,
DecompressionError,
DownloadError,
ImageSetNotFoundError,
IntegrityError,
NoDownloaderConfiguredError,
)
from .manifest import (
BootImageSpec,
BootImagesManifest,
CompressedSource,
ImageSetSpec,
load_manifest,
)
from .provider import BootImageProvider
__all__ = [
"AssetDownloader",
"BootImageError",
"BootImageProvider",
"BootImageSpec",
"BootImagesManifest",
"CompressedSource",
"DecompressionError",
"DownloadError",
"ImageSetNotFoundError",
"ImageSetSpec",
"IntegrityError",
"LicenseGatedDownloader",
"LocalDirectoryDownloader",
"NoDownloaderConfiguredError",
"build_downloader_from_env",
"get_default_provider",
"load_manifest",
"reset_default_provider",
]
_DEFAULT_MANIFEST_PATH = Path(__file__).parent / "manifest.json"
_DEFAULT_CACHE_DIR = Path("/var/cache/velxio/boot-images")
_provider_cache: Optional[BootImageProvider] = None
def get_default_provider() -> BootImageProvider:
"""Lazy process-wide singleton.
Built once from the bundled ``manifest.json`` + ``build_downloader_from_env()``
+ ``VELXIO_BOOT_IMAGE_CACHE_DIR`` (or the default mount path).
Re-imported tests should call :func:`reset_default_provider`
between fixtures.
"""
global _provider_cache
if _provider_cache is None:
manifest = load_manifest(_DEFAULT_MANIFEST_PATH)
cache_dir = Path(
os.environ.get(
"VELXIO_BOOT_IMAGE_CACHE_DIR",
str(_DEFAULT_CACHE_DIR),
)
)
_provider_cache = BootImageProvider(
manifest=manifest,
downloader=build_downloader_from_env(),
cache_dir=cache_dir,
)
return _provider_cache
def reset_default_provider() -> None:
"""Test helper — clears the lazy singleton so the next
``get_default_provider()`` call rebuilds from current env."""
global _provider_cache
_provider_cache = None