velxio/backend/app/services/boot_images/manifest.py

140 lines
4.6 KiB
Python
Raw Normal View History

feat(sim): boot_images module + Pi 3 emulation restored Pi 3 simulation had been broken since at least April 2026 (51 fail-events / 24h per docs/PI3_EMULATION_BROKEN.md). Two distinct defects compounded: 1. qemu_manager.py hard-coded paths for kernel8.img, a device-tree blob (under a DOS 8.3 short name!), and a 5.4 GiB Raspberry Pi OS SD image — none of which shipped in the repo or were pulled at image build. 2. qemu-system-arm + qemu-utils were missing from the Docker image entirely, so even with the boot files in place QEMU couldn't launch. Add both to Dockerfile.standalone (~200 MB). The architecture fix is a new `app.services.boot_images` module: * Manifest-driven (boot_images/manifest.json, versioned in repo, declares SHA256 + size for each file, supports an optional `compressed.{encoding,sha256,size_bytes}` block for assets shipped as .zst). * `BootImageProvider` materialises files lazily, atomically (temp + rename), verifies SHA256 pre- AND post-decompression, caches under /var/cache/velxio/boot-images, serialises concurrent get() calls per image set via asyncio.Lock. * `AssetDownloader` Protocol with two impls: - `LicenseGatedDownloader` — same flow ESP32 / RISC-V QEMU libs use (VELXIO_BINARY_BASE_URL + VELXIO_LICENSE_KEY). - `LocalDirectoryDownloader` — for tests + in-prod use where the licence-module storage is already on the same filesystem (saves the loopback HTTP roundtrip on a 1.4 GiB blob). * `build_downloader_from_env()` picks one — local-dir wins if both sets of env vars are present, so the prod box short-circuits to direct disk reads automatically. * Lifespan hook in qemu_manager.py pre-warms the cache on container boot so first-time user requests don't pay the 30-60 s download + decompress latency. Adding a future board kind (Pi 4 / Pi 5) is now: upload assets via upload-binary.sh, append an entry to manifest.json, register a lifespan pre-warm in the new board's service module. Zero edits to provider.py / downloader.py. Manifest entries for raspberry-pi-3: kernel8.img 9 695 883 bytes (uncompressed) bcm2710-rpi-3-b.dtb 34 687 bytes (uncompressed) raspios-trixie-armhf.img 5 729 419 264 bytes raw / 1 488 002 803 bytes .zst on wire (zstd -19) source: 2026-04-21 build from raspberrypi.com Tests: 21 new unit tests covering manifest parsing, integrity helpers, both downloaders, and the provider's idempotent / concurrent / integrity / decompression / warmup paths. In-process FakeDownloader keeps the suite under 1 s and httpx-free. Docs: new docs/BOOT_IMAGES.md describes the architecture, on-disk layout, named-volume operation, and the procedure for adding a new image set.
2026-05-16 10:41:01 +07:00
"""Boot-image manifest schema + JSON loader.
The manifest is the single source of truth for "what files are expected
to exist on disk before QEMU is launched for board X". It lives in
``manifest.json`` next to this module and is committed to the repo so
a refactor that bumps a kernel image is visible in code review.
Each ``ImageSetSpec`` corresponds to one board kind (e.g.
``raspberry-pi-3``). Adding a new board is appending a JSON entry; no
Python edit required.
"""
from __future__ import annotations
import json
from collections.abc import Mapping
from dataclasses import dataclass
from pathlib import Path
from .errors import ImageSetNotFoundError
@dataclass(frozen=True, slots=True)
class CompressedSource:
"""Wire-format description for assets that ship compressed.
``encoding`` is the algorithm identifier ('zstd' is the only one
implemented today). ``sha256`` and ``size_bytes`` describe the file
AS DOWNLOADED; the post-decompression hash lives on the owning
:class:`BootImageSpec` so verification can run before AND after
decompression.
"""
encoding: str
sha256: str
size_bytes: int
@dataclass(frozen=True, slots=True)
class BootImageSpec:
"""One file the provider must materialise on disk.
``name`` is the final filename in the cache directory and what
callers ask for. ``asset_id`` is the downloader-side identifier
(e.g. the path component of the licence-gated URL or the filename
inside a local directory). Decoupling the two lets the on-disk
layout stay stable while the upstream asset name evolves.
"""
name: str
asset_id: str
sha256: str
size_bytes: int
version: str | None = None
compressed: CompressedSource | None = None
@dataclass(frozen=True, slots=True)
class ImageSetSpec:
"""A coherent group of boot files for a single board kind."""
id: str
description: str
images: tuple[BootImageSpec, ...]
def image(self, name: str) -> BootImageSpec:
for img in self.images:
if img.name == name:
return img
raise KeyError(f"image {name!r} not declared in set {self.id!r}")
@dataclass(frozen=True, slots=True)
class BootImagesManifest:
"""Top-level manifest object — loaded once at process start."""
version: int
image_sets: Mapping[str, ImageSetSpec]
def get(self, set_id: str) -> ImageSetSpec:
try:
return self.image_sets[set_id]
except KeyError as exc:
known = ", ".join(sorted(self.image_sets)) or "(empty)"
raise ImageSetNotFoundError(
f"image_set {set_id!r} not in manifest. Known sets: {known}"
) from exc
def load_manifest(path: Path) -> BootImagesManifest:
"""Parse the manifest JSON at ``path``.
Raises ``ValueError`` if any required key is missing or any sha256
is malformed fail-fast at process start beats a confusing error
at first download.
"""
data = json.loads(path.read_text(encoding="utf-8"))
version = int(data["version"])
sets_raw = data.get("image_sets", {})
if not isinstance(sets_raw, Mapping):
raise ValueError("manifest.image_sets must be an object")
sets: dict[str, ImageSetSpec] = {}
for set_id, set_raw in sets_raw.items():
images: list[BootImageSpec] = []
for img_raw in set_raw.get("images", []):
compressed = None
if img_raw.get("compressed"):
c = img_raw["compressed"]
compressed = CompressedSource(
encoding=str(c["encoding"]),
sha256=_check_sha256(c["sha256"]),
size_bytes=int(c["size_bytes"]),
)
images.append(
BootImageSpec(
name=str(img_raw["name"]),
asset_id=str(img_raw["asset_id"]),
sha256=_check_sha256(img_raw["sha256"]),
size_bytes=int(img_raw["size_bytes"]),
version=(
str(img_raw["version"]) if img_raw.get("version") else None
),
compressed=compressed,
)
)
sets[set_id] = ImageSetSpec(
id=str(set_id),
description=str(set_raw.get("description", "")),
images=tuple(images),
)
return BootImagesManifest(version=version, image_sets=sets)
def _check_sha256(value: str) -> str:
v = value.strip().lower()
if len(v) != 64 or not all(c in "0123456789abcdef" for c in v):
raise ValueError(f"invalid sha256: {value!r}")
return v