velxio/test/backend/unit/test_compile_dedup.py

142 lines
4.9 KiB
Python
Raw Normal View History

perf(compile): dedup, concurrency limits, and persistent build dir for ESP-IDF Three coordinated fixes that together close the "ESP-IDF compile takes 5-7 min every time" gap and prevent the failure mode where a user clicking compile multiple times spawns six ninja processes that peel each other apart on a modest VPS. What was wrong - /compile/start generated a fresh uuid4 every call, so 6 clicks = 6 independent builds racing each other. Saw load average 30 on the prod VPS during a real BMP280 attempt today. - No concurrency limit anywhere; asyncio.create_task() fired without gating. - ccache was wired in last week (PR #149) but reported 18,350 cacheable calls and **0 hits** because the build dir was a fresh tempfile.TemporaryDirectory(prefix='espidf_') per compile. The random /tmp/espidf_<random>/ path baked into -I and -fmacro-prefix-map flags → different command line every compile → ccache hash miss every time. What this PR does 1. Job deduplication (`backend/app/api/routes/compile.py`) - New `_job_key(files, board_fqbn)` returns SHA-256 of normalised file names + contents + board. Order-independent. - New `JOB_BY_KEY: dict[str, str]` indexes hash → job_id. - `compile_start` checks JOB_BY_KEY before spawning a new task; if a job for this exact content is already pending or running, returns the existing job_id (logs `[compile] dedup hit — reusing job <id>`). - `_purge_expired_jobs` evicts both COMPILE_JOBS and JOB_BY_KEY, keeping the index consistent. Edge case where two jobs share a key (old finished, new running) is handled — only evict the key entry if it still points at the purged job. 2. Concurrency control (`backend/app/api/routes/compile.py`) - `_COMPILE_SEMAPHORE = asyncio.Semaphore(2)` global cap on simultaneous compiles. - `_target_lock(board_fqbn)` returns a per-target asyncio.Lock so concurrent compiles to the SAME board (sharing the persistent build dir) serialise. Different boards still run in parallel up to the semaphore cap. - `_compile_job` acquires sema → per-target lock → flips state to `running` → calls `_run_compile`. Pending state now accurately reflects "queued waiting for resources". 3. Persistent build dir (`backend/app/services/espidf_compiler.py`) - New `_prepare_persistent_project_dir(idf_target)` materialises `/var/lib/velxio-build/<target>/project/` from the template on first use; on subsequent compiles it wipes only `main/` and `user_libs/` (the per-compile parts) and leaves `build/` alone so ninja's incremental cache + ccache .o files survive. - Toolchain version sentinel (`.idf_version`) wipes the whole target dir if the ESP-IDF or arduino-esp32 version changes — cached objects from the old toolchain are no longer ABI-compatible. - `compile()` is now a thin dispatcher: persistent path or fallback to the legacy `tempfile.TemporaryDirectory()` flow. The actual build logic was extracted into `_compile_in_dir()` so both paths share one implementation, no duplication. - Escape hatch: `VELXIO_PERSISTENT_BUILD_DIR=0` env var falls back to the tempfile path without rebuilding the image. Critical for production safety. 4. ccache normalisation (`Dockerfile.standalone`) - + `ENV CCACHE_BASEDIR=/var/lib/velxio-build` makes ccache canonicalise absolute paths under that prefix when computing the cache key. Robustens hits against any future subdir rearrangement. 5. Docker compose (`docker-compose.yml`) - + named volume `velxio-build:/var/lib/velxio-build` so the persistent build dir survives `docker compose up -d --build`. - + env `VELXIO_PERSISTENT_BUILD_DIR=1` (default ON; users disable without rebuilding). Expected impact - Cold first compile per container per target: unchanged (~5-7 min). - Same sketch re-compiled: ~2-5 s (everything cached). - Different sketch, same target: ~5-30 s (only user code + new lib steps rebuild; ESP-IDF base hits cache). - Different sketch with new libraries: ~30-90 s (new lib component compiles; rest hits cache). - Concurrent clicks on same example: 1 build, others poll the same job_id. No more six-ninja meltdown. Tests - `test/backend/unit/test_compile_dedup.py` covers `_job_key` stability + variance and `_purge_expired_jobs` consistency (including the "two jobs share a key" edge case). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 13:33:11 +07:00
"""
Tests for the compile-job deduplication logic in routes/compile.py.
Covers:
- _job_key() stability across invocations + variance with content/board
- _purge_expired_jobs() cleans both COMPILE_JOBS and JOB_BY_KEY consistently
Does NOT exercise the full FastAPI route or the ESP-IDF toolchain those are
covered by integration tests. This file is fast (no I/O, no toolchain).
Run from the repo root:
python -m pytest test/backend/unit/test_compile_dedup.py -v
"""
import sys
import time
import unittest
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent.parent.parent / 'backend'))
from app.api.routes import compile as compile_module
class JobKeyTests(unittest.TestCase):
def setUp(self):
# Reset module-level state before each test so they don't leak.
compile_module.COMPILE_JOBS.clear()
compile_module.JOB_BY_KEY.clear()
def test_key_stable_across_invocations(self):
files = [{'name': 'sketch.ino', 'content': 'void setup(){}'}]
k1 = compile_module._job_key(files, 'esp32:esp32:esp32')
k2 = compile_module._job_key(files, 'esp32:esp32:esp32')
self.assertEqual(k1, k2)
def test_key_changes_with_content(self):
k1 = compile_module._job_key(
[{'name': 'sketch.ino', 'content': 'void setup(){}'}],
'esp32:esp32:esp32',
)
k2 = compile_module._job_key(
[{'name': 'sketch.ino', 'content': 'void loop(){}'}],
'esp32:esp32:esp32',
)
self.assertNotEqual(k1, k2)
def test_key_changes_with_filename(self):
k1 = compile_module._job_key(
[{'name': 'sketch.ino', 'content': 'X'}],
'esp32:esp32:esp32',
)
k2 = compile_module._job_key(
[{'name': 'other.ino', 'content': 'X'}],
'esp32:esp32:esp32',
)
self.assertNotEqual(k1, k2)
def test_key_changes_with_board(self):
files = [{'name': 'sketch.ino', 'content': 'X'}]
k1 = compile_module._job_key(files, 'esp32:esp32:esp32')
k2 = compile_module._job_key(files, 'esp32:esp32:esp32c3')
self.assertNotEqual(k1, k2)
def test_key_independent_of_file_order(self):
files_a = [
{'name': 'a.ino', 'content': 'X'},
{'name': 'b.h', 'content': 'Y'},
]
files_b = [
{'name': 'b.h', 'content': 'Y'},
{'name': 'a.ino', 'content': 'X'},
]
k1 = compile_module._job_key(files_a, 'esp32:esp32:esp32')
k2 = compile_module._job_key(files_b, 'esp32:esp32:esp32')
self.assertEqual(k1, k2)
class PurgeExpiredJobsTests(unittest.TestCase):
def setUp(self):
compile_module.COMPILE_JOBS.clear()
compile_module.JOB_BY_KEY.clear()
def test_purge_drops_done_jobs_past_ttl(self):
old_finished = time.time() - compile_module.JOB_TTL_S - 10
compile_module.COMPILE_JOBS['old-id'] = {
'state': 'done',
'started_at': old_finished - 60,
'finished_at': old_finished,
'key': 'k1',
}
compile_module.JOB_BY_KEY['k1'] = 'old-id'
compile_module._purge_expired_jobs()
self.assertNotIn('old-id', compile_module.COMPILE_JOBS)
self.assertNotIn('k1', compile_module.JOB_BY_KEY)
def test_purge_keeps_running_jobs(self):
# No finished_at; state=running. Should never be purged.
compile_module.COMPILE_JOBS['running-id'] = {
'state': 'running',
'started_at': time.time() - 10000,
'key': 'k2',
}
compile_module.JOB_BY_KEY['k2'] = 'running-id'
compile_module._purge_expired_jobs()
self.assertIn('running-id', compile_module.COMPILE_JOBS)
self.assertIn('k2', compile_module.JOB_BY_KEY)
def test_purge_does_not_evict_key_pointing_at_newer_job(self):
# Edge case: an old finished job and a newer running job share the
# same key. JOB_BY_KEY[key] points at the newer one. Purging the old
# job must NOT clear the key (it would orphan the running job from
# future dedup hits).
old_finished = time.time() - compile_module.JOB_TTL_S - 10
compile_module.COMPILE_JOBS['old-id'] = {
'state': 'done',
'started_at': old_finished - 60,
'finished_at': old_finished,
'key': 'shared-key',
}
compile_module.COMPILE_JOBS['new-id'] = {
'state': 'running',
'started_at': time.time() - 5,
'key': 'shared-key',
}
compile_module.JOB_BY_KEY['shared-key'] = 'new-id'
compile_module._purge_expired_jobs()
self.assertNotIn('old-id', compile_module.COMPILE_JOBS)
self.assertIn('new-id', compile_module.COMPILE_JOBS)
# Crucially, the key still points at the running job.
self.assertEqual(compile_module.JOB_BY_KEY.get('shared-key'), 'new-id')
if __name__ == '__main__':
unittest.main()