Backend Architecture¶
Overview¶
The Python backend follows Cosmic Python ("Architecture Patterns with Python") adapted for a single-user Decky plugin. Code is split into four layers with a strictly enforced dependency direction:
services/— orchestration. Business logic and the public callable surface.adapters/— I/O. Everything that touches the network, the filesystem, the clock, or Steam.domain/— pure compute. Functions in, values out; no I/O, no state mutation, no service/adapter imports.lib/— cross-cutting utilities independent of every other layer.models/— data shapes (TypedDicts, dataclasses) independent of every other layer.
Services depend on Protocols (defined in services/protocols/), never on concrete adapter classes. Adapters
implement those Protocols. bootstrap/ is the composition root — the only place where concrete adapters meet services.
main.py owns the Decky lifecycle and the callable surface; it holds no business logic.
class Plugin:
# No base classes — pure composition
# Owns: the Decky lifecycle (_main / _unload) and the callable surface
# Delegates: all business logic to services, all I/O to adapters
Dependency Diagram¶
main.py (Plugin — Decky lifecycle + callable routing)
↓ calls
bootstrap/ (composition root: adapters.bootstrap() builds adapters, services.wire_services() builds services)
↓ creates
┌─────────────────────────────────────────────────────────┐
│ Adapters (own all I/O — implement Protocols) │
│ RommHttpAdapter / RommApiAdapter — RomM REST │
│ SteamConfigAdapter — Steam VDF, grid dir, Steam Input │
│ SteamGridDbAdapter / SgdbArtworkCacheAdapter — SGDB │
│ PersistenceAdapter (+ persister adapters) — JSON I/O │
│ SqliteUnitOfWork (+ repository adapters) — SQLite I/O │
│ CoverArtFileStore / DownloadFile │
│ FirmwareFile / MigrationFile / RomFile / SaveFile │
│ RetroDeckPaths / RetroArchConfig / RetroArchCoreInfo │
│ CoreResolver (ES-DE es_systems.xml) │
│ PlatformCoreReaderAdapter (settings platform_cores) │
│ SystemClock / SystemUuidGen / AsyncioSleeper │
│ HostnameAdapter / PathProbe / PluginMetadata │
└────────────────────────┬────────────────────────────────┘
│ injected via *ServiceConfig
┌────────────────────────▼────────────────────────────────┐
│ Services (depend on Protocols, not concrete adapters) │
│ LibraryService SaveService │
│ DownloadService PlaytimeService │
│ FirmwareService SteamGridService │
│ MetadataService AchievementsService │
│ MigrationService GameDetailService │
│ ArtworkService RomRemovalService │
│ ShortcutRemovalService SettingsService │
│ CoreService ConnectionService │
│ StartupHealingService LaunchGateService │
│ SessionLifecycleService │
│ RomAdoptionService RomInstallRecorder │
└────────────────────────┬────────────────────────────────┘
│ depend on
┌────────────────────────▼────────────────────────────────┐
│ Protocols (services/protocols/) — grouped topically: │
│ transport / determinism / persistence / paths / │
│ infra / files / cross_service │
└─────────────────────────────────────────────────────────┘
Domain (domain/) — pure compute, imported by services and adapters; imports nothing above it.
Arrow direction: depends-on (A -> B means A uses B).
The XxxServiceConfig constructor pattern¶
Every service takes a single config keyword argument — a frozen dataclass named <ServiceName>Config. All
dependencies live in the config: Protocol-typed adapters, infrastructure seams (event loop, logger, Clock, UuidGen,
Sleeper), persistence callbacks, and settings-derived values. There are no bare-param or mixed constructors.
sync_service = LibraryService(
config=LibraryServiceConfig(
romm_api=..., # Protocol-typed adapter
steam_config=..., # Protocol-typed adapter
clock=..., # Clock Protocol
uuid_gen=..., # UuidGen Protocol
sleeper=..., # Sleeper Protocol
uow_factory=..., # UnitOfWorkFactory Protocol (roms / sync_runs / kv_config / rom_metadata)
artwork=..., # cross-service Protocol-typed peer
# ...
),
)
Outer services keep the Service token in both names (SteamGridService + SteamGridServiceConfig). Sub-services may
use role-based names without the token when it reads more naturally (SyncEngine + SyncEngineConfig,
SyncOrchestrator + SyncOrchestratorConfig).
Module Responsibilities¶
Services (py_modules/services/)¶
Two services are large enough to be decomposed into sub-service packages (services/library/ and services/saves/);
the rest are single modules. A service over ~1000 LOC is the decomposition signal.
| Module | Domain |
|---|---|
library/ |
LibraryService façade — fetch ROMs, preview/apply sync, per-unit shortcut delivery, roms/SyncRun writes + queries (decomposed; see below) |
saves/ |
SaveService aggregate — .srm upload/download, conflict detection, slots, versions (decomposed; see below) |
downloads.py |
DownloadService — ZIP extraction, M3U, progress, bounded-concurrency download queue (Semaphore(2) + reserved-bytes pre-flight); cancel/cleanup never deletes a live install. The occupancy pre-flight sits beside the disk-space one and refuses rather than writing over content the plugin did not place (DownloadTargetGateFn → RomAdoptionService) |
rom_adoption.py |
RomAdoptionService — the download-target gate (describe what occupies the path, refuse with the comparison, or clear it on an explicit replace), adoption of on-disk content as an install, and the user-triggered content check against RomM's per-file checksums. A directory's files are held to the place RomM names: RomFile.is_top_level compares rom.full_path against file_path, so the two are one coordinate system and the ROM-relative path is a subtraction — an entry the payload does not locate falls back to a by-name search, and one whose relative path escapes the ROM directory is refused rather than looked up. Anything whose name is one RomM reads as an archive is held to its contents instead, because RomM's digest for such a file describes what is inside (its current scanner accumulates over every member, the pre-4.9.0 one took the largest) while file_size_bytes stays the container's: neither the container's digest nor its size is compared, and the local side comes from the ZIP central directory (name, uncompressed size, CRC32, all free). One member inside — the same bytes under every rule RomM has used — is compared against the file-level digest; several members are compared one by one when the payload carries archive_members, and are unverifiable when it does not, since the single number is then either a composite or the largest member with nothing saying which. A size or CRC32 disagreement is reported without decompressing; the MD5 is what earns a match. A container this adapter cannot open leaves the entry unverifiable rather than accused — except a single-member archive whose member size the loose file on disk matches exactly, which is the unpacked case. Each difference is one {name, detail} line, and unverifiable carries a message per reason (no checksums published / could not be read / one checksum for the whole archive). Never deletes on its own initiative; the replace leg is guarded by is_safe_rom_path and only removes what cannot be replaced atomically. Adoption runs the #1298 sibling supersede through the SiblingSupersedeFn seam (DownloadService owns the selection rule; ADR-0021 §5), ordered validate → supersede → record, so no other version is deleted until every refusal has been ruled out. See ADR-0028 |
rom_install_recorder.py |
RomInstallRecorder — the one writer of a rom_installs row and of the shortcut bake behind it (launchable verdict, the upsert, the fs_size_bytes write-back, the launch command resolved through the per-game core pin + disc pick, and the applied_launch_options memo). Both a completed download and an adoption reach an install through here, so an adopted row cannot drift from a downloaded one |
firmware.py |
FirmwareService — BIOS registry, downloads, per-core filtering; get_firmware_status ships per-platform bios_level (ok/partial/missing via domain.bios.compute_bios_level) + required_count/required_downloaded/server_count/local_count so the System page reads the decision off the payload (#461) |
session_lifecycle.py |
SessionLifecycleService — post-exit orchestration (playtime + post-exit save sync + achievement sync + migration refresh) |
migration.py |
MigrationService — RetroDECK path-change detection + file migration, save-sort change detection + conflict resolution. A home changed again before migrating accumulates a pending-home set (_previous + a _hops array) instead of overwriting the marker, and collection matches each tracked record by longest-prefix against every pending home, so files under an intermediate home are never stranded (#1042; pure transition/remap kernel in domain/migration_paths.py) |
steamgrid.py |
SteamGridService — SteamGridDB fetch, cache, icons |
artwork.py |
ArtworkService — cover art download into the per-ROM cover cache ({runtime}/covers/{rom_id}.png, the single source of truth for a ROM's cover), cover-cache invalidation against the persisted cover_source fingerprint (refresh_changed_covers, the per-unit pass that re-downloads changed covers and feeds the frontend's tile re-apply, #1386; a ts-only fingerprint change revalidates the cached bytes with a conditional request via the {rom_id}.cover-meta.json validator sidecar instead of re-downloading, #1454), publish of the active version's cache cover onto the Steam grid as {app_id}p.png (a copy so every sibling keeps its own file, ADR-0021 / #1346), the read-only get_artwork_base64 query, the cache-first fetch_cover_base64 used by the version picker (a cache miss downloads the ROM's cover from RomM — works for a server-only version with no local row), cache/staging cleanup + startup orphan pruning, the full grid-image sweep on shortcut removal (remove_artwork_files deletes every portrait/wide/hero/logo/icon × png/jpg/jpeg form for the removed appId, not just {app_id}p.png), and the user-triggered orphaned grid-image cleanup (cleanup_orphaned_grid_images, Danger Zone): deletes grid files whose parsed appId is in the non-Steam-shortcut range (domain/artwork_paths.is_shortcut_app_id, [0x80000000, 0xFFFFFFFF] — store-game custom art is never a candidate) and absent from the frontend's full live-shortcut scan (RomM-owned AND foreign), refusing outright (incomplete_scan) when any bound roms.shortcut_app_id is missing from the submitted live set — a provably partial scan deletes nothing; dry-run mode feeds the QAM's two-tap confirm count |
game_detail.py |
GameDetailService — game detail page data aggregation |
game_process.py |
GameProcessService — the Stop Game ladder: find the RetroDECK flatpak's host processes, ask each to exit once, wait out a bounded grace window, force the survivors. Steam's TerminateApp cannot reach a portal-started flatpak, so the kill is backend-side; the never-re-request rule is a save-safety invariant (see State-aware Resume button) |
playtime.py |
PlaytimeService — session recording into rom_playtime + the rom_playtime_sessions outbox, native play-session ingest (session-end fold+enqueue+flush to /api/play-sessions grouped per stored device_id with a bounded-retry quarantine + pull-only reconcile-on-view that restores total_seconds, session_count, and last_played via monotonic clamps, ADR-0018 / #903); owns the durable kv_config["playtime_scope_notice"] re-sign-in flag (a reconcile 403 sets it, a later 200 or a fresh sign-in clears it) surfaced via get_playtime_scope_notice |
achievements.py |
AchievementsService — progress, caching, RA username |
settings.py |
SettingsService — settings reads/writes, Steam Input config |
rom_removal.py |
RomRemovalService — ROM file deletion + rom_installs cleanup via the UoW; keeps the roms row, playtime, and saves per ADR-0007. Deletion runs under a descriptor-relative no-follow claim whose discipline follows the recovery bundle, not the caller: content-bound whenever a bundle exists to bind the hashes to, identity-only when none does — a user-initiated uninstall, or a cleanup run with recovery off. Only the identity-only form may adopt interrupted staging, and only it leases per unlink rather than over the whole tree — see Removed-game cleanup. One removal per rom_id at a time, across both entry points: a bulk uninstall reads its install list and claims every rom_id on the loop thread before dispatching its worker, so the single-ROM guard is what refuses a press during a bulk run, and an already-claimed ROM is what refuses the bulk run itself. Both refusals are in_progress; the bulk one carries no removal payload, which is the frontend's existing refusal discriminant. The claim set has a single writer — every mutation brackets the run_in_executor call instead of happening inside it — which is what keeps it lock-free, and services/ may not import threading anyway (.importlinter, no-stdlib-io-in-services). Start/finish logged with duration; per-file uninstall_progress frames for multi-file ROMs |
cores.py |
CoreService — per-platform emulator-info lookup (get_platform_core_info — the classified emulators list + emulator_data_available), per-game core pin/clear (roms.emulator_override), per-platform core write (settings.json platform_cores) + fan-out re-bake; a pin's LABEL is validated against the bakeable options before any write and may name a standalone emulator or a libretro core; see Core and Emulator Selection |
version_switch.py |
VersionSwitchService — the game-detail version picker (ADR-0021). get_version_list(app_id) resolves the bound ROM's sibling group by roms.sibling_group_key (migration-010 index) and merges the local rows with RomM's live get_rom(bound).sibling_roms view — versions not yet synced are marked synced: false; returns each version's markers (active = the bound row, is_default = the resolution chain's pick with EMPTY install/binding filters, installed, and switchable) plus an additive server_query_failed flag when the server view can't be fetched. switchable is the SAME membership authority switch_version decides by (domain.sibling_group.target_in_sibling_group): a local target is a sibling_group_key match (the component keys encode membership, ADR-0022); a server-only target is judged by canonical compatibility — parse the bound key to {source}:{value} and require the target's id at that source to be absent-or-equal (the persisted key doubles as the group's canonical summary). A RomM sibling in a conflicting metadata match — a locally-synced ROM under a different sibling_group_key (#1359) or a never-synced ROM carrying a different id at the bound canonical source (#1360) — is listed but switchable: false and dropped from the default ranking, so the picker disables the row instead of offering a switch the backend would reject with not_in_group. A romm: bound key (no metadata source) admits no server-only target; a target that merely lacks the canonical id is in-group and adopts the bound group's key on switch (#1368 / ADR-0022). switch_version(app_id, target_rom_id, allow_stranded) (@migration_blocked) moves the group's binding to the target — a pure roms write (the repository's collision-unbind clears the old representative); no Steam call, no name change, no save migration (ADR-0021 §2/§4). Applies to downloaded games too (#1298): on success it returns target_installed + the target install's full launch_options ("" for an uninstalled target, the ADR-0009 placeholder) which the frontend confirm-writes onto the sticky shortcut. Switching away from a downloaded version whose local saves drift is a soft block (unsynced_saves, carrying server_reachable / unsynced_rom_id / unsynced_version_name) so the picker can offer sync-first / switch-anyway / cancel; allow_stranded=true is the "Switch anyway" override, honoured even offline. Guards, each canonical {success, reason, message}: not_found / download_in_progress (any group member has an active download — cancel first) / not_in_group / bound_elsewhere / unsynced_saves (soft) / server_unreachable / invalid_target (a server-only target that is a sibling but whose RomM detail the Rom aggregate rejects). The drift + reachability probes run in their own short reads before any write UoW opens; the write UoW re-checks membership + bound-elsewhere (TOCTOU) and the relaunch resolver re-bakes the command outside it (both open their own UoW; ADR-0006). A server-only target is persisted from its RomM detail first via domain.shortcut_data.extract_version_metadata, adopting the bound group's sibling_group_key (not its own coalesce-first key) so the next sync re-canonicalizes the whole component together (ADR-0022) |
active_core_resolver.py |
ActiveCoreResolver — the single per-ROM read seam: active_core_for_rom(rom_id) folds the per-game DB override + the per-platform settings.json core over the live-es_systems.xml system default (three layers: per-game override → per-platform core → live es_systems default → None, the plain launch; no bundled snapshot). Each layer resolves a LABEL to a bakeable EmulatorInvocation (libretro or standalone) via get_emulator_options / label_to_invocation. Every per-game core read + every launch bake draws from it |
shortcut_removal.py |
ShortcutRemovalService — shortcut removal; unbinds the ROM in roms (keeps the row per ADR-0007) |
metadata.py |
MetadataService — ROM metadata reads from rom_metadata (7-day TTL): the per-ROM get_rom_metadata lookup, the paged get_metadata_cache_page(offset, limit) the frontend loads on plugin start (returns {items: {str(rom_id): entry}, total}, page and total read under one short read UoW), and the app_id → rom_id map the launcher resolves session ROMs through. Paging replaced a single whole-cache dump so a large library (5–10k ROMs) never pushes a multi-MB response through the size-limited callable bridge in one frame (#1025); the frontend pages at 500/page until total is reached |
launch_gate.py |
LaunchGateService — pre-launch gate (rom lookup, install check, save status) |
startup_healing.py |
StartupHealingService — prunes stale rom_installs rows against disk on load (via the UoW) + reconciles orphaned running SyncRuns (a hard crash leaves a running row → marked errored) + get_installed_relaunch_options() builds the startup launch-options reconcile items (see StartupHealingService notes) |
connection.py |
ConnectionService — connection test + RomM minimum-version gate + Client API Token lifecycle (mint/establish via credentials, validate/store a user-pasted token, or exchange a short-lived pairing code for a token — the latter two for OIDC accounts; host-bound to the minting origin; see ConnectionService notes) |
protocols/ |
Protocol interfaces grouped by concern (see Protocol Interfaces) |
Version-list liveness¶
VersionSwitchService.get_version_list separates retained local membership from current RomM availability. On each lazy
Game Page load, the bound detail request proves the bound id live and supplies the direct sibling ids; only non-bound
local members absent from that direct sibling view are exact-id probed through RommRomReader.get_rom_once, the
adapter's single-attempt short-timeout request_once path, concurrently on the existing executor fan-out. Only
RommNotFoundError produces vanished: true. All other probe failures and malformed/falsy responses fail open. No
result is persisted.
The response carries vanished on every version and bound_vanished even when multi_version is false. A singleton
also carries its bound_version, which lets a definitively vanished synced binding expose scoped cleanup without
inventing a multi-version picker. Retained vanished rows keep the independently-computed switchable membership verdict
but are excluded from default ranking. A bound-id 404 is a successful entity verdict (bound_vanished: true,
server_query_failed: false), not a global reachability signal. Artwork and save reads do not contribute availability
evidence; fetch_cover_base64 remains a nullable data callable.
Version-switch target liveness¶
VersionSwitchService.switch_version obtains a fresh exact-id verdict immediately before every actual binding move onto
an already-local target. The save-stranding guard runs first, so its initial soft block performs no target request; both
follow-up choices (Sync now retry and Switch anyway) re-enter the service and receive the same liveness guard.
allow_stranded bypasses only save stranding. The active-target no-op, invalid local context, active download,
non-member target, and target bound to another shortcut all stop before the request because none can enter the binding
write.
The local-target request uses RommRomReader.get_rom_once on the worker executor: one attempt, three-second timeout, no
retry-progress event, and no open UoW. Only RommNotFoundError returns the canonical version_vanished refusal.
Timeout, transport/DNS/SSL, authentication, server errors, and malformed or empty successful payloads are logged and
fail open, preserving the fast offline switch path. The existing short write UoW still rechecks membership and
bound-elsewhere after the request; the request narrows the race but cannot form a transaction across RomM and SQLite.
A server-only target still needs its full detail before it can be validated and persisted. That mandatory fetch keeps
the normal retry/classification policy and is not preceded by a redundant exact-id probe; only its typed 404 is peeled
into version_vanished. A refusal performs no binding, row/install/applied-launch-options write, launch-command
resolution, event, cache invalidation, sync, or completion-stamp update.
Explicit removed-game cleanup (services/prune/)¶
| Module | Role |
|---|---|
service.py |
Callable facade, atomic preview/run admission, and claimed frontend action leases |
executor.py |
Serial per-group liveness, recovery, Steam-action, reconciliation, and finalization state machine |
preview.py |
Local generation-gated candidate snapshot, complete affected-group disclosure, sizing, and paging |
recovery.py |
Lossless aggregate snapshots, recovery artifact assembly, and sealed-state comparison |
registry.py |
Short SQLite reads, action/final race validation, reconciliation, and final cascade delete |
requests.py |
Preview/option decoding, bounded selections, and lossless bounded Steam snapshot validation |
PruneService is the only path that deliberately deletes retained roms aggregate roots. A bulk preview is local-only:
for each platform it requires a non-empty completed fetch generation and selects rows whose last_fetch_id differs,
including NULL row generations. An inline preview may nominate one concrete retained ROM without generation evidence.
Both forms return serialized-byte-budgeted pages plus an ephemeral fingerprint. A page may therefore contain fewer than
the requested row limit, and the next offset advances by the rows actually returned. Pages include every member of an
affected sibling group, with generation candidates marked separately, so whole-game deletion cannot reach an undisclosed
row. start_prune consumes a finalized preview-bound installed-content selection. The frontend stages that selection in
bounded pages, so wire bounds do not cap the total selected set. Start atomically refuses any registered conflicting
callable and reserves the run before rebuilding the preview; concurrent starts cannot consume one token twice, and
shutdown owns/cancels an admitted refresh before it can spawn a run. Each sync, download, migration, version-switch,
save-write, session, uninstall, connection-identity change (including a successful connection test), or cache-mutation
callable registers for its full lifetime before its first await. Detached status, download, and playtime tasks transfer
the claim to their task lifetime. Core/disc writes, launch evaluation, and Steam Input application are included because
they mutate recovered state. Frontend-owned shortcut removal, core/disc writes, version switches, uninstalls, SGDB/icon
application, download completion, home migration, startup healing, pre-launch healing, and every post-sync Steam branch
(launch options, collections, playtime, and overview metadata) hold tokenized conflict leases through their final Steam
write and bounded release. Active continuations heartbeat those leases once per minute. A global frontend registry
signals cooperative cancellation by component owner or on plugin dismount. Every backend wait captures the current
plugin/owner mount generation before it starts; teardown tombstones that generation synchronously, so a lease-bearing
response that arrives afterward is released without running its continuation. Only a genuine plugin/component remount
opens a new generation. Cancellation stops every not-yet-started Steam mutation and lease renewal, but explicit backend
release waits for any already-started non-cancellable Steam promise to settle. An unresolved operation stops renewing
after a bounded five minutes; the backend's five-minute no-heartbeat expiry is the abandonment backstop if it never
settles. Launch funnels carry the admission captured at the original Play action through every gate, modal, and
launch-options confirmation wait. The version picker likewise rechecks its captured owner admission after save-sync and
modal waits before any successor switch_version mutation, so an unmounted chain cannot resume under a new picker. Each
non-empty sync_stale event carries its own lease through the paced removal tail; a later sync_complete lease
overlaps and joins that same promise, so success composes both leases while a post-stale backend failure still leaves
the tail covered. A terminal prune result that needs repoint publication likewise acquires its lease before event
delivery while the old run is active; the frontend holds it across release acknowledgement and cover publication. Event
delivery failure releases a token that never reached the frontend. This closes the reciprocal admission race, and each
path refuses while a prune claim is active. Migration and active-library-sync decorators additionally guard preview and
start.
The executor processes sibling groups serially and catches ordinary exceptions per group. It rejects multiple shortcut
bindings and active downloads, pins the preview's canonical RomM origin/token-origin/user namespace, probes every local
group member with the single-attempt three-second get_rom_once, and treats only a typed RommNotFoundError from that
same namespace as destructive authority. A connection/server/user change before, during, or after an exact-ID request is
uncertain. Live, malformed, wrong-id, transport, authentication, timeout, server, and unknown outcomes retain data. Long
recovery and frontend round trips are followed by another exact-ID proof before local source removal; every repoint
re-proves both the vanished source and live target after the frontend action, including a vanished source that is not
itself a generation candidate. The natural live repoint target uses the same filename-stem projection and
resolve_group_representative ranking as the version picker with empty installed/bound preference sets. Repoint
selection is independent of row deletion.
Recovery is coordinated through ordered ROM save locks, acquired with no UoW open. The lock set is retried until shared save ownership is stable. Short read UoWs close before filesystem work, and no RomM request, event emission, or frontend wait runs under those locks. The bundle is sealed before mutation. Aggregate, save-inventory, bundle/source, controller, and fresh frontend Steam state are validated before the irreversible Steam action. After that action the service reacquires the same recovery-set locks and repeats the guards immediately before local finalization; quarantine ownership is separately projected only for rows being deleted. The locks remain held through save quarantine, filesystem-only installed-content removal, and the final parent cascade. Source sets record expected presence or absence, the root's sealed no-follow identity and regular-file hash, and every descendant identity, mount ID, and regular-file hash. Every resolvable exclusive save path is represented even when it was absent. Traversal refuses nested mount transitions. After the anchored root rename, deletion obtains kernel read leases for every regular root/descendant before removing any entry and holds them through the last descriptor/hash check and unlink. An existing writer, an unsupported lease, or any inability to establish exclusion restores/retains the source rather than deleting bytes not in recovery. The controller VDF's held claimed inode uses the same writer exclusion from its final identity/hash validation through claim unlink on both normal publication and collision/rollback discard; setup failure preserves the claim and teardown failure after unlink is an ambiguous mutation. Content excluded from the bundle and recovery-off saves still receive a final complete no-follow claim; their remove/quarantine uses anchored parents and atomic no-replace rename rather than a raw path fallback, so a concurrently created backup is never overwritten. Writer-exclusion teardown faults after mutation are reported as ambiguous, not exact success. Controller rewrites revalidate the held claimed inode before every restore/discard branch and retain a newer claimed inode at a surfaced path when a concurrent Steam file wins publication. Every exclusive current-save path is expected absent after quarantine, regardless of whether it existed during inventory, and the whole set is collectively rechecked after all filesystem work and immediately before the aggregate cascade. Validation and claim decoding share one held bundle descriptor and return a digest that every later guard compares to the cached claims. Recovery root, staging, bundles, destination files, and the sealed bundle remain descriptor-anchored through copy, metadata application, hashing, cross-directory rename, validation, and both-parent fsync. Failure cleanup uses the same mount-aware descriptor-relative remover; if cleanup cannot prove a safe tree, it leaves and reports the full anchored staging path instead of recursively entering uncertain data. Adapter outcomes carry actual and durability-ambiguous changes into the mutation ledger even when a later item or parent fsync fails. The final SQLite delete is one short UoW that revalidates complete row/binding state and invalidates intersecting collection stamps; platform stamps are deliberately preserved (removed-game-cleanup.md).
Frontend Steam events use a claim/complete protocol. A claim checks the run, token, discriminant, appId, target, exact
single-binding group, and current binding before the frontend rechecks the live rom-launcher executable and mutates
Steam. The lease is monotonic-clock bounded and rechecked after asynchronous validation; an identical repeat claim is
idempotent, while mismatched or expired claims cannot authorize a mutation. Repoint commits through the normal
version-switch authority first; shortcut removal is immediately reconciled to an unbound local row after Steam confirms
absence. A claimed action whose completion is lost, or a RemoveShortcut/launch-options write that was attempted but
could not be confirmed, is an explicit ambiguous partial. A pre-mutation refusal remains an ordinary failure. A later
run can confirm a shortcut is already absent and reconcile without repeating Steam removal. A later guard failure is
likewise an explicit partial result with the committed action and actual mutation categories recorded, not a claim
that the group was unchanged. Cancellation can stop final guards and every later group; shielding starts only with the
first irreversible local mutation and awaits that phase to a known result. Cancellation remains authoritative if that
shielded child faults: the current group's truthful fault/ledger result is recorded and no later group starts. Terminal
result strings/arrays are bounded and chunks are built to a serialized byte budget. Every action/progress/completion
frame carries the originating preview ID, allowing the frontend to adopt a matching run even if the successful start
response is delayed or lost while still rejecting foreign frames; completion finalizes only a contiguous chunk sequence.
An accepted contiguous terminal sequence seals that run against every later action, progress, or completion frame, and
the modal exposes its terminal controls immediately even if the start response is still pending. Committed repoint
publication performs a bounded/retried backend release acknowledgement before gated cover/status work. Mixed runs
continue unrelated groups.
LibraryService decomposition (services/library/)¶
The library sync subsystem is a façade over three sub-services that coordinate through a shared LibrarySyncStateBox:
| Module | Role |
|---|---|
service.py |
LibraryService façade — public callable surface; wires the sub-services and delegates |
fetcher.py |
LibraryFetcher — read-only RomM roundtrips: list platforms/collections, the incremental/full pagination loop, per-unit work-queue construction |
sync_orchestrator.py |
SyncOrchestrator — preview (read-only), the per-unit apply pipeline, cancel, the heartbeat clock, progress emission |
reporter.py |
SyncReporter — post-apply finalisation (artwork filenames, per-unit roms upsert + SyncRun lifecycle) and the roms-derived queries |
_state.py |
LibrarySyncStateBox — shared mutable in-flight sync state; single source of truth threaded through every sub-service, and the sole owner of the run-lifecycle pair (sync_state / current_sync_id) via its verb methods |
The pipeline is split fetch (read-only) / apply (owns persistence): the fetcher never mutates the roms registry or
rom_metadata, and the reporter's per-unit commit upserts each acked ROM's roms row and stamps its cached
rom_metadata in the same write Unit of Work (Rom row first, then metadata — FK-safe). So a preview never mutates
state, and an interrupted apply leaves only the units it already committed — incremental, per-unit delivery.
Where the synced-ROM state lives. The registry of synced ROMs, the last-sync timestamp, and the sync stats are
SQLite, not JSON. The reporter upserts each acked ROM into the roms table via Rom.synced(...) / update_cover_path
/ assign_sgdb_id (artwork and steamgrid patch cover_path / sgdb_id on the same aggregate during the per-unit
commit) and, in the same write UoW, stamps the ROM's cached rom_metadata (build_rom_metadata maps the live RomM
metadatum — Rom row saved first so the rom_id FK holds); the orchestrator drives the SyncRun lifecycle (start at
apply-dispatch, complete / mark_cancelled / mark_interrupted / mark_errored at finalize). The terminal
sync_complete event (and its progress frame) is emitted LAST — reporter.emit_sync_complete, called by the
orchestrator only AFTER that terminal SyncRun status is persisted — so a frontend stats refetch triggered by the event
reads the fresh run status instead of racing the write (the #39 lag); reporter.finalize_per_unit_run keeps only the
stale-unbind + sync_collections emit and returns the app-id maps. sync_stats.roms is a registry-derived
bound-shortcut count computed at read time (the ROMs still bound to a shortcut in roms, i.e. shortcut_app_id not
NULL), not a stored scalar. The old JSON shortcut_registry / last_sync / sync_stats are gone from this path; all
writes go through the roms / sync_runs Repository Protocols behind a narrow Unit of Work (per
ADR-0006 the UoW
spans only the DB write, never the up-to-60s frontend ack). The platform slug → display_name map resolves live from
RomM each sync and is cached in a kv_config row for offline reads. Removing a shortcut unbinds the ROM
(Rom.unbind_shortcut() NULLs shortcut_app_id, keeping the row and its per-ROM children) rather than deleting it, per
ADR-0007.
The full schema and aggregate model are in Database Design.
Delta-restricted apply: emit only new + changed
(ADR-0025).
Between the sibling-group collapse and chunking, _sync_one_unit runs domain/sync_diff.py::classify_roms over the
unit's collapsed entries against the bound-shortcut registry and emits only the delta — new + changed (plus rebind
entries, which always move their binding). "Changed" is content, not identity-only: on top of the identity triple (name
/ fs_name / platform_slug) the classifier compares each item's freshly built target launch_options against the
applied_launch_options recorded on its roms row (the launch command last written to that shortcut). A match on both
is content-unchanged and skipped — it never reaches the frontend, so no Set* walk and no ~2 s AppDetails confirm
poll — while an install/uninstall or core/disc pin change that leaves identity untouched still flips the item to
changed. A NULL recorded value (a pre-migration-015 row, or a freshly created row not yet recorded) is unknown and
never skipped, so the first post-upgrade sync re-applies exactly as before, records values, and only later syncs skip.
Skipped rows are still committed (chunking routes their groups to chunk 0's leftover), so no DB work is dropped and
the per-platform completion stamp still rides the final chunk only — a platform is stamped exactly when its whole delta
is durable, and an empty-delta platform emits one empty chunk that commits every row and writes the stamp. The recorded
value is refreshed by six writer sites (sync ack-commit, download-complete, adopt-complete, uninstall, RetroDECK-home
migration re-resolve, version switch), each recording the value the frontend just wrote onto the shortcut; a missed
writer only ever causes a harmless spurious re-touch, never a wrong skip (the benign-failure asymmetry). The per-unit
sync_apply_unit's unit_total is therefore the delta size, so the progress counter shows net progress and a resume
converges quickly.
Cover-cache invalidation: the cover_source fingerprint (#1386). The per-ROM cover cache is only valid while the
server's cover is unchanged, and the delta apply never re-downloads a skipped ROM's cover — so each roms row records a
cover_source fingerprint: the full RomM cover source string (path_cover_large else path_cover_small, the embedded
?ts=… cache-buster included) whose bytes the cache file holds, compared as an opaque string (catches both a ts bump
and a path change). Two compare layers run per unit, both in ArtworkService:
- Apply-path covers (
_resolve_cached_cover): a stored fingerprint that differs from the fresh one blocks both the cache-hit reuse and the grid→cache seed, forcing a fresh download. - The invalidation pass (
refresh_changed_covers, called by_sync_one_unitbefore the cover download): for every BOUND fetched ROM whose stored fingerprint differs — delta-skipped ROMs included — it re-downloads the cache file (atomic tmp+rename), republishes the grid{app_id}p.pngcopy, persists the new fingerprint in a small write UoW (an observed server fact, deliberately NOT ack-coupled likeapplied_launch_options), and collects{rom_id, app_id}. The pass scans against the unit's bound-row registry projection (_read_apply_registry, which carriescover_source) — the same read the group collapse diffs against, so it opens no per-ROM DB lookups. That list rides the unit's firstsync_apply_unitchunk ascover_refreshes, clipped to the session-budget headroom left after the chunk's own projected cost (each refresh ≈ one transient cover,COVER_TRANSIENT_KB) — clipped, never pausing the run: the grid files are already updated, a Steam restart shows the rest. The frontend re-applies each entry viaSetCustomArtworkForAppat the 50 ms cadence before the chunk ack — without that push the tile stays stale in-session, sincert_custom_image_mtimeonly bumps throughSetCustomArtworkForAppor a client restart. - The preview-side count (
sync_preview→domain/cover_refresh.py::count_cover_refreshes):classify_romsis deliberately cover-blind (ADR-0025), so a cover-only server change yields an empty shortcut delta — and the QAM's preview flow used to short-circuit on "no changes", meaning the apply (and with it the invalidation pass) never ran and the fingerprint stayed stale forever. The preview therefore counts the fingerprint mismatches itself, with the SAME pure kernel the apply pass scans by (scan_cover_refresh_candidates— shared viadomain/cover_refresh.py, so preview count and apply set cannot diverge), as an in-memory compare of the fetched union (platform and collection units alike) against the registry projection the classify already read — no extra DB pass, no downloads, no writes. The result rides the summary as the additivecover_refresh_count(absent/0 tolerated by old consumers); the frontend treats a cover-only preview (all diffs zero, count > 0) as apply-able, showing "No shortcut changes — N cover updates" with the normal Apply/Cancel confirm. NULL-fingerprint rows are not counted: their adopt-vs-download fork needs a cache-file check the preview must not perform, and the adopt is invisible to the user. The wholesale platform-skip gate needs no cover awareness — a cover change bumps the ROM'supdated_at, which already forces the fetch; a skipped platform's registry-reconstructed thin ROMs carry no cover fields and count 0 by construction. - The empty-unit chunk vehicle: a cover-only unit has an empty apply delta, but
build_unit_chunksyields exactly one empty chunk for it (ADR-0023's empty-unit round-trip), so thecover_refresheslist still rides async_apply_unitframe withshortcuts: []; the frontend processes the refreshes and acks the empty chunk, and the commit advances the unit normally. No special-casing exists on either side.
A NULL fingerprint (every pre-migration-016 row) with an existing cache file is adopted: the fresh fingerprint is
persisted without a download, so the upgrade never mass re-downloads a library; a NULL with no cache file downloads on
the apply path as before. The fingerprint is only ever advanced when the cache is actually confirmed against the server
— a fresh download, a cache reuse/seed (the reporter's commit merges the confirmed value from the unit's
pending_cover_sources staging, else preserves the existing one), or the NULL-adopt — so a failed download keeps the
old fingerprint and the change is retried next sync. A cover-only change never re-applies the shortcut itself (that
apply path writes launch options under the ADR-0025 invariant — pure churn for a cover).
url_cover fallback on a 404 RomM asset (#1450). When the RomM-local cover asset returns an HTTP 404 (the server's
own cover resources are missing while its web UI still renders from url_cover), the cover download retries once
against the ROM's url_cover — an external metadata-provider CDN (SteamGridDB / IGDB / …) — before giving up. The retry
lives in ArtworkService._download_cover_atomic: only a RommNotFoundError (404) with a non-empty url_cover triggers
it, so a transient transport error keeps today's retry-ladder behaviour and never falls back, and an empty/absent
url_cover is exactly today's failure (warning, gray tile). The cover routes are exactly why the entity-verdict rule in
what makes a 404 an entity verdict carves them out: a
static-mount miss answers with the generic route-404 body, so demanding an entity answer there would disarm this
fallback. The fallback fetch goes through a separate, bearer-free adapter seam
(RommRomReader.download_cover_from_url → RommHttpAdapter.download_external): the host-bound RomM bearer must never
reach a third-party origin, so only the plugin User-Agent is attached (the CDN behind Cloudflare Bot Fight Mode also
403s the default Python-urllib UA). Because url_cover is untrusted server-supplied input, download_external
scheme-allowlists it to http/https before any fetch — a file:/data:/ftp:/scheme-relative URL is rejected with
a RommApiError so a file:///etc/passwd never reaches urlopen (the broader bounded-SSRF hardening — post-DNS
private/link-local IP blocking and per-redirect-hop revalidation — is deferred to #1182). The fingerprint records the
source actually applied — url_cover on a fallback, threaded back through the download_artwork applied_sources
accumulator (sync path) or the direct persist (refresh_changed_covers / refresh_cover) — so the compare stays
truthful: because the fresh RomM path_cover string never equals the stored url_cover, a later fixed RomM asset (or a
changed url_cover) is always re-checked, at the cost of re-downloading the fallback ROM's cover each sync until the
RomM asset is repaired.
Conditional-request revalidation on a ts-only change (#1454). A server-side rescan re-stamps every ROM's
updated_at — and thus the ?ts= in every cover path — without touching the cover files, so the cover_source
fingerprint changes library-wide and the #1386 compare would re-download every cover. Instead, when a fingerprint
change is ts-only (the paths are equal once the ts query param is stripped —
domain/cover_refresh.cover_ts_only_change) and a stored validator exists, the cached bytes are revalidated with a
conditional request rather than re-downloaded: every regular download records the response's HTTP validator (ETag /
Last-Modified) in a {rom_id}.cover-meta.json sidecar beside the cache file, and a ts-only change re-requests with
If-None-Match (else If-Modified-Since) through the normal authenticated RomM adapter path
(RommRomReader.download_cover → RommHttpAdapter.download_conditional, bearer + UA as today). A 304 Not Modified
keeps the cached bytes, adopts the fresh fingerprint (so the next sync is clean — this is the whole point), and
refreshes the validator if the 304 carried one, with no grid republish and no in-session tile re-apply (the tile
is already current); a 200 replaces the bytes, validator, and fingerprint. Validators are an optional capability
— a ROM with no stored validator (first sync, or a server/proxy that strips them) plain-downloads and seeds one for next
time, and a conditional-request transport error is exactly today's failed-download path (retry ladder, old bytes + old
fingerprint kept). The sidecar lives beside the cache file (pruned with it on shortcut removal and by the orphan sweep);
the #1450 url_cover/CDN fallback path is out of scope for revalidation and keeps its once-per-sync re-fetch. No RomM
minimum-version change — the floor is untouched.
Unstamped-platform re-run: the restamp_platform_count signal (#1416). A heartbeat-timeout run's late-ack recovery
(ADR-0023) leaves a platform complete but unstamped: the timed-out apply cleared the stamp at its start and its late
ack re-binds the chunk without re-writing it (the late-ack path never passes a platform_stamp). The wholesale-skip
gate then full-fetches that platform on every future sync (no stamp = no skip authority), and — because its shortcut
delta is otherwise empty — the QAM's preview would short-circuit on "no changes", so the re-walk that would re-stamp it
never runs and the run's interrupted status lingers indefinitely (only an apply run records a fresh SyncRun).
sync_preview therefore counts the enabled platforms lacking a PlatformSyncState stamp (_count_unstamped_platforms,
a side-effect-free read) and rides it as the additive summary field restamp_platform_count (absent/0 tolerated by old
consumers). The frontend treats a restamp-only preview (all diffs zero, count > 0) as apply-able — "No changes —
finishing an interrupted sync." with the normal Apply/Cancel confirm — mirroring the cover-only flow. The gated apply
then runs the unstamped platform (the skip gate never skips it), and its 0-delta empty final chunk re-writes the stamp
and records a fresh completed SyncRun, healing both symptoms. The stamp write stays pipeline-owned: the preview only
counts unstamped platforms, it never stamps (check_sync_lifecycle_owner / the final-chunk-only rule).
Per-unit apply is chunked, durable per chunk
(ADR-0023). The
delta emit list is split into fixed-size chunks (_APPLY_CHUNK_SIZE, 200) by the pure
domain/sync_chunking.py::build_unit_chunks, and the orchestrator processes them one at a time: emit sync_apply_unit
→ wait for the ack → commit that chunk's roms rows durably → next chunk. Each sync_apply_unit carries chunk_index
/ chunk_count / chunk_offset / unit_total, and shortcuts is the chunk slice; the reporter's per-chunk commit is
the same group-aware two-pass write (every fetched sibling upserted, only representatives bound; Rom row then
rom_metadata, FK-safe) over that chunk's row subset, so a committed chunk is crash-safe on its own and a mid-unit
failure forfeits only the in-flight chunk. Chunks cut only at sibling-group boundaries (overflowing 200 to keep a
game's dumps whole so they never straddle two commits); no-emit groups and unmatched leftovers ride chunk 0, and an
empty unit is one empty chunk (the empty round-trip still commits its unbound rows). The whole-unit staging
(pending_sync / pending_all_roms / pending_cover_sources) is set once; only the per-chunk coordination is re-armed
each chunk. The motivating field crash is #797: a
3084-shortcut unit emitted in one frame lost ~24 minutes of work when steamwebhelper OOM-crashed before any ack — a
200-chunk caps both the bridge payload (~200 KB vs ~3 MB) and the crash blast radius (~2 min vs 24+).
Per-chunk wait: timeout vs. cancel. The orchestrator emits each chunk's sync_apply_unit, then waits on
unit_complete_event (heartbeat-clocked) for the frontend's report_unit_results ack. When the wait returns None the
teardown branches on the cause (#1052) — and either way, every chunk committed before this one stays committed:
- User cancel (
is_cancelling()already True at the return) — the in-flight chunk is intentionally discarded: clear the staging, nullunit_complete_event, and clearactive_unit_id+active_chunk_index. A stray late ack then no-ops. - Heartbeat timeout (still RUNNING) — the frontend has already created this chunk's Steam shortcuts and will fire a
late
report_unit_results, but in production after the run has already wound down. The orchestrator moves the abandoned chunk into anabandoned_chunkstash on the box (stash_abandoned_chunk): its run/unit/chunk identity plus this chunk's ROMs (only the abandoned chunk, never the whole unit, themetadatumsource), while it keeps the whole-unit staging (pending_sync/pending_all_roms/pending_cover_sources) live for the recovery commit to read and clears the dispatch identity (unit_complete_event+active_unit_id+active_chunk_index). It then marks the runinterruptedand flips CANCELLING so the loop stops. The stash lives outside the run-lifecycle state and deliberately survivesfinish_run(which nullscurrent_sync_id), so a late ack arriving after teardown can still recover it (#1367 — an earlier design keptactive_unit_idlive and lost the ack once the run wound down).SyncReporter.report_unit_resultsthen matches the stash by identity (take_abandoned_chunk) and drivescommit_unit_resultsitself over the stashed rows — every fetched sibling upserted (identity + metadata), only the acked representatives bound, and never aplatform_stamp(a timed-out platform is incomplete) — persisting the delivered bindings instead of leaving orphan shortcuts that the next sync re-creates as duplicates. The stash has a bounded lifetime: the next run'stry_begin_runclears it, so a frontend that crashes and never acks just leaves inert data. The committed binding is mapped by the next sync's existing-shortcut scan, so no active orphan deletion is needed (a Steam shortcut is the sole record of its tile).
Session-budget gate: GC-before-measure at each chunk boundary
(ADR-0024).
Steam's SharedJSContext renderer OOM-crashes at ~2.45–2.53 GB RSS and never self-recovers within a session; each
created shortcut costs 0.7–1.5 MB permanently (measured on-device). Chunking (ADR-0023) makes hitting that cliff a cheap
resume; this gate stops the run before it, at a chunk boundary, as a controlled pause. At each chunk boundary the
orchestrator reads the renderer's RSS (RendererRssFn, adapters/renderer_rss.py — the max VmRSS across
steamwebhelper processes from /proc) and, when that raw reading is at/above GC_SKIP_BELOW_KB (1.5 GB), forces a
renderer GC (RendererGcFn, adapters/renderer_gc.py — an HeapProfiler.collectGarbage over the CEF debugger on
localhost:8080, so the reading reflects settled heap not transient garbage that Steam's measured-unreliable natural GC
hasn't reclaimed) and re-reads; below the floor the raw reading (which can only over-estimate) already clears every
threshold, so the ~5 s GC is skipped and a small sync pays zero GC cost. It then runs the pure
domain/session_budget.py::gate_decision over the chunk's composition-priced cost (chunk_worst_cost_kb): now that
the emitted chunk is new + changed only (the delta-restricted apply above), a create is priced at the worst-case create
rate plus its transient cover term while a changed/rebind item is priced at the lighter Set*-walk rate — the gate no
longer prices every item as a cover-applying create. Pause iff rss + chunk_cost ≥ limit; the pause log line names the
composition ("… + N creates + M updates projects …"). The frontend decides create-vs-update itself via its
existing-shortcut scan, so a small backend/frontend mismatch only ever overprices (worst-case safe). Both modes are
predictive and differ only in the limit line: every later chunk projects against cliff − margin (≈2.2 GB,
keeping the anti-thrash safety margin), while the run's very first chunk projects against CLIFF_KB (≈2.45 GB)
instead. Forward progress must be guaranteed (the run has to apply at least one chunk or it loops forever on a
no-progress pause), so the first chunk is allowed to spend into the safety margin — but the predictive projection
still stops it before the crash line. A resume's first chunk therefore proceeds only when its worst-case peak stays
below the cliff (≈1.95 GB for a full 200-item chunk of cover-applying creates, each priced create + cover) and can never
be projected past it; a resume attempted without a Steam restart re-pauses cleanly rather than driving a chunk into the
cliff. On a pause it sets run_paused + a distinct interrupt_reason and requests cancel — the loop returns cleanly
with prior chunks committed and the terminal write records the new terminal status paused (migration 014, its own
status distinct from a crash's interrupted; both resumable, but the split lets the UI say "(paused)"). Completed
platforms keep their PlatformSyncState stamps, so Resume Sync redoes only the remainder. Every step is fail-open:
an unavailable RSS reading (no steamwebhelper, unreadable /proc) skips the gate — and short-circuits further GC
attempts — for the rest of the run (logged once), a seam error is caught locally, and a failed GC only makes the reading
less precise; measurement never blocks a sync. The same seams feed the UI surfaces: sync_preview returns
pause_likely (a predict_run_crosses prognosis pricing only new creates + changed updates, never fully-unchanged
items, so an unchanged re-sync never warns), a clean run's sync_complete carries restart_recommended
(post_run_advisory, RSS > ~1.8 GB, read GC-first), and the get_session_budget_status callable returns a live RSS
reading (no GC) plus the three fixed threshold lines (warn_kb ≈1.8 GB, ceiling_kb ≈2.2 GB, cliff_kb ≈2.45 GB) for
the persistent QAM banners (a blue "paused" banner, a yellow high-heap banner) and the always-on "Steam memory" status
row. The row's value text is traffic-light coloured against those three thresholds — green / yellow (warn_kb) / red
(ceiling_kb) — so the frontend holds no threshold magic numbers, and while a sync runs (or a paused banner is showing)
the row polls the callable (~5 s during a sync, ~10 s while paused) so the number tracks the climbing RSS and the blue
paused banner notices once a Steam restart frees memory. That notice is driven by resume_ready on the callable
(domain.session_budget.resume_would_proceed: rss + RESUME_HEADROOM_CHUNKS × FULL_CHUNK_WORST_KB < ceiling — room for
TWO worst-case chunks, ≈1.2 GB bar, because a one-chunk bar sits exactly on the pause point where Steam's own small
frees flicker the verdict; None when RSS is unreadable) — when it flips true the blue banner reads "Steam memory is
free again — press Resume Sync" and hides the restart button. The callable also carries the paused run's progress
(run_done_items / run_total_items), so that banner reads "1200 of 2001 games done": the counters are run-scoped
fields on LibrarySyncStateBox, stamped with the plan's ROM total and grown by the delta-restricted apply's SKIPPED
entries (already correct on their shortcut), each wholesale-skipped unit's ROMs, and every COMMITTED chunk's acked items
— an emitted-but-uncommitted chunk (cancel / heartbeat timeout / the pause itself) never counts, so the number can't
over-report. They live in the backend deliberately: the plugin process survives the Steam restart the banner asks for,
while the frontend reloads. In-memory only — a plugin reload wipes them and both come back None, which the banner
renders by dropping the sentence rather than showing a zero. That row also shows the last run's signed RSS growth,
appended inline after the value ("X.X GB · last run ±Y"), measured at EVERY terminal (completed / paused / cancelled /
interrupted) so a paused run reads as its own consumption-so-far rather than a prior clean run's: a RAW read taken
unconditionally at run start is the baseline (run_start_rss_kb — captured before any chunk, so even a
fully-incremental-skip run still records one and reports ≈ +0.0 GB), the terminal RSS read is the end, and
session_memory_delta differences them (an approximation for information only, which a raw start baseline is fine for).
The value is retained in last_run_delta_kb so get_session_budget_status surfaces it on a QAM remount (in-memory
only, lost on reload, no migration; None when either endpoint was unmeasurable, so a stale delta is never shown); the
UI reads it from that callable, so it is deliberately NOT put on the sync_complete wire. Both banners also offer a
Restart Steam now button that calls SteamClient.User.StartRestart directly from the frontend — a deterministic
full client restart that resets the renderer's per-session budget to the ~430 MB baseline. The button is disabled while
a game is running and hard-guarded on click (isAnyAppRunning) so a restart can never close a game. The RSS reader and
GC trigger are wired through SyncOrchestratorConfig; the gate's per-item cost is a parameter, and because the apply
now pushes each created shortcut's cover through Steam's artwork API (SetCustomArtworkForApp, transiently resident but
GC-reclaimable — hence the GC-before-measure), the orchestrator prices each create at the worst-case create rate
plus the transient cover term (COVER_TRANSIENT_KB) at both the chunk gate and the preview prognosis, while a
changed item stays at the lighter update rate.
Run/unit/chunk identity on the ack (#1041). Every sync_apply_unit event carries the run_id (the run's
current_sync_id UUID), the unit_id (the WorkUnit.id), and the chunk_index; the frontend echoes all three back on
the report_unit_results ack. The orchestrator stamps the dispatched unit's id into active_unit_id and the current
chunk into active_chunk_index just before it emits, and report_unit_results validates the ack against them: the
run_id must match current_sync_id, the unit_id must match active_unit_id (both compared by string value, since a
platform's id is numeric and a collection's is a string), and the chunk_index must match active_chunk_index (as
an int). An ack that fails the active-unit check falls through to the abandoned-chunk stash
(take_abandoned_chunk): a heartbeat-timed-out chunk's late ack — which in production arrives after the run wound down,
so active_unit_id / active_chunk_index are already cleared and current_sync_id is null — matches the stash by
identity and drives the recovery commit (the per-chunk timeout branch above, #1367). Only an ack matching neither
the active chunk nor a stash — a late ack from a cancelled run arriving while a fresh run is in flight, a stray ack
for a different unit, or a superseded chunk — is ignored (logged at debug, returns
{success: True, count: 0, ignored: True}): it is neither recorded, signalled, nor committed, so it can never be
credited to the wrong run/unit/chunk. active_unit_id + active_chunk_index are cleared once the chunk commits, is
cancelled, or times out (its identity moves into the stash). On the frontend side, the unit handler does not send the
ack at all once cancel has been requested — the first line of defence against a cancelled run's bindings landing in
whatever run started next.
Run identity on the cancel (#1198). cancel_sync(run_id) is run-scoped, mirroring the ack-path
_ack_matches_active_unit check above. A Cancel click meant for run N can land after run N finalized to IDLE
(current_sync_id nulled) and run N+1 started fresh — an unscoped cancel would flip run N+1 to CANCELLING (a sync the
user never cancelled), which would abort it and report cancelled. So when the requested run_id is truthy and does
not match current_sync_id, the cancel is ignored (logged at INFO, returns
{success: True, message: "Cancel
ignored (stale run)"}) — sync_state is left RUNNING. A matching id (or no active
run) flips RUNNING → CANCELLING as before. A falsy/None run_id cancels unconditionally — legacy callers and
the "no active run id captured yet" safety case — so cancel is never made less reliable. Every cancel logs one INFO line
recording the requested run_id, the active current_sync_id, and the sync_state at call time. The frontend sources
the run id from the backend-fed sync_progress store — its additive runId field (see below) — and passes it on the
Cancel click; in the pre-run "Fetching library…" window the store's runId is still "", which maps to the
unconditional cancel path rather than a stale prior-run id the backend would reject as a cross-run mismatch.
sync_cancel_preview is not run-scoped: it only clears the pending preview delta (pending_delta), never touches
sync_state, and sync_apply_delta independently validates the preview_id, so a stale preview-cancel cannot abort a
fresh sync.
Group-aware sync — one Steam shortcut per sibling group
(ADR-0021).
A game frequently exists in a RomM library as several dumps of the same title (region / language / revision variants).
These share a sibling group (same roms.sibling_group_key, derived client-side by domain/sibling_group.py: a
connected component over RomM's sibling_roms edges, keyed by the highest-priority metadata source the component agrees
on — compute_component_group_keys, stamped onto the raw dicts before the shortcut build at both the preview and
per-unit call sites, ADR-0022). The sync pipeline treats a group as one game = at most one new Steam shortcut; the
sibling row holding shortcut_app_id is the group's active version.
- Persist all siblings, emit one shortcut per group. The reporter's per-unit commit upserts a
romsrow (identity + version metadata, via the sync UPSERT) for every fetched ROM of the unit, but only the group's representative carries a Steam-shortcut binding — a non-representative sibling is a tracked, unbound row. Persisting the whole group is what the version picker (#1297) and the incremental-skip gate need; binding only the representative is what stops the same-named-collision and duplicate-shortcut bugs the ADR describes. - The collapse happens at the same
build_shortcuts_databoundary in both paths (domain/sync_diff.py::collapse_sibling_groups). Preview and apply build the full per-ROM shortcut list, then collapse it to one entry per group against the bound-row registry (preview reads_read_preview_baseline, apply reads_read_apply_registryper unit). The collapse takes an explicitcomplete_group_viewflag: a sibling group is per-platform, so the whole-library preview union and each platform apply unit see a group's whole membership (complete_group_view=True) and collapse identically — the preview counts can never diverge from what those units create (the #1292 counts-vs-reality bug class). A collection apply unit spans platforms and may fetch only one unbound sibling of a group whose bound member it never fetched, so it is a partial view (complete_group_view=False): a group that already holds a binding anywhere is grandfathered untouched — never rebound onto the partial member, never given a second shortcut — because the group's real representative rides its own platform unit in the same run. (Inferring "bound sibling absent ⇒ vanished" from a partial view would rebind a live installed game onto an uninstalled sibling — #1296.)classify_romsruns over the collapsed set, so its new / changed / unchanged / stale buckets count games, not dumps, and an unbound sibling stops reading as a perpetual "new". - Resolution chain (
domain/sibling_resolution.py::resolve_group_representative, total + shuffle-stable): an installed sibling wins; else an existing binding; else RomM's per-user default (is_main_sibling); else the 1G1R ranking — prerelease demotion > region priority > revision (newest) > alphabeticalfs_name_no_ext>rom_id. The first three legs are membership filters; the rest are the total order applied inside the surviving leg. Prerelease demotion ranks first: a member whose structuredtagsname a draft build (Alpha / Beta / Proto / Sample / Demo, case-insensitive, tolerant of a trailing number;Unl/Aftermarket/ unknown tags are neutral) is demoted below every retail sibling across regions, so a finished(Japan)release beats a(USA) (Beta). Revision ranks just after region: within one region the newestrevisionwins (natural compare; base dump = lowest), but never lifts a lower-ranked region (a(USA)base still beats a(Europe) (Rev 9)). The alphabetical leg also keeps a base dump ahead of a filename-only re-dump ((Virtual Console),(Extended Edition)) RomM does not parse into a tag. Region priority ranks a version by its bestregionsentry against a fixed build-time order —World > USA > Europe > Japan, every other named region after these (alphabetically), a no-region version last (a fixed constant, not language/system detection) — which the user may re-head with a singlepreferred_regionsetting ("auto"= the fixed order; any other value lifts that region to the top). The leg is evaluated at resolution time, so the setting takes effect on the next sync and an existing binding shields its group (the installed/binding legs win before region priority is consulted). Used wherever one version must be chosen (a new group's shortcut, a rebind's target). The QAM dropdown that setspreferred_regionis populated from fixed anchors (Default, World, USA, Europe, Japan) plus the distinct regions read from the local library (get_known_regionsoverroms.regions— no server call), and a confirmation modal states the apply-at-next-sync / no-rename semantics before persisting. - Canonical name at mint (
domain/sibling_resolution.py::canonical_group_name): a NEW group's shortcut is named after the member ranked first by the pure order (prerelease demotion > region priority > revision > alphabetical >rom_id), ignoring the installed/binding/default filters — so a Japanese default still binds Japan but the shortcut carries the USA name (never majority voting: two Japan dumps + one USA yields the USA name). This is mint-time only: the name is sticky forever after (ADR-0021 §2), so a rebind/grandfathered entry carries the persisted bound name verbatim and a live shortcut is never renamed. The appId is minted from the canonical name by the frontendAddShortcut; the DB binding lands on the representativerom_id; the reporter is agnostic to the name↔rom mismatch (it binds onrom_id+ the acked appId, and persists each sibling's own RomM name frompending_all_roms).preferred_regionis read from settings bysync_orchestratorand threaded into every collapse call site (preview + apply) as the same value within a run. - Rebind, not remove. When every bound sibling of a group vanishes from the server but the group still has fetched
members — and the collapse sees the group's whole membership (a platform unit or the preview,
complete_group_view=True; a partial collection view never rebinds) — it emits one rebind entry keyed to the vanished sibling'srom_id(so the frontend reuses its existing shortcut through therom_id → appIdmap — no delete + recreate churn), carrying the representative's launch bake and abind_rom_idmarker. The commit translates the ack throughbind_rom_id, so the DB binding moves onto the representative; the collision-safeRomsave then unbinds the vanished row. The shortcut's appId, artwork, collections and playtime all survive — only the active version changes. A whole-group disappearance still routes through the normal stale path, and the #1036 committed-appId guard is unchanged (the reused appId is committed this run, so it is never emitted for removal). - Grandfathering. A group that already carries several bound shortcuts (a library synced before this model) keeps them all — each surviving bound sibling stays its own tracked shortcut; no new shortcut is minted for a group with a binding. Convergence to one-shortcut-per-group happens naturally as the user uninstalls duplicates.
- Downstream group semantics. The incremental-skip gate compares RomM's platform
rom_countagainst the persisted rows carrying the completion stamp's fetch generation — bound and unbound alike, so skip parity on platforms with sibling groups holds (a NULLsibling_group_keyon such a row still forces a backfill full-fetch), while a row for arom_idthe server has since dropped is excluded rather than inflating the count forever (#1504; the row itself is retained per ADR-0007 — nothing is deleted). The backfill gate reads the same generation: only a row the last complete fetch returned may demand a full fetch, because a dropped row is never returned again and no fetch could ever fill its key in — counting it would wedge the platform into a full fetch on every sync, forever. A stamp with no generation predates the contract and falls back to counting — and backfilling for — every row. Artwork downloads only for the emitted representatives (+ grandfathered bound siblings), never eager sibling covers. Steam-collection membership resolves each RomM collectionrom_idwith a group fallback — an unbound sibling maps to its group's bound sibling's appId — so collecting or favouriting any version puts the game's single shortcut in the collection.
Same-named collections union into one Steam collection (#1503). RomM enforces name uniqueness only per-table and
per-(name, user_id), so two enabled collections can share a display name — a standard collection and a smart/virtual
collection, or (multi-user) another account's public collection the list endpoints return. Steam's collection namespace
is by-name (RomM: [<name>] (host)), so both must resolve to the one Steam collection. The finalize accumulator
(_state.py pending_collection_memberships) is therefore keyed by a collision-free (collection_kind, collection_id)
identity — the same identity the #742 completion stamp uses — with the display name carried in the
CollectionMembership value; the real-sync and preview write paths share one key builder so they cannot drift. The
reporter (_resolve_collection_memberships) then groups the accumulator's entries by name and unions their resolved
appId sets (order-preserving, de-duplicated across collections; each collection's own resolution already dedups within),
emitting the unchanged by-name romm_collection_app_ids: {name → [appId]} contract — a single-collection name unions a
set of one and is byte-for-byte the pre-#1503 output. The union runs after the owner-scope filter (below): once a
foreign collection is dropped from the work queue it never reaches the accumulator, so "Mine" narrows what is unioned
rather than changing how the union works.
Collection kinds — one virtual kind covers RomM's browsable virtual types (#1538). get_collections and
build_work_queue group collections into three internal kinds: standard, smart, and virtual. RomM's own UI calls
the ownership-carrying first kind Standard (the plugin's earlier name for it was user, a misnomer renamed
internally by #1539 — display "My" → "Standard"; see the enabled-bucket migration note below). The virtual kind is
RomM's ownerless VirtualCollection (base64 id, no user_id, no stable updated_at — never stamped), and it carries a
virtual_type sub-field on each collection dict/setting so the UI can label the row. RomM's VirtualCollection has
five type values, but the plugin syncs only the two RomM itself surfaces as browsable collections: IGDB
franchise and the default IGDB collection (series). genre, company, and mode are intentionally excluded —
RomM treats genre/company as ROM filter facets (not collections) and mode as neither, so they never appear in
RomM's Collections view. The supported set is a single constant (services/library/fetcher._SUPPORTED_VIRTUAL_TYPES);
the fetcher fetches each supported type (per-type fail-open) and merges them under the one virtual bucket. Because the
type is baked into the base64 id, ids are globally unique across types, so one enabled-bucket keyed by id cannot
collide. The owner filter, stamp-exclusion, and per-unit ROM-fetch dispatch all stay a single kind == "virtual"
branch, not fanned out per type. On disk the enabled-collections bucket was renamed franchise → virtual by the
lossless settings.json migration v10 → v11 (domain/state_migrations._migrate_v10_to_v11): it renames the bucket
key while preserving every enabled id, so a previously-enabled franchise collection stays enabled and no re-login is
required. (The historical v2 → v3 split still produces the franchise bucket; the v10 → v11 step renames it afterwards,
so that frozen step is untouched.) The ownership-carrying bucket was likewise renamed user → standard by the lossless
settings.json migration v12 → v13 (domain/state_migrations._migrate_v12_to_v13, same merge-into-existing
semantics), and old collection_sync_state completion stamps keyed collection_kind = 'user' are rewritten to
'standard' by SQLite migration 022 so an unchanged standard collection still takes the incremental skip across the
upgrade (#1539).
Batch collection enable — save_collections_sync for a filtered subset (#1539). The Collections tab adds a name
search and (on Virtual) a per-type filter, and its Enable All / Disable All act on the current filter. When the view
is a bounded subset (a search or per-type filter is active) the frontend calls save_collections_sync with the matched
ids, the kind, and the enabled flag — a single settings write that stamps every id into the kind bucket, so the whole
kind is never touched. An unknown kind or a non-list id argument is rejected with the canonical failure shape; an empty
id list is a success no-op. The unfiltered whole-kind case still uses set_all_collections_sync (which re-fetches
the kind from the server) so a large id list never crosses the WebSocket bridge; the frontend gates that whole-kind call
behind a confirm. Both live on LibraryFetcher.
Collection owner-scope filter — "Mine" is a sync scope, not just a display filter (#1532). RomM's collection list
endpoints return the signed-in user's own collections plus every other user's public collection. The QAM
Show
collections control writes collection_owner_scope ("own" / "all", default "all" — the value stays own;
only the QAM label changed to "Mine" in #1539); get_collections also tags each row with is_own so the frontend can
hide foreign ones under "Mine". Ownership is a pure predicate (domain/collection_owner.is_own_collection): a
collection is own when it is a virtual collection (RomM's VirtualCollection model carries no user_id column —
these are global/derived and belong to no one, so they always survive), when the plugin's own identity is unknown (the
degrade-to-"All" fallback), or when the collection's user_id equals the stored romm_user_id; only standard and
smart collections carry a user_id to compare. build_work_queue applies the same predicate: under "own" with a
known identity it drops foreign standard/smart units from the queue — so a foreign collection enabled earlier is never
synced — while virtual units and every unit under "all" pass through unchanged. The scope filters over the
per-kind enable state without mutating it, so switching back to "all" restores the prior enables. Because an unknown
identity never filters, the feature is non-breaking: it silently no-ops until romm_user_id is stamped (see the
ConnectionService lazy-identity note), then activates — no re-login required.
Incremental skip — the per-platform completion stamp is the sole authority. A platform unit skips only when its
PlatformSyncState stamp exists
(ADR-0023); the
stamp's completed_at is the "unchanged since" reference for the updated_after server-delta probe. A completed
SyncRun's last_sync is deliberately not a fallback: a run-scoped timestamp cannot see a platform whose shortcuts
were locally removed and only partially re-applied afterwards, so trusting it can silently skip a platform with missing
shortcuts. No stamp means a full fetch — including, once, every platform's first sync after this contract shipped. The
reporter writes the stamp when a platform work unit's last apply chunk commits — atomically in the same write UoW as
the chunk's roms upserts — so a platform that fully synced inside a run the user later cancelled/crashed still skips
on the next run instead of re-walking every already-applied game through CEF. All the existing guards still gate the
skip (zero-bound-rows, the sibling_group_key backfill for rows carrying the stamp's generation, the updated_after
server-delta check, the persisted-row count match); additionally the stamp's rom_count must still equal the server's
platform rom_count — a server-side count change invalidates it.
The stamp's contract is stamp exists ⟺ the platform's most recent apply attempt ran to completion, so a stale stamp
can never skip a half-mirrored platform. Because unbinding keeps the roms row (ADR-0007), a platform's persisted-row
count survives a partial re-apply or a local removal, so a surviving stamp with a matching rom_count would otherwise
let the skip drop the un-recreated games (the #1025 silent gap). Two rules keep the contract true: the orchestrator
clears the stamp at a platform unit's apply start (once the fetch succeeded and the apply is about to emit its first
chunk) and only the final chunk re-writes it, so an apply interrupted by a crash / cancel / heartbeat-timeout before the
final chunk leaves none; and the local destructive flows — DangerZone remove-all and per-platform removals (via
report_removal_results) plus the Steam-UI-deletion reconcile (reconcile_live_shortcuts) — delete the touched
platforms' stamps in the same write UoW as the unbind. The reporter's server-side stale removal is the deliberate
exception (it leaves the stamp, since a server-dropped ROM lowers RomM's rom_count and the count guard catches it).
"Force Full Sync" (clear_sync_cache) clears every stamp (and resets the recorded applied_launch_options to NULL),
which is the entire full-re-fetch + full-re-apply arm — the stamps are the fetcher's sole skip authority. The
sync_runs history is deliberately preserved (#1318): it feeds no skip gate and is the source of the "Last sync"
display, so deleting it forced nothing and only blanked the panel to "Never" right after a reset.
Single-owner run lifecycle (#1202). The run-lifecycle pair — sync_state (idle/running/cancelling) and
current_sync_id — is mutated only through four verb methods on LibrarySyncStateBox, never by direct field
assignment from a sub-service. Confining those two writes to the box makes run admission, cancellation, and termination
a single compare-and-swap on the plugin's one event loop, so a rapid Sync/Cancel can't interleave a stale terminal with
a fresh run's start and leave a half-reset id:
try_begin_run(run_id)— the admission guard. Compare-and-swap: returnsFalse(no state change) if a run is already in flight, else flips IDLE → RUNNING and stampscurrent_sync_id. Every entry point (start_sync,sync_preview,sync_apply_delta) goes through it, so an overlapping second Sync/Apply is rejected with{success: False, reason: "sync_in_progress"}instead of beginning a concurrent run that would double the terminal events. A rejected apply does not consumepending_delta, so the still-valid preview survives for the legitimate apply.request_cancel(run_id)— the run-scoped cancel routing (the #1198 / #1200 logic, centralized):"no_sync"when idle,"stale"when a truthyrun_iddoesn't match the active run (cancel ignored, run left RUNNING), else flips to CANCELLING. A falsyrun_idcancels unconditionally.cancel_syncandshutdowndelegate here.finish_run(run_id)— the terminal. Compare-and-reset: resets to IDLE + nullscurrent_sync_idonly ifrun_idstill owns the slot; a late, foreign, or doubled terminal is a no-op and can never null a fresher run. Every exit path (success, cancel, error, zero-unit) of the apply pipeline and the preview funnels into onefinally: box.finish_run(run_id)per run — the scattered per-unit / reporter resets are gone.is_in_flight()/is_cancelling()— read-only state probes for the per-unit loop and checkpoints.
sync_preview adds a final cancel checkpoint after the unit loop, immediately before it stages pending_delta, so
a cancel landing in that last window routes into the cancelled branch ({success: False, reason: "cancelled"},
pending_delta left None) rather than staging a delta the user already cancelled. The confinement is enforced by
scripts/check_sync_lifecycle_owner.py (an AST gate that fails CI if sync_state / current_sync_id is assigned
anywhere outside _state.py).
sync_progress carries runId. Every sync_progress event (and the persisted get_sync_status snapshot) now
includes an additive runId: str field — str(current_sync_id or "") — so the frontend reads the active run id from
the authoritative backend store rather than minting or threading its own. The idle default snapshot carries runId: "".
The same sync_plan capture point also clears the frontend's per-run cancel flag (_cancelRequested). The per-unit
handler resets that flag at its own start, but an incrementally-skipped unit never runs that handler, so a skip-only
run could otherwise carry a stale cancel from a prior cancelled run; resetting once per run on sync_plan is the
reliable reset.
Sync time estimate and live ETA (frontend)¶
The QAM's time readout is a two-stage design layered on the sync_progress stream. It is pure frontend logic
(src/utils/syncEstimate.ts and src/utils/syncEta.ts, both unit-tested); the backend supplies the plan (per-unit
weights + planned totals, via sync_plan) and the applying frames.
- The plan is skip-aware (#1382). At plan time the fetcher stamps every platform
WorkUnitwith two estimate-only riders read in one short UoW (_read_plan_estimates+ the puredomain/skip_prediction.py):predicted_skip— whether the wholesale incremental-skip gate is expected to skip the platform, replaying the gate's local conditions only (completion stamp present, the stamped count and the count of rows carrying the stamp's fetch generation both match the server'srom_count, bound rows exist, no sibling-group-key backfill pending for a row of that generation; the gate'slist_roms_updated_afterserver check is deliberately not replayed — no network at plan time) — andcollapsed_count, the persisted post-collapse shortcut count, mirroring the collapse's lane selection (ADR-0021):max(1, bound rows)per sibling group — so a grandfathered legacy group with multiple independently-bound duplicates (§5) prices one shortcut per bound sibling, not one per group — plus one per keyless row. Both of those two ride thesync_planpayload conditionally-present (absent on collections, never-synced platforms, and failed reads);collapsed_countis additionally gated on the platform's completion stamp (#1412, mirroring theget_platformsgarnish below) — a never-synced platform holds only PARTIAL collection-sibling rows (ADR-0021), so an ungated count would weight the ETA below the true work, and without a stamp the frontend falls back to the rawrom_count. The payload'stotal_romsstays the raw pre-collapse total (backward compat); an additivetotal_estimated_itemssums0for predicted skips, elsecollapsed_count ?? rom_count. A third rider,bound_count(#1511), counts the unit's known ROMs that already carry ashortcut_app_id, and is the one rider that rides both unit kinds. On a platform it counts the persisted rows, read in the same short UoW the skip prediction already needed that count for, and is not stamp-gated: a bound row genuinely has a Steam shortcut whether or not the mirror is complete, and zero persisted rows honestly means "every planned item is a create". On a collection it counts the bound members of the completion stamp's storedmember_rom_ids(the same member set the skip replays), in one short read UoW covering every collection unit — no ROM fetch. The two sides are deliberately asymmetric on the empty case: a platform reports0, an unstamped or virtual collection is omitted. A collection's membership exists only in its stamp, and virtual collections are never stampable (CollectionSyncState.stampaccepts onlystandard/smart), so0there would claim knowledge that does not exist. Absent and0price identically today; the distinction keeps the field honest for later consumers, so do not collapse it into consistency. A collection's stored member set may be stale if membership changed since the stamp — accepted and bounded, since this is estimate-only and a freshness probe would mean network I/O at plan time. A fourth rider,new_shortcut_count(#1517), is the create-side complement ofcollapsed_count: the shortcuts the next apply must mint rather than update — sibling groups with no binding anywhere, unbound keyless rows, and every server ROM the local mirror holds no row for (rom_count − persisted rows, clamped at zero). It is platform-only — a collection's rows belong to their platform's unit, so counting creates on a collection too would price the same shortcuts twice — and likebound_countit is not stamp-gated: an unbound group genuinely has no shortcut and a ROM with no local row genuinely has to be created, whether or not the mirror is complete. The frontend takes it as its create term directly instead of deriving creates by subtractingbound_countfrom the unit's weight (see the composition-priced seed below). Hard constraint (ADR-0023): the prediction never feeds the actual skip decision —_try_unit_incremental_skipat fetch time remains the sole skip authority, so a mis-prediction can only make the estimate read long or short, never mis-apply. A Force Full Sync needs no special case:clear_sync_cachedeletes every stamp before the run, so a forced plan predicts no skips and drops everycollapsed_count— the unit is then weighed at the full pre-collapserom_count, butbound_countandnew_shortcut_countboth survive the clear (they read the rows, not the stamp) and keep the forced re-apply priced by composition (#1517). The same collapsed counts also garnishget_platforms(an optional per-platformcollapsed_count), so the platform toggles show the number of games a synced platform actually produces rather than the raw server file count. That garnish is gated on the platform's completion stamp (_read_collapsed_counts, #1412): the count is emitted only for slugs that currently carry aPlatformSyncStatestamp — the stamp exists iff the local mirror is complete, which is exactly when a post-collapse count is meaningful. A never-synced platform legitimately holds only PARTIAL rows (cross-platform collection siblings persist per ADR-0021), so an ungated count would shadow the true server total; with no stamp the field is absent and the toggle label falls back to the rawrom_count. Clearing the stamp (local removal / Force Full Sync) reverts the label to the server total until the next completed sync re-stamps. - Static walk-cost ceiling (pre-run seed). Before a run — in the preview, and again as the initial "up to X min" the
instant a skip-preview run starts — the estimate is a pure cost model over three independent terms (#1511),
because a run is three independent phases and one blended per-item rate cannot describe a mix of them:
created × NEW_ITEM_SEC+changed × UPDATED_ITEM_SEC+(created + cover_refresh_count) × COVER_DOWNLOAD_SEC+ a flat fixed-overhead allowance. Each constant is calibrated to its own measured mean with a ~15% ceiling margin (0.36 s per created shortcut against a measured 0.314; 0.13 s per update against 0.109; 0.15 s per cover download against 0.132 cold), and the allowance (45 s) covers the run's genuinely fixed cost — the one-time shortcut scan, the multi-page ROM/save fetches, the inter-chunk gaps, finalize (measured 17–24 s). So the seed still reads long, never short, but it now reads long by a margin rather than by a factor. Two consequences worth stating: an update carries no artwork cost (the apply loop gates cover application oncreated), and a cached cover is not a term — warm covers cost 0.0018 s, 73x cheaper than cold, and pricing them would need a backend cache probe in the preview path that is deliberately not added, so every create is priced as needing a cold download. - Composition-priced plan seed. The skip-preview seed prices the plan per unit by composition, not as one
blended item count (#1511): a predicted-skip unit costs nothing, the unit's already-bound ROMs (
bound_count) take the update rate, and the shortcuts that genuinely have to be minted (new_shortcut_count) take the create rate. Pricing every planned item as a create over-read by ~4x on the common case — any re-sync, and every Force Full Sync, which clears the completion stamps but unbinds nothing and is therefore an all-updates run. - Why the create term is read, not subtracted (#1517). The two rates are priced from independent counts; the
create term is
new_shortcut_countdirectly, neveritems − bound_count. The subtraction over-read a Force Full Sync of a sibling-heavy platform by ~2.5x: a forced run dropscollapsed_count, so the unit is weighed at its pre-collapserom_count, and on a platform carrying sibling groups (ADR-0021) that raw count exceeds the real shortcut count — one row per group is bound, its duplicates are not. Subtracting the bound rows priced every unbound duplicate as a phantom new shortcut, at the dear create rate plus a cover download it would never perform, even though a sibling duplicate never produces a shortcut.new_shortcut_countreports no creates there instead, because the clear takes the stamps and not the bindings, so no group is left without one. The symmetric hazard is a never- synced platform holding only PARTIAL collection-sibling rows (ADR-0021): counting creates from the known rows alone would price a handful of items for a whole platform — a short read, the one direction this estimate may not err in. Therom_count − persisted rowsterm prices those unmirrored ROMs as creates, so the seed stays at (or above) the full platform. A unit whosenew_shortcut_countis absent (collections, older backends) falls back to theitems − bound_countsubtraction, the pre-#1517 behaviour. - Collections still price by bound members. Platform and collection units are priced the same way on an ordinary
re-sync, so a collection-heavy library does not reinstate the over-read through its collections. A Force Full Sync
is the exception for collections:
clear_sync_cacheclearscollection_sync_statewholesale, and a collection's member set lives only in that stamp, so every collection unit is unstamped for that run and reverts to create pricing. Platforms are unaffected — theirbound_countandnew_shortcut_countare deliberately not stamp-gated. This follows directly from theNone-not-0rule above and errs long, the safe direction; correcting it would mean asserting membership the plan does not have. A unit whosebound_countis absent (older backend, unstamped or virtual collection) prices as all creates, the pre-#1511 behaviour. The live-countdown weights are unaffected and staypredicted_skip ? 0 : (collapsed_count ?? rom_count). - Measured live countdown (takes over within seconds). Once the apply is underway,
syncEta.tsmeasures the real rate from the applying frames — one throttled sample per second over a ~30 s sliding window — and projectsremaining = (planned_total − processed) / rate, rendered rounded up ("9 min left") so it never promises less time than it expects. It replaces the static seed as soon as the window spans enough real time to trust the slope (a couple of samples across a few seconds). Because it reflects the actual mix of cheap-update vs. full-create work, it is far closer to reality than the ceiling and ticks down as the run proceeds. - Estimator-owned sticky deadline. The estimator, not the UI, owns the displayed value: each fresh measurement
re-anchors an absolute wall-clock deadline, and the countdown renders
max(0, deadline − now). This is what keeps the readout smooth. The raw measurement re-arms to "not ready" between measurement segments, and a run's tail of small units each finishes inside the readiness window — so a raw snapshot would blink back to the static "up to X" seed for the whole tail; holding the last good deadline through those gaps keeps the countdown honestly ticking down instead. - Segment break across fetch gaps. Applying frames arrive roughly every second during real apply work, so a silence longer than ~10 s is always a unit or fetch boundary, never apply progress. Crossing it starts a fresh measurement segment (the prior samples are discarded), so the slope is never measured across the gap — pairing a pre-gap sample with a post-gap one would be a tiny item delta over a long span, an absurd rate that would briefly spike the countdown.
- Stalled-prefix trim (shorter stalls). A stall too short to break the segment used to poison the window head instead (#1511). The applying stage is entered — and its first frame emitted — before the one-time shortcut scan, which then blocks for ~9–14 s (it scales with the existing library, not the delta). That frame became sample #1 and paired with the first post-scan sample: a couple of items across the whole scan, a rate ~10x below the real one, which the sticky deadline then held until the 30 s window slid past — the reported "6 min → 2 min" collapse. Leading samples are therefore dropped when the head gap is both wide (> 3 s, several sampling cadences) and unproductive (fewer than one item per sampling interval, below anything the 50 ms-paced apply loop produces). Both conditions are required: a wide gap that carried real throughput is slow work, not a stall, and discarding it would measure only the fastest stretch. Trimming can legitimately leave a single sample — "no measurement yet" is the honest answer while a stall is all the window has seen, so the static seed keeps standing until clean samples accumulate.
The coarse QAM progress bar reuses the same run weights (weightedCoarseFraction, syncEta.ts): each unit's bar width
is its item-weight share — skip-aware at plan time, corrected to the real delta as units dispatch — with the within-unit
fill scaled by the running unit's share, so a predicted-skip platform occupies no width and a huge platform fills the
bar in proportion to its real work. It falls back to the old equal-per-unit index weighting when no plan is measured
(QAM opened mid-run before any sync_plan, an older backend) or the plan can't apportion (unit-count mismatch, all-zero
weights).
Two run-scoped latches keep the coarse fraction monotonic so the bar only ever moves forward. A run's leading
zero-weight units still refresh covers, so rather than pinning the bar to zero they each claim an equal 1/totalUnits
slice as a floor, with the weighted shares compressed into the band above it (#1506); that floor is held at its
high-water mark, so raising a mispredicted skip off zero can't shorten the prefix and retract it. On top of that, the
returned fraction itself is latched at the run's high-water mark by latchedCoarseFraction — the wrapper MainPage
actually calls, keeping weightedCoarseFraction a pure reader (#1509): when observeUnitTotal corrects a mispredicted
trailing skip up (0 → its real delta) it correctly grows the countdown's totalRoms (the run really is longer), but
that shrinks the bar's completed/total ratio and would retract width already shown, so the bar holds while the countdown
lengthens. Both latches touch only the bar output; the live ETA still reads the corrected total. The null fallbacks
(no run, unit-count mismatch, zero total weight) pass through the wrapper un-latched.
The within-unit fill is itself split into three monotonic sub-slices (#1407, withinUnitFraction in
src/utils/syncProgress.ts): fetch (FETCH_SHARE 15%) → covers (COVERS_SHARE 25%) → apply (APPLY_SHARE 60%). A
unit is worked in that order — paginate the ROM list, download/refresh cover art, then create the shortcuts — and each
phase fills its own slice by its own current/total, with a later phase's floor sitting at the sum of the earlier
phases' shares. So the bar advances continuously through a unit's fetch and cover phases instead of resting frozen at
the unit floor until applying, and it never jumps backwards at a phase boundary even though each phase restarts
current/total from zero (each phase's frames land in a strictly-higher band than the phase before). The phase is
tagged on the sync_progress payload's additive subStage field (camelCase, matching the sibling totalSteps /
runId keys — the emit_progress Python kwarg is sub_stage, the emitted key is subStage): the fetcher's per-page
frames carry subStage: "fetch", the artwork cover-refresh and download frames carry subStage: "covers" (both
share the covers slice), and the frontend-driven apply frames are keyed on the applying stage alone — so a merged
apply frame still carrying a stale subStage is unaffected. Within the covers slice the fill is not strictly
monotonic: the covers phase runs two sequential passes — the cover-cache refresh first, then the download for the apply
set — each counting current/total from zero, so a mixed unit (changed covers to refresh and new covers to
download) can dip backwards within the covers band at the refresh→download handover, bounded to the covers share of
that unit's slice. The fetch→covers→apply phase handovers themselves stay monotonic (the common cases don't dip either:
a first full sync refreshes nothing, and a steady incremental sync with no cover changes runs neither pass). A frame
with no sub-stage (the per-unit fetch anchor, or a pre-#1407 backend) rests at the unit floor, the old behaviour.
emit_progress writes subStage into both the emitted event and the persisted get_sync_status snapshot, so it rides
the QAM-remount re-seed path too.
The whole thing is an approximation by design — though a narrow one since the plan went skip-aware: seed weights and the
applying frames usually both count post-collapse shortcuts now, the raw pre-collapse rom_count survives only as the
fallback where the backend doesn't know better, and a unit the plan mis-predicted re-corrects on its first
sync_apply_unit (observeUnitTotal). Estimate degradation is strictly one-directional: a wrong prediction or a stale
collapsed count makes the readout run long or short, never changes what the sync applies. The readout wording ("up to",
rounded-up countdown) keeps it an estimate, never a guarantee.
Save-sync serialization (device gate)¶
Every save-sync run on the device passes through a single device-level serialization gate (SaveSyncGate, in
services/saves/sync_engine/_gate.py). Only one save-sync run is in flight at a time per device: when a second
trigger fires while one is running, it queues — it waits for the in-flight run to finish rather than running
alongside it. An asyncio.Lock is the serializer/queue; the gate owns only the bounded-acquire discipline around it (no
run-lifecycle state, no run ids, no cancellation — those are out of scope here).
The wait is bounded so a stuck run never traps the launch path. Each of the four trigger methods on SyncEngine
wraps its run body in bounded_run(timeout=…) with a per-trigger budget — pre_launch_sync (30 s), post_exit_sync
(60 s), sync_rom_saves (15 s), sync_all_saves (60 s). If the gate can't be acquired within the budget the call
returns its own fallthrough instead of blocking, and all four fallthroughs carry the same busy shape:
reason: "sync_busy", no additive offline flag. A busy gate is a local wait — nothing on that path observed the
server — so it never borrows a server-reachability slug and never sets the offline flag (#1625); the session-end toast
would otherwise announce "Server offline" about a reachable server. Each caller routes the skip on success: False
alone: the launch gate maps it to sync_failed (fallback-launch confirm, so Play is never trapped), the session-end
toast keys on the reason. The lock is never leaked on timeout — a timed-out acquire releases any photo-finish hold
before raising.
The gate sits outside the per-ROM lock (SyncEngine.rom_lock(rom_id)): the device gate admits one run at a time
across the whole device, then each ROM still takes its own rom_lock for the read-mutate-write of its
RomSaveSyncState. The cheap stateless early-out (is_save_sync_enabled()) runs before the gate, so a
disabled-feature call never queues behind an in-flight run just to report it's off.
DownloadService notes¶
RomM exposes three mutually exclusive file-layout flags on every ROM detail. They control how the server stores files and how the API serves them. The plugin maps each layout to a local on-disk path:
| RomM flag | RomM server layout | What fs_name is |
Plugin local layout |
|---|---|---|---|
has_simple_single_file |
roms/<platform>/<file> — one file, flat |
the filename | flat in platform folder: roms/<platform>/<file> |
has_nested_single_file |
roms/<platform>/<folder>/<file> — one file in a per-game folder |
the folder name | flat in platform folder: roms/<platform>/<file> |
has_multiple_files |
per-game folder with multiple files (multi-disc, BIN+CUE, etc.) | the ZIP/folder name | extracted into per-game subfolder named after the launch file: roms/<platform>/<launch-file-name>/... |
has_nested_single_file quirk: fs_name is the parent folder name, not the filename. The actual filename with
extension lives in files[0].file_name. The plugin reads from files[0].file_name so the downloaded ROM lands with the
correct extension (e.g. Game.chd, not the extension-less folder name Game). A defensive helper falls back to
fs_name and warns if files is empty or missing.
Why nested-single is flattened locally: a nested-single-file ROM has no sidecars by definition — RomM would mark it
has_multiple_files if any companion files existed. The parent folder adds no value at the local layer, so the plugin
drops it and stores the ROM directly in the platform folder, matching the simple-single-file layout. Multi-file ROMs
keep their per-game subfolder because they contain multiple related files that belong together.
Extract-vs-flat gate keys on len(files) > 1, not on has_multiple_files: the plugin decides ZIP-extract vs
single-file download with the is_multi_file_download helper (domain/rom_files.py), which returns
len(files) > 1 OR has_multiple_files. This mirrors RomM's own download gate, which zips whenever the total file
count is not exactly 1. RomM computes has_multiple_files from top-level files only, so the two counts disagree for
a nested layout: a canonical Switch game (base file at the root plus update/ and dlc/ in subfolders) has exactly one
top-level file (has_multiple_files=False, has_nested_single_file=True) yet more than one total file, so RomM serves
a ZIP. Keying on has_multiple_files alone would take the single-file path and write the ZIP bytes verbatim into one
unreadable .nsp. The boolean is kept as a defensive fallback for payloads that omit files; a genuine nested-single
ROM has len(files) == 1 and correctly stays on the flat single-file path.
ES-DE directory-collapse rename: a multi-file ROM is extracted into a staging folder named after the ROM's
identity — fs_name_no_ext, falling back to splitext(fs_name) (resolve_extract_dir_name in domain/rom_files.py)
— never after files[0]. That distinction matters for a has_nested_single_file folder game served as a ZIP (a PS3
title whose first listed file is an arbitrary inner asset like a music file): resolve_local_file_name returns that
asset's name for the local filename, but the extract directory must carry the game's identity or the whole install —
launch file_path, rom_dir, and the folder-boot launch bake — inherits the wrong name. The staging folder is then
renamed after the detected launch file including its extension (e.g. Final Fantasy VII (USA).m3u/ containing
Final Fantasy VII (USA).m3u). ES-DE only collapses a directory into a single game entry when the folder name matches
the launch file's full name with extension; without the rename a multi-disc game shows in ES-DE as a folder plus loose
disc files. The launch file is only known after extraction (an .m3u may be auto-generated — see below), so the rename
happens last, after launch-file detection, via es_de_collapse_rename (domain/rom_files.py) + the
DownloadFileStore.move_dir whole-directory move. On a name collision (target already exists) the rename is skipped and
the staging folder is kept — never clobbered or merged. Existing installs from before this feature keep their old folder
layout until re-downloaded.
Folder-boot launch target (PS3): for a PS3 folder game detect_launch_file picks the nested
…/PS3_GAME/USRDIR/EBOOT.BIN as the install's file_path — the correct launch file identity (save path, core, and
displayed filename all derive from it). But RPCS3's directory-boot wants the game folder, not the EBOOT, so the
baked launch_options carries the game directory instead. This is a bake-time path override (folder_boot_root,
domain/rom_files.py) applied in the DiscLaunchResolver seam, never a file_path rewrite: file_path stays the
EBOOT anchor while only the argument baked into the shortcut becomes the folder
(ADR-0019, see
Core and Emulator Selection).
Folder-boot launch target (PS3): for a PS3 folder game detect_launch_file picks the nested
…/PS3_GAME/USRDIR/EBOOT.BIN as the install's file_path — the correct launch file identity (save path, core, and
displayed filename all derive from it). But RPCS3's directory-boot wants the game folder, not the EBOOT, and
RetroDECK's run_game.sh reinterprets a directory %ROM% as a "directory as a file" (run_game.sh:63-67) so it can
never launch a bare folder. Two overrides fire together, both keyed on the same folder_boot_root fact and neither a
file_path rewrite: the baked path becomes the game folder (folder_boot_root, domain/rom_files.py, in the
DiscLaunchResolver seam), and the baked invocation becomes a direct sandbox command that bypasses
run_game.sh — flatpak run --command=<launcher> net.retrodeck.retrodeck <args> "<folder>", resolved in
ActiveCoreResolver.active_emulator_for_rom (standalone + folder-boot install → CoreResolver.resolve_sandbox_launcher
gives the /app/…/component_launcher.sh sandbox path → EmulatorInvocation.direct). file_path stays the EBOOT anchor
(ADR-0019, see
Core and Emulator Selection). The multi-file download path also
suppresses M3U generation and heals a .txt-suffixed PS3_DISC.SFB for a folder-boot dump (domain
folder_boot_layout_root + DownloadService._maybe_heal_ps3_sfb_io).
M3U generation rule (needs_m3u in domain/rom_files.py): a game-named <fs_name_no_ext>.m3u is auto-generated
(when no .m3u already exists) for multi-disc ROMs — two or more disc files of any kind (.cue/.chd/.iso) — so
the emulator can switch discs, and for single-disc bin/cue ROMs — exactly one .cue — so the extract dir is
renamed after a game-named playlist rather than a generically-named cue (disc1.cue/). Single-disc .chd/.iso arrive
as single-file downloads that never reach the extraction path, so they get no playlist.
M3U is platform-gated on ES-DE's own extension list (ADR-0013).
The file-count rule above only runs when the ROM's system actually supports .m3u. RomM bundles a platform-blind .m3u
into the ZIP for every multi-file game, including cartridge systems (Switch .nsp, Xbox 360 .iso) whose emulators
have no playlist concept — so an extension-only heuristic wrongly produced a <Game>.m3u/ folder that never collapsed.
The plugin now asks whether ES-DE lists .m3u as a supported extension for that system, read from the same
es_systems.xml ES-DE uses to decide directory-collapse, via CoreResolver.system_supports_m3u(system) exposed through
the SystemM3uSupportFn Protocol (services/protocols/) and threaded into DownloadService from bootstrap. When the
answer is False, no .m3u is generated and the bundled one is never chosen as the launch file
(detect_launch_file skips its .m3u preference), so selection falls through to the real game file and the folder is
named <Game>.nsp/ / <Game>.iso/ instead. The capability crosses the service/domain seam as a plain bool — the
domain functions (needs_m3u, detect_launch_file) take m3u_supported, never a system name or an adapter. The
bundled .m3u is left inert on disk, never deleted. When es_systems.xml cannot be found the answer defaults to
False (safe: a missing playlist only degrades disc-switching, a wrong one breaks the launch).
Launch-target validation¶
detect_launch_file ends in "largest file by size". When none of its format-specific rules match, whatever happens to
be biggest becomes the launch target, is written into RomInstall.file_path, and becomes the shortcut's launch command
— with nothing checking whether the system can act on it. A PS3 title distributed as .pkg + .rap is the reported
case (#1582): a PKG is an installer, the game is still
sealed inside it, no EBOOT.BIN exists anywhere in the download, so every rule misses and the multi-gigabyte package is
baked. The download reports success and the failure only surfaces when the user presses play.
The verdict is decided once, at record time. _record_install_io — the single seam both the single-file and
multi-file download paths pass through — calls is_launchable_target (domain/rom_files.py) and records the answer on
RomInstall.launchable. The check reads the target system's live ES-DE accept-list through the
SystemSupportedExtensionsFn Protocol, the same seam DiscLaunchResolver intersects the disc set with. Two cases pass
without consulting it:
- An empty accept-list — the source could not answer (unknown system, no ES-DE installation). A missing answer must never turn a working install into an unlaunchable one, so it accepts.
- A folder-boot layout (
FOLDER_BOOT_MARKERS) — the baked target is the game directory, not the nestedEBOOT.BINthatfile_pathrecords, and ES-DE spells the directory case.ps3dir. The marker match is positive evidence that the plugin recognised the layout, so no extension is examined. Without this carve-out every working PS3 dump would be rejected:file_pathends in.bin, which ps3's accept-list (.desktop .iso .ps3 .ps3dir) does not carry.
Everything else is decided by the recorded launch file's extension. .pkg is absent from ps3's list; a bare track
.bin is absent from dreamcast's (.cdi .chd .cue .dat .elf .gdi .iso .lst .m3u .7z .zip), which is why a multi-file
GDI rip without a .cue is caught by the same rule.
The files are always kept. An unlaunchable download is a real install: the row is written, the ROM is uninstallable through the normal path, and nothing is deleted. Refusing the install would discard a package whose remaining use is exactly to be installed by hand in the emulator — the documented RetroDECK procedure for PSN titles — leaving the user worse off than the silent failure this replaces. What is withheld is only the launch command.
One seam withholds it everywhere. DiscLaunchResolver.resolve_bake_path returns "" for an install with
launchable is False, before any disc work, and build_launch_options renders an empty path as the empty launch
command. Every launch-bake site already draws its path from that resolver — library sync's installed_paths map,
download-complete's re-bake, the core-change re-bake, the startup relaunch-options heal, the disc picker — so none of
them can compose a command for content nothing can boot, and none needed a guard of its own. An unlaunchable install
stays in the installed_paths map (it is downloaded, and collapse_sibling_groups reads the key set to choose a
sibling group's representative) and maps to the empty path.
Empty is the established uninstalled state, not one invented here: addShortcut leaves a new shortcut's options
untouched when the command is "" (src/utils/steamShortcuts.ts), the sync update/adoption path writes "" explicitly
(rewriteShortcutIdentity, src/utils/syncManager.ts), and an uninstall records "" as the ROM's
applied_launch_options (services/rom_removal.py). An empty launch command therefore means two things, and nothing
may infer which: "not downloaded" and "downloaded but not launchable" are indistinguishable at the Steam-shortcut
layer — the shortcut holds "" either way. Only the data layer separates them (no rom_installs row versus a row with
launchable = 0), so a consumer that needs to tell them apart must ask the install record, never read emptiness as "not
downloaded". The rule is restated on build_launch_options itself, where a later reader will hit it.
The frontend closes the loop in two places: the shared launch gate blocks with no_launch_target before any save-sync
work — for both the Play button and the global launch watcher — and the game-detail page's ROM File section states
that the download has no launchable format and that the files are on disk. Re-checking a recorded verdict once the user
has installed the package in the emulator is separate work
(#1654).
Where the knowledge comes from. The accept-list is read from the plugin's own es_systems.xml parser
(CoreResolver.get_supported_extensions), wired in as DownloadServiceConfig.system_extensions. That parser duplicates
one emu-atlas already has, and the resolver adoption is meant to remove it — so the consuming call site is deliberately
a single self._system_extensions(system) behind one Protocol-typed config field. Swapping the source is a change to
that wiring line.
Filesystem writes go through DownloadFileAdapter. ZIP extraction is ZIP-slip protected and streamed: extract_zip
copies each member in chunks and reports byte progress through an optional callback, so a multi-file ROM emits
download_progress frames with status: "extracting" (bytes_downloaded/total_bytes over the uncompressed
total, resumable: false) after the transfer hits 100%. The frontend reuses the same event — no new event name — to
switch the download button and QAM queue into the non-cancellable Extracting… phase. Single-file downloads never
emit it.
Bounded concurrency + reserved-bytes pre-flight: at most two ROMs transfer at once, gated by an
asyncio.Semaphore(2) around the transfer + post-IO critical section. start_download enters the queue with status
queued and reserves the download's required bytes in _reserved_bytes[rom_id]; _do_download flips the status to
downloading only once it acquires the semaphore (emitting a download_progress status: "queued" frame first if it
has to wait), and releases the reservation in its finally. The disk pre-flight accounts for siblings' outstanding
reservations (free_space - sum(reserved) < required) so two concurrent downloads that each fit alone but not together
can't both pass — the second is rejected with an insufficient_space failure.
Cancel reaches the UI and never destroys a live install: cancelling a download emits a terminal download_progress
status: "cancelled" frame so the frontend resets the button out of its downloading state (the cancel path used to be
silent). Because executor threads run to completion regardless of cancellation, a cancel that loses the race to a
just-committed install is reconciled (_reconcile_post_io awaits the in-flight post-IO future): if the install
committed, the download is surfaced as completed (launch options baked, download_complete emitted) rather than
torn down. _cleanup_partial_download removes only the transient transfer artifacts (.zip.tmp / .tmp) and, for
a multi-file ROM that did not commit, the extract dir(s) this download created — it never deletes the bare
target_path, so a re-download that fails mid-stream (or a cancel that lost the race) cannot destroy a pre-existing or
just-committed install.
Sibling supersede — one downloaded version per group
(ADR-0021
§4 amendment, #1298): before a download begins, start_download (and resume_download) strips any other installed
member of the ROM's sibling group so a game keeps at most one copy on disk. _conflicting_sibling_install_ids reads the
group in one short UoW (indexed iter_by_group_key) and returns members that are installed and either unbound or
bound to the same shortcut — a member bound to a different shortcut (a grandfathered duplicate, ADR-0021 §5) is
exempt and never removed. Each superseded install goes through the canonical RomRemovalService.remove_rom (files +
rom_installs row; saves untouched per ADR-0007), the membership read UoW closed first so the removal's own UoW
does not nest (ADR-0006); a not_installed result raced clean and is skipped, any other removal failure aborts the
download so the invariant stays honest. A superseded sibling's paused queue entry is evicted (_evict_if_paused) so a
stale .tmp can't later resume into a second install. resume_download additionally refuses (superseded, entry
dropped) when a switch has moved the group's binding to a different member since the pause — it won't re-download a
version the picker has already left. The in-progress claim is taken before the supersede await and released on every
early exit so a concurrent start_download for the same rom is rejected rather than racing past it.
ConnectionService notes¶
The minimum-version gate is SemVer-aware on SYSTEM.VERSION. domain.version.meets_min_version compares the
numeric core against _MIN_REQUIRED_VERSION; when the core equals the floor, a -alpha / -beta suffix
(case-insensitive, optional .N build number) ranks below the release and is rejected — so 4.9.0-beta.3 fails at
floor 4.9.0 while 4.9.1-beta passes. development and a missing version bypass the gate.
A Client API Token is bound to the server it was minted against. When the token is minted, the canonical origin of
romm_url (full scheme://host[:port], default ports folded out, path/query dropped — lib/url_host.normalize_origin)
is stored alongside it as romm_api_token_origin. RommHttpAdapter.auth_header() attaches the bearer only when
that stored origin matches the current romm_url origin; on a mismatch it raises TokenHostMismatchError instead of
sending the credential to a host the token was not minted for. The error is non-retryable and maps to a config_error
failure (Your saved RomM login is for a different server. Sign in again to continue.), so every data flow fails fast
until the user re-signs-in. https://h and http://h are deliberately different origins — a plaintext downgrade is
a different destination, not the same one. A legacy token minted before origin stamping carries
romm_api_token_origin =
None and is treated as un-bound: it is still attached (never blocked) so existing installs
keep working until their next sign-in stamps the origin.
Sign-in ordering: validate → probe → mint → persist (one atomic save). establish_token trims the entered URL and
rejects a non-http(s) value before any network call. It then holds the candidate URL in memory only — clearing the
stored token in memory first so the version probe never carries the old server's bearer to the candidate host — and
persists nothing until the mint succeeds. On any failure (probe unreachable, version too old, forbidden/error mint, no
usable token, or a disk error) the in-memory auth state is rolled back to the previous working URL + token, and because
disk was never touched the prior working credentials survive a failed sign-in. Only a successful mint commits
romm_url + SSL flag + token + id + origin to disk in a single save_settings() call.
The old-token DELETE is origin-guarded and provenance-guarded. RomM scopes a Client API Token to the account, and
re-auth deletes the device's previous token. That DELETE is only fired when the old token's stored origin matches the
new URL's origin (same-server re-auth) — replaying it against a different server would delete an unrelated token there,
so the DELETE is skipped (and logged) when the origins differ or the old origin is unknown. It is also skipped
whenever the previous token's romm_api_token_source is "user": a token the user pasted belongs to the user, not to
this device, and must never be revoked by the plugin (a user token also carries no romm_api_token_id, so there is
nothing to DELETE by anyway — the source check states the intent). The DELETE uses Basic auth from the one-time
credentials, unaffected by the cleared bearer.
A token's provenance is recorded in romm_api_token_source. The value is "minted" for a token the plugin minted
from a username/password (establish_token and the startup migrate_legacy_credentials) and "user" for a token the
user pasted (establish_user_token). It is part of the snapshot/restore auth-state set, so a failed sign-in rolls it
back with the rest of the auth state. Settings migration v9 → v10 seeds it — "minted" when a token already exists (a
pre-v10 install could only hold minted tokens), else None.
Pasted-token sign-in (establish_user_token) — the OIDC path. OIDC / SSO accounts have no password to mint from, so
the user creates a Client API Token in RomM's web UI and pastes it. establish_user_token mirrors establish_token's
validate → probe → gate → validate → persist-on-success-only shape, but the credential is the pasted token rather than a
fresh mint, so there is no mint and no server-side DELETE of any prior token. The entered URL is trimmed and
rejected if non-http(s); a blank/whitespace token is rejected as config_error before any network call. The candidate
URL and the pasted token are held in memory only, with romm_api_token_origin stamped to the candidate origin and
romm_api_token_source = "user", so the auth-header guard attaches this token (not the old server's bearer) to the
validation probe. Validation is an authenticated GET /api/users/me: a 401 means the token is invalid or revoked, a 403
means it authenticates but lacks a required scope (the plugin cannot introspect a token's granted scopes, since
/api/users/me's oauth_scopes reflects the user's role, not the token's grants, so this connect-time authenticated
probe is the only validation). Both map to the canonical auth_failed failure with a token-specific message. On any
failure the in-memory auth state is rolled back and disk is never touched; only a successful validation commits URL +
SSL flag + token + id = None + origin + source in a single save_settings(). The token value is never logged. The
device-forget-on-origin-change and playtime-scope-notice clear run on the success path exactly as they do for
establish_token. The /api/users/me validation + persist tail is shared with the paired-token path below via the
private _validate_and_persist_user_token helper (both hand the plugin a token to validate and store with "user"
provenance), so their observable behaviour stays identical.
Pairing-code sign-in (establish_paired_token) — the OIDC path without pasting. The same OIDC accounts can sign in
by entering a short-lived RomM pairing code instead of pasting the token: the plugin exchanges the code for the token
over the public, unauthenticated POST /api/client-tokens/exchange endpoint (the one-time code is itself the
credential). It mirrors establish_user_token's validate URL → probe version → gate → obtain-credential → validate via
/api/users/me → persist-on-success-only shape, but the credential is fetched by the exchange rather than pasted, and
the exchange runs with the token trio cleared in memory (like establish_token) so no old bearer leaks to the
candidate host during the unauthenticated call. The code is normalized the way RomM normalizes it — all whitespace and
- stripped, then uppercased ("ab-cd ef23" → "ABCDEF23") — and a code that is blank after normalization is a
config_error before any network call. The exchange is never auto-retried: a pairing code is single-use, and a
replay would burn both the code and RomM's per-client rate limit, so the transport's unauthenticated_post_json skips
with_retry. Each RomM rejection maps to a distinct, actionable message: an invalid/expired/used code, a
token-no-longer-exists 404, and a disabled-owner 403 all carry auth_failed; a 429 carries the bespoke rate_limited
reason. On success the freshly rotated raw_token (the exchange regenerates the token server-side, so any previously
copied raw value stops working) is host-bound in memory and run through the shared _validate_and_persist_user_token
tail, so it is persisted with "user" provenance and id = None exactly like a pasted token — no mint, no server-side
DELETE. The pairing code and the returned token are never logged.
The registered device id is forgotten only on a genuine origin change. A device registered with RomM
(POST /api/devices, its id stored in kv_config["device_id"]) is bound to the server it was registered against —
RomM's negotiate save-sync transport hard-404s a foreign device id. So a successful sign-in that genuinely changes
origin forgets the stored device id (SaveService.forget_device → DeviceRegistry.forget_device, wired as the
DeviceForgetFn injected into ConnectionService), so the next save-sync run re-registers against the new server. The
decision is is_origin_change(old_token_origin, new_url) in lib/url_host — both origins are normalized and the id is
forgotten only when both are known (parseable) and differ. This is deliberately the opposite failure posture to
the old-token DELETE guard (same_origin, which fails closed): a same-server re-sign-in keeps the device identity,
including a token swap on the unchanged URL, URL-formatting variants, and a None/unstamped old origin (an unknown old
origin is treated as unknown, not different — never a change). Without this, a same-server token swap dropped the
identity and the next post-exit sync flagged a spurious conflict for a save this device itself synced (#1437). The
old_token_origin comparison input is captured from the pre-clear auth-state snapshot, so the leak-safety in-memory
token clear (which zeroes the origin) never poisons the comparison. The forget is local-only (the row on the old
server is left for RomM's machine-id dedup to reconcile on a later re-registration) and best-effort on the success
path — a failed local clear never turns a good sign-in into a failure, and a failed sign-in (in-memory snapshot
restored) keeps the still-current server's id. ensure_device_registered never deletes the id on any failure, so a
permission-degraded (403/timeout) session leaves the identity intact. This is Phase 0a of the RomM Device Sync negotiate
adoption (ADR-0016).
The signed-in user's own id (romm_user_id) is bound to the token — stamped at sign-in, backfilled lazily, cleared on
sign-out. It drives the collection owner-scope filter (build_work_queue + get_collections, described in the
LibraryService section above). Every sign-in path re-derives it from the freshly authenticated token so it can never
linger for a different user or a different server: the mint path (establish_token) probes GET /api/users/me after
host-binding the minted token (the mint response carries only the token id, not the user id); the pasted/paired paths
reuse the id from the /api/users/me validation probe they already run (no second call). In every case the id is set
in memory before the token-persist save, so it rides the same single atomic save_settings() — the sign-in write
shape is unchanged. It is part of the snapshot/restore auth-state set, so a failed sign-in restores the previous id, and
it is cleared in each path's in-memory auth clear so a probe failure or a malformed payload leaves it None rather than
stale — degrading the "Mine" filter to "All" until the next backfill. Existing installs (a valid token minted before
this setting existed) carry no id; test_connection lazily backfills it — when a token is present and the id is
missing, the successful connection check probes /api/users/me and persists the id (its own save), so the filter
activates without a re-login. A known id needs no network on later checks. Every identity read is best-effort: a failure
never fails the sign-in or the connection check.
The no-sign-in URL change path (SettingsService.save_server_url) deliberately does not touch the token, so pointing
the URL at a different origin leaves the stored token's origin mismatched and the auth-header guard makes subsequent
data flows fail fast with config_error until the user signs in again. Because every data flow (including device
registration) is inert until that sign-in, a stale device id from this path cannot be used before the sign-in clears it.
Sign-out (sign_out) is a local forget — never a server-side delete. sign_out clears the token quad
(romm_api_token / romm_api_token_id / romm_api_token_origin / romm_api_token_source) in the in-memory settings
dict and persists them in a single save_settings() (the same atomic-write funnel _persist_token uses), keeping
romm_url and the SSL flag so the user need not re-enter them. It mirrors the sign-in paths' persist discipline: the
auth state is snapshotted first, and a failed save rolls the in-memory quad back to the snapshot and returns the
canonical failure shape (via error_response), so a disk error never strands the user with a half-forgotten but
still-valid token. Only on a successful save does it drop the cached RomM server version (set_version(None)) so no
stale value lingers. It is synchronous — no run_in_executor, no network — and idempotent (signing out when already
signed out still returns success). It never issues the server-side token DELETE: a minted token deliberately lacks
me.write (deleting it would require re-entering the password, which sign-out does not have), and a user-supplied token
belongs to the user. That is why the direct re-authentication path stays as Sign in again (not sign-out-then-in):
establish_token's same-origin minted-token cleanup (#1038) only fires while the old token id is still stored, so
signing out first would strand the old minted token on RomM (which caps tokens per user). After sign-out,
test_connection returns the canonical "Not signed in" config_error because romm_url survives but the token is
gone. Signing out (like re-signing-in) while a sync or download is in flight needs no guard: once the token is gone the
in-flight operation simply fails authentication and surfaces its normal error — deliberate, not a race to defend.
Server-supplied paths are validated, fail-stop on traversal: every server-supplied path component — the firmware
file_name, the ROM platform slug, and post-extraction URL-decoded ZIP member names — is checked through
lib/path_safety (safe_join for realpath containment, safe_path_component for single-component names) before any
write. A traversal attempt (../, an absolute path, or a %2e%2e%2f-encoded ZIP member that decodes to ../ after the
pre-decode ZIP-slip check passes) aborts the whole download rather than skipping the offending entry:
already-extracted members are cleaned up (no half-installed ROM), a canonical
{"success": false, "reason": "path_traversal", "message": ...} failure is returned, and the download_failed event
fires so the UI doesn't hang on "downloading". Firmware downloads surface the same canonical failure from
download_firmware.
StartupHealingService notes¶
Beyond the disk-prune and orphaned-SyncRun reconciliation, this service owns the startup launch-options reconcile
(#1043). launch_options (the full Steam-shortcut launch command) is written only event-driven — at sync, at
download-complete, and on RetroDECK-home migration (ADR-0009) — so any path that misses its bake leaves an installed
shortcut stuck on the "" placeholder, and bin/rom-launcher then runs with no args and exits non-zero. There was no
backstop short of a Force Full Sync or uninstall/reinstall.
get_installed_relaunch_options() is the read half of the fix: a 0-arg read that returns [{app_id, launch_options}]
for every ROM that is both installed (has a rom_installs row) and bound (its roms.shortcut_app_id is set).
It snapshots the install/ROM rows in one short read UoW, then re-bakes each command outside that UoW through the
same active_core / disc_resolver seams every other bake site uses — resolving inside the iteration UoW would
deadlock, since ActiveCoreResolver.active_core_for_rom opens its own UoW (the per-connection write lock is not
re-entrant). Uninstalled and unbound ROMs are skipped by construction. The callable is read-only and not
migration-gated.
The frontend pulls this on mount, once the backend is proven reachable (it reuses the app-id/metadata init's
retry/backoff, not a second loop), and confirm-sets each entry via the existing setLaunchOptionsConfirmed
fire-then-poll. The pass is idempotent and appId-safe: re-confirming a correct command matches the read-back
instantly, and a launch_options write does not change the shortcut's appId, so artwork, collections, and the shortcut
identity survive. This heals drift from every cause at the next plugin load.
Adapters (py_modules/adapters/)¶
Adapters own all I/O and implement the Protocols defined in services/protocols/. Selected adapters:
| Module | Role |
|---|---|
romm/http.py |
RommHttpAdapter — HTTP transport: auth, SSL, retry, User-Agent, platform map |
romm/romm_api.py |
RommApiAdapter — RomM REST surface (saves, ROMs, platforms, firmware, devices, play-sessions) over the HTTP transport |
steam_config.py |
SteamConfigAdapter — Steam VDF read/write, grid dir, shortcut icon write, Steam Input config |
steamgriddb.py |
SteamGridDbAdapter — SteamGridDB REST client |
sgdb_artwork_cache.py |
SgdbArtworkCacheAdapter — on-disk SGDB artwork cache |
cover_art_file_store.py |
CoverArtFileStoreAdapter — RomM cover art I/O across the per-ROM cover cache and the Steam grid dir (download, copy_file publish/seed, read, prune) |
persistence.py |
PersistenceAdapter + per-domain persister adapters — settings.json read/write plus the one-time legacy save_sync_state.json read that feeds the bootstrap settings fold |
repositories/ |
SqliteUnitOfWork + per-aggregate repository adapters — SQLite I/O (the live persistence path; see Database Design) |
sqlite_migrations.py |
apply_migrations — schema migration runner (db/migrations/NNN_*.sql, PRAGMA user_version) |
download_file.py |
DownloadFileAdapter — download filesystem |
firmware_file.py / migration_file.py / rom_files.py / save_file.py |
per-subtree filesystem adapters (BIOS, RetroDECK migration, ROM removal, local saves) |
retrodeck_paths.py |
RetroDeckPathsAdapter — reads retrodeck.json for ROMs/saves/BIOS/home paths |
recovery_bundle.py |
RecoveryBundleAdapter — safe recovery-root derivation, exact source measurement, verified copies, checksums/human manifests, fsync, staging cleanup, and atomic bundle sealing |
prune_artifacts.py |
PruneArtifactAdapter — per-ROM cover/validator and SteamGridDB cache discovery/removal |
steam_recovery.py |
SteamRecoveryAdapter — per-shortcut Steam grid, Steam Input, and controller-setting recovery inventory plus post-confirmation cleanup |
retroarch_config.py |
RetroArchConfigAdapter — reads retroarch.cfg save-sort flags |
retroarch_core_info.py |
RetroArchCoreInfoAdapter — reads RetroArch .info files (corename, metadata) |
es_de_config.py |
CoreResolver — ES-DE es_systems.xml (system-layer default core + available cores); the gamelist is no longer read or written |
gavel_native.py |
GavelNativeAdapter — loads the compiled romm-gavel core (py_modules/native/libgavel-x86_64-linux.so) via ctypes; is itself the ResolveUploadConflictFn seam and provides the ComputeSyncActionFn seam, the two save-sync decisions (no Python fallback) |
system_clock.py / system_uuid_gen.py / asyncio_sleeper.py |
concrete Clock / UuidGen / Sleeper seams |
hostname.py / path_probe.py / plugin_metadata.py / debug_logger.py |
hostname, path-exists probe, package.json name/version reader, settings-aware debug logger |
renderer_rss.py / renderer_gc.py |
RendererRssFn — max steamwebhelper VmRSS from /proc; RendererGcFn (HeapProfiler.collectGarbage) over the CEF debugger. The session-budget measure + settle seams (ADR-0024). The "free memory" action is a frontend SteamClient.User.StartRestart, not a backend adapter |
game_process.py |
GameProcessControl — resolves a flatpak app's live instances via the per-user registry (info / bwrapinfo.json) plus the /proc child walk, reporting each tree's PIDs and argv separately, and signals them. Direct reads + os.kill, no subprocess; fail-soft on every read |
RommHttpAdapter notes: what makes a 404 an entity verdict¶
RommNotFoundError means "RomM's entity layer says this entity does not exist", and downstream that reading is
authority: the removed-game cleanup deletes on it, the version picker marks a version vanished on it, save-sync drops a
stale device registration on it. A bare HTTP 404 does not carry that meaning on its own — FastAPI answers a
misconfigured path prefix with the same status and a generic {"detail": "Not Found"} body, and a reverse proxy in
front of RomM (Cloudflare Tunnel, Traefik) answers a misroute with an HTML or empty one. So on the API routes the
adapter raises RommNotFoundError only for a response that proves it came from RomM's entity layer: a JSON content type
whose body parses to an object carrying a detail string that is neither blank nor FastAPI's stock Not Found (matched
case-insensitively — a real entity answer always names the entity, so it is never the bare phrase). The requested id is
deliberately not parsed back out of that detail — its wording moves between RomM releases while the generic default
stays put, so blocklisting the default is the robust test. Every other 404 shape degrades to a plain RommApiError,
which classify_error maps to server_unreachable, so infrastructure can no longer authorize a deletion and each
caller fails open on it instead.
The byte-stream routes — download, download_conditional, download_external — opt out via
translate_http_error(..., asset_route=True) and keep the plain status mapping. Their 404 answers about a file, not
an entity, so it is never deletion-authority grade; and it has to keep raising RommNotFoundError, because RomM serves
its cover resources from a static mount where a genuinely missing cover answers with exactly that generic route-404
body, and the sole consumer reacts by refetching from the ROM's url_cover (the fallback above) — non-destructive and
self-correcting rather than a deletion. Each of the three call sites carries that constraint as a comment: a future
unification of the two classes would silently disarm the fallback, so both sides are pinned by tests.
Every deletion-authority probe reaches the network through the JSON-API entry points, never a byte-stream one:
get_rom / get_rom_once (request / request_once) behind the version picker's liveness and vanished probes,
list_saves (request) behind the save-status, copies and slot-setup reads, and update_device (put_json) behind
the device re-registration. The one consumer that branches on a byte-stream 404 is the url_cover fallback.
The entity-answer shape is captured from RomM 5.1.0; the supported floor is 4.9.0. If a 4.9.x entity-404 ever carries a
different body shape (no JSON content type, no detail string), every entity-404 on that server degrades to
server_unreachable: the cleanup never confirms a ROM gone, no version is marked vanished, no stale device registration
is dropped. That is the fail-open direction by design — if a 4.9.x user reports exactly that symptom set, this paragraph
is the explanation, and the fix is a version-aware entity check, never a return to trusting the bare status.
PersistenceAdapter notes¶
- File locking: write methods acquire an exclusive
fcntl.flockbefore touching the file, preventing concurrent writes from corrupting state. - Schema versioning: every state file written includes a
versionfield. On read, a mismatch causes the file to be treated as absent (cache discarded, state reset to defaults) rather than loading incompatible data. - Crash-safe atomic writes:
settings.jsonis written with the durable write-tmp →fsync(tmp)→os.replace()→fsync(dir)recipe. The temp file's bytes are forced to disk before the rename, and the directory entry the rename creates is forced to disk after it. This closes the power-loss window on the Steam Deck's ext4: without the fsyncs, a crash after the rename but before the kernel flushed could leave a truncated or emptysettings.json— which boot rewrites every run, so the window recurred. The directory fsync is best-effort: on the rare filesystem that rejects it, the error is logged at debug and swallowed (the content is already durable via the temp-file fsync). - Corrupt-file quarantine (never silently factory-reset): a
FileNotFoundErroron read is a legitimate first run — defaults are returned silently, no backup, no flag. An unparseablesettings.json(aJSONDecodeError, e.g. a truncated file from a prior crash) is the data-loss hazard: returning defaults silently would let the immediate bootstrap save overwrite the corrupt file, wiping the user's RomM URL, API token, SGDB key, and platform/collection selections with no trace. Instead the adapter logs the corruption loudly at error level, renames the unparseable file aside tosettings.json.corrupt-<ts>(the<ts>is the injectedClock's epoch seconds — filesystem-safe), and sets a transient in-memorycorrupt_resetflag before returning defaults. If the backup rename itself fails (e.g. permissions), the error is logged and defaults are still returned so boot never crashes. Bootstrap reads that transient flag after migration and — before the immediate save — folds it into the settings dict as a persistent_settings_reset_noticemarker ({"backed_up_to": <basename>}), so it survives a plugin reload. The frontend reads it via the non-consumingget_settings_reset_noticecallable and surfaces a persistent notice — a QAMPanelSectionbanner (with a Dismiss button) plus a game-detailWarningCard(informational; its copy points the user to the QAM to dismiss) — not a toast — telling the user their settings were reset and where the backup landed so they can re-enter the server URL and sign in. The marker is cleared only by an explicit user acknowledgement: the QAM Dismiss button callsdismiss_settings_reset_notice, which pops_settings_reset_noticeand persists; the frontend clears the shared store on success so the banner and every game-detail card disappear at once. Sign-in does not clear the notice — the user decides when they have read it. - Version never down-stamps: on write, the
versionfield is stamped tomax(stored_version, _SETTINGS_VERSION). A file written by a newer plugin (stored version > current) is preserved as-is rather than down-stamped, so a later re-upgrade does not re-run migrations against down-stamped data. An absent or older version is stamped up to the current_SETTINGS_VERSION.
GavelNativeAdapter notes — the compiled save-sync core¶
Both save-sync decisions run through the compiled romm-gavel core rather than in-tree Python:
- the full per-
(rom, filename, slot)sync action (Skip/Upload/Download/Conflict— the matrix in Save File Sync Architecture), and - the upload-409 resolution (the decision, on a
409from RomM'sadd_save, to eitherdownloadthe server head or surface aconflict).
GavelNativeAdapter loads py_modules/native/libgavel-x86_64-linux.so via ctypes at construction and binds both
symbols. It is itself the ResolveUploadConflictFn seam, and its compute_sync_action method is the
ComputeSyncActionFn seam (both in services/protocols/infra.py); both are injected through SaveServiceConfig →
SyncEngineConfig → MatrixExecutor, which calls the decision once per file in iter_matrix_outcomes and the 409
backstop in _handle_upload_409 (services/saves/sync_engine/matrix.py).
- What crosses the boundary: the adapter marshals the caller's raw dict shapes onto the core's caller-owned structs.
Two conversions are the adapter's, not the core's, and both exist so an "unknown" stays a flag instead of becoming a
value: a server save's ISO-8601
updated_atbecomes epoch seconds plus ahas_updated_atflag (an unparseable timestamp arrives as a cleared flag, never as a substitute instant, so it cannot win head selection), and an absent size becomes a clearedhas_sizeflag (0is exactly what the corrupt-local guard reacts to, so it cannot double as "no size recorded"). Presence of the local file rides on the pointer alone — a file that exists but could not be measured is a real case, distinct from a missing one. The core answers with the chosenserver_save_id, which the adapter resolves back to the caller's own save dict soDownload/Conflictcarry the full record their consumers read. - What ships:
py_modules/native/libgavel-x86_64-linux.so(romm-gavelv1.0.1, a freestanding build with zero library dependencies — it loads on any x86_64 Linux), vendored verbatim from the upstream release with a pinned SHA-256 checksum. The C ABI has been part of upstream's promise sincev1.0.0: struct layouts, signatures and enumerator values now cost a major bump to change, which is what makes pinning a compiled artifact meaningful. Provenance and the update procedure live innative/README.md. The checksum is re-verified by CI and the release smoke test asserts the.sois present in the plugin zip, so both a swapped binary and a dropped artifact fail the pipeline. - No fallback: if the library cannot load,
GavelNativeLoadErrorpropagates sobootstrap()aborts and the plugin stays inert — the same "fatal until the environment is fixed" posture as the SQLite migration gate. There is no Python implementation of either decision to fall back to:domain/sync_action.pyholds only theSyncActionvocabulary the core answers in. What holds the shipped binary to the contract is the vendored gavel vectors — the ladder family intests/adapters/test_gavel_native.py, the decision table intests/adapters/test_gavel_native_table_vectors.py— alongside the hand-enumerated cases intests/adapters/test_gavel_native_decision_table.py, the property tier intests/adapters/test_gavel_native_property.py, and the boundary cases that pin the two marshalling conversions above.
FirmwareService notes — the vendored BIOS registry¶
The BIOS registry — which firmware files each platform and libretro core want, with the hashes and sizes that identify
them — is the data FirmwareService reads (via domain/bios.py) to classify what a platform needs and whether a local
file is the right one. It ships as defaults/bios_registry.json and is vendored verbatim from an
emu-atlas release, where the registry and its generator now live — there is
no in-tree generator, and regeneration is a dev-time, offline step upstream (from the libretro libretro-core-info and
libretro-database sources).
- What ships:
defaults/bios_registry.json(emu-atlasv0.1.0, upstream pathatlas/data/bios_registry.json), copied verbatim with a pinned SHA-256 checksum. Provenance and the update procedure live indefaults/README.md. The checksum is re-verified by CI (.github/workflows/ci.yml, mirrored inmise run gate) and the release smoke test asserts the registry is present in the plugin zip, so both a hand-edited snapshot and a dropped file fail the pipeline. - Never hand-edit it: a manual edit would silently diverge from the released snapshot and break the checksum gate. A
registry change lands by pulling a newer emu-atlas release, re-pinning the checksum, and bumping the tag in
defaults/README.md. - Path resolution: the
.sois resolved relative to the adapter module (__file__), mirroringsqlite_migrations.MIGRATIONS_DIR— a fatal-load bundled resource must resolve identically in the installed plugin (<plugin>/py_modules/native/…) and in the repo checkout the real-bootstrap()test tiers run from, where the plugin dir is a baretmp_path.
Domain (py_modules/domain/)¶
Domain modules contain pure logic with no I/O and no Decky imports. They take inputs and return outputs; anything
stateless and I/O-free that would otherwise sit in a service lives here. Domain is stdlib + self only — it imports no
other internal layer (lib and models included). Aggregate roots and the enforcement that keeps them honest are
documented in Database Design. Selected modules:
| Module | Role |
|---|---|
sync_action.py |
The SyncAction union (Skip / Upload / Download / Conflict) the whole vertical dispatches on and the compiled core answers in. The decisions themselves are made in the core, not here. See Save File Sync Architecture. |
sync_diff.py |
ROM classification and platform/collection diff computation for the sync preview |
cover_refresh.py |
cover-fingerprint compare kernel (#1386) — scan_cover_refresh_candidates / count_cover_refreshes, shared by the apply-path invalidation pass and the preview's cover-work count so the two can never diverge |
preview_delta.py |
PreviewDelta shape for the sync preview |
work_unit.py |
WorkUnit — the per-unit sync work item |
rom_save_sync_state.py |
RomSaveSyncState aggregate + FileSyncState value object — per-ROM save-sync state, backed by rom_save_sync_states + rom_save_files |
save_path.py / save_attribution.py / save_status*.py / save_extensions.py |
save path resolution, uploader attribution, status DTO building |
firmware_paths.py / bios.py |
BIOS path computation and status formatting; bios.py holds the BIOS status dataclasses (AvailableCore, BiosFileEntry, BiosStatus) and owns the ok/partial/missing CLASSIFICATION boundary (compute_bios_level / compute_bios_label) — the single source of truth all surfaces read; phrasing + color stay UI-layer |
iso_time.py |
parse_iso / parse_iso_to_epoch — ISO-8601 timestamp parsing (stdlib only) |
achievements.py |
achievement progress computation |
shortcut_data.py |
shortcut data building (registry entries, shortcut dicts) |
game_instance.py |
GameInstance (one live sandbox instance: its signal-target pids + argv) and match_instance_for_launch_path — which live instance is running a given ROM, so Stop Game signals that tree and no other |
steam_categories.py |
Steam collection name computation |
sgdb_artwork.py |
SGDB asset-type/endpoint maps and to_signed_app_id |
installed_roms.py / rom_files.py |
installed-ROM detection, M3U generation, launch-file detection |
retroarch_core_info.py |
parse_core_info — pure parser for RetroArch .info files |
state_migrations.py |
migrate_settings (settings.json) + fold_legacy_save_sync_settings (one-time legacy save_sync_state.json fold) |
sync_state.py |
SyncState enum (idle, running, cancelling) |
emulator_tag.py / version.py |
emulator-tag formatting, version parsing, core-change detection |
Config-source parsers follow a dedicated domain+adapter template (pure parse in domain, I/O in adapter, callback Protocol into services). The full pattern, source catalog, and decisions log are on the Config Source Parsers page.
Models (py_modules/models/)¶
TypedDicts and dataclasses describing on-disk and in-flight data shapes (state.py, metadata.py). Models import
nothing from the other layers.
Other¶
| File | Role |
|---|---|
main.py |
Plugin class — Decky lifecycle (_main/_unload) and the callable surface (one async def per @callable) |
bootstrap/ |
Composition root — adapters.bootstrap() builds adapters, services.wire_services() builds services |
lib/errors.py |
Exception hierarchy (RommApiError, classify_error) |
lib/list_result.py |
ErrorCode and the canonical callable failure shape |
Composition Root (bootstrap/)¶
The composition root is a package of two halves, one per phase, plus an __init__.py that is namespace and re-exports
only — consumers write from bootstrap import … and never deep-import a submodule:
-
adapters.py— ownsbootstrap(), which builds every adapter, applies the SQLite schema migrations, and loads + migratessettings.json(folding in the one-time legacysave_sync_state.jsonsettings) so the settings persister binds the live mutablesettingsdict at construction. Returns a typedBootstrapResultcarrying four bundles (adapters,stores,callbacks,runtime_adapters) plus a smallhandlesstruct for Plugin-only outputs. The bundle dataclasses are defined here too — they are the vocabulary the second half consumes. -
services.py— ownsWiringConfigandwire_services(), which takes the four bundles plusmin_required_versionand constructs every service, injecting each one's*ServiceConfig. Returns a dict of named service instances.
The two-phase split exists because adapter instantiation and state loading happen first (bootstrap()), then main.py
composes the runtime bundle (event loop, decky.emit) and calls wire_services(). Services receive the settings dict
(the only field on StateBundle) plus the SQLite Unit-of-Work factory / repository handles for all relational state —
no plural in-memory state dicts remain. Some services are constructed before others to satisfy ordering constraints
(e.g. MigrationService before SaveService so save sync observes fresh save-sort state). Forward references between
peers are threaded via LateBinding.
Per the process-boundary rule, adapter instantiation never happens in main.py, and no service wiring happens in
bootstrap/'s caller other than via wire_services(). Both modules are governed by the ~1000-LOC decomposition
threshold (scripts/check_module_size.py), neither is grandfathered.
Protocol Interfaces¶
Services depend on Protocols, never on concrete adapter implementations. The Protocols live in the services/protocols/
package, organised topically (consumers always deep-import from services.protocols import X):
transport— external system clients:RommApi(and its narrowed facetsRommSaveApi,RommRomReader,RommDeviceApi,RommFirmwareApi,RommPlaytimeApi,RommLibraryApi,RommConnectionApi,RommPlatformReader,RommAchievementsApi,RommSyncApi,RommVersion),SteamConfigStore,SteamGridDbApi.determinism—Clock/UuidGen/Sleepertest seams.persistence—SettingsPersister,PluginMetadataReader.paths—RetroDeckPaths,SystemResolver,CoreInfoProvider,CoreResolverFn,CoreNameProviderFn,RetroArchConfigReader,RetroArchCoreInfoReader,RetroArchSaveSortingProvider,PlatformCoreReader.infra— cross-cutting callable seams:EventEmitter,DebugLogger,PathExistsReader,HostnameReader,PendingSyncReader,DownloadQueueCleanup.files— filesystem seams:CoverArtFileStore,DownloadFileStore,FirmwareFileStore,MigrationFileStore,RomFileStore,SaveFileStore,SgdbArtworkCache.cross_service— narrowly-typed multi-method seams one service exposes to another so services stay independent:BiosChecker,AchievementsReader,ArtworkManager,ArtworkRemover,RetryStrategy,MigrationPendingFn,SaveSortChangeFn,DeviceForgetFn,DeviceIdProvider(server device id,SaveService.get_device_id→ PlaytimeService),PlaytimeScopeNoticeClearFn(PlaytimeService clears its re-sign-in notice on a fresh sign-in from ConnectionService), theLaunchGate*andSession*seams.
Protocol names carry a suffix that signals shape (…Reader, …Provider/…Fn, …Store, …Cache, …Persister; bare
names for pervasive primitives like Clock).
RommApiAdapter implements RommApi over RommHttpAdapter, targeting RomM 4.9.0+ endpoints.
Boundary Enforcement¶
Four CI-gated layers keep the dependency direction and the call-site rules from drifting. Aggregate-specific enforcement
(the @cosmic_aggregate decorator and the field-assignment check) is documented in
Database Design.
1. import-linter (CI-enforced)¶
.importlinter declares the layer contracts:
# Services must not import concrete adapter implementations (Protocols OK)
[importlinter:contract:no-adapter-impl-in-services]
type = forbidden
source_modules = services
forbidden_modules = adapters
# Adapters must not import services
[importlinter:contract:no-services-in-adapters]
type = forbidden
source_modules = adapters
forbidden_modules = services
# Utilities (lib/) must not import services, adapters, or domain
[importlinter:contract:utilities-independence]
type = forbidden
source_modules = lib
forbidden_modules = services, adapters, domain
# Domain is pure compute — no dependency on any other internal layer
[importlinter:contract:domain-independence]
type = forbidden
source_modules = domain
forbidden_modules = services, adapters, lib, models
# Domain is stdlib + self only — no vendored third-party packages
[importlinter:contract:domain-stdlib-only]
type = forbidden
source_modules = domain
forbidden_modules = _vendor
# Models must not import services, adapters, domain, or lib
[importlinter:contract:models-independence]
type = forbidden
source_modules = models
forbidden_modules = services, adapters, domain, lib
# Services must not import stdlib I/O / non-deterministic primitives directly
[importlinter:contract:no-stdlib-io-in-services]
type = forbidden
source_modules = services
forbidden_modules = random, subprocess, threading, requests, time, uuid
# Services must be independent of each other
[importlinter:contract:service-independence]
type = independence
modules = services.library, services.saves, services.playtime, ...
Run with PYTHONPATH=py_modules lint-imports (or mise run lint). CI gates on this.
The service-independence modules list is hand-enumerated, so scripts/check_service_independence_contract.py
(bundled into mise run lint and gated in CI) derives the expected services from py_modules/services/ and fails if
the contract omits a service or carries a stale entry — keeping the list self-healing rather than silently rotting.
2. Cosmic Python call bans¶
scripts/check_cosmic_call_bans.sh (also bundled into mise run lint) complements the import-level guardrail at the
call site: services may not call datetime.now() / asyncio.sleep() / time.time() / time.monotonic() /
uuid.uuid4() / random.* directly — they inject the corresponding Clock / Sleeper / UuidGen Protocol instead.
3. Aggregate field-assignment check¶
scripts/check_aggregate_field_assignment.py (also bundled into mise run lint) is a small custom AST linter that
enforces the mutation-only-via-methods rule for aggregates — a rule no type checker can express directly. It
collects the class names decorated with @cosmic_aggregate in domain/ (currently the 9 aggregate roots), then scans
services/ for <aggregate>.<field> = ... assignments and fails CI on any it finds. The escape hatch is a trailing
# pragma: no aggregate-check on the offending line. Full detail in Database Design.
4. Failure-shape dialect gate¶
scripts/check_failure_shape.py --check (also bundled into mise run lint) is a small custom AST linter that enforces
the canonical failure shape for dict-returning callables — every success: False return in services/ must carry
both reason and message and must not carry the legacy error_code key or a second error key. It collapses the
three dialects that previously coexisted (error_code, error, and slug-less ad-hoc dicts) onto one vocabulary. The
two documented carve-outs (discriminated-status unions — a status key with no success; and partial-success payloads
carrying an additive server_query_failed / recommended_action flag) are pattern-exempt. Run without --check for
the report-mode inventory grouped by classification. The routing slugs come from lib.list_result.ErrorCode (the Lean
enum) plus bespoke plain-string reasons for non-server-reachability guards.
5. Enforced: underscore prefix¶
All internal methods use a _ prefix; public callables (exposed to the frontend via callable()) have none. main.py
callable methods delegate directly to the corresponding service method. Even synchronous callable bodies are async def
— Decky's callable framework requires it.
This is no longer just a convention — basedpyright enforces it with reportPrivateUsage = "error", so accessing a
_-prefixed name from outside its owning class is a hard type error. Tests are exempt via an executionEnvironments
override (white-box testing — inspecting and rebinding a system-under-test's private state — is an accepted pattern).
One corollary: a method one sub-service calls on a peer is part of that peer's public surface and carries no
underscore, which keeps reportPrivateUsage coherent with the saves-style peer-injection carve-out.
Service Dependency Summary¶
Every service receives its dependencies through a single *ServiceConfig dataclass. Cross-service dependencies are
Protocol-typed (services never import each other's concrete classes). Selected wiring:
| Service | Key injected dependencies |
|---|---|
| LibraryService | RommLibraryApi, SteamConfigStore, ArtworkManager, Clock/UuidGen/Sleeper, SettingsPersister, UnitOfWorkFactory (roms / sync_runs / kv_config / rom_metadata) |
| MetadataService | UnitOfWorkFactory (reads rom_metadata / roms) |
| SaveService | RommApi, RetryStrategy, SaveFileStore, UnitOfWorkFactory (rom_save_sync_states / rom_save_files), Clock, RetroDeckPaths, core-name/active-core providers, migration-detect callbacks |
| DownloadService | RommApi, DownloadFileStore, RetroDeckPaths, Clock/Sleeper, RomInstallRecorder + DownloadTargetGateFn cross-service seams |
| FirmwareService | RommApi, FirmwareFileStore, CoreInfoProvider, RetroDeckPaths, UnitOfWorkFactory (firmware_cache) |
| SteamGridService | SteamGridDbApi, RommApi, SteamConfigStore, SgdbArtworkCache, UnitOfWorkFactory (sgdb_id on roms), PendingSyncReader |
| MigrationService | MigrationFileStore, RetroDeckPaths, save-sort/active-core/core-name providers, BIOS-index callback |
| GameDetailService | BiosChecker, AchievementsReader (cross-service), Clock, UnitOfWorkFactory (one read UoW over roms / rom_installs / rom_save_sync_states / rom_metadata / kv_config), plus PathExistsReader + RetroDeckPaths + SystemResolver for the single target-path stat |
| AchievementsService | RommAchievementsApi, Clock, DebugLogger, UnitOfWorkFactory (reads ra_id from roms) |
| SettingsService | SteamConfigStore, SettingsPersister, UnitOfWorkFactory (reads bound shortcut_app_ids from roms) |
| PlaytimeService | RommPlaytimeApi, RetryStrategy, DeviceIdProvider (server device id, satisfied by SaveService.get_device_id), Clock, UnitOfWorkFactory (reads/writes rom_playtime + rom_playtime_sessions, and the kv_config scope-notice flag). Exposes clear_scope_notice to ConnectionService (via PlaytimeScopeNoticeClearFn) so a fresh sign-in drops the re-sign-in notice |
| RomAdoptionService | RommRomReader, DownloadFileStore (the stat, the directory scan, the hashing), RetroDeckPaths, SystemResolver, SystemM3uSupportFn, RomInstallRecorder peer, EventEmitter + Clock (throttled verify_progress frames) |
| RomInstallRecorder | Clock, UnitOfWorkFactory (rom_installs upsert + the roms size / applied-launch-options writes), SystemSupportedExtensionsFn (the launchable verdict), ActiveCoreReader + DiscResolver (the launch bake) |
| RomRemovalService | RomFileStore, RetroDeckPaths, DownloadQueueCleanup peer, UnitOfWorkFactory (reads/deletes rom_installs), Clock + EventEmitter (removal duration logging and uninstall_progress frames) |
| ShortcutRemovalService | SteamConfigStore, ArtworkRemover peer, UnitOfWorkFactory (unbinds via roms, offline name via kv_config) |
| SessionLifecycleService | Session* cross-service seams (playtime / post-exit sync / achievement sync / migration reader) |
| LaunchGateService | LaunchGateRomLookup, LaunchGateInstalledChecker, LaunchGateSaveStatusReader cross-service seams |
| GameProcessService | GameProcessControl (instance discovery + signals), RomLaunchPathReader (the launch target each live instance is matched against, from the same resolver that bakes it), Sleeper (the grace window — the service owns no clock), and RetroDECK's flatpak app id from the single domain.shortcut_data.RETRODECK_APP_ID constant the launch command is built from |
| ConnectionService | RommConnectionApi, SettingsPersister, min_required_version, DeviceForgetFn (cross-service — forgets the device id on an origin change), PlaytimeScopeNoticeClearFn (cross-service — clears the playtime re-sign-in notice on a fresh sign-in) |
| VersionSwitchService | RommRomReader (live sibling_roms view + per-sibling detail), Clock, UnitOfWorkFactory (resolves a sibling group via iter_by_group_key and moves its binding on roms), settings (reads preferred_region for the default-badge ranking), SaveDriftProbeFn + ReachabilityProbeFn (the switch-away save-stranding soft block), RomRelaunchItemReader (re-bakes a switched-onto install's launch_options), ActiveDownloadRomIdsFn (refuse a switch while a group member is downloading) |
Most services also receive the settings dict (StateBundle's only field), the runtime infrastructure (event loop,
logger, the DebugLogger Protocol), and the UnitOfWorkFactory for relational state through their config. The old
in-memory state / metadata_cache / save_sync_state / shortcut_registry dicts are gone — every relational
read/write goes through the Unit of Work.