This package is an operator composition tool. policy validates delivery bounds, manifest inventories/verifies explicit files, and build runs the locked trusted checkout build and copies only tracked application/deployment sources, fresh production dependencies and separately fingerprinted public runtime inputs. It does not import domain ORM/repositories, write application state, implement a second migration path, execute student code or perform Sandbox privilege work. scripts/check_architecture.py enforces its application import boundary.

All dependency resolution for a release uses existing lockfiles and offline caches. Backend source is shipped with locked production dependencies rather than invoking an unpinned isolated wheel build backend. uvx/pip and package startup path hooks are not delivery runtime entry points. Build inputs specify a verified public runtime inventory and the three existing image catalogs; an independent expected SHA-256 binds each input. Real input paths/receipts are local operator data and do not belong in Git.

python3.13 -m scripts.release build --repo /native/checkout \
  --inputs /private/build-inputs.json --output /private/new-bundle \
  --work /private/new-build-work

The checkout must be clean and committed, build output/work directories new, and the output filesystem have the required reserve. The resulting receipt contains the source and manifest fingerprints, file count and byte total, with installed=false and execution_verified=false. Installation and service acceptance remain separate operations. See deploy/release/README.md for the versioned installation contract and authority split.

preflight reads installation eligibility only. install-files requires root, a completely root-controlled verified input tree and the fixed target profile; it installs protected versions/images and bounded OS identities, without table migrations, application accounts, services or the active-version switch. Repeated identical files converge; unknown modes/owners, changed bytes or an incomplete staging directory require operator inspection and are never removed.

generate-configuration composes exact public contracts with separately supplied 0600 node and endpoint inputs. It emits a new private directory and a hash-bound receipt, never prints tokens, never changes a supplied password and never silently overwrites an earlier identity. API, publisher, import, migration and identity operations have distinct required database authority; Worker holds only its own MQ/control identity, Root receives no business credentials. Backend URLs use the existing asyncpg driver. No uvx/pip or online runtime installation is used.

install-configuration validates the entire group before writing any member; check-configuration checks installed bytes, same-source admission, identity and hardware/kernel/node scheduler bindings. activate requires that check and stopped control ports, no factory socket and no retained task/identity/capture resources. It atomically selects an installed directory and starts no process; draining, migrations and actual health remain separate operations. It does not erase retained journals or release resources lacking cleanup proof. Every private group also has a content-bound configuration_id. Installing a second group for the same source preserves the first group byte-for-byte. activate RELEASE --manifest-sha256 SHA --configuration-id BATCH selects an explicit validated batch only after ordinary application units and independent task resources are stopped. check-active checks the protected batch/code binding and current measurement snapshot, with no liveness/readiness claim.

provision-database-identities is an explicit local database operator step. It uses the locked asyncpg dependency, bounded connections/statements and a 120-second overall deadline. It installs only fixed independent logins/NOLOGIN groups and the application database, verifies every supplied credential and refuses unexpected existing authority. There is no application table DDL or automatic account bootstrap: the migration login owns the database without global superuser/create-role/create-db/bypass-RLS rights; Backend Alembic remains the only schema/grant migration entry. Runtime logins cannot grant/set group authority. The migration login inherits the narrow NOLOGIN function-owner role and can SET that role, so unchanged Alembic can grant/rename its functions after ownership transfer. Membership has no ADMIN option, adds no global role-creation privilege, and is never given to runtime logins. See PostgreSQL role membership and function ownership.

Always bootstrap validation from a separately trusted tool copy and an out-of-band manifest SHA. Do not execute a checker supplied inside an unverified bundle to establish trust in that same bundle. A verified delivery includes bin/kf-delivery and its source for later offline operations. Command success/receipt generation does not claim Linux execution, working service roles or installation acceptance; these require the recorded independent VM evidence.

generate-proxy is a separate, offline composition entry. Its private 0600 profile has exactly schema_version: 1, one canonical HTTPS DNS origin, certificate_sha256 and key_sha256. TLS inputs are independently supplied later at /etc/kf-proxy/tls/fullchain.pem and key.pem; the generator does not read, create or verify a private key. It emits a new private NGINX configuration and receipt tied to the verified release. No arbitrary upstream, include, path, credentials or resource overrides are accepted. Only browser API families target the API loopback port; execution-control paths are refused. ZIP uploads have their own canonical UUID route. Syntax, actual TLS/stream limits, static routes, permissions and service health require subsequent VM acceptance; the generator explicitly returns those capabilities as unverified. The separately installed kf-proxy worker identity must be distinct from API, Coordinator, import, judging and database/broker identities. It has no business credentials; generation alone never creates that account or starts NGINX.

Operator responsibilities are separate: build/verify are read-only with respect to system state; preflight observes eligibility; file/config installation and activation require the root operator; DB provisioning requires a separately supplied local DB operator; schema/grants and first account use the existing Backend Alembic/account CLIs. Configuration validation never starts or restarts a service. The later bounded service lifecycle must prove drain/cleanup before maintenance; none of these entries authorizes a Sandbox isolation bypass or silently substitutes a healthy result when lifecycle evidence is absent.

Proxy directive references: NGINX core, proxy module, and TLS module.

python3.13 -m scripts.release.services generate composes five ordinary system units from a verified bundle and matching private configuration receipt. The matching service launch modules must themselves be in that bundle. Rendering uses fixed users, resource ceilings, finite restart/stop limits and an explicit release/configuration pre-start binding. It never writes system units, starts a process, migrates a schema or claims readiness. Actual units, mount/PID/cgroup views, Unix DB socket access, dependency roles and effective limits require VM installation/acceptance. Root factory and database/broker lifecycle are separate.

API/outbox/import share the established content-storage identity; individual PID namespaces prevent same-UID inspection of another service's environment. This requires the fixed systemd profile and verified kernel namespace support. Rust Coordinator/Worker use distinct ordinary identities and systemd credentials. Every ordinary launch checks effective service/node cgroup memory, swap, CPU and PID limits, file/descriptor/stack limits, UID/GID and absence of capabilities before exec. The three Python services also require PID 1 in their private PID namespace; import requires a loopback-only network view. Ignored namespace directives therefore refuse startup. These checks still require actual VM verification; they do not attest Sandbox isolation or service readiness. The import environment includes the same validated admission policy as API.

service_bootstrap verifies bounded Rust credential bytes, copies them to a private service runtime directory with the existing required 0600 mode and then launches only the fixed ordinary binary. It accepts no student executable, command override or privileged Sandbox operation. Tokens/URLs are not command arguments or unit text. The outbox unit uses the existing actual confirmed publisher module; the old scaffold check entry is not treated as a delivery loop. Python launches use a fixed module/argument list and only the generated service environment fields; inherited debugging/proxy/provider credentials are omitted.

python3.13 -m scripts.release.service_install DIRECTORY --receipt-sha256 SHA is the separate root operator step for those five ordinary units. It requires the selected installed configuration, the canonical configuration receipt from the generator, and matching service launch/installer bytes in that delivery. All app units must be stopped. It stores immutable batches under the protected /etc/kf-onlinejudge/units/RELEASE.CONFIGURATION/ directory, then updates only known root-owned system unit pointers and performs a bounded daemon reload. Unknown regular files, foreign pointers, changed bytes and unrecorded old batches refuse replacement. Previously installed batches remain available; interruption may leave a mixed set of pointers, but each pre-start check rejects an unselected binding. Repeat installation can finish the same verified batch after stopping all app units. Loaded/inactive unit readback starts no service and makes no claim about effective runtime limits, readiness, Sandbox gates or VM acceptance.

python3.13 -m scripts.release.factory_service --output DIRECTORY composes the separate Root factory unit only after selection and protected configuration checks. It requires matching composition code in that delivery, uses the fixed Root binary and empty business environment, and preserves the supervisor reservations and finite lifetime. It does not inherit the ordinary service namespace/filter flags. Its pre-start kf-sandbox prepare-factory performs privileged cgroup preparation inside Sandbox: the exact node/UID pool, single supervisor process, empty task/lease/capture resources and factory lock are required. A retained boot/inode record recognizes only its own empty delegation; an unknown group, same-boot loss or conflicting record refuses. New-boot recreation requires empty persistent resources. No journal/lease is deleted and preparation grants no execution permit. Neither composition nor lexical tests are Root/kernel/service acceptance; install, restart, failure recovery and actual execution remain VM-only. The separate operator step factory_service --install DIRECTORY --receipt-sha256 SHA requires that generated receipt, the selected protected configuration and matching installer bytes. It keeps immutable Root-unit batches under /etc/kf-judge/units/, updates only recorded Root-owned pointers with all app units stopped, and checks loaded/inactive systemd metadata. Foreign files/links and partial unknown batches are refused, old batches remain retained. This step neither starts the factory nor prepares its cgroup; only the later bounded startup invokes Sandbox preparation.

python3.13 -m scripts.release.dependencies --connections PRIVATE_JSON --output DIRECTORY composes the fixed loopback PostgreSQL/RabbitMQ configurations and ordinary units from the selected release. Broker passwords must be identical to the selected outbox/Worker batch. The private definition file contains salted password hashes, two users without management tags, one vhost/queue and separate publish/consume permissions. Definitions import is required on a blank broker; no guest account or default password is configured. The management plugin remains disabled. PostgreSQL local/loopback paths both use SCRAM, including the Unix import socket. Configuration includes bounded connections, query/temp budgets and log controls.

Both dependency units reserve 512 MiB, zero swap and bounded CPU/PIDs/files, restart and lifetime. The receipt requires separate 2-GiB PostgreSQL and 512-MiB broker storage mounts. These mounts are not created or checked by composition. RabbitMQ memory/disk alarms and PostgreSQL WAL/temp settings are not total hard storage/memory guarantees. Required real filesystem/cgroup/queue/auth readbacks, account ownership, cluster initialization, cookie/definition permissions and startup/restart are still explicit installation/VM steps; no service is started. Dependency pre-start checks require the selected release/batch and read back the ordinary UID, zero capabilities, no-new-privileges, process/cgroup/node budgets, private data ownership, a separate data filesystem mount and its actual size. A parent filesystem or an alarm setting therefore cannot silently pass as the required dedicated hard storage capacity. Config parsing alone does not verify these conditions or broker semantics on an already-initialized node.

Dependency directive references: PostgreSQL resources, RabbitMQ limits, definition import, and RabbitMQ access control.

python3.13 -m scripts.release.dependency_install DIRECTORY --receipt-sha256 SHA --erlang-cookie PRIVATE_FILE --cookie-sha256 SHA installs the selected dependency snapshot with all app/dependency services stopped. It requires already provisioned dedicated ext4 mounts at the two fixed data paths, with their fixed capacity and rw,nosuid,nodev,noexec options. It does not create, format, resize or mount disks. A root-owned empty mount (apart from an empty private lost+found) may be assigned to its dedicated account; existing data is never recursively reassigned. Ordinary UID/GID 1203/1204 must be locked, without supplementary authority, and have the fixed PostgreSQL/RabbitMQ service homes. Conflicting OS accounts are not adopted.

The generated files are installed under /etc/kf-dependencies/RELEASE.CONFIGURATION/. This separate root-owned searchable tree lets each service group read only its 0640 configuration files; the application configuration tree remains private. The definition file's salted credentials, user privileges and queue budgets must match the selected outbox/Worker batch. The operator supplies an explicit private 256-bit Erlang cookie; repeat installation requires the same ordinary-owned 0400 cookie and does not implicitly rotate it. No cookie/password bytes are reported. System unit pointers may replace only recorded root-owned dependency snapshots. The command retains old batches, verifies loaded/inactive metadata, starts no service and initializes no database. Cluster initialization, actual effective limits, broker authorization and startup/readiness still require VM acceptance.

python3.13 -m scripts.release.dependency_initialize --password-file PRIVATE_FILE --password-sha256 SHA is the separate explicit PostgreSQL cluster initialization step. It requires the selected dependency snapshot, stopped services, the dedicated storage mount and an operator-supplied hexadecimal bootstrap password file. Only a fresh unrecorded target is initialized. The fixed ordinary PostgreSQL identity runs initdb in a transient network-isolated unit with node/service/process/disk budgets checked before execution. systemd passes the password as a private file credential; its bytes are neither unit arguments nor environment values. Both local and host authentication use SCRAM; incomplete data is retained for explicit recovery, not silently deleted or reused. The initializer never starts a database server or migrates business tables; Alembic and role provisioning remain separate.

A root-owned initialization receipt binds the physical cluster identity and initial credential fingerprint. Repeat initialization requires that recorded identity and does not reset a password or claim successful DB authentication. The operator reads the transient unit result and confirms stop; foreign/residual units and unrecorded/partial targets are refused. These are implementation interfaces, not actual VM acceptance. See PostgreSQL initdb and systemd credentials for the underlying interfaces.

python3.13 -m scripts.release.node_health performs the installed service admission-precondition checks in a fixed root operator unit. It uses bounded ordinary broker/factory/import observer units and accepts no executable, account, port or command override. Each helper checks its effective node/service/process budgets, identity and no-new-privileges/capability boundary. Unknown or concurrent units are refused; transient helper stop is read back. This is an operator CLI, not an additional public HTTP control endpoint, and has no schema/state writer.

The projection checks selected unit pointers and effective identities/limits, actual DB-role/column privileges, the import owner's fresh policy-bound heartbeat through its Unix socket, and broker queue semantics/permissions/vhost limits via ordinary-cookie read-only CLI queries. Coordinator owns the authenticated bounded execution/cleanup counters; truncated or unconfirmed data never proves a drain. Its /internal/operations/ routes require platform authority and are not proxied publicly. Counters contain no submission IDs, source, archives or credential bytes.

Factory health is a separate version-1 authenticated observation, rate/call-count bounded for the finite factory lifetime. It reads current supervisor/delegation authorization, the exact runtime-config digest, live helpers, remaining requests and lifetime, without recovery, task allocation, ledger writes or limit repair. The old allocation request bytes are unchanged. Health explicitly reports isolation_verified=false and execution_gate=required_per_execution: every real execution still needs its namespace/root/identity/seccomp/exec acknowledgements. Admission readiness requires confirmed empty execution/cleanup state and enough factory requests/lifetime for the configured maximum job. It does not certify TLS/browser behavior or replace the independent VM's execution acceptance.

The optional explicit --probe-notification sends one small confirmed notification for a fresh identifier checked absent by Coordinator. It reads no message body, creates no evaluation/grade and starts no program; ordinary periodic observation does not publish messages. Probe failure is reported separately. Underlying interfaces: RabbitMQ CLI, PostgreSQL privilege functions.

python3.13 -m scripts.release.lifecycle --action start|resume|drain|stop|restart is the explicit fixed-unit maintenance entry. Root publishes a protected, versioned availability record bound to the selected release/configuration. The installed API checks that record before accepting mutations; an absent, unknown, closed or wrong binding returns 503. Existing reads, logout, cancellation and receipt-bound archive uploads retain their ordinary identity/lease checks. Health alone cannot open this barrier, and service startup does not create schemas, accounts or credentials.

Drain first closes new mutations, keeps the API available to already accepted work, and polls the Coordinator, outbox, import, broker and factory owners within a finite deadline. Only their untruncated empty projections permit ordinary service shutdown. Final owner counts, actual stopped process groups, delegated tasks/identity/capture resources and absent factory socket must all agree before quiescence is published. Timeout preserves unfinished rows, leases, queues and replay journals; no forced grade or cleanup proof is written. drain leaves DB/MQ running, stop additionally stops them, and restart drains before a fresh start.

Fresh start requires stopped known application units. resume explicitly checks known units that are already running after an interrupted operation; it does not adopt foreign units or repair unknown resources. Both keep admission closed until full service preconditions pass and opening is read back. Failed final readiness closes the barrier again. A bounded Root maintenance receipt records manager operations and cleanup results. Root factory termination uses a signal notice and the existing bounded guardian/recovery path; ordinary import termination finishes its current authorized operation before claiming another. These interfaces require actual independent-VM lifecycle acceptance and are not installation/safety proof.

python3.13 -m scripts.release.journal separately installs a fixed KF journal namespace profile. It requires a preprovisioned dedicated 64-MiB ext4 mount at /var/log/journal/MACHINE_ID.kf-onlinejudge, with the same private hard-volume mount rules used for dependencies. It never creates/formats/mounts a disk, changes the default system journal, adopts old unrecorded log data or starts a daemon. Unknown configuration/overrides and partial installations are retained and refused.

The profile fixes 32-MiB persistent/8-MiB runtime rotation budgets, bounded files, lines and seven-day retention. Its namespaced daemon override limits CPU, memory, swap and PIDs. Rotation settings are not a hard filesystem guarantee; the dedicated volume must also be observed. The check_ready interface reads actual namespace service/process/cgroup state. Ordinary application, DB/MQ and Root-factory templates now select this namespace. Their protected pre-start binding checks require its recorded configuration and physical hard volume, while retaining ordinary private PID views and the Root factory's existing permission context. Installing the profile does not start or replace running services. Real routing, privacy and retention/capacity behavior remain C+D/F VM acceptance work. The profile relies on the pinned systemd 262 namespace interface and journal configuration.

The read-only status observer may see a ProtectSystem mount as read-only. That does not prove the logger can write: readiness separately reads the fixed known daemon's kernel mountinfo, matches its mount device to the protected volume and requires both mount and superblock writability. Main PID, invocation and process start time are checked again to reject a changed process. No writable exception, ptrace/setns capability or wider Python PID visibility is added to observation. Underlying read interface: Linux proc mountinfo implementation.

Production API logging uses a closed event vocabulary and omits arbitrary library messages, arguments, paths and exception text. Uvicorn's separate error handlers are redirected through that formatter; installed API launchers disable access logging so request query strings are not copied into service logs. Development diagnostics remain separate. These ordinary formatting checks are not evidence of actual daemon, HTTP or upload log privacy; that is still a VM acceptance case.

Build-input version 2 adds an explicitly fingerprinted public NGINX 1.30.5 runtime catalog. Version 1 remains available for older engineering bundles without proxy support; there is no implicit proxy default. The separate proxy_runtime module permits only the binary and bounded shared libraries, excludes TLS and configuration files, verifies every byte/mode and ships them under payload/proxy. This separate library search tree avoids silently replacing existing application, database or language-image libraries. Binary version/signature/ELF-closure evidence is recorded by the operator; accepting a structural catalog is not TLS/server or Linux-safety acceptance. Actual proxy installation and execution remain separate.

python3.13 -m scripts.release.proxy_install --source DIRECTORY --receipt-sha256 SHA --certificate PRIVATE_CERTIFICATE --private-key PRIVATE_KEY installs the explicitly supplied protected proxy snapshot and matching unencrypted TLS files. It requires the same release/API-origin batch, a distinct locked UID/GID 1205 (kf-proxy) without supplementary groups, and an already provisioned private 128-MiB ext4 volume at /var/lib/kf-onlinejudge/proxy. Provision that volume's ownership/mode explicitly; the installer does not format, mount or recursively reassign unknown data. Existing TLS material must match its fingerprint; this entry does not silently rotate keys or claim DNS/public-browser trust.

Proxy modules separate public runtime verification, fixed service rendering, Root installation, ordinary launch guarding and read-only HTTPS observation. The ordinary process retains only CAP_NET_BIND_SERVICE for 443, no-new-privileges, private PID visibility and actual CPU/memory/swap/PIDs/file bounds. It receives no application DB/MQ/control credentials. Installation preserves older snapshots, refuses foreign system-unit pointers, reads back loaded/inactive metadata and starts no server. Maintenance includes the proxy in start/drain/stop ordering. Readiness uses fixed loopback TLS with the supplied certificate as an explicit local trust anchor and checks hostname/leaf identity plus the API live response; it does not substitute for actual browser CA/HTTPS/security acceptance in the VM. The exact /api/v1/health/live route permits GET/HEAD for this bounded process liveness check. Other health, internal, execution and control routes remain outside the public proxy allowlist; liveness does not attest judging readiness.

Private cold backups use a separate format from public release bundles. python3.13 -m kf_oj.infra.artifacts.backup is an ordinary read-only artifact inventory command. Run it with the selected API database/storage identity after the owner-confirmed drain and while PostgreSQL is still available. It reads every ready artifact registry object through a repeatable-read, read-only transaction and hash-checks existing source/case bytes with a 64-KiB buffer. It creates no storage directory or lock and changes no metadata, grade, session or grant. Its output is private and does not itself prove that writers have stopped.

After the confirmed drain, stop the selected database and broker and verify all their processes have exited. python3.13 -m scripts.release.backup capture --backup-id bk-<32-lowercase-hex> --references PRIVATE_INVENTORY --references-sha256 SHA launches one fixed Root backup operator. It requires the same selected release/configuration, this boot's successful owner-derived cut, quiescent admission, empty task resources, no surviving service identities and clean PostgreSQL shutdown. The operator has only CAP_DAC_READ_SEARCH, no-new- privileges, 256-MiB memory, zero swap, 32 tasks, 50% CPU, a 256-MiB individual-file limit and a 30-minute deadline. Only the preprovisioned private backup volume is writable; this entry does not format/mount a volume or execute captured content.

Provision a separate Root-owned 0700 ext4 volume of 16 GiB at /var/lib/kf-onlinejudge/backups, with nosuid,nodev,noexec and no shared mount. The capture retains all source domains: the cold PostgreSQL cluster, source and testcase store, private import staging, all retained release/image versions, application/dependency/proxy configuration, factory replay history and operation receipts. Every entry has a bounded digest/size and original identity/mode; captured copies remain Root-only. Registered source/case references must all match captured files. Source modification, links, special files, unknown identities, insufficient space or an incomplete cut are refused. A failed target is retained for explicit inspection and never published as a successful backup.

The bound is 7 GiB and 50,000 entries per backup, at most two backup directories, with a seven-day declared retention period. Capacity refusal does not silently delete an old/unknown backup. backup verify --backup-id ID --manifest-sha256 SHA checks an explicitly supplied private operator receipt and every private member; it performs no restoration or secret/epoch reuse. Actual retention pruning, controlled recovery into a clean node and measured recovery time require their separate implementation and independent VM acceptance. Backup data and manifests contain private credentials/source/testcase data and must stay outside Git and CI artifacts. Parsing, ordinary adapter tests or a manifest digest are not cold-backup/recovery acceptance evidence.