Build and operate a KF-OnlineJudge node with locked dependencies and verified offline materials. The tools cover installation, configuration, HTTPS, maintenance and cold recovery. Full system and recovery acceptance is still in progress.

For the target platform and installation order, read deploy/release/README.md.

Build a release

python3.13 -m scripts.release build --repo /native/checkout \
  --inputs /private/build-inputs.json --output /private/new-bundle \
  --work /private/new-build-work

Start from a clean committed checkout. Output and work directories must be new, and their filesystem must have the required reserve. The build uses lockfiles and offline caches, then copies tracked application/deployment sources, fresh locked production dependencies and separately fingerprinted public runtimes. Backend source ships with those dependencies; uvx/pip and startup hooks are not runtime entry points.

Build inputs contain explicit inventories and expected SHA-256 digests for public runtimes and the three image catalogs. Keep real input paths and receipts outside Git. The build receipt records source and manifest digests, file count and byte total with installed=false and execution_verified=false.

Use a separately trusted checker and an out-of-band manifest SHA to validate an incoming bundle. A verified delivery includes bin/kf-delivery for later offline operations. Running an unverified bundle's checker cannot establish trust in that bundle.

Authority and service identities

Operation	Authority
Build, verify and preflight	Read-only system access
File/configuration installation and activation	Root operator
Database identity provisioning	Supplied local DB operator
Schema and grants	Backend Alembic
Application accounts	Backend account CLI
Execution state	Coordinator
Privileged task preparation and execution	Rust Sandbox

policy validates bounds, manifest inventories and verifies files, and build assembles the release. Release tools do not import domain ORM/repositories or provide a second schema migration path. scripts/check_architecture.py checks the application import boundary. Installation and configuration commands start no services; the maintenance commands below control their lifecycle.

API/outbox/import share the content-storage identity and use private PID views. Coordinator and Worker have distinct ordinary identities. service_bootstrap validates Rust credential bytes and writes a private runtime copy with mode 0600 before launching the fixed binary. Tokens and URLs stay out of arguments and unit text. Python launchers accept fixed modules, arguments and generated environment fields; inherited debugging, proxy and provider credentials are omitted. Import also requires a loopback-only network view and the same admission policy as API.

Every ordinary launch checks UID/GID, cap0, no-new-privileges, node/service cgroup limits and process file/descriptor/stack limits. The Python services require PID 1 inside their private namespace. Unsupported or ignored isolation settings refuse startup. Rust Worker has no database credentials. Root factory uses an empty business environment and gets no database, MQ or platform identity.

Install files and configuration

preflight checks eligibility. install-files requires Root, a Root-controlled verified input tree and the fixed target profile. It installs protected versions, images and OS identities. Repeat installation checks identical files; unknown ownership/modes, changed bytes and incomplete staging directories require inspection.

generate-configuration reads separate 0600 node and endpoint inputs and writes a new private batch. API, publisher, import, migration and account operations use their required DB authority; Worker receives its own MQ/control identity. Backend URLs use asyncpg. The tool preserves supplied passwords and earlier identities.

install-configuration validates the entire batch before writing. A batch has a content-bound configuration_id; installing another for the same source retains the first byte-for-byte. check-configuration validates installed bytes, admission, identity and hardware/kernel/node scheduling bindings.

Run activate RELEASE --manifest-sha256 SHA --configuration-id BATCH with the explicit batch. activate requires stopped application units, control ports and factory socket, and empty task/identity/capture resources. Root journals and batches are retained. Interruption between pointer writes requires inspection. check-active checks code/configuration selection and the measurement snapshot.

PostgreSQL and RabbitMQ

Prepare dependencies

python3.13 -m scripts.release.dependencies --connections PRIVATE_JSON --output DIRECTORY generates fixed loopback configurations and ordinary units for the selected release. Broker passwords must match the outbox/Worker batch. Its private definition contains salted hashes, two users without management tags, one vhost/queue and separate publish/consume permissions. A blank broker must import these definitions; management and guest/default accounts stay disabled.

PostgreSQL uses SCRAM on local and loopback paths, including the import Unix socket. Its configuration bounds connections, queries, temporary storage, WAL and logs. Both dependency services have 512 MiB, zero swap and finite CPU/PIDs/files, restart and lifetime limits. Memory/disk alarms supplement the hard limits.

Install snapshots

python3.13 -m scripts.release.dependency_install DIRECTORY --receipt-sha256 SHA --erlang-cookie PRIVATE_FILE --cookie-sha256 SHA installs a snapshot while all application and dependency services are stopped.

Prepare dedicated ext4 data mounts first, with fixed capacities and rw,nosuid,nodev,noexec options:

Service	Storage	Ordinary UID/GID
PostgreSQL	2 GiB	1203
RabbitMQ	512 MiB	1204

The installer does not create, format, resize or mount disks. It can assign an empty Root-owned mount, allowing only an empty private lost+found, to the dedicated account. Existing data is never recursively reassigned. Accounts must be locked, have the fixed service homes and lack supplementary authority.

Snapshots live in /etc/kf-dependencies/RELEASE.CONFIGURATION/. Each service group reads its own 0640 files; application configuration remains private. Definition hashes, permissions and queue budgets must match the selected outbox/Worker batch. Supply a private 256-bit Erlang cookie. Repeat installation requires the same ordinary-owned 0400 cookie and leaves its value unchanged.

Only recorded Root-owned unit pointers can be replaced. Old snapshots remain; loaded/inactive metadata is read back. Before execution, each dependency checks its UID, cap0, no-new-privileges, process/service/node budgets, private data ownership and the dedicated filesystem's mount identity and size.

Initialize PostgreSQL

python3.13 -m scripts.release.dependency_initialize --password-file PRIVATE_FILE --password-sha256 SHA initializes a fresh unrecorded cluster on the selected snapshot's dedicated mount, with services stopped. Supply a hexadecimal bootstrap password file. The ordinary PostgreSQL identity runs initdb in a bounded, network-isolated transient unit. systemd supplies the private password file as a credential, outside arguments and environment values. Local and host authentication use SCRAM.

The Root-owned receipt records the physical cluster identity and credential fingerprint. Repeat initialization requires that identity and does not reset its password. The operator checks the unit result and confirms stop; foreign/residual units and partial/unrecorded targets are refused. Retain incomplete data for inspection. This command starts no DB server and migrates no tables.

Provision database identities

provision-database-identities uses locked asyncpg, bounded connections/statements and a 120-second deadline. It installs fixed independent logins/NOLOGIN groups and the application database, checks every supplied credential and rejects unexpected authority. Backend Alembic remains responsible for table DDL and grants.

The migration login owns the database and can inherit/SET the narrow NOLOGIN function-owner role so Alembic can grant or rename transferred functions. It has no superuser/create-role/create-db/bypass-RLS rights or ADMIN membership option. Runtime logins cannot grant/set group authority and never receive that membership.

Application units and Root factory

python3.13 -m scripts.release.services generate renders five ordinary units from a verified bundle and matching configuration receipt. Users, budgets, restart/stop limits and pre-start release/batch checks are fixed. The outbox runs the confirmed publisher module. Service launch code must belong to the selected delivery.

python3.13 -m scripts.release.service_install DIRECTORY --receipt-sha256 SHA installs those units with all application services stopped. Snapshots are retained in /etc/kf-onlinejudge/units/RELEASE.CONFIGURATION/. The installer updates only known Root-owned pointers and performs a bounded daemon reload. Foreign pointers, regular files, changed bytes and unknown partial batches refuse replacement. A stopped repeat installation can finish the same verified batch. Every pre-start check rejects a pointer left on an unselected batch.

python3.13 -m scripts.release.factory_service --output DIRECTORY renders the separate Root unit after selection and protected configuration checks. The unit uses the fixed binary, empty business environment, supervisor reservations and finite lifetime, with the factory's own namespace/filter context.

factory_service --install DIRECTORY --receipt-sha256 SHA installs the generated receipt and unit under /etc/kf-judge/units/, preserving earlier snapshots and checking loaded/inactive systemd metadata. Application units must be stopped; foreign files/links and unknown partial batches are refused.

On startup, kf-sandbox prepare-factory checks the exact node/UID pool, a single supervisor, the factory lock and empty task/lease/capture resources before creating cgroup delegation. A boot/inode record identifies only its own empty delegation. Unknown groups, same-boot loss and conflicting records refuse startup; a new boot requires empty persistent resources. Preparation retains journals and leases. Each subsequent execution still needs the full Sandbox isolation check.

HTTPS entry

generate-proxy reads a 0600 profile containing schema_version: 1, canonical DNS origin, certificate_sha256 and key_sha256. It generates a new NGINX snapshot and receipt for the verified release. It accepts fixed upstreams, paths and limits. Browser API families reach only API loopback; execution-control routes are refused. ZIP uploads use a separate canonical UUID route.

python3.13 -m scripts.release.proxy_install --source DIRECTORY --receipt-sha256 SHA --certificate PRIVATE_CERTIFICATE --private-key PRIVATE_KEY installs the selected release/API-origin snapshot and matching unencrypted TLS material at /etc/kf-proxy/tls/fullchain.pem and key.pem.

Prepare a 128-MiB private ext4 volume at /var/lib/kf-onlinejudge/proxy and the distinct locked UID/GID 1205 (kf-proxy), without supplementary groups. The installer checks ownership/mode and matching existing TLS fingerprints; it does not format/mount data volumes, reassign unknown data or rotate keys implicitly.

The ordinary proxy keeps only CAP_NET_BIND_SERVICE for 443, no-new-privileges, private PID visibility and CPU/memory/swap/PIDs/file limits. It has no DB/MQ/control credentials. Maintenance includes it in start/drain/stop order. Readiness verifies loopback TLS with the supplied certificate, hostname/leaf identity and API live response. /api/v1/health/live allows GET/HEAD; other health/internal/control routes remain outside the allowlist. Browser CA trust needs separate verification.

Build-input version 2 specifies the NGINX 1.30.5 catalog. Version 1 supports older bundles without a proxy. proxy_runtime checks every binary/library byte and mode, excludes TLS/configuration files and ships a separate tree under payload/proxy. Record package signature, version and ELF-closure verification with the inputs.

Logs

python3.13 -m scripts.release.journal installs the fixed KF journal namespace profile on a preprovisioned 64-MiB ext4 volume at /var/log/journal/MACHINE_ID.kf-onlinejudge. It keeps the default system journal unchanged and rejects unknown configuration, overrides, old unrecorded data and partial installs.

The profile uses 32-MiB persistent and 8-MiB runtime rotation budgets, bounded files/lines and seven-day retention. Its daemon has separate CPU/memory/swap/PIDs limits. Application, DB/MQ and Root units use this namespace and check the recorded profile and physical volume before starting.

check_ready reads the daemon's process, cgroup and mount state. An observer's read-only ProtectSystem view does not establish daemon writability: the check matches the daemon's mount device to the protected volume and checks mount and superblock writability. It rechecks PID, invocation and start time to detect a changed process, without ptrace/setns or a wider observer PID view.

Production API logs use fixed event names and omit arbitrary library messages, arguments, paths and exception text. Uvicorn errors use that formatter; installed API launchers disable access logging so queries are not copied to logs. Development diagnostics remain separate. Verify real routing, privacy and capacity on the installed node.

NGINX sends access records to the namespace's local syslog socket. Records contain only method, status, response byte count and request duration; request paths, queries, cookies and hostname are omitted.

Readiness and maintenance

python3.13 -m scripts.release.node_health checks admission preconditions in one fixed Root operator unit. Ordinary broker/factory/import observers have finite budgets; each checks identity, limits and no-new-privileges/capabilities. The CLI rejects unknown/concurrent helpers, reads back their stop and accepts no executable, account, port or command override.

Checks include selected units and effective limits, DB-role/column privileges, the import owner's fresh policy-bound Unix-socket heartbeat and read-only broker queue/permissions/vhost queries. Coordinator supplies authenticated execution and cleanup counts through /internal/operations/. Truncated or unconfirmed counts cannot establish a drain. These internal routes require platform authority and return no submission IDs, source, archives or credentials.

Factory health is a version-1 authenticated, rate/call-count-bounded observation. It reports supervision/delegation authorization, config digest, live helpers, remaining requests and lifetime. It performs no recovery, allocation, ledger write or limit repair. Request bytes remain unchanged. It reports isolation_verified=false and execution_gate=required_per_execution.

Readiness requires confirmed empty execution/cleanup state and enough requests and lifetime for the maximum job. --probe-notification optionally sends one small confirmed notification for an identifier checked absent by Coordinator. It reads no body, creates no evaluation and runs no program. Periodic observation does not publish; probe failures are reported separately.

python3.13 -m scripts.release.lifecycle --action start|resume|drain|stop|restart controls the fixed units and a Root-protected release/batch availability record:

Action	Behaviour
`start`	Requires stopped known application units; opens admission after readiness passes
`resume`	Validates known running units after an interrupted operation
`drain`	Closes new mutations, waits for owner-confirmed cleanup and leaves DB/MQ running
`stop`	Completes the drain, then stops DB/MQ
`restart`	Drains before a fresh start

API returns 503 for an absent, unknown, closed or wrong-bound availability record. Reads, logout, cancellation and receipt-bound archive uploads keep their existing identity/lease checks. Both startup paths keep admission closed until the opening is read back; failed final readiness closes it again.

Drain keeps API available to accepted work and polls Coordinator, outbox, import, broker and factory within a finite deadline. Untruncated owner counts, stopped process groups, empty tasks/identity/capture resources and an absent factory socket must agree before quiescence is published. Timeout preserves rows, leases, queues and replay journals. Root uses its signal/guardian recovery path; import finishes its current authorized operation before claiming another. The maintenance receipt records manager operations and cleanup results.

Cold backups

Collect the private artifact inventory with python3.13 -m kf_oj.infra.artifacts.backup after owner-confirmed drain, while PostgreSQL is available. Use the selected API DB/storage identity. A read-only, repeatable-read transaction checks ready registry objects and source/case digests with a 64-KiB buffer. The command creates no directory/lock and changes no state.

Run lifecycle --action stop and verify processes have exited. A quiescent stop reuses only this boot's successful selected owner-derived drain receipt and rechecks application cgroups and Root task cleanup.

python3.13 -m scripts.release.backup capture --backup-id bk-<32-lowercase-hex> --references PRIVATE_INVENTORY --references-sha256 SHA captures a cold snapshot after checking the selected batch, owner-derived cut, quiescent admission, empty tasks, stopped service identities and clean PostgreSQL shutdown.

Prepare a separate Root-owned 0700, 16-GiB ext4 volume at /var/lib/kf-onlinejudge/backups, with nosuid,nodev,noexec and no shared mount. The fixed backup operator has CAP_DAC_READ_SEARCH, no-new-privileges, 256 MiB, zero swap, 32 tasks, 50% CPU, a 256-MiB file limit and a 30-minute deadline. Only that volume is writable.

Backup domains include PostgreSQL, source/case storage, private import staging, retained releases/images, configuration, replay history and operation receipts. Every member records bounded digest/size and original identity/mode; copies remain Root-only. Registry references must match. Links, special files, changing sources, unknown identities, insufficient space and incomplete cuts are refused. Failed targets remain unpublished for inspection.

Limit	Value
Per snapshot	7 GiB and 50,000 entries
Stored snapshots	At most two
Declared retention	Seven days

backup verify --backup-id ID --manifest-sha256 SHA checks the supplied private receipt and every member. backup prune --backup-id EXPIRED_ID --manifest-sha256 SHA --keep-backup-id NEWER_ID --keep-manifest-sha256 SHA verifies both inventories, requires an expired target and a newer unexpired retained copy, then removes only the named known inventory under the maintenance lock. It retains the last verified copy and refuses unknown/partial trees. Capacity refusal deletes nothing.

Private v3 backups bind the stopped broker cookie, current DB bootstrap fingerprint and recovery history. Earlier private v1/v2 prototypes are refused. These private versions do not alter judging canonical bytes or historical reports. Keep all backup data and manifests out of Git and CI artifacts.

Restore to a clean node

Install a verified public bundle on a different clean boot/node. Generate and select a newly attested batch, then install dependencies with a new broker cookie on empty provisioned volumes. Keep services stopped and do not initialize a new cluster over restored data. Use fresh Worker UUID, measurement nonce/epoch, control tokens and all DB/MQ passwords.

python3.13 -m scripts.release.restore --recovery-id rr-<32-lowercase-hex> --backup-id ID --manifest-sha256 SHA runs the cold data step. Compatibility requires identical migration bytes, target platform/toolchains and runtime images; schema-changing upgrades need a separate Backend Alembic procedure.

The operator reads the backup through a read-only mount, checks the private receipt and every byte, then copies to fixed targets under 512 MiB, zero swap, 32 tasks, 50% CPU and a 30-minute deadline. Original identities/modes and replay IDs are preserved. Old configuration and maintenance receipts go into inactive recovery/ID/history/. The new boot's Sandbox preparation owns cgroup recreation. Unknown/nonempty targets and differing existing files are refused; partial restores remain stopped for inspection.

data-restored.json records only byte/identity restoration. Services stay stopped and DB credential rotation and execution verification remain false.

Rotate database credentials

python3.13 -m scripts.release.recovery_database --recovery-id ID --data-receipt-sha256 SHA --password-file PRIVATE_NEW_BOOTSTRAP --password-sha256 SHA requires this boot's selected complete data receipt and a new 256-bit bootstrap password with a different fingerprint from the backup. Backup capture requires a complete initialization or recovery marker for the actual cluster, allowing a restored node to be backed up again. Endpoints come from current protected configuration.

One ordinary kf-postgres temporary unit uses 512 MiB, zero swap, 64 tasks, 100% CPU, a four-minute deadline, cap0/no-new-privileges and private PID/network/runtime namespaces. TCP is disabled; the private 0700 socket uses local peer authentication for this OS identity only. Fixed service logins use SCRAM.

The process checks PostgreSQL settings, roles, memberships and database ownership. It rotates the five service logins and postgres in one transaction, preserves authority and authenticates every new credential. It changes no schema, grade, lease or measurement history. SQL/error utility text is suppressed. Confirm server exit and empty resources before publishing identity-recovered.json.

Recovery and startup share a fixed lock. Startup holds a read-only shared lock through admission and validates linked prepared/data/identity receipts in the bounded Root-private recovery directories. Missing/incomplete/foreign receipts keep admission closed, including a committed credential change without confirmed shutdown. Historical receipts remain valid after reboot or version change but do not establish current readiness. They stay in inactive history on later restores.

Finish recovery with Backend Alembic verification, new TLS material, service readiness and actual four-language/history checks on the second VM. The 30-minute end-to-end target still requires measured acceptance.

Health query deadlines

Fixed health helpers include interpreter startup in their wall-clock deadline. Outbox and import queries have 60 seconds each; factory status has 30 seconds and broker metadata has 45 seconds. Node readiness has 300 seconds for the sequential checks. Each helper keeps its fixed CPU, memory, process and privilege limits. These operator deadlines do not change student execution budgets. An expired or failed query keeps admission closed.

The broker helper has a 64 MiB per-file ceiling for Erlang JIT mappings. OTP 27.3.4 starts with a 32 MiB JIT block. Broker JSON output remains limited to 32 KiB; other operator helpers keep the 16 MiB file ceiling.

Observe resources and queues

python3.13 -m scripts.release.node_observation uses a finite read-only Root cap0, no-new-privileges helper: 128 MiB, zero swap, 16 tasks, 25% CPU, 360 seconds and one named unit. It reads memory/PIDs current/peak, CPU throttling, memory events, device IO and filesystem bytes/inodes without enumerating private content. Existing owners supply service identities, ACLs, leases/cleanup, broker/outbox/import backlog and factory capacity. Notification probing stays off.

Output includes source/batch, sample time and duration. Readings cover a bounded time window, not an atomic backup cut. Missing fields, stopped services and failed owner queries are explicit incomplete observations. Busy admission can be false in a complete sample; isolation_verified remains false. Keep operational output outside public API, Git and CI artifacts.

References

PostgreSQL: role membership, function ownership, initdb, peer authentication, resources, privilege functions.
RabbitMQ: limits, definition import, access control, CLI.
NGINX: core, proxy module, TLS module, syslog.
systemd: credentials, namespace interface, journal configuration.
Linux: proc mountinfo.