docs(release): organize installation and operator guides by task

deploy/README.md

@@ -1,10 +1,12 @@
 # 本地开发依赖
 
-管理员账号 bootstrap/恢复、有界题包解析服务、不可变发布/下架、审计与回收步骤见
-[题目维护运维说明](admin-problems.md)。该说明与服务模板用于受控部署准备，不表示已部署生产。
+开发环境用 Compose 提供数据库和消息队列。单机交付使用独立的受限服务，安装入口见
+[单机交付说明](release/README.md)。管理员账号、题包导入、发布、下架和审计见
+[题目维护运维说明](admin-problems.md)。
 
-`compose.dev.yml` 只提供 PostgreSQL 和 RabbitMQ，端口绑定 localhost。
-先复制 `.env.example` 为 `.env`，然后在仓库根目录运行：
+## 启动开发依赖
+
+将 `.env.example` 复制为 `.env`，在仓库根目录运行：
 
 ```bash
 make infra-config
@@ -12,75 +14,73 @@ make infra-up
 make infra-down
 ```
 
-`infra-down` 保留数据卷；不要把示例口令用于正式部署。
-API 存活检查和前端预览不依赖这些容器，可以先独立启动。
-该 Compose 不运行判题 Worker 或不可信代码。
+`compose.dev.yml` 只启动 PostgreSQL 和 RabbitMQ，端口绑定 localhost。
+`infra-down` 保留数据卷。API 存活检查和前端预览可以先独立启动。
+不要在这个 Compose 或宿主机上运行学生代码，也不要将示例口令用于正式部署。
 
-镜像固定具体维护版本（PostgreSQL 18.6、RabbitMQ 4.3.6），正式实验前另锁镜像摘要。
-配置检查只验证 Compose 结构；成功启动、实际健康与备份恢复需要单独验证。
-后续受控主机服务和最小权限沙箱启动器配置放在本目录。
+开发镜像固定 PostgreSQL 18.6 和 RabbitMQ 4.3.6；正式实验还需锁定镜像摘要。
+配置检查只检查 Compose 结构。服务健康和备份恢复要在实际环境验证。
+镜像用法见官方 [PostgreSQL](https://hub.docker.com/_/postgres) 和
+[RabbitMQ](https://hub.docker.com/_/rabbitmq) 说明。
 
-官方镜像说明：[PostgreSQL](https://hub.docker.com/_/postgres)、
-[RabbitMQ](https://hub.docker.com/_/rabbitmq)。
+## 单机 helper 工厂
 
-### 单机 helper 工厂
+`kf-sandbox factory --config /etc/kf-judge/runtime.json` 启动 Root 工厂。
+普通 Worker 连接配置中的 `state_root/factory/control.sock`。
+安装 `deploy/kf-sandbox-factory.service` 示例前，管理员需核对固定二进制、不可写的
+GNU 镜像与配置、Worker UID/GID，以及 Root 所有且与监督器 cgroup 分离的任务委派根。
+工厂读回服务和任务预算，每次编译、测试点运行都经过完整隔离检查。
 
-`kf-sandbox factory --config /etc/kf-judge/runtime.json` 是 Root 服务入口；
-普通 Worker 使用配置中 `state_root/factory/control.sock` 的普通用户客户端。
-`deploy/kf-sandbox-factory.service` 给出有限服务预算示例，安装前由可信管理员
-核对固定二进制、不可写 GNU 镜像/配置、Worker UID/GID，以及**与服务自身 cgroup
-分离**的 Root 所有任务委派根。工厂实际读回服务 cgroup 限额，应用/读回任务总预算；
-每次编译和测试点仍须通过已有完整隔离门控。服务文件本身不证明这些条件成立。
+工厂无等待队列，只接收有界 run ID。槽位满、重复 ID 和未知字段立即拒绝。
+每次请求创建全新的单线程 helper，继承已认证连接和工厂生命线。
+额外控制帧、断连或生命线消失会终止任务树。ID 先进入持久记录，最多 4096 个，
+跨重启继续拒绝重放。清理失败时保留身份租约并停止分配；恢复只处理已记录的
+归属和 inode，未知资源保留并拒绝启动。
 
-工厂只接收有界 run ID，无排队：槽位满、重复 ID 或未知字段立即拒绝。
-每次申请使用全新单线程 helper，固定继承已认证的连接与工厂生命线；
-额外控制帧、断开或生命线消失会终止整棵任务树。ID 先写入有限持久记录
-（最多 4096 个，跨重启拒绝重放）；容量用尽需要管理员在所有任务停止、已核对
-回收后安排新记录周期，不自动删除记录。清理失败保留身份租约并停止新分配；
-重启只根据已记录归属和 inode 回收，未知/外来内容保留并拒绝启动。
+### 轮换记录周期
 
-工厂 `--max-requests` 由可信 Root 启动参数选择，范围1–4096，默认4096；
-它限制当前记录周期的受理次数，不改变学生预算或已记录 ID 的拒绝规则。
-`kf-sandbox rotate-factory-cycle --config ROOT_CONFIG --cycle-id ID` 是独立的
-Root 维护入口：在受限监督器作用域中，核对受保护配置/镜像、取得工厂排他锁，
-并读回任务、身份、捕获目录和委派 cgroup 为空、控制 socket 已消失后，才把
-当前请求记录保留到 Root 私有 `factory/cycles/`，fsync文件与目录后清空当前记录。
-最多保留64个有界周期；历史 ID 在后续启动仍用于拒绝重放。正常启动、重启、
-寿命结束和容量用尽不会调用轮换或删除任何历史记录。归档完成但当前记录未清空
-的中断状态拒绝启动，需在停止/排空后显式使用同一个 cycle ID 重试；未知或
-不完整记录保留并拒绝，不能通过删日志恢复。该入口不启动服务，也不证明隔离。
-新安装路径下的实际轮换、崩溃恢复、跨周期重放拒绝与资源读回仍需独立 VM 验收。
+可信 Root 参数 `--max-requests` 设置本周期受理上限，范围 1 到 4096，默认 4096。
+容量用尽后，先停止任务并核对回收，再由管理员维护记录周期。
 
-`kf-worker judge-submission` 是受控本地适配入口。常驻 `kf-worker serve --config`
-使用有界 AMQP 通知和经过服务身份认证的 Coordinator claim；配置与依赖见
-[判题工程说明](../judge/README.md#单机持久化协调与常驻-worker)。
-它验证不可变清单摘要，C17/C++20 仅映射到固定 GNU 命令，编译一次；
-成功产物经完整清理后封印并计算 SHA-256，每点分别申请 helper、stdin 与工作区。
-期望值逐点读取，留在比较端，不传给工厂/用户文件系统。
-新增锁定 `sha2 0.10.9` 仅供 Worker 验证输入及编译产物摘要；
-Sandbox 同版本仅为独立 VM 测试数据生成的开发依赖，Engine 不依赖加密或 OS 库。
+`kf-sandbox rotate-factory-cycle --config ROOT_CONFIG --cycle-id ID` 在受限监督器作用域
+中核对配置和镜像、取得工厂排他锁，确认任务、身份、捕获目录、委派 cgroup 均为空，
+且控制 socket 已消失，才归档请求记录到 Root 私有 `factory/cycles/`。
+归档文件和目录经 fsync 后清空当前记录，最多保留 64 个周期；历史 ID 继续用于拒绝重放。
 
-### 单机提交服务的启动与停止
+正常启动、重启、寿命结束和容量用尽不会自动轮换或删除历史记录。
+若归档完成后中断，先排空，再显式用同一 cycle ID 重试。未知或不完整记录需要检查，
+不能删日志续用工厂。轮换入口不启动服务。新安装路径的轮换、崩溃恢复和跨周期拒绝
+仍需独立 VM 验收。
 
-1. Backend Alembic 迁移；创建分别仅继承平台/协调器 NOLOGIN 角色的专用登录账号，
-   平台角色只可插入初始评测，执行状态仅由 Coordinator 写入。
-2. 普通平台服务持有私有工件目录；使用应用服务导入不可变题目和创建测试账号。
-3. 验证固定 GNU 镜像、Root 配置及工厂服务/任务 cgroup 的真实上限，再启动工厂。
-4. 独立启动 API、Coordinator、outbox publisher 和普通 Worker；单机 HTTP/AMQP
-   仅数值 loopback，PostgreSQL loopback/Unix socket。任何入口都不自动创建表。
-5. Coordinator readiness 验证数据库和权限；仍需独立 VM 验证真实执行门控。
-   Root 工厂不可持有平台/MQ/数据库口令，用户运行镜像不含服务文件。
-6. 先向 Worker 发送 SIGTERM，使当前任务取消/清理，再停止 publisher/API/Coordinator，
-   最后停止工厂和本地依赖。强制死亡后的旧 attempt 必须撤销并取得实际停止证明；
-   `cleanup_failed` 不等于普通终态，先核对保留资源再由管理员恢复工厂/Worker。
+## 单机提交服务的启动与停止
 
-平台配置私有 `KF_OJ_ARTIFACT_ROOT`、有界字节/文件容量以及
-`KF_OJ_JUDGE_ADMISSION`（完整可信 JSON）；其工具链、编译/运行上限、总体作业期限、
-并发/prefetch/retry 必须与 Coordinator/Worker 一致。单机使用并发1、prefetch1，
-不能仅提高 Worker 数量绕过节点容量。源码最多256KiB，测试点最多256个；
-stdout/stderr 各不超过 file_bytes，合计不超过 workdir_bytes。
+`kf-worker judge-submission` 是受控本地适配入口；常驻
+`kf-worker serve --config` 使用有界 AMQP 通知和认证后的 Coordinator claim。
+配置见[判题工程说明](../judge/README.md#单机持久化协调与常驻-worker)。
 
-迁移账号与平台账号分开配置，命令均在 `backend/` 运行：
+Worker 校验不可变清单摘要，将 C17/C++20 映射到固定 GNU 命令，编译一次。
+产物经完整清理后封印并计算 SHA-256；每个测试点单独申请 helper、stdin 和工作区。
+期望输出留在比较端。Worker 用 `sha2 0.10.9` 校验输入与产物；Sandbox 的同版本
+开发依赖仅生成独立 VM 测试数据，Engine 不依赖加密或 OS 库。
+
+### 启动顺序
+
+1. 用独立迁移账号执行 Backend Alembic。平台与协调器登录账号分别继承各自的
+   NOLOGIN 角色；平台只插入初始评测，执行状态由 Coordinator 写入。
+2. 为平台准备私有工件目录，通过应用服务导入不可变题目并创建测试账号。
+3. 验证 GNU 镜像、Root 配置、工厂服务和任务 cgroup 的实际限额，启动工厂。
+4. 独立启动 API、Coordinator、outbox publisher 和普通 Worker。HTTP/AMQP 使用
+   数值 loopback，PostgreSQL 使用 loopback/Unix socket；启动入口不创建表。
+5. 检查 Coordinator 数据库权限与 readiness，再在独立 VM 验证执行隔离。
+   Root 工厂不持有平台/MQ/数据库口令，用户运行镜像不含服务文件。
+
+平台配置包括私有 `KF_OJ_ARTIFACT_ROOT`、工件容量和可信
+`KF_OJ_JUDGE_ADMISSION`。工具链、编译/运行预算、作业期限、并发/prefetch/retry
+必须与 Coordinator/Worker 一致。单机并发 1、prefetch 1，共享任务身份池。
+源码最多 256 KiB，测试点最多 256 个；stdout/stderr 各不超过 file_bytes，
+合计不超过 workdir_bytes。增加 Worker 数量不会扩大节点槽位。
+
+以下开发命令均在 `backend/` 运行；迁移账号与平台账号分别配置：
 
 ```sh
 uv run --locked alembic upgrade head           # 私有迁移账号，仅执行受控 schema 操作
@@ -91,70 +91,77 @@ uv run --locked uvicorn kf_oj.api.app:create_app --factory --host 127.0.0.1 --po
 uv run --locked python -m kf_oj.infra.messaging.publisher
 ```
 
-每个账号操作/导入目录都由管理员准备，不从浏览器接受私有路径。导入格式见
-[题目模块](../backend/src/kf_oj/modules/problems/README.md)，不可变版本以 UUID
-在题库/题面链接和提交请求中传递。更新题目需导入新版本，不覆盖已有提交的题目版本。
-临时验收使用合成数据和测试口令，实际业务数据与服务配置都保存在仓库外的私有目录。
+账号和导入目录由管理员准备，浏览器不能传入私有路径。导入格式见
+[题目模块](../backend/src/kf_oj/modules/problems/README.md)。题库、题面和提交使用
+不可变版本 UUID；更新需发布新版本。临时验收使用合成数据和测试口令，业务数据与
+配置放在仓库外。
 
-取消另需 API 私有配置 `KF_OJ_COORDINATOR_URL=http://127.0.0.1:8100` 与
-`KF_OJ_COORDINATOR_TOKEN`（Coordinator 的 platform_token）。所有服务令牌私下生成，
-不可互用或放进前端环境。Coordinator/Worker 私有 JSON 为0600、单链接、有界文件；
-Root 镜像/配置/二进制由 root 持有，任务委派子树独立于监督器自身 cgroup。
-运行前实际读回整体节点、服务和任务 memory/swap/pids/CPU/文件/运行期限上限，
-并通过真实隔离门控；配置解析、readiness、存活探测都不能替代执行验收。
+取消功能还需 API 配置 `KF_OJ_COORDINATOR_URL=http://127.0.0.1:8100` 和
+`KF_OJ_COORDINATOR_TOKEN`，后者对应 Coordinator 的 platform_token。
+独立生成服务令牌，保存在私有配置中。Coordinator/Worker JSON 为 0600、单链接、
+有界文件；Root 持有镜像、配置和二进制。运行前读回节点、服务和任务的
+memory/swap/pids/CPU/文件/期限上限。
 
-前端从 `frontend/` 启动 `pnpm dev`，默认127.0.0.1:5173；平台明确允许相同浏览器
-Origin，`/api` 同源代理。生产需另行配置 HTTPS、安全 cookie、SPA 回退与 `/api`
-反向代理，本仓库不自动部署。用户以受控账号登录，在题面选择支持语言、提交源码，
-随后查看同一 submission UUID 的持久化状态。等待/取消中/系统失败都不等于成功成绩。
+开发前端在 `frontend/` 运行 `pnpm dev`，默认 127.0.0.1:5173。
+平台明确允许相同 Origin，`/api` 使用同源代理。生产入口还需 HTTPS、安全 Cookie、
+SPA 回退和 API 反向代理。登录后在题面提交源码，按同一 submission UUID 查看状态；
+等待、取消中和系统失败各有独立状态。
 
-broker 暂停时 API 仍能持久化 queued/outbox；恢复后 publisher confirm 与 Coordinator
-补偿重新投递。Worker/Coordinator 重启使用原私有配置、工件、日志与 DB，不删除
-attempt 或强改状态。旧租约不能覆盖新结果，有限基础设施重试最终收敛。
-容量满时返回429或503；先确认待执行/清理状态，恢复服务或由管理员扩充经过核对的
-存储容量，不删除仍被提交/题目引用的工件。`kf-artifacts --limit 100` 仅回收满足年龄
-要求的无引用工件和已标记删除项；它不会通过删除用户源码来腾出容量。
-恢复 Root 必须核对 journal/inode/实际排空；未知资源保留，管理员处理自己确认的
-故障来源后再重启。不要递归清空任务目录，也不要把 cleanup_failed 改成已成功。
+### 停止与故障恢复
 
-这些入口和顺序不表示生产部署已完成。隔离/超限与用户程序仅在无 NIC、无宿主共享
-挂载、无业务凭据且整体上限已读回的独立测试 VM 内验收。开发 Compose 仍只提供可信
-数据库和 broker，不在宿主或业务服务器运行用户代码。workflow 如需启用由指定
-GitHub 私有仓库承载，并设置有限并发；Gitea 只保存逐阶段工程提交。
+先向 Worker 发送 SIGTERM，使当前任务取消并清理，再停止 publisher/API/Coordinator，
+最后停止工厂和本地依赖。强制死亡后的旧 attempt 必须撤销并取得停止证明。
+`cleanup_failed` 保留未确认资源，管理员检查故障来源后再恢复工厂/Worker。
+不要递归清空任务目录或强改结果。
 
+broker 暂停期间，API 保留 queued/outbox；恢复后 publisher confirm 与 Coordinator
+补偿重新投递。重启 Worker/Coordinator 时保留配置、工件、日志、DB 和 attempt。
+旧租约不能覆盖新结果，基础设施重试有次数上限。
+
+容量满时返回 429 或 503。先检查待执行和清理状态，再恢复服务或核对扩容。
+`kf-artifacts --limit 100` 只回收达到年龄要求的无引用工件和已标记删除项，
+保留仍被题目或提交引用的源码。Root 恢复还需核对 journal、inode 和实际排空。
 
 ## 四语言节点的受控升级
 
-新增公开示例 `judge.admission.example.json` 为显式 v1/v2 注册表，
-`sandbox.managed.runtime.example.json` 为一槽位的共享 Root 配置；原 v1 示例保持原语义。
-所有 `aaaaaaaa…` 镜像摘要均是无权限占位值，不能当作已验镜像。管理员制作固定镜像后，
-对 manifest.json 的实际字节计算 SHA-256，逐文件验证版本/字节/权限，并将相同完整身份
-写入准入注册表与 Root runtime_images。Root 镜像路径仅在私有可信配置中保存。
-配置解析通过或 check-runtime 可见仍不允许跳过实际隔离门控。
+`judge.admission.example.json` 是显式 v1/v2 注册表，
+`sandbox.managed.runtime.example.json` 使用一槽位共享 Root 配置。
+原 v1 示例语义保留。示例中的 `aaaaaaaa…` 是无权限占位摘要，必须换成固定镜像
+manifest.json 的实际 SHA-256，并逐文件验证版本、字节和权限。
+准入注册表与 Root runtime_images 使用同一完整身份；Root 镜像路径仅放入私有配置。
 
-1. 停止创建新提交/消费投递，向所有普通 Worker 发 SIGTERM；保留 DB、清单、工件和 journal。
-2. 取得工厂实际 stopped/cleanup_complete 证明，读回空任务 cgroup 和 owned 资源；
-   未排空或存在未知资源时停止升级，保留证据，不递归清空任务目录。
-3. 使用独立迁移账号执行 Backend Alembic upgrade；新语言数据存在时旧迁移降级会拒绝，
-   不删除数据以实现回退。平台与 Coordinator 继续使用各自最小权限角色，Worker 无 DB 账号。
-4. 安装固定版本的 Backend/Coordinator/Worker/Sandbox；同时更新三个普通服务的同一
-   `judge_admission`/`admission` 注册表，Root 配置单独由管理员持有。旧消费者不混用新协议。
-5. 为整体节点、控制面各服务、工厂和任务设置有限 CPU/memory/swap/pids/文件/期限，
-   读回内核值。节点至少预留一份512MiB编译任务以及独立控制面；Java运行示例总预算384MiB，
-   Xmx不能代替整体预算。并发1/prefetch1，共享UID池，不因第二Worker存在而扩大节点槽位。
-6. 在独立无NIC、无宿主共享挂载、无业务凭据 VM 用合成题目验证四语言、故障与实际排空，
-   再恢复工厂/Coordinator/API/publisher/Worker。生产启用需单独安排，仓库不自动部署。
+1. 停止创建提交和消费投递，向普通 Worker 发 SIGTERM，保留 DB、清单、工件和 journal。
+2. 确认工厂 stopped/cleanup_complete，读回空任务 cgroup 和 owned 资源。
+   发现未知资源或未排空时停止升级并保留证据。
+3. 用独立账号执行 Backend Alembic upgrade。含新语言数据时旧迁移拒绝降级；
+   保留这些数据。平台与 Coordinator 使用各自最小权限角色，Worker 无 DB 账号。
+4. 安装固定版本 Backend/Coordinator/Worker/Sandbox，同时更新普通服务的同一
+   `judge_admission`/`admission`。Root 配置单独保管，旧消费者不混用新协议。
+5. 设置并读回节点、控制面、工厂和任务的 CPU/memory/swap/pids/文件/期限上限。
+   至少预留一份 512 MiB 编译任务和独立控制面；Java 运行示例总预算 384 MiB，
+   Xmx 只是堆上限。并发 1、prefetch 1，使用共享 UID 池。
+6. 在断网、无宿主共享挂载、无业务凭据的受限独立 VM 验证四语言、故障和排空，
+   再恢复服务。生产启用单独安排。
 
-请求不在线安装 Python/Java 依赖，不运行 pip/uvx，不接收自定义解释器参数/JAR/agents。
-Worker 私有配置可设置 `control_timeout_ms`（1000–5000，缺省2000）作为 Coordinator
-单次控制请求的期限。资源受限节点应依据实际领取/续租/停止耗时选择，并核对租约和
-停止恢复；这不改变学生阶段/作业期限、重试次数或工件下载的固定2秒期限，连接仍限1秒。
-Python main.py 仅标准库；Java Main.java 使用默认包 Main，最多256个有界编译class。
-题目语言与预算以公开版本配置为准，旧题目和草稿保留，Rust继续明确不支持。
-新版检查阶段CE/运行RE/真实限额与SystemError的区别见 [判题说明](../judge/README.md#python--java-与明确协议版本)。
+Python `main.py` 仅支持标准库。Java `Main.java` 使用默认包 Main，最多 256 个
+有界编译 class。请求不在线安装依赖，不运行 pip/uvx，不接收解释器参数、JAR 或 agents。
+题目版本决定语言和预算，旧题目、草稿和报告保留；Rust 暂不支持。
+CE、RE、真实超限与 SystemError 的区别见
+[判题说明](../judge/README.md#python--java-与明确协议版本)。
 
-停止/故障恢复沿原单机流程：停止Worker使任务全树排空，旧租约/report拒绝后恢复；
-publisher/broker中断保留outbox，有限重试而非清空记录。恢复旧软件前应先排空v2任务，
-保留不可变v2版本和报告供兼容消费者读取；不能把旧配置静默转换为宽松新能力。
-这些公开配置不含业务令牌。数据库/AMQP/服务令牌分别在仓库外私下生成，Root不持有它们。
-需要CI时只在GitHub私有仓库 `KiriAky107/KF-OnlineJudge` 运行并设置有限并发，Gitea保存源码。
+Worker 的 `control_timeout_ms` 可设为 1000 到 5000，缺省 2000。
+按实际领取、续租和停止耗时选择，并核对租约与恢复流程。
+该值不改变学生阶段/作业期限、重试次数、工件下载的固定 2 秒期限或 1 秒连接期限。
+
+回退旧软件前先排空 v2 任务，保留 v2 不可变版本和报告供兼容消费者读取。
+恢复时沿用停止与故障流程，保持 outbox 和有限重试。
+公开示例不含业务令牌；数据库、AMQP 和服务令牌分别在仓库外生成，Root 不持有它们。
+
+## 验收与 CI
+
+所有学生源码、隔离、超限和故障测试都在无 NIC、无宿主共享挂载、无业务凭据且
+整体上限已读回的独立 VM 运行。配置解析、readiness 和存活探测不能替代执行验收。
+仓库不自动部署生产。
+
+需要 workflow 时，在 GitHub 私有仓库 `KiriAky107/KF-OnlineJudge` 设置有限并发执行。
+Gitea 保存工程提交，不运行 workflow。

deploy/release/README.md

@@ -1,107 +1,134 @@
-# Single-node delivery contract
+# Single-node delivery
 
-`contract.v1.json` describes the first pre-release installation interface. It is
-separate from judging v1/v2 wire formats and never rewrites historical reports.
-The reference target is Arch/CachyOS Linux x86_64-v3, glibc 2.44, systemd and cgroup
-v2. Images and service libraries are bound to explicit file/catalog fingerprints;
-support for other distributions requires a separately verified delivery profile.
+The single-node profile packages the application, four judging runtimes and the
+installation tools for offline use. `contract.v1.json` defines the installation
+interface. Judging v1/v2 reports keep their existing formats.
 
-The target has at least 4 GiB RAM and 12 GiB installation/data capacity. Judging
-and problem-import concurrency remain one each. Configured service and execution
-budgets, effective kernel limits and free capacity must also pass preflight; total
-RAM alone does not prove enough capacity or verified execution isolation.
-Preflight requires at least 12 GiB of filesystem capacity and derives required
-free bytes from the verified application, protected runtime-image copies and
-separate GNU legacy copy, plus 4 GiB of mutable-data/operation reserve. It does
-not assume that a nominal 12-GiB filesystem exposes 12 GiB free after metadata.
+Full installation and recovery acceptance is in progress. Validate a release on
+an independent test node before using it for production.
 
-Release payloads are directory bundles with a bounded manifest: at most 50,000
-single-link regular files, 256 MiB per file, 4 GiB total and 8 MiB metadata. The
-manifest SHA-256 is supplied separately by the operator. Links, extra members,
-private paths and unexpected file changes are refused. Built payloads and real
-configuration never belong in Git. Validation is read-only and does not claim
-that any service has been installed, started or verified:
+## Requirements
+
+| Item | Required profile |
+| --- | --- |
+| OS | Arch/CachyOS, Linux x86_64-v3, glibc 2.44, systemd 262, cgroup v2 |
+| Memory | At least 4 GiB, with enough free capacity for the configured services and tasks |
+| Storage | At least 12 GiB filesystem capacity; preflight calculates the required free space |
+| Concurrency | One judging job and one problem import |
+| Dependencies | PostgreSQL 18.6 and the fixed broker runtime, prepared from verified offline materials |
+
+Other distributions need a separately verified profile. Preflight checks the
+effective kernel limits and available capacity as well as nominal RAM and disk
+size. Its free-space calculation includes the application, protected runtime
+images, the separate GNU legacy image and 4 GiB for mutable data and operations.
+
+## Verify a bundle
+
+Use a separately trusted checker and the manifest digest supplied with the release:
 
 ```sh
 python3.13 -m scripts.release contract deploy/release/contract.v1.json
 python3.13 -m scripts.release verify /path/to/bundle --manifest-sha256 SHA256_FROM_RECEIPT
 ```
 
-Application versions live below `/opt/kf-onlinejudge/releases`; `current` selects
-an explicitly installed version. Private operator/service configuration lives in
-`/etc/kf-onlinejudge` and `/etc/kf-judge`, mutable application data under
-`/var/lib/kf-onlinejudge`, and protected Sandbox state/images under
-`/var/lib/kf-judge`. The delegated task cgroup is separate from the Root factory's
-own supervision cgroup. Application processes do not initialize tables on start;
-Backend Alembic remains the only migration entry point.
-All application services and the separate delegated task subtree live beneath
-`kfonlinejudge.slice`: 3840 MiB memory, no swap, 384 processes/threads and two CPU
-equivalents. The remaining physical RAM is reserved for the kernel/minimal OS.
-The task subtree is `/sys/fs/cgroup/kfonlinejudge.slice/kf-judge-tasks`, a sibling
-of the Root factory supervision unit under that same total node budget. Its
-installation template does not prove the effective limits; actual service and
-task limits are read back in the independent acceptance environment.
+A bundle contains at most 50,000 single-link regular files, 256 MiB per file,
+4 GiB in total and 8 MiB of metadata. Verification rejects links, extra members,
+private paths and changed bytes. Keep bundles and real configuration outside Git.
 
-Database login identities are separate for migration, API/outbox, Coordinator,
-problem import and local identity operations. Rust Worker and Root factory do not
-hold database credentials. Root factory has no MQ/platform/SSH/GitHub identity;
-privileged execution remains inside Sandbox. Public and admin browser traffic
-share the HTTPS site; execution-control interfaces remain internal.
+Verification is read-only. Installation, running services and execution isolation
+each need their own acceptance evidence.
 
-Upgrade and restore must retain immutable problem versions, submitted source,
-audit and original judging reports. A consistent backup cut has a zero confirmed
-data-loss target; restoring from already prepared release/backup materials has a
-30-minute recovery target, including service health and actual four-language
-verification. These are acceptance targets, not results inferred from this file.
-Installation/service/restore evidence is required before this pre-release can be
-treated as an accepted delivery; the contract itself does not deploy production.
+## Directories and authority
 
-The node input for `generate-configuration` has exactly `schema_version: 1`,
-`origin` (one HTTPS origin), `worker_id` (canonical nonzero UUID), and
-`measurement_nonce` (null or a fresh operator-generated SHA-256-sized nonce).
-It is a 0600 single-link file. Endpoint input is a separate 0600 JSON object
-with exactly `schema_version: 1`, `database` and `amqp`: database keys are
-`api`, `coordinator`, `import`, `migration`, `identity`; broker keys are `outbox`
-and `worker`. Login names and numeric loopback/socket locations are validated by
-the tool. Every endpoint requires its own operator-supplied private credential;
-there is no working example password or shared default. Import uses the reachable
-`/run/postgresql` Unix socket; all other control connections use numeric loopback.
+| Location | Contents |
+| --- | --- |
+| `/opt/kf-onlinejudge/releases` | Installed application versions |
+| `current` | Explicitly selected application version |
+| `/etc/kf-onlinejudge` and `/etc/kf-judge` | Private operator and service configuration |
+| `/var/lib/kf-onlinejudge` | Mutable application data |
+| `/var/lib/kf-judge` | Protected Sandbox state and runtime images |
 
-Generated admission uses the actual verified catalogs for C17/C++20, CPython
-3.13.16 and OpenJDK 21.0.12.1+1-LTS and the unchanged public per-language budgets.
-The protected GNU legacy image is a separate directory: it retains the original
-v1 header and identical verified GNU executable/library bytes, while the three
-v2 catalog identities remain untouched. Shared Worker/task UID/GID pool and
-execution/prefetch/import slots remain one. No canonical judging report or
-historical runtime identity is converted by the installer.
+Backend Alembic is the only schema migration entry point. Database identities are
+separate for migration, API/outbox, Coordinator, problem import and local account
+operations. Rust Worker has no database identity; the Root factory has no
+database, MQ, platform, SSH or GitHub credentials. Sandbox owns privileged
+execution. Public and admin pages share the HTTPS origin; control interfaces stay
+internal.
 
-Comparability defaults to unavailable (`measurement_epoch=null`). A trusted
-operator may explicitly use `--attest-measurement` only after actual node/service
-limits and the intended execution environment have been verified. The attestation
-binds a fresh nonce, worker, source, exact images, budgets, CPU/microcode, kernel
-and node scheduling snapshot. `check-configuration` refuses an old binding when
-those values change; it does not manufacture replacement metrics for old reports.
-Before any service resume after such a change, drain execution, generate/install
-a new complete private group with a new nonce and independently revalidate the
-environment. Raw configuration checks are not performance or isolation evidence.
-Configuration batches have independent hash-bound `cfg-...` identities; files
-and receipts contain both the source release and batch identifier. A fresh
-nonce/identity can therefore be installed for the same source without replacing
-an old configuration. Preparation never selects the new batch automatically.
-`activate` requires an explicit `--configuration-id`, a fully validated group,
-all application units inactive, stopped control ports and no retained task
-resources. Root journals and all old batches remain intact. It binds the code
-pointer and private batch in `/etc/kf-onlinejudge/active.json`; `check-active`
-refuses inconsistent pointers, stale hardware bindings or changed bytes.
-Interruption between the two atomic pointer writes requires operator review;
-no service starts or claims readiness during that mismatch. A stopped stale
-batch can be retired without accepting it for execution under a changed kernel.
+Application services and the task subtree share `kfonlinejudge.slice`, capped at
+3840 MiB, zero swap, 384 processes/threads and two CPU equivalents. The remaining
+physical memory is reserved for the kernel and minimal OS. Tasks use
+`/sys/fs/cgroup/kfonlinejudge.slice/kf-judge-tasks`, separate from the Root
+supervisor's cgroup. Read back service and task limits on the target node.
 
-The installation entry points are available with `python3.13 -m scripts.release
---help`. Use them in order: verify/preflight, install-files, generate-configuration,
-install-configuration, explicit DB identity provision, controlled Backend Alembic
-and account CLI operations, then activate and the separately verified bounded
-service lifecycle. These operations never contact a package index and do not
-install onto a business host on their own. OS dependencies (systemd 262, matching
-glibc ABI bytes, PostgreSQL 18.6 and the fixed broker runtime) must already be
-present from separately verified offline materials.
+## Prepare configuration
+
+Supply two 0600 single-link JSON files to `generate-configuration`:
+
+| Input | Fields |
+| --- | --- |
+| Node | `schema_version: 1`, `origin`, `worker_id`, `measurement_nonce` |
+| Endpoints | `schema_version: 1`, `database`, `amqp` |
+
+`origin` is one HTTPS origin. `worker_id` is a canonical nonzero UUID.
+`measurement_nonce` is null or a fresh operator-generated SHA-256-sized nonce.
+The database object has `api`, `coordinator`, `import`, `migration` and `identity`
+keys; the broker object has `outbox` and `worker` keys. Supply a different private
+credential for every endpoint. Import uses `/run/postgresql`; other control
+connections use numeric loopback. The tool checks login names and locations.
+
+The generated admission profile binds C17/C++20, CPython 3.13.16 and OpenJDK
+21.0.12.1+1-LTS to verified catalogs and the existing per-language budgets. The
+GNU legacy image retains its v1 header and verified executable/library bytes in
+a separate directory. Worker, task identities and execution/prefetch/import
+slots remain one. Historical reports and runtime identities remain unchanged.
+
+## Select a version
+
+Configuration batches have content-bound `cfg-...` identities. Preparation keeps
+earlier batches and does not select a new one. `activate` requires an explicit
+`--configuration-id`, a fully validated batch, inactive application units,
+stopped control ports and no retained task resources.
+
+Selection writes the code pointer and `/etc/kf-onlinejudge/active.json`. A crash
+between those atomic writes can leave a mismatch; stop and inspect it before
+starting services. `check-active` rejects inconsistent pointers, changed bytes
+and stale hardware bindings. Root journals and earlier batches remain intact.
+A stopped stale batch can be retired without accepting it under a changed kernel.
+
+## Performance comparability
+
+Comparability defaults to unavailable (`measurement_epoch=null`). Use
+`--attest-measurement` after verifying the node's limits and execution environment.
+It binds a fresh nonce, worker, source, images, budgets, CPU/microcode, kernel and
+node scheduling snapshot. `check-configuration` rejects a changed binding.
+
+After an environment change, drain execution, install a complete batch with a new
+nonce and revalidate before resuming. Earlier reports retain their original
+measurements. Configuration validation alone does not prove comparable performance
+or execution isolation.
+
+## Installation order
+
+Use `python3.13 -m scripts.release
+--help` for the command interface. Prepare the OS dependencies and dedicated data
+volumes, then follow this order:
+
+1. Verify the bundle and run preflight.
+2. Install files and runtime images.
+3. Generate and install a private configuration batch.
+4. Activate that batch while services and tasks are stopped.
+5. Install the journal profile and selected dependency snapshots.
+6. Initialize PostgreSQL, provision DB identities, run Backend Alembic and create
+   application accounts; stop the initialization database before installing units.
+7. Install ordinary units, the Root factory and the HTTPS snapshot/TLS material.
+8. Start the bounded service lifecycle and verify readiness and actual execution.
+
+The tools use offline materials and do not contact a package index.
+
+See the [operator command reference](../../scripts/release/README.md) for service,
+HTTPS, backup and recovery operations. Upgrades and restores must preserve
+immutable problem versions, submitted source, audit and original judging reports.
+A confirmed consistent backup cut has a zero data-loss target. Recovery from
+prepared release and backup materials has a 30-minute target, including health
+and four-language verification. Both targets require measured acceptance results.

scripts/release/README.md

@@ -1,20 +1,13 @@
 # Offline release tooling
 
-This package is an operator composition tool. `policy` validates delivery bounds,
-`manifest` inventories/verifies explicit files, and `build` runs the locked trusted
-checkout build and copies only tracked application/deployment sources, fresh
-production dependencies and separately fingerprinted public runtime inputs.
-It does not import domain ORM/repositories, write application state, implement a
-second migration path, execute student code or perform Sandbox privilege work.
-`scripts/check_architecture.py` enforces its application import boundary.
+Build and operate a KF-OnlineJudge node with locked dependencies and verified
+offline materials. The tools cover installation, configuration, HTTPS, maintenance
+and cold recovery. Full system and recovery acceptance is still in progress.
 
-All dependency resolution for a release uses existing lockfiles and offline
-caches. Backend source is shipped with locked production dependencies rather
-than invoking an unpinned isolated wheel build backend. uvx/pip and package
-startup path hooks are not delivery runtime entry points. Build inputs specify a
-verified public runtime inventory and the three existing image catalogs; an
-independent expected SHA-256 binds each input. Real input paths/receipts are local
-operator data and do not belong in Git.
+For the target platform and installation order, read
+[`deploy/release/README.md`](../../deploy/release/README.md).
+
+## Build a release
 
 ```sh
 python3.13 -m scripts.release build --repo /native/checkout \
@@ -22,504 +15,428 @@ python3.13 -m scripts.release build --repo /native/checkout \
   --work /private/new-build-work
 ```
 
-The checkout must be clean and committed, build output/work directories new,
-and the output filesystem have the required reserve. The resulting receipt
-contains the source and manifest fingerprints, file count and byte total, with
-`installed=false` and `execution_verified=false`. Installation and service
-acceptance remain separate operations. See `deploy/release/README.md` for the
-versioned installation contract and authority split.
+Start from a clean committed checkout. Output and work directories must be new,
+and their filesystem must have the required reserve. The build uses lockfiles and
+offline caches, then copies tracked application/deployment sources, fresh locked
+production dependencies and separately fingerprinted public runtimes. Backend
+source ships with those dependencies; uvx/pip and startup hooks are not runtime
+entry points.
 
-`preflight` reads installation eligibility only. `install-files` requires root,
-a completely root-controlled verified input tree and the fixed target profile;
-it installs protected versions/images and bounded OS identities, without table
-migrations, application accounts, services or the active-version switch.
-Repeated identical files converge; unknown modes/owners, changed bytes or an
-incomplete staging directory require operator inspection and are never removed.
+Build inputs contain explicit inventories and expected SHA-256 digests for public
+runtimes and the three image catalogs. Keep real input paths and receipts outside
+Git. The build receipt records source and manifest digests, file count and byte
+total with `installed=false` and `execution_verified=false`.
 
-`generate-configuration` composes exact public contracts with separately supplied
-0600 node and endpoint inputs. It emits a new private directory and a hash-bound
-receipt, never prints tokens, never changes a supplied password and never silently
-overwrites an earlier identity. API, publisher, import, migration and identity
-operations have distinct required database authority; Worker holds only its own
-MQ/control identity, Root receives no business credentials. Backend URLs use the
-existing asyncpg driver. No uvx/pip or online runtime installation is used.
+Use a separately trusted checker and an out-of-band manifest SHA to validate an
+incoming bundle. A verified delivery includes `bin/kf-delivery` for later offline
+operations. Running an unverified bundle's checker cannot establish trust in that
+bundle.
 
-`install-configuration` validates the entire group before writing any member;
-`check-configuration` checks installed bytes, same-source admission, identity and
-hardware/kernel/node scheduler bindings. `activate` requires that check and
-stopped control ports, no factory socket and no retained task/identity/capture
-resources. It atomically selects an installed directory and starts no process;
-draining, migrations and actual health remain separate operations. It does not
-erase retained journals or release resources lacking cleanup proof.
-Every private group also has a content-bound `configuration_id`. Installing a
-second group for the same source preserves the first group byte-for-byte.
-`activate RELEASE --manifest-sha256 SHA --configuration-id BATCH` selects an
-explicit validated batch only after ordinary application units and independent
-task resources are stopped. `check-active` checks the protected batch/code
-binding and current measurement snapshot, with no liveness/readiness claim.
+## Authority and service identities
 
-`provision-database-identities` is an explicit local database operator step.
-It uses the locked asyncpg dependency, bounded connections/statements and a
-120-second overall deadline. It installs only fixed independent logins/NOLOGIN
-groups and the application database, verifies every supplied credential and
-refuses unexpected existing authority. There is no application table DDL or
-automatic account bootstrap: the migration login owns the database without
-global superuser/create-role/create-db/bypass-RLS rights; Backend Alembic remains
-the only schema/grant migration entry. Runtime logins cannot grant/set group
-authority. The migration login inherits the narrow NOLOGIN function-owner role
-and can SET that role, so unchanged Alembic can grant/rename its functions after
-ownership transfer. Membership has no ADMIN option, adds no global role-creation
-privilege, and is never given to runtime logins. See
-[PostgreSQL role membership](https://www.postgresql.org/docs/18/role-membership.html)
-and [function ownership](https://www.postgresql.org/docs/18/sql-alterfunction.html).
+| Operation | Authority |
+| --- | --- |
+| Build, verify and preflight | Read-only system access |
+| File/configuration installation and activation | Root operator |
+| Database identity provisioning | Supplied local DB operator |
+| Schema and grants | Backend Alembic |
+| Application accounts | Backend account CLI |
+| Execution state | Coordinator |
+| Privileged task preparation and execution | Rust Sandbox |
 
-Always bootstrap validation from a separately trusted tool copy and an out-of-band
-manifest SHA. Do not execute a checker supplied inside an unverified bundle to
-establish trust in that same bundle. A verified delivery includes `bin/kf-delivery`
-and its source for later offline operations. Command success/receipt generation
-does not claim Linux execution, working service roles or installation acceptance;
-these require the recorded independent VM evidence.
+`policy` validates bounds, `manifest` inventories and verifies files, and `build`
+assembles the release. Release tools do not import domain ORM/repositories or
+provide a second schema migration path. `scripts/check_architecture.py` checks
+the application import boundary. Installation and configuration commands start
+no services; the maintenance commands below control their lifecycle.
 
-`generate-proxy` is a separate, offline composition entry. Its private 0600
-profile has exactly `schema_version: 1`, one canonical HTTPS DNS `origin`,
-`certificate_sha256` and `key_sha256`. TLS inputs are independently supplied
-later at `/etc/kf-proxy/tls/fullchain.pem` and `key.pem`; the generator
-does not read, create or verify a private key. It emits a new private NGINX
-configuration and receipt tied to the verified release. No arbitrary upstream,
-include, path, credentials or resource overrides are accepted. Only browser API
-families target the API loopback port; execution-control paths are refused. ZIP
-uploads have their own canonical UUID route. Syntax, actual TLS/stream limits,
-static routes, permissions and service health require subsequent VM acceptance;
-the generator explicitly returns those capabilities as unverified.
-The separately installed `kf-proxy` worker identity must be distinct from API,
-Coordinator, import, judging and database/broker identities. It has no business
-credentials; generation alone never creates that account or starts NGINX.
+API/outbox/import share the content-storage identity and use private PID views.
+Coordinator and Worker have distinct ordinary identities. `service_bootstrap`
+validates Rust credential bytes and writes a private runtime copy with mode 0600
+before launching the fixed binary. Tokens and URLs
+stay out of arguments and unit text. Python launchers accept fixed modules,
+arguments and generated environment fields; inherited debugging, proxy and
+provider credentials are omitted. Import also requires a loopback-only network
+view and the same admission policy as API.
 
-Operator responsibilities are separate: build/verify are read-only with respect
-to system state; preflight observes eligibility; file/config installation and
-activation require the root operator; DB provisioning requires a separately
-supplied local DB operator; schema/grants and first account use the existing
-Backend Alembic/account CLIs. Configuration validation never starts or restarts
-a service. The later bounded service lifecycle must prove drain/cleanup before
-maintenance; none of these entries authorizes a Sandbox isolation bypass or
-silently substitutes a healthy result when lifecycle evidence is absent.
+Every ordinary launch checks UID/GID, cap0, no-new-privileges, node/service cgroup
+limits and process file/descriptor/stack limits. The Python services require PID 1
+inside their private namespace. Unsupported or ignored isolation settings refuse
+startup. Rust Worker has no database credentials. Root factory uses an empty
+business environment and gets no database, MQ or platform identity.
 
-Proxy directive references: [NGINX core](https://nginx.org/en/docs/http/ngx_http_core_module.html),
-[proxy module](https://nginx.org/en/docs/http/ngx_http_proxy_module.html), and
-[TLS module](https://nginx.org/en/docs/http/ngx_http_ssl_module.html).
+## Install files and configuration
 
-`python3.13 -m scripts.release.services generate` composes five ordinary system
-units from a verified bundle and matching private configuration receipt. The
-matching service launch modules must themselves be in that bundle. Rendering
-uses fixed users, resource ceilings, finite restart/stop limits and an explicit
-release/configuration pre-start binding. It never writes system units, starts a
-process, migrates a schema or claims readiness. Actual units, mount/PID/cgroup
-views, Unix DB socket access, dependency roles and effective limits require VM
-installation/acceptance. Root factory and database/broker lifecycle are separate.
+`preflight` checks eligibility. `install-files` requires Root, a Root-controlled
+verified input tree and the fixed target profile. It installs protected versions,
+images and OS identities. Repeat installation checks identical files; unknown
+ownership/modes, changed bytes and incomplete staging directories require inspection.
 
-API/outbox/import share the established content-storage identity; individual PID
-namespaces prevent same-UID inspection of another service's environment. This
-requires the fixed systemd profile and verified kernel namespace support. Rust
-Coordinator/Worker use distinct ordinary identities and systemd credentials.
-Every ordinary launch checks effective service/node cgroup memory, swap, CPU and
-PID limits, file/descriptor/stack limits, UID/GID and absence of capabilities
-before exec. The three Python services also require PID 1 in their private PID
-namespace; import requires a loopback-only network view. Ignored namespace
-directives therefore refuse startup. These checks still require actual VM
-verification; they do not attest Sandbox isolation or service readiness.
-The import environment includes the same validated admission policy as API.
+`generate-configuration` reads separate 0600 node and endpoint inputs and writes
+a new private batch. API, publisher, import, migration and account operations use
+their required DB authority; Worker receives its own MQ/control identity. Backend
+URLs use asyncpg. The tool preserves supplied passwords and earlier identities.
 
-`service_bootstrap` verifies bounded Rust credential bytes, copies them to a
-private service runtime directory with the existing required 0600 mode and then
-launches only the fixed ordinary binary. It accepts no student executable,
-command override or privileged Sandbox operation. Tokens/URLs are not command
-arguments or unit text. The outbox unit uses the existing actual confirmed
-publisher module; the old scaffold check entry is not treated as a delivery loop.
-Python launches use a fixed module/argument list and only the generated service
-environment fields; inherited debugging/proxy/provider credentials are omitted.
+`install-configuration` validates the entire batch before writing. A batch has a
+content-bound `configuration_id`; installing another for the same source retains
+the first byte-for-byte. `check-configuration` validates installed bytes,
+admission, identity and hardware/kernel/node scheduling bindings.
 
-`python3.13 -m scripts.release.service_install DIRECTORY --receipt-sha256 SHA`
-is the separate root operator step for those five ordinary units. It requires
-the selected installed configuration, the canonical configuration receipt from
-the generator, and matching service launch/installer bytes in that delivery.
-All app units must be stopped. It stores immutable batches under the protected
-`/etc/kf-onlinejudge/units/RELEASE.CONFIGURATION/` directory, then updates only
-known root-owned system unit pointers and performs a bounded daemon reload.
-Unknown regular files, foreign pointers, changed bytes and unrecorded old batches
-refuse replacement. Previously installed batches remain available; interruption
-may leave a mixed set of pointers, but each pre-start check rejects an unselected
-binding. Repeat installation can finish the same verified batch after stopping
-all app units. Loaded/inactive unit readback starts no service and makes no
-claim about effective runtime limits, readiness, Sandbox gates or VM acceptance.
+Run `activate RELEASE --manifest-sha256 SHA --configuration-id BATCH` with the
+explicit batch. `activate` requires stopped application units, control ports and
+factory socket, and empty task/identity/capture resources. Root journals and
+batches are retained. Interruption between pointer writes requires inspection.
+`check-active` checks code/configuration selection and the measurement snapshot.
 
-`python3.13 -m scripts.release.factory_service --output DIRECTORY` composes the
-separate Root factory unit only after selection and protected configuration checks.
-It requires matching composition code in that delivery, uses the fixed Root
-binary and empty business environment, and preserves the supervisor reservations
-and finite lifetime. It does not inherit the ordinary service namespace/filter
-flags. Its pre-start `kf-sandbox prepare-factory` performs privileged cgroup
-preparation inside Sandbox: the exact node/UID pool, single supervisor process,
-empty task/lease/capture resources and factory lock are required. A retained
-boot/inode record recognizes only its own empty delegation; an unknown group,
-same-boot loss or conflicting record refuses. New-boot recreation requires empty
-persistent resources. No journal/lease is deleted and preparation grants no
-execution permit. Neither composition nor lexical tests are Root/kernel/service
-acceptance; install, restart, failure recovery and actual execution remain VM-only.
-The separate operator step `factory_service --install DIRECTORY --receipt-sha256 SHA`
-requires that generated receipt, the selected protected configuration and matching
-installer bytes. It keeps immutable Root-unit batches under `/etc/kf-judge/units/`,
-updates only recorded Root-owned pointers with all app units stopped, and checks
-loaded/inactive systemd metadata. Foreign files/links and partial unknown batches
-are refused, old batches remain retained. This step neither starts the factory nor
-prepares its cgroup; only the later bounded startup invokes Sandbox preparation.
+## PostgreSQL and RabbitMQ
+
+### Prepare dependencies
 
 `python3.13 -m scripts.release.dependencies --connections PRIVATE_JSON --output DIRECTORY`
-composes the fixed loopback PostgreSQL/RabbitMQ configurations and ordinary units
-from the selected release. Broker passwords must be identical to the selected
-outbox/Worker batch. The private definition file contains salted password hashes,
-two users without management tags, one vhost/queue and separate publish/consume
-permissions. Definitions import is required on a blank broker; no guest account
-or default password is configured. The management plugin remains disabled.
-PostgreSQL local/loopback paths both use SCRAM, including the Unix import socket.
-Configuration includes bounded connections, query/temp budgets and log controls.
+generates fixed loopback configurations and ordinary units for the selected
+release. Broker passwords must match the outbox/Worker batch. Its private
+definition contains salted hashes, two users without management tags, one
+vhost/queue and separate publish/consume permissions. A blank broker must import
+these definitions; management and guest/default accounts stay disabled.
 
-Both dependency units reserve 512 MiB, zero swap and bounded CPU/PIDs/files,
-restart and lifetime. The receipt requires separate 2-GiB PostgreSQL and 512-MiB
-broker storage mounts. These mounts are not created or checked by composition.
-RabbitMQ memory/disk alarms and PostgreSQL WAL/temp settings are not total hard
-storage/memory guarantees. Required real filesystem/cgroup/queue/auth readbacks,
-account ownership, cluster initialization, cookie/definition permissions and
-startup/restart are still explicit installation/VM steps; no service is started.
-Dependency pre-start checks require the selected release/batch and read back the
-ordinary UID, zero capabilities, no-new-privileges, process/cgroup/node budgets,
-private data ownership, a separate data filesystem mount and its actual size.
-A parent filesystem or an alarm setting therefore cannot silently pass as the
-required dedicated hard storage capacity. Config parsing alone does not verify
-these conditions or broker semantics on an already-initialized node.
+PostgreSQL uses SCRAM on local and loopback paths, including the import Unix
+socket. Its configuration bounds connections, queries, temporary storage, WAL and
+logs. Both dependency services have 512 MiB, zero swap and finite CPU/PIDs/files,
+restart and lifetime limits. Memory/disk alarms supplement the hard limits.
 
-Dependency directive references: [PostgreSQL resources](https://www.postgresql.org/docs/18/runtime-config-resource.html),
-[RabbitMQ limits](https://www.rabbitmq.com/docs/limits),
-[definition import](https://www.rabbitmq.com/docs/definitions),
-and [RabbitMQ access control](https://www.rabbitmq.com/docs/access-control).
+### Install snapshots
 
 `python3.13 -m scripts.release.dependency_install DIRECTORY --receipt-sha256 SHA
---erlang-cookie PRIVATE_FILE --cookie-sha256 SHA` installs the selected dependency
-snapshot with all app/dependency services stopped. It requires already provisioned
-dedicated ext4 mounts at the two fixed data paths, with their fixed capacity and
-`rw,nosuid,nodev,noexec` options. It does not create, format, resize or mount disks.
-A root-owned empty mount (apart from an empty private `lost+found`) may be assigned
-to its dedicated account; existing data is never recursively reassigned. Ordinary
-UID/GID 1203/1204 must be locked, without supplementary authority, and have the
-fixed PostgreSQL/RabbitMQ service homes. Conflicting OS accounts are not adopted.
+--erlang-cookie PRIVATE_FILE --cookie-sha256 SHA` installs a snapshot while all
+application and dependency services are stopped.
 
-The generated files are installed under `/etc/kf-dependencies/RELEASE.CONFIGURATION/`.
-This separate root-owned searchable tree lets each service group read only its
-0640 configuration files; the application configuration tree remains private.
-The definition file's salted credentials, user privileges and queue budgets must
-match the selected outbox/Worker batch. The operator supplies an explicit private
-256-bit Erlang cookie; repeat installation requires the same ordinary-owned 0400
-cookie and does not implicitly rotate it. No cookie/password bytes are reported.
-System unit pointers may replace only recorded root-owned dependency snapshots.
-The command retains old batches, verifies loaded/inactive metadata, starts no
-service and initializes no database. Cluster initialization, actual effective
-limits, broker authorization and startup/readiness still require VM acceptance.
+Prepare dedicated ext4 data mounts first, with fixed capacities and
+`rw,nosuid,nodev,noexec` options:
+
+| Service | Storage | Ordinary UID/GID |
+| --- | --- | --- |
+| PostgreSQL | 2 GiB | 1203 |
+| RabbitMQ | 512 MiB | 1204 |
+
+The installer does not create, format, resize or mount disks. It can assign an
+empty Root-owned mount, allowing only an empty private `lost+found`, to the
+dedicated account. Existing data is never recursively reassigned. Accounts must
+be locked, have the fixed service homes and lack supplementary authority.
+
+Snapshots live in `/etc/kf-dependencies/RELEASE.CONFIGURATION/`. Each service group
+reads its own 0640 files; application configuration remains private. Definition
+hashes, permissions and queue budgets must match the selected outbox/Worker batch.
+Supply a private 256-bit Erlang cookie. Repeat installation requires the same
+ordinary-owned 0400 cookie and leaves its value unchanged.
+
+Only recorded Root-owned unit pointers can be replaced. Old snapshots remain;
+loaded/inactive metadata is read back. Before execution, each dependency checks
+its UID, cap0, no-new-privileges, process/service/node budgets, private data
+ownership and the dedicated filesystem's mount identity and size.
+
+### Initialize PostgreSQL
 
 `python3.13 -m scripts.release.dependency_initialize --password-file PRIVATE_FILE
---password-sha256 SHA` is the separate explicit PostgreSQL cluster initialization
-step. It requires the selected dependency snapshot, stopped services, the dedicated
-storage mount and an operator-supplied hexadecimal bootstrap password file. Only
-a fresh unrecorded target is initialized. The fixed ordinary PostgreSQL identity
-runs `initdb` in a transient network-isolated unit with node/service/process/disk
-budgets checked before execution. systemd passes the password as a private file
-credential; its bytes are neither unit arguments nor environment values. Both
-local and host authentication use SCRAM; incomplete data is retained for explicit
-recovery, not silently deleted or reused. The initializer never starts a database
-server or migrates business tables; Alembic and role provisioning remain separate.
+--password-sha256 SHA` initializes a fresh unrecorded cluster on the selected
+snapshot's dedicated mount, with services stopped. Supply a hexadecimal bootstrap
+password file. The ordinary PostgreSQL identity runs `initdb` in a bounded,
+network-isolated transient unit. systemd supplies the private password file as a
+credential, outside arguments and environment values. Local and host
+authentication use SCRAM.
 
-A root-owned initialization receipt binds the physical cluster identity and initial
-credential fingerprint. Repeat initialization requires that recorded identity and
-does not reset a password or claim successful DB authentication. The operator
-reads the transient unit result and confirms stop; foreign/residual units and
-unrecorded/partial targets are refused. These are implementation interfaces, not
-actual VM acceptance. See [PostgreSQL initdb](https://www.postgresql.org/docs/18/app-initdb.html)
-and [systemd credentials](https://systemd.io/CREDENTIALS/) for the underlying interfaces.
+The Root-owned receipt records the physical cluster identity and credential
+fingerprint. Repeat initialization requires that identity and does not reset its
+password. The operator checks the unit result and confirms stop; foreign/residual
+units and partial/unrecorded targets are refused. Retain incomplete data for
+inspection. This command starts no DB server and migrates no tables.
 
-`python3.13 -m scripts.release.node_health` performs the installed service
-admission-precondition checks in a fixed root operator unit. It uses bounded
-ordinary broker/factory/import observer units and accepts no executable, account,
-port or command override. Each helper checks its effective node/service/process
-budgets, identity and no-new-privileges/capability boundary. Unknown or concurrent
-units are refused; transient helper stop is read back. This is an operator CLI,
-not an additional public HTTP control endpoint, and has no schema/state writer.
+### Provision database identities
 
-The projection checks selected unit pointers and effective identities/limits,
-actual DB-role/column privileges, the import owner's fresh policy-bound heartbeat
-through its Unix socket, and broker queue semantics/permissions/vhost limits via
-ordinary-cookie read-only CLI queries. Coordinator owns the authenticated bounded
-execution/cleanup counters; truncated or unconfirmed data never proves a drain.
-Its `/internal/operations/` routes require platform authority and are not proxied
-publicly. Counters contain no submission IDs, source, archives or credential bytes.
+`provision-database-identities` uses locked asyncpg, bounded connections/statements
+and a 120-second deadline. It installs fixed independent logins/NOLOGIN groups and
+the application database, checks every supplied credential and rejects unexpected
+authority. Backend Alembic remains responsible for table DDL and grants.
 
-Factory health is a separate version-1 authenticated observation, rate/call-count
-bounded for the finite factory lifetime. It reads current supervisor/delegation
-authorization, the exact runtime-config digest, live helpers, remaining requests
-and lifetime, without recovery, task allocation, ledger writes or limit repair.
-The old allocation request bytes are unchanged. Health explicitly reports
-`isolation_verified=false` and `execution_gate=required_per_execution`: every real
-execution still needs its namespace/root/identity/seccomp/exec acknowledgements.
-Admission readiness requires confirmed empty execution/cleanup state and enough
-factory requests/lifetime for the configured maximum job. It does not certify
-TLS/browser behavior or replace the independent VM's execution acceptance.
+The migration login owns the database and can inherit/SET the narrow NOLOGIN
+function-owner role so Alembic can grant or rename transferred functions. It has
+no superuser/create-role/create-db/bypass-RLS rights or ADMIN membership option.
+Runtime logins cannot grant/set group authority and never receive that membership.
 
-The optional explicit `--probe-notification` sends one small confirmed notification
-for a fresh identifier checked absent by Coordinator. It reads no message body,
-creates no evaluation/grade and starts no program; ordinary periodic observation
-does not publish messages. Probe failure is reported separately. Underlying
-interfaces: [RabbitMQ CLI](https://www.rabbitmq.com/docs/man/rabbitmqctl.8),
-[PostgreSQL privilege functions](https://www.postgresql.org/docs/18/functions-info.html).
+## Application units and Root factory
 
-`python3.13 -m scripts.release.lifecycle --action start|resume|drain|stop|restart`
-is the explicit fixed-unit maintenance entry. Root publishes a protected, versioned
-availability record bound to the selected release/configuration. The installed API
-checks that record before accepting mutations; an absent, unknown, closed or wrong
-binding returns 503. Existing reads, logout, cancellation and receipt-bound archive
-uploads retain their ordinary identity/lease checks. Health alone cannot open this
-barrier, and service startup does not create schemas, accounts or credentials.
+`python3.13 -m scripts.release.services generate` renders five ordinary units from
+a verified bundle and matching configuration receipt. Users, budgets, restart/stop
+limits and pre-start release/batch checks are fixed. The outbox runs the confirmed
+publisher module. Service launch code must belong to the selected delivery.
 
-Drain first closes new mutations, keeps the API available to already accepted
-work, and polls the Coordinator, outbox, import, broker and factory owners within
-a finite deadline. Only their untruncated empty projections permit ordinary
-service shutdown. Final owner counts, actual stopped process groups, delegated
-tasks/identity/capture resources and absent factory socket must all agree before
-quiescence is published. Timeout preserves unfinished rows, leases, queues and
-replay journals; no forced grade or cleanup proof is written. `drain` leaves DB/MQ
-running, `stop` additionally stops them, and `restart` drains before a fresh start.
+`python3.13 -m scripts.release.service_install DIRECTORY --receipt-sha256 SHA`
+installs those units with all application services stopped. Snapshots are retained
+in `/etc/kf-onlinejudge/units/RELEASE.CONFIGURATION/`. The installer updates only
+known Root-owned pointers and performs a bounded daemon reload. Foreign pointers,
+regular files, changed bytes and unknown partial batches refuse replacement.
+A stopped repeat installation can finish the same verified batch. Every pre-start
+check rejects a pointer left on an unselected batch.
 
-Fresh start requires stopped known application units. `resume` explicitly checks
-known units that are already running after an interrupted operation; it does not
-adopt foreign units or repair unknown resources. Both keep admission closed until
-full service preconditions pass and opening is read back. Failed final readiness
-closes the barrier again. A bounded Root maintenance receipt records manager
-operations and cleanup results. Root factory termination uses a signal notice and
-the existing bounded guardian/recovery path; ordinary import termination finishes
-its current authorized operation before claiming another. These interfaces require
-actual independent-VM lifecycle acceptance and are not installation/safety proof.
+`python3.13 -m scripts.release.factory_service --output DIRECTORY` renders the
+separate Root unit after selection and protected configuration checks. The unit
+uses the fixed binary, empty business environment, supervisor reservations and
+finite lifetime, with the factory's own namespace/filter context.
 
-`python3.13 -m scripts.release.journal` separately installs a fixed KF journal
-namespace profile. It requires a preprovisioned dedicated 64-MiB ext4 mount at
-`/var/log/journal/MACHINE_ID.kf-onlinejudge`, with the same private hard-volume
-mount rules used for dependencies. It never creates/formats/mounts a disk, changes
-the default system journal, adopts old unrecorded log data or starts a daemon.
-Unknown configuration/overrides and partial installations are retained and refused.
+`factory_service --install DIRECTORY --receipt-sha256 SHA` installs the generated
+receipt and unit under `/etc/kf-judge/units/`, preserving earlier snapshots and
+checking loaded/inactive systemd metadata. Application units must be stopped;
+foreign files/links and unknown partial batches are refused.
 
-The profile fixes 32-MiB persistent/8-MiB runtime rotation budgets, bounded files,
-lines and seven-day retention. Its namespaced daemon override limits CPU, memory,
-swap and PIDs. Rotation settings are not a hard filesystem guarantee; the dedicated
-volume must also be observed. The `check_ready` interface reads actual namespace
-service/process/cgroup state. Ordinary application, DB/MQ and Root-factory
-templates now select this namespace. Their protected pre-start binding checks
-require its recorded configuration and physical hard volume, while retaining
-ordinary private PID views and the Root factory's existing permission context.
-Installing the profile does not start or replace running services. Real routing,
-privacy and retention/capacity behavior remain C+D/F VM acceptance work.
-The profile relies on the pinned systemd 262
-[namespace interface](https://github.com/systemd/systemd/blob/v262/man/systemd.exec.xml)
-and [journal configuration](https://github.com/systemd/systemd/blob/v262/man/journald.conf.xml).
+On startup, `kf-sandbox prepare-factory` checks the exact node/UID pool, a single
+supervisor, the factory lock and empty task/lease/capture resources before creating
+cgroup delegation. A boot/inode record identifies only its own empty delegation.
+Unknown groups, same-boot loss and conflicting records refuse startup; a new boot
+requires empty persistent resources. Preparation retains journals and leases.
+Each subsequent execution still needs the full Sandbox isolation check.
 
-The read-only status observer may see a `ProtectSystem` mount as read-only. That
-does not prove the logger can write: readiness separately reads the fixed known
-daemon's kernel mountinfo, matches its mount device to the protected volume and
-requires both mount and superblock writability. Main PID, invocation and process
-start time are checked again to reject a changed process. No writable exception,
-ptrace/setns capability or wider Python PID visibility is added to observation.
-Underlying read interface: [Linux proc mountinfo implementation](https://github.com/torvalds/linux/blob/v6.18/fs/proc_namespace.c).
+## HTTPS entry
 
-Production API logging uses a closed event vocabulary and omits arbitrary library
-messages, arguments, paths and exception text. Uvicorn's separate error handlers
-are redirected through that formatter; installed API launchers disable access
-logging so request query strings are not copied into service logs. Development
-diagnostics remain separate. These ordinary formatting checks are not evidence
-of actual daemon, HTTP or upload log privacy; that is still a VM acceptance case.
-
-Build-input version 2 adds an explicitly fingerprinted public NGINX 1.30.5
-runtime catalog. Version 1 remains available for older engineering bundles without
-proxy support; there is no implicit proxy default. The separate `proxy_runtime`
-module permits only the binary and bounded shared libraries, excludes TLS and
-configuration files, verifies every byte/mode and ships them under `payload/proxy`.
-This separate library search tree avoids silently replacing existing application,
-database or language-image libraries. Binary version/signature/ELF-closure evidence
-is recorded by the operator; accepting a structural catalog is not TLS/server or
-Linux-safety acceptance. Actual proxy installation and execution remain separate.
+`generate-proxy` reads a 0600 profile containing `schema_version: 1`, canonical DNS
+`origin`, `certificate_sha256` and `key_sha256`. It generates a new NGINX snapshot
+and receipt for the verified release. It accepts fixed upstreams, paths and limits.
+Browser API families reach only API loopback; execution-control routes are refused.
+ZIP uploads use a separate canonical UUID route.
 
 `python3.13 -m scripts.release.proxy_install --source DIRECTORY
 --receipt-sha256 SHA --certificate PRIVATE_CERTIFICATE --private-key PRIVATE_KEY`
-installs the explicitly supplied protected proxy snapshot and matching unencrypted
-TLS files. It requires the same release/API-origin batch, a distinct locked
-UID/GID 1205 (`kf-proxy`) without supplementary groups, and an already provisioned
-private 128-MiB ext4 volume at `/var/lib/kf-onlinejudge/proxy`. Provision that
-volume's ownership/mode explicitly; the installer does not format, mount or
-recursively reassign unknown data. Existing TLS material must match its fingerprint;
-this entry does not silently rotate keys or claim DNS/public-browser trust.
+installs the selected release/API-origin snapshot and matching unencrypted TLS
+material at `/etc/kf-proxy/tls/fullchain.pem` and `key.pem`.
 
-Proxy modules separate public runtime verification, fixed service rendering,
-Root installation, ordinary launch guarding and read-only HTTPS observation.
-The ordinary process retains only `CAP_NET_BIND_SERVICE` for 443, no-new-privileges,
-private PID visibility and actual CPU/memory/swap/PIDs/file bounds. It receives
-no application DB/MQ/control credentials. Installation preserves older snapshots,
-refuses foreign system-unit pointers, reads back loaded/inactive metadata and
-starts no server. Maintenance includes the proxy in start/drain/stop ordering.
-Readiness uses fixed loopback TLS with the supplied certificate as an explicit
-local trust anchor and checks hostname/leaf identity plus the API live response;
-it does not substitute for actual browser CA/HTTPS/security acceptance in the VM.
-The exact `/api/v1/health/live` route permits GET/HEAD for this bounded process
-liveness check. Other health, internal, execution and control routes remain
-outside the public proxy allowlist; liveness does not attest judging readiness.
+Prepare a 128-MiB private ext4 volume at `/var/lib/kf-onlinejudge/proxy` and the
+distinct locked UID/GID 1205 (`kf-proxy`), without supplementary groups. The
+installer checks ownership/mode and matching existing TLS fingerprints; it does
+not format/mount data volumes, reassign unknown data or rotate keys implicitly.
 
-Private cold backups use a separate format from public release bundles.
-The private v3 format binds the stopped broker cookie and current DB bootstrap
-fingerprints and retains
-recovery history as another private domain. A restore can reject distribution-secret
-reuse without dropping the inactive configuration from an earlier recovery. It
-changes no public judging v1/v2 canonical bytes or historical report. The earlier
-private v1/v2 prototypes were never system-accepted and are refused by this format.
-`python3.13 -m kf_oj.infra.artifacts.backup` is an ordinary read-only artifact
-inventory command. Run it with the selected API database/storage identity after
-the owner-confirmed drain and while PostgreSQL is still available. It reads every
-ready artifact registry object through a repeatable-read, read-only transaction
-and hash-checks existing source/case bytes with a 64-KiB buffer. It creates no
-storage directory or lock and changes no metadata, grade, session or grant.
-Its output is private and does not itself prove that writers have stopped.
+The ordinary proxy keeps only `CAP_NET_BIND_SERVICE` for 443, no-new-privileges,
+private PID visibility and CPU/memory/swap/PIDs/file limits. It has no DB/MQ/control
+credentials. Maintenance includes it in start/drain/stop order. Readiness verifies
+loopback TLS with the supplied certificate, hostname/leaf identity and API live
+response. `/api/v1/health/live` allows GET/HEAD; other health/internal/control
+routes remain outside the allowlist. Browser CA trust needs separate verification.
+
+Build-input version 2 specifies the NGINX 1.30.5 catalog. Version 1 supports older
+bundles without a proxy. `proxy_runtime` checks every binary/library byte and mode,
+excludes TLS/configuration files and ships a separate tree under `payload/proxy`.
+Record package signature, version and ELF-closure verification with the inputs.
+
+## Logs
+
+`python3.13 -m scripts.release.journal` installs the fixed KF journal namespace
+profile on a preprovisioned 64-MiB ext4 volume at
+`/var/log/journal/MACHINE_ID.kf-onlinejudge`. It keeps the default system journal
+unchanged and rejects unknown configuration, overrides, old unrecorded data and
+partial installs.
+
+The profile uses 32-MiB persistent and 8-MiB runtime rotation budgets, bounded
+files/lines and seven-day retention. Its daemon has separate CPU/memory/swap/PIDs
+limits. Application, DB/MQ and Root units use this namespace and check the recorded
+profile and physical volume before starting.
+
+`check_ready` reads the daemon's process, cgroup and mount state. An observer's
+read-only `ProtectSystem` view does not establish daemon writability: the check
+matches the daemon's mount device to the protected volume and checks mount and
+superblock writability. It rechecks PID, invocation and start time to detect a
+changed process, without ptrace/setns or a wider observer PID view.
+
+Production API logs use fixed event names and omit arbitrary library messages,
+arguments, paths and exception text. Uvicorn errors use that formatter; installed
+API launchers disable access logging so queries are not copied to logs.
+Development diagnostics remain separate. Verify real routing, privacy and capacity
+on the installed node.
+
+## Readiness and maintenance
+
+`python3.13 -m scripts.release.node_health` checks admission preconditions in one
+fixed Root operator unit. Ordinary broker/factory/import observers have finite
+budgets; each checks identity, limits and no-new-privileges/capabilities. The CLI
+rejects unknown/concurrent helpers, reads back their stop and accepts no executable,
+account, port or command override.
+
+Checks include selected units and effective limits, DB-role/column privileges,
+the import owner's fresh policy-bound Unix-socket heartbeat and read-only broker
+queue/permissions/vhost queries. Coordinator supplies authenticated execution and
+cleanup counts through `/internal/operations/`. Truncated or unconfirmed counts
+cannot establish a drain. These internal routes require platform authority and
+return no submission IDs, source, archives or credentials.
+
+Factory health is a version-1 authenticated, rate/call-count-bounded observation.
+It reports supervision/delegation authorization, config digest, live helpers,
+remaining requests and lifetime. It performs no recovery, allocation, ledger write
+or limit repair. Request bytes remain unchanged. It reports
+`isolation_verified=false` and `execution_gate=required_per_execution`.
+
+Readiness requires confirmed empty execution/cleanup state and enough requests
+and lifetime for the maximum job. `--probe-notification` optionally sends one
+small confirmed notification for an identifier checked absent by Coordinator. It
+reads no body, creates no evaluation and runs no program. Periodic observation
+does not publish; probe failures are reported separately.
+
+`python3.13 -m scripts.release.lifecycle --action start|resume|drain|stop|restart`
+controls the fixed units and a Root-protected release/batch availability record:
+
+| Action | Behaviour |
+| --- | --- |
+| `start` | Requires stopped known application units; opens admission after readiness passes |
+| `resume` | Validates known running units after an interrupted operation |
+| `drain` | Closes new mutations, waits for owner-confirmed cleanup and leaves DB/MQ running |
+| `stop` | Completes the drain, then stops DB/MQ |
+| `restart` | Drains before a fresh start |
+
+API returns 503 for an absent, unknown, closed or wrong-bound availability record.
+Reads, logout, cancellation and receipt-bound archive uploads keep their existing
+identity/lease checks. Both startup paths keep admission closed until the opening
+is read back; failed final readiness closes it again.
+
+Drain keeps API available to accepted work and polls Coordinator, outbox, import,
+broker and factory within a finite deadline. Untruncated owner counts, stopped
+process groups, empty tasks/identity/capture resources and an absent factory socket
+must agree before quiescence is published. Timeout preserves rows, leases, queues
+and replay journals. Root uses its signal/guardian recovery path; import finishes
+its current authorized operation before claiming another. The maintenance receipt
+records manager operations and cleanup results.
+
+## Cold backups
+
+Collect the private artifact inventory with
+`python3.13 -m kf_oj.infra.artifacts.backup` after owner-confirmed drain, while
+PostgreSQL is available. Use the selected API DB/storage identity. A read-only,
+repeatable-read transaction checks ready registry objects and source/case digests
+with a 64-KiB buffer. The command creates no directory/lock and changes no state.
+
+Run `lifecycle --action stop` and verify processes have exited. A quiescent stop
+reuses only this boot's successful selected owner-derived drain receipt and
+rechecks application cgroups and Root task cleanup.
 
-After collecting the inventory, run `lifecycle --action stop`. For an already
-quiescent node it reuses only this boot's successful selected owner-derived drain
-receipt, rechecks every application cgroup and the Root task cleanup, then stops
-the selected database and broker. It does not query owners already stopped or
-adopt a stale, failed or unproven cut. Verify all their processes have exited.
 `python3.13 -m scripts.release.backup capture
 --backup-id bk-<32-lowercase-hex> --references PRIVATE_INVENTORY
---references-sha256 SHA` launches one fixed Root backup operator. It requires the
-same selected release/configuration, this boot's successful owner-derived cut,
-quiescent admission, empty task resources, no surviving service identities and
-clean PostgreSQL shutdown. The operator has only `CAP_DAC_READ_SEARCH`, no-new-
-privileges, 256-MiB memory, zero swap, 32 tasks, 50% CPU, a 256-MiB individual-file
-limit and a 30-minute deadline. Only the preprovisioned private backup volume is
-writable; this entry does not format/mount a volume or execute captured content.
+--references-sha256 SHA` captures a cold snapshot after checking the selected
+batch, owner-derived cut, quiescent admission, empty tasks, stopped service
+identities and clean PostgreSQL shutdown.
 
-Provision a separate Root-owned 0700 ext4 volume of 16 GiB at
+Prepare a separate Root-owned 0700, 16-GiB ext4 volume at
 `/var/lib/kf-onlinejudge/backups`, with `nosuid,nodev,noexec` and no shared mount.
-The capture retains all source domains: the cold PostgreSQL cluster, source and
-testcase store, private import staging, all retained release/image versions,
-application/dependency/proxy configuration, factory replay history and operation
-receipts. Every entry has a bounded digest/size and original identity/mode;
-captured copies remain Root-only. Registered source/case references must all
-match captured files. Source modification, links, special files, unknown
-identities, insufficient space or an incomplete cut are refused. A failed target
-is retained for explicit inspection and never published as a successful backup.
+The fixed backup operator has `CAP_DAC_READ_SEARCH`, no-new-privileges, 256 MiB,
+zero swap, 32 tasks, 50% CPU, a 256-MiB file limit and a 30-minute deadline. Only
+that volume is writable.
 
-The bound is 7 GiB and 50,000 entries per backup, at most two backup directories,
-with a seven-day declared retention period. Capacity refusal does not silently
-delete an old/unknown backup. `backup verify --backup-id ID
---manifest-sha256 SHA` checks an explicitly supplied private operator receipt and
-every private member; it performs no restoration or secret/epoch reuse. Actual
-pruning uses `backup prune --backup-id EXPIRED_ID --manifest-sha256 SHA
---keep-backup-id NEWER_ID --keep-manifest-sha256 SHA`. Both private inventories
-must verify completely; the named target must be expired and the preserved copy
-must be newer and unexpired. It removes only the known inventory under the
-maintenance lock, never the last verified copy, an unknown tree or a partial
-backup. Verification and pruning use the same bounded operator as capture.
-Controlled recovery into a clean node and measured recovery time require their
-separate implementation and independent VM acceptance. Capture, expiry pruning
-and interruption behavior also still require actual VM acceptance.
-Backup data and manifests contain private credentials/source/testcase data and
-must stay outside Git and CI artifacts. Parsing, ordinary adapter tests or a
-manifest digest are not cold-backup/recovery acceptance evidence.
+Backup domains include PostgreSQL, source/case storage, private import staging,
+retained releases/images, configuration, replay history and operation receipts.
+Every member records bounded digest/size and original identity/mode; copies remain
+Root-only. Registry references must match. Links, special files, changing sources,
+unknown identities, insufficient space and incomplete cuts are refused. Failed
+targets remain unpublished for inspection.
+
+| Limit | Value |
+| --- | --- |
+| Per snapshot | 7 GiB and 50,000 entries |
+| Stored snapshots | At most two |
+| Declared retention | Seven days |
+
+`backup verify --backup-id ID
+--manifest-sha256 SHA` checks the supplied private receipt and every member.
+`backup prune --backup-id EXPIRED_ID --manifest-sha256 SHA
+--keep-backup-id NEWER_ID --keep-manifest-sha256 SHA` verifies both inventories,
+requires an expired target and a newer unexpired retained copy, then removes only
+the named known inventory under the maintenance lock. It retains the last verified
+copy and refuses unknown/partial trees. Capacity refusal deletes nothing.
+
+Private v3 backups bind the stopped broker cookie, current DB bootstrap fingerprint
+and recovery history. Earlier private v1/v2 prototypes are refused. These private
+versions do not alter judging canonical bytes or historical reports. Keep all
+backup data and manifests out of Git and CI artifacts.
+
+## Restore to a clean node
+
+Install a verified public bundle on a different clean boot/node. Generate and
+select a newly attested batch, then install dependencies with a new broker cookie
+on empty provisioned volumes. Keep services stopped and do not initialize a new
+cluster over restored data. Use fresh Worker UUID, measurement nonce/epoch,
+control tokens and all DB/MQ passwords.
 
 `python3.13 -m scripts.release.restore --recovery-id rr-<32-lowercase-hex>
---backup-id ID --manifest-sha256 SHA` performs the cold **data** step on a
-different clean boot/node. First install a verified public bundle, generate and
-select an attested fresh configuration, and explicitly install the dependency
-snapshot with a new broker cookie on empty preprovisioned volumes. Keep all
-services stopped; do not initialize a new PostgreSQL cluster over restored data.
-The target must have new Worker UUID, measurement nonce/epoch, control tokens
-and all DB/MQ passwords. Cold restoration currently requires identical migration
-bytes, target platform/toolchains and runtime images; schema-changing upgrades
-need their separate Backend Alembic procedure.
+--backup-id ID --manifest-sha256 SHA` runs the cold data step. Compatibility
+requires identical migration bytes, target platform/toolchains and runtime images;
+schema-changing upgrades need a separate Backend Alembic procedure.
 
-The fixed restore operator reads the backup volume through a read-only mount
-view, under 512-MiB memory, zero swap, 32 tasks, 50% CPU and a 30-minute deadline.
-It verifies the separate private receipt and every source byte before copying
-to fixed targets. It preserves original data identities/modes and every retained
-factory replay ID; a new boot's existing Sandbox preparation owns delegation
-recreation. It copies old application/judge/dependency/proxy configuration and
-maintenance receipts into private inactive `recovery/ID/history/`, never over
-the selected current configuration. Unknown/nonempty data targets and differing
-existing files are refused. Failed partial data stays stopped for explicit
-inspection; it is never silently adopted or erased.
+The operator reads the backup through a read-only mount, checks the private receipt
+and every byte, then copies to fixed targets under 512 MiB, zero swap, 32 tasks,
+50% CPU and a 30-minute deadline. Original identities/modes and replay IDs are
+preserved. Old configuration and maintenance receipts go into inactive
+`recovery/ID/history/`. The new boot's Sandbox preparation owns cgroup recreation.
+Unknown/nonempty targets and differing existing files are refused; partial restores
+remain stopped for inspection.
 
-The `data-restored.json` receipt proves only byte/identity restoration. The
-operator starts no service, changes no schema/grant/grade/lease and reports DB
-credential rotation and execution verification as false.
+`data-restored.json` records only byte/identity restoration. Services stay stopped
+and DB credential rotation and execution verification remain false.
+
+### Rotate database credentials
 
 `python3.13 -m scripts.release.recovery_database --recovery-id ID
 --data-receipt-sha256 SHA --password-file PRIVATE_NEW_BOOTSTRAP
---password-sha256 SHA` dispatches the credential step only after that boot's
-selected complete data receipt. Supply a new 256-bit bootstrap password; its
-fingerprint must differ from the separately verified private backup's current
-bootstrap fingerprint. Backup capture requires one complete initialization or
-recovery marker for the actual cluster, so a restored node can be backed up again.
-Current
-service endpoints are loaded from the new protected configuration, not backup
-URLs. Missing/different authority, an active cluster, unknown unit or incomplete
-cleanup is refused; this is an explicit operation, not service startup behavior.
+--password-sha256 SHA` requires this boot's selected complete data receipt and a
+new 256-bit bootstrap password with a different fingerprint from the backup.
+Backup capture requires a complete initialization or recovery marker for the
+actual cluster, allowing a restored node to be backed up again. Endpoints come
+from current protected configuration.
 
-One ordinary `kf-postgres` temporary unit gets 512-MiB memory, zero swap, 64
-tasks, 100% CPU, a four-minute deadline, cap0/no-new-privileges and private PID/
-network/runtime namespaces. It uses a private 0700 socket directory with TCP
-listening disabled. The [PostgreSQL peer mechanism](https://www.postgresql.org/docs/18/auth-peer.html)
-maps only this local OS identity to the fixed maintenance database login; fixed
-service logins use SCRAM for their credential checks. No trust authentication,
-external database listener or runtime Root business credential is introduced.
+One ordinary `kf-postgres` temporary unit uses 512 MiB, zero swap, 64 tasks, 100%
+CPU, a four-minute deadline, cap0/no-new-privileges and private PID/network/runtime
+namespaces. TCP is disabled; the private 0700 socket uses local peer authentication
+for this OS identity only. Fixed service logins use SCRAM.
 
-The process checks effective PostgreSQL memory/time/socket/logging settings and
-the exact existing roles/memberships/database owner. It rotates only passwords
-for the five fixed service logins and `postgres` in one transaction, verifies
-unchanged role authority and authenticates each new credential. It grants no
-membership, creates no database/table, invokes no business repair and does not
-rewrite grades, leases or measurement history. SQL/error output containing
-utility statements is suppressed. Temporary-server exit and empty resources
-must be confirmed before the dispatcher writes `identity-recovered.json`.
+The process checks PostgreSQL settings, roles, memberships and database ownership.
+It rotates the five service logins and `postgres` in one transaction, preserves
+authority and authenticates every new credential. It changes no schema, grade,
+lease or measurement history. SQL/error utility text is suppressed. Confirm
+server exit and empty resources before publishing `identity-recovered.json`.
 
-Startup and credential recovery share the cold data step's fixed lock. Startup
-holds a read-only shared lock through admission and validates the linked prepared,
-data and identity receipts in each bounded Root-private recovery directory.
-Missing/incomplete/foreign receipts refuse startup; a committed password change
-without a verified shutdown receipt cannot open admission. Complete historical
-receipts remain valid after reboot or a later selected version; they do not make
-the current services ready. Previous recovery history is backed up and restored
-into inactive history, never used as the new node's completion receipt.
+Recovery and startup share a fixed lock. Startup holds a read-only shared lock
+through admission and validates linked prepared/data/identity receipts in the
+bounded Root-private recovery directories. Missing/incomplete/foreign receipts
+keep admission closed, including a committed credential change without confirmed
+shutdown. Historical receipts remain valid after reboot or version change but do
+not establish current readiness. They stay in inactive history on later restores.
 
-Recovery still needs Backend Alembic verification, new TLS inputs, service
-readiness and actual four-language/history checks in the second VM. Neither
-data nor credential receipts assert execution safety or healthy public service.
-These system steps and the 30-minute end-to-end recovery target remain pending
-independent VM acceptance.
+Finish recovery with Backend Alembic verification, new TLS material, service
+readiness and actual four-language/history checks on the second VM. The 30-minute
+end-to-end target still requires measured acceptance.
 
-`python3.13 -m scripts.release.node_observation` dispatches one finite,
-read-only local observation through its fixed Root cap0/no-new-privileges helper
-(128 MiB, zero swap, 16 tasks, 25% CPU, 120 seconds, single named unit).
-It reads kernel memory/PIDs current and peak, CPU throttling, memory events,
-aggregate device IO and filesystem bytes/inodes without enumerating private
-source, hidden cases or credential files. Current selected service identities,
-ACLs, execution leases/cleanup, broker/outbox/import backlog and finite factory
-capacity come from the existing status owners. Notification probing stays off;
-it does not publish, claim, acknowledge, close admission or repair state.
+## Observe resources and queues
 
-The result includes source/configuration binding, a sample time and observation
-duration. Independent readings are a bounded time window, not an atomic backup
-cut. Missing kernel fields, stopped services or failed owner queries are explicit
-incomplete observations; there is no invented zero/healthy fallback. Busy
-admission can remain false in a complete observation. `isolation_verified`
-remains false: every real execution still needs its own Sandbox gate. Private
-operational output is for the local operator and must stay outside public API,
-Git and CI artifacts. Actual resource/queue observations and long-run evidence
-still require the independent VM.
+`python3.13 -m scripts.release.node_observation` uses a finite read-only Root cap0,
+no-new-privileges helper: 128 MiB, zero swap, 16 tasks, 25% CPU, 120 seconds and one
+named unit. It reads memory/PIDs current/peak, CPU throttling, memory events, device
+IO and filesystem bytes/inodes without enumerating private content. Existing owners
+supply service identities, ACLs, leases/cleanup, broker/outbox/import backlog and
+factory capacity. Notification probing stays off.
+
+Output includes source/batch, sample time and duration. Readings cover a bounded
+time window, not an atomic backup cut. Missing fields, stopped services and failed
+owner queries are explicit incomplete observations. Busy admission can be false
+in a complete sample; `isolation_verified` remains false. Keep operational output
+outside public API, Git and CI artifacts.
+
+## References
+
+- PostgreSQL: [role membership](https://www.postgresql.org/docs/18/role-membership.html),
+  [function ownership](https://www.postgresql.org/docs/18/sql-alterfunction.html),
+  [initdb](https://www.postgresql.org/docs/18/app-initdb.html),
+  [peer authentication](https://www.postgresql.org/docs/18/auth-peer.html),
+  [resources](https://www.postgresql.org/docs/18/runtime-config-resource.html),
+  [privilege functions](https://www.postgresql.org/docs/18/functions-info.html).
+- RabbitMQ: [limits](https://www.rabbitmq.com/docs/limits),
+  [definition import](https://www.rabbitmq.com/docs/definitions),
+  [access control](https://www.rabbitmq.com/docs/access-control),
+  [CLI](https://www.rabbitmq.com/docs/man/rabbitmqctl.8).
+- NGINX: [core](https://nginx.org/en/docs/http/ngx_http_core_module.html),
+  [proxy module](https://nginx.org/en/docs/http/ngx_http_proxy_module.html),
+  [TLS module](https://nginx.org/en/docs/http/ngx_http_ssl_module.html).
+- systemd: [credentials](https://systemd.io/CREDENTIALS/),
+  [namespace interface](https://github.com/systemd/systemd/blob/v262/man/systemd.exec.xml),
+  [journal configuration](https://github.com/systemd/systemd/blob/v262/man/journald.conf.xml).
+- Linux: [proc mountinfo](https://github.com/torvalds/linux/blob/v6.18/fs/proc_namespace.c).