diff --git a/deploy/README.md b/deploy/README.md index db01f0d..11d980f 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -1,10 +1,12 @@ # 本地开发依赖 -管理员账号 bootstrap/恢复、有界题包解析服务、不可变发布/下架、审计与回收步骤见 -[题目维护运维说明](admin-problems.md)。该说明与服务模板用于受控部署准备，不表示已部署生产。 +开发环境用 Compose 提供数据库和消息队列。单机交付使用独立的受限服务，安装入口见 +[单机交付说明](release/README.md)。管理员账号、题包导入、发布、下架和审计见 +[题目维护运维说明](admin-problems.md)。 -`compose.dev.yml` 只提供 PostgreSQL 和 RabbitMQ，端口绑定 localhost。 -先复制 `.env.example` 为 `.env`，然后在仓库根目录运行： +## 启动开发依赖 + +将 `.env.example` 复制为 `.env`，在仓库根目录运行： ```bash make infra-config @@ -12,75 +14,73 @@ make infra-up make infra-down ``` -`infra-down` 保留数据卷；不要把示例口令用于正式部署。 -API 存活检查和前端预览不依赖这些容器，可以先独立启动。 -该 Compose 不运行判题 Worker 或不可信代码。 +`compose.dev.yml` 只启动 PostgreSQL 和 RabbitMQ，端口绑定 localhost。 +`infra-down` 保留数据卷。API 存活检查和前端预览可以先独立启动。 +不要在这个 Compose 或宿主机上运行学生代码，也不要将示例口令用于正式部署。 -镜像固定具体维护版本（PostgreSQL 18.6、RabbitMQ 4.3.6），正式实验前另锁镜像摘要。 -配置检查只验证 Compose 结构；成功启动、实际健康与备份恢复需要单独验证。 -后续受控主机服务和最小权限沙箱启动器配置放在本目录。 +开发镜像固定 PostgreSQL 18.6 和 RabbitMQ 4.3.6；正式实验还需锁定镜像摘要。 +配置检查只检查 Compose 结构。服务健康和备份恢复要在实际环境验证。 +镜像用法见官方 [PostgreSQL](https://hub.docker.com/_/postgres) 和 +[RabbitMQ](https://hub.docker.com/_/rabbitmq) 说明。 -官方镜像说明：[PostgreSQL](https://hub.docker.com/_/postgres)、 -[RabbitMQ](https://hub.docker.com/_/rabbitmq)。 +## 单机 helper 工厂 -### 单机 helper 工厂 +`kf-sandbox factory --config /etc/kf-judge/runtime.json` 启动 Root 工厂。 +普通 Worker 连接配置中的 `state_root/factory/control.sock`。 +安装 `deploy/kf-sandbox-factory.service` 示例前，管理员需核对固定二进制、不可写的 +GNU 镜像与配置、Worker UID/GID，以及 Root 所有且与监督器 cgroup 分离的任务委派根。 +工厂读回服务和任务预算，每次编译、测试点运行都经过完整隔离检查。 -`kf-sandbox factory --config /etc/kf-judge/runtime.json` 是 Root 服务入口； -普通 Worker 使用配置中 `state_root/factory/control.sock` 的普通用户客户端。 -`deploy/kf-sandbox-factory.service` 给出有限服务预算示例，安装前由可信管理员 -核对固定二进制、不可写 GNU 镜像/配置、Worker UID/GID，以及**与服务自身 cgroup -分离**的 Root 所有任务委派根。工厂实际读回服务 cgroup 限额，应用/读回任务总预算； -每次编译和测试点仍须通过已有完整隔离门控。服务文件本身不证明这些条件成立。 +工厂无等待队列，只接收有界 run ID。槽位满、重复 ID 和未知字段立即拒绝。 +每次请求创建全新的单线程 helper，继承已认证连接和工厂生命线。 +额外控制帧、断连或生命线消失会终止任务树。ID 先进入持久记录，最多 4096 个， +跨重启继续拒绝重放。清理失败时保留身份租约并停止分配；恢复只处理已记录的 +归属和 inode，未知资源保留并拒绝启动。 -工厂只接收有界 run ID，无排队：槽位满、重复 ID 或未知字段立即拒绝。 -每次申请使用全新单线程 helper，固定继承已认证的连接与工厂生命线； -额外控制帧、断开或生命线消失会终止整棵任务树。ID 先写入有限持久记录 -（最多 4096 个，跨重启拒绝重放）；容量用尽需要管理员在所有任务停止、已核对 -回收后安排新记录周期，不自动删除记录。清理失败保留身份租约并停止新分配； -重启只根据已记录归属和 inode 回收，未知/外来内容保留并拒绝启动。 +### 轮换记录周期 -工厂 `--max-requests` 由可信 Root 启动参数选择，范围1–4096，默认4096； -它限制当前记录周期的受理次数，不改变学生预算或已记录 ID 的拒绝规则。 -`kf-sandbox rotate-factory-cycle --config ROOT_CONFIG --cycle-id ID` 是独立的 -Root 维护入口：在受限监督器作用域中，核对受保护配置/镜像、取得工厂排他锁， -并读回任务、身份、捕获目录和委派 cgroup 为空、控制 socket 已消失后，才把 -当前请求记录保留到 Root 私有 `factory/cycles/`，fsync文件与目录后清空当前记录。 -最多保留64个有界周期；历史 ID 在后续启动仍用于拒绝重放。正常启动、重启、 -寿命结束和容量用尽不会调用轮换或删除任何历史记录。归档完成但当前记录未清空 -的中断状态拒绝启动，需在停止/排空后显式使用同一个 cycle ID 重试；未知或 -不完整记录保留并拒绝，不能通过删日志恢复。该入口不启动服务，也不证明隔离。 -新安装路径下的实际轮换、崩溃恢复、跨周期重放拒绝与资源读回仍需独立 VM 验收。 +可信 Root 参数 `--max-requests` 设置本周期受理上限，范围 1 到 4096，默认 4096。 +容量用尽后，先停止任务并核对回收，再由管理员维护记录周期。 -`kf-worker judge-submission` 是受控本地适配入口。常驻 `kf-worker serve --config` -使用有界 AMQP 通知和经过服务身份认证的 Coordinator claim；配置与依赖见 -[判题工程说明](../judge/README.md#单机持久化协调与常驻-worker)。 -它验证不可变清单摘要，C17/C++20 仅映射到固定 GNU 命令，编译一次； -成功产物经完整清理后封印并计算 SHA-256，每点分别申请 helper、stdin 与工作区。 -期望值逐点读取，留在比较端，不传给工厂/用户文件系统。 -新增锁定 `sha2 0.10.9` 仅供 Worker 验证输入及编译产物摘要； -Sandbox 同版本仅为独立 VM 测试数据生成的开发依赖，Engine 不依赖加密或 OS 库。 +`kf-sandbox rotate-factory-cycle --config ROOT_CONFIG --cycle-id ID` 在受限监督器作用域 +中核对配置和镜像、取得工厂排他锁，确认任务、身份、捕获目录、委派 cgroup 均为空， +且控制 socket 已消失，才归档请求记录到 Root 私有 `factory/cycles/`。 +归档文件和目录经 fsync 后清空当前记录，最多保留 64 个周期；历史 ID 继续用于拒绝重放。 -### 单机提交服务的启动与停止 +正常启动、重启、寿命结束和容量用尽不会自动轮换或删除历史记录。 +若归档完成后中断，先排空，再显式用同一 cycle ID 重试。未知或不完整记录需要检查， +不能删日志续用工厂。轮换入口不启动服务。新安装路径的轮换、崩溃恢复和跨周期拒绝 +仍需独立 VM 验收。 -1. Backend Alembic 迁移；创建分别仅继承平台/协调器 NOLOGIN 角色的专用登录账号， - 平台角色只可插入初始评测，执行状态仅由 Coordinator 写入。 -2. 普通平台服务持有私有工件目录；使用应用服务导入不可变题目和创建测试账号。 -3. 验证固定 GNU 镜像、Root 配置及工厂服务/任务 cgroup 的真实上限，再启动工厂。 -4. 独立启动 API、Coordinator、outbox publisher 和普通 Worker；单机 HTTP/AMQP - 仅数值 loopback，PostgreSQL loopback/Unix socket。任何入口都不自动创建表。 -5. Coordinator readiness 验证数据库和权限；仍需独立 VM 验证真实执行门控。 - Root 工厂不可持有平台/MQ/数据库口令，用户运行镜像不含服务文件。 -6. 先向 Worker 发送 SIGTERM，使当前任务取消/清理，再停止 publisher/API/Coordinator， - 最后停止工厂和本地依赖。强制死亡后的旧 attempt 必须撤销并取得实际停止证明； - `cleanup_failed` 不等于普通终态，先核对保留资源再由管理员恢复工厂/Worker。 +## 单机提交服务的启动与停止 -平台配置私有 `KF_OJ_ARTIFACT_ROOT`、有界字节/文件容量以及 -`KF_OJ_JUDGE_ADMISSION`（完整可信 JSON）；其工具链、编译/运行上限、总体作业期限、 -并发/prefetch/retry 必须与 Coordinator/Worker 一致。单机使用并发1、prefetch1， -不能仅提高 Worker 数量绕过节点容量。源码最多256KiB，测试点最多256个； -stdout/stderr 各不超过 file_bytes，合计不超过 workdir_bytes。 +`kf-worker judge-submission` 是受控本地适配入口；常驻 +`kf-worker serve --config` 使用有界 AMQP 通知和认证后的 Coordinator claim。 +配置见[判题工程说明](../judge/README.md#单机持久化协调与常驻-worker)。 -迁移账号与平台账号分开配置，命令均在 `backend/` 运行： +Worker 校验不可变清单摘要，将 C17/C++20 映射到固定 GNU 命令，编译一次。 +产物经完整清理后封印并计算 SHA-256；每个测试点单独申请 helper、stdin 和工作区。 +期望输出留在比较端。Worker 用 `sha2 0.10.9` 校验输入与产物；Sandbox 的同版本 +开发依赖仅生成独立 VM 测试数据，Engine 不依赖加密或 OS 库。 + +### 启动顺序 + +1. 用独立迁移账号执行 Backend Alembic。平台与协调器登录账号分别继承各自的 + NOLOGIN 角色；平台只插入初始评测，执行状态由 Coordinator 写入。 +2. 为平台准备私有工件目录，通过应用服务导入不可变题目并创建测试账号。 +3. 验证 GNU 镜像、Root 配置、工厂服务和任务 cgroup 的实际限额，启动工厂。 +4. 独立启动 API、Coordinator、outbox publisher 和普通 Worker。HTTP/AMQP 使用 + 数值 loopback，PostgreSQL 使用 loopback/Unix socket；启动入口不创建表。 +5. 检查 Coordinator 数据库权限与 readiness，再在独立 VM 验证执行隔离。 + Root 工厂不持有平台/MQ/数据库口令，用户运行镜像不含服务文件。 + +平台配置包括私有 `KF_OJ_ARTIFACT_ROOT`、工件容量和可信 +`KF_OJ_JUDGE_ADMISSION`。工具链、编译/运行预算、作业期限、并发/prefetch/retry +必须与 Coordinator/Worker 一致。单机并发 1、prefetch 1，共享任务身份池。 +源码最多 256 KiB，测试点最多 256 个；stdout/stderr 各不超过 file_bytes， +合计不超过 workdir_bytes。增加 Worker 数量不会扩大节点槽位。 + +以下开发命令均在 `backend/` 运行；迁移账号与平台账号分别配置： ```sh uv run --locked alembic upgrade head # 私有迁移账号，仅执行受控 schema 操作 @@ -91,70 +91,77 @@ uv run --locked uvicorn kf_oj.api.app:create_app --factory --host 127.0.0.1 --po uv run --locked python -m kf_oj.infra.messaging.publisher ``` -每个账号操作/导入目录都由管理员准备，不从浏览器接受私有路径。导入格式见 -[题目模块](../backend/src/kf_oj/modules/problems/README.md)，不可变版本以 UUID -在题库/题面链接和提交请求中传递。更新题目需导入新版本，不覆盖已有提交的题目版本。 -临时验收使用合成数据和测试口令，实际业务数据与服务配置都保存在仓库外的私有目录。 +账号和导入目录由管理员准备，浏览器不能传入私有路径。导入格式见 +[题目模块](../backend/src/kf_oj/modules/problems/README.md)。题库、题面和提交使用 +不可变版本 UUID；更新需发布新版本。临时验收使用合成数据和测试口令，业务数据与 +配置放在仓库外。 -取消另需 API 私有配置 `KF_OJ_COORDINATOR_URL=http://127.0.0.1:8100` 与 -`KF_OJ_COORDINATOR_TOKEN`（Coordinator 的 platform_token）。所有服务令牌私下生成， -不可互用或放进前端环境。Coordinator/Worker 私有 JSON 为0600、单链接、有界文件； -Root 镜像/配置/二进制由 root 持有，任务委派子树独立于监督器自身 cgroup。 -运行前实际读回整体节点、服务和任务 memory/swap/pids/CPU/文件/运行期限上限， -并通过真实隔离门控；配置解析、readiness、存活探测都不能替代执行验收。 +取消功能还需 API 配置 `KF_OJ_COORDINATOR_URL=http://127.0.0.1:8100` 和 +`KF_OJ_COORDINATOR_TOKEN`，后者对应 Coordinator 的 platform_token。 +独立生成服务令牌，保存在私有配置中。Coordinator/Worker JSON 为 0600、单链接、 +有界文件；Root 持有镜像、配置和二进制。运行前读回节点、服务和任务的 +memory/swap/pids/CPU/文件/期限上限。 -前端从 `frontend/` 启动 `pnpm dev`，默认127.0.0.1:5173；平台明确允许相同浏览器 -Origin，`/api` 同源代理。生产需另行配置 HTTPS、安全 cookie、SPA 回退与 `/api` -反向代理，本仓库不自动部署。用户以受控账号登录，在题面选择支持语言、提交源码， -随后查看同一 submission UUID 的持久化状态。等待/取消中/系统失败都不等于成功成绩。 +开发前端在 `frontend/` 运行 `pnpm dev`，默认 127.0.0.1:5173。 +平台明确允许相同 Origin，`/api` 使用同源代理。生产入口还需 HTTPS、安全 Cookie、 +SPA 回退和 API 反向代理。登录后在题面提交源码，按同一 submission UUID 查看状态； +等待、取消中和系统失败各有独立状态。 -broker 暂停时 API 仍能持久化 queued/outbox；恢复后 publisher confirm 与 Coordinator -补偿重新投递。Worker/Coordinator 重启使用原私有配置、工件、日志与 DB，不删除 -attempt 或强改状态。旧租约不能覆盖新结果，有限基础设施重试最终收敛。 -容量满时返回429或503；先确认待执行/清理状态，恢复服务或由管理员扩充经过核对的 -存储容量，不删除仍被提交/题目引用的工件。`kf-artifacts --limit 100` 仅回收满足年龄 -要求的无引用工件和已标记删除项；它不会通过删除用户源码来腾出容量。 -恢复 Root 必须核对 journal/inode/实际排空；未知资源保留，管理员处理自己确认的 -故障来源后再重启。不要递归清空任务目录，也不要把 cleanup_failed 改成已成功。 +### 停止与故障恢复 -这些入口和顺序不表示生产部署已完成。隔离/超限与用户程序仅在无 NIC、无宿主共享 -挂载、无业务凭据且整体上限已读回的独立测试 VM 内验收。开发 Compose 仍只提供可信 -数据库和 broker，不在宿主或业务服务器运行用户代码。workflow 如需启用由指定 -GitHub 私有仓库承载，并设置有限并发；Gitea 只保存逐阶段工程提交。 +先向 Worker 发送 SIGTERM，使当前任务取消并清理，再停止 publisher/API/Coordinator， +最后停止工厂和本地依赖。强制死亡后的旧 attempt 必须撤销并取得停止证明。 +`cleanup_failed` 保留未确认资源，管理员检查故障来源后再恢复工厂/Worker。 +不要递归清空任务目录或强改结果。 +broker 暂停期间，API 保留 queued/outbox；恢复后 publisher confirm 与 Coordinator +补偿重新投递。重启 Worker/Coordinator 时保留配置、工件、日志、DB 和 attempt。 +旧租约不能覆盖新结果，基础设施重试有次数上限。 + +容量满时返回 429 或 503。先检查待执行和清理状态，再恢复服务或核对扩容。 +`kf-artifacts --limit 100` 只回收达到年龄要求的无引用工件和已标记删除项， +保留仍被题目或提交引用的源码。Root 恢复还需核对 journal、inode 和实际排空。 ## 四语言节点的受控升级 -新增公开示例 `judge.admission.example.json` 为显式 v1/v2 注册表， -`sandbox.managed.runtime.example.json` 为一槽位的共享 Root 配置；原 v1 示例保持原语义。 -所有 `aaaaaaaa…` 镜像摘要均是无权限占位值，不能当作已验镜像。管理员制作固定镜像后， -对 manifest.json 的实际字节计算 SHA-256，逐文件验证版本/字节/权限，并将相同完整身份 -写入准入注册表与 Root runtime_images。Root 镜像路径仅在私有可信配置中保存。 -配置解析通过或 check-runtime 可见仍不允许跳过实际隔离门控。 +`judge.admission.example.json` 是显式 v1/v2 注册表， +`sandbox.managed.runtime.example.json` 使用一槽位共享 Root 配置。 +原 v1 示例语义保留。示例中的 `aaaaaaaa…` 是无权限占位摘要，必须换成固定镜像 +manifest.json 的实际 SHA-256，并逐文件验证版本、字节和权限。 +准入注册表与 Root runtime_images 使用同一完整身份；Root 镜像路径仅放入私有配置。 -1. 停止创建新提交/消费投递，向所有普通 Worker 发 SIGTERM；保留 DB、清单、工件和 journal。 -2. 取得工厂实际 stopped/cleanup_complete 证明，读回空任务 cgroup 和 owned 资源； - 未排空或存在未知资源时停止升级，保留证据，不递归清空任务目录。 -3. 使用独立迁移账号执行 Backend Alembic upgrade；新语言数据存在时旧迁移降级会拒绝， - 不删除数据以实现回退。平台与 Coordinator 继续使用各自最小权限角色，Worker 无 DB 账号。 -4. 安装固定版本的 Backend/Coordinator/Worker/Sandbox；同时更新三个普通服务的同一 - `judge_admission`/`admission` 注册表，Root 配置单独由管理员持有。旧消费者不混用新协议。 -5. 为整体节点、控制面各服务、工厂和任务设置有限 CPU/memory/swap/pids/文件/期限， - 读回内核值。节点至少预留一份512MiB编译任务以及独立控制面；Java运行示例总预算384MiB， - Xmx不能代替整体预算。并发1/prefetch1，共享UID池，不因第二Worker存在而扩大节点槽位。 -6. 在独立无NIC、无宿主共享挂载、无业务凭据 VM 用合成题目验证四语言、故障与实际排空， - 再恢复工厂/Coordinator/API/publisher/Worker。生产启用需单独安排，仓库不自动部署。 +1. 停止创建提交和消费投递，向普通 Worker 发 SIGTERM，保留 DB、清单、工件和 journal。 +2. 确认工厂 stopped/cleanup_complete，读回空任务 cgroup 和 owned 资源。 + 发现未知资源或未排空时停止升级并保留证据。 +3. 用独立账号执行 Backend Alembic upgrade。含新语言数据时旧迁移拒绝降级； + 保留这些数据。平台与 Coordinator 使用各自最小权限角色，Worker 无 DB 账号。 +4. 安装固定版本 Backend/Coordinator/Worker/Sandbox，同时更新普通服务的同一 + `judge_admission`/`admission`。Root 配置单独保管，旧消费者不混用新协议。 +5. 设置并读回节点、控制面、工厂和任务的 CPU/memory/swap/pids/文件/期限上限。 + 至少预留一份 512 MiB 编译任务和独立控制面；Java 运行示例总预算 384 MiB， + Xmx 只是堆上限。并发 1、prefetch 1，使用共享 UID 池。 +6. 在断网、无宿主共享挂载、无业务凭据的受限独立 VM 验证四语言、故障和排空， + 再恢复服务。生产启用单独安排。 -请求不在线安装 Python/Java 依赖，不运行 pip/uvx，不接收自定义解释器参数/JAR/agents。 -Worker 私有配置可设置 `control_timeout_ms`（1000–5000，缺省2000）作为 Coordinator -单次控制请求的期限。资源受限节点应依据实际领取/续租/停止耗时选择，并核对租约和 -停止恢复；这不改变学生阶段/作业期限、重试次数或工件下载的固定2秒期限，连接仍限1秒。 -Python main.py 仅标准库；Java Main.java 使用默认包 Main，最多256个有界编译class。 -题目语言与预算以公开版本配置为准，旧题目和草稿保留，Rust继续明确不支持。 -新版检查阶段CE/运行RE/真实限额与SystemError的区别见 [判题说明](../judge/README.md#python--java-与明确协议版本)。 +Python `main.py` 仅支持标准库。Java `Main.java` 使用默认包 Main，最多 256 个 +有界编译 class。请求不在线安装依赖，不运行 pip/uvx，不接收解释器参数、JAR 或 agents。 +题目版本决定语言和预算，旧题目、草稿和报告保留；Rust 暂不支持。 +CE、RE、真实超限与 SystemError 的区别见 +[判题说明](../judge/README.md#python--java-与明确协议版本)。 -停止/故障恢复沿原单机流程：停止Worker使任务全树排空，旧租约/report拒绝后恢复； -publisher/broker中断保留outbox，有限重试而非清空记录。恢复旧软件前应先排空v2任务， -保留不可变v2版本和报告供兼容消费者读取；不能把旧配置静默转换为宽松新能力。 -这些公开配置不含业务令牌。数据库/AMQP/服务令牌分别在仓库外私下生成，Root不持有它们。 -需要CI时只在GitHub私有仓库 `KiriAky107/KF-OnlineJudge` 运行并设置有限并发，Gitea保存源码。 +Worker 的 `control_timeout_ms` 可设为 1000 到 5000，缺省 2000。 +按实际领取、续租和停止耗时选择，并核对租约与恢复流程。 +该值不改变学生阶段/作业期限、重试次数、工件下载的固定 2 秒期限或 1 秒连接期限。 + +回退旧软件前先排空 v2 任务，保留 v2 不可变版本和报告供兼容消费者读取。 +恢复时沿用停止与故障流程，保持 outbox 和有限重试。 +公开示例不含业务令牌；数据库、AMQP 和服务令牌分别在仓库外生成，Root 不持有它们。 + +## 验收与 CI + +所有学生源码、隔离、超限和故障测试都在无 NIC、无宿主共享挂载、无业务凭据且 +整体上限已读回的独立 VM 运行。配置解析、readiness 和存活探测不能替代执行验收。 +仓库不自动部署生产。 + +需要 workflow 时，在 GitHub 私有仓库 `KiriAky107/KF-OnlineJudge` 设置有限并发执行。 +Gitea 保存工程提交，不运行 workflow。 diff --git a/deploy/release/README.md b/deploy/release/README.md index cfb897c..95bc01d 100644 --- a/deploy/release/README.md +++ b/deploy/release/README.md @@ -1,107 +1,134 @@ -# Single-node delivery contract +# Single-node delivery -`contract.v1.json` describes the first pre-release installation interface. It is -separate from judging v1/v2 wire formats and never rewrites historical reports. -The reference target is Arch/CachyOS Linux x86_64-v3, glibc 2.44, systemd and cgroup -v2. Images and service libraries are bound to explicit file/catalog fingerprints; -support for other distributions requires a separately verified delivery profile. +The single-node profile packages the application, four judging runtimes and the +installation tools for offline use. `contract.v1.json` defines the installation +interface. Judging v1/v2 reports keep their existing formats. -The target has at least 4 GiB RAM and 12 GiB installation/data capacity. Judging -and problem-import concurrency remain one each. Configured service and execution -budgets, effective kernel limits and free capacity must also pass preflight; total -RAM alone does not prove enough capacity or verified execution isolation. -Preflight requires at least 12 GiB of filesystem capacity and derives required -free bytes from the verified application, protected runtime-image copies and -separate GNU legacy copy, plus 4 GiB of mutable-data/operation reserve. It does -not assume that a nominal 12-GiB filesystem exposes 12 GiB free after metadata. +Full installation and recovery acceptance is in progress. Validate a release on +an independent test node before using it for production. -Release payloads are directory bundles with a bounded manifest: at most 50,000 -single-link regular files, 256 MiB per file, 4 GiB total and 8 MiB metadata. The -manifest SHA-256 is supplied separately by the operator. Links, extra members, -private paths and unexpected file changes are refused. Built payloads and real -configuration never belong in Git. Validation is read-only and does not claim -that any service has been installed, started or verified: +## Requirements + +| Item | Required profile | +| --- | --- | +| OS | Arch/CachyOS, Linux x86_64-v3, glibc 2.44, systemd 262, cgroup v2 | +| Memory | At least 4 GiB, with enough free capacity for the configured services and tasks | +| Storage | At least 12 GiB filesystem capacity; preflight calculates the required free space | +| Concurrency | One judging job and one problem import | +| Dependencies | PostgreSQL 18.6 and the fixed broker runtime, prepared from verified offline materials | + +Other distributions need a separately verified profile. Preflight checks the +effective kernel limits and available capacity as well as nominal RAM and disk +size. Its free-space calculation includes the application, protected runtime +images, the separate GNU legacy image and 4 GiB for mutable data and operations. + +## Verify a bundle + +Use a separately trusted checker and the manifest digest supplied with the release: ```sh python3.13 -m scripts.release contract deploy/release/contract.v1.json python3.13 -m scripts.release verify /path/to/bundle --manifest-sha256 SHA256_FROM_RECEIPT ``` -Application versions live below `/opt/kf-onlinejudge/releases`; `current` selects -an explicitly installed version. Private operator/service configuration lives in -`/etc/kf-onlinejudge` and `/etc/kf-judge`, mutable application data under -`/var/lib/kf-onlinejudge`, and protected Sandbox state/images under -`/var/lib/kf-judge`. The delegated task cgroup is separate from the Root factory's -own supervision cgroup. Application processes do not initialize tables on start; -Backend Alembic remains the only migration entry point. -All application services and the separate delegated task subtree live beneath -`kfonlinejudge.slice`: 3840 MiB memory, no swap, 384 processes/threads and two CPU -equivalents. The remaining physical RAM is reserved for the kernel/minimal OS. -The task subtree is `/sys/fs/cgroup/kfonlinejudge.slice/kf-judge-tasks`, a sibling -of the Root factory supervision unit under that same total node budget. Its -installation template does not prove the effective limits; actual service and -task limits are read back in the independent acceptance environment. +A bundle contains at most 50,000 single-link regular files, 256 MiB per file, +4 GiB in total and 8 MiB of metadata. Verification rejects links, extra members, +private paths and changed bytes. Keep bundles and real configuration outside Git. -Database login identities are separate for migration, API/outbox, Coordinator, -problem import and local identity operations. Rust Worker and Root factory do not -hold database credentials. Root factory has no MQ/platform/SSH/GitHub identity; -privileged execution remains inside Sandbox. Public and admin browser traffic -share the HTTPS site; execution-control interfaces remain internal. +Verification is read-only. Installation, running services and execution isolation +each need their own acceptance evidence. -Upgrade and restore must retain immutable problem versions, submitted source, -audit and original judging reports. A consistent backup cut has a zero confirmed -data-loss target; restoring from already prepared release/backup materials has a -30-minute recovery target, including service health and actual four-language -verification. These are acceptance targets, not results inferred from this file. -Installation/service/restore evidence is required before this pre-release can be -treated as an accepted delivery; the contract itself does not deploy production. +## Directories and authority -The node input for `generate-configuration` has exactly `schema_version: 1`, -`origin` (one HTTPS origin), `worker_id` (canonical nonzero UUID), and -`measurement_nonce` (null or a fresh operator-generated SHA-256-sized nonce). -It is a 0600 single-link file. Endpoint input is a separate 0600 JSON object -with exactly `schema_version: 1`, `database` and `amqp`: database keys are -`api`, `coordinator`, `import`, `migration`, `identity`; broker keys are `outbox` -and `worker`. Login names and numeric loopback/socket locations are validated by -the tool. Every endpoint requires its own operator-supplied private credential; -there is no working example password or shared default. Import uses the reachable -`/run/postgresql` Unix socket; all other control connections use numeric loopback. +| Location | Contents | +| --- | --- | +| `/opt/kf-onlinejudge/releases` | Installed application versions | +| `current` | Explicitly selected application version | +| `/etc/kf-onlinejudge` and `/etc/kf-judge` | Private operator and service configuration | +| `/var/lib/kf-onlinejudge` | Mutable application data | +| `/var/lib/kf-judge` | Protected Sandbox state and runtime images | -Generated admission uses the actual verified catalogs for C17/C++20, CPython -3.13.16 and OpenJDK 21.0.12.1+1-LTS and the unchanged public per-language budgets. -The protected GNU legacy image is a separate directory: it retains the original -v1 header and identical verified GNU executable/library bytes, while the three -v2 catalog identities remain untouched. Shared Worker/task UID/GID pool and -execution/prefetch/import slots remain one. No canonical judging report or -historical runtime identity is converted by the installer. +Backend Alembic is the only schema migration entry point. Database identities are +separate for migration, API/outbox, Coordinator, problem import and local account +operations. Rust Worker has no database identity; the Root factory has no +database, MQ, platform, SSH or GitHub credentials. Sandbox owns privileged +execution. Public and admin pages share the HTTPS origin; control interfaces stay +internal. -Comparability defaults to unavailable (`measurement_epoch=null`). A trusted -operator may explicitly use `--attest-measurement` only after actual node/service -limits and the intended execution environment have been verified. The attestation -binds a fresh nonce, worker, source, exact images, budgets, CPU/microcode, kernel -and node scheduling snapshot. `check-configuration` refuses an old binding when -those values change; it does not manufacture replacement metrics for old reports. -Before any service resume after such a change, drain execution, generate/install -a new complete private group with a new nonce and independently revalidate the -environment. Raw configuration checks are not performance or isolation evidence. -Configuration batches have independent hash-bound `cfg-...` identities; files -and receipts contain both the source release and batch identifier. A fresh -nonce/identity can therefore be installed for the same source without replacing -an old configuration. Preparation never selects the new batch automatically. -`activate` requires an explicit `--configuration-id`, a fully validated group, -all application units inactive, stopped control ports and no retained task -resources. Root journals and all old batches remain intact. It binds the code -pointer and private batch in `/etc/kf-onlinejudge/active.json`; `check-active` -refuses inconsistent pointers, stale hardware bindings or changed bytes. -Interruption between the two atomic pointer writes requires operator review; -no service starts or claims readiness during that mismatch. A stopped stale -batch can be retired without accepting it for execution under a changed kernel. +Application services and the task subtree share `kfonlinejudge.slice`, capped at +3840 MiB, zero swap, 384 processes/threads and two CPU equivalents. The remaining +physical memory is reserved for the kernel and minimal OS. Tasks use +`/sys/fs/cgroup/kfonlinejudge.slice/kf-judge-tasks`, separate from the Root +supervisor's cgroup. Read back service and task limits on the target node. -The installation entry points are available with `python3.13 -m scripts.release ---help`. Use them in order: verify/preflight, install-files, generate-configuration, -install-configuration, explicit DB identity provision, controlled Backend Alembic -and account CLI operations, then activate and the separately verified bounded -service lifecycle. These operations never contact a package index and do not -install onto a business host on their own. OS dependencies (systemd 262, matching -glibc ABI bytes, PostgreSQL 18.6 and the fixed broker runtime) must already be -present from separately verified offline materials. +## Prepare configuration + +Supply two 0600 single-link JSON files to `generate-configuration`: + +| Input | Fields | +| --- | --- | +| Node | `schema_version: 1`, `origin`, `worker_id`, `measurement_nonce` | +| Endpoints | `schema_version: 1`, `database`, `amqp` | + +`origin` is one HTTPS origin. `worker_id` is a canonical nonzero UUID. +`measurement_nonce` is null or a fresh operator-generated SHA-256-sized nonce. +The database object has `api`, `coordinator`, `import`, `migration` and `identity` +keys; the broker object has `outbox` and `worker` keys. Supply a different private +credential for every endpoint. Import uses `/run/postgresql`; other control +connections use numeric loopback. The tool checks login names and locations. + +The generated admission profile binds C17/C++20, CPython 3.13.16 and OpenJDK +21.0.12.1+1-LTS to verified catalogs and the existing per-language budgets. The +GNU legacy image retains its v1 header and verified executable/library bytes in +a separate directory. Worker, task identities and execution/prefetch/import +slots remain one. Historical reports and runtime identities remain unchanged. + +## Select a version + +Configuration batches have content-bound `cfg-...` identities. Preparation keeps +earlier batches and does not select a new one. `activate` requires an explicit +`--configuration-id`, a fully validated batch, inactive application units, +stopped control ports and no retained task resources. + +Selection writes the code pointer and `/etc/kf-onlinejudge/active.json`. A crash +between those atomic writes can leave a mismatch; stop and inspect it before +starting services. `check-active` rejects inconsistent pointers, changed bytes +and stale hardware bindings. Root journals and earlier batches remain intact. +A stopped stale batch can be retired without accepting it under a changed kernel. + +## Performance comparability + +Comparability defaults to unavailable (`measurement_epoch=null`). Use +`--attest-measurement` after verifying the node's limits and execution environment. +It binds a fresh nonce, worker, source, images, budgets, CPU/microcode, kernel and +node scheduling snapshot. `check-configuration` rejects a changed binding. + +After an environment change, drain execution, install a complete batch with a new +nonce and revalidate before resuming. Earlier reports retain their original +measurements. Configuration validation alone does not prove comparable performance +or execution isolation. + +## Installation order + +Use `python3.13 -m scripts.release +--help` for the command interface. Prepare the OS dependencies and dedicated data +volumes, then follow this order: + +1. Verify the bundle and run preflight. +2. Install files and runtime images. +3. Generate and install a private configuration batch. +4. Activate that batch while services and tasks are stopped. +5. Install the journal profile and selected dependency snapshots. +6. Initialize PostgreSQL, provision DB identities, run Backend Alembic and create + application accounts; stop the initialization database before installing units. +7. Install ordinary units, the Root factory and the HTTPS snapshot/TLS material. +8. Start the bounded service lifecycle and verify readiness and actual execution. + +The tools use offline materials and do not contact a package index. + +See the [operator command reference](../../scripts/release/README.md) for service, +HTTPS, backup and recovery operations. Upgrades and restores must preserve +immutable problem versions, submitted source, audit and original judging reports. +A confirmed consistent backup cut has a zero data-loss target. Recovery from +prepared release and backup materials has a 30-minute target, including health +and four-language verification. Both targets require measured acceptance results. diff --git a/scripts/release/README.md b/scripts/release/README.md index 63eb851..ea83a6e 100644 --- a/scripts/release/README.md +++ b/scripts/release/README.md @@ -1,20 +1,13 @@ # Offline release tooling -This package is an operator composition tool. `policy` validates delivery bounds, -`manifest` inventories/verifies explicit files, and `build` runs the locked trusted -checkout build and copies only tracked application/deployment sources, fresh -production dependencies and separately fingerprinted public runtime inputs. -It does not import domain ORM/repositories, write application state, implement a -second migration path, execute student code or perform Sandbox privilege work. -`scripts/check_architecture.py` enforces its application import boundary. +Build and operate a KF-OnlineJudge node with locked dependencies and verified +offline materials. The tools cover installation, configuration, HTTPS, maintenance +and cold recovery. Full system and recovery acceptance is still in progress. -All dependency resolution for a release uses existing lockfiles and offline -caches. Backend source is shipped with locked production dependencies rather -than invoking an unpinned isolated wheel build backend. uvx/pip and package -startup path hooks are not delivery runtime entry points. Build inputs specify a -verified public runtime inventory and the three existing image catalogs; an -independent expected SHA-256 binds each input. Real input paths/receipts are local -operator data and do not belong in Git. +For the target platform and installation order, read +[`deploy/release/README.md`](../../deploy/release/README.md). + +## Build a release ```sh python3.13 -m scripts.release build --repo /native/checkout \ @@ -22,504 +15,428 @@ python3.13 -m scripts.release build --repo /native/checkout \ --work /private/new-build-work ``` -The checkout must be clean and committed, build output/work directories new, -and the output filesystem have the required reserve. The resulting receipt -contains the source and manifest fingerprints, file count and byte total, with -`installed=false` and `execution_verified=false`. Installation and service -acceptance remain separate operations. See `deploy/release/README.md` for the -versioned installation contract and authority split. +Start from a clean committed checkout. Output and work directories must be new, +and their filesystem must have the required reserve. The build uses lockfiles and +offline caches, then copies tracked application/deployment sources, fresh locked +production dependencies and separately fingerprinted public runtimes. Backend +source ships with those dependencies; uvx/pip and startup hooks are not runtime +entry points. -`preflight` reads installation eligibility only. `install-files` requires root, -a completely root-controlled verified input tree and the fixed target profile; -it installs protected versions/images and bounded OS identities, without table -migrations, application accounts, services or the active-version switch. -Repeated identical files converge; unknown modes/owners, changed bytes or an -incomplete staging directory require operator inspection and are never removed. +Build inputs contain explicit inventories and expected SHA-256 digests for public +runtimes and the three image catalogs. Keep real input paths and receipts outside +Git. The build receipt records source and manifest digests, file count and byte +total with `installed=false` and `execution_verified=false`. -`generate-configuration` composes exact public contracts with separately supplied -0600 node and endpoint inputs. It emits a new private directory and a hash-bound -receipt, never prints tokens, never changes a supplied password and never silently -overwrites an earlier identity. API, publisher, import, migration and identity -operations have distinct required database authority; Worker holds only its own -MQ/control identity, Root receives no business credentials. Backend URLs use the -existing asyncpg driver. No uvx/pip or online runtime installation is used. +Use a separately trusted checker and an out-of-band manifest SHA to validate an +incoming bundle. A verified delivery includes `bin/kf-delivery` for later offline +operations. Running an unverified bundle's checker cannot establish trust in that +bundle. -`install-configuration` validates the entire group before writing any member; -`check-configuration` checks installed bytes, same-source admission, identity and -hardware/kernel/node scheduler bindings. `activate` requires that check and -stopped control ports, no factory socket and no retained task/identity/capture -resources. It atomically selects an installed directory and starts no process; -draining, migrations and actual health remain separate operations. It does not -erase retained journals or release resources lacking cleanup proof. -Every private group also has a content-bound `configuration_id`. Installing a -second group for the same source preserves the first group byte-for-byte. -`activate RELEASE --manifest-sha256 SHA --configuration-id BATCH` selects an -explicit validated batch only after ordinary application units and independent -task resources are stopped. `check-active` checks the protected batch/code -binding and current measurement snapshot, with no liveness/readiness claim. +## Authority and service identities -`provision-database-identities` is an explicit local database operator step. -It uses the locked asyncpg dependency, bounded connections/statements and a -120-second overall deadline. It installs only fixed independent logins/NOLOGIN -groups and the application database, verifies every supplied credential and -refuses unexpected existing authority. There is no application table DDL or -automatic account bootstrap: the migration login owns the database without -global superuser/create-role/create-db/bypass-RLS rights; Backend Alembic remains -the only schema/grant migration entry. Runtime logins cannot grant/set group -authority. The migration login inherits the narrow NOLOGIN function-owner role -and can SET that role, so unchanged Alembic can grant/rename its functions after -ownership transfer. Membership has no ADMIN option, adds no global role-creation -privilege, and is never given to runtime logins. See -[PostgreSQL role membership](https://www.postgresql.org/docs/18/role-membership.html) -and [function ownership](https://www.postgresql.org/docs/18/sql-alterfunction.html). +| Operation | Authority | +| --- | --- | +| Build, verify and preflight | Read-only system access | +| File/configuration installation and activation | Root operator | +| Database identity provisioning | Supplied local DB operator | +| Schema and grants | Backend Alembic | +| Application accounts | Backend account CLI | +| Execution state | Coordinator | +| Privileged task preparation and execution | Rust Sandbox | -Always bootstrap validation from a separately trusted tool copy and an out-of-band -manifest SHA. Do not execute a checker supplied inside an unverified bundle to -establish trust in that same bundle. A verified delivery includes `bin/kf-delivery` -and its source for later offline operations. Command success/receipt generation -does not claim Linux execution, working service roles or installation acceptance; -these require the recorded independent VM evidence. +`policy` validates bounds, `manifest` inventories and verifies files, and `build` +assembles the release. Release tools do not import domain ORM/repositories or +provide a second schema migration path. `scripts/check_architecture.py` checks +the application import boundary. Installation and configuration commands start +no services; the maintenance commands below control their lifecycle. -`generate-proxy` is a separate, offline composition entry. Its private 0600 -profile has exactly `schema_version: 1`, one canonical HTTPS DNS `origin`, -`certificate_sha256` and `key_sha256`. TLS inputs are independently supplied -later at `/etc/kf-proxy/tls/fullchain.pem` and `key.pem`; the generator -does not read, create or verify a private key. It emits a new private NGINX -configuration and receipt tied to the verified release. No arbitrary upstream, -include, path, credentials or resource overrides are accepted. Only browser API -families target the API loopback port; execution-control paths are refused. ZIP -uploads have their own canonical UUID route. Syntax, actual TLS/stream limits, -static routes, permissions and service health require subsequent VM acceptance; -the generator explicitly returns those capabilities as unverified. -The separately installed `kf-proxy` worker identity must be distinct from API, -Coordinator, import, judging and database/broker identities. It has no business -credentials; generation alone never creates that account or starts NGINX. +API/outbox/import share the content-storage identity and use private PID views. +Coordinator and Worker have distinct ordinary identities. `service_bootstrap` +validates Rust credential bytes and writes a private runtime copy with mode 0600 +before launching the fixed binary. Tokens and URLs +stay out of arguments and unit text. Python launchers accept fixed modules, +arguments and generated environment fields; inherited debugging, proxy and +provider credentials are omitted. Import also requires a loopback-only network +view and the same admission policy as API. -Operator responsibilities are separate: build/verify are read-only with respect -to system state; preflight observes eligibility; file/config installation and -activation require the root operator; DB provisioning requires a separately -supplied local DB operator; schema/grants and first account use the existing -Backend Alembic/account CLIs. Configuration validation never starts or restarts -a service. The later bounded service lifecycle must prove drain/cleanup before -maintenance; none of these entries authorizes a Sandbox isolation bypass or -silently substitutes a healthy result when lifecycle evidence is absent. +Every ordinary launch checks UID/GID, cap0, no-new-privileges, node/service cgroup +limits and process file/descriptor/stack limits. The Python services require PID 1 +inside their private namespace. Unsupported or ignored isolation settings refuse +startup. Rust Worker has no database credentials. Root factory uses an empty +business environment and gets no database, MQ or platform identity. -Proxy directive references: [NGINX core](https://nginx.org/en/docs/http/ngx_http_core_module.html), -[proxy module](https://nginx.org/en/docs/http/ngx_http_proxy_module.html), and -[TLS module](https://nginx.org/en/docs/http/ngx_http_ssl_module.html). +## Install files and configuration -`python3.13 -m scripts.release.services generate` composes five ordinary system -units from a verified bundle and matching private configuration receipt. The -matching service launch modules must themselves be in that bundle. Rendering -uses fixed users, resource ceilings, finite restart/stop limits and an explicit -release/configuration pre-start binding. It never writes system units, starts a -process, migrates a schema or claims readiness. Actual units, mount/PID/cgroup -views, Unix DB socket access, dependency roles and effective limits require VM -installation/acceptance. Root factory and database/broker lifecycle are separate. +`preflight` checks eligibility. `install-files` requires Root, a Root-controlled +verified input tree and the fixed target profile. It installs protected versions, +images and OS identities. Repeat installation checks identical files; unknown +ownership/modes, changed bytes and incomplete staging directories require inspection. -API/outbox/import share the established content-storage identity; individual PID -namespaces prevent same-UID inspection of another service's environment. This -requires the fixed systemd profile and verified kernel namespace support. Rust -Coordinator/Worker use distinct ordinary identities and systemd credentials. -Every ordinary launch checks effective service/node cgroup memory, swap, CPU and -PID limits, file/descriptor/stack limits, UID/GID and absence of capabilities -before exec. The three Python services also require PID 1 in their private PID -namespace; import requires a loopback-only network view. Ignored namespace -directives therefore refuse startup. These checks still require actual VM -verification; they do not attest Sandbox isolation or service readiness. -The import environment includes the same validated admission policy as API. +`generate-configuration` reads separate 0600 node and endpoint inputs and writes +a new private batch. API, publisher, import, migration and account operations use +their required DB authority; Worker receives its own MQ/control identity. Backend +URLs use asyncpg. The tool preserves supplied passwords and earlier identities. -`service_bootstrap` verifies bounded Rust credential bytes, copies them to a -private service runtime directory with the existing required 0600 mode and then -launches only the fixed ordinary binary. It accepts no student executable, -command override or privileged Sandbox operation. Tokens/URLs are not command -arguments or unit text. The outbox unit uses the existing actual confirmed -publisher module; the old scaffold check entry is not treated as a delivery loop. -Python launches use a fixed module/argument list and only the generated service -environment fields; inherited debugging/proxy/provider credentials are omitted. +`install-configuration` validates the entire batch before writing. A batch has a +content-bound `configuration_id`; installing another for the same source retains +the first byte-for-byte. `check-configuration` validates installed bytes, +admission, identity and hardware/kernel/node scheduling bindings. -`python3.13 -m scripts.release.service_install DIRECTORY --receipt-sha256 SHA` -is the separate root operator step for those five ordinary units. It requires -the selected installed configuration, the canonical configuration receipt from -the generator, and matching service launch/installer bytes in that delivery. -All app units must be stopped. It stores immutable batches under the protected -`/etc/kf-onlinejudge/units/RELEASE.CONFIGURATION/` directory, then updates only -known root-owned system unit pointers and performs a bounded daemon reload. -Unknown regular files, foreign pointers, changed bytes and unrecorded old batches -refuse replacement. Previously installed batches remain available; interruption -may leave a mixed set of pointers, but each pre-start check rejects an unselected -binding. Repeat installation can finish the same verified batch after stopping -all app units. Loaded/inactive unit readback starts no service and makes no -claim about effective runtime limits, readiness, Sandbox gates or VM acceptance. +Run `activate RELEASE --manifest-sha256 SHA --configuration-id BATCH` with the +explicit batch. `activate` requires stopped application units, control ports and +factory socket, and empty task/identity/capture resources. Root journals and +batches are retained. Interruption between pointer writes requires inspection. +`check-active` checks code/configuration selection and the measurement snapshot. -`python3.13 -m scripts.release.factory_service --output DIRECTORY` composes the -separate Root factory unit only after selection and protected configuration checks. -It requires matching composition code in that delivery, uses the fixed Root -binary and empty business environment, and preserves the supervisor reservations -and finite lifetime. It does not inherit the ordinary service namespace/filter -flags. Its pre-start `kf-sandbox prepare-factory` performs privileged cgroup -preparation inside Sandbox: the exact node/UID pool, single supervisor process, -empty task/lease/capture resources and factory lock are required. A retained -boot/inode record recognizes only its own empty delegation; an unknown group, -same-boot loss or conflicting record refuses. New-boot recreation requires empty -persistent resources. No journal/lease is deleted and preparation grants no -execution permit. Neither composition nor lexical tests are Root/kernel/service -acceptance; install, restart, failure recovery and actual execution remain VM-only. -The separate operator step `factory_service --install DIRECTORY --receipt-sha256 SHA` -requires that generated receipt, the selected protected configuration and matching -installer bytes. It keeps immutable Root-unit batches under `/etc/kf-judge/units/`, -updates only recorded Root-owned pointers with all app units stopped, and checks -loaded/inactive systemd metadata. Foreign files/links and partial unknown batches -are refused, old batches remain retained. This step neither starts the factory nor -prepares its cgroup; only the later bounded startup invokes Sandbox preparation. +## PostgreSQL and RabbitMQ + +### Prepare dependencies `python3.13 -m scripts.release.dependencies --connections PRIVATE_JSON --output DIRECTORY` -composes the fixed loopback PostgreSQL/RabbitMQ configurations and ordinary units -from the selected release. Broker passwords must be identical to the selected -outbox/Worker batch. The private definition file contains salted password hashes, -two users without management tags, one vhost/queue and separate publish/consume -permissions. Definitions import is required on a blank broker; no guest account -or default password is configured. The management plugin remains disabled. -PostgreSQL local/loopback paths both use SCRAM, including the Unix import socket. -Configuration includes bounded connections, query/temp budgets and log controls. +generates fixed loopback configurations and ordinary units for the selected +release. Broker passwords must match the outbox/Worker batch. Its private +definition contains salted hashes, two users without management tags, one +vhost/queue and separate publish/consume permissions. A blank broker must import +these definitions; management and guest/default accounts stay disabled. -Both dependency units reserve 512 MiB, zero swap and bounded CPU/PIDs/files, -restart and lifetime. The receipt requires separate 2-GiB PostgreSQL and 512-MiB -broker storage mounts. These mounts are not created or checked by composition. -RabbitMQ memory/disk alarms and PostgreSQL WAL/temp settings are not total hard -storage/memory guarantees. Required real filesystem/cgroup/queue/auth readbacks, -account ownership, cluster initialization, cookie/definition permissions and -startup/restart are still explicit installation/VM steps; no service is started. -Dependency pre-start checks require the selected release/batch and read back the -ordinary UID, zero capabilities, no-new-privileges, process/cgroup/node budgets, -private data ownership, a separate data filesystem mount and its actual size. -A parent filesystem or an alarm setting therefore cannot silently pass as the -required dedicated hard storage capacity. Config parsing alone does not verify -these conditions or broker semantics on an already-initialized node. +PostgreSQL uses SCRAM on local and loopback paths, including the import Unix +socket. Its configuration bounds connections, queries, temporary storage, WAL and +logs. Both dependency services have 512 MiB, zero swap and finite CPU/PIDs/files, +restart and lifetime limits. Memory/disk alarms supplement the hard limits. -Dependency directive references: [PostgreSQL resources](https://www.postgresql.org/docs/18/runtime-config-resource.html), -[RabbitMQ limits](https://www.rabbitmq.com/docs/limits), -[definition import](https://www.rabbitmq.com/docs/definitions), -and [RabbitMQ access control](https://www.rabbitmq.com/docs/access-control). +### Install snapshots `python3.13 -m scripts.release.dependency_install DIRECTORY --receipt-sha256 SHA ---erlang-cookie PRIVATE_FILE --cookie-sha256 SHA` installs the selected dependency -snapshot with all app/dependency services stopped. It requires already provisioned -dedicated ext4 mounts at the two fixed data paths, with their fixed capacity and -`rw,nosuid,nodev,noexec` options. It does not create, format, resize or mount disks. -A root-owned empty mount (apart from an empty private `lost+found`) may be assigned -to its dedicated account; existing data is never recursively reassigned. Ordinary -UID/GID 1203/1204 must be locked, without supplementary authority, and have the -fixed PostgreSQL/RabbitMQ service homes. Conflicting OS accounts are not adopted. +--erlang-cookie PRIVATE_FILE --cookie-sha256 SHA` installs a snapshot while all +application and dependency services are stopped. -The generated files are installed under `/etc/kf-dependencies/RELEASE.CONFIGURATION/`. -This separate root-owned searchable tree lets each service group read only its -0640 configuration files; the application configuration tree remains private. -The definition file's salted credentials, user privileges and queue budgets must -match the selected outbox/Worker batch. The operator supplies an explicit private -256-bit Erlang cookie; repeat installation requires the same ordinary-owned 0400 -cookie and does not implicitly rotate it. No cookie/password bytes are reported. -System unit pointers may replace only recorded root-owned dependency snapshots. -The command retains old batches, verifies loaded/inactive metadata, starts no -service and initializes no database. Cluster initialization, actual effective -limits, broker authorization and startup/readiness still require VM acceptance. +Prepare dedicated ext4 data mounts first, with fixed capacities and +`rw,nosuid,nodev,noexec` options: + +| Service | Storage | Ordinary UID/GID | +| --- | --- | --- | +| PostgreSQL | 2 GiB | 1203 | +| RabbitMQ | 512 MiB | 1204 | + +The installer does not create, format, resize or mount disks. It can assign an +empty Root-owned mount, allowing only an empty private `lost+found`, to the +dedicated account. Existing data is never recursively reassigned. Accounts must +be locked, have the fixed service homes and lack supplementary authority. + +Snapshots live in `/etc/kf-dependencies/RELEASE.CONFIGURATION/`. Each service group +reads its own 0640 files; application configuration remains private. Definition +hashes, permissions and queue budgets must match the selected outbox/Worker batch. +Supply a private 256-bit Erlang cookie. Repeat installation requires the same +ordinary-owned 0400 cookie and leaves its value unchanged. + +Only recorded Root-owned unit pointers can be replaced. Old snapshots remain; +loaded/inactive metadata is read back. Before execution, each dependency checks +its UID, cap0, no-new-privileges, process/service/node budgets, private data +ownership and the dedicated filesystem's mount identity and size. + +### Initialize PostgreSQL `python3.13 -m scripts.release.dependency_initialize --password-file PRIVATE_FILE ---password-sha256 SHA` is the separate explicit PostgreSQL cluster initialization -step. It requires the selected dependency snapshot, stopped services, the dedicated -storage mount and an operator-supplied hexadecimal bootstrap password file. Only -a fresh unrecorded target is initialized. The fixed ordinary PostgreSQL identity -runs `initdb` in a transient network-isolated unit with node/service/process/disk -budgets checked before execution. systemd passes the password as a private file -credential; its bytes are neither unit arguments nor environment values. Both -local and host authentication use SCRAM; incomplete data is retained for explicit -recovery, not silently deleted or reused. The initializer never starts a database -server or migrates business tables; Alembic and role provisioning remain separate. +--password-sha256 SHA` initializes a fresh unrecorded cluster on the selected +snapshot's dedicated mount, with services stopped. Supply a hexadecimal bootstrap +password file. The ordinary PostgreSQL identity runs `initdb` in a bounded, +network-isolated transient unit. systemd supplies the private password file as a +credential, outside arguments and environment values. Local and host +authentication use SCRAM. -A root-owned initialization receipt binds the physical cluster identity and initial -credential fingerprint. Repeat initialization requires that recorded identity and -does not reset a password or claim successful DB authentication. The operator -reads the transient unit result and confirms stop; foreign/residual units and -unrecorded/partial targets are refused. These are implementation interfaces, not -actual VM acceptance. See [PostgreSQL initdb](https://www.postgresql.org/docs/18/app-initdb.html) -and [systemd credentials](https://systemd.io/CREDENTIALS/) for the underlying interfaces. +The Root-owned receipt records the physical cluster identity and credential +fingerprint. Repeat initialization requires that identity and does not reset its +password. The operator checks the unit result and confirms stop; foreign/residual +units and partial/unrecorded targets are refused. Retain incomplete data for +inspection. This command starts no DB server and migrates no tables. -`python3.13 -m scripts.release.node_health` performs the installed service -admission-precondition checks in a fixed root operator unit. It uses bounded -ordinary broker/factory/import observer units and accepts no executable, account, -port or command override. Each helper checks its effective node/service/process -budgets, identity and no-new-privileges/capability boundary. Unknown or concurrent -units are refused; transient helper stop is read back. This is an operator CLI, -not an additional public HTTP control endpoint, and has no schema/state writer. +### Provision database identities -The projection checks selected unit pointers and effective identities/limits, -actual DB-role/column privileges, the import owner's fresh policy-bound heartbeat -through its Unix socket, and broker queue semantics/permissions/vhost limits via -ordinary-cookie read-only CLI queries. Coordinator owns the authenticated bounded -execution/cleanup counters; truncated or unconfirmed data never proves a drain. -Its `/internal/operations/` routes require platform authority and are not proxied -publicly. Counters contain no submission IDs, source, archives or credential bytes. +`provision-database-identities` uses locked asyncpg, bounded connections/statements +and a 120-second deadline. It installs fixed independent logins/NOLOGIN groups and +the application database, checks every supplied credential and rejects unexpected +authority. Backend Alembic remains responsible for table DDL and grants. -Factory health is a separate version-1 authenticated observation, rate/call-count -bounded for the finite factory lifetime. It reads current supervisor/delegation -authorization, the exact runtime-config digest, live helpers, remaining requests -and lifetime, without recovery, task allocation, ledger writes or limit repair. -The old allocation request bytes are unchanged. Health explicitly reports -`isolation_verified=false` and `execution_gate=required_per_execution`: every real -execution still needs its namespace/root/identity/seccomp/exec acknowledgements. -Admission readiness requires confirmed empty execution/cleanup state and enough -factory requests/lifetime for the configured maximum job. It does not certify -TLS/browser behavior or replace the independent VM's execution acceptance. +The migration login owns the database and can inherit/SET the narrow NOLOGIN +function-owner role so Alembic can grant or rename transferred functions. It has +no superuser/create-role/create-db/bypass-RLS rights or ADMIN membership option. +Runtime logins cannot grant/set group authority and never receive that membership. -The optional explicit `--probe-notification` sends one small confirmed notification -for a fresh identifier checked absent by Coordinator. It reads no message body, -creates no evaluation/grade and starts no program; ordinary periodic observation -does not publish messages. Probe failure is reported separately. Underlying -interfaces: [RabbitMQ CLI](https://www.rabbitmq.com/docs/man/rabbitmqctl.8), -[PostgreSQL privilege functions](https://www.postgresql.org/docs/18/functions-info.html). +## Application units and Root factory -`python3.13 -m scripts.release.lifecycle --action start|resume|drain|stop|restart` -is the explicit fixed-unit maintenance entry. Root publishes a protected, versioned -availability record bound to the selected release/configuration. The installed API -checks that record before accepting mutations; an absent, unknown, closed or wrong -binding returns 503. Existing reads, logout, cancellation and receipt-bound archive -uploads retain their ordinary identity/lease checks. Health alone cannot open this -barrier, and service startup does not create schemas, accounts or credentials. +`python3.13 -m scripts.release.services generate` renders five ordinary units from +a verified bundle and matching configuration receipt. Users, budgets, restart/stop +limits and pre-start release/batch checks are fixed. The outbox runs the confirmed +publisher module. Service launch code must belong to the selected delivery. -Drain first closes new mutations, keeps the API available to already accepted -work, and polls the Coordinator, outbox, import, broker and factory owners within -a finite deadline. Only their untruncated empty projections permit ordinary -service shutdown. Final owner counts, actual stopped process groups, delegated -tasks/identity/capture resources and absent factory socket must all agree before -quiescence is published. Timeout preserves unfinished rows, leases, queues and -replay journals; no forced grade or cleanup proof is written. `drain` leaves DB/MQ -running, `stop` additionally stops them, and `restart` drains before a fresh start. +`python3.13 -m scripts.release.service_install DIRECTORY --receipt-sha256 SHA` +installs those units with all application services stopped. Snapshots are retained +in `/etc/kf-onlinejudge/units/RELEASE.CONFIGURATION/`. The installer updates only +known Root-owned pointers and performs a bounded daemon reload. Foreign pointers, +regular files, changed bytes and unknown partial batches refuse replacement. +A stopped repeat installation can finish the same verified batch. Every pre-start +check rejects a pointer left on an unselected batch. -Fresh start requires stopped known application units. `resume` explicitly checks -known units that are already running after an interrupted operation; it does not -adopt foreign units or repair unknown resources. Both keep admission closed until -full service preconditions pass and opening is read back. Failed final readiness -closes the barrier again. A bounded Root maintenance receipt records manager -operations and cleanup results. Root factory termination uses a signal notice and -the existing bounded guardian/recovery path; ordinary import termination finishes -its current authorized operation before claiming another. These interfaces require -actual independent-VM lifecycle acceptance and are not installation/safety proof. +`python3.13 -m scripts.release.factory_service --output DIRECTORY` renders the +separate Root unit after selection and protected configuration checks. The unit +uses the fixed binary, empty business environment, supervisor reservations and +finite lifetime, with the factory's own namespace/filter context. -`python3.13 -m scripts.release.journal` separately installs a fixed KF journal -namespace profile. It requires a preprovisioned dedicated 64-MiB ext4 mount at -`/var/log/journal/MACHINE_ID.kf-onlinejudge`, with the same private hard-volume -mount rules used for dependencies. It never creates/formats/mounts a disk, changes -the default system journal, adopts old unrecorded log data or starts a daemon. -Unknown configuration/overrides and partial installations are retained and refused. +`factory_service --install DIRECTORY --receipt-sha256 SHA` installs the generated +receipt and unit under `/etc/kf-judge/units/`, preserving earlier snapshots and +checking loaded/inactive systemd metadata. Application units must be stopped; +foreign files/links and unknown partial batches are refused. -The profile fixes 32-MiB persistent/8-MiB runtime rotation budgets, bounded files, -lines and seven-day retention. Its namespaced daemon override limits CPU, memory, -swap and PIDs. Rotation settings are not a hard filesystem guarantee; the dedicated -volume must also be observed. The `check_ready` interface reads actual namespace -service/process/cgroup state. Ordinary application, DB/MQ and Root-factory -templates now select this namespace. Their protected pre-start binding checks -require its recorded configuration and physical hard volume, while retaining -ordinary private PID views and the Root factory's existing permission context. -Installing the profile does not start or replace running services. Real routing, -privacy and retention/capacity behavior remain C+D/F VM acceptance work. -The profile relies on the pinned systemd 262 -[namespace interface](https://github.com/systemd/systemd/blob/v262/man/systemd.exec.xml) -and [journal configuration](https://github.com/systemd/systemd/blob/v262/man/journald.conf.xml). +On startup, `kf-sandbox prepare-factory` checks the exact node/UID pool, a single +supervisor, the factory lock and empty task/lease/capture resources before creating +cgroup delegation. A boot/inode record identifies only its own empty delegation. +Unknown groups, same-boot loss and conflicting records refuse startup; a new boot +requires empty persistent resources. Preparation retains journals and leases. +Each subsequent execution still needs the full Sandbox isolation check. -The read-only status observer may see a `ProtectSystem` mount as read-only. That -does not prove the logger can write: readiness separately reads the fixed known -daemon's kernel mountinfo, matches its mount device to the protected volume and -requires both mount and superblock writability. Main PID, invocation and process -start time are checked again to reject a changed process. No writable exception, -ptrace/setns capability or wider Python PID visibility is added to observation. -Underlying read interface: [Linux proc mountinfo implementation](https://github.com/torvalds/linux/blob/v6.18/fs/proc_namespace.c). +## HTTPS entry -Production API logging uses a closed event vocabulary and omits arbitrary library -messages, arguments, paths and exception text. Uvicorn's separate error handlers -are redirected through that formatter; installed API launchers disable access -logging so request query strings are not copied into service logs. Development -diagnostics remain separate. These ordinary formatting checks are not evidence -of actual daemon, HTTP or upload log privacy; that is still a VM acceptance case. - -Build-input version 2 adds an explicitly fingerprinted public NGINX 1.30.5 -runtime catalog. Version 1 remains available for older engineering bundles without -proxy support; there is no implicit proxy default. The separate `proxy_runtime` -module permits only the binary and bounded shared libraries, excludes TLS and -configuration files, verifies every byte/mode and ships them under `payload/proxy`. -This separate library search tree avoids silently replacing existing application, -database or language-image libraries. Binary version/signature/ELF-closure evidence -is recorded by the operator; accepting a structural catalog is not TLS/server or -Linux-safety acceptance. Actual proxy installation and execution remain separate. +`generate-proxy` reads a 0600 profile containing `schema_version: 1`, canonical DNS +`origin`, `certificate_sha256` and `key_sha256`. It generates a new NGINX snapshot +and receipt for the verified release. It accepts fixed upstreams, paths and limits. +Browser API families reach only API loopback; execution-control routes are refused. +ZIP uploads use a separate canonical UUID route. `python3.13 -m scripts.release.proxy_install --source DIRECTORY --receipt-sha256 SHA --certificate PRIVATE_CERTIFICATE --private-key PRIVATE_KEY` -installs the explicitly supplied protected proxy snapshot and matching unencrypted -TLS files. It requires the same release/API-origin batch, a distinct locked -UID/GID 1205 (`kf-proxy`) without supplementary groups, and an already provisioned -private 128-MiB ext4 volume at `/var/lib/kf-onlinejudge/proxy`. Provision that -volume's ownership/mode explicitly; the installer does not format, mount or -recursively reassign unknown data. Existing TLS material must match its fingerprint; -this entry does not silently rotate keys or claim DNS/public-browser trust. +installs the selected release/API-origin snapshot and matching unencrypted TLS +material at `/etc/kf-proxy/tls/fullchain.pem` and `key.pem`. -Proxy modules separate public runtime verification, fixed service rendering, -Root installation, ordinary launch guarding and read-only HTTPS observation. -The ordinary process retains only `CAP_NET_BIND_SERVICE` for 443, no-new-privileges, -private PID visibility and actual CPU/memory/swap/PIDs/file bounds. It receives -no application DB/MQ/control credentials. Installation preserves older snapshots, -refuses foreign system-unit pointers, reads back loaded/inactive metadata and -starts no server. Maintenance includes the proxy in start/drain/stop ordering. -Readiness uses fixed loopback TLS with the supplied certificate as an explicit -local trust anchor and checks hostname/leaf identity plus the API live response; -it does not substitute for actual browser CA/HTTPS/security acceptance in the VM. -The exact `/api/v1/health/live` route permits GET/HEAD for this bounded process -liveness check. Other health, internal, execution and control routes remain -outside the public proxy allowlist; liveness does not attest judging readiness. +Prepare a 128-MiB private ext4 volume at `/var/lib/kf-onlinejudge/proxy` and the +distinct locked UID/GID 1205 (`kf-proxy`), without supplementary groups. The +installer checks ownership/mode and matching existing TLS fingerprints; it does +not format/mount data volumes, reassign unknown data or rotate keys implicitly. -Private cold backups use a separate format from public release bundles. -The private v3 format binds the stopped broker cookie and current DB bootstrap -fingerprints and retains -recovery history as another private domain. A restore can reject distribution-secret -reuse without dropping the inactive configuration from an earlier recovery. It -changes no public judging v1/v2 canonical bytes or historical report. The earlier -private v1/v2 prototypes were never system-accepted and are refused by this format. -`python3.13 -m kf_oj.infra.artifacts.backup` is an ordinary read-only artifact -inventory command. Run it with the selected API database/storage identity after -the owner-confirmed drain and while PostgreSQL is still available. It reads every -ready artifact registry object through a repeatable-read, read-only transaction -and hash-checks existing source/case bytes with a 64-KiB buffer. It creates no -storage directory or lock and changes no metadata, grade, session or grant. -Its output is private and does not itself prove that writers have stopped. +The ordinary proxy keeps only `CAP_NET_BIND_SERVICE` for 443, no-new-privileges, +private PID visibility and CPU/memory/swap/PIDs/file limits. It has no DB/MQ/control +credentials. Maintenance includes it in start/drain/stop order. Readiness verifies +loopback TLS with the supplied certificate, hostname/leaf identity and API live +response. `/api/v1/health/live` allows GET/HEAD; other health/internal/control +routes remain outside the allowlist. Browser CA trust needs separate verification. + +Build-input version 2 specifies the NGINX 1.30.5 catalog. Version 1 supports older +bundles without a proxy. `proxy_runtime` checks every binary/library byte and mode, +excludes TLS/configuration files and ships a separate tree under `payload/proxy`. +Record package signature, version and ELF-closure verification with the inputs. + +## Logs + +`python3.13 -m scripts.release.journal` installs the fixed KF journal namespace +profile on a preprovisioned 64-MiB ext4 volume at +`/var/log/journal/MACHINE_ID.kf-onlinejudge`. It keeps the default system journal +unchanged and rejects unknown configuration, overrides, old unrecorded data and +partial installs. + +The profile uses 32-MiB persistent and 8-MiB runtime rotation budgets, bounded +files/lines and seven-day retention. Its daemon has separate CPU/memory/swap/PIDs +limits. Application, DB/MQ and Root units use this namespace and check the recorded +profile and physical volume before starting. + +`check_ready` reads the daemon's process, cgroup and mount state. An observer's +read-only `ProtectSystem` view does not establish daemon writability: the check +matches the daemon's mount device to the protected volume and checks mount and +superblock writability. It rechecks PID, invocation and start time to detect a +changed process, without ptrace/setns or a wider observer PID view. + +Production API logs use fixed event names and omit arbitrary library messages, +arguments, paths and exception text. Uvicorn errors use that formatter; installed +API launchers disable access logging so queries are not copied to logs. +Development diagnostics remain separate. Verify real routing, privacy and capacity +on the installed node. + +## Readiness and maintenance + +`python3.13 -m scripts.release.node_health` checks admission preconditions in one +fixed Root operator unit. Ordinary broker/factory/import observers have finite +budgets; each checks identity, limits and no-new-privileges/capabilities. The CLI +rejects unknown/concurrent helpers, reads back their stop and accepts no executable, +account, port or command override. + +Checks include selected units and effective limits, DB-role/column privileges, +the import owner's fresh policy-bound Unix-socket heartbeat and read-only broker +queue/permissions/vhost queries. Coordinator supplies authenticated execution and +cleanup counts through `/internal/operations/`. Truncated or unconfirmed counts +cannot establish a drain. These internal routes require platform authority and +return no submission IDs, source, archives or credentials. + +Factory health is a version-1 authenticated, rate/call-count-bounded observation. +It reports supervision/delegation authorization, config digest, live helpers, +remaining requests and lifetime. It performs no recovery, allocation, ledger write +or limit repair. Request bytes remain unchanged. It reports +`isolation_verified=false` and `execution_gate=required_per_execution`. + +Readiness requires confirmed empty execution/cleanup state and enough requests +and lifetime for the maximum job. `--probe-notification` optionally sends one +small confirmed notification for an identifier checked absent by Coordinator. It +reads no body, creates no evaluation and runs no program. Periodic observation +does not publish; probe failures are reported separately. + +`python3.13 -m scripts.release.lifecycle --action start|resume|drain|stop|restart` +controls the fixed units and a Root-protected release/batch availability record: + +| Action | Behaviour | +| --- | --- | +| `start` | Requires stopped known application units; opens admission after readiness passes | +| `resume` | Validates known running units after an interrupted operation | +| `drain` | Closes new mutations, waits for owner-confirmed cleanup and leaves DB/MQ running | +| `stop` | Completes the drain, then stops DB/MQ | +| `restart` | Drains before a fresh start | + +API returns 503 for an absent, unknown, closed or wrong-bound availability record. +Reads, logout, cancellation and receipt-bound archive uploads keep their existing +identity/lease checks. Both startup paths keep admission closed until the opening +is read back; failed final readiness closes it again. + +Drain keeps API available to accepted work and polls Coordinator, outbox, import, +broker and factory within a finite deadline. Untruncated owner counts, stopped +process groups, empty tasks/identity/capture resources and an absent factory socket +must agree before quiescence is published. Timeout preserves rows, leases, queues +and replay journals. Root uses its signal/guardian recovery path; import finishes +its current authorized operation before claiming another. The maintenance receipt +records manager operations and cleanup results. + +## Cold backups + +Collect the private artifact inventory with +`python3.13 -m kf_oj.infra.artifacts.backup` after owner-confirmed drain, while +PostgreSQL is available. Use the selected API DB/storage identity. A read-only, +repeatable-read transaction checks ready registry objects and source/case digests +with a 64-KiB buffer. The command creates no directory/lock and changes no state. + +Run `lifecycle --action stop` and verify processes have exited. A quiescent stop +reuses only this boot's successful selected owner-derived drain receipt and +rechecks application cgroups and Root task cleanup. -After collecting the inventory, run `lifecycle --action stop`. For an already -quiescent node it reuses only this boot's successful selected owner-derived drain -receipt, rechecks every application cgroup and the Root task cleanup, then stops -the selected database and broker. It does not query owners already stopped or -adopt a stale, failed or unproven cut. Verify all their processes have exited. `python3.13 -m scripts.release.backup capture --backup-id bk-<32-lowercase-hex> --references PRIVATE_INVENTORY ---references-sha256 SHA` launches one fixed Root backup operator. It requires the -same selected release/configuration, this boot's successful owner-derived cut, -quiescent admission, empty task resources, no surviving service identities and -clean PostgreSQL shutdown. The operator has only `CAP_DAC_READ_SEARCH`, no-new- -privileges, 256-MiB memory, zero swap, 32 tasks, 50% CPU, a 256-MiB individual-file -limit and a 30-minute deadline. Only the preprovisioned private backup volume is -writable; this entry does not format/mount a volume or execute captured content. +--references-sha256 SHA` captures a cold snapshot after checking the selected +batch, owner-derived cut, quiescent admission, empty tasks, stopped service +identities and clean PostgreSQL shutdown. -Provision a separate Root-owned 0700 ext4 volume of 16 GiB at +Prepare a separate Root-owned 0700, 16-GiB ext4 volume at `/var/lib/kf-onlinejudge/backups`, with `nosuid,nodev,noexec` and no shared mount. -The capture retains all source domains: the cold PostgreSQL cluster, source and -testcase store, private import staging, all retained release/image versions, -application/dependency/proxy configuration, factory replay history and operation -receipts. Every entry has a bounded digest/size and original identity/mode; -captured copies remain Root-only. Registered source/case references must all -match captured files. Source modification, links, special files, unknown -identities, insufficient space or an incomplete cut are refused. A failed target -is retained for explicit inspection and never published as a successful backup. +The fixed backup operator has `CAP_DAC_READ_SEARCH`, no-new-privileges, 256 MiB, +zero swap, 32 tasks, 50% CPU, a 256-MiB file limit and a 30-minute deadline. Only +that volume is writable. -The bound is 7 GiB and 50,000 entries per backup, at most two backup directories, -with a seven-day declared retention period. Capacity refusal does not silently -delete an old/unknown backup. `backup verify --backup-id ID ---manifest-sha256 SHA` checks an explicitly supplied private operator receipt and -every private member; it performs no restoration or secret/epoch reuse. Actual -pruning uses `backup prune --backup-id EXPIRED_ID --manifest-sha256 SHA ---keep-backup-id NEWER_ID --keep-manifest-sha256 SHA`. Both private inventories -must verify completely; the named target must be expired and the preserved copy -must be newer and unexpired. It removes only the known inventory under the -maintenance lock, never the last verified copy, an unknown tree or a partial -backup. Verification and pruning use the same bounded operator as capture. -Controlled recovery into a clean node and measured recovery time require their -separate implementation and independent VM acceptance. Capture, expiry pruning -and interruption behavior also still require actual VM acceptance. -Backup data and manifests contain private credentials/source/testcase data and -must stay outside Git and CI artifacts. Parsing, ordinary adapter tests or a -manifest digest are not cold-backup/recovery acceptance evidence. +Backup domains include PostgreSQL, source/case storage, private import staging, +retained releases/images, configuration, replay history and operation receipts. +Every member records bounded digest/size and original identity/mode; copies remain +Root-only. Registry references must match. Links, special files, changing sources, +unknown identities, insufficient space and incomplete cuts are refused. Failed +targets remain unpublished for inspection. + +| Limit | Value | +| --- | --- | +| Per snapshot | 7 GiB and 50,000 entries | +| Stored snapshots | At most two | +| Declared retention | Seven days | + +`backup verify --backup-id ID +--manifest-sha256 SHA` checks the supplied private receipt and every member. +`backup prune --backup-id EXPIRED_ID --manifest-sha256 SHA +--keep-backup-id NEWER_ID --keep-manifest-sha256 SHA` verifies both inventories, +requires an expired target and a newer unexpired retained copy, then removes only +the named known inventory under the maintenance lock. It retains the last verified +copy and refuses unknown/partial trees. Capacity refusal deletes nothing. + +Private v3 backups bind the stopped broker cookie, current DB bootstrap fingerprint +and recovery history. Earlier private v1/v2 prototypes are refused. These private +versions do not alter judging canonical bytes or historical reports. Keep all +backup data and manifests out of Git and CI artifacts. + +## Restore to a clean node + +Install a verified public bundle on a different clean boot/node. Generate and +select a newly attested batch, then install dependencies with a new broker cookie +on empty provisioned volumes. Keep services stopped and do not initialize a new +cluster over restored data. Use fresh Worker UUID, measurement nonce/epoch, +control tokens and all DB/MQ passwords. `python3.13 -m scripts.release.restore --recovery-id rr-<32-lowercase-hex> ---backup-id ID --manifest-sha256 SHA` performs the cold **data** step on a -different clean boot/node. First install a verified public bundle, generate and -select an attested fresh configuration, and explicitly install the dependency -snapshot with a new broker cookie on empty preprovisioned volumes. Keep all -services stopped; do not initialize a new PostgreSQL cluster over restored data. -The target must have new Worker UUID, measurement nonce/epoch, control tokens -and all DB/MQ passwords. Cold restoration currently requires identical migration -bytes, target platform/toolchains and runtime images; schema-changing upgrades -need their separate Backend Alembic procedure. +--backup-id ID --manifest-sha256 SHA` runs the cold data step. Compatibility +requires identical migration bytes, target platform/toolchains and runtime images; +schema-changing upgrades need a separate Backend Alembic procedure. -The fixed restore operator reads the backup volume through a read-only mount -view, under 512-MiB memory, zero swap, 32 tasks, 50% CPU and a 30-minute deadline. -It verifies the separate private receipt and every source byte before copying -to fixed targets. It preserves original data identities/modes and every retained -factory replay ID; a new boot's existing Sandbox preparation owns delegation -recreation. It copies old application/judge/dependency/proxy configuration and -maintenance receipts into private inactive `recovery/ID/history/`, never over -the selected current configuration. Unknown/nonempty data targets and differing -existing files are refused. Failed partial data stays stopped for explicit -inspection; it is never silently adopted or erased. +The operator reads the backup through a read-only mount, checks the private receipt +and every byte, then copies to fixed targets under 512 MiB, zero swap, 32 tasks, +50% CPU and a 30-minute deadline. Original identities/modes and replay IDs are +preserved. Old configuration and maintenance receipts go into inactive +`recovery/ID/history/`. The new boot's Sandbox preparation owns cgroup recreation. +Unknown/nonempty targets and differing existing files are refused; partial restores +remain stopped for inspection. -The `data-restored.json` receipt proves only byte/identity restoration. The -operator starts no service, changes no schema/grant/grade/lease and reports DB -credential rotation and execution verification as false. +`data-restored.json` records only byte/identity restoration. Services stay stopped +and DB credential rotation and execution verification remain false. + +### Rotate database credentials `python3.13 -m scripts.release.recovery_database --recovery-id ID --data-receipt-sha256 SHA --password-file PRIVATE_NEW_BOOTSTRAP ---password-sha256 SHA` dispatches the credential step only after that boot's -selected complete data receipt. Supply a new 256-bit bootstrap password; its -fingerprint must differ from the separately verified private backup's current -bootstrap fingerprint. Backup capture requires one complete initialization or -recovery marker for the actual cluster, so a restored node can be backed up again. -Current -service endpoints are loaded from the new protected configuration, not backup -URLs. Missing/different authority, an active cluster, unknown unit or incomplete -cleanup is refused; this is an explicit operation, not service startup behavior. +--password-sha256 SHA` requires this boot's selected complete data receipt and a +new 256-bit bootstrap password with a different fingerprint from the backup. +Backup capture requires a complete initialization or recovery marker for the +actual cluster, allowing a restored node to be backed up again. Endpoints come +from current protected configuration. -One ordinary `kf-postgres` temporary unit gets 512-MiB memory, zero swap, 64 -tasks, 100% CPU, a four-minute deadline, cap0/no-new-privileges and private PID/ -network/runtime namespaces. It uses a private 0700 socket directory with TCP -listening disabled. The [PostgreSQL peer mechanism](https://www.postgresql.org/docs/18/auth-peer.html) -maps only this local OS identity to the fixed maintenance database login; fixed -service logins use SCRAM for their credential checks. No trust authentication, -external database listener or runtime Root business credential is introduced. +One ordinary `kf-postgres` temporary unit uses 512 MiB, zero swap, 64 tasks, 100% +CPU, a four-minute deadline, cap0/no-new-privileges and private PID/network/runtime +namespaces. TCP is disabled; the private 0700 socket uses local peer authentication +for this OS identity only. Fixed service logins use SCRAM. -The process checks effective PostgreSQL memory/time/socket/logging settings and -the exact existing roles/memberships/database owner. It rotates only passwords -for the five fixed service logins and `postgres` in one transaction, verifies -unchanged role authority and authenticates each new credential. It grants no -membership, creates no database/table, invokes no business repair and does not -rewrite grades, leases or measurement history. SQL/error output containing -utility statements is suppressed. Temporary-server exit and empty resources -must be confirmed before the dispatcher writes `identity-recovered.json`. +The process checks PostgreSQL settings, roles, memberships and database ownership. +It rotates the five service logins and `postgres` in one transaction, preserves +authority and authenticates every new credential. It changes no schema, grade, +lease or measurement history. SQL/error utility text is suppressed. Confirm +server exit and empty resources before publishing `identity-recovered.json`. -Startup and credential recovery share the cold data step's fixed lock. Startup -holds a read-only shared lock through admission and validates the linked prepared, -data and identity receipts in each bounded Root-private recovery directory. -Missing/incomplete/foreign receipts refuse startup; a committed password change -without a verified shutdown receipt cannot open admission. Complete historical -receipts remain valid after reboot or a later selected version; they do not make -the current services ready. Previous recovery history is backed up and restored -into inactive history, never used as the new node's completion receipt. +Recovery and startup share a fixed lock. Startup holds a read-only shared lock +through admission and validates linked prepared/data/identity receipts in the +bounded Root-private recovery directories. Missing/incomplete/foreign receipts +keep admission closed, including a committed credential change without confirmed +shutdown. Historical receipts remain valid after reboot or version change but do +not establish current readiness. They stay in inactive history on later restores. -Recovery still needs Backend Alembic verification, new TLS inputs, service -readiness and actual four-language/history checks in the second VM. Neither -data nor credential receipts assert execution safety or healthy public service. -These system steps and the 30-minute end-to-end recovery target remain pending -independent VM acceptance. +Finish recovery with Backend Alembic verification, new TLS material, service +readiness and actual four-language/history checks on the second VM. The 30-minute +end-to-end target still requires measured acceptance. -`python3.13 -m scripts.release.node_observation` dispatches one finite, -read-only local observation through its fixed Root cap0/no-new-privileges helper -(128 MiB, zero swap, 16 tasks, 25% CPU, 120 seconds, single named unit). -It reads kernel memory/PIDs current and peak, CPU throttling, memory events, -aggregate device IO and filesystem bytes/inodes without enumerating private -source, hidden cases or credential files. Current selected service identities, -ACLs, execution leases/cleanup, broker/outbox/import backlog and finite factory -capacity come from the existing status owners. Notification probing stays off; -it does not publish, claim, acknowledge, close admission or repair state. +## Observe resources and queues -The result includes source/configuration binding, a sample time and observation -duration. Independent readings are a bounded time window, not an atomic backup -cut. Missing kernel fields, stopped services or failed owner queries are explicit -incomplete observations; there is no invented zero/healthy fallback. Busy -admission can remain false in a complete observation. `isolation_verified` -remains false: every real execution still needs its own Sandbox gate. Private -operational output is for the local operator and must stay outside public API, -Git and CI artifacts. Actual resource/queue observations and long-run evidence -still require the independent VM. +`python3.13 -m scripts.release.node_observation` uses a finite read-only Root cap0, +no-new-privileges helper: 128 MiB, zero swap, 16 tasks, 25% CPU, 120 seconds and one +named unit. It reads memory/PIDs current/peak, CPU throttling, memory events, device +IO and filesystem bytes/inodes without enumerating private content. Existing owners +supply service identities, ACLs, leases/cleanup, broker/outbox/import backlog and +factory capacity. Notification probing stays off. + +Output includes source/batch, sample time and duration. Readings cover a bounded +time window, not an atomic backup cut. Missing fields, stopped services and failed +owner queries are explicit incomplete observations. Busy admission can be false +in a complete sample; `isolation_verified` remains false. Keep operational output +outside public API, Git and CI artifacts. + +## References + +- PostgreSQL: [role membership](https://www.postgresql.org/docs/18/role-membership.html), + [function ownership](https://www.postgresql.org/docs/18/sql-alterfunction.html), + [initdb](https://www.postgresql.org/docs/18/app-initdb.html), + [peer authentication](https://www.postgresql.org/docs/18/auth-peer.html), + [resources](https://www.postgresql.org/docs/18/runtime-config-resource.html), + [privilege functions](https://www.postgresql.org/docs/18/functions-info.html). +- RabbitMQ: [limits](https://www.rabbitmq.com/docs/limits), + [definition import](https://www.rabbitmq.com/docs/definitions), + [access control](https://www.rabbitmq.com/docs/access-control), + [CLI](https://www.rabbitmq.com/docs/man/rabbitmqctl.8). +- NGINX: [core](https://nginx.org/en/docs/http/ngx_http_core_module.html), + [proxy module](https://nginx.org/en/docs/http/ngx_http_proxy_module.html), + [TLS module](https://nginx.org/en/docs/http/ngx_http_ssl_module.html). +- systemd: [credentials](https://systemd.io/CREDENTIALS/), + [namespace interface](https://github.com/systemd/systemd/blob/v262/man/systemd.exec.xml), + [journal configuration](https://github.com/systemd/systemd/blob/v262/man/journald.conf.xml). +- Linux: [proc mountinfo](https://github.com/torvalds/linux/blob/v6.18/fs/proc_namespace.c).