Kronecker/KF-OnlineJudge
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

docs(release): organize installation and operator guides by task

Browse Source
This commit is contained in:
Kronecker
2026-10-14 10:14:35 +08:00
parent 16bcce3f04
commit e742c3543d
3 changed files with 618 additions and 667 deletions
Download Patch File Download Diff File Expand all files Collapse all files
deploy/README.md
+125 -118
View File
@@ -1,10 +1,12 @@
# 本地开发依赖 # 本地开发依赖
管理员账号 bootstrap/恢复、有界题包解析服务、不可变发布/下架、审计与回收步骤见 开发环境用 Compose 提供数据库和消息队列。单机交付使用独立的受限服务，安装入口见
[题目维护运维说明](admin-problems.md)。该说明与服务模板用于受控部署准备，不表示已部署生产。 [单机交付说明](release/README.md)。管理员账号、题包导入、发布、下架和审计见
[题目维护运维说明](admin-problems.md)。
`compose.dev.yml` 只提供 PostgreSQL 和 RabbitMQ，端口绑定 localhost。 ## 启动开发依赖
先复制 `.env.example` 为 `.env`，然后在仓库根目录运行：
将 `.env.example` 复制为 `.env`，在仓库根目录运行：
```bash ```bash
make infra-config make infra-config
@@ -12,75 +14,73 @@ make infra-up
make infra-down make infra-down
``` ```
`infra-down` 保留数据卷；不要把示例口令用于正式部署。 `compose.dev.yml` 只启动 PostgreSQL 和 RabbitMQ，端口绑定 localhost。
API 存活检查和前端预览不依赖这些容器，可以先独立启动。 `infra-down` 保留数据卷。API 存活检查和前端预览可以先独立启动。
该 Compose 不运行判题 Worker 或不可信代码。 不要在这个 Compose 或宿主机上运行学生代码，也不要将示例口令用于正式部署。
镜像固定具体维护版本（PostgreSQL 18.6、RabbitMQ 4.3.6），正式实验前另锁镜像摘要。 开发镜像固定 PostgreSQL 18.6 和 RabbitMQ 4.3.6；正式实验还需锁定镜像摘要。
配置检查只验证 Compose 结构；成功启动、实际健康与备份恢复需要单独验证。 配置检查只检查 Compose 结构。服务健康和备份恢复要在实际环境验证。
后续受控主机服务和最小权限沙箱启动器配置放在本目录。 镜像用法见官方 [PostgreSQL](https://hub.docker.com/_/postgres) 和
[RabbitMQ](https://hub.docker.com/_/rabbitmq) 说明。
官方镜像说明：[PostgreSQL](https://hub.docker.com/_/postgres)、 ## 单机 helper 工厂
[RabbitMQ](https://hub.docker.com/_/rabbitmq)。
### 单机 helper 工厂 `kf-sandbox factory --config /etc/kf-judge/runtime.json` 启动 Root 工厂。
普通 Worker 连接配置中的 `state_root/factory/control.sock`。
安装 `deploy/kf-sandbox-factory.service` 示例前，管理员需核对固定二进制、不可写的
GNU 镜像与配置、Worker UID/GID，以及 Root 所有且与监督器 cgroup 分离的任务委派根。
工厂读回服务和任务预算，每次编译、测试点运行都经过完整隔离检查。
`kf-sandbox factory --config /etc/kf-judge/runtime.json` 是 Root 服务入口； 工厂无等待队列，只接收有界 run ID。槽位满、重复 ID 和未知字段立即拒绝。
普通 Worker 使用配置中 `state_root/factory/control.sock` 的普通用户客户端。 每次请求创建全新的单线程 helper，继承已认证连接和工厂生命线。
`deploy/kf-sandbox-factory.service` 给出有限服务预算示例，安装前由可信管理员 额外控制帧、断连或生命线消失会终止任务树。ID 先进入持久记录，最多 4096 个，
核对固定二进制、不可写 GNU 镜像/配置、Worker UID/GID，以及**与服务自身 cgroup 跨重启继续拒绝重放。清理失败时保留身份租约并停止分配；恢复只处理已记录的
分离**的 Root 所有任务委派根。工厂实际读回服务 cgroup 限额，应用/读回任务总预算； 归属和 inode，未知资源保留并拒绝启动。
每次编译和测试点仍须通过已有完整隔离门控。服务文件本身不证明这些条件成立。
工厂只接收有界 run ID，无排队：槽位满、重复 ID 或未知字段立即拒绝。 ### 轮换记录周期
每次申请使用全新单线程 helper，固定继承已认证的连接与工厂生命线；
额外控制帧、断开或生命线消失会终止整棵任务树。ID 先写入有限持久记录
（最多 4096 个，跨重启拒绝重放）；容量用尽需要管理员在所有任务停止、已核对
回收后安排新记录周期，不自动删除记录。清理失败保留身份租约并停止新分配；
重启只根据已记录归属和 inode 回收，未知/外来内容保留并拒绝启动。
工厂 `--max-requests` 由可信 Root 启动参数选择，范围1–4096，默认4096； 可信 Root 参数 `--max-requests` 设置本周期受理上限，范围 1 到 4096，默认 4096。
它限制当前记录周期的受理次数，不改变学生预算或已记录 ID 的拒绝规则。 容量用尽后，先停止任务并核对回收，再由管理员维护记录周期。
`kf-sandbox rotate-factory-cycle --config ROOT_CONFIG --cycle-id ID` 是独立的
Root 维护入口：在受限监督器作用域中，核对受保护配置/镜像、取得工厂排他锁，
并读回任务、身份、捕获目录和委派 cgroup 为空、控制 socket 已消失后，才把
当前请求记录保留到 Root 私有 `factory/cycles/`，fsync文件与目录后清空当前记录。
最多保留64个有界周期；历史 ID 在后续启动仍用于拒绝重放。正常启动、重启、
寿命结束和容量用尽不会调用轮换或删除任何历史记录。归档完成但当前记录未清空
的中断状态拒绝启动，需在停止/排空后显式使用同一个 cycle ID 重试；未知或
不完整记录保留并拒绝，不能通过删日志恢复。该入口不启动服务，也不证明隔离。
新安装路径下的实际轮换、崩溃恢复、跨周期重放拒绝与资源读回仍需独立 VM 验收。
`kf-worker judge-submission` 是受控本地适配入口。常驻 `kf-worker serve --config` `kf-sandbox rotate-factory-cycle --config ROOT_CONFIG --cycle-id ID` 在受限监督器作用域
使用有界 AMQP 通知和经过服务身份认证的 Coordinator claim；配置与依赖见 中核对配置和镜像、取得工厂排他锁，确认任务、身份、捕获目录、委派 cgroup 均为空，
[判题工程说明](../judge/README.md#单机持久化协调与常驻-worker)。 且控制 socket 已消失，才归档请求记录到 Root 私有 `factory/cycles/`。
它验证不可变清单摘要，C17/C++20 仅映射到固定 GNU 命令，编译一次； 归档文件和目录经 fsync 后清空当前记录，最多保留 64 个周期；历史 ID 继续用于拒绝重放。
成功产物经完整清理后封印并计算 SHA-256，每点分别申请 helper、stdin 与工作区。
期望值逐点读取，留在比较端，不传给工厂/用户文件系统。
新增锁定 `sha2 0.10.9` 仅供 Worker 验证输入及编译产物摘要；
Sandbox 同版本仅为独立 VM 测试数据生成的开发依赖，Engine 不依赖加密或 OS 库。
### 单机提交服务的启动与停止 正常启动、重启、寿命结束和容量用尽不会自动轮换或删除历史记录。
若归档完成后中断，先排空，再显式用同一 cycle ID 重试。未知或不完整记录需要检查，
不能删日志续用工厂。轮换入口不启动服务。新安装路径的轮换、崩溃恢复和跨周期拒绝
仍需独立 VM 验收。
1. Backend Alembic 迁移；创建分别仅继承平台/协调器 NOLOGIN 角色的专用登录账号， ## 单机提交服务的启动与停止
平台角色只可插入初始评测，执行状态仅由 Coordinator 写入。
2. 普通平台服务持有私有工件目录；使用应用服务导入不可变题目和创建测试账号。
3. 验证固定 GNU 镜像、Root 配置及工厂服务/任务 cgroup 的真实上限，再启动工厂。
4. 独立启动 API、Coordinator、outbox publisher 和普通 Worker；单机 HTTP/AMQP
仅数值 loopback，PostgreSQL loopback/Unix socket。任何入口都不自动创建表。
5. Coordinator readiness 验证数据库和权限；仍需独立 VM 验证真实执行门控。
Root 工厂不可持有平台/MQ/数据库口令，用户运行镜像不含服务文件。
6. 先向 Worker 发送 SIGTERM，使当前任务取消/清理，再停止 publisher/API/Coordinator，
最后停止工厂和本地依赖。强制死亡后的旧 attempt 必须撤销并取得实际停止证明；
`cleanup_failed` 不等于普通终态，先核对保留资源再由管理员恢复工厂/Worker。
平台配置私有 `KF_OJ_ARTIFACT_ROOT`、有界字节/文件容量以及 `kf-worker judge-submission` 是受控本地适配入口；常驻
`KF_OJ_JUDGE_ADMISSION`（完整可信 JSON）；其工具链、编译/运行上限、总体作业期限、 `kf-worker serve --config` 使用有界 AMQP 通知和认证后的 Coordinator claim。
并发/prefetch/retry 必须与 Coordinator/Worker 一致。单机使用并发1、prefetch1， 配置见[判题工程说明](../judge/README.md#单机持久化协调与常驻-worker)。
不能仅提高 Worker 数量绕过节点容量。源码最多256KiB，测试点最多256个；
stdout/stderr 各不超过 file_bytes，合计不超过 workdir_bytes。
迁移账号与平台账号分开配置，命令均在 `backend/` 运行： Worker 校验不可变清单摘要，将 C17/C++20 映射到固定 GNU 命令，编译一次。
产物经完整清理后封印并计算 SHA-256；每个测试点单独申请 helper、stdin 和工作区。
期望输出留在比较端。Worker 用 `sha2 0.10.9` 校验输入与产物；Sandbox 的同版本
开发依赖仅生成独立 VM 测试数据，Engine 不依赖加密或 OS 库。
### 启动顺序
1. 用独立迁移账号执行 Backend Alembic。平台与协调器登录账号分别继承各自的
NOLOGIN 角色；平台只插入初始评测，执行状态由 Coordinator 写入。
2. 为平台准备私有工件目录，通过应用服务导入不可变题目并创建测试账号。
3. 验证 GNU 镜像、Root 配置、工厂服务和任务 cgroup 的实际限额，启动工厂。
4. 独立启动 API、Coordinator、outbox publisher 和普通 Worker。HTTP/AMQP 使用
数值 loopback，PostgreSQL 使用 loopback/Unix socket；启动入口不创建表。
5. 检查 Coordinator 数据库权限与 readiness，再在独立 VM 验证执行隔离。
Root 工厂不持有平台/MQ/数据库口令，用户运行镜像不含服务文件。
平台配置包括私有 `KF_OJ_ARTIFACT_ROOT`、工件容量和可信
`KF_OJ_JUDGE_ADMISSION`。工具链、编译/运行预算、作业期限、并发/prefetch/retry
必须与 Coordinator/Worker 一致。单机并发 1、prefetch 1，共享任务身份池。
源码最多 256 KiB，测试点最多 256 个；stdout/stderr 各不超过 file_bytes，
合计不超过 workdir_bytes。增加 Worker 数量不会扩大节点槽位。
以下开发命令均在 `backend/` 运行；迁移账号与平台账号分别配置：
```sh ```sh
uv run --locked alembic upgrade head # 私有迁移账号，仅执行受控 schema 操作 uv run --locked alembic upgrade head # 私有迁移账号，仅执行受控 schema 操作
@@ -91,70 +91,77 @@ uv run --locked uvicorn kf_oj.api.app:create_app --factory --host 127.0.0.1 --po
uv run --locked python -m kf_oj.infra.messaging.publisher uv run --locked python -m kf_oj.infra.messaging.publisher
``` ```
每个账号操作/导入目录都由管理员准备，不从浏览器接受私有路径。导入格式见 账号和导入目录由管理员准备，浏览器不能传入私有路径。导入格式见
[题目模块](../backend/src/kf_oj/modules/problems/README.md)，不可变版本以 UUID [题目模块](../backend/src/kf_oj/modules/problems/README.md)。题库、题面和提交使用
在题库/题面链接和提交请求中传递。更新题目需导入新版本，不覆盖已有提交的题目版本。 不可变版本 UUID；更新需发布新版本。临时验收使用合成数据和测试口令，业务数据与
临时验收使用合成数据和测试口令，实际业务数据与服务配置都保存在仓库外的私有目录。 配置放在仓库外。
取消另需 API 私有配置 `KF_OJ_COORDINATOR_URL=http://127.0.0.1:8100` 与 取消功能还需 API 配置 `KF_OJ_COORDINATOR_URL=http://127.0.0.1:8100` 和
`KF_OJ_COORDINATOR_TOKEN`（Coordinator 的 platform_token）。所有服务令牌私下生成， `KF_OJ_COORDINATOR_TOKEN`，后者对应 Coordinator 的 platform_token。
不可互用或放进前端环境。Coordinator/Worker 私有 JSON 为0600、单链接、有界文件； 独立生成服务令牌，保存在私有配置中。Coordinator/Worker JSON 为 0600、单链接、
Root 镜像/配置/二进制由 root 持有，任务委派子树独立于监督器自身 cgroup。 有界文件；Root 持有镜像、配置和二进制。运行前读回节点、服务和任务的
运行前实际读回整体节点、服务和任务 memory/swap/pids/CPU/文件/运行期限上限， memory/swap/pids/CPU/文件/期限上限。
并通过真实隔离门控；配置解析、readiness、存活探测都不能替代执行验收。
前端从 `frontend/` 启动 `pnpm dev`，默认127.0.0.1:5173；平台明确允许相同浏览器 开发前端在 `frontend/` 运行 `pnpm dev`，默认 127.0.0.1:5173。
Origin，`/api` 同源代理。生产需另行配置 HTTPS、安全 cookie、SPA 回退与 `/api` 平台明确允许相同 Origin，`/api` 使用同源代理。生产入口还需 HTTPS、安全 Cookie、
反向代理，本仓库不自动部署。用户以受控账号登录，在题面选择支持语言、提交源码， SPA 回退和 API 反向代理。登录后在题面提交源码，按同一 submission UUID 查看状态；
随后查看同一 submission UUID 的持久化状态。等待/取消中/系统失败都不等于成功成绩。 等待、取消中和系统失败各有独立状态。
broker 暂停时 API 仍能持久化 queued/outbox；恢复后 publisher confirm 与 Coordinator ### 停止与故障恢复
补偿重新投递。Worker/Coordinator 重启使用原私有配置、工件、日志与 DB，不删除
attempt 或强改状态。旧租约不能覆盖新结果，有限基础设施重试最终收敛。
容量满时返回429或503；先确认待执行/清理状态，恢复服务或由管理员扩充经过核对的
存储容量，不删除仍被提交/题目引用的工件。`kf-artifacts --limit 100` 仅回收满足年龄
要求的无引用工件和已标记删除项；它不会通过删除用户源码来腾出容量。
恢复 Root 必须核对 journal/inode/实际排空；未知资源保留，管理员处理自己确认的
故障来源后再重启。不要递归清空任务目录，也不要把 cleanup_failed 改成已成功。
这些入口和顺序不表示生产部署已完成。隔离/超限与用户程序仅在无 NIC、无宿主共享 先向 Worker 发送 SIGTERM，使当前任务取消并清理，再停止 publisher/API/Coordinator，
挂载、无业务凭据且整体上限已读回的独立测试 VM 内验收。开发 Compose 仍只提供可信 最后停止工厂和本地依赖。强制死亡后的旧 attempt 必须撤销并取得停止证明。
数据库和 broker，不在宿主或业务服务器运行用户代码。workflow 如需启用由指定 `cleanup_failed` 保留未确认资源，管理员检查故障来源后再恢复工厂/Worker。
GitHub 私有仓库承载，并设置有限并发；Gitea 只保存逐阶段工程提交。 不要递归清空任务目录或强改结果。
broker 暂停期间，API 保留 queued/outbox；恢复后 publisher confirm 与 Coordinator
补偿重新投递。重启 Worker/Coordinator 时保留配置、工件、日志、DB 和 attempt。
旧租约不能覆盖新结果，基础设施重试有次数上限。
容量满时返回 429 或 503。先检查待执行和清理状态，再恢复服务或核对扩容。
`kf-artifacts --limit 100` 只回收达到年龄要求的无引用工件和已标记删除项，
保留仍被题目或提交引用的源码。Root 恢复还需核对 journal、inode 和实际排空。
## 四语言节点的受控升级 ## 四语言节点的受控升级
新增公开示例 `judge.admission.example.json` 为显式 v1/v2 注册表， `judge.admission.example.json` 是显式 v1/v2 注册表，
`sandbox.managed.runtime.example.json` 为一槽位的共享 Root 配置；原 v1 示例保持原语义。 `sandbox.managed.runtime.example.json` 使用一槽位共享 Root 配置。
所有 `aaaaaaaa…` 镜像摘要均是无权限占位值，不能当作已验镜像。管理员制作固定镜像后， 原 v1 示例语义保留。示例中的 `aaaaaaaa…` 是无权限占位摘要，必须换成固定镜像
对 manifest.json 的实际字节计算 SHA-256，逐文件验证版本/字节/权限，并将相同完整身份 manifest.json 的实际 SHA-256，并逐文件验证版本、字节和权限。
写入准入注册表与 Root runtime_images。Root 镜像路径仅在私有可信配置中保存。 准入注册表与 Root runtime_images 使用同一完整身份；Root 镜像路径仅放入私有配置。
配置解析通过或 check-runtime 可见仍不允许跳过实际隔离门控。
1. 停止创建新提交/消费投递，向所有普通 Worker 发 SIGTERM；保留 DB、清单、工件和 journal。 1. 停止创建提交和消费投递，向普通 Worker 发 SIGTERM，保留 DB、清单、工件和 journal。
2. 取得工厂实际 stopped/cleanup_complete 证明，读回空任务 cgroup 和 owned 资源； 2. 确认工厂 stopped/cleanup_complete，读回空任务 cgroup 和 owned 资源。
未排空或存在未知资源时停止升级，保留证据，不递归清空任务目录。 发现未知资源或未排空时停止升级并保留证据。
3. 使用独立迁移账号执行 Backend Alembic upgrade；新语言数据存在时旧迁移降级会拒绝， 3. 用独立账号执行 Backend Alembic upgrade。含新语言数据时旧迁移拒绝降级；
不删除数据以实现回退。平台与 Coordinator 继续使用各自最小权限角色，Worker 无 DB 账号。 保留这些数据。平台与 Coordinator 使用各自最小权限角色，Worker 无 DB 账号。
4. 安装固定版本的 Backend/Coordinator/Worker/Sandbox；同时更新三个普通服务的同一 4. 安装固定版本 Backend/Coordinator/Worker/Sandbox，同时更新普通服务的同一
`judge_admission`/`admission` 注册表，Root 配置单独由管理员持有。旧消费者不混用新协议。 `judge_admission`/`admission`。Root 配置单独保管，旧消费者不混用新协议。
5. 为整体节点、控制面各服务、工厂和任务设置有限 CPU/memory/swap/pids/文件/期限， 5. 设置并读回节点、控制面、工厂和任务的 CPU/memory/swap/pids/文件/期限上限。
读回内核值。节点至少预留一份512MiB编译任务以及独立控制面；Java运行示例总预算384MiB， 至少预留一份 512 MiB 编译任务和独立控制面；Java 运行示例总预算 384 MiB，
Xmx不能代替整体预算。并发1/prefetch1，共享UID池，不因第二Worker存在而扩大节点槽位。 Xmx 只是堆上限。并发 1、prefetch 1，使用共享 UID 池。
6. 在独立无NIC、无宿主共享挂载、无业务凭据 VM 用合成题目验证四语言、故障与实际排空， 6. 在断网、无宿主共享挂载、无业务凭据的受限独立 VM 验证四语言、故障和排空，
再恢复工厂/Coordinator/API/publisher/Worker。生产启用需单独安排，仓库不自动部署。 再恢复服务。生产启用单独安排。
请求不在线安装 Python/Java 依赖，不运行 pip/uvx，不接收自定义解释器参数/JAR/agents。 Python `main.py` 仅支持标准库。Java `Main.java` 使用默认包 Main，最多 256 个
Worker 私有配置可设置 `control_timeout_ms`（1000–5000，缺省2000）作为 Coordinator 有界编译 class。请求不在线安装依赖，不运行 pip/uvx，不接收解释器参数、JAR 或 agents。
单次控制请求的期限。资源受限节点应依据实际领取/续租/停止耗时选择，并核对租约和 题目版本决定语言和预算，旧题目、草稿和报告保留；Rust 暂不支持。
停止恢复；这不改变学生阶段/作业期限、重试次数或工件下载的固定2秒期限，连接仍限1秒。 CE、RE、真实超限与 SystemError 的区别见
Python main.py 仅标准库；Java Main.java 使用默认包 Main，最多256个有界编译class。 [判题说明](../judge/README.md#python--java-与明确协议版本)。
题目语言与预算以公开版本配置为准，旧题目和草稿保留，Rust继续明确不支持。
新版检查阶段CE/运行RE/真实限额与SystemError的区别见 [判题说明](../judge/README.md#python--java-与明确协议版本)。
停止/故障恢复沿原单机流程：停止Worker使任务全树排空，旧租约/report拒绝后恢复； Worker 的 `control_timeout_ms` 可设为 1000 到 5000，缺省 2000。
publisher/broker中断保留outbox，有限重试而非清空记录。恢复旧软件前应先排空v2任务， 按实际领取、续租和停止耗时选择，并核对租约与恢复流程。
保留不可变v2版本和报告供兼容消费者读取；不能把旧配置静默转换为宽松新能力。 该值不改变学生阶段/作业期限、重试次数、工件下载的固定 2 秒期限或 1 秒连接期限。
这些公开配置不含业务令牌。数据库/AMQP/服务令牌分别在仓库外私下生成，Root不持有它们。
需要CI时只在GitHub私有仓库 `KiriAky107/KF-OnlineJudge` 运行并设置有限并发，Gitea保存源码。 回退旧软件前先排空 v2 任务，保留 v2 不可变版本和报告供兼容消费者读取。
恢复时沿用停止与故障流程，保持 outbox 和有限重试。
公开示例不含业务令牌；数据库、AMQP 和服务令牌分别在仓库外生成，Root 不持有它们。
## 验收与 CI
所有学生源码、隔离、超限和故障测试都在无 NIC、无宿主共享挂载、无业务凭据且
整体上限已读回的独立 VM 运行。配置解析、readiness 和存活探测不能替代执行验收。
仓库不自动部署生产。
需要 workflow 时，在 GitHub 私有仓库 `KiriAky107/KF-OnlineJudge` 设置有限并发执行。
Gitea 保存工程提交，不运行 workflow。
deploy/release/README.md
+119 -92
View File
@@ -1,107 +1,134 @@
# Single-node delivery contract # Single-node delivery
`contract.v1.json` describes the first pre-release installation interface. It is The single-node profile packages the application, four judging runtimes and the
separate from judging v1/v2 wire formats and never rewrites historical reports. installation tools for offline use. `contract.v1.json` defines the installation
The reference target is Arch/CachyOS Linux x86_64-v3, glibc 2.44, systemd and cgroup interface. Judging v1/v2 reports keep their existing formats.
v2. Images and service libraries are bound to explicit file/catalog fingerprints;
support for other distributions requires a separately verified delivery profile.
The target has at least 4 GiB RAM and 12 GiB installation/data capacity. Judging Full installation and recovery acceptance is in progress. Validate a release on
and problem-import concurrency remain one each. Configured service and execution an independent test node before using it for production.
budgets, effective kernel limits and free capacity must also pass preflight; total
RAM alone does not prove enough capacity or verified execution isolation.
Preflight requires at least 12 GiB of filesystem capacity and derives required
free bytes from the verified application, protected runtime-image copies and
separate GNU legacy copy, plus 4 GiB of mutable-data/operation reserve. It does
not assume that a nominal 12-GiB filesystem exposes 12 GiB free after metadata.
Release payloads are directory bundles with a bounded manifest: at most 50,000 ## Requirements
single-link regular files, 256 MiB per file, 4 GiB total and 8 MiB metadata. The
manifest SHA-256 is supplied separately by the operator. Links, extra members, | Item | Required profile |
private paths and unexpected file changes are refused. Built payloads and real | --- | --- |
configuration never belong in Git. Validation is read-only and does not claim | OS | Arch/CachyOS, Linux x86_64-v3, glibc 2.44, systemd 262, cgroup v2 |
that any service has been installed, started or verified: | Memory | At least 4 GiB, with enough free capacity for the configured services and tasks |
| Storage | At least 12 GiB filesystem capacity; preflight calculates the required free space |
| Concurrency | One judging job and one problem import |
| Dependencies | PostgreSQL 18.6 and the fixed broker runtime, prepared from verified offline materials |
Other distributions need a separately verified profile. Preflight checks the
effective kernel limits and available capacity as well as nominal RAM and disk
size. Its free-space calculation includes the application, protected runtime
images, the separate GNU legacy image and 4 GiB for mutable data and operations.
## Verify a bundle
Use a separately trusted checker and the manifest digest supplied with the release:
```sh ```sh
python3.13 -m scripts.release contract deploy/release/contract.v1.json python3.13 -m scripts.release contract deploy/release/contract.v1.json
python3.13 -m scripts.release verify /path/to/bundle --manifest-sha256 SHA256_FROM_RECEIPT python3.13 -m scripts.release verify /path/to/bundle --manifest-sha256 SHA256_FROM_RECEIPT
``` ```
Application versions live below `/opt/kf-onlinejudge/releases`; `current` selects A bundle contains at most 50,000 single-link regular files, 256 MiB per file,
an explicitly installed version. Private operator/service configuration lives in 4 GiB in total and 8 MiB of metadata. Verification rejects links, extra members,
`/etc/kf-onlinejudge` and `/etc/kf-judge`, mutable application data under private paths and changed bytes. Keep bundles and real configuration outside Git.
`/var/lib/kf-onlinejudge`, and protected Sandbox state/images under
`/var/lib/kf-judge`. The delegated task cgroup is separate from the Root factory's
own supervision cgroup. Application processes do not initialize tables on start;
Backend Alembic remains the only migration entry point.
All application services and the separate delegated task subtree live beneath
`kfonlinejudge.slice`: 3840 MiB memory, no swap, 384 processes/threads and two CPU
equivalents. The remaining physical RAM is reserved for the kernel/minimal OS.
The task subtree is `/sys/fs/cgroup/kfonlinejudge.slice/kf-judge-tasks`, a sibling
of the Root factory supervision unit under that same total node budget. Its
installation template does not prove the effective limits; actual service and
task limits are read back in the independent acceptance environment.
Database login identities are separate for migration, API/outbox, Coordinator, Verification is read-only. Installation, running services and execution isolation
problem import and local identity operations. Rust Worker and Root factory do not each need their own acceptance evidence.
hold database credentials. Root factory has no MQ/platform/SSH/GitHub identity;
privileged execution remains inside Sandbox. Public and admin browser traffic
share the HTTPS site; execution-control interfaces remain internal.
Upgrade and restore must retain immutable problem versions, submitted source, ## Directories and authority
audit and original judging reports. A consistent backup cut has a zero confirmed
data-loss target; restoring from already prepared release/backup materials has a
30-minute recovery target, including service health and actual four-language
verification. These are acceptance targets, not results inferred from this file.
Installation/service/restore evidence is required before this pre-release can be
treated as an accepted delivery; the contract itself does not deploy production.
The node input for `generate-configuration` has exactly `schema_version: 1`, | Location | Contents |
`origin` (one HTTPS origin), `worker_id` (canonical nonzero UUID), and | --- | --- |
`measurement_nonce` (null or a fresh operator-generated SHA-256-sized nonce). | `/opt/kf-onlinejudge/releases` | Installed application versions |
It is a 0600 single-link file. Endpoint input is a separate 0600 JSON object | `current` | Explicitly selected application version |
with exactly `schema_version: 1`, `database` and `amqp`: database keys are | `/etc/kf-onlinejudge` and `/etc/kf-judge` | Private operator and service configuration |
`api`, `coordinator`, `import`, `migration`, `identity`; broker keys are `outbox` | `/var/lib/kf-onlinejudge` | Mutable application data |
and `worker`. Login names and numeric loopback/socket locations are validated by | `/var/lib/kf-judge` | Protected Sandbox state and runtime images |
the tool. Every endpoint requires its own operator-supplied private credential;
there is no working example password or shared default. Import uses the reachable
`/run/postgresql` Unix socket; all other control connections use numeric loopback.
Generated admission uses the actual verified catalogs for C17/C++20, CPython Backend Alembic is the only schema migration entry point. Database identities are
3.13.16 and OpenJDK 21.0.12.1+1-LTS and the unchanged public per-language budgets. separate for migration, API/outbox, Coordinator, problem import and local account
The protected GNU legacy image is a separate directory: it retains the original operations. Rust Worker has no database identity; the Root factory has no
v1 header and identical verified GNU executable/library bytes, while the three database, MQ, platform, SSH or GitHub credentials. Sandbox owns privileged
v2 catalog identities remain untouched. Shared Worker/task UID/GID pool and execution. Public and admin pages share the HTTPS origin; control interfaces stay
execution/prefetch/import slots remain one. No canonical judging report or internal.
historical runtime identity is converted by the installer.
Comparability defaults to unavailable (`measurement_epoch=null`). A trusted Application services and the task subtree share `kfonlinejudge.slice`, capped at
operator may explicitly use `--attest-measurement` only after actual node/service 3840 MiB, zero swap, 384 processes/threads and two CPU equivalents. The remaining
limits and the intended execution environment have been verified. The attestation physical memory is reserved for the kernel and minimal OS. Tasks use
binds a fresh nonce, worker, source, exact images, budgets, CPU/microcode, kernel `/sys/fs/cgroup/kfonlinejudge.slice/kf-judge-tasks`, separate from the Root
and node scheduling snapshot. `check-configuration` refuses an old binding when supervisor's cgroup. Read back service and task limits on the target node.
those values change; it does not manufacture replacement metrics for old reports.
Before any service resume after such a change, drain execution, generate/install
a new complete private group with a new nonce and independently revalidate the
environment. Raw configuration checks are not performance or isolation evidence.
Configuration batches have independent hash-bound `cfg-...` identities; files
and receipts contain both the source release and batch identifier. A fresh
nonce/identity can therefore be installed for the same source without replacing
an old configuration. Preparation never selects the new batch automatically.
`activate` requires an explicit `--configuration-id`, a fully validated group,
all application units inactive, stopped control ports and no retained task
resources. Root journals and all old batches remain intact. It binds the code
pointer and private batch in `/etc/kf-onlinejudge/active.json`; `check-active`
refuses inconsistent pointers, stale hardware bindings or changed bytes.
Interruption between the two atomic pointer writes requires operator review;
no service starts or claims readiness during that mismatch. A stopped stale
batch can be retired without accepting it for execution under a changed kernel.
The installation entry points are available with `python3.13 -m scripts.release ## Prepare configuration
--help`. Use them in order: verify/preflight, install-files, generate-configuration,
install-configuration, explicit DB identity provision, controlled Backend Alembic Supply two 0600 single-link JSON files to `generate-configuration`:
and account CLI operations, then activate and the separately verified bounded
service lifecycle. These operations never contact a package index and do not | Input | Fields |
install onto a business host on their own. OS dependencies (systemd 262, matching | --- | --- |
glibc ABI bytes, PostgreSQL 18.6 and the fixed broker runtime) must already be | Node | `schema_version: 1`, `origin`, `worker_id`, `measurement_nonce` |
present from separately verified offline materials. | Endpoints | `schema_version: 1`, `database`, `amqp` |
`origin` is one HTTPS origin. `worker_id` is a canonical nonzero UUID.
`measurement_nonce` is null or a fresh operator-generated SHA-256-sized nonce.
The database object has `api`, `coordinator`, `import`, `migration` and `identity`
keys; the broker object has `outbox` and `worker` keys. Supply a different private
credential for every endpoint. Import uses `/run/postgresql`; other control
connections use numeric loopback. The tool checks login names and locations.
The generated admission profile binds C17/C++20, CPython 3.13.16 and OpenJDK
21.0.12.1+1-LTS to verified catalogs and the existing per-language budgets. The
GNU legacy image retains its v1 header and verified executable/library bytes in
a separate directory. Worker, task identities and execution/prefetch/import
slots remain one. Historical reports and runtime identities remain unchanged.
## Select a version
Configuration batches have content-bound `cfg-...` identities. Preparation keeps
earlier batches and does not select a new one. `activate` requires an explicit
`--configuration-id`, a fully validated batch, inactive application units,
stopped control ports and no retained task resources.
Selection writes the code pointer and `/etc/kf-onlinejudge/active.json`. A crash
between those atomic writes can leave a mismatch; stop and inspect it before
starting services. `check-active` rejects inconsistent pointers, changed bytes
and stale hardware bindings. Root journals and earlier batches remain intact.
A stopped stale batch can be retired without accepting it under a changed kernel.
## Performance comparability
Comparability defaults to unavailable (`measurement_epoch=null`). Use
`--attest-measurement` after verifying the node's limits and execution environment.
It binds a fresh nonce, worker, source, images, budgets, CPU/microcode, kernel and
node scheduling snapshot. `check-configuration` rejects a changed binding.
After an environment change, drain execution, install a complete batch with a new
nonce and revalidate before resuming. Earlier reports retain their original
measurements. Configuration validation alone does not prove comparable performance
or execution isolation.
## Installation order
Use `python3.13 -m scripts.release
--help` for the command interface. Prepare the OS dependencies and dedicated data
volumes, then follow this order:
1. Verify the bundle and run preflight.
2. Install files and runtime images.
3. Generate and install a private configuration batch.
4. Activate that batch while services and tasks are stopped.
5. Install the journal profile and selected dependency snapshots.
6. Initialize PostgreSQL, provision DB identities, run Backend Alembic and create
application accounts; stop the initialization database before installing units.
7. Install ordinary units, the Root factory and the HTTPS snapshot/TLS material.
8. Start the bounded service lifecycle and verify readiness and actual execution.
The tools use offline materials and do not contact a package index.
See the [operator command reference](../../scripts/release/README.md) for service,
HTTPS, backup and recovery operations. Upgrades and restores must preserve
immutable problem versions, submitted source, audit and original judging reports.
A confirmed consistent backup cut has a zero data-loss target. Recovery from
prepared release and backup materials has a 30-minute target, including health
and four-language verification. Both targets require measured acceptance results.
scripts/release/README.md
+374 -457
View File
@@ -1,20 +1,13 @@
# Offline release tooling # Offline release tooling
This package is an operator composition tool. `policy` validates delivery bounds, Build and operate a KF-OnlineJudge node with locked dependencies and verified
`manifest` inventories/verifies explicit files, and `build` runs the locked trusted offline materials. The tools cover installation, configuration, HTTPS, maintenance
checkout build and copies only tracked application/deployment sources, fresh and cold recovery. Full system and recovery acceptance is still in progress.
production dependencies and separately fingerprinted public runtime inputs.
It does not import domain ORM/repositories, write application state, implement a
second migration path, execute student code or perform Sandbox privilege work.
`scripts/check_architecture.py` enforces its application import boundary.
All dependency resolution for a release uses existing lockfiles and offline For the target platform and installation order, read
caches. Backend source is shipped with locked production dependencies rather [`deploy/release/README.md`](../../deploy/release/README.md).
than invoking an unpinned isolated wheel build backend. uvx/pip and package
startup path hooks are not delivery runtime entry points. Build inputs specify a ## Build a release
verified public runtime inventory and the three existing image catalogs; an
independent expected SHA-256 binds each input. Real input paths/receipts are local
operator data and do not belong in Git.
```sh ```sh
python3.13 -m scripts.release build --repo /native/checkout \ python3.13 -m scripts.release build --repo /native/checkout \
@@ -22,504 +15,428 @@ python3.13 -m scripts.release build --repo /native/checkout \
--work /private/new-build-work --work /private/new-build-work
``` ```
The checkout must be clean and committed, build output/work directories new, Start from a clean committed checkout. Output and work directories must be new,
and the output filesystem have the required reserve. The resulting receipt and their filesystem must have the required reserve. The build uses lockfiles and
contains the source and manifest fingerprints, file count and byte total, with offline caches, then copies tracked application/deployment sources, fresh locked
`installed=false` and `execution_verified=false`. Installation and service production dependencies and separately fingerprinted public runtimes. Backend
acceptance remain separate operations. See `deploy/release/README.md` for the source ships with those dependencies; uvx/pip and startup hooks are not runtime
versioned installation contract and authority split. entry points.
`preflight` reads installation eligibility only. `install-files` requires root, Build inputs contain explicit inventories and expected SHA-256 digests for public
a completely root-controlled verified input tree and the fixed target profile; runtimes and the three image catalogs. Keep real input paths and receipts outside
it installs protected versions/images and bounded OS identities, without table Git. The build receipt records source and manifest digests, file count and byte
migrations, application accounts, services or the active-version switch. total with `installed=false` and `execution_verified=false`.
Repeated identical files converge; unknown modes/owners, changed bytes or an
incomplete staging directory require operator inspection and are never removed.
`generate-configuration` composes exact public contracts with separately supplied Use a separately trusted checker and an out-of-band manifest SHA to validate an
0600 node and endpoint inputs. It emits a new private directory and a hash-bound incoming bundle. A verified delivery includes `bin/kf-delivery` for later offline
receipt, never prints tokens, never changes a supplied password and never silently operations. Running an unverified bundle's checker cannot establish trust in that
overwrites an earlier identity. API, publisher, import, migration and identity bundle.
operations have distinct required database authority; Worker holds only its own
MQ/control identity, Root receives no business credentials. Backend URLs use the
existing asyncpg driver. No uvx/pip or online runtime installation is used.
`install-configuration` validates the entire group before writing any member; ## Authority and service identities
`check-configuration` checks installed bytes, same-source admission, identity and
hardware/kernel/node scheduler bindings. `activate` requires that check and
stopped control ports, no factory socket and no retained task/identity/capture
resources. It atomically selects an installed directory and starts no process;
draining, migrations and actual health remain separate operations. It does not
erase retained journals or release resources lacking cleanup proof.
Every private group also has a content-bound `configuration_id`. Installing a
second group for the same source preserves the first group byte-for-byte.
`activate RELEASE --manifest-sha256 SHA --configuration-id BATCH` selects an
explicit validated batch only after ordinary application units and independent
task resources are stopped. `check-active` checks the protected batch/code
binding and current measurement snapshot, with no liveness/readiness claim.
`provision-database-identities` is an explicit local database operator step. | Operation | Authority |
It uses the locked asyncpg dependency, bounded connections/statements and a | --- | --- |
120-second overall deadline. It installs only fixed independent logins/NOLOGIN | Build, verify and preflight | Read-only system access |
groups and the application database, verifies every supplied credential and | File/configuration installation and activation | Root operator |
refuses unexpected existing authority. There is no application table DDL or | Database identity provisioning | Supplied local DB operator |
automatic account bootstrap: the migration login owns the database without | Schema and grants | Backend Alembic |
global superuser/create-role/create-db/bypass-RLS rights; Backend Alembic remains | Application accounts | Backend account CLI |
the only schema/grant migration entry. Runtime logins cannot grant/set group | Execution state | Coordinator |
authority. The migration login inherits the narrow NOLOGIN function-owner role | Privileged task preparation and execution | Rust Sandbox |
and can SET that role, so unchanged Alembic can grant/rename its functions after
ownership transfer. Membership has no ADMIN option, adds no global role-creation
privilege, and is never given to runtime logins. See
[PostgreSQL role membership](https://www.postgresql.org/docs/18/role-membership.html)
and [function ownership](https://www.postgresql.org/docs/18/sql-alterfunction.html).
Always bootstrap validation from a separately trusted tool copy and an out-of-band `policy` validates bounds, `manifest` inventories and verifies files, and `build`
manifest SHA. Do not execute a checker supplied inside an unverified bundle to assembles the release. Release tools do not import domain ORM/repositories or
establish trust in that same bundle. A verified delivery includes `bin/kf-delivery` provide a second schema migration path. `scripts/check_architecture.py` checks
and its source for later offline operations. Command success/receipt generation the application import boundary. Installation and configuration commands start
does not claim Linux execution, working service roles or installation acceptance; no services; the maintenance commands below control their lifecycle.
these require the recorded independent VM evidence.
`generate-proxy` is a separate, offline composition entry. Its private 0600 API/outbox/import share the content-storage identity and use private PID views.
profile has exactly `schema_version: 1`, one canonical HTTPS DNS `origin`, Coordinator and Worker have distinct ordinary identities. `service_bootstrap`
`certificate_sha256` and `key_sha256`. TLS inputs are independently supplied validates Rust credential bytes and writes a private runtime copy with mode 0600
later at `/etc/kf-proxy/tls/fullchain.pem` and `key.pem`; the generator before launching the fixed binary. Tokens and URLs
does not read, create or verify a private key. It emits a new private NGINX stay out of arguments and unit text. Python launchers accept fixed modules,
configuration and receipt tied to the verified release. No arbitrary upstream, arguments and generated environment fields; inherited debugging, proxy and
include, path, credentials or resource overrides are accepted. Only browser API provider credentials are omitted. Import also requires a loopback-only network
families target the API loopback port; execution-control paths are refused. ZIP view and the same admission policy as API.
uploads have their own canonical UUID route. Syntax, actual TLS/stream limits,
static routes, permissions and service health require subsequent VM acceptance;
the generator explicitly returns those capabilities as unverified.
The separately installed `kf-proxy` worker identity must be distinct from API,
Coordinator, import, judging and database/broker identities. It has no business
credentials; generation alone never creates that account or starts NGINX.
Operator responsibilities are separate: build/verify are read-only with respect Every ordinary launch checks UID/GID, cap0, no-new-privileges, node/service cgroup
to system state; preflight observes eligibility; file/config installation and limits and process file/descriptor/stack limits. The Python services require PID 1
activation require the root operator; DB provisioning requires a separately inside their private namespace. Unsupported or ignored isolation settings refuse
supplied local DB operator; schema/grants and first account use the existing startup. Rust Worker has no database credentials. Root factory uses an empty
Backend Alembic/account CLIs. Configuration validation never starts or restarts business environment and gets no database, MQ or platform identity.
a service. The later bounded service lifecycle must prove drain/cleanup before
maintenance; none of these entries authorizes a Sandbox isolation bypass or
silently substitutes a healthy result when lifecycle evidence is absent.
Proxy directive references: [NGINX core](https://nginx.org/en/docs/http/ngx_http_core_module.html), ## Install files and configuration
[proxy module](https://nginx.org/en/docs/http/ngx_http_proxy_module.html), and
[TLS module](https://nginx.org/en/docs/http/ngx_http_ssl_module.html).
`python3.13 -m scripts.release.services generate` composes five ordinary system `preflight` checks eligibility. `install-files` requires Root, a Root-controlled
units from a verified bundle and matching private configuration receipt. The verified input tree and the fixed target profile. It installs protected versions,
matching service launch modules must themselves be in that bundle. Rendering images and OS identities. Repeat installation checks identical files; unknown
uses fixed users, resource ceilings, finite restart/stop limits and an explicit ownership/modes, changed bytes and incomplete staging directories require inspection.
release/configuration pre-start binding. It never writes system units, starts a
process, migrates a schema or claims readiness. Actual units, mount/PID/cgroup
views, Unix DB socket access, dependency roles and effective limits require VM
installation/acceptance. Root factory and database/broker lifecycle are separate.
API/outbox/import share the established content-storage identity; individual PID `generate-configuration` reads separate 0600 node and endpoint inputs and writes
namespaces prevent same-UID inspection of another service's environment. This a new private batch. API, publisher, import, migration and account operations use
requires the fixed systemd profile and verified kernel namespace support. Rust their required DB authority; Worker receives its own MQ/control identity. Backend
Coordinator/Worker use distinct ordinary identities and systemd credentials. URLs use asyncpg. The tool preserves supplied passwords and earlier identities.
Every ordinary launch checks effective service/node cgroup memory, swap, CPU and
PID limits, file/descriptor/stack limits, UID/GID and absence of capabilities
before exec. The three Python services also require PID 1 in their private PID
namespace; import requires a loopback-only network view. Ignored namespace
directives therefore refuse startup. These checks still require actual VM
verification; they do not attest Sandbox isolation or service readiness.
The import environment includes the same validated admission policy as API.
`service_bootstrap` verifies bounded Rust credential bytes, copies them to a `install-configuration` validates the entire batch before writing. A batch has a
private service runtime directory with the existing required 0600 mode and then content-bound `configuration_id`; installing another for the same source retains
launches only the fixed ordinary binary. It accepts no student executable, the first byte-for-byte. `check-configuration` validates installed bytes,
command override or privileged Sandbox operation. Tokens/URLs are not command admission, identity and hardware/kernel/node scheduling bindings.
arguments or unit text. The outbox unit uses the existing actual confirmed
publisher module; the old scaffold check entry is not treated as a delivery loop.
Python launches use a fixed module/argument list and only the generated service
environment fields; inherited debugging/proxy/provider credentials are omitted.
`python3.13 -m scripts.release.service_install DIRECTORY --receipt-sha256 SHA` Run `activate RELEASE --manifest-sha256 SHA --configuration-id BATCH` with the
is the separate root operator step for those five ordinary units. It requires explicit batch. `activate` requires stopped application units, control ports and
the selected installed configuration, the canonical configuration receipt from factory socket, and empty task/identity/capture resources. Root journals and
the generator, and matching service launch/installer bytes in that delivery. batches are retained. Interruption between pointer writes requires inspection.
All app units must be stopped. It stores immutable batches under the protected `check-active` checks code/configuration selection and the measurement snapshot.
`/etc/kf-onlinejudge/units/RELEASE.CONFIGURATION/` directory, then updates only
known root-owned system unit pointers and performs a bounded daemon reload.
Unknown regular files, foreign pointers, changed bytes and unrecorded old batches
refuse replacement. Previously installed batches remain available; interruption
may leave a mixed set of pointers, but each pre-start check rejects an unselected
binding. Repeat installation can finish the same verified batch after stopping
all app units. Loaded/inactive unit readback starts no service and makes no
claim about effective runtime limits, readiness, Sandbox gates or VM acceptance.
`python3.13 -m scripts.release.factory_service --output DIRECTORY` composes the ## PostgreSQL and RabbitMQ
separate Root factory unit only after selection and protected configuration checks.
It requires matching composition code in that delivery, uses the fixed Root ### Prepare dependencies
binary and empty business environment, and preserves the supervisor reservations
and finite lifetime. It does not inherit the ordinary service namespace/filter
flags. Its pre-start `kf-sandbox prepare-factory` performs privileged cgroup
preparation inside Sandbox: the exact node/UID pool, single supervisor process,
empty task/lease/capture resources and factory lock are required. A retained
boot/inode record recognizes only its own empty delegation; an unknown group,
same-boot loss or conflicting record refuses. New-boot recreation requires empty
persistent resources. No journal/lease is deleted and preparation grants no
execution permit. Neither composition nor lexical tests are Root/kernel/service
acceptance; install, restart, failure recovery and actual execution remain VM-only.
The separate operator step `factory_service --install DIRECTORY --receipt-sha256 SHA`
requires that generated receipt, the selected protected configuration and matching
installer bytes. It keeps immutable Root-unit batches under `/etc/kf-judge/units/`,
updates only recorded Root-owned pointers with all app units stopped, and checks
loaded/inactive systemd metadata. Foreign files/links and partial unknown batches
are refused, old batches remain retained. This step neither starts the factory nor
prepares its cgroup; only the later bounded startup invokes Sandbox preparation.
`python3.13 -m scripts.release.dependencies --connections PRIVATE_JSON --output DIRECTORY` `python3.13 -m scripts.release.dependencies --connections PRIVATE_JSON --output DIRECTORY`
composes the fixed loopback PostgreSQL/RabbitMQ configurations and ordinary units generates fixed loopback configurations and ordinary units for the selected
from the selected release. Broker passwords must be identical to the selected release. Broker passwords must match the outbox/Worker batch. Its private
outbox/Worker batch. The private definition file contains salted password hashes, definition contains salted hashes, two users without management tags, one
two users without management tags, one vhost/queue and separate publish/consume vhost/queue and separate publish/consume permissions. A blank broker must import
permissions. Definitions import is required on a blank broker; no guest account these definitions; management and guest/default accounts stay disabled.
or default password is configured. The management plugin remains disabled.
PostgreSQL local/loopback paths both use SCRAM, including the Unix import socket.
Configuration includes bounded connections, query/temp budgets and log controls.
Both dependency units reserve 512 MiB, zero swap and bounded CPU/PIDs/files, PostgreSQL uses SCRAM on local and loopback paths, including the import Unix
restart and lifetime. The receipt requires separate 2-GiB PostgreSQL and 512-MiB socket. Its configuration bounds connections, queries, temporary storage, WAL and
broker storage mounts. These mounts are not created or checked by composition. logs. Both dependency services have 512 MiB, zero swap and finite CPU/PIDs/files,
RabbitMQ memory/disk alarms and PostgreSQL WAL/temp settings are not total hard restart and lifetime limits. Memory/disk alarms supplement the hard limits.
storage/memory guarantees. Required real filesystem/cgroup/queue/auth readbacks,
account ownership, cluster initialization, cookie/definition permissions and
startup/restart are still explicit installation/VM steps; no service is started.
Dependency pre-start checks require the selected release/batch and read back the
ordinary UID, zero capabilities, no-new-privileges, process/cgroup/node budgets,
private data ownership, a separate data filesystem mount and its actual size.
A parent filesystem or an alarm setting therefore cannot silently pass as the
required dedicated hard storage capacity. Config parsing alone does not verify
these conditions or broker semantics on an already-initialized node.
Dependency directive references: [PostgreSQL resources](https://www.postgresql.org/docs/18/runtime-config-resource.html), ### Install snapshots
[RabbitMQ limits](https://www.rabbitmq.com/docs/limits),
[definition import](https://www.rabbitmq.com/docs/definitions),
and [RabbitMQ access control](https://www.rabbitmq.com/docs/access-control).
`python3.13 -m scripts.release.dependency_install DIRECTORY --receipt-sha256 SHA `python3.13 -m scripts.release.dependency_install DIRECTORY --receipt-sha256 SHA
--erlang-cookie PRIVATE_FILE --cookie-sha256 SHA` installs the selected dependency --erlang-cookie PRIVATE_FILE --cookie-sha256 SHA` installs a snapshot while all
snapshot with all app/dependency services stopped. It requires already provisioned application and dependency services are stopped.
dedicated ext4 mounts at the two fixed data paths, with their fixed capacity and
`rw,nosuid,nodev,noexec` options. It does not create, format, resize or mount disks.
A root-owned empty mount (apart from an empty private `lost+found`) may be assigned
to its dedicated account; existing data is never recursively reassigned. Ordinary
UID/GID 1203/1204 must be locked, without supplementary authority, and have the
fixed PostgreSQL/RabbitMQ service homes. Conflicting OS accounts are not adopted.
The generated files are installed under `/etc/kf-dependencies/RELEASE.CONFIGURATION/`. Prepare dedicated ext4 data mounts first, with fixed capacities and
This separate root-owned searchable tree lets each service group read only its `rw,nosuid,nodev,noexec` options:
0640 configuration files; the application configuration tree remains private.
The definition file's salted credentials, user privileges and queue budgets must | Service | Storage | Ordinary UID/GID |
match the selected outbox/Worker batch. The operator supplies an explicit private | --- | --- | --- |
256-bit Erlang cookie; repeat installation requires the same ordinary-owned 0400 | PostgreSQL | 2 GiB | 1203 |
cookie and does not implicitly rotate it. No cookie/password bytes are reported. | RabbitMQ | 512 MiB | 1204 |
System unit pointers may replace only recorded root-owned dependency snapshots.
The command retains old batches, verifies loaded/inactive metadata, starts no The installer does not create, format, resize or mount disks. It can assign an
service and initializes no database. Cluster initialization, actual effective empty Root-owned mount, allowing only an empty private `lost+found`, to the
limits, broker authorization and startup/readiness still require VM acceptance. dedicated account. Existing data is never recursively reassigned. Accounts must
be locked, have the fixed service homes and lack supplementary authority.
Snapshots live in `/etc/kf-dependencies/RELEASE.CONFIGURATION/`. Each service group
reads its own 0640 files; application configuration remains private. Definition
hashes, permissions and queue budgets must match the selected outbox/Worker batch.
Supply a private 256-bit Erlang cookie. Repeat installation requires the same
ordinary-owned 0400 cookie and leaves its value unchanged.
Only recorded Root-owned unit pointers can be replaced. Old snapshots remain;
loaded/inactive metadata is read back. Before execution, each dependency checks
its UID, cap0, no-new-privileges, process/service/node budgets, private data
ownership and the dedicated filesystem's mount identity and size.
### Initialize PostgreSQL
`python3.13 -m scripts.release.dependency_initialize --password-file PRIVATE_FILE `python3.13 -m scripts.release.dependency_initialize --password-file PRIVATE_FILE
--password-sha256 SHA` is the separate explicit PostgreSQL cluster initialization --password-sha256 SHA` initializes a fresh unrecorded cluster on the selected
step. It requires the selected dependency snapshot, stopped services, the dedicated snapshot's dedicated mount, with services stopped. Supply a hexadecimal bootstrap
storage mount and an operator-supplied hexadecimal bootstrap password file. Only password file. The ordinary PostgreSQL identity runs `initdb` in a bounded,
a fresh unrecorded target is initialized. The fixed ordinary PostgreSQL identity network-isolated transient unit. systemd supplies the private password file as a
runs `initdb` in a transient network-isolated unit with node/service/process/disk credential, outside arguments and environment values. Local and host
budgets checked before execution. systemd passes the password as a private file authentication use SCRAM.
credential; its bytes are neither unit arguments nor environment values. Both
local and host authentication use SCRAM; incomplete data is retained for explicit
recovery, not silently deleted or reused. The initializer never starts a database
server or migrates business tables; Alembic and role provisioning remain separate.
A root-owned initialization receipt binds the physical cluster identity and initial The Root-owned receipt records the physical cluster identity and credential
credential fingerprint. Repeat initialization requires that recorded identity and fingerprint. Repeat initialization requires that identity and does not reset its
does not reset a password or claim successful DB authentication. The operator password. The operator checks the unit result and confirms stop; foreign/residual
reads the transient unit result and confirms stop; foreign/residual units and units and partial/unrecorded targets are refused. Retain incomplete data for
unrecorded/partial targets are refused. These are implementation interfaces, not inspection. This command starts no DB server and migrates no tables.
actual VM acceptance. See [PostgreSQL initdb](https://www.postgresql.org/docs/18/app-initdb.html)
and [systemd credentials](https://systemd.io/CREDENTIALS/) for the underlying interfaces.
`python3.13 -m scripts.release.node_health` performs the installed service ### Provision database identities
admission-precondition checks in a fixed root operator unit. It uses bounded
ordinary broker/factory/import observer units and accepts no executable, account,
port or command override. Each helper checks its effective node/service/process
budgets, identity and no-new-privileges/capability boundary. Unknown or concurrent
units are refused; transient helper stop is read back. This is an operator CLI,
not an additional public HTTP control endpoint, and has no schema/state writer.
The projection checks selected unit pointers and effective identities/limits, `provision-database-identities` uses locked asyncpg, bounded connections/statements
actual DB-role/column privileges, the import owner's fresh policy-bound heartbeat and a 120-second deadline. It installs fixed independent logins/NOLOGIN groups and
through its Unix socket, and broker queue semantics/permissions/vhost limits via the application database, checks every supplied credential and rejects unexpected
ordinary-cookie read-only CLI queries. Coordinator owns the authenticated bounded authority. Backend Alembic remains responsible for table DDL and grants.
execution/cleanup counters; truncated or unconfirmed data never proves a drain.
Its `/internal/operations/` routes require platform authority and are not proxied
publicly. Counters contain no submission IDs, source, archives or credential bytes.
Factory health is a separate version-1 authenticated observation, rate/call-count The migration login owns the database and can inherit/SET the narrow NOLOGIN
bounded for the finite factory lifetime. It reads current supervisor/delegation function-owner role so Alembic can grant or rename transferred functions. It has
authorization, the exact runtime-config digest, live helpers, remaining requests no superuser/create-role/create-db/bypass-RLS rights or ADMIN membership option.
and lifetime, without recovery, task allocation, ledger writes or limit repair. Runtime logins cannot grant/set group authority and never receive that membership.
The old allocation request bytes are unchanged. Health explicitly reports
`isolation_verified=false` and `execution_gate=required_per_execution`: every real
execution still needs its namespace/root/identity/seccomp/exec acknowledgements.
Admission readiness requires confirmed empty execution/cleanup state and enough
factory requests/lifetime for the configured maximum job. It does not certify
TLS/browser behavior or replace the independent VM's execution acceptance.
The optional explicit `--probe-notification` sends one small confirmed notification ## Application units and Root factory
for a fresh identifier checked absent by Coordinator. It reads no message body,
creates no evaluation/grade and starts no program; ordinary periodic observation
does not publish messages. Probe failure is reported separately. Underlying
interfaces: [RabbitMQ CLI](https://www.rabbitmq.com/docs/man/rabbitmqctl.8),
[PostgreSQL privilege functions](https://www.postgresql.org/docs/18/functions-info.html).
`python3.13 -m scripts.release.lifecycle --action start|resume|drain|stop|restart` `python3.13 -m scripts.release.services generate` renders five ordinary units from
is the explicit fixed-unit maintenance entry. Root publishes a protected, versioned a verified bundle and matching configuration receipt. Users, budgets, restart/stop
availability record bound to the selected release/configuration. The installed API limits and pre-start release/batch checks are fixed. The outbox runs the confirmed
checks that record before accepting mutations; an absent, unknown, closed or wrong publisher module. Service launch code must belong to the selected delivery.
binding returns 503. Existing reads, logout, cancellation and receipt-bound archive
uploads retain their ordinary identity/lease checks. Health alone cannot open this
barrier, and service startup does not create schemas, accounts or credentials.
Drain first closes new mutations, keeps the API available to already accepted `python3.13 -m scripts.release.service_install DIRECTORY --receipt-sha256 SHA`
work, and polls the Coordinator, outbox, import, broker and factory owners within installs those units with all application services stopped. Snapshots are retained
a finite deadline. Only their untruncated empty projections permit ordinary in `/etc/kf-onlinejudge/units/RELEASE.CONFIGURATION/`. The installer updates only
service shutdown. Final owner counts, actual stopped process groups, delegated known Root-owned pointers and performs a bounded daemon reload. Foreign pointers,
tasks/identity/capture resources and absent factory socket must all agree before regular files, changed bytes and unknown partial batches refuse replacement.
quiescence is published. Timeout preserves unfinished rows, leases, queues and A stopped repeat installation can finish the same verified batch. Every pre-start
replay journals; no forced grade or cleanup proof is written. `drain` leaves DB/MQ check rejects a pointer left on an unselected batch.
running, `stop` additionally stops them, and `restart` drains before a fresh start.
Fresh start requires stopped known application units. `resume` explicitly checks `python3.13 -m scripts.release.factory_service --output DIRECTORY` renders the
known units that are already running after an interrupted operation; it does not separate Root unit after selection and protected configuration checks. The unit
adopt foreign units or repair unknown resources. Both keep admission closed until uses the fixed binary, empty business environment, supervisor reservations and
full service preconditions pass and opening is read back. Failed final readiness finite lifetime, with the factory's own namespace/filter context.
closes the barrier again. A bounded Root maintenance receipt records manager
operations and cleanup results. Root factory termination uses a signal notice and
the existing bounded guardian/recovery path; ordinary import termination finishes
its current authorized operation before claiming another. These interfaces require
actual independent-VM lifecycle acceptance and are not installation/safety proof.
`python3.13 -m scripts.release.journal` separately installs a fixed KF journal `factory_service --install DIRECTORY --receipt-sha256 SHA` installs the generated
namespace profile. It requires a preprovisioned dedicated 64-MiB ext4 mount at receipt and unit under `/etc/kf-judge/units/`, preserving earlier snapshots and
`/var/log/journal/MACHINE_ID.kf-onlinejudge`, with the same private hard-volume checking loaded/inactive systemd metadata. Application units must be stopped;
mount rules used for dependencies. It never creates/formats/mounts a disk, changes foreign files/links and unknown partial batches are refused.
the default system journal, adopts old unrecorded log data or starts a daemon.
Unknown configuration/overrides and partial installations are retained and refused.
The profile fixes 32-MiB persistent/8-MiB runtime rotation budgets, bounded files, On startup, `kf-sandbox prepare-factory` checks the exact node/UID pool, a single
lines and seven-day retention. Its namespaced daemon override limits CPU, memory, supervisor, the factory lock and empty task/lease/capture resources before creating
swap and PIDs. Rotation settings are not a hard filesystem guarantee; the dedicated cgroup delegation. A boot/inode record identifies only its own empty delegation.
volume must also be observed. The `check_ready` interface reads actual namespace Unknown groups, same-boot loss and conflicting records refuse startup; a new boot
service/process/cgroup state. Ordinary application, DB/MQ and Root-factory requires empty persistent resources. Preparation retains journals and leases.
templates now select this namespace. Their protected pre-start binding checks Each subsequent execution still needs the full Sandbox isolation check.
require its recorded configuration and physical hard volume, while retaining
ordinary private PID views and the Root factory's existing permission context.
Installing the profile does not start or replace running services. Real routing,
privacy and retention/capacity behavior remain C+D/F VM acceptance work.
The profile relies on the pinned systemd 262
[namespace interface](https://github.com/systemd/systemd/blob/v262/man/systemd.exec.xml)
and [journal configuration](https://github.com/systemd/systemd/blob/v262/man/journald.conf.xml).
The read-only status observer may see a `ProtectSystem` mount as read-only. That ## HTTPS entry
does not prove the logger can write: readiness separately reads the fixed known
daemon's kernel mountinfo, matches its mount device to the protected volume and
requires both mount and superblock writability. Main PID, invocation and process
start time are checked again to reject a changed process. No writable exception,
ptrace/setns capability or wider Python PID visibility is added to observation.
Underlying read interface: [Linux proc mountinfo implementation](https://github.com/torvalds/linux/blob/v6.18/fs/proc_namespace.c).
Production API logging uses a closed event vocabulary and omits arbitrary library `generate-proxy` reads a 0600 profile containing `schema_version: 1`, canonical DNS
messages, arguments, paths and exception text. Uvicorn's separate error handlers `origin`, `certificate_sha256` and `key_sha256`. It generates a new NGINX snapshot
are redirected through that formatter; installed API launchers disable access and receipt for the verified release. It accepts fixed upstreams, paths and limits.
logging so request query strings are not copied into service logs. Development Browser API families reach only API loopback; execution-control routes are refused.
diagnostics remain separate. These ordinary formatting checks are not evidence ZIP uploads use a separate canonical UUID route.
of actual daemon, HTTP or upload log privacy; that is still a VM acceptance case.
Build-input version 2 adds an explicitly fingerprinted public NGINX 1.30.5
runtime catalog. Version 1 remains available for older engineering bundles without
proxy support; there is no implicit proxy default. The separate `proxy_runtime`
module permits only the binary and bounded shared libraries, excludes TLS and
configuration files, verifies every byte/mode and ships them under `payload/proxy`.
This separate library search tree avoids silently replacing existing application,
database or language-image libraries. Binary version/signature/ELF-closure evidence
is recorded by the operator; accepting a structural catalog is not TLS/server or
Linux-safety acceptance. Actual proxy installation and execution remain separate.
`python3.13 -m scripts.release.proxy_install --source DIRECTORY `python3.13 -m scripts.release.proxy_install --source DIRECTORY
--receipt-sha256 SHA --certificate PRIVATE_CERTIFICATE --private-key PRIVATE_KEY` --receipt-sha256 SHA --certificate PRIVATE_CERTIFICATE --private-key PRIVATE_KEY`
installs the explicitly supplied protected proxy snapshot and matching unencrypted installs the selected release/API-origin snapshot and matching unencrypted TLS
TLS files. It requires the same release/API-origin batch, a distinct locked material at `/etc/kf-proxy/tls/fullchain.pem` and `key.pem`.
UID/GID 1205 (`kf-proxy`) without supplementary groups, and an already provisioned
private 128-MiB ext4 volume at `/var/lib/kf-onlinejudge/proxy`. Provision that
volume's ownership/mode explicitly; the installer does not format, mount or
recursively reassign unknown data. Existing TLS material must match its fingerprint;
this entry does not silently rotate keys or claim DNS/public-browser trust.
Proxy modules separate public runtime verification, fixed service rendering, Prepare a 128-MiB private ext4 volume at `/var/lib/kf-onlinejudge/proxy` and the
Root installation, ordinary launch guarding and read-only HTTPS observation. distinct locked UID/GID 1205 (`kf-proxy`), without supplementary groups. The
The ordinary process retains only `CAP_NET_BIND_SERVICE` for 443, no-new-privileges, installer checks ownership/mode and matching existing TLS fingerprints; it does
private PID visibility and actual CPU/memory/swap/PIDs/file bounds. It receives not format/mount data volumes, reassign unknown data or rotate keys implicitly.
no application DB/MQ/control credentials. Installation preserves older snapshots,
refuses foreign system-unit pointers, reads back loaded/inactive metadata and
starts no server. Maintenance includes the proxy in start/drain/stop ordering.
Readiness uses fixed loopback TLS with the supplied certificate as an explicit
local trust anchor and checks hostname/leaf identity plus the API live response;
it does not substitute for actual browser CA/HTTPS/security acceptance in the VM.
The exact `/api/v1/health/live` route permits GET/HEAD for this bounded process
liveness check. Other health, internal, execution and control routes remain
outside the public proxy allowlist; liveness does not attest judging readiness.
Private cold backups use a separate format from public release bundles. The ordinary proxy keeps only `CAP_NET_BIND_SERVICE` for 443, no-new-privileges,
The private v3 format binds the stopped broker cookie and current DB bootstrap private PID visibility and CPU/memory/swap/PIDs/file limits. It has no DB/MQ/control
fingerprints and retains credentials. Maintenance includes it in start/drain/stop order. Readiness verifies
recovery history as another private domain. A restore can reject distribution-secret loopback TLS with the supplied certificate, hostname/leaf identity and API live
reuse without dropping the inactive configuration from an earlier recovery. It response. `/api/v1/health/live` allows GET/HEAD; other health/internal/control
changes no public judging v1/v2 canonical bytes or historical report. The earlier routes remain outside the allowlist. Browser CA trust needs separate verification.
private v1/v2 prototypes were never system-accepted and are refused by this format.
`python3.13 -m kf_oj.infra.artifacts.backup` is an ordinary read-only artifact Build-input version 2 specifies the NGINX 1.30.5 catalog. Version 1 supports older
inventory command. Run it with the selected API database/storage identity after bundles without a proxy. `proxy_runtime` checks every binary/library byte and mode,
the owner-confirmed drain and while PostgreSQL is still available. It reads every excludes TLS/configuration files and ships a separate tree under `payload/proxy`.
ready artifact registry object through a repeatable-read, read-only transaction Record package signature, version and ELF-closure verification with the inputs.
and hash-checks existing source/case bytes with a 64-KiB buffer. It creates no
storage directory or lock and changes no metadata, grade, session or grant. ## Logs
Its output is private and does not itself prove that writers have stopped.
`python3.13 -m scripts.release.journal` installs the fixed KF journal namespace
profile on a preprovisioned 64-MiB ext4 volume at
`/var/log/journal/MACHINE_ID.kf-onlinejudge`. It keeps the default system journal
unchanged and rejects unknown configuration, overrides, old unrecorded data and
partial installs.
The profile uses 32-MiB persistent and 8-MiB runtime rotation budgets, bounded
files/lines and seven-day retention. Its daemon has separate CPU/memory/swap/PIDs
limits. Application, DB/MQ and Root units use this namespace and check the recorded
profile and physical volume before starting.
`check_ready` reads the daemon's process, cgroup and mount state. An observer's
read-only `ProtectSystem` view does not establish daemon writability: the check
matches the daemon's mount device to the protected volume and checks mount and
superblock writability. It rechecks PID, invocation and start time to detect a
changed process, without ptrace/setns or a wider observer PID view.
Production API logs use fixed event names and omit arbitrary library messages,
arguments, paths and exception text. Uvicorn errors use that formatter; installed
API launchers disable access logging so queries are not copied to logs.
Development diagnostics remain separate. Verify real routing, privacy and capacity
on the installed node.
## Readiness and maintenance
`python3.13 -m scripts.release.node_health` checks admission preconditions in one
fixed Root operator unit. Ordinary broker/factory/import observers have finite
budgets; each checks identity, limits and no-new-privileges/capabilities. The CLI
rejects unknown/concurrent helpers, reads back their stop and accepts no executable,
account, port or command override.
Checks include selected units and effective limits, DB-role/column privileges,
the import owner's fresh policy-bound Unix-socket heartbeat and read-only broker
queue/permissions/vhost queries. Coordinator supplies authenticated execution and
cleanup counts through `/internal/operations/`. Truncated or unconfirmed counts
cannot establish a drain. These internal routes require platform authority and
return no submission IDs, source, archives or credentials.
Factory health is a version-1 authenticated, rate/call-count-bounded observation.
It reports supervision/delegation authorization, config digest, live helpers,
remaining requests and lifetime. It performs no recovery, allocation, ledger write
or limit repair. Request bytes remain unchanged. It reports
`isolation_verified=false` and `execution_gate=required_per_execution`.
Readiness requires confirmed empty execution/cleanup state and enough requests
and lifetime for the maximum job. `--probe-notification` optionally sends one
small confirmed notification for an identifier checked absent by Coordinator. It
reads no body, creates no evaluation and runs no program. Periodic observation
does not publish; probe failures are reported separately.
`python3.13 -m scripts.release.lifecycle --action start|resume|drain|stop|restart`
controls the fixed units and a Root-protected release/batch availability record:
| Action | Behaviour |
| --- | --- |
| `start` | Requires stopped known application units; opens admission after readiness passes |
| `resume` | Validates known running units after an interrupted operation |
| `drain` | Closes new mutations, waits for owner-confirmed cleanup and leaves DB/MQ running |
| `stop` | Completes the drain, then stops DB/MQ |
| `restart` | Drains before a fresh start |
API returns 503 for an absent, unknown, closed or wrong-bound availability record.
Reads, logout, cancellation and receipt-bound archive uploads keep their existing
identity/lease checks. Both startup paths keep admission closed until the opening
is read back; failed final readiness closes it again.
Drain keeps API available to accepted work and polls Coordinator, outbox, import,
broker and factory within a finite deadline. Untruncated owner counts, stopped
process groups, empty tasks/identity/capture resources and an absent factory socket
must agree before quiescence is published. Timeout preserves rows, leases, queues
and replay journals. Root uses its signal/guardian recovery path; import finishes
its current authorized operation before claiming another. The maintenance receipt
records manager operations and cleanup results.
## Cold backups
Collect the private artifact inventory with
`python3.13 -m kf_oj.infra.artifacts.backup` after owner-confirmed drain, while
PostgreSQL is available. Use the selected API DB/storage identity. A read-only,
repeatable-read transaction checks ready registry objects and source/case digests
with a 64-KiB buffer. The command creates no directory/lock and changes no state.
Run `lifecycle --action stop` and verify processes have exited. A quiescent stop
reuses only this boot's successful selected owner-derived drain receipt and
rechecks application cgroups and Root task cleanup.
After collecting the inventory, run `lifecycle --action stop`. For an already
quiescent node it reuses only this boot's successful selected owner-derived drain
receipt, rechecks every application cgroup and the Root task cleanup, then stops
the selected database and broker. It does not query owners already stopped or
adopt a stale, failed or unproven cut. Verify all their processes have exited.
`python3.13 -m scripts.release.backup capture `python3.13 -m scripts.release.backup capture
--backup-id bk-<32-lowercase-hex> --references PRIVATE_INVENTORY --backup-id bk-<32-lowercase-hex> --references PRIVATE_INVENTORY
--references-sha256 SHA` launches one fixed Root backup operator. It requires the --references-sha256 SHA` captures a cold snapshot after checking the selected
same selected release/configuration, this boot's successful owner-derived cut, batch, owner-derived cut, quiescent admission, empty tasks, stopped service
quiescent admission, empty task resources, no surviving service identities and identities and clean PostgreSQL shutdown.
clean PostgreSQL shutdown. The operator has only `CAP_DAC_READ_SEARCH`, no-new-
privileges, 256-MiB memory, zero swap, 32 tasks, 50% CPU, a 256-MiB individual-file
limit and a 30-minute deadline. Only the preprovisioned private backup volume is
writable; this entry does not format/mount a volume or execute captured content.
Provision a separate Root-owned 0700 ext4 volume of 16 GiB at Prepare a separate Root-owned 0700, 16-GiB ext4 volume at
`/var/lib/kf-onlinejudge/backups`, with `nosuid,nodev,noexec` and no shared mount. `/var/lib/kf-onlinejudge/backups`, with `nosuid,nodev,noexec` and no shared mount.
The capture retains all source domains: the cold PostgreSQL cluster, source and The fixed backup operator has `CAP_DAC_READ_SEARCH`, no-new-privileges, 256 MiB,
testcase store, private import staging, all retained release/image versions, zero swap, 32 tasks, 50% CPU, a 256-MiB file limit and a 30-minute deadline. Only
application/dependency/proxy configuration, factory replay history and operation that volume is writable.
receipts. Every entry has a bounded digest/size and original identity/mode;
captured copies remain Root-only. Registered source/case references must all
match captured files. Source modification, links, special files, unknown
identities, insufficient space or an incomplete cut are refused. A failed target
is retained for explicit inspection and never published as a successful backup.
The bound is 7 GiB and 50,000 entries per backup, at most two backup directories, Backup domains include PostgreSQL, source/case storage, private import staging,
with a seven-day declared retention period. Capacity refusal does not silently retained releases/images, configuration, replay history and operation receipts.
delete an old/unknown backup. `backup verify --backup-id ID Every member records bounded digest/size and original identity/mode; copies remain
--manifest-sha256 SHA` checks an explicitly supplied private operator receipt and Root-only. Registry references must match. Links, special files, changing sources,
every private member; it performs no restoration or secret/epoch reuse. Actual unknown identities, insufficient space and incomplete cuts are refused. Failed
pruning uses `backup prune --backup-id EXPIRED_ID --manifest-sha256 SHA targets remain unpublished for inspection.
--keep-backup-id NEWER_ID --keep-manifest-sha256 SHA`. Both private inventories
must verify completely; the named target must be expired and the preserved copy | Limit | Value |
must be newer and unexpired. It removes only the known inventory under the | --- | --- |
maintenance lock, never the last verified copy, an unknown tree or a partial | Per snapshot | 7 GiB and 50,000 entries |
backup. Verification and pruning use the same bounded operator as capture. | Stored snapshots | At most two |
Controlled recovery into a clean node and measured recovery time require their | Declared retention | Seven days |
separate implementation and independent VM acceptance. Capture, expiry pruning
and interruption behavior also still require actual VM acceptance. `backup verify --backup-id ID
Backup data and manifests contain private credentials/source/testcase data and --manifest-sha256 SHA` checks the supplied private receipt and every member.
must stay outside Git and CI artifacts. Parsing, ordinary adapter tests or a `backup prune --backup-id EXPIRED_ID --manifest-sha256 SHA
manifest digest are not cold-backup/recovery acceptance evidence. --keep-backup-id NEWER_ID --keep-manifest-sha256 SHA` verifies both inventories,
requires an expired target and a newer unexpired retained copy, then removes only
the named known inventory under the maintenance lock. It retains the last verified
copy and refuses unknown/partial trees. Capacity refusal deletes nothing.
Private v3 backups bind the stopped broker cookie, current DB bootstrap fingerprint
and recovery history. Earlier private v1/v2 prototypes are refused. These private
versions do not alter judging canonical bytes or historical reports. Keep all
backup data and manifests out of Git and CI artifacts.
## Restore to a clean node
Install a verified public bundle on a different clean boot/node. Generate and
select a newly attested batch, then install dependencies with a new broker cookie
on empty provisioned volumes. Keep services stopped and do not initialize a new
cluster over restored data. Use fresh Worker UUID, measurement nonce/epoch,
control tokens and all DB/MQ passwords.
`python3.13 -m scripts.release.restore --recovery-id rr-<32-lowercase-hex> `python3.13 -m scripts.release.restore --recovery-id rr-<32-lowercase-hex>
--backup-id ID --manifest-sha256 SHA` performs the cold **data** step on a --backup-id ID --manifest-sha256 SHA` runs the cold data step. Compatibility
different clean boot/node. First install a verified public bundle, generate and requires identical migration bytes, target platform/toolchains and runtime images;
select an attested fresh configuration, and explicitly install the dependency schema-changing upgrades need a separate Backend Alembic procedure.
snapshot with a new broker cookie on empty preprovisioned volumes. Keep all
services stopped; do not initialize a new PostgreSQL cluster over restored data.
The target must have new Worker UUID, measurement nonce/epoch, control tokens
and all DB/MQ passwords. Cold restoration currently requires identical migration
bytes, target platform/toolchains and runtime images; schema-changing upgrades
need their separate Backend Alembic procedure.
The fixed restore operator reads the backup volume through a read-only mount The operator reads the backup through a read-only mount, checks the private receipt
view, under 512-MiB memory, zero swap, 32 tasks, 50% CPU and a 30-minute deadline. and every byte, then copies to fixed targets under 512 MiB, zero swap, 32 tasks,
It verifies the separate private receipt and every source byte before copying 50% CPU and a 30-minute deadline. Original identities/modes and replay IDs are
to fixed targets. It preserves original data identities/modes and every retained preserved. Old configuration and maintenance receipts go into inactive
factory replay ID; a new boot's existing Sandbox preparation owns delegation `recovery/ID/history/`. The new boot's Sandbox preparation owns cgroup recreation.
recreation. It copies old application/judge/dependency/proxy configuration and Unknown/nonempty targets and differing existing files are refused; partial restores
maintenance receipts into private inactive `recovery/ID/history/`, never over remain stopped for inspection.
the selected current configuration. Unknown/nonempty data targets and differing
existing files are refused. Failed partial data stays stopped for explicit
inspection; it is never silently adopted or erased.
The `data-restored.json` receipt proves only byte/identity restoration. The `data-restored.json` records only byte/identity restoration. Services stay stopped
operator starts no service, changes no schema/grant/grade/lease and reports DB and DB credential rotation and execution verification remain false.
credential rotation and execution verification as false.
### Rotate database credentials
`python3.13 -m scripts.release.recovery_database --recovery-id ID `python3.13 -m scripts.release.recovery_database --recovery-id ID
--data-receipt-sha256 SHA --password-file PRIVATE_NEW_BOOTSTRAP --data-receipt-sha256 SHA --password-file PRIVATE_NEW_BOOTSTRAP
--password-sha256 SHA` dispatches the credential step only after that boot's --password-sha256 SHA` requires this boot's selected complete data receipt and a
selected complete data receipt. Supply a new 256-bit bootstrap password; its new 256-bit bootstrap password with a different fingerprint from the backup.
fingerprint must differ from the separately verified private backup's current Backup capture requires a complete initialization or recovery marker for the
bootstrap fingerprint. Backup capture requires one complete initialization or actual cluster, allowing a restored node to be backed up again. Endpoints come
recovery marker for the actual cluster, so a restored node can be backed up again. from current protected configuration.
Current
service endpoints are loaded from the new protected configuration, not backup
URLs. Missing/different authority, an active cluster, unknown unit or incomplete
cleanup is refused; this is an explicit operation, not service startup behavior.
One ordinary `kf-postgres` temporary unit gets 512-MiB memory, zero swap, 64 One ordinary `kf-postgres` temporary unit uses 512 MiB, zero swap, 64 tasks, 100%
tasks, 100% CPU, a four-minute deadline, cap0/no-new-privileges and private PID/ CPU, a four-minute deadline, cap0/no-new-privileges and private PID/network/runtime
network/runtime namespaces. It uses a private 0700 socket directory with TCP namespaces. TCP is disabled; the private 0700 socket uses local peer authentication
listening disabled. The [PostgreSQL peer mechanism](https://www.postgresql.org/docs/18/auth-peer.html) for this OS identity only. Fixed service logins use SCRAM.
maps only this local OS identity to the fixed maintenance database login; fixed
service logins use SCRAM for their credential checks. No trust authentication,
external database listener or runtime Root business credential is introduced.
The process checks effective PostgreSQL memory/time/socket/logging settings and The process checks PostgreSQL settings, roles, memberships and database ownership.
the exact existing roles/memberships/database owner. It rotates only passwords It rotates the five service logins and `postgres` in one transaction, preserves
for the five fixed service logins and `postgres` in one transaction, verifies authority and authenticates every new credential. It changes no schema, grade,
unchanged role authority and authenticates each new credential. It grants no lease or measurement history. SQL/error utility text is suppressed. Confirm
membership, creates no database/table, invokes no business repair and does not server exit and empty resources before publishing `identity-recovered.json`.
rewrite grades, leases or measurement history. SQL/error output containing
utility statements is suppressed. Temporary-server exit and empty resources
must be confirmed before the dispatcher writes `identity-recovered.json`.
Startup and credential recovery share the cold data step's fixed lock. Startup Recovery and startup share a fixed lock. Startup holds a read-only shared lock
holds a read-only shared lock through admission and validates the linked prepared, through admission and validates linked prepared/data/identity receipts in the
data and identity receipts in each bounded Root-private recovery directory. bounded Root-private recovery directories. Missing/incomplete/foreign receipts
Missing/incomplete/foreign receipts refuse startup; a committed password change keep admission closed, including a committed credential change without confirmed
without a verified shutdown receipt cannot open admission. Complete historical shutdown. Historical receipts remain valid after reboot or version change but do
receipts remain valid after reboot or a later selected version; they do not make not establish current readiness. They stay in inactive history on later restores.
the current services ready. Previous recovery history is backed up and restored
into inactive history, never used as the new node's completion receipt.
Recovery still needs Backend Alembic verification, new TLS inputs, service Finish recovery with Backend Alembic verification, new TLS material, service
readiness and actual four-language/history checks in the second VM. Neither readiness and actual four-language/history checks on the second VM. The 30-minute
data nor credential receipts assert execution safety or healthy public service. end-to-end target still requires measured acceptance.
These system steps and the 30-minute end-to-end recovery target remain pending
independent VM acceptance.
`python3.13 -m scripts.release.node_observation` dispatches one finite, ## Observe resources and queues
read-only local observation through its fixed Root cap0/no-new-privileges helper
(128 MiB, zero swap, 16 tasks, 25% CPU, 120 seconds, single named unit).
It reads kernel memory/PIDs current and peak, CPU throttling, memory events,
aggregate device IO and filesystem bytes/inodes without enumerating private
source, hidden cases or credential files. Current selected service identities,
ACLs, execution leases/cleanup, broker/outbox/import backlog and finite factory
capacity come from the existing status owners. Notification probing stays off;
it does not publish, claim, acknowledge, close admission or repair state.
The result includes source/configuration binding, a sample time and observation `python3.13 -m scripts.release.node_observation` uses a finite read-only Root cap0,
duration. Independent readings are a bounded time window, not an atomic backup no-new-privileges helper: 128 MiB, zero swap, 16 tasks, 25% CPU, 120 seconds and one
cut. Missing kernel fields, stopped services or failed owner queries are explicit named unit. It reads memory/PIDs current/peak, CPU throttling, memory events, device
incomplete observations; there is no invented zero/healthy fallback. Busy IO and filesystem bytes/inodes without enumerating private content. Existing owners
admission can remain false in a complete observation. `isolation_verified` supply service identities, ACLs, leases/cleanup, broker/outbox/import backlog and
remains false: every real execution still needs its own Sandbox gate. Private factory capacity. Notification probing stays off.
operational output is for the local operator and must stay outside public API,
Git and CI artifacts. Actual resource/queue observations and long-run evidence Output includes source/batch, sample time and duration. Readings cover a bounded
still require the independent VM. time window, not an atomic backup cut. Missing fields, stopped services and failed
owner queries are explicit incomplete observations. Busy admission can be false
in a complete sample; `isolation_verified` remains false. Keep operational output
outside public API, Git and CI artifacts.
## References
- PostgreSQL: [role membership](https://www.postgresql.org/docs/18/role-membership.html),
[function ownership](https://www.postgresql.org/docs/18/sql-alterfunction.html),
[initdb](https://www.postgresql.org/docs/18/app-initdb.html),
[peer authentication](https://www.postgresql.org/docs/18/auth-peer.html),
[resources](https://www.postgresql.org/docs/18/runtime-config-resource.html),
[privilege functions](https://www.postgresql.org/docs/18/functions-info.html).
- RabbitMQ: [limits](https://www.rabbitmq.com/docs/limits),
[definition import](https://www.rabbitmq.com/docs/definitions),
[access control](https://www.rabbitmq.com/docs/access-control),
[CLI](https://www.rabbitmq.com/docs/man/rabbitmqctl.8).
- NGINX: [core](https://nginx.org/en/docs/http/ngx_http_core_module.html),
[proxy module](https://nginx.org/en/docs/http/ngx_http_proxy_module.html),
[TLS module](https://nginx.org/en/docs/http/ngx_http_ssl_module.html).
- systemd: [credentials](https://systemd.io/CREDENTIALS/),
[namespace interface](https://github.com/systemd/systemd/blob/v262/man/systemd.exec.xml),
[journal configuration](https://github.com/systemd/systemd/blob/v262/man/journald.conf.xml).
- Linux: [proc mountinfo](https://github.com/torvalds/linux/blob/v6.18/fs/proc_namespace.c).