Compare commits

...
Author SHA1 Message Date
yxx 5c2441464d feat(export): 交付 Markdown → HTML 导出服务
实现 Export Service 完整生命周期:mistune AST → Document AST → HtmlExporter 渲染完整 HTML5,异步任务注册表 + 取消 + 24h 产物过期。新增 5 个 /api/exports 端点与 15 项测试;pdf/docx 与函数图像静态渲染留待后续 PR。
2026-09-04 09:02:33 +08:00
Kronecker 2e496462a9 Merge pull request 'feat(provider): 完成阶段 E 多协议模型接入、国内预设与能力路由' (#15) from feat/provider-routing into main
Reviewed-on: #15
2026-09-04 07:35:50 +08:00
admin a75d81a7d9 fix(benchmark): 记录实际Embedding空间与逐样本回退信息 2026-09-04 07:25:12 +08:00
admin 5dd5a46aae fix(provider): 修复索引事务回滚与工具名分片并同步主分支 2026-09-04 07:15:52 +08:00
admin 1fe75e3fd2 feat(provider): 完成阶段E协议适配、国内预设与模型路由 2026-09-04 06:19:32 +08:00
Kronecker d31cd842c5 Merge pull request 'Feat/knowledge retrieval core' (#14) from feat/knowledge-retrieval-core into main
Reviewed-on: #14
2026-09-04 00:20:55 +08:00
yxxandClaude Code abccb328fc fix(backend): FTS 阈值过滤处理 span=0 退化场景
全部命中 bm25 相同时归一化皆为 1.0,阈值超过 1.0 应无命中,
与旧 normalize_scores 语义对齐。

Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-09-03 23:57:00 +08:00
yxx ae65c64c8f Merge remote-tracking branch 'origin/main' into feat/knowledge-retrieval-core
# Conflicts:
#	README.md
#	backend/app/routes.py
#	docs/architecture/AI笔记软件技术栈说明-团队版-v2.3.md
#	docs/development/Knowledge与Retrieval-Core开发说明.md
2026-09-03 23:45:52 +08:00
yxxandClaude Code 3bd475dc15 fix(backend): 落实 PR #12 评审意见
- P1 事件循环让出:run_rag 在样本边界 await asyncio.sleep(0),运行中取消/进度/SSE 可及时调度
- P2 SSE 终止事件:历史回放期间识别终止事件并结束流,try/finally 保证订阅清理
- P2 FTS 截断:fts 走数据库侧精确分页与计数,阈值经 bm25 截止值换算,不再受 5000 条固定截断
- P2 仅块标注:expected_block_ids 从块反查所属笔记,避免合法样本被判零分

Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-09-03 23:37:09 +08:00
Kronecker 78e8e3e33b Merge pull request 'feat(mcp): 完善独立 MCP 服务器配置中心与连接生命周期管理' (#13) from feat/mcp-server-registry into main
Reviewed-on: #13
2026-09-03 22:45:55 +08:00
admin 9b8b10cdb1 fix(mcp): 修复配置导入、凭据管理与协议边界
修复生命周期锁阻塞事件循环、旧连接回调误停新连接及超时契约不一致。

补齐 MCP JSON 兼容导入、密钥拆分与失败重试,修复 Header 大小写草稿丢失,迁移大小写敏感的环境变量凭据。

在 SSE 行拼接前限制缓冲大小,增加并发、迁移和流式输入回归测试;忽略本机 MCP 数据及 server.json/servers.json。

验证:后端 185 项、前端 54 项测试通过,前端生产构建、相关文件 Ruff 与暂存差异检查通过。
2026-09-03 22:38:35 +08:00
yxxandClaude Code 3898530585 docs: 补齐 PR #11 评审要求的文档同步
- 技术栈说明实施状态:RAG Benchmark 标记为已完成、Agent Benchmark 暂缓
- 新增 Benchmark 开发说明,并登记到文档索引
- README 回归基线更新为后端 157 / 前端 29

Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-09-03 22:35:45 +08:00
yxxandClaude Code fcc601fcf3 fix(backend): 落实 PR #11 第二轮评审意见
- Benchmark 容量淘汰只删终态 run,满容量且全活动时返回 BENCHMARK_CAPACITY_EXCEEDED
- 创建 run 前校验索引兼容性(BENCHMARK_INDEX_INCOMPATIBLE)
- 取消 run 补发 RunCancelled 终止事件;失败分支脱敏(BENCHMARK_RUN_FAILED)
- 失败样本计入汇总分母,报告输出 total/successful/failed/failure_rate
- load_dataset 按文件名隔离无关损坏文件,顶层非对象拒绝
- FTS score_threshold 先于计数/分页,total 与 items 一致
- Benchmark SSE 支持 Last-Event-ID 游标
- 移除 Agent Benchmark 501 占位接口
- 同步第二阶段接口契约与开发说明文档

Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-09-03 22:20:11 +08:00
admin 2f7066aa92 添加MCP客户端超时配置和连接管理改进
添加了MCP客户端的超时配置功能,包括启动超时和工具调用超时参数。
改进了HTTP客户端和标准IO客户端的超时处理机制,确保请求在指定时间内完成或取消。
增加了对MCP服务器数量的限制,防止配置过多服务器导致系统不稳定。
增强了错误处理机制,当连接异常时能够正确清理资源并移除桥接主机。
添加了对大型MCP消息的大小验证,防止过大的请求导致系统问题。
优化了密钥更改后的处理流程,确保在修改密钥时停用服务器并要求重新测试。
2026-09-03 16:16:58 +08:00
admin 7d5f4023a9 feat(mcp): complete remote transports and configuration workflow 2026-09-03 15:25:41 +08:00
admin 2dc984401d feat(mcp): add standalone server registry
Implement C.1 stdio MCP server CRUD, encrypted environment secrets, command digest approval, connection tests, lifecycle recovery, and dynamic tool registration. Add the standalone frontend configuration center, contracts, regression tests, and development documentation.
2026-09-03 14:49:49 +08:00
admin ed2e867db1 feat(frontend): complete MCP plugin management UI 2026-09-03 14:22:29 +08:00
yxxandClaude Code c6cde2500b fix(backend): 落实 PR #9 评审意见
- 检索调优参数(rrf_k/rerank/rerank_candidates/score_threshold)透传到引擎实际执行
- Recall 去重,避免同一 Note 多 Block 重复导致 Recall 超 1
- RAG 运行改为后台异步执行:创建即 queued + 202,支持取消与 SSE 实时事件
- 数据集元数据校验,坏文件隔离跳过;citation_required 语义修正
- modes 空/重复校验;配置快照记录模型版本与索引元信息

Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-09-02 23:20:34 +08:00
admin 1e32b2e0f4 docs: 同步第二阶段进度与扩展复盘 2026-09-02 20:22:06 +08:00
Kronecker ff3da5d6b1 Merge pull request 'feat(extension): 实现 Plugin Command 与 Settings Contribution' (#10) from feat/plugin-command-settings into main
Reviewed-on: #10
2026-09-02 20:05:42 +08:00
admin d39ae727c1 fix(extension): 收紧插件命令运行时契约 2026-09-02 20:03:39 +08:00
admin eb3464b522 fix(extension): 向MCP命令传递插件设置 2026-09-02 18:23:32 +08:00
admin c06b962743 fix(extension): 按Schema资源作用域校验引用 2026-09-02 15:47:54 +08:00
admin 022c3226c7 fix(extension): 保护插件凭据引用与删除事务 2026-09-02 15:31:39 +08:00
admin 9e680a0239 fix(extension): 完成MCP命令目标并收紧Schema边界 2026-09-02 14:52:03 +08:00
admin c3ef9dfa44 fix(extension): 收紧插件密钥与命令清单边界 2026-09-02 14:08:47 +08:00
admin 6a08ad898e feat(extension): 实现插件命令与设置贡献 2026-09-02 12:52:40 +08:00
yxxandClaude Code 0006e91e67 chore(backend): 移除误提交的验收笔记
验收笔记此前被误纳入 benchmark 提交,现摘除跟踪,文件保留在本地磁盘。

Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-09-01 23:35:11 +08:00
yxx 866febec21 feat: benchmark功能开发完成 2026-09-01 23:30:11 +08:00
yxx 9b50b8f0ce feat: 完成benchmark后端功能 2026-09-01 23:19:28 +08:00
Kronecker eb940e6590 Merge pull request 'feat(extension): 接入 stdio MCP Bridge 与 Plugin Host' (#8) from feat/mcp-plugin-host into main
Reviewed-on: #8
2026-09-01 22:10:56 +08:00
admin e37ac7b0a4 fix(extension): 强化 MCP 参数与生产运行门禁 2026-09-01 21:25:05 +08:00
admin 574b113827 fix(extension): 完善 MCP 参数与运行安全边界 2026-09-01 16:06:23 +08:00
admin 1132a4cece fix(extension): 修复 MCP Host 资源与协议边界 2026-09-01 12:11:30 +08:00
admin aedb1c1267 docs(extension): 补充阶段C MCP开发说明 2026-09-01 11:32:47 +08:00
admin fc4b7b9495 feat(extension): 接入 stdio MCP Plugin Host 2026-09-01 11:32:05 +08:00
Kronecker 83782f1d0a Merge pull request 'feat(agent): 持久化 Agent Trace 并支持 SSE 断点恢复' (#7) from feat/agent-trace-persistence into main
Reviewed-on: #7
2026-09-01 10:36:43 +08:00
admin 0e8d4b7b9f fix(agent): 保留持久化Run完整内容 2026-09-01 10:05:58 +08:00
admin a5b709a46f docs: 重组文档目录并补充CI/CD细则 2026-09-01 09:55:40 +08:00
admin 49dbacb296 docs(agent): 记录Trace问题与修复方案 2026-09-01 00:49:30 +08:00
admin 3cb197aafe feat(agent): 持久化Trace并支持SSE恢复 2026-09-01 00:39:37 +08:00
admin 8da75d4420 feat(workspace): 接入真实Vault数据链路 2026-08-31 21:38:56 +08:00
admin 84077feb18 docs: 清理接口文档行尾格式 2026-08-31 20:27:23 +08:00
admin bb8091f2e6 docs(api): 规划第二阶段统一接口契约 2026-08-31 20:26:53 +08:00
admin 0b20bad0a8 docs: 更新第二阶段技术栈基线 2026-08-31 20:08:14 +08:00
admin 9559fda5f9 docx:添加第二阶段团队分工表
添加详细的第二阶段开发计划文档,包括:

- 阶段目标和总体分工安排
- 各成员具体职责和任务分配(范涵宇、杨星萱、吉海燕)
- 技术实现方案和架构设计
- 跨模块协作关系和接口定义
- 优先级划分(P0/P1/P2)和验收标准
- 项目演示Demo规划和完成定义
2026-08-31 19:53:44 +08:00
admin fb1da4d00a merge: 补充前后端代码注释与TODO约定 2026-08-30 23:03:33 +08:00
admin 2becaf0c65 docs: 建立代码注释与TODO维护约定 2026-08-30 23:00:11 +08:00
admin 49c856a69c chore(frontend): 补充状态与服务边界注释 2026-08-30 22:59:58 +08:00
admin 629a6bda9c chore(backend): 补充核心流程注释与待办 2026-08-30 22:59:45 +08:00
Kronecker 8e4f4e6d1a Merge pull request 'feat(frontend): 统一页面视觉并完善 GitHub 代码主题' (#6) from feat/frontend-visual-polish into main
Reviewed-on: #6
2026-08-30 22:39:15 +08:00
admin df044d7888 docs(frontend): 补充 Shiki 预览与选择器回归说明 2026-08-30 22:31:51 +08:00
admin 9bcb3bf3a6 fix(frontend): 修复 Shiki 主题选择器并添加真实预览 2026-08-30 22:31:21 +08:00
admin 206a8f5b0a docs(frontend): 记录 GitHub 代码主题配置 2026-08-30 20:42:00 +08:00
admin 3595021d80 feat(frontend): 添加 GitHub 代码块主题设置 2026-08-30 20:41:42 +08:00
admin 7803ae5110 chore(frontend): 将暂定品牌名统一为 NotesAgent 2026-08-30 20:25:51 +08:00
admin 668139706b fix(frontend): 提升 Markdown 表格与列表对比度 2026-08-30 20:23:22 +08:00
admin 7fa2e9c404 docs(frontend): 记录视觉优化与动效约束 2026-08-30 20:14:22 +08:00
admin ef03a8f745 feat(frontend): 统一页面视觉与轻量动效 2026-08-30 20:13:29 +08:00
admin 2c84a98474 docs: 添加第一阶段测试验证手册 2026-08-30 15:20:12 +08:00
Kronecker f75d486e00 Merge pull request 'docs: 同步当前工程实现与验证基线' (#5) from fix/frontend-review-findings into main
Reviewed-on: #5
2026-08-30 15:16:17 +08:00
Kronecker 82cb396685 Merge branch 'main' into fix/frontend-review-findings 2026-08-30 15:16:07 +08:00
admin 3872ef3304 docs: 同步当前工程实现与验证基线 2026-08-30 15:15:00 +08:00
Kronecker b70aac1934 Merge pull request 'Fix/frontend review findings' (#4) from fix/frontend-review-findings into main
Reviewed-on: #4
2026-08-30 15:06:05 +08:00
admin bec308b5d1 feat(frontend): 完成智能体页面汉化 2026-08-30 11:04:13 +08:00
admin 0bb78e004b docs(provider): 记录本地密钥加密边界 2026-08-30 10:57:14 +08:00
admin 352975d753 feat(frontend): 支持直接配置模型API密钥 2026-08-30 10:57:05 +08:00
admin 0836807aa2 feat(provider): 添加API密钥加密存储 2026-08-30 10:56:57 +08:00
admin 0842413d29 docs(provider): 补充DeepSeek开发凭据配置 2026-08-30 10:48:35 +08:00
admin 31e29f25bb fix(frontend): 提示模型凭据缺失与鉴权失败 2026-08-30 10:48:25 +08:00
admin dc5ee76bed fix(provider): 修复DeepSeek开发凭据解析 2026-08-30 10:48:12 +08:00
admin 340bfbbd07 docs(provider): 补充模型发现与凭据边界说明 2026-08-30 10:44:13 +08:00
admin 0f6938c6a2 feat(frontend): 支持提供商预设与自动获取模型 2026-08-30 10:44:04 +08:00
admin f7d864bc4d feat(provider): 添加厂商预设与模型发现接口 2026-08-30 10:43:55 +08:00
admin 11cb384115 docs(frontend): 记录文件切换与代码块调整 2026-08-30 10:29:23 +08:00
admin f993ee9657 fix(frontend): 默认展开代码块编辑器 2026-08-30 10:29:15 +08:00
admin 1564df434d fix(frontend): 修复文件树点击切换竞态 2026-08-30 10:29:08 +08:00
admin 99a0595dbc docs(frontend): 记录编辑器交互修复与回归验证 2026-08-30 10:25:10 +08:00
admin 5a4084de1e fix(frontend): 修复Markdown工具栏选区交互 2026-08-30 10:25:00 +08:00
admin 693c86c24d fix(frontend): 修复可视化编辑器文件切换时序 2026-08-30 10:24:47 +08:00
admin 9bbe4d3c86 docs(frontend): 补充Markdown插入工具说明 2026-08-30 10:15:35 +08:00
admin d7a640a147 feat(frontend): 扩展Markdown插入工具并汉化块菜单 2026-08-30 10:15:27 +08:00
admin 9b55ccb02a docs(frontend): 记录Markdown工具栏增强 2026-08-30 10:07:40 +08:00
admin 8ee3adae6a feat(frontend): 完善Markdown选区格式工具栏 2026-08-30 10:07:31 +08:00
admin 9d223be5ca docs(frontend): 补充写作体验优化说明 2026-08-30 09:57:31 +08:00
admin c5193626a5 feat(frontend): 引入可视化Markdown写作与Shiki高亮 2026-08-30 09:57:22 +08:00
admin c33610a295 feat(frontend): 统一界面图标体系 2026-08-30 09:57:04 +08:00
Kronecker 32f6bd6468 Merge pull request 'Fix/frontend review findings' (#3) from fix/frontend-review-findings into main
Reviewed-on: #3
2026-08-30 00:16:34 +08:00
admin bba3041867 docs(frontend): 补充合并终审修复记录 2026-08-30 00:14:21 +08:00
admin 0dbd32a757 fix(frontend): 防止切换文件丢稿并修复主题恢复 2026-08-30 00:14:10 +08:00
admin a7772430d5 docs(frontend): 记录全页面审阅问题与修复 2026-08-29 23:59:22 +08:00
admin cd405da21c fix(frontend): 接通设置状态与搜索降级 2026-08-29 23:59:22 +08:00
admin d6f6b7e6e2 fix(frontend): 修复任务关联字段持久化 2026-08-29 23:59:11 +08:00
admin 4f9ebc55ec fix(frontend): 清理Agent跨Run权限状态 2026-08-29 23:59:11 +08:00
admin afd49ba00f fix(frontend): 修复文件树与编辑器状态一致性 2026-08-29 23:59:11 +08:00
admin 5aa0026bb9 fix(frontend): 完善Markdown与Chat事件渲染 2026-08-29 23:59:01 +08:00
admin e5f803c364 feat(frontend): 接通完整页面路由与桌面壳层 2026-08-29 22:49:53 +08:00
admin 5192a8b4e8 feat(frontend): 完成主题与设置页面 2026-08-29 22:49:44 +08:00
admin 3239bab696 feat(frontend): 完成Skill与Plugin管理页面 2026-08-29 22:49:44 +08:00
admin 3c9142533e feat(frontend): 完成任务管理页面 2026-08-29 22:49:33 +08:00
admin 6b362ccb6f feat(frontend): 完成Agent运行与Trace页面 2026-08-29 22:49:32 +08:00
admin 48dc6ce75b feat(frontend): 完成知识对话页面 2026-08-29 22:49:32 +08:00
admin 02e19be8bb feat(frontend): 完成知识库搜索页面 2026-08-29 22:48:58 +08:00
admin 610fc77b0f fix(frontend): 修复合并审阅发现的构建与契约问题
恢复 Vue TypeScript 生产构建,补齐可运行页面壳子,并修复文件树与 SSE 状态问题。

按 FastAPI Wire Contract 统一 Service DTO 映射,同时补充前端开发说明和问题修复复盘。
2026-08-29 12:10:33 +08:00
Kronecker c6c28e4ebe Merge pull request 'feat(frontend): 搭建桌面端基础界面与 Workspace' (#2) from feat/frontend-workspace into main
Reviewed-on: #2
2026-08-29 11:47:07 +08:00
219 changed files with 29958 additions and 2006 deletions
+9
View File
@@ -13,6 +13,15 @@ backend/**/__pycache__/
backend/.env
# 运行期生成的 SQLite 索引(vault 下的 Markdown 测试数据需提交)
backend/data/*.db*
backend/data/credentials/
# 运行期导出的 HTML/PDF/DOCX 产物(不提交)
backend/data/exports/
# 阶段验收笔记(验收用,不提交)
backend/data/vault/验收/
# 本机 MCP 配置、授权状态及服务器工作目录不得提交。
backend/data/mcp/
server.json
servers.json
# Editors and operating systems
.idea/
+43 -6
View File
@@ -2,19 +2,21 @@
> 本文件用于团队开发期间快速配置环境和启动项目,不是正式的项目 README。
> 当前基线:2026-09-03。第一阶段 Web 联调前后端已经完成;第二阶段已完成 Workspace 去 Mock、Agent Trace 持久化与 SSE 恢复、stdio MCP Bridge、隔离 Plugin Host、Plugin Command/Settings,以及独立 MCP Server 配置中心 C.1stdio、Streamable HTTP 与旧 SSE 兼容)、RAG Benchmark 与 Markdown → HTML 导出。真实音频、Provider 协议增强、Agent Benchmark、PDF/DOCX 导出、主题包、Trace 可视化、Mermaid 与函数图像仍在后续开发;Tauri Host、Stronghold、原生多 Vault 文件系统和 Sync Server 尚未接入。
## 当前目录
```text
NotesAgent/
├── frontend/ Vue 3 + TypeScript + Vite 前端
├── backend/ FastAPI + Pydantic 后端
├── docs/ 分工与技术栈说明
├── docs/ 架构、契约、开发说明、协作规范与问题复盘
└── server sync/ 云同步服务预留目录,当前未实现
```
## 开发环境
当前前后端壳子需要:
当前开发版需要:
| 环境 | 要求 | 说明 |
| --- | --- | --- |
@@ -34,7 +36,7 @@ python --version
uv --version
```
当前壳子暂不需要 Rust 和 Tauri。开始桌面端集成后,再按照 `docs/AI笔记软件技术栈说明-团队版-v2.2.md` 安装 Rust Toolchain 与 Tauri CLI。
当前 Web 联调不需要 Rust 和 Tauri。开始桌面端集成后,再按照 `docs/architecture/AI笔记软件技术栈说明-团队版-v2.3.md` 安装 Rust Toolchain 与 Tauri CLI。
## 首次初始化
@@ -74,6 +76,14 @@ uv run uvicorn app.main:app --reload --host 127.0.0.1 --port 8000
- API 文档:<http://127.0.0.1:8000/docs>
- OpenAPI JSON<http://127.0.0.1:8000/openapi.json>
#### 开发环境使用外部模型
在“设置 → 模型提供商”中选择 DeepSeek 或 OpenAI 预设后,直接在密码输入框填写 API Key。前端只在提交期间持有该值,不写入 Pinia 或 localStorageAI Core 将其加密保存到本机 `backend/data/credentials/`Provider 配置只保留内部 Credential ID。
该目录同时包含本地开发用主密钥和密文,并已加入 `.gitignore`。这提供本地静态加密和完整性校验,但不能替代操作系统凭据库。开始 Tauri 桌面集成后,应将存储实现迁移到 Stronghold,保留现有 Credential API 与 Provider 接口边界。
无界面或自动化环境仍可使用 `DEEPSEEK_API_KEY``OPENAI_API_KEY``AINOTE_CREDENTIAL_<ID>` 注入;设置页保存的本地密钥优先,环境变量仅在本地未保存对应 Credential ID 时作为回退。密钥不得写入仓库文件、README、Issue、提交信息或聊天记录。
### 终端二:启动前端
```powershell
@@ -101,8 +111,34 @@ cd frontend
pnpm build
```
前端单元与组件测试:
```powershell
cd frontend
pnpm test
```
当前回归基线为后端 467 项测试、前端 32 项测试,且 TypeScript 类型检查和生产构建通过。测试数量会随功能增长,以本地实际输出和 CI 为准。
构建产物位于 `frontend/dist`,该目录不提交到 Git。
## 文档导航
| 文档 | 用途 |
| --- | --- |
| [文档总索引](docs/README.md) | 文档分类、阅读顺序和维护规则 |
| [技术栈说明](docs/architecture/AI笔记软件技术栈说明-团队版-v2.3.md) | 目标架构、第二阶段技术边界与模块依赖 |
| [第二阶段分工表](docs/architecture/第二阶段团队分工表.md) | 第二阶段人员职责、任务顺序、协作关系与验收项 |
| [后端接口契约](docs/contracts/后端接口契约-开发版.md) | HTTP/SSE 接口、错误和当前实现状态 |
| [第二阶段接口契约](docs/contracts/第二阶段接口契约-开发版.md) | 第二阶段公共 DTO、计划接口、SSE、错误码与联调顺序 |
| [AI Core 与 Agent Core](docs/development/AI-Core与Agent-Core开发说明.md) | Provider、Agent、Tool、Permission 与 Extension Core |
| [MCP Bridge 与 Plugin Host](docs/development/MCP-Bridge与Plugin-Host开发说明.md) | stdio MCP、隔离进程、Tool 映射、状态与错误边界 |
| [Plugin Command 与 Settings](docs/development/Plugin-Command与Settings开发说明.md) | Command Registry、Settings Schema、Secret 引用与联调边界 |
| [Plugin Command 与 Settings 复盘](docs/retrospectives/Plugin-Command与Settings问题与修复复盘.md) | 阶段 D 连续审阅发现的安全、事务、Schema 与运行时契约问题 |
| [Git 使用细则](docs/guides/Git使用细则-团队开发版.md) | 分支、提交、PR、Review 与合并流程 |
| [CI/CD 细则](docs/guides/CI-CD细则-团队开发版.md) | Gitea 流水线、质量门禁、产物、发布与回滚规则 |
| [Agent Trace 复盘](docs/retrospectives/Agent-Core第二阶段问题与修复复盘.md) | Agent 持久化、SSE 恢复、事件契约与脱敏问题复盘 |
## 日常开发注意事项
- Python 依赖统一修改 `backend/pyproject.toml`,修改后执行 `uv sync`
@@ -111,6 +147,7 @@ pnpm build
- API 默认监听 `127.0.0.1:8000`,前端默认监听 `127.0.0.1:5173`
- 后端附件目录默认是 `backend/data/attachments`,可通过 `APP_ATTACHMENTS_PATH` 覆盖;该目录由桌面 Host 管理。
- 跨模块接口发生变化时,需要同步更新前后端类型和 `docs` 中的接口说明。
- 当前前后端接口清单`docs/后端接口契约-开发版.md`OpenAPI `/openapi.json` 为准。
- 前端页面、交互、状态管理和第一阶段验收要求见 `docs/前端页面需求说明-开发版.md`
- 分支、提交、Pull Request、Review 和冲突处理规范见 `docs/Git使用细则-团队开发版.md`
- 当前已实现接口`docs/contracts/后端接口契约-开发版.md`第二阶段规划接口见 `docs/contracts/第二阶段接口契约-开发版.md`;已实现能力`/openapi.json` 为准。
- 前端页面、交互、状态管理及当前阶段后续页面需求见 `docs/contracts/前端页面需求说明-开发版.md`
- 分支、提交、Pull Request、Review 和冲突处理规范见 `docs/guides/Git使用细则-团队开发版.md`
- CI 检查、产物、发布和回滚规范见 `docs/guides/CI-CD细则-团队开发版.md`
+15 -4
View File
@@ -1,7 +1,9 @@
# Backend
# Notes Agent Backend
FastAPI + Pydantic 的本地 AI Core / Agent Core。项目使用 uv 管理依赖和虚拟环境。
当前实现包含 Knowledge/Retrieval、Chat、Agent Runtime、Tool/Permission、Skill/Plugin、stdio MCP Host、Plugin Command/Settings、Provider Adapter、任务、索引和开发阶段凭据加密存储。Provider 支持 Mock、OpenAI Chat/OpenAI-Compatible 与 OllamaOpenAI Responses、Anthropic Messages、操作系统级 Plugin 沙箱和真实语音模型仍属于后续阶段。
```powershell
uv sync
uv run uvicorn app.main:app --reload --host 127.0.0.1 --port 8000
@@ -13,9 +15,18 @@ uv run uvicorn app.main:app --reload --host 127.0.0.1 --port 8000
- 健康检查:<http://127.0.0.1:8000/health>
- API 文档:<http://127.0.0.1:8000/docs>
- OpenAPI<http://127.0.0.1:8000/openapi.json>
团队接口清单见 `../docs/后端接口契约-开发版.md`,机器可读契约以运行时的 `/openapi.json` 为准。
运行回归测试:
AI Core 与 Agent Core 的模块边界、Mock Provider 和 Tool Calling 调试方式见 `../docs/AI-Core与Agent-Core开发说明.md`
```powershell
uv run pytest
```
Knowledge Core 与 Retrieval Core 的模块边界、数据模型、接口与检索流程见 `../docs/Knowledge与Retrieval-Core开发说明.md`
当前基线为 136 项测试通过。Provider API Key 可通过前端设置页写入,也可用 `OPENAI_API_KEY``DEEPSEEK_API_KEY``AINOTE_CREDENTIAL_<ID>` 注入;不要把真实密钥写入仓库。`plugin.*` 是 Plugin Settings 的保留凭据命名空间,通用 Provider 凭据接口不能读写
团队接口清单见 `../docs/contracts/后端接口契约-开发版.md`,机器可读契约以运行时的 `/openapi.json` 为准。
AI Core 与 Agent Core 的模块边界、Mock Provider 和 Tool Calling 调试方式见 `../docs/development/AI-Core与Agent-Core开发说明.md`
Knowledge Core 与 Retrieval Core 的模块边界、数据模型、接口与检索流程见 `../docs/development/Knowledge与Retrieval-Core开发说明.md`
+4 -3
View File
@@ -160,10 +160,11 @@ def read_attachment(arguments: AttachmentReadArguments, _: ToolExecutionContext)
return attachment_service.read_attachment(**arguments.model_dump())
def transcribe_audio(arguments: AudioTranscribeArguments, _: ToolExecutionContext) -> dict:
return transcription_service.create_transcription(
async def transcribe_audio(arguments: AudioTranscribeArguments, _: ToolExecutionContext) -> dict:
job = await transcription_service.create_transcription(
arguments.attachment_id, arguments.language
).model_dump(mode="json")
)
return job.model_dump(mode="json")
def _register(
+11
View File
@@ -1,3 +1,5 @@
"""Agent 工具权限策略与一次性确认票据。"""
import asyncio
from dataclasses import dataclass
from enum import Enum
@@ -51,6 +53,7 @@ class PermissionPolicy:
def mode_for(self, permission: str | None) -> PermissionMode:
if permission is None:
return PermissionMode.allow
# 未登记权限一律拒绝,防止扩展通过拼写错误或新权限绕过策略。
return self._rules.get(permission, PermissionMode.deny)
@@ -63,6 +66,8 @@ class PermissionTicket:
class PermissionManager:
"""管理当前进程内的确认请求与会话级授权。"""
def __init__(self, policy: PermissionPolicy) -> None:
self.policy = policy
self._pending: dict[tuple[str, str], PermissionTicket] = {}
@@ -94,10 +99,16 @@ class PermissionManager:
if ticket is None or ticket.future.done():
return False
if decision == "allow_session":
# 会话授权只存在于进程内,应用重启后按默认策略重新确认。
self._session_grants.add(ticket.permission)
ticket.future.set_result(decision)
return True
def get_ticket(self, run_id: str, request_id: str) -> PermissionTicket | None:
"""只读返回待确认票据,供 Trace 记录权限类型;不暴露 Future 给接口层。"""
return self._pending.get((run_id, request_id))
def cancel_run(self, run_id: str) -> None:
for key, ticket in list(self._pending.items()):
if ticket.run_id == run_id:
+195 -33
View File
@@ -1,3 +1,5 @@
"""Agent 运行时:负责模型轮次、工具调用、权限确认与事件发布。"""
from __future__ import annotations
import asyncio
@@ -5,17 +7,20 @@ import json
from collections.abc import AsyncIterator
from dataclasses import dataclass, field
from datetime import datetime, timezone
from time import perf_counter
from typing import TYPE_CHECKING
from uuid import uuid4
from app.agent.permissions import PermissionManager, PermissionMode
from app.agent.tools import ToolExecutionContext, ToolNotFoundError, ToolRegistry
from app.agent.trace_repository import AgentTraceRepository, sanitize_trace_value
from app.contracts import (
AgentEvent,
AgentEventType,
AgentRun,
AgentRunCreateRequest,
AgentRunStatus,
AgentTraceResponse,
Citation,
Message,
MessageRole,
@@ -50,6 +55,8 @@ MAX_TOOL_CALLS_PER_TURN = 50
@dataclass(slots=True)
class RunRecord:
"""单次运行的可变上下文,仅由 AgentRuntime 持有。"""
run: AgentRun
request: AgentRunCreateRequest
skill_config: AgentConfiguration | None = None
@@ -57,20 +64,25 @@ class RunRecord:
events: list[AgentEvent] = field(default_factory=list)
subscribers: set[asyncio.Queue[AgentEvent]] = field(default_factory=set)
task: asyncio.Task[None] | None = None
next_sequence: int = 0
class AgentRuntime:
"""进程内 Agent 编排器;对外返回深拷贝,避免调用方修改运行状态。"""
def __init__(
self,
providers: ProviderRegistry,
tools: ToolRegistry,
permissions: PermissionManager,
skills: SkillRuntime | None = None,
trace_repository: AgentTraceRepository | None = None,
) -> None:
self.providers = providers
self.tools = tools
self.permissions = permissions
self.skills = skills
self.trace_repository = trace_repository or AgentTraceRepository()
self._records: dict[str, RunRecord] = {}
async def create_run(self, request: AgentRunCreateRequest) -> AgentRun:
@@ -98,6 +110,7 @@ class AgentRuntime:
)
allowed_tools = list(request.allowed_tools)
if skill_config is not None:
# 同时指定 Skill 与工具白名单时取交集,避免 Skill 扩大调用权限。
allowed_tools = (
[name for name in skill_config.allowed_tools if name in allowed_tools]
if allowed_tools
@@ -109,22 +122,38 @@ class AgentRuntime:
skill_config=skill_config,
allowed_tools=allowed_tools,
)
self.trace_repository.create_run(
run,
request,
self._config_snapshot(record),
)
self._records[run.run_id] = record
record.task = asyncio.create_task(self._execute(record), name=run.run_id)
return run.model_copy(deep=True)
def get_run(self, run_id: str) -> AgentRun:
return self._get_record(run_id).run.model_copy(deep=True)
record = self._records.get(run_id)
if record is not None:
return record.run.model_copy(deep=True)
run = self.trace_repository.recover_interrupted(run_id)
if run is None:
raise AgentRunNotFoundError(run_id)
return run.model_copy(deep=True)
def list_runs(self, limit: int, offset: int) -> tuple[list[AgentRun], int]:
records = sorted(
self._records.values(), key=lambda item: item.run.created_at, reverse=True
)
items = [item.run.model_copy(deep=True) for item in records[offset : offset + limit]]
return items, len(records)
items, total = self.trace_repository.list_runs(limit=limit, offset=offset)
recovered = [
self.trace_repository.recover_interrupted(item.run_id) or item
if item.run_id not in self._records
else self._records[item.run_id].run.model_copy(deep=True)
for item in items
]
return recovered, total
async def cancel(self, run_id: str) -> AgentRun:
record = self._get_record(run_id)
record = self._records.get(run_id)
if record is None:
return self.get_run(run_id)
if record.run.status in TERMINAL_STATUSES:
return record.run.model_copy(deep=True)
record.run.cancelled = True
@@ -137,21 +166,53 @@ class AgentRuntime:
return record.run.model_copy(deep=True)
def resolve_permission(self, run_id: str, request_id: str, decision: str) -> bool:
self._get_record(run_id)
return self.permissions.resolve(run_id, request_id, decision)
record = self._records.get(run_id)
if record is None:
return False
ticket = self.permissions.get_ticket(run_id, request_id)
resolved = self.permissions.resolve(run_id, request_id, decision)
if resolved:
self._publish(
record,
AgentEventType.permission_resolved,
{
"request_id": request_id,
"permission": ticket.permission if ticket else None,
"decision": decision,
},
)
return resolved
async def events(self, run_id: str) -> AsyncIterator[AgentEvent]:
record = self._get_record(run_id)
async def events(
self, run_id: str, *, after_sequence: int = -1
) -> AsyncIterator[AgentEvent]:
record = self._records.get(run_id)
run = self.get_run(run_id)
if record is None:
for event in self.trace_repository.list_events(
run_id, after_sequence=after_sequence
):
yield event
return
# 先注册订阅再读持久化历史;同一事件循环内没有 await,不会丢失交界事件。
queue: asyncio.Queue[AgentEvent] = asyncio.Queue()
record.subscribers.add(queue)
history = [event.model_copy(deep=True) for event in record.events]
history = self.trace_repository.list_events(
run_id, after_sequence=after_sequence
)
last_sequence = after_sequence
try:
for event in history:
last_sequence = event.sequence
yield event
if record.run.status in TERMINAL_STATUSES:
if run.status in TERMINAL_STATUSES:
return
while True:
event = await queue.get()
if event.sequence <= last_sequence:
continue
last_sequence = event.sequence
yield event.model_copy(deep=True)
if event.event in {
AgentEventType.run_completed,
@@ -163,7 +224,9 @@ class AgentRuntime:
record.subscribers.discard(queue)
async def wait(self, run_id: str) -> AgentRun:
record = self._get_record(run_id)
record = self._records.get(run_id)
if record is None:
return self.get_run(run_id)
if record.task:
try:
await asyncio.shield(record.task)
@@ -171,6 +234,17 @@ class AgentRuntime:
pass
return record.run.model_copy(deep=True)
def get_trace(
self, run_id: str, *, after_sequence: int, limit: int
) -> AgentTraceResponse:
self.get_run(run_id)
trace = self.trace_repository.get_trace(
run_id, after_sequence=after_sequence, limit=limit
)
if trace is None:
raise AgentRunNotFoundError(run_id)
return trace
async def _execute(self, record: RunRecord) -> None:
try:
async with asyncio.timeout(record.request.run_timeout_seconds):
@@ -201,15 +275,51 @@ class AgentRuntime:
for step in range(1, record.request.max_steps + 1):
record.run.current_step = step
record.run.updated_at = datetime.now(timezone.utc)
turn = await provider.complete(
ModelRequest(
provider_id=record.request.provider_id,
model=record.request.model,
system=(record.skill_config.system_prompt if record.skill_config else None),
messages=messages,
tools=allowed_tools,
metadata=self._request_metadata(record),
model_call_id = f"model_call_{uuid4().hex}"
started_at = perf_counter()
self._publish(
record,
AgentEventType.model_call_started,
{
"model_call_id": model_call_id,
"step": step,
"provider_id": record.request.provider_id,
"model": record.request.model,
},
)
try:
turn = await provider.complete(
ModelRequest(
provider_id=record.request.provider_id,
model=record.request.model,
system=(record.skill_config.system_prompt if record.skill_config else None),
messages=messages,
tools=allowed_tools,
metadata=self._request_metadata(record),
)
)
except Exception as exc:
self._publish(
record,
AgentEventType.model_call_failed,
{
"model_call_id": model_call_id,
"duration_ms": int((perf_counter() - started_at) * 1000),
"error_code": getattr(exc, "code", type(exc).__name__),
},
)
raise
self._publish(
record,
AgentEventType.model_call_completed,
{
"model_call_id": model_call_id,
"duration_ms": int((perf_counter() - started_at) * 1000),
"finish_reason": "tool_calls" if turn.tool_calls else "stop",
"input_tokens": turn.input_tokens,
"output_tokens": turn.output_tokens,
"tool_call_count": len(turn.tool_calls),
},
)
record.run.token_usage += turn.input_tokens + turn.output_tokens
self._publish(
@@ -243,11 +353,12 @@ class AgentRuntime:
messages.append(
Message(role=MessageRole.assistant, content=turn.text or "", tool_calls=calls)
)
# 工具可以并发执行,但结果按模型原始调用顺序写回上下文,保证轮次可复现。
semaphore = asyncio.Semaphore(record.request.max_concurrent_tools)
async def execute(call: ToolCall) -> ToolResult:
async with semaphore:
return await self._execute_tool(record, call)
return await self._execute_tool(record, call, model_call_id)
results = await asyncio.gather(*(execute(call) for call in calls))
for call, result in zip(calls, results):
@@ -280,8 +391,13 @@ class AgentRuntime:
self._fail(record, "MAX_STEPS_EXCEEDED", "Agent reached its maximum step count.")
async def _execute_tool(self, record: RunRecord, call: ToolCall) -> ToolResult:
self._publish(record, AgentEventType.tool_call, call.model_dump(mode="json"))
async def _execute_tool(
self, record: RunRecord, call: ToolCall, parent_model_call_id: str
) -> ToolResult:
started_at = perf_counter()
call_data = call.model_dump(mode="json")
call_data["parent_model_call_id"] = parent_model_call_id
self._publish(record, AgentEventType.tool_call, call_data)
try:
registered = self.tools.get(call.name)
except ToolNotFoundError:
@@ -295,7 +411,9 @@ class AgentRuntime:
error_code="TOOL_NOT_ALLOWED",
error_message="Tool is not included in allowed_tools.",
)
self._publish(record, AgentEventType.tool_result, result.model_dump(mode="json"))
self._publish_tool_result(
record, result, parent_model_call_id, started_at
)
return result
permission = registered.definition.permission if registered else None
@@ -307,12 +425,15 @@ class AgentRuntime:
error_code="NETWORK_NOT_ALLOWED",
error_message="Agent run does not allow network tools.",
)
self._publish(record, AgentEventType.tool_result, result.model_dump(mode="json"))
self._publish_tool_result(
record, result, parent_model_call_id, started_at
)
return result
mode = self.permissions.mode_for(permission)
if mode == PermissionMode.deny:
result = self._permission_denied(call)
elif mode == PermissionMode.confirm and permission:
# 运行状态必须在等待期间可见,前端才能展示并处理权限确认卡片。
ticket = self.permissions.create_ticket(record.run.run_id, permission)
record.run.status = AgentRunStatus.waiting_permission
self._publish(
@@ -337,11 +458,13 @@ class AgentRuntime:
error_code="PERMISSION_TIMEOUT",
error_message="Tool permission confirmation timed out.",
)
self._publish(
record, AgentEventType.tool_result, result.model_dump(mode="json")
self._publish_tool_result(
record, result, parent_model_call_id, started_at
)
return result
record.run.status = AgentRunStatus.running
record.run.updated_at = datetime.now(timezone.utc)
self.trace_repository.save_run(record.run)
result = (
await self._invoke_tool(record, call)
if decision in {"allow_once", "allow_session"}
@@ -350,13 +473,31 @@ class AgentRuntime:
else:
result = await self._invoke_tool(record, call)
self._publish(record, AgentEventType.tool_result, result.model_dump(mode="json"))
self._publish_tool_result(record, result, parent_model_call_id, started_at)
return result
def _publish_tool_result(
self,
record: RunRecord,
result: ToolResult,
parent_model_call_id: str,
started_at: float,
) -> None:
data = result.model_dump(mode="json")
data["parent_model_call_id"] = parent_model_call_id
data["duration_ms"] = int((perf_counter() - started_at) * 1000)
self._publish(record, AgentEventType.tool_result, data)
async def _invoke_tool(self, record: RunRecord, call: ToolCall) -> ToolResult:
try:
return await asyncio.wait_for(
self.tools.execute(call, ToolExecutionContext(run_id=record.run.run_id)),
self.tools.execute(
call,
ToolExecutionContext(
run_id=record.run.run_id,
tool_call_id=call.tool_call_id,
),
),
timeout=record.request.tool_timeout_seconds,
)
except TimeoutError:
@@ -400,14 +541,19 @@ class AgentRuntime:
def _publish(
self, record: RunRecord, event_type: AgentEventType, data: dict[str, object]
) -> None:
sanitized = sanitize_trace_value(data)
assert isinstance(sanitized, dict)
event = AgentEvent(
event=event_type,
run_id=record.run.run_id,
sequence=len(record.events),
data=data,
sequence=record.next_sequence,
data=sanitized,
timestamp=datetime.now(timezone.utc),
)
record.next_sequence += 1
record.events.append(event)
self.trace_repository.append_event(record.run, event)
# 内存只保留实时订阅窗口;完整审计轨迹由 SQLite 保存。
if len(record.events) > MAX_EVENTS_PER_RUN:
del record.events[: len(record.events) - MAX_EVENTS_PER_RUN]
for queue in record.subscribers:
@@ -421,6 +567,21 @@ class AgentRuntime:
metadata["retrieval"] = record.skill_config.retrieval.model_dump(mode="json")
return metadata
def _config_snapshot(self, record: RunRecord) -> dict[str, object]:
provider = self.providers.get(record.request.provider_id).config
return {
"provider_id": record.request.provider_id,
"provider_type": provider.provider_type.value,
"model": record.request.model,
"capabilities": [item.value for item in provider.capabilities],
"skill_id": record.request.skill_id,
"allowed_tools": list(record.allowed_tools),
"max_steps": record.request.max_steps,
"token_budget": record.request.token_budget,
"allow_network": record.request.allow_network,
"metadata": record.request.metadata,
}
def _collect_citations(self, record: RunRecord, result: ToolResult) -> None:
if not result.success or not isinstance(result.output, dict):
return
@@ -448,6 +609,7 @@ class AgentRuntime:
raise AgentRunNotFoundError(run_id) from exc
def _prune_records(self) -> None:
# 只清理终态记录,绝不为了容量取消仍在执行或等待授权的任务。
overflow = len(self._records) - MAX_RUN_RECORDS + 1
if overflow <= 0:
return
+53 -19
View File
@@ -1,4 +1,7 @@
"""Agent 工具注册与执行边界。"""
import inspect
import threading
from dataclasses import dataclass
from time import perf_counter
from typing import Any, Awaitable, Callable
@@ -8,6 +11,7 @@ from jsonschema import Draft202012Validator
from jsonschema.exceptions import ValidationError as JsonSchemaValidationError
from app.contracts import ToolCall, ToolDefinition, ToolResult
from app.schema_security import reject_external_schema_references
ToolExecutor = Callable[[BaseModel, "ToolExecutionContext"], Any | Awaitable[Any]]
@@ -15,6 +19,7 @@ ToolExecutor = Callable[[BaseModel, "ToolExecutionContext"], Any | Awaitable[Any
@dataclass(frozen=True, slots=True)
class ToolExecutionContext:
run_id: str
tool_call_id: str | None = None
@dataclass(slots=True)
@@ -28,9 +33,21 @@ class ToolNotFoundError(LookupError):
pass
class ToolExecutionError(RuntimeError):
"""Executor 可预期失败,保留领域错误码而不是折叠成通用异常。"""
def __init__(self, code: str, message: str) -> None:
super().__init__(message)
self.code = code
self.message = message
class ToolRegistry:
"""统一校验工具入参并隔离执行异常,避免单个工具击穿 Agent 主循环。"""
def __init__(self) -> None:
self._tools: dict[str, RegisteredTool] = {}
self._lock = threading.RLock()
def register(
self,
@@ -38,33 +55,40 @@ class ToolRegistry:
arguments_model: type[BaseModel],
executor: ToolExecutor,
) -> None:
if definition.name in self._tools:
raise ValueError(f"Tool already registered: {definition.name}")
self._tools[definition.name] = RegisteredTool(
definition=definition,
arguments_model=arguments_model,
executor=executor,
)
Draft202012Validator.check_schema(definition.parameters)
reject_external_schema_references(definition.parameters)
with self._lock:
if definition.name in self._tools:
raise ValueError(f"Tool already registered: {definition.name}")
self._tools[definition.name] = RegisteredTool(
definition=definition,
arguments_model=arguments_model,
executor=executor,
)
def unregister(self, name: str) -> None:
self._tools.pop(name, None)
with self._lock:
self._tools.pop(name, None)
def contains(self, name: str) -> bool:
return name in self._tools
with self._lock:
return name in self._tools
def get(self, name: str) -> RegisteredTool:
try:
return self._tools[name]
except KeyError as exc:
raise ToolNotFoundError(name) from exc
with self._lock:
try:
return self._tools[name]
except KeyError as exc:
raise ToolNotFoundError(name) from exc
def definitions(self, allowed: list[str] | None = None) -> list[ToolDefinition]:
names = set(allowed) if allowed is not None else None
return [
item.definition.model_copy(deep=True)
for name, item in self._tools.items()
if names is None or name in names
]
with self._lock:
return [
item.definition.model_copy(deep=True)
for name, item in self._tools.items()
if names is None or name in names
]
async def execute(self, call: ToolCall, context: ToolExecutionContext) -> ToolResult:
started = perf_counter()
@@ -80,6 +104,7 @@ class ToolRegistry:
)
try:
# JSON Schema 约束模型可见的协议,Pydantic 再完成运行时类型转换。
Draft202012Validator(registered.definition.parameters).validate(call.arguments)
arguments = registered.arguments_model.model_validate(call.arguments)
except (ValidationError, JsonSchemaValidationError) as exc:
@@ -103,7 +128,16 @@ class ToolRegistry:
output=output,
duration_ms=round((perf_counter() - started) * 1000),
)
except Exception as exc: # Tool failures are isolated from the Agent loop.
except ToolExecutionError as exc:
return ToolResult(
tool_call_id=call.tool_call_id,
name=call.name,
success=False,
error_code=exc.code,
error_message=exc.message,
duration_ms=round((perf_counter() - started) * 1000),
)
except Exception as exc: # 工具失败转换成结构化结果,由模型决定是否降级或重试。
return ToolResult(
tool_call_id=call.tool_call_id,
name=call.name,
+372
View File
@@ -0,0 +1,372 @@
"""Agent Run/Event 持久化与 Trace 查询。
SQLite 中的事件是 SSE、前端 Trace 和 Benchmark 的共同事实来源。写入前统一脱敏和
限长,避免 Secret 或无限大的 Tool Result 进入审计数据。
"""
from __future__ import annotations
import json
import re
from datetime import datetime, timezone
from typing import Any
from app.contracts import (
AgentEvent,
AgentEventType,
AgentRun,
AgentRunCreateRequest,
AgentRunStatus,
AgentTraceResponse,
AgentTraceSummary,
)
from app.database.db import connect, transaction
MAX_TRACE_STRING = 4_096
MAX_TRACE_COLLECTION = 100
MAX_TRACE_DEPTH = 8
_SECRET_KEYS = {
"api_key",
"apikey",
"authorization",
"access_token",
"refresh_token",
"client_secret",
"password",
"secret",
"token",
}
_SECRET_KEY_SUFFIXES = ("_api_key", "_password", "_secret")
_TERMINAL_VALUES = {
AgentRunStatus.completed.value,
AgentRunStatus.failed.value,
AgentRunStatus.cancelled.value,
}
_BEARER_PATTERN = re.compile(r"(?i)\bBearer\s+[^\s,;]+")
_API_KEY_PATTERN = re.compile(r"\bsk-[A-Za-z0-9_-]{8,}\b")
def sanitize_trace_value(
value: Any, *, depth: int = 0, apply_limits: bool = True
) -> Any:
"""递归净化持久化数据;可按审计用途限制体积,Secret 始终脱敏。"""
if apply_limits and depth >= MAX_TRACE_DEPTH:
return "[MAX_DEPTH]"
if isinstance(value, dict):
sanitized: dict[str, Any] = {}
for index, (key, item) in enumerate(value.items()):
if apply_limits and index >= MAX_TRACE_COLLECTION:
sanitized["__truncated__"] = True
break
normalized = str(key).casefold().replace("-", "_")
sanitized[str(key)] = (
"[REDACTED]"
if normalized in _SECRET_KEYS
or normalized.endswith(_SECRET_KEY_SUFFIXES)
else sanitize_trace_value(
item, depth=depth + 1, apply_limits=apply_limits
)
)
return sanitized
if isinstance(value, (list, tuple)):
source_items = value[:MAX_TRACE_COLLECTION] if apply_limits else value
items = [
sanitize_trace_value(
item, depth=depth + 1, apply_limits=apply_limits
)
for item in source_items
]
if apply_limits and len(value) > MAX_TRACE_COLLECTION:
items.append("[TRUNCATED]")
return items
if isinstance(value, str):
value = _BEARER_PATTERN.sub("Bearer [REDACTED]", value)
value = _API_KEY_PATTERN.sub("[REDACTED]", value)
if apply_limits and len(value) > MAX_TRACE_STRING:
return f"{value[:MAX_TRACE_STRING]}...[TRUNCATED]"
return value
if value is None or isinstance(value, (str, int, float, bool)):
return value
return sanitize_trace_value(
str(value), depth=depth + 1, apply_limits=apply_limits
)
class AgentTraceRepository:
def create_run(
self,
run: AgentRun,
request: AgentRunCreateRequest,
config_snapshot: dict[str, Any],
) -> None:
conn = connect()
try:
with transaction(conn):
conn.execute(
"""
INSERT INTO agent_runs(
run_id, status, run_json, request_json, config_snapshot_json,
created_at, updated_at
) VALUES (?, ?, ?, ?, ?, ?, ?)
""",
(
run.run_id,
run.status.value,
self._serialize_run(run),
json.dumps(
sanitize_trace_value(request.model_dump(mode="json")),
ensure_ascii=False,
),
json.dumps(
sanitize_trace_value(config_snapshot), ensure_ascii=False
),
run.created_at.isoformat(),
run.updated_at.isoformat(),
),
)
finally:
conn.close()
def save_run(self, run: AgentRun) -> None:
conn = connect()
try:
with transaction(conn):
self._update_run(conn, run)
finally:
conn.close()
def append_event(self, run: AgentRun, event: AgentEvent) -> None:
"""在同一事务中保存最新 Run 和事件;复写同一序号时保持幂等。"""
conn = connect()
try:
with transaction(conn):
self._update_run(conn, run)
conn.execute(
"""
INSERT INTO agent_events(run_id, sequence, event, data_json, timestamp)
VALUES (?, ?, ?, ?, ?)
ON CONFLICT(run_id, sequence) DO NOTHING
""",
(
event.run_id,
event.sequence,
event.event.value,
json.dumps(event.data, ensure_ascii=False),
event.timestamp.isoformat(),
),
)
finally:
conn.close()
def get_run(self, run_id: str) -> AgentRun | None:
conn = connect()
try:
row = conn.execute(
"SELECT run_json FROM agent_runs WHERE run_id = ?", (run_id,)
).fetchone()
return AgentRun.model_validate_json(row["run_json"]) if row else None
finally:
conn.close()
def list_runs(self, limit: int, offset: int) -> tuple[list[AgentRun], int]:
conn = connect()
try:
total = int(conn.execute("SELECT COUNT(*) FROM agent_runs").fetchone()[0])
rows = conn.execute(
"""
SELECT run_json FROM agent_runs
ORDER BY created_at DESC LIMIT ? OFFSET ?
""",
(limit, offset),
).fetchall()
return [AgentRun.model_validate_json(row["run_json"]) for row in rows], total
finally:
conn.close()
def list_events(
self, run_id: str, *, after_sequence: int = -1, limit: int | None = None
) -> list[AgentEvent]:
conn = connect()
try:
sql = """
SELECT event, sequence, data_json, timestamp
FROM agent_events
WHERE run_id = ? AND sequence > ?
ORDER BY sequence
"""
params: tuple[Any, ...] = (run_id, after_sequence)
if limit is not None:
sql += " LIMIT ?"
params += (limit,)
return [self._event_from_row(run_id, row) for row in conn.execute(sql, params)]
finally:
conn.close()
def get_trace(
self, run_id: str, *, after_sequence: int, limit: int
) -> AgentTraceResponse | None:
conn = connect()
try:
row = conn.execute(
"""
SELECT run_json, config_snapshot_json
FROM agent_runs WHERE run_id = ?
""",
(run_id,),
).fetchone()
if row is None:
return None
run = AgentRun.model_validate_json(row["run_json"])
event_rows = conn.execute(
"""
SELECT event, sequence, data_json, timestamp
FROM agent_events
WHERE run_id = ? AND sequence > ?
ORDER BY sequence LIMIT ?
""",
(run_id, after_sequence, limit + 1),
).fetchall()
has_more = len(event_rows) > limit
items = [
self._event_from_row(run_id, item) for item in event_rows[:limit]
]
counts = {
item["event"]: int(item["count"])
for item in conn.execute(
"""
SELECT event, COUNT(*) AS count
FROM agent_events WHERE run_id = ? GROUP BY event
""",
(run_id,),
)
}
tool_errors = int(
conn.execute(
"""
SELECT COUNT(*) FROM agent_events
WHERE run_id = ? AND event = 'ToolResult'
AND json_extract(data_json, '$.success') = 0
""",
(run_id,),
).fetchone()[0]
)
errors = (
counts.get(AgentEventType.run_failed.value, 0)
+ counts.get(AgentEventType.model_call_failed.value, 0)
+ tool_errors
)
duration_ms = max(
0, int((run.updated_at - run.created_at).total_seconds() * 1000)
)
return AgentTraceResponse(
run_id=run_id,
status=run.status,
items=items,
next_sequence=items[-1].sequence if items else after_sequence,
has_more=has_more,
summary=AgentTraceSummary(
model_calls=counts.get(AgentEventType.model_call_started.value, 0),
tool_calls=counts.get(AgentEventType.tool_call.value, 0),
duration_ms=duration_ms,
token_usage=run.token_usage,
errors=errors,
),
config_snapshot=json.loads(row["config_snapshot_json"]),
)
finally:
conn.close()
def recover_interrupted(self, run_id: str) -> AgentRun | None:
"""把上个进程遗留的非终态 Run 收束为失败,并追加可回放终止事件。"""
conn = connect()
try:
with transaction(conn):
row = conn.execute(
"SELECT run_json, status FROM agent_runs WHERE run_id = ?", (run_id,)
).fetchone()
if row is None:
return None
run = AgentRun.model_validate_json(row["run_json"])
if row["status"] in _TERMINAL_VALUES:
return run
run.status = AgentRunStatus.failed
run.error_code = "AGENT_PROCESS_RESTARTED"
run.error_message = "Agent process restarted before the run completed."
run.updated_at = datetime.now(timezone.utc)
next_sequence = int(
conn.execute(
"""
SELECT COALESCE(MAX(sequence), -1) + 1
FROM agent_events WHERE run_id = ?
""",
(run_id,),
).fetchone()[0]
)
event = AgentEvent(
event=AgentEventType.run_failed,
run_id=run_id,
sequence=next_sequence,
data={
"code": run.error_code,
"message": run.error_message,
},
timestamp=run.updated_at,
)
self._update_run(conn, run)
conn.execute(
"""
INSERT INTO agent_events(run_id, sequence, event, data_json, timestamp)
VALUES (?, ?, ?, ?, ?)
""",
(
run_id,
next_sequence,
event.event.value,
json.dumps(event.data, ensure_ascii=False),
event.timestamp.isoformat(),
),
)
return run
finally:
conn.close()
@staticmethod
def _update_run(conn, run: AgentRun) -> None:
cursor = conn.execute(
"""
UPDATE agent_runs
SET status = ?, run_json = ?, updated_at = ?
WHERE run_id = ?
""",
(
run.status.value,
AgentTraceRepository._serialize_run(run),
run.updated_at.isoformat(),
run.run_id,
),
)
if cursor.rowcount != 1:
raise LookupError(run.run_id)
@staticmethod
def _event_from_row(run_id: str, row) -> AgentEvent:
return AgentEvent(
event=AgentEventType(row["event"]),
run_id=run_id,
sequence=int(row["sequence"]),
data=json.loads(row["data_json"]),
timestamp=datetime.fromisoformat(row["timestamp"]),
)
@staticmethod
def _serialize_run(run: AgentRun) -> str:
# Run 是重启后 GET/list 的完整事实;只做 Secret 脱敏,不套用 Trace 摘要限长。
return json.dumps(
sanitize_trace_value(
run.model_dump(mode="json"), apply_limits=False
),
ensure_ascii=False,
)
+8
View File
@@ -0,0 +1,8 @@
"""Benchmark 服务:RAG / Agent 数据集注册、指标计算与运行管理。
模块划分:
- metrics.py 纯函数指标(Hit@K / Recall@K / MRR / CitationHit / 分位数)
- datasets.py 受控目录的 Dataset 注册与校验
- rag.py RAG Benchmark Runner(调用 retrieval.engine.search
- service.py 运行注册表、配置快照与报告组装
"""
+198
View File
@@ -0,0 +1,198 @@
"""Benchmark Dataset 注册:从受控目录加载 JSON 数据集并校验。
Dataset 只能来自配置目录(settings.benchmark_datasets_path),API 不接受调用方提交
任意文件路径。目录不存在或为空时按「无数据集」处理,不报错。
"""
from __future__ import annotations
import hashlib
import json
from dataclasses import dataclass, field
from pathlib import Path
from pydantic import BaseModel, Field, ValidationError
from app.config import get_settings
from app.contracts import (
BenchmarkDatasetInfo,
BenchmarkKind,
RAGDatasetCase,
)
from app.errors import ApiError
@dataclass
class RAGDataset:
"""内存中的 RAG 数据集:元信息 + 已校验的 Case 列表 + 内容哈希。"""
dataset_id: str
kind: BenchmarkKind
version: str
description: str
cases: list[RAGDatasetCase] = field(default_factory=list)
content_hash: str = ""
class _DatasetMeta(BaseModel):
"""Dataset 元数据的最小校验模型。
list_datasets 用它逐文件校验元信息字段结构,把「合法 JSON 但字段类型错误」
(如 cases: 42)这类损坏文件隔离掉,而不是让 len() 抛 TypeError 拖垮整个列表。
"""
dataset_id: str = Field(min_length=1)
kind: str = ""
version: str = ""
description: str = ""
cases: list = Field(default_factory=list)
def _datasets_dir() -> Path:
return get_settings().benchmark_datasets_path
def _dataset_files() -> list[Path]:
directory = _datasets_dir()
if not directory.is_dir():
return []
return sorted(directory.glob("*.json"))
def _content_hash(raw: bytes) -> str:
return "sha256:" + hashlib.sha256(raw).hexdigest()
def _read_json(path: Path) -> tuple[dict, bytes]:
"""读取并解析 JSON 文件,返回 (dict, 原始字节);非法 JSON 抛 BENCHMARK_DATASET_INVALID。"""
try:
raw_bytes = path.read_bytes()
return json.loads(raw_bytes.decode("utf-8")), raw_bytes
except (json.JSONDecodeError, OSError, UnicodeDecodeError) as exc:
raise ApiError(
422,
"BENCHMARK_DATASET_INVALID",
f"Dataset file is not valid JSON: {path.name}",
{"path": str(path)},
) from exc
def _dataset_from_raw(raw: dict, raw_bytes: bytes, kind: BenchmarkKind) -> RAGDataset:
"""把单个数据集 JSON 解析为 RAGDataset,非法结构抛 BENCHMARK_DATASET_INVALID。"""
dataset_id = raw.get("dataset_id")
if not isinstance(dataset_id, str) or not dataset_id:
raise ApiError(
422,
"BENCHMARK_DATASET_INVALID",
"Dataset must declare a non-empty string 'dataset_id'.",
{},
)
file_kind = raw.get("kind", kind.value)
if file_kind != kind.value:
raise ApiError(
422,
"BENCHMARK_DATASET_INVALID",
f"Dataset kind mismatch: expected '{kind.value}', got '{file_kind}'.",
{"dataset_id": dataset_id},
)
raw_cases = raw.get("cases")
if not isinstance(raw_cases, list) or not raw_cases:
raise ApiError(
422,
"BENCHMARK_DATASET_INVALID",
"Dataset 'cases' must be a non-empty list.",
{"dataset_id": dataset_id},
)
cases: list[RAGDatasetCase] = []
for index, case in enumerate(raw_cases):
try:
parsed = RAGDatasetCase.model_validate(case)
except ValidationError as exc:
raise ApiError(
422,
"BENCHMARK_DATASET_INVALID",
f"Dataset case #{index} is invalid.",
{"dataset_id": dataset_id, "case_index": index, "errors": exc.errors()},
) from exc
# 每个 Case 至少要声明一个期望 id,否则无法计算命中/召回
if not parsed.expected_note_ids and not parsed.expected_block_ids:
raise ApiError(
422,
"BENCHMARK_DATASET_INVALID",
f"Dataset case '{parsed.case_id}' must declare expected_note_ids or expected_block_ids.",
{"dataset_id": dataset_id, "case_id": parsed.case_id},
)
# citation_required=true 时必须声明 expected_block_ids,否则无法计算 Citation Hit Rate
if parsed.citation_required and not parsed.expected_block_ids:
raise ApiError(
422,
"BENCHMARK_DATASET_INVALID",
f"Dataset case '{parsed.case_id}' requires expected_block_ids when citation_required is true.",
{"dataset_id": dataset_id, "case_id": parsed.case_id},
)
cases.append(parsed)
return RAGDataset(
dataset_id=dataset_id,
kind=kind,
version=str(raw.get("version", "")),
description=str(raw.get("description", "")),
cases=cases,
content_hash=_content_hash(raw_bytes),
)
def list_datasets(kind: BenchmarkKind) -> list[BenchmarkDatasetInfo]:
"""枚举受控目录下指定 kind 的数据集元信息(不含 Case 内容)。
逐文件用 _DatasetMeta 校验元信息字段结构,单个损坏文件隔离跳过而非整体失败,
保证列表接口健壮;损坏细节由 load_dataset 抛出。
"""
infos: list[BenchmarkDatasetInfo] = []
for path in _dataset_files():
try:
raw, raw_bytes = _read_json(path)
meta = _DatasetMeta.model_validate(raw)
except (ApiError, ValidationError):
continue
if meta.kind not in ("", kind.value):
continue
infos.append(
BenchmarkDatasetInfo(
dataset_id=meta.dataset_id,
kind=kind,
version=meta.version,
description=meta.description,
case_count=len(meta.cases),
content_hash=_content_hash(raw_bytes),
)
)
return infos
def load_dataset(dataset_id: str, kind: BenchmarkKind) -> RAGDataset:
"""按文件名加载并校验数据集;找不到抛 BENCHMARK_DATASET_NOT_FOUND。
只读取与请求 dataset_id 同名的文件({dataset_id}.json),无关文件的损坏(JSON 语法
错误、UTF-8 解码错误、顶层非对象)不会阻断目标数据集加载;只有目标文件本身损坏
才抛 BENCHMARK_DATASET_INVALID。按现有文件 stem 精确匹配,不拼接调用方传入的路径。
"""
for path in _dataset_files():
if path.stem != dataset_id:
continue
raw, raw_bytes = _read_json(path)
if not isinstance(raw, dict):
raise ApiError(
422,
"BENCHMARK_DATASET_INVALID",
"Dataset top-level must be a JSON object.",
{"dataset_id": dataset_id, "path": path.name},
)
return _dataset_from_raw(raw, raw_bytes, kind)
raise ApiError(
404,
"BENCHMARK_DATASET_NOT_FOUND",
f"Benchmark dataset does not exist: {dataset_id}",
{"dataset_id": dataset_id, "kind": kind.value},
)
+58
View File
@@ -0,0 +1,58 @@
"""Benchmark 指标纯函数。
所有指标只依赖「按相关性降序的 retrieved id 列表」和「期望 id 集合」,不接触任何
外部状态,便于单元测试与未来 Agent Benchmark 复用。retrieved 顺序越靠前越相关。
"""
from __future__ import annotations
def hit_at_k(retrieved: list[str], expected: set[str], k: int) -> bool:
"""前 k 个结果里是否命中任意期望 id(用于 Hit@1 / Hit@5)。"""
return any(item in expected for item in retrieved[:k])
def recall_at_k(retrieved: list[str], expected: set[str], k: int) -> float:
"""前 k 个结果召回的期望 id 占比;期望为空时视为 0。
结果先去重:检索结果是 Block 级,同一 Note 可能经多个 Block 重复出现,
直接逐项计数会把同一 Note 算多次、导致 Recall 超过 1。
"""
if not expected:
return 0.0
return len(set(retrieved[:k]) & expected) / len(expected)
def reciprocal_rank(retrieved: list[str], expected: set[str]) -> float:
"""首个命中的倒数排名;未命中返回 0。rank 从 1 开始。"""
for rank, item in enumerate(retrieved, start=1):
if item in expected:
return 1.0 / rank
return 0.0
def citation_hit(retrieved_block_ids: list[str], expected: set[str]) -> bool:
"""首条结果的 block_id 是否为期望引用块(Citation Hit Rate 的逐 Case 判据)。"""
if not retrieved_block_ids or not expected:
return False
return retrieved_block_ids[0] in expected
def mean(values: list[float]) -> float:
return sum(values) / len(values) if values else 0.0
def percentile(values: list[float], p: float) -> float:
"""线性插值分位数(p ∈ [0, 100]),用于 P50 / P95 延迟。空列表返回 0。"""
if not values:
return 0.0
ordered = sorted(values)
if len(ordered) == 1:
return ordered[0]
rank = (len(ordered) - 1) * (p / 100.0)
lo = int(rank)
hi = lo + 1
if hi >= len(ordered):
return ordered[-1]
frac = rank - lo
return ordered[lo] + (ordered[hi] - ordered[lo]) * frac
+163
View File
@@ -0,0 +1,163 @@
"""RAG Benchmark Runner:调用检索引擎对数据集逐 Case 求值并聚合指标。
只读操作,直接复用 app.retrieval.engine 的 search(),不旁路检索链路。指标按
(mode, case, repeat) 逐样本计算,再按 mode 聚合;失败样本按零分计入质量指标分母,
避免把执行失败误判为检索质量(同时保留 total/successful/failed/failure_rate)。
"""
from __future__ import annotations
import asyncio
import logging
import time
from collections.abc import Callable
from app import repository
from app.benchmarks import metrics as m
from app.benchmarks.datasets import RAGDataset
from app.contracts import (
RAGCaseResult,
RAGDatasetCase,
RAGMetrics,
RAGRunRequest,
SearchMode,
SearchRequest,
)
from app.retrieval.engine import engine
from app.retrieval.provenance import capture_embedding
logger = logging.getLogger(__name__)
class BenchmarkCancelled(Exception):
"""运行在 Case 之间被取消时抛出,用于中断后台执行并标记 cancelled。"""
async def run_rag(
dataset: RAGDataset,
request: RAGRunRequest,
on_case: Callable[[RAGCaseResult, int, int], None] | None = None,
should_cancel: Callable[[], bool] | None = None,
) -> tuple[dict[str, RAGMetrics], list[RAGCaseResult]]:
"""执行 RAG Benchmark,返回 (按 mode 聚合的指标, 全部逐样本结果)。
on_case 在每个样本求值完成后回调 (result, done, total),供上层更新进度与事件。
should_cancel 在每个样本开始前被检查;返回 True 时抛出 BenchmarkCancelled 中断运行。
"""
total = len(request.modes) * len(dataset.cases) * request.repeat
done = 0
results: list[RAGCaseResult] = []
for mode in request.modes:
for case in dataset.cases:
expected_notes = _expected_notes(case)
for repeat in range(request.repeat):
# 让出事件循环:使运行中取消、SSE 进度与并发 API 请求能及时得到调度
await asyncio.sleep(0)
if should_cancel is not None and should_cancel():
raise BenchmarkCancelled()
result = await _evaluate_one(case, mode, request, repeat, expected_notes)
results.append(result)
done += 1
if on_case is not None:
on_case(result, done, total)
metrics_by_mode = {mode.value: _aggregate(results, mode) for mode in request.modes}
return metrics_by_mode, results
def _expected_notes(case: RAGDatasetCase) -> set[str]:
"""返回笔记级期望 id;仅标注块 ID 时从块反查所属笔记,避免把标注缺失误判为检索失败。"""
if case.expected_note_ids:
return set(case.expected_note_ids)
return {hit.note_id for hit in repository.get_block_hits(case.expected_block_ids)}
async def _evaluate_one(
case: RAGDatasetCase,
mode: SearchMode,
request: RAGRunRequest,
repeat: int,
expected_notes: set[str],
) -> RAGCaseResult:
search_request = SearchRequest(
query=case.query,
mode=mode,
limit=request.retrieval.top_k,
include_snippet=False,
rrf_k=request.retrieval.rrf_k,
rerank=request.retrieval.rerank,
rerank_candidates=request.retrieval.rerank_candidates,
score_threshold=request.retrieval.score_threshold,
)
start = time.perf_counter()
embedding = {}
try:
with capture_embedding() as embedding:
response = await engine.search(search_request)
latency_ms = (time.perf_counter() - start) * 1000.0
except Exception as exc: # 单个样本失败不中断整个 Benchmark
# 详细异常只进日志,公开响应只带项目错误码与安全消息,避免泄露路径/SQL 等敏感信息
logger.warning(
"RAG case evaluation failed: case=%s mode=%s", case.case_id, mode.value,
exc_info=exc,
)
return RAGCaseResult(
embedding=embedding,
case_id=case.case_id,
mode=mode,
repeat=repeat,
latency_ms=(time.perf_counter() - start) * 1000.0,
citation_applicable=case.citation_required,
error="RAG case evaluation failed.",
error_code="BENCHMARK_CASE_EVALUATION_FAILED",
)
retrieved_note_ids = [item.note_id for item in response.items]
retrieved_block_ids = [item.block_id for item in response.items]
expected_blocks = set(case.expected_block_ids)
k = request.retrieval.top_k
return RAGCaseResult(
embedding=embedding,
case_id=case.case_id,
mode=mode,
repeat=repeat,
latency_ms=latency_ms,
retrieved_note_ids=retrieved_note_ids,
retrieved_block_ids=retrieved_block_ids,
hit_at_1=m.hit_at_k(retrieved_note_ids, expected_notes, 1),
hit_at_5=m.hit_at_k(retrieved_note_ids, expected_notes, 5),
recall=m.recall_at_k(retrieved_note_ids, expected_notes, k),
reciprocal_rank=m.reciprocal_rank(retrieved_note_ids, expected_notes),
citation_hit=m.citation_hit(retrieved_block_ids, expected_blocks),
citation_applicable=case.citation_required,
)
def _aggregate(cases: list[RAGCaseResult], mode: SearchMode) -> RAGMetrics:
samples = [c for c in cases if c.mode == mode]
total = len(samples)
failed = sum(1 for c in samples if c.error is not None)
successful = total - failed
if total == 0:
return RAGMetrics()
# 延迟只统计成功样本;失败样本按零分计入质量指标分母,避免汇总虚高
latencies = [c.latency_ms for c in samples if c.error is None]
citation_samples = [c for c in samples if c.citation_applicable]
return RAGMetrics(
hit_at_1=m.mean([1.0 if (c.error is None and c.hit_at_1) else 0.0 for c in samples]),
hit_at_5=m.mean([1.0 if (c.error is None and c.hit_at_5) else 0.0 for c in samples]),
recall_at_k=m.mean([c.recall if c.error is None else 0.0 for c in samples]),
mrr=m.mean([c.reciprocal_rank if c.error is None else 0.0 for c in samples]),
citation_hit_rate=m.mean(
[1.0 if (c.error is None and c.citation_hit) else 0.0 for c in citation_samples]
),
p50_latency_ms=m.percentile(latencies, 50.0),
p95_latency_ms=m.percentile(latencies, 95.0),
total_cases=total,
successful_cases=successful,
failed_cases=failed,
failure_rate=failed / total,
)
+349
View File
@@ -0,0 +1,349 @@
"""Benchmark 服务:运行注册表、配置快照与报告组装。
RAG Benchmark 采用「创建即返回 queued、后台 Task 异步执行」的模式(与 index_service
的 rebuild 一致):POST 创建后立即返回 202 queued 的 BenchmarkRun,由受管 asyncio.Task
在后台逐 Case 求值,进度与事件实时写入内存注册表,供 SSE 订阅。运行记录、事件与报告
暂存内存(_runs/_events/_reports),不持久化到 SQLite;后续接入异步任务队列时再落库。
"""
from __future__ import annotations
import asyncio
import logging
import sys
from datetime import datetime, timezone
from uuid import uuid4
from app import repository
from app.benchmarks import datasets
from app.benchmarks.datasets import RAGDataset
from app.benchmarks.rag import BenchmarkCancelled, run_rag
from app.config import get_settings
from app.contracts import (
BenchmarkEvent,
BenchmarkEventType,
BenchmarkKind,
BenchmarkReport,
BenchmarkRun,
BenchmarkStatus,
RAGCaseResult,
RAGMetrics,
RAGRunRequest,
SearchMode,
)
from app.errors import ApiError
from app.retrieval.engine import engine
logger = logging.getLogger(__name__)
_runs: dict[str, BenchmarkRun] = {}
_events: dict[str, list[BenchmarkEvent]] = {}
_reports: dict[str, BenchmarkReport] = {}
_tasks: dict[str, asyncio.Task] = {}
_subscribers: dict[str, list[asyncio.Queue[BenchmarkEvent]]] = {}
_cancel_flags: dict[str, asyncio.Event] = {}
MAX_RUNS = 100
def _now() -> datetime:
return datetime.now(timezone.utc)
def _forget(run_id: str) -> None:
"""移除一条 run 的全部内存态;仅在 run 处于终态时调用,避免打断活动任务。"""
_runs.pop(run_id, None)
_events.pop(run_id, None)
_reports.pop(run_id, None)
_tasks.pop(run_id, None)
_subscribers.pop(run_id, None)
_cancel_flags.pop(run_id, None)
def _evict_terminal() -> bool:
"""超过容量时淘汰最旧的终态 run;全部为活动 run 无法淘汰时返回 False。
绝不能删除仍在运行(queued/running)的 run:那会连带移除其 _cancel_flags 与
_subscribers,使后台 Task 访问时抛出 KeyError。
"""
terminal = (BenchmarkStatus.completed, BenchmarkStatus.failed, BenchmarkStatus.cancelled)
while len(_runs) >= MAX_RUNS:
victim = next(
(rid for rid, run in _runs.items() if run.status in terminal), None
)
if victim is None:
return False
_forget(victim)
return True
def _config_snapshot(request: RAGRunRequest, dataset: RAGDataset) -> dict:
"""记录运行时的模型 / 索引 / 环境信息,保证报告可解释、可复现。"""
settings = get_settings()
return {
"dataset_id": dataset.dataset_id,
"dataset_hash": dataset.content_hash,
"dataset_version": dataset.version,
"modes": [m.value for m in request.modes],
"retrieval": request.retrieval.model_dump(),
"repeat": request.repeat,
"embedding": {"policy": "per_case", "details": "cases[].embedding"},
"local_embedding": {
"model_id": engine.embedding.model_id,
"version": engine.embedding.version,
"dim": engine.embedding.dim,
},
"reranker": {
"model_id": engine.reranker.model_id,
"version": engine.reranker.version,
},
"index_meta": repository.get_index_meta(),
"app": {"version": settings.version, "environment": settings.environment},
"python": sys.version.split()[0],
"metadata": request.metadata,
}
async def _validate_index_compatibility(request: RAGRunRequest) -> None:
"""创建 RAG Run 前校验索引已建立且与当前 Embedding 模型/维度兼容。
空索引或不兼容索引会让所有模式得到全 0 指标,把环境/索引错误误判为检索质量差,
故在创建时即拒绝,返回 BENCHMARK_INDEX_INCOMPATIBLE。
"""
stats = repository.stats()
meta = repository.get_index_meta()
needs_vector = any(m in (SearchMode.vector, SearchMode.hybrid) for m in request.modes)
reasons: list[str] = []
if stats["blocks"] == 0:
reasons.append("index is empty (no indexed blocks; run /api/index/rebuild first)")
if needs_vector:
if meta.get("embedding_model") != engine.embedding.model_id:
reasons.append(
f"embedding model mismatch: index={meta.get('embedding_model')!r}, "
f"engine={engine.embedding.model_id!r}"
)
if meta.get("embedding_dim") != str(engine.embedding.dim):
reasons.append(
f"embedding dimension mismatch: index={meta.get('embedding_dim')!r}, "
f"engine={engine.embedding.dim}"
)
if await engine.vector_store.count() == 0:
reasons.append("vector index is empty")
if reasons:
raise ApiError(
409,
"BENCHMARK_INDEX_INCOMPATIBLE",
"Benchmark index is not built or is incompatible with the current retrieval engine.",
{"reasons": reasons},
)
async def create_rag_run(request: RAGRunRequest) -> BenchmarkRun:
"""创建一次 RAG Benchmark,立即返回 queued 的 BenchmarkRun,由后台 Task 执行。"""
dataset = datasets.load_dataset(request.dataset_id, BenchmarkKind.rag)
await _validate_index_compatibility(request)
# 容量检查:先淘汰终态 run 腾空间;满容量且全为活动 run 时拒绝创建
if not _evict_terminal():
raise ApiError(
429,
"BENCHMARK_CAPACITY_EXCEEDED",
"Benchmark run capacity exceeded; wait for active runs to finish.",
{},
)
run_id = "benchmark_" + uuid4().hex[:12]
snapshot = _config_snapshot(request, dataset)
run = BenchmarkRun(
run_id=run_id,
kind=BenchmarkKind.rag,
dataset_id=dataset.dataset_id,
dataset_hash=dataset.content_hash,
status=BenchmarkStatus.queued,
progress=0.0,
config_snapshot=snapshot,
created_at=_now(),
)
_runs[run_id] = run
_events[run_id] = []
_subscribers[run_id] = []
_cancel_flags[run_id] = asyncio.Event()
_tasks[run_id] = asyncio.create_task(_execute_rag(run_id, request, dataset, snapshot))
return run
async def _execute_rag(
run_id: str, request: RAGRunRequest, dataset: RAGDataset, snapshot: dict
) -> None:
"""后台执行 RAG Benchmark,实时更新进度/事件,结束后写入报告并关闭订阅。"""
cancel_event = _cancel_flags[run_id]
def emit(event_type: BenchmarkEventType, data: dict) -> None:
sequence = len(_events[run_id])
event = BenchmarkEvent(
event=event_type, run_id=run_id, sequence=sequence, data=data, timestamp=_now()
)
_events[run_id].append(event)
for queue in _subscribers.get(run_id, []):
queue.put_nowait(event)
def finish() -> None:
_subscribers.pop(run_id, None)
_cancel_flags.pop(run_id, None)
_runs[run_id] = _runs[run_id].model_copy(
update={"status": BenchmarkStatus.running, "started_at": _now()}
)
emit(
BenchmarkEventType.run_started,
{"dataset_id": dataset.dataset_id, "modes": [m.value for m in request.modes]},
)
total = len(request.modes) * len(dataset.cases) * request.repeat
def on_case(result: RAGCaseResult, done: int, _total: int) -> None:
progress = done / total if total else 1.0
_runs[run_id] = _runs[run_id].model_copy(update={"progress": progress})
emit(BenchmarkEventType.case_completed, result.model_dump(mode="json"))
try:
metrics_by_mode, results = await run_rag(
dataset,
request,
on_case=on_case,
should_cancel=cancel_event.is_set,
)
except BenchmarkCancelled:
_runs[run_id] = _runs[run_id].model_copy(
update={
"status": BenchmarkStatus.cancelled,
"progress": 1.0,
"completed_at": _now(),
}
)
emit(BenchmarkEventType.run_cancelled, {"status": BenchmarkStatus.cancelled.value})
_reports[run_id] = BenchmarkReport(
run_id=run_id,
kind=BenchmarkKind.rag,
dataset_id=dataset.dataset_id,
dataset_hash=dataset.content_hash,
status=BenchmarkStatus.cancelled,
config_snapshot=snapshot,
)
finish()
return
except Exception as exc: # 单次运行失败不拖垮服务,记录错误后结束
# 详细异常只进日志,公开响应仅带项目错误码与安全消息,避免泄露路径/SQL 等敏感信息
logger.exception("Benchmark run failed: run_id=%s", run_id)
_runs[run_id] = _runs[run_id].model_copy(
update={
"status": BenchmarkStatus.failed,
"progress": 1.0,
"error": "Benchmark run failed.",
"error_code": "BENCHMARK_RUN_FAILED",
"completed_at": _now(),
}
)
emit(
BenchmarkEventType.run_failed,
{"error": "Benchmark run failed.", "error_code": "BENCHMARK_RUN_FAILED"},
)
_reports[run_id] = BenchmarkReport(
run_id=run_id,
kind=BenchmarkKind.rag,
dataset_id=dataset.dataset_id,
dataset_hash=dataset.content_hash,
status=BenchmarkStatus.failed,
config_snapshot=snapshot,
error="Benchmark run failed.",
error_code="BENCHMARK_RUN_FAILED",
)
finish()
return
metrics = {mode: m.model_dump() for mode, m in metrics_by_mode.items()}
_runs[run_id] = _runs[run_id].model_copy(
update={
"status": BenchmarkStatus.completed,
"progress": 1.0,
"metrics": metrics,
"completed_at": _now(),
}
)
emit(BenchmarkEventType.run_completed, {"metrics": metrics})
_reports[run_id] = BenchmarkReport(
run_id=run_id,
kind=BenchmarkKind.rag,
dataset_id=dataset.dataset_id,
dataset_hash=dataset.content_hash,
status=BenchmarkStatus.completed,
config_snapshot=snapshot,
metrics=metrics,
cases=results,
)
finish()
def list_runs(
kind: BenchmarkKind | None = None,
status: BenchmarkStatus | None = None,
limit: int = 50,
offset: int = 0,
) -> tuple[list[BenchmarkRun], int]:
runs = list(_runs.values())
if kind is not None:
runs = [r for r in runs if r.kind == kind]
if status is not None:
runs = [r for r in runs if r.status == status]
runs.sort(key=lambda r: r.created_at, reverse=True)
total = len(runs)
return runs[offset : offset + limit], total
def get_run(run_id: str) -> BenchmarkRun | None:
return _runs.get(run_id)
def get_report(run_id: str) -> BenchmarkReport | None:
return _reports.get(run_id)
def get_events(run_id: str) -> list[BenchmarkEvent]:
return _events.get(run_id, [])
def cancel_run(run_id: str) -> BenchmarkRun | None:
"""取消运行:对 queued/running 设置取消标志,后台 Task 在 Case 边界检查后置为 cancelled。"""
run = _runs.get(run_id)
if run is None:
return None
if run.status in (BenchmarkStatus.queued, BenchmarkStatus.running):
_cancel_flags[run_id].set()
return run
def subscribe(run_id: str) -> asyncio.Queue[BenchmarkEvent] | None:
"""订阅运行事件流;运行已结束(completed/failed/cancelled)时返回 None。"""
run = _runs.get(run_id)
if run is None or run.status in (
BenchmarkStatus.completed,
BenchmarkStatus.failed,
BenchmarkStatus.cancelled,
):
return None
queue: asyncio.Queue[BenchmarkEvent] = asyncio.Queue()
_subscribers.setdefault(run_id, []).append(queue)
return queue
def unsubscribe(run_id: str, queue: asyncio.Queue[BenchmarkEvent]) -> None:
subscribers = _subscribers.get(run_id)
if subscribers and queue in subscribers:
subscribers.remove(queue)
async def wait_for_run(run_id: str) -> BenchmarkRun:
"""等待后台任务结束(测试/轮询用);无任务时直接返回当前状态。"""
task = _tasks.get(run_id)
if task is not None:
await task
return _runs.get(run_id)
+6
View File
@@ -24,6 +24,8 @@ class Settings:
db_path: Path
vault_path: Path
attachments_path: Path
benchmark_datasets_path: Path
exports_path: Path
@lru_cache
@@ -41,4 +43,8 @@ def get_settings() -> Settings:
attachments_path=Path(
os.getenv("APP_ATTACHMENTS_PATH", str(data_dir / "attachments"))
),
benchmark_datasets_path=Path(
os.getenv("APP_BENCHMARK_DATASETS_PATH", str(data_dir / "benchmarks"))
),
exports_path=Path(os.getenv("APP_EXPORTS_PATH", str(data_dir / "exports"))),
)
+35 -5
View File
@@ -3,26 +3,39 @@ from dataclasses import dataclass
from app.agent import AgentRuntime, PermissionManager, PermissionPolicy, ToolRegistry
from app.agent.builtin_tools import register_builtin_tools
from app.contracts import ModelCapability, ProviderConfig, ProviderType
from app.config import BACKEND_DIR
from app.config import BACKEND_DIR, get_settings
from app.extensions import PluginRuntime, SkillRuntime
from app.extensions.mcp_registry import McpServerRegistry
from app.providers import MockProvider, ProviderFactory, ProviderRegistry
from app.providers.credentials import EnvironmentCredentialResolver
from app.providers.routing import ModelRoutingService
from app.providers.credentials import (
ChainedCredentialResolver,
EncryptedCredentialStore,
EnvironmentCredentialResolver,
)
@dataclass(frozen=True)
class ApplicationContainer:
providers: ProviderRegistry
provider_factory: ProviderFactory
model_routing: ModelRoutingService
credentials: EncryptedCredentialStore
tools: ToolRegistry
permissions: PermissionManager
skills: SkillRuntime
plugins: PluginRuntime
mcp_servers: McpServerRegistry
agent: AgentRuntime
def build_container() -> ApplicationContainer:
provider_factory = ProviderFactory(EnvironmentCredentialResolver())
providers = ProviderRegistry()
settings = get_settings()
credentials = EncryptedCredentialStore()
provider_factory = ProviderFactory(
ChainedCredentialResolver(credentials, EnvironmentCredentialResolver())
)
providers = ProviderRegistry(provider_factory)
providers.register(
ProviderConfig(
provider_id="mock",
@@ -42,10 +55,24 @@ def build_container() -> ApplicationContainer:
tools = ToolRegistry()
register_builtin_tools(tools)
plugins = PluginRuntime(tools)
plugins = PluginRuntime(
tools,
credentials=credentials,
# 当前 Python Host 尚无 OS 沙箱。生产构建必须保持关闭,直到
# Tauri/Rust Host 能签发绑定命令摘要的可信启动许可。
allow_unsandboxed_mcp=settings.environment == "development",
)
plugins.install(BACKEND_DIR / "extensions" / "plugins" / "text-tools")
plugins.enable("text-tools")
mcp_servers = McpServerRegistry(
tools,
credentials,
settings.data_dir,
allow_process_launch=settings.environment == "development",
)
mcp_servers.restore_enabled()
skills = SkillRuntime(tools)
skills.install(BACKEND_DIR / "extensions" / "skills" / "knowledge-assistant")
skills.enable("knowledge-assistant")
@@ -61,10 +88,13 @@ def build_container() -> ApplicationContainer:
return ApplicationContainer(
providers=providers,
provider_factory=provider_factory,
model_routing=ModelRoutingService(providers, provider_factory.credentials),
credentials=credentials,
tools=tools,
permissions=permissions,
skills=skills,
plugins=plugins,
mcp_servers=mcp_servers,
agent=agent,
)
+722 -6
View File
@@ -1,8 +1,15 @@
from datetime import datetime
from enum import Enum
from typing import Any, Literal
from typing import Annotated, Any, Literal
from pydantic import BaseModel, ConfigDict, Field
from pydantic import (
BaseModel,
ConfigDict,
Field,
SecretStr,
field_validator,
model_validator,
)
class Contract(BaseModel):
@@ -31,6 +38,48 @@ class OperationResponse(Contract):
message: str | None = None
# Workspace boundary (single configured Vault in Web development mode)
class WorkspaceInfo(Contract):
vault_id: str = "default"
name: str
path: str
file_count: int = 0
indexed_note_count: int = 0
requires_refresh: bool = False
class WorkspaceEntry(Contract):
entry_id: str
name: str
path: str
type: Literal["file", "folder"]
note_id: str | None = None
children: list["WorkspaceEntry"] = Field(default_factory=list)
class WorkspaceSnapshot(Contract):
workspace: WorkspaceInfo
items: list[WorkspaceEntry] = Field(default_factory=list)
class WorkspaceOpenRequest(Contract):
path: str | None = None
class FolderCreateRequest(Contract):
parent: str = ""
name: str = Field(min_length=1)
class FolderRenameRequest(Contract):
path: str
new_name: str = Field(min_length=1)
class FolderDeleteRequest(Contract):
path: str
# Notes and retrieval
class NoteBlock(Contract):
block_id: str
@@ -79,6 +128,10 @@ class NoteMoveRequest(Contract):
folder: str
class NoteRenameRequest(Contract):
file_name: str = Field(min_length=1)
class SearchMode(str, Enum):
fts = "fts"
vector = "vector"
@@ -98,6 +151,12 @@ class SearchRequest(Contract):
limit: int = Field(default=20, ge=1, le=100)
offset: int = Field(default=0, ge=0)
include_snippet: bool = True
# 检索调优参数(Benchmark 与 Skill 共用):控制 RRF / 精排 / 候选池 / 分数阈值。
# rerank_candidates=None 表示对全部候选精排(保留原有行为),Benchmark 传显式值。
rrf_k: int = Field(default=60, ge=1)
rerank: bool = True
rerank_candidates: int | None = Field(default=None, ge=1)
score_threshold: float = Field(default=0.0, ge=0.0)
class Citation(Contract):
@@ -153,7 +212,7 @@ class ToolDefinition(Contract):
description: str
parameters: dict[str, Any] = Field(default_factory=dict)
permission: str | None = None
source: Literal["builtin", "plugin"] = "builtin"
source: Literal["builtin", "plugin", "mcp_server"] = "builtin"
class ToolCall(Contract):
@@ -184,6 +243,8 @@ class ModelCapability(str, Enum):
streaming = "streaming"
structured_output = "structured_output"
embedding = "embedding"
transcription = "transcription"
speaker_matching = "speaker_matching"
class ModelRequest(Contract):
@@ -283,6 +344,10 @@ class AgentEventType(str, Enum):
permission_required = "PermissionRequired"
usage = "Usage"
citation = "Citation"
model_call_started = "ModelCallStarted"
model_call_completed = "ModelCallCompleted"
model_call_failed = "ModelCallFailed"
permission_resolved = "PermissionResolved"
run_completed = "RunCompleted"
run_failed = "RunFailed"
run_cancelled = "RunCancelled"
@@ -296,6 +361,24 @@ class AgentEvent(Contract):
timestamp: datetime
class AgentTraceSummary(Contract):
model_calls: int = 0
tool_calls: int = 0
duration_ms: int = 0
token_usage: int = 0
errors: int = 0
class AgentTraceResponse(Contract):
run_id: str
status: AgentRunStatus
items: list[AgentEvent] = Field(default_factory=list)
next_sequence: int
has_more: bool = False
summary: AgentTraceSummary = Field(default_factory=AgentTraceSummary)
config_snapshot: dict[str, Any] = Field(default_factory=dict)
class PermissionDecisionRequest(Contract):
decision: Literal["allow_once", "allow_session", "deny"]
@@ -349,6 +432,10 @@ class ExtensionInstallRequest(Contract):
class PluginBackend(Contract):
type: Literal["mcp", "internal_rpc", "none"] = "none"
transport: Literal["stdio", "http", "none"] = "none"
command: str | None = None
args: list[str] = Field(default_factory=list)
startup_timeout_seconds: int = Field(default=10, ge=1, le=60)
tool_timeout_seconds: int = Field(default=30, ge=1, le=600)
class PluginContribution(Contract):
@@ -392,6 +479,285 @@ class PluginListResponse(Contract):
items: list[Plugin] = Field(default_factory=list)
class PluginHostState(str, Enum):
stopped = "stopped"
starting = "starting"
ready = "ready"
unhealthy = "unhealthy"
error = "error"
class PluginHostStatus(Contract):
plugin_id: str
backend_type: Literal["mcp", "internal_rpc", "none"]
transport: Literal["stdio", "http", "none"]
status: PluginHostState
tools_count: int = 0
started_at: datetime | None = None
last_seen_at: datetime | None = None
protocol_version: str | None = None
server_name: str | None = None
server_version: str | None = None
error: str | None = None
# Independent user-managed MCP Server Registry. This is deliberately separate
# from Plugin manifests: a server can contribute tools without being a Plugin.
class McpServerTransport(str, Enum):
stdio = "stdio"
streamable_http = "streamable_http"
sse = "sse"
class McpServerConfig(Contract):
name: str = Field(min_length=1, max_length=80)
transport: McpServerTransport = McpServerTransport.stdio
command: str | None = Field(default=None, max_length=1024)
args: list[str] = Field(default_factory=list, max_length=64)
url: str | None = Field(default=None, max_length=4096)
headers: dict[str, str] = Field(default_factory=dict)
environment: dict[str, str] = Field(default_factory=dict)
secret_environment_keys: list[str] = Field(default_factory=list)
secret_header_keys: list[str] = Field(default_factory=list)
permissions: list[str] = Field(default_factory=list)
startup_timeout_seconds: float = Field(default=15, ge=1, le=120)
tool_timeout_seconds: float = Field(default=30, ge=1, le=300)
class McpServerCreateRequest(McpServerConfig):
pass
class McpServerUpdateRequest(McpServerConfig):
version: int = Field(ge=1)
class McpServerSecretWriteRequest(Contract):
secret: SecretStr = Field(min_length=1, max_length=32768)
class McpServerSecretStatus(Contract):
key: str
configured: bool
class McpServerTrustRequest(Contract):
command_digest: str = Field(min_length=64, max_length=64)
class McpServerStatus(Contract):
enabled: bool = False
status: PluginHostState = PluginHostState.stopped
tools_count: int = 0
protocol_version: str | None = None
remote_server_name: str | None = None
remote_server_version: str | None = None
error: str | None = None
last_tested_at: datetime | None = None
last_test_succeeded: bool | None = None
class McpServer(McpServerStatus):
server_id: str
version: int
name: str
transport: McpServerTransport
command: str | None = None
args: list[str] = Field(default_factory=list)
url: str | None = None
headers: dict[str, str] = Field(default_factory=dict)
environment: dict[str, str] = Field(default_factory=dict)
permissions: list[str] = Field(default_factory=list)
startup_timeout_seconds: float
tool_timeout_seconds: float
secret_environment: dict[str, bool] = Field(default_factory=dict)
secret_headers: dict[str, bool] = Field(default_factory=dict)
trusted: bool = False
command_digest: str
command_summary: str
class McpServerListResponse(Contract):
items: list[McpServer] = Field(default_factory=list)
class McpToolSummary(Contract):
name: str
remote_name: str
description: str
permission: str | None = None
class McpToolSummaryListResponse(Contract):
items: list[McpToolSummary] = Field(default_factory=list)
class PluginCommandLocation(str, Enum):
command_palette = "command_palette"
context_menu = "context_menu"
toolbar = "toolbar"
class PluginCommand(Contract):
command_id: str
plugin_id: str
title: str
description: str = ""
icon: str | None = None
locations: list[PluginCommandLocation] = Field(default_factory=list)
when: list[str] = Field(default_factory=list)
parameters: dict[str, Any] = Field(default_factory=dict)
enabled: bool = True
class PluginCommandListResponse(Contract):
items: list[PluginCommand] = Field(default_factory=list)
class PluginCommandContext(Contract):
vault_id: str | None = None
note_id: str | None = None
file_path: str | None = None
selection: str | None = None
class PluginCommandExecuteRequest(Contract):
arguments: dict[str, Any] = Field(default_factory=dict)
context: PluginCommandContext = Field(default_factory=PluginCommandContext)
class PluginNotificationEffectPayload(Contract):
level: Literal["info", "success", "warning", "error"] = "info"
message: str = Field(min_length=1, max_length=4096)
class PluginNavigateEffectPayload(Contract):
route: Literal[
"vault-entry",
"workspace",
"search",
"chat",
"agent",
"tasks",
"skills",
"plugins",
"themes",
"settings",
]
class PluginRefreshEffectPayload(Contract):
scope: Literal["workspace", "commands", "settings", "plugins"]
class PluginJobEffectPayload(Contract):
job_id: str = Field(
min_length=1,
max_length=128,
pattern=r"^[A-Za-z0-9][A-Za-z0-9._:-]*$",
)
class PluginNoEffectPayload(Contract):
pass
class PluginNoEffect(Contract):
type: Literal["none"] = "none"
payload: PluginNoEffectPayload = Field(default_factory=PluginNoEffectPayload)
class PluginNotificationEffect(Contract):
type: Literal["notification"] = "notification"
payload: PluginNotificationEffectPayload
class PluginNavigateEffect(Contract):
type: Literal["navigate"] = "navigate"
payload: PluginNavigateEffectPayload
class PluginRefreshEffect(Contract):
type: Literal["refresh"] = "refresh"
payload: PluginRefreshEffectPayload
class PluginJobEffect(Contract):
type: Literal["job"] = "job"
payload: PluginJobEffectPayload
PluginCommandEffect = Annotated[
PluginNoEffect
| PluginNotificationEffect
| PluginNavigateEffect
| PluginRefreshEffect
| PluginJobEffect,
Field(discriminator="type"),
]
PLUGIN_COMMAND_EFFECT_TYPES = (
PluginNoEffect,
PluginNotificationEffect,
PluginNavigateEffect,
PluginRefreshEffect,
PluginJobEffect,
)
class PluginCommandResult(Contract):
command_id: str
status: Literal["completed"] = "completed"
effect: PluginCommandEffect = Field(default_factory=PluginNoEffect)
class PluginSettingType(str, Enum):
string = "string"
number = "number"
boolean = "boolean"
select = "select"
secret = "secret"
class PluginSettingField(Contract):
key: str
label: str
description: str = ""
type: PluginSettingType
required: bool = False
default: Any | None = None
minimum: float | None = None
maximum: float | None = None
options: list[str] = Field(default_factory=list)
class PluginSecretState(Contract):
configured: bool = False
class PluginSettingsSchema(Contract):
plugin_id: str
schema_version: int = Field(ge=1)
fields: list[PluginSettingField] = Field(default_factory=list)
values: dict[str, Any] = Field(default_factory=dict)
secrets: dict[str, PluginSecretState] = Field(default_factory=dict)
class PluginSettingsUpdateRequest(Contract):
schema_version: int = Field(ge=1)
values: dict[str, Any] = Field(default_factory=dict)
class PluginSecretWriteRequest(Contract):
secret: SecretStr
class PluginSecretStatus(Contract):
plugin_id: str
key: str
configured: bool
class PluginPermissionGrantRequest(Contract):
permissions: list[str] = Field(default_factory=list)
@@ -406,7 +772,24 @@ class ProviderType(str, Enum):
ollama = "ollama"
class ProviderConfig(Contract):
class ProviderConnectionFields(Contract):
base_url: str | None = None
credential_id: str | None = None
@field_validator("base_url")
@classmethod
def provider_url(cls, value: str | None) -> str | None:
if value is None:
return value
from urllib.parse import urlsplit
parsed = urlsplit(value)
if (parsed.scheme not in {"http", "https"} or not parsed.hostname or
parsed.username or parsed.password or parsed.query or parsed.fragment):
raise ValueError("Base URL requires HTTP(S), without credentials, query or fragment")
return value.rstrip("/")
class ProviderConfig(ProviderConnectionFields):
provider_id: str
provider_type: ProviderType
name: str
@@ -417,7 +800,7 @@ class ProviderConfig(Contract):
capabilities: list[ModelCapability] = Field(default_factory=list)
class ProviderCreateRequest(Contract):
class ProviderCreateRequest(ProviderConnectionFields):
provider_type: ProviderType
name: str
base_url: str | None = None
@@ -426,7 +809,8 @@ class ProviderCreateRequest(Contract):
enabled: bool = True
class ProviderUpdateRequest(Contract):
class ProviderUpdateRequest(ProviderConnectionFields):
provider_type: ProviderType | None = None
name: str | None = None
base_url: str | None = None
default_model: str | None = None
@@ -438,6 +822,102 @@ class ProviderListResponse(Contract):
items: list[ProviderConfig] = Field(default_factory=list)
class ProviderPreset(Contract):
preset_id: str
name: str
provider_type: ProviderType
base_url: str
default_credential_id: str | None = None
requires_credential: bool = True
logo_id: str = "custom"
description: str = ""
capabilities: list[ModelCapability] = Field(default_factory=list)
class ModelBinding(Contract):
provider_id: str = Field(min_length=1, max_length=128)
model: str = Field(min_length=1, max_length=256)
endpoint: str = Field(min_length=1, max_length=256)
dimensions: int | None = Field(default=None, ge=1, le=16384)
@field_validator("endpoint")
@classmethod
def relative_endpoint(cls, value: str) -> str:
# An endpoint is a path on the selected provider, never a second origin.
import re
if not re.fullmatch(r"/[A-Za-z0-9_/-]+", value) or value.startswith("//"):
raise ValueError("endpoint must be an absolute API path on the provider")
return value
@field_validator("model", "provider_id")
@classmethod
def non_blank(cls, value: str) -> str:
if not value.strip():
raise ValueError("value must not be blank")
return value.strip()
class ModelRoutingConfig(Contract):
version: int = Field(default=0, ge=0)
embedding: ModelBinding | None = None
transcription: ModelBinding | None = None
speaker_matching: ModelBinding | None = None
class LocalBackendStatus(Contract):
capability: Literal["embedding", "transcription", "speaker_matching"]
status: Literal["placeholder", "not_installed", "ready"]
message: str
class ModelRoutingResponse(Contract):
config: ModelRoutingConfig
local_backends: list[LocalBackendStatus]
class EmbeddingRequest(Contract):
texts: list[str] = Field(min_length=1, max_length=256)
@field_validator("texts")
@classmethod
def bound_texts(cls, value: list[str]) -> list[str]:
if sum(len(text) for text in value) > 200_000:
raise ValueError("embedding input is too large")
return value
class EmbeddingResult(Contract):
vectors: list[list[float]]
source: Literal["api", "local"]
model_id: str
dimensions: int
fallback_reason: str | None = None
class SpeakerMatchRequest(Contract):
attachment_id: str
reference_attachment_id: str
class SpeakerMatchResult(Contract):
score: float = Field(ge=0, le=1, allow_inf_nan=False)
source: Literal["api", "local"]
fallback_reason: str | None = None
class ProviderPresetListResponse(Contract):
items: list[ProviderPreset] = Field(default_factory=list)
class CredentialWriteRequest(Contract):
api_key: SecretStr = Field(min_length=1, max_length=8192)
class CredentialStatus(Contract):
credential_id: str
configured: bool
class ModelInfo(Contract):
model: str
display_name: str
@@ -515,6 +995,8 @@ class TranscriptionJob(Contract):
error_code: str | None = None
error_message: str | None = None
created_at: datetime
source: Literal["api", "local", "sidecar"] | None = None
fallback_reason: str | None = None
class IndexStatus(Contract):
@@ -536,3 +1018,237 @@ class IndexJob(Contract):
status: Literal["queued", "running", "completed", "failed"]
scope: Literal["all", "notes", "vectors"]
created_at: datetime
# Benchmark
class BenchmarkKind(str, Enum):
rag = "rag"
agent = "agent"
class BenchmarkStatus(str, Enum):
queued = "queued"
running = "running"
completed = "completed"
failed = "failed"
cancelled = "cancelled"
class RAGDatasetCase(Contract):
case_id: str
query: str = Field(min_length=1)
expected_note_ids: list[str] = Field(default_factory=list)
expected_block_ids: list[str] = Field(default_factory=list)
citation_required: bool = False
tags: list[str] = Field(default_factory=list)
class RAGRetrievalConfig(Contract):
"""RAG Benchmark 的检索参数。top_k 映射到 SearchRequest.limit
其余参数透传到 SearchRequest,由检索引擎实际执行。"""
top_k: int = Field(default=10, ge=1, le=100)
rrf_k: int = Field(default=60, ge=1)
rerank: bool = True
rerank_candidates: int = Field(default=20, ge=1)
score_threshold: float = Field(default=0.0, ge=0.0)
class RAGRunRequest(Contract):
dataset_id: str = Field(min_length=1)
modes: list[SearchMode] = Field(
default_factory=lambda: [SearchMode.fts, SearchMode.vector, SearchMode.hybrid],
min_length=1,
)
retrieval: RAGRetrievalConfig = Field(default_factory=RAGRetrievalConfig)
repeat: int = Field(default=1, ge=1, le=10)
metadata: dict[str, Any] = Field(default_factory=dict)
@field_validator("modes")
@classmethod
def _no_duplicate_modes(cls, value: list[SearchMode]) -> list[SearchMode]:
if len(value) != len(set(value)):
raise ValueError("modes must not contain duplicates")
return value
class RAGMetrics(Contract):
hit_at_1: float = 0.0
hit_at_5: float = 0.0
recall_at_k: float = 0.0
mrr: float = 0.0
citation_hit_rate: float = 0.0
p50_latency_ms: float = 0.0
p95_latency_ms: float = 0.0
# 样本构成:失败样本按零分计入质量指标,汇总不虚高;报告据此可知实际分母
total_cases: int = 0
successful_cases: int = 0
failed_cases: int = 0
failure_rate: float = 0.0
class BenchmarkDatasetInfo(Contract):
dataset_id: str
kind: BenchmarkKind
version: str
description: str = ""
case_count: int
content_hash: str
class BenchmarkDatasetListResponse(Contract):
items: list[BenchmarkDatasetInfo] = Field(default_factory=list)
class BenchmarkRun(Contract):
run_id: str
kind: BenchmarkKind
dataset_id: str
dataset_hash: str
status: BenchmarkStatus
progress: float | None = None
metrics: dict[str, Any] | None = None
config_snapshot: dict[str, Any] = Field(default_factory=dict)
error: str | None = None
error_code: str | None = None
created_at: datetime
started_at: datetime | None = None
completed_at: datetime | None = None
class BenchmarkRunListResponse(Contract):
items: list[BenchmarkRun] = Field(default_factory=list)
page: PageMeta = Field(default_factory=PageMeta)
class BenchmarkEventType(str, Enum):
run_started = "RunStarted"
case_completed = "CaseCompleted"
run_completed = "RunCompleted"
run_failed = "RunFailed"
run_cancelled = "RunCancelled"
class BenchmarkEvent(Contract):
event: BenchmarkEventType
run_id: str
sequence: int
data: dict[str, Any] = Field(default_factory=dict)
timestamp: datetime
class RAGCaseResult(Contract):
embedding: dict[str, Any] = Field(default_factory=dict)
case_id: str
mode: SearchMode
repeat: int
latency_ms: float
retrieved_note_ids: list[str] = Field(default_factory=list)
retrieved_block_ids: list[str] = Field(default_factory=list)
hit_at_1: bool = False
hit_at_5: bool = False
recall: float = 0.0
reciprocal_rank: float = 0.0
citation_hit: bool = False
# 该 Case 是否声明了 expected_block_ids(决定是否计入 citation_hit_rate 分母)
citation_applicable: bool = False
error: str | None = None
error_code: str | None = None
class BenchmarkReport(Contract):
run_id: str
kind: BenchmarkKind
dataset_id: str
dataset_hash: str
status: BenchmarkStatus
config_snapshot: dict[str, Any] = Field(default_factory=dict)
metrics: dict[str, Any] = Field(default_factory=dict)
cases: list[RAGCaseResult] = Field(default_factory=list)
error: str | None = None
error_code: str | None = None
# Export(多格式文档导出)
class ExportStatus(str, Enum):
queued = "queued"
running = "running"
completed = "completed"
failed = "failed"
cancelled = "cancelled"
class ExportFormat(str, Enum):
html = "html"
pdf = "pdf"
docx = "docx"
class ExportSourceType(str, Enum):
note = "note"
markdown = "markdown"
class ExportSource(Contract):
"""导出源:note 引用已索引笔记,markdown 用于未保存预览(不持久化)。"""
type: ExportSourceType
note_id: str | None = None
markdown: str | None = None
@model_validator(mode="after")
def _validate_source(self) -> "ExportSource":
if self.type == ExportSourceType.note and not self.note_id:
raise ValueError("note source requires note_id")
if self.type == ExportSourceType.markdown and not self.markdown:
raise ValueError("markdown source requires markdown")
return self
class ExportOptions(Contract):
theme_id: str = "light"
include_title: bool = True
include_metadata: bool = False
page_size: str = "A4"
code_theme: str = "github-light"
class ExportRequest(Contract):
source: ExportSource
format: ExportFormat
options: ExportOptions = Field(default_factory=ExportOptions)
class ExportProgress(Contract):
phase: str
current: int
total: int
percent: float | None = None
message: str | None = None
class ExportFile(Contract):
file_name: str
mime_type: str
size: int
sha256: str
expires_at: datetime
class ExportJob(Contract):
job_id: str
status: ExportStatus
format: ExportFormat
progress: ExportProgress | None = None
file: ExportFile | None = None
warnings: list[str] = Field(default_factory=list)
error: str | None = None
error_code: str | None = None
created_at: datetime
started_at: datetime | None = None
completed_at: datetime | None = None
class ExportJobListResponse(Contract):
items: list[ExportJob] = Field(default_factory=list)
page: PageMeta = Field(default_factory=PageMeta)
+27
View File
@@ -69,6 +69,33 @@ MIGRATIONS: list[str] = [
);
CREATE INDEX IF NOT EXISTS idx_tasks_status_due ON tasks(status, due_at);
""",
# v3: 第二阶段 Agent TraceRun 与事件事实持久化,供 SSE 恢复和 Benchmark 复用。
"""
CREATE TABLE IF NOT EXISTS agent_runs (
run_id TEXT PRIMARY KEY,
status TEXT NOT NULL,
run_json TEXT NOT NULL,
request_json TEXT NOT NULL,
config_snapshot_json TEXT NOT NULL DEFAULT '{}',
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_agent_runs_created
ON agent_runs(created_at DESC);
CREATE INDEX IF NOT EXISTS idx_agent_runs_status
ON agent_runs(status, updated_at DESC);
CREATE TABLE IF NOT EXISTS agent_events (
run_id TEXT NOT NULL REFERENCES agent_runs(run_id) ON DELETE CASCADE,
sequence INTEGER NOT NULL,
event TEXT NOT NULL,
data_json TEXT NOT NULL DEFAULT '{}',
timestamp TEXT NOT NULL,
PRIMARY KEY (run_id, sequence)
);
CREATE INDEX IF NOT EXISTS idx_agent_events_type
ON agent_events(run_id, event, sequence);
""",
]
+5 -1
View File
@@ -36,7 +36,11 @@ async def validation_error_handler(_: Request, exc: RequestValidationError) -> J
error=ErrorDetail(
code="VALIDATION_ERROR",
message="Request validation failed.",
details={"errors": exc.errors()},
# Pydantic ctx can contain exception objects; input may contain API keys.
details={"errors": [
{key: error[key] for key in ("type", "loc", "msg") if key in error}
for error in exc.errors()
]},
)
)
return JSONResponse(status_code=422, content=jsonable_encoder(body))
+8
View File
@@ -0,0 +1,8 @@
"""Export Service:多格式文档导出(首批 HTML)。
模块划分:
- document.py Document AST 内部协议 + DocumentExporter Protocol + ExportResult
- markdown.py mistune → Document AST 解析
- exporters/html.py HtmlExporterDocument AST → HTML5
- service.py 导出任务注册表、后台执行、取消与文件生命周期
"""
+46
View File
@@ -0,0 +1,46 @@
"""Document AST:导出器的内部中间表示(Internal Protocol,不放入 contracts.py)。
契约 §10.3 规定节点用稳定判别字段 node_id / type / attributes / children / text
类型专有信息统一放 attributes(如 heading 的 level、link 的 href、image 的 src)。
导出器据此递归渲染,对无法表示的节点记 warning,不静默丢弃。
"""
from __future__ import annotations
from typing import Any, Protocol
from pydantic import BaseModel, ConfigDict, Field
from app.contracts import ExportOptions
class DocumentNode(BaseModel):
"""递归文档节点;type 取契约 §10.3 首批 node type 之一。"""
model_config = ConfigDict(extra="forbid")
type: str
node_id: str
attributes: dict[str, Any] = Field(default_factory=dict)
children: list["DocumentNode"] = Field(default_factory=list)
text: str = ""
class Document(DocumentNode):
"""根节点,type 固定为 document。"""
type: str = "document"
class DocumentExporter(Protocol):
"""导出器协议(契约 §10.3):把 Document AST 渲染为指定格式的产物。"""
async def export(self, document: Document, options: ExportOptions) -> "ExportResult": ...
class ExportResult(BaseModel):
model_config = ConfigDict(extra="forbid")
content: bytes
mime_type: str
warnings: list[str] = Field(default_factory=list)
+1
View File
@@ -0,0 +1 @@
"""Export 渲染器:Document AST → 具体格式产物。"""
+216
View File
@@ -0,0 +1,216 @@
"""HtmlExporterDocument AST → 完整 HTML5 文档(内嵌基础 CSS)。
对无法静态表达的节点(mermaid / function_plot)渲染为占位代码块并记 warning,不静默丢失;
严重内容缺失由 service 层以 EXPORT_UNSUPPORTED_CONTENT 判定,本层只负责逐节点渲染。
"""
from __future__ import annotations
import html
from datetime import datetime
from app.contracts import ExportOptions
from app.export.document import Document, DocumentNode, ExportResult
_MERMAID_WARNING = "mermaid 需前端渲染,已保留为占位代码块"
_FUNCTION_PLOT_WARNING = "函数图像渲染将在后续版本提供,已保留为占位代码块"
_BASE_CSS = """
body { margin: 0; background: #f6f7f9; color: #1f2328; font: 15px/1.7 -apple-system, 'Segoe UI', 'Microsoft YaHei', sans-serif; }
article { max-width: 860px; margin: 0 auto; padding: 40px 48px; background: #fff; }
article.theme-dark { background: #0d1117; color: #c9d1d9; }
h1, h2, h3, h4, h5, h6 { line-height: 1.3; margin: 1.4em 0 0.6em; }
h1.title { margin-top: 0; }
p { margin: 0.6em 0; }
a { color: #0969da; }
code { font-family: 'JetBrains Mono', Consolas, monospace; font-size: 0.9em; background: #f0f1f3; padding: 0.15em 0.35em; border-radius: 3px; }
pre { background: #f6f8fa; padding: 14px 16px; border-radius: 6px; overflow-x: auto; }
pre.code-theme-github-dark { background: #0d1117; color: #c9d1d9; }
pre code { background: none; padding: 0; }
pre.mermaid, pre.function-plot { border: 1px dashed #d0d7de; }
blockquote { margin: 0.8em 0; padding: 0.2em 1em; border-left: 4px solid #d0d7de; color: #57606a; }
img { max-width: 100%; }
table { border-collapse: collapse; margin: 0.8em 0; }
th, td { border: 1px solid #d0d7de; padding: 6px 12px; }
th { background: #f6f8fa; }
dl.metadata { font-size: 0.85em; color: #57606a; border-top: 1px solid #eaeef2; border-bottom: 1px solid #eaeef2; padding: 0.6em 0; }
dl.metadata dt { display: inline; font-weight: 600; margin-right: 0.4em; }
dl.metadata dd { display: inline; margin: 0 1.2em 0 0; }
.math, .math-block { overflow-x: auto; padding: 0.4em 0; }
.task-list-item { list-style: none; }
.task-list-item input { margin-right: 0.4em; }
hr { border: none; border-top: 1px solid #d0d7de; margin: 1.4em 0; }
""".strip()
class HtmlExporter:
"""实现 DocumentExporter:递归渲染 Document AST 为完整 HTML5 文档。"""
async def export(self, document: Document, options: ExportOptions) -> ExportResult:
self._options = options
warnings: list[str] = []
body = self._render_children(document.children, warnings)
content = self._assemble(document, options, body, warnings)
return ExportResult(
content=content.encode("utf-8"), mime_type="text/html", warnings=warnings
)
def _assemble(
self, document: Document, options: ExportOptions, body: str, warnings: list[str]
) -> str:
title = str(document.attributes.get("title") or "")
parts = [
"<!doctype html>",
'<html lang="zh-CN">',
"<head>",
'<meta charset="utf-8">',
'<meta name="viewport" content="width=device-width, initial-scale=1">',
]
if title:
parts.append(f"<title>{html.escape(title)}</title>")
parts.append(f"<style>{_BASE_CSS}</style>")
parts.append("</head>")
parts.append("<body>")
parts.append(f'<article class="theme-{html.escape(options.theme_id)}">')
if options.include_title and title:
parts.append(f'<h1 class="title">{html.escape(title)}</h1>')
if options.include_metadata:
metadata = document.attributes.get("metadata")
if metadata:
parts.append(self._render_metadata(metadata))
parts.append(body)
parts.append("</article>")
parts.append("</body>")
parts.append("</html>")
return "\n".join(parts) + "\n"
def _render_metadata(self, metadata: dict) -> str:
entries = ["<dl", ' class="metadata">']
for key, value in metadata.items():
entries.append(f"<dt>{html.escape(str(key))}</dt>")
entries.append(f"<dd>{html.escape(self._fmt_meta_value(value))}</dd>")
entries.append("</dl>")
return "".join(entries)
@staticmethod
def _fmt_meta_value(value: object) -> str:
if isinstance(value, datetime):
return value.isoformat()
if isinstance(value, list):
return ", ".join(str(item) for item in value)
return str(value)
def _render_children(self, children: list[DocumentNode], warnings: list[str]) -> str:
return "".join(self._render_node(child, warnings) for child in children)
def _render_node(self, node: DocumentNode, warnings: list[str]) -> str:
handler = getattr(self, f"_render_{node.type}", None)
if handler is not None:
return handler(node, warnings)
warnings.append(f"无法表示的节点类型已跳过:{node.type}")
return ""
# --- 块级 ---
def _render_heading(self, node: DocumentNode, warnings: list[str]) -> str:
level = max(1, min(6, int(node.attributes.get("level", 1))))
return f"<h{level}>{self._render_children(node.children, warnings)}</h{level}>"
def _render_paragraph(self, node: DocumentNode, warnings: list[str]) -> str:
return f"<p>{self._render_children(node.children, warnings)}</p>"
def _render_blockquote(self, node: DocumentNode, warnings: list[str]) -> str:
return f"<blockquote>{self._render_children(node.children, warnings)}</blockquote>"
def _render_list(self, node: DocumentNode, warnings: list[str]) -> str:
tag = "ol" if node.attributes.get("ordered") else "ul"
return f"<{tag}>{self._render_children(node.children, warnings)}</{tag}>"
def _render_list_item(self, node: DocumentNode, warnings: list[str]) -> str:
inner = self._render_children(node.children, warnings)
if node.attributes.get("task"):
checked = " checked" if node.attributes.get("checked") else ""
return (
'<li class="task-list-item">'
f'<input type="checkbox" disabled{checked}>{inner}</li>'
)
return f"<li>{inner}</li>"
def _render_table(self, node: DocumentNode, warnings: list[str]) -> str:
rows = node.children
head_rows = [r for r in rows if r.attributes.get("head")]
body_rows = [r for r in rows if not r.attributes.get("head")]
parts = ["<table>"]
if head_rows:
parts.append("<thead>")
parts.extend(self._render_node(r, warnings) for r in head_rows)
parts.append("</thead>")
if body_rows:
parts.append("<tbody>")
parts.extend(self._render_node(r, warnings) for r in body_rows)
parts.append("</tbody>")
parts.append("</table>")
return "".join(parts)
def _render_table_row(self, node: DocumentNode, warnings: list[str]) -> str:
return f"<tr>{self._render_children(node.children, warnings)}</tr>"
def _render_table_cell(self, node: DocumentNode, warnings: list[str]) -> str:
tag = "th" if node.attributes.get("head") else "td"
return f"<{tag}>{self._render_children(node.children, warnings)}</{tag}>"
def _render_code_block(self, node: DocumentNode, warnings: list[str]) -> str:
lang = str(node.attributes.get("language") or "")
code = html.escape(node.text)
lang_cls = f' class="language-{html.escape(lang)}"' if lang else ""
theme = html.escape(self._options.code_theme)
return f'<pre class="code-theme-{theme}"><code{lang_cls}>{code}</code></pre>'
def _render_thematic_break(self, node: DocumentNode, warnings: list[str]) -> str:
return "<hr>"
def _render_mermaid(self, node: DocumentNode, warnings: list[str]) -> str:
warnings.append(_MERMAID_WARNING)
return f'<pre class="mermaid">{html.escape(node.text)}</pre>'
def _render_function_plot(self, node: DocumentNode, warnings: list[str]) -> str:
warnings.append(_FUNCTION_PLOT_WARNING)
return f'<pre class="function-plot">{html.escape(node.text)}</pre>'
def _render_math_block(self, node: DocumentNode, warnings: list[str]) -> str:
return f'<div class="math-block">$${html.escape(node.text)}$$</div>'
# --- 行内 ---
def _render_text(self, node: DocumentNode, warnings: list[str]) -> str:
return html.escape(node.text)
def _render_emphasis(self, node: DocumentNode, warnings: list[str]) -> str:
return f"<em>{self._render_children(node.children, warnings)}</em>"
def _render_strong(self, node: DocumentNode, warnings: list[str]) -> str:
return f"<strong>{self._render_children(node.children, warnings)}</strong>"
def _render_link(self, node: DocumentNode, warnings: list[str]) -> str:
href = html.escape(str(node.attributes.get("href") or ""))
title = str(node.attributes.get("title") or "")
attrs = [f'href="{href}"']
if title:
attrs.append(f'title="{html.escape(title)}"')
return f"<a {' '.join(attrs)}>{self._render_children(node.children, warnings)}</a>"
def _render_codespan(self, node: DocumentNode, warnings: list[str]) -> str:
return f"<code>{html.escape(node.text)}</code>"
def _render_image(self, node: DocumentNode, warnings: list[str]) -> str:
src = html.escape(str(node.attributes.get("src") or ""))
alt = html.escape(str(node.attributes.get("alt") or ""))
title = str(node.attributes.get("title") or "")
attrs = [f'src="{src}"', f'alt="{alt}"']
if title:
attrs.append(f'title="{html.escape(title)}"')
return f"<img {' '.join(attrs)}>"
def _render_math_inline(self, node: DocumentNode, warnings: list[str]) -> str:
return f"\\({html.escape(node.text)}\\)"
def _render_linebreak(self, node: DocumentNode, warnings: list[str]) -> str:
return "<br>"
+214
View File
@@ -0,0 +1,214 @@
"""Markdown → Document AST:用 mistune 的 ast renderer 产出通用 token,再映射为内部节点。
选用 mistune 内置 'ast' renderer 而非自写 BaseRenderer,是因为 mistune 的行内渲染按
字符串拼接、无法承载结构化子节点;ast renderer 直接给出带 children/attrs/raw 的 token
树,映射层只做 token → DocumentNode 的搬运,不掺入任何 HTML。
"""
from __future__ import annotations
import mistune
from app.export.document import Document, DocumentNode
_PLUGINS = ["table", "math", "url", "task_lists"]
# fenced code 语言分流:命中则转为专用节点,其余按普通代码块
_MERMAID_LANG = "mermaid"
_FUNCTION_PLOT_LANGS = {"function_plot", "functionplot"}
def parse_document(markdown: str) -> Document:
"""把 Markdown 文本解析为 Document AST 根节点。"""
renderer = mistune.create_markdown(renderer="ast", plugins=_PLUGINS)
tokens = renderer(markdown)
mapper = _AstMapper()
return Document(node_id=mapper.next_id(), children=mapper.map_blocks(tokens))
class _AstMapper:
"""token 树 → DocumentNode 树的映射器;node_id 按遍历顺序递增,无需跨请求稳定。"""
def __init__(self) -> None:
self._seq = 0
def next_id(self) -> str:
self._seq += 1
return f"node_{self._seq:03d}"
def map_blocks(self, tokens: list[dict]) -> list[DocumentNode]:
nodes: list[DocumentNode] = []
for token in tokens:
node = self.map_block(token)
if node is not None:
nodes.append(node)
return nodes
def map_block(self, token: dict) -> DocumentNode | None:
kind = token["type"]
if kind == "heading":
return DocumentNode(
type="heading",
node_id=self.next_id(),
attributes={"level": token["attrs"]["level"]},
children=self.map_inline(token.get("children", [])),
)
if kind in ("paragraph", "block_text"):
# block_text 是列表项内的段落块,仍按 paragraph 表达,由 list_item 包裹
return DocumentNode(
type="paragraph",
node_id=self.next_id(),
children=self.map_inline(token.get("children", [])),
)
if kind == "list":
return DocumentNode(
type="list",
node_id=self.next_id(),
attributes={"ordered": bool(token.get("attrs", {}).get("ordered"))},
children=[self.map_list_item(child) for child in token.get("children", [])],
)
if kind == "block_code":
return self._map_code(token)
if kind == "block_quote":
return DocumentNode(
type="blockquote",
node_id=self.next_id(),
children=self.map_blocks(token.get("children", [])),
)
if kind == "table":
return self._map_table(token)
if kind == "block_math":
return DocumentNode(
type="math_block", node_id=self.next_id(), text=token.get("raw", "")
)
if kind == "thematic_break":
return DocumentNode(type="thematic_break", node_id=self.next_id())
if kind == "blank_line":
return None
# 未知块级 token(如 block_html)保守保留原文,避免静默丢失
raw = token.get("raw", "")
if raw:
return DocumentNode(type="paragraph", node_id=self.next_id(), text=raw)
return None
def map_list_item(self, token: dict) -> DocumentNode:
"""列表项:block_text 展平为行内子节点,嵌套 list 保留为子节点。"""
attributes: dict = {}
if token["type"] == "task_list_item":
attributes = {"task": True, "checked": bool(token.get("attrs", {}).get("checked"))}
children: list[DocumentNode] = []
for child in token.get("children", []):
if child["type"] == "block_text":
children.extend(self.map_inline(child.get("children", [])))
elif child["type"] == "list":
children.append(self.map_block(child))
else:
node = self.map_block(child)
if node is not None:
children.append(node)
return DocumentNode(
type="list_item", node_id=self.next_id(), attributes=attributes, children=children
)
def map_inline(self, tokens: list[dict]) -> list[DocumentNode]:
nodes: list[DocumentNode] = []
for token in tokens:
node = self.map_inline_token(token)
if node is not None:
nodes.append(node)
return nodes
def map_inline_token(self, token: dict) -> DocumentNode | None:
kind = token["type"]
if kind == "text":
return DocumentNode(type="text", node_id=self.next_id(), text=token.get("raw", ""))
if kind == "strong":
return DocumentNode(
type="strong", node_id=self.next_id(),
children=self.map_inline(token.get("children", [])),
)
if kind == "emphasis":
return DocumentNode(
type="emphasis", node_id=self.next_id(),
children=self.map_inline(token.get("children", [])),
)
if kind == "link":
attrs = token.get("attrs", {})
attributes = {"href": attrs.get("url", "")}
if attrs.get("title"):
attributes["title"] = attrs["title"]
return DocumentNode(
type="link", node_id=self.next_id(), attributes=attributes,
children=self.map_inline(token.get("children", [])),
)
if kind == "codespan":
return DocumentNode(type="codespan", node_id=self.next_id(), text=token.get("raw", ""))
if kind == "image":
attrs = token.get("attrs", {})
attributes = {"src": attrs.get("src", "")}
if attrs.get("alt"):
attributes["alt"] = attrs["alt"]
if attrs.get("title"):
attributes["title"] = attrs["title"]
return DocumentNode(type="image", node_id=self.next_id(), attributes=attributes)
if kind == "inline_math":
return DocumentNode(
type="math_inline", node_id=self.next_id(), text=token.get("raw", "")
)
if kind == "softbreak":
# HTML 中换行会折叠为空白,软换行按空格表达
return DocumentNode(type="text", node_id=self.next_id(), text=" ")
if kind == "linebreak":
return DocumentNode(type="linebreak", node_id=self.next_id())
# 未知行内 token 保守保留原文
raw = token.get("raw", "")
if raw:
return DocumentNode(type="text", node_id=self.next_id(), text=raw)
return None
def _map_code(self, token: dict) -> DocumentNode:
info = (token.get("attrs", {}).get("info") or "").strip()
lang = info.split()[0].lower() if info else ""
code = token.get("raw", "").rstrip("\n")
if lang == _MERMAID_LANG:
return DocumentNode(type="mermaid", node_id=self.next_id(), text=code)
if lang in _FUNCTION_PLOT_LANGS:
return DocumentNode(type="function_plot", node_id=self.next_id(), text=code)
attributes = {"language": lang} if lang else {}
return DocumentNode(
type="code_block", node_id=self.next_id(), attributes=attributes, text=code
)
def _map_table(self, token: dict) -> DocumentNode:
rows: list[DocumentNode] = []
for child in token.get("children", []):
if child["type"] == "table_head":
rows.append(self._map_table_row(child, head=True))
elif child["type"] == "table_body":
for row in child.get("children", []):
if row["type"] == "table_row":
rows.append(self._map_table_row(row, head=False))
elif child["type"] == "table_row":
rows.append(self._map_table_row(child, head=False))
return DocumentNode(type="table", node_id=self.next_id(), children=rows)
def _map_table_row(self, token: dict, *, head: bool) -> DocumentNode:
cells: list[DocumentNode] = []
for cell in token.get("children", []):
if cell["type"] != "table_cell":
continue
attrs = cell.get("attrs", {})
cell_attributes = {"head": bool(attrs.get("head", head))}
if attrs.get("align"):
cell_attributes["align"] = attrs["align"]
cells.append(
DocumentNode(
type="table_cell",
node_id=self.next_id(),
attributes=cell_attributes,
children=self.map_inline(cell.get("children", [])),
)
)
return DocumentNode(
type="table_row", node_id=self.next_id(), attributes={"head": head}, children=cells
)
+272
View File
@@ -0,0 +1,272 @@
"""Export 服务:任务注册表、后台渲染、取消与产物生命周期。
与 Benchmark 一致采用「创建即返回 queued、后台 Task 异步执行」的内存模型:任务与产物
暂存内存与 exports 目录,不持久化到 SQLite。导出是单阶段渲染,无 SSE 事件流,取消主要
在渲染前/后让出执行权的边界生效;产物带 24h 过期时间,过期后不可下载。
"""
from __future__ import annotations
import asyncio
import hashlib
import logging
import re
from datetime import datetime, timedelta, timezone
from pathlib import Path
from uuid import uuid4
from app.config import get_settings
from app.contracts import (
ExportFile,
ExportFormat,
ExportJob,
ExportOptions,
ExportProgress,
ExportRequest,
ExportSource,
ExportSourceType,
ExportStatus,
)
from app.errors import ApiError
from app.export.document import Document
from app.export.exporters.html import HtmlExporter
from app.export.markdown import parse_document
from app.services import note_service
logger = logging.getLogger(__name__)
_jobs: dict[str, ExportJob] = {}
_tasks: dict[str, asyncio.Task] = {}
_cancel_flags: dict[str, asyncio.Event] = {}
MAX_JOBS = 100
# markdown 源大小上限,防止未保存预览塞爆内存/产物
MAX_MARKDOWN_CHARS = 200_000
# 产物有效期
FILE_TTL = timedelta(hours=24)
_INVALID_FILE_CHARS = re.compile(r'[\\/:*?"<>|]')
class ExportCancelled(Exception):
"""导出在渲染前被取消时抛出,用于标记 cancelled。"""
def _now() -> datetime:
return datetime.now(timezone.utc)
def _safe_download_name(title: str) -> str:
"""清洗标题得到安全的下载文件名;空标题回退到 export。"""
name = _INVALID_FILE_CHARS.sub("_", title).strip() or "export"
return name[:80]
def _forget(job_id: str) -> None:
_jobs.pop(job_id, None)
_tasks.pop(job_id, None)
_cancel_flags.pop(job_id, None)
def _evict_terminal() -> bool:
"""超过容量时淘汰最旧的终态任务;全为活动任务无法淘汰时返回 False。"""
terminal = (ExportStatus.completed, ExportStatus.failed, ExportStatus.cancelled)
while len(_jobs) >= MAX_JOBS:
victim = next((jid for jid, job in _jobs.items() if job.status in terminal), None)
if victim is None:
return False
_forget(victim)
return True
async def _resolve_source(source: ExportSource) -> tuple[str, str, dict | None]:
"""把导出源解析为 (markdown, title, metadata)metadata 仅 note 源提供。"""
if source.type == ExportSourceType.note:
note = await note_service.get_note(source.note_id)
if note is None:
raise ApiError(
404,
"EXPORT_SOURCE_NOT_FOUND",
"note not found",
{"note_id": source.note_id},
)
metadata = {
"file_path": note.file_path,
"tags": note.tags,
"created_at": note.created_at,
"updated_at": note.updated_at,
}
return note.markdown, note.title, metadata
markdown = source.markdown or ""
if not markdown.strip():
raise ApiError(400, "EXPORT_OPTIONS_INVALID", "markdown source must not be empty")
if len(markdown) > MAX_MARKDOWN_CHARS:
raise ApiError(
400,
"EXPORT_OPTIONS_INVALID",
f"markdown source exceeds {MAX_MARKDOWN_CHARS} characters",
{"size": len(markdown), "limit": MAX_MARKDOWN_CHARS},
)
return markdown, "", None
async def create_export(request: ExportRequest) -> ExportJob:
"""创建导出任务,立即返回 queued 的 ExportJob,由后台 Task 渲染。"""
if request.format != ExportFormat.html:
raise ApiError(
400,
"EXPORT_FORMAT_UNSUPPORTED",
"PDF/DOCX 暂未实现,当前仅支持 HTML",
{"format": request.format.value},
)
markdown, title, metadata = await _resolve_source(request.source)
if not _evict_terminal():
raise ApiError(
429,
"EXPORT_CAPACITY_EXCEEDED",
"Export capacity exceeded; wait for active jobs to finish.",
{},
)
job_id = "export_" + uuid4().hex[:12]
job = ExportJob(
job_id=job_id,
status=ExportStatus.queued,
format=request.format,
created_at=_now(),
)
_jobs[job_id] = job
_cancel_flags[job_id] = asyncio.Event()
_tasks[job_id] = asyncio.create_task(
_execute(job_id, markdown, title, metadata, request.options)
)
return job
async def _execute(
job_id: str,
markdown: str,
title: str,
metadata: dict | None,
options: ExportOptions,
) -> None:
"""后台渲染:解析 → 导出 → 写文件 → 挂载产物元信息。"""
cancel_event = _cancel_flags[job_id]
_jobs[job_id] = _jobs[job_id].model_copy(
update={
"status": ExportStatus.running,
"started_at": _now(),
"progress": ExportProgress(phase="rendering", current=0, total=1, percent=0.0),
}
)
try:
# 让出一次,使「创建后立即取消」的 queued 任务能及时进入 cancelled
await asyncio.sleep(0)
if cancel_event.is_set():
raise ExportCancelled()
document = parse_document(markdown)
document.attributes["title"] = title
if metadata:
document.attributes["metadata"] = metadata
exporter = HtmlExporter()
result = await exporter.export(document, options)
if cancel_event.is_set():
raise ExportCancelled()
out_dir = get_settings().exports_path
out_dir.mkdir(parents=True, exist_ok=True)
path = out_dir / f"{job_id}.html"
path.write_bytes(result.content)
completed_at = _now()
_jobs[job_id] = _jobs[job_id].model_copy(
update={
"status": ExportStatus.completed,
"progress": ExportProgress(
phase="completed", current=1, total=1, percent=1.0
),
"file": ExportFile(
file_name=f"{_safe_download_name(title)}.html",
mime_type=result.mime_type,
size=len(result.content),
sha256=hashlib.sha256(result.content).hexdigest(),
expires_at=completed_at + FILE_TTL,
),
"warnings": result.warnings,
"completed_at": completed_at,
}
)
except ExportCancelled:
_jobs[job_id] = _jobs[job_id].model_copy(
update={
"status": ExportStatus.cancelled,
"completed_at": _now(),
}
)
except Exception as exc: # 渲染失败不拖垮服务,只记日志与项目错误码
logger.exception("Export failed: job_id=%s", job_id)
_jobs[job_id] = _jobs[job_id].model_copy(
update={
"status": ExportStatus.failed,
"error": "Export render failed.",
"error_code": "EXPORT_RENDER_FAILED",
"completed_at": _now(),
}
)
finally:
_cancel_flags.pop(job_id, None)
def list_exports(
status: ExportStatus | None = None,
format: ExportFormat | None = None,
limit: int = 50,
offset: int = 0,
) -> tuple[list[ExportJob], int]:
jobs = list(_jobs.values())
if status is not None:
jobs = [j for j in jobs if j.status == status]
if format is not None:
jobs = [j for j in jobs if j.format == format]
jobs.sort(key=lambda j: j.created_at, reverse=True)
total = len(jobs)
return jobs[offset : offset + limit], total
def get_export(job_id: str) -> ExportJob | None:
return _jobs.get(job_id)
def cancel_export(job_id: str) -> ExportJob | None:
"""取消导出:仅 queued/running 可取消,后台 Task 在让出边界标记 cancelled。"""
job = _jobs.get(job_id)
if job is None:
return None
if job.status in (ExportStatus.queued, ExportStatus.running):
_cancel_flags[job_id].set()
return job
def get_export_file(job_id: str) -> Path:
"""返回可下载产物的存储路径;未完成返回 404、过期返回 410。"""
job = _jobs.get(job_id)
if job is None:
raise ApiError(404, "EXPORT_JOB_NOT_FOUND", "export job not found", {"job_id": job_id})
if job.status != ExportStatus.completed or job.file is None:
raise ApiError(
404, "EXPORT_JOB_NOT_FOUND", "export file not ready", {"job_id": job_id}
)
if job.file.expires_at <= _now():
raise ApiError(410, "EXPORT_FILE_EXPIRED", "export file has expired", {"job_id": job_id})
return get_settings().exports_path / f"{job_id}.html"
async def wait_for_export(job_id: str) -> ExportJob | None:
"""等待后台任务结束(测试/轮询用);无任务时直接返回当前状态。"""
task = _tasks.get(job_id)
if task is not None:
await task
return _jobs.get(job_id)
+11 -7
View File
@@ -1,8 +1,12 @@
from app.extensions.runtime import (
AgentConfiguration,
ExtensionError,
PluginRuntime,
SkillRuntime,
)
from app.extensions.errors import ExtensionError
from app.extensions.runtime import AgentConfiguration, PluginRuntime, SkillRuntime
from app.extensions.mcp import McpBridge, McpBridgeError
__all__ = ["AgentConfiguration", "ExtensionError", "PluginRuntime", "SkillRuntime"]
__all__ = [
"AgentConfiguration",
"ExtensionError",
"McpBridge",
"McpBridgeError",
"PluginRuntime",
"SkillRuntime",
]
+847
View File
@@ -0,0 +1,847 @@
"""Plugin Command Registry 与 Settings/Secret 命名空间存储。"""
from __future__ import annotations
import asyncio
import hashlib
import inspect
import json
import math
import re
import threading
from collections import deque
from dataclasses import dataclass
from datetime import UTC, datetime
from pathlib import Path
from time import perf_counter
from typing import Any, Awaitable, Callable, Literal
from jsonschema import Draft202012Validator
from jsonschema.exceptions import SchemaError, ValidationError as JsonSchemaValidationError
from pydantic import BaseModel, ConfigDict, Field, model_validator
from app.config import get_settings
from app.contracts import (
PLUGIN_COMMAND_EFFECT_TYPES,
PluginCommand,
PluginCommandContext,
PluginCommandEffect,
PluginCommandLocation,
PluginCommandResult,
PluginSecretState,
PluginSecretStatus,
PluginSettingField,
PluginSettingType,
PluginSettingsSchema,
)
from app.extensions.errors import ExtensionError
from app.providers.credentials import CredentialStoreError, EncryptedCredentialStore
from app.schema_security import (
SchemaReferenceError,
reject_external_schema_references,
)
_CONTRIBUTION_ID = re.compile(r"^[a-z0-9][a-z0-9._-]*$")
_SETTING_KEY = re.compile(r"^[a-z][a-z0-9._-]{0,127}$")
_HOST_ICONS = {"bolt", "document", "edit", "link", "refresh", "search", "setting"}
_WHEN_TOKENS = {
"workspace.has_vault",
"editor.has_note",
"editor.has_selection",
}
_CONTEXT_KEYS = {"vault_id", "note_id", "file_path", "selection"}
_WHEN_CONTEXT = {
"workspace.has_vault": "vault_id",
"editor.has_note": "note_id",
"editor.has_selection": "selection",
}
class PluginCommandSpec(BaseModel):
"""包内 commands.yaml 的宿主侧声明,不直接暴露 handler。"""
model_config = ConfigDict(extra="forbid")
command_id: str
title: str
description: str = ""
icon: str | None = None
locations: list[PluginCommandLocation] = Field(default_factory=list)
when: list[str] = Field(default_factory=list)
context: list[Literal["vault_id", "note_id", "file_path", "selection"]] = Field(
default_factory=list
)
parameters: dict[str, Any] = Field(
default_factory=lambda: {
"type": "object",
"properties": {},
"additionalProperties": False,
}
)
permission: str | None = None
secrets: list[str] = Field(default_factory=list)
handler: Literal["echo", "uppercase_selection"] | None = None
mcp_tool: str | None = None
timeout_seconds: int = Field(default=30, ge=1, le=120)
@model_validator(mode="after")
def validate_execution_target(self) -> "PluginCommandSpec":
if (self.handler is None) == (self.mcp_tool is None):
raise ValueError("Command must declare exactly one handler or mcp_tool target.")
return self
CommandExecutor = Callable[
[dict[str, Any], dict[str, Any]],
PluginCommandEffect | Awaitable[PluginCommandEffect],
]
PluginSecretResolver = Callable[[str], str | None]
@dataclass(slots=True)
class _RegisteredCommand:
command: PluginCommand
spec: PluginCommandSpec
executor: CommandExecutor
@dataclass(frozen=True, slots=True)
class PluginCommandAuditEvent:
"""不记录参数与上下文的轻量审计事件,避免把正文或 Secret 写入日志。"""
command_id: str
plugin_id: str
status: Literal["completed", "failed"]
duration_ms: int
error_code: str | None
created_at: datetime
class CommandRegistry:
"""只发布已启用 Plugin 的受控 Command Contribution。"""
def __init__(self) -> None:
self._commands: dict[str, _RegisteredCommand] = {}
self._audit: deque[PluginCommandAuditEvent] = deque(maxlen=500)
self._lock = threading.RLock()
def register(
self,
plugin_id: str,
spec: PluginCommandSpec,
executor: CommandExecutor,
) -> None:
validate_command_spec(plugin_id, spec)
command = PluginCommand(
command_id=spec.command_id,
plugin_id=plugin_id,
title=spec.title,
description=spec.description,
icon=spec.icon,
locations=spec.locations,
when=spec.when,
parameters=spec.parameters,
enabled=True,
)
with self._lock:
if spec.command_id in self._commands:
raise ExtensionError(
"PLUGIN_COMMAND_CONFLICT",
f"Plugin command is already registered: {spec.command_id}",
status_code=409,
details={"command_id": spec.command_id},
)
self._commands[spec.command_id] = _RegisteredCommand(command, spec, executor)
def unregister(self, command_id: str) -> None:
with self._lock:
self._commands.pop(command_id, None)
def contains(self, command_id: str) -> bool:
with self._lock:
return command_id in self._commands
def list(self, location: PluginCommandLocation | None = None) -> list[PluginCommand]:
with self._lock:
items = [
item.command.model_copy(deep=True)
for item in self._commands.values()
if location is None or location in item.command.locations
]
return sorted(items, key=lambda item: item.command_id)
def audit_events(self) -> list[PluginCommandAuditEvent]:
"""返回有界审计快照;事件刻意不包含 arguments/context/effect。"""
with self._lock:
return list(self._audit)
async def execute(
self,
command_id: str,
arguments: dict[str, Any],
context: PluginCommandContext,
) -> PluginCommandResult:
with self._lock:
registered = self._commands.get(command_id)
if registered is None:
raise ExtensionError(
"PLUGIN_COMMAND_NOT_FOUND",
f"Plugin command is not registered or enabled: {command_id}",
status_code=404,
details={"command_id": command_id},
)
started_at = perf_counter()
try:
Draft202012Validator(registered.spec.parameters).validate(arguments)
except JsonSchemaValidationError as exc:
error = ExtensionError(
"PLUGIN_COMMAND_ARGUMENT_INVALID",
"Plugin command arguments do not match the declared schema.",
details={"command_id": command_id, "path": list(exc.path)},
)
self._record_audit(registered, started_at, error.code)
raise error from exc
raw_context = context.model_dump(exclude_none=True)
missing = [
token
for token in registered.spec.when
if not raw_context.get(_WHEN_CONTEXT[token])
]
if missing:
error = ExtensionError(
"PLUGIN_COMMAND_CONTEXT_INVALID",
"Plugin command context does not satisfy its when conditions.",
details={"command_id": command_id, "missing": missing},
)
self._record_audit(registered, started_at, error.code)
raise error
scoped_context = {
key: raw_context[key]
for key in registered.spec.context
if key in raw_context
}
try:
effect = registered.executor(dict(arguments), scoped_context)
if inspect.isawaitable(effect):
effect = await asyncio.wait_for(
effect, timeout=registered.spec.timeout_seconds
)
except TimeoutError as exc:
error = ExtensionError(
"PLUGIN_COMMAND_TIMEOUT",
"Plugin command execution timed out.",
status_code=504,
details={"command_id": command_id},
)
self._record_audit(registered, started_at, error.code)
raise error from exc
except ExtensionError as exc:
self._record_audit(registered, started_at, exc.code)
raise
except Exception as exc:
error = ExtensionError(
"PLUGIN_COMMAND_EXECUTION_FAILED",
"Plugin command execution failed.",
status_code=502,
details={"command_id": command_id},
)
self._record_audit(registered, started_at, error.code)
raise error from exc
if not isinstance(effect, PLUGIN_COMMAND_EFFECT_TYPES):
error = ExtensionError(
"PLUGIN_COMMAND_RESULT_INVALID",
"Plugin command returned an invalid effect.",
status_code=502,
details={"command_id": command_id},
)
self._record_audit(registered, started_at, error.code)
raise error
try:
encoded_effect = json.dumps(effect.model_dump(mode="json"), ensure_ascii=False)
except (TypeError, ValueError) as exc:
error = ExtensionError(
"PLUGIN_COMMAND_RESULT_INVALID",
"Plugin command returned a non-serializable effect.",
status_code=502,
details={"command_id": command_id},
)
self._record_audit(registered, started_at, error.code)
raise error from exc
if len(encoded_effect.encode("utf-8")) > 64 * 1024:
error = ExtensionError(
"PLUGIN_COMMAND_RESULT_TOO_LARGE",
"Plugin command effect exceeds the 64 KiB response limit.",
status_code=502,
details={"command_id": command_id},
)
self._record_audit(registered, started_at, error.code)
raise error
self._record_audit(registered, started_at, None)
return PluginCommandResult(command_id=command_id, effect=effect)
def _record_audit(
self,
registered: _RegisteredCommand,
started_at: float,
error_code: str | None,
) -> None:
event = PluginCommandAuditEvent(
command_id=registered.command.command_id,
plugin_id=registered.command.plugin_id,
status="failed" if error_code else "completed",
duration_ms=max(0, round((perf_counter() - started_at) * 1000)),
error_code=error_code,
created_at=datetime.now(UTC),
)
with self._lock:
self._audit.append(event)
class PluginSettingsDefinition(BaseModel):
model_config = ConfigDict(extra="forbid")
section_id: str
schema_version: int = Field(ge=1)
fields: list[PluginSettingField] = Field(default_factory=list)
class PluginSettingsStore:
"""非敏感值写入插件命名空间;Secret 只保存加密凭据引用。"""
def __init__(self, credentials: EncryptedCredentialStore) -> None:
self.credentials = credentials
self._lock = threading.RLock()
@staticmethod
def _path() -> Path:
return get_settings().data_dir / "plugins" / "settings.json"
def get(
self, plugin_id: str, definition: PluginSettingsDefinition
) -> PluginSettingsSchema:
with self._lock:
entry = self._entry(self._read(), plugin_id)
stored_values = entry.get("values", {})
secret_refs = entry.get("secret_refs", {})
if not isinstance(stored_values, dict) or not isinstance(secret_refs, dict):
raise self._storage_format_error(plugin_id)
validated_refs = self._validate_secret_refs(plugin_id, secret_refs)
values = {
field.key: field.default
for field in definition.fields
if field.type != PluginSettingType.secret and field.default is not None
}
allowed_values = {
field.key
for field in definition.fields
if field.type != PluginSettingType.secret
}
fields = {field.key: field for field in definition.fields}
for key, value in stored_values.items():
if key not in allowed_values:
continue
try:
_validate_setting_value(fields[key], value)
except ExtensionError as exc:
raise self._storage_format_error(plugin_id) from exc
values[key] = value
secrets: dict[str, PluginSecretState] = {}
for field in definition.fields:
if field.type != PluginSettingType.secret:
continue
reference = validated_refs.get(field.key)
secrets[field.key] = PluginSecretState(
configured=isinstance(reference, str) and self._has_secret(reference)
)
return PluginSettingsSchema(
plugin_id=plugin_id,
schema_version=definition.schema_version,
fields=definition.fields,
values=values,
secrets=secrets,
)
def runtime_values(
self, plugin_id: str, definition: PluginSettingsDefinition
) -> dict[str, Any]:
"""返回可供 Command 使用的完整普通设置,并拦截未配置的必填项。"""
schema = self.get(plugin_id, definition)
missing = [
field.key
for field in definition.fields
if field.required
and field.type != PluginSettingType.secret
and field.key not in schema.values
]
if missing:
raise ExtensionError(
"PLUGIN_SETTINGS_REQUIRED",
"Required Plugin settings have not been configured.",
status_code=409,
details={"plugin_id": plugin_id, "fields": missing},
)
return schema.values
def update(
self,
plugin_id: str,
definition: PluginSettingsDefinition,
schema_version: int,
values: dict[str, Any],
) -> PluginSettingsSchema:
if schema_version != definition.schema_version:
raise ExtensionError(
"PLUGIN_SETTINGS_VERSION_CONFLICT",
"Plugin settings schema version is out of date.",
status_code=409,
details={
"plugin_id": plugin_id,
"requested_version": schema_version,
"current_version": definition.schema_version,
},
)
fields = {field.key: field for field in definition.fields}
unknown = sorted(set(values) - set(fields))
if unknown:
raise ExtensionError(
"PLUGIN_SETTINGS_FIELD_INVALID",
"Plugin settings contain unknown fields.",
details={"plugin_id": plugin_id, "fields": unknown},
)
secret_keys = sorted(
key for key in values if fields[key].type == PluginSettingType.secret
)
if secret_keys:
raise ExtensionError(
"PLUGIN_SETTINGS_FIELD_INVALID",
"Secret fields must use the dedicated Secret endpoint.",
details={"plugin_id": plugin_id, "fields": secret_keys},
)
for key, value in values.items():
_validate_setting_value(fields[key], value)
with self._lock:
data = self._read()
entry = self._entry(data, plugin_id, create=True)
current = entry.get("values", {})
if not isinstance(current, dict):
raise self._storage_format_error(plugin_id)
entry["values"] = current
current.update(values)
effective = {
field.key: field.default
for field in definition.fields
if field.type != PluginSettingType.secret and field.default is not None
}
effective.update(current)
missing = [
field.key
for field in definition.fields
if field.required
and field.type != PluginSettingType.secret
and field.key not in effective
]
if missing:
raise ExtensionError(
"PLUGIN_SETTINGS_FIELD_INVALID",
"Required Plugin settings are missing.",
details={"plugin_id": plugin_id, "fields": missing},
)
entry["schema_version"] = definition.schema_version
self._write(data)
return self.get(plugin_id, definition)
def put_secret(
self,
plugin_id: str,
definition: PluginSettingsDefinition,
key: str,
secret: str,
) -> PluginSecretStatus:
_secret_field(definition, plugin_id, key)
if not secret:
raise ExtensionError(
"PLUGIN_SECRET_VALUE_INVALID",
"Plugin secret cannot be empty.",
details={"plugin_id": plugin_id, "key": key},
)
if len(secret.encode("utf-8")) > 64 * 1024:
raise ExtensionError(
"PLUGIN_SECRET_VALUE_INVALID",
"Plugin secret exceeds the 64 KiB limit.",
details={"plugin_id": plugin_id, "key": key},
)
reference = _secret_reference(plugin_id, key)
with self._lock:
data = self._read()
entry = self._entry(data, plugin_id, create=True)
refs = entry.get("secret_refs", {})
if not isinstance(refs, dict):
raise self._storage_format_error(plugin_id)
self._validate_secret_refs(plugin_id, refs)
entry["secret_refs"] = refs
try:
previous = self.credentials.resolve(reference)
self.credentials.put(reference, secret)
except CredentialStoreError as exc:
raise ExtensionError(
"PLUGIN_SECRET_STORE_ERROR", str(exc), status_code=500
) from exc
refs[key] = reference
entry["schema_version"] = definition.schema_version
try:
self._write(data)
except ExtensionError:
# 普通设置落盘失败时恢复凭据旧值,避免产生不可达的新 Secret。
try:
if previous is None:
self.credentials.delete(reference)
else:
self.credentials.put(reference, previous)
except CredentialStoreError:
pass
raise
return PluginSecretStatus(plugin_id=plugin_id, key=key, configured=True)
def delete_secret(
self,
plugin_id: str,
definition: PluginSettingsDefinition,
key: str,
) -> PluginSecretStatus:
_secret_field(definition, plugin_id, key)
with self._lock:
data = self._read()
entry = self._entry(data, plugin_id)
refs = entry.get("secret_refs", {})
if not isinstance(refs, dict):
raise self._storage_format_error(plugin_id)
self._validate_secret_refs(plugin_id, refs)
reference = _secret_reference(plugin_id, key)
had_reference = refs.pop(key, None) is not None
if plugin_id in data and had_reference:
self._write(data)
try:
self.credentials.delete(reference)
except CredentialStoreError as exc:
if had_reference:
refs[key] = reference
try:
self._write(data)
except ExtensionError as rollback_exc:
raise ExtensionError(
"PLUGIN_STORAGE_ERROR",
"Plugin Secret deletion failed and its reference could not be restored.",
status_code=500,
details={"plugin_id": plugin_id, "key": key},
) from rollback_exc
raise ExtensionError(
"PLUGIN_SECRET_STORE_ERROR", str(exc), status_code=500
) from exc
return PluginSecretStatus(plugin_id=plugin_id, key=key, configured=False)
def resolve_secret(
self, plugin_id: str, definition: PluginSettingsDefinition, key: str
) -> str | None:
_secret_field(definition, plugin_id, key)
with self._lock:
entry = self._entry(self._read(), plugin_id)
refs = entry.get("secret_refs", {})
if not isinstance(refs, dict):
raise self._storage_format_error(plugin_id)
reference = self._validate_secret_refs(plugin_id, refs).get(key)
try:
return self.credentials.resolve(reference) if isinstance(reference, str) else None
except CredentialStoreError as exc:
raise ExtensionError(
"PLUGIN_SECRET_STORE_ERROR", str(exc), status_code=500
) from exc
def remove_plugin(self, plugin_id: str) -> None:
with self._lock:
data = self._read()
entry = data.pop(plugin_id, None)
references: list[str] = []
if entry is not None and not isinstance(entry, dict):
raise self._storage_format_error(plugin_id)
if entry is not None:
refs = entry.get("secret_refs", {})
if not isinstance(refs, dict):
raise self._storage_format_error(plugin_id)
references = list(self._validate_secret_refs(plugin_id, refs).values())
if entry is not None:
self._write(data)
try:
self.credentials.delete_many(references)
except CredentialStoreError as exc:
if entry is not None:
data[plugin_id] = entry
try:
self._write(data)
except ExtensionError as rollback_exc:
raise ExtensionError(
"PLUGIN_STORAGE_ERROR",
"Plugin uninstall failed and its Settings namespace could not be restored.",
status_code=500,
details={"plugin_id": plugin_id},
) from rollback_exc
raise ExtensionError(
"PLUGIN_SECRET_STORE_ERROR", str(exc), status_code=500
) from exc
def _validate_secret_refs(
self, plugin_id: str, refs: dict[Any, Any]
) -> dict[str, str]:
validated: dict[str, str] = {}
for key, reference in refs.items():
if (
not isinstance(key, str)
or not _SETTING_KEY.fullmatch(key)
or not isinstance(reference, str)
or reference != _secret_reference(plugin_id, key)
):
raise self._storage_format_error(plugin_id)
validated[key] = reference
return validated
def _has_secret(self, reference: str) -> bool:
try:
return self.credentials.has(reference)
except CredentialStoreError as exc:
raise ExtensionError(
"PLUGIN_SECRET_STORE_ERROR", str(exc), status_code=500
) from exc
@staticmethod
def _storage_format_error(plugin_id: str) -> ExtensionError:
return ExtensionError(
"PLUGIN_STORAGE_ERROR",
"Plugin settings namespace has an invalid format.",
status_code=500,
details={"plugin_id": plugin_id},
)
def _entry(
self,
data: dict[str, dict[str, Any]],
plugin_id: str,
*,
create: bool = False,
) -> dict[str, Any]:
entry = data.get(plugin_id)
if entry is None:
if create:
data[plugin_id] = {}
return data[plugin_id]
return {}
if not isinstance(entry, dict):
raise self._storage_format_error(plugin_id)
return entry
def _read(self) -> dict[str, dict[str, Any]]:
path = self._path()
if not path.exists():
return {}
try:
value = json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError) as exc:
raise ExtensionError(
"PLUGIN_STORAGE_ERROR",
"Plugin settings storage cannot be loaded.",
status_code=500,
) from exc
if not isinstance(value, dict):
raise ExtensionError(
"PLUGIN_STORAGE_ERROR",
"Plugin settings storage has an invalid format.",
status_code=500,
)
return value
def _write(self, value: dict[str, dict[str, Any]]) -> None:
path = self._path()
temporary = path.with_suffix(".tmp")
try:
path.parent.mkdir(parents=True, exist_ok=True)
temporary.write_text(
json.dumps(value, ensure_ascii=False, sort_keys=True),
encoding="utf-8",
)
temporary.replace(path)
except OSError as exc:
try:
temporary.unlink(missing_ok=True)
except OSError:
pass
raise ExtensionError(
"PLUGIN_STORAGE_ERROR",
"Plugin settings storage cannot be written.",
status_code=500,
) from exc
def validate_settings_definition(
plugin_id: str, definition: PluginSettingsDefinition
) -> None:
if not _CONTRIBUTION_ID.fullmatch(definition.section_id):
raise _settings_schema_error(plugin_id, "Settings section id is invalid.")
if not definition.section_id.startswith(f"{plugin_id}."):
raise _settings_schema_error(
plugin_id, "Settings section id must use the Plugin namespace."
)
keys: set[str] = set()
for field in definition.fields:
if not _SETTING_KEY.fullmatch(field.key) or field.key in keys:
raise _settings_schema_error(plugin_id, f"Invalid or duplicate setting key: {field.key}")
keys.add(field.key)
if field.type == PluginSettingType.select and not field.options:
raise _settings_schema_error(plugin_id, f"Select setting requires options: {field.key}")
if field.type != PluginSettingType.select and field.options:
raise _settings_schema_error(plugin_id, f"Only select settings accept options: {field.key}")
if field.type != PluginSettingType.number and (
field.minimum is not None or field.maximum is not None
):
raise _settings_schema_error(plugin_id, f"Only number settings accept bounds: {field.key}")
if any(
bound is not None and not math.isfinite(bound)
for bound in (field.minimum, field.maximum)
):
raise _settings_schema_error(
plugin_id, f"Number setting bounds must be finite: {field.key}"
)
if field.minimum is not None and field.maximum is not None and field.minimum > field.maximum:
raise _settings_schema_error(plugin_id, f"Setting bounds are reversed: {field.key}")
if field.type == PluginSettingType.secret and field.default is not None:
raise _settings_schema_error(plugin_id, f"Secret settings cannot declare defaults: {field.key}")
if field.default is not None:
try:
_validate_setting_value(field, field.default)
except ExtensionError as exc:
raise _settings_schema_error(plugin_id, exc.message) from exc
def validate_command_spec(plugin_id: str, spec: PluginCommandSpec) -> None:
if not _CONTRIBUTION_ID.fullmatch(spec.command_id) or not spec.command_id.startswith(
f"{plugin_id}."
):
raise ExtensionError(
"PLUGIN_COMMAND_INVALID",
"Plugin command id must be valid and use the Plugin namespace.",
details={"plugin_id": plugin_id, "command_id": spec.command_id},
)
if not spec.locations:
raise ExtensionError(
"PLUGIN_COMMAND_INVALID",
"Plugin command must declare at least one location.",
details={"command_id": spec.command_id},
)
if len(spec.locations) != len(set(spec.locations)):
raise ExtensionError("PLUGIN_COMMAND_INVALID", "Plugin command locations must be unique.")
if len(spec.when) != len(set(spec.when)) or len(spec.context) != len(set(spec.context)):
raise ExtensionError(
"PLUGIN_COMMAND_INVALID",
"Plugin command when/context entries must be unique.",
)
if len(spec.secrets) != len(set(spec.secrets)):
raise ExtensionError(
"PLUGIN_COMMAND_INVALID",
"Plugin command Secret entries must be unique.",
details={"command_id": spec.command_id},
)
unknown_when = sorted(set(spec.when) - _WHEN_TOKENS)
if unknown_when:
raise ExtensionError(
"PLUGIN_COMMAND_INVALID",
"Plugin command declares unsupported when tokens.",
details={"command_id": spec.command_id, "when": unknown_when},
)
required_context = {_WHEN_CONTEXT[token] for token in spec.when}
if not required_context.issubset(set(spec.context)):
raise ExtensionError(
"PLUGIN_COMMAND_INVALID",
"Plugin command context must include every field required by when.",
details={"command_id": spec.command_id},
)
if not set(spec.context).issubset(_CONTEXT_KEYS):
raise ExtensionError("PLUGIN_COMMAND_INVALID", "Plugin command context is invalid.")
if spec.icon and spec.icon not in _HOST_ICONS:
raise ExtensionError(
"PLUGIN_COMMAND_INVALID",
"Plugin command icon is not a supported Host icon.",
details={"command_id": spec.command_id, "icon": spec.icon},
)
if spec.parameters.get("type", "object") != "object":
raise ExtensionError("PLUGIN_COMMAND_INVALID", "Command parameters must be an object schema.")
try:
Draft202012Validator.check_schema(spec.parameters)
reject_external_schema_references(spec.parameters)
except (SchemaReferenceError, SchemaError) as exc:
message = exc.message if isinstance(exc, SchemaError) else str(exc)
raise ExtensionError(
"PLUGIN_COMMAND_INVALID",
f"Plugin command parameters contain invalid JSON Schema: {message}",
) from exc
def _validate_setting_value(field: PluginSettingField, value: Any) -> None:
valid = False
if field.type == PluginSettingType.string:
valid = isinstance(value, str) and len(value.encode("utf-8")) <= 64 * 1024
elif field.type == PluginSettingType.number:
valid = (
(isinstance(value, int) and not isinstance(value, bool))
or (isinstance(value, float) and math.isfinite(value))
)
elif field.type == PluginSettingType.boolean:
valid = isinstance(value, bool)
elif field.type == PluginSettingType.select:
valid = isinstance(value, str) and value in field.options
if not valid:
raise ExtensionError(
"PLUGIN_SETTINGS_FIELD_INVALID",
f"Plugin setting has an invalid value: {field.key}",
details={"key": field.key},
)
if field.type == PluginSettingType.number:
if field.minimum is not None and value < field.minimum:
raise ExtensionError(
"PLUGIN_SETTINGS_FIELD_INVALID",
f"Plugin setting is below its minimum: {field.key}",
details={"key": field.key, "minimum": field.minimum},
)
if field.maximum is not None and value > field.maximum:
raise ExtensionError(
"PLUGIN_SETTINGS_FIELD_INVALID",
f"Plugin setting is above its maximum: {field.key}",
details={"key": field.key, "maximum": field.maximum},
)
def _secret_field(
definition: PluginSettingsDefinition, plugin_id: str, key: str
) -> PluginSettingField:
field = next((item for item in definition.fields if item.key == key), None)
if field is None or field.type != PluginSettingType.secret:
raise ExtensionError(
"PLUGIN_SECRET_FIELD_NOT_FOUND",
f"Plugin secret field does not exist: {key}",
status_code=404,
details={"plugin_id": plugin_id, "key": key},
)
return field
def _secret_reference(plugin_id: str, key: str) -> str:
digest = hashlib.sha256(f"{plugin_id}\0{key}".encode("utf-8")).hexdigest()
return f"plugin.{digest}"
def _settings_schema_error(plugin_id: str, message: str) -> ExtensionError:
return ExtensionError(
"PLUGIN_SETTINGS_SCHEMA_INVALID",
message,
details={"plugin_id": plugin_id},
)
+21
View File
@@ -0,0 +1,21 @@
from __future__ import annotations
from typing import Any
class ExtensionError(RuntimeError):
"""Extension Core 对 API 暴露的稳定领域错误。"""
def __init__(
self,
code: str,
message: str,
*,
status_code: int = 422,
details: dict[str, Any] | None = None,
) -> None:
super().__init__(message)
self.code = code
self.message = message
self.status_code = status_code
self.details = details or {}
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+725 -78
View File
@@ -1,21 +1,44 @@
from __future__ import annotations
import re
import threading
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Literal
from uuid import uuid4
import yaml
from jsonschema import Draft202012Validator
from jsonschema.exceptions import SchemaError
from pydantic import BaseModel, ConfigDict, Field, ValidationError, create_model
from jsonschema.exceptions import (
SchemaError,
ValidationError as JsonSchemaValidationError,
)
from pydantic import (
BaseModel,
ConfigDict,
Field,
TypeAdapter,
ValidationError,
create_model,
)
from app.agent.tools import ToolExecutionContext, ToolRegistry
from app.agent.tools import ToolExecutionContext, ToolExecutionError, ToolRegistry
from app.agent.permissions import KNOWN_PERMISSIONS
from app.contracts import (
ModelCapability,
Plugin,
PluginCommand,
PluginCommandContext,
PluginCommandEffect,
PluginNoEffect,
PluginNotificationEffect,
PluginCommandLocation,
PluginCommandResult,
PluginManifest,
PluginHostStatus,
PluginSecretStatus,
PluginSettingType,
PluginSettingsSchema,
PluginStatus,
RetrievalConfig,
Skill,
@@ -23,26 +46,26 @@ from app.contracts import (
SkillStatus,
ToolDefinition,
)
from app.extensions.contributions import (
CommandRegistry,
PluginCommandSpec,
PluginSecretResolver,
PluginSettingsDefinition,
PluginSettingsStore,
validate_command_spec,
validate_settings_definition,
)
from app.extensions.errors import ExtensionError
from app.extensions.mcp import McpBridge, McpBridgeError, McpDiscoveredTool
from app.providers.credentials import EncryptedCredentialStore
from app.schema_security import (
SchemaReferenceError,
reject_external_schema_references,
)
_EXTENSION_ID = re.compile(r"^[a-z0-9][a-z0-9._-]*$")
class ExtensionError(RuntimeError):
def __init__(
self,
code: str,
message: str,
*,
status_code: int = 422,
details: dict[str, Any] | None = None,
) -> None:
super().__init__(message)
self.code = code
self.message = message
self.status_code = status_code
self.details = details or {}
@dataclass(frozen=True, slots=True)
class AgentConfiguration:
skill_id: str
@@ -67,6 +90,7 @@ class SkillRuntime:
self._records: dict[str, _SkillRecord] = {}
def install(self, package_path: str | Path) -> Skill:
# TODO(extension): 将安装记录持久化,应用重启后从可信包目录恢复状态。
root = _package_dir(package_path)
raw = _read_yaml(root / "skill.yaml")
if "id" in raw and "skill_id" not in raw:
@@ -234,24 +258,70 @@ class DeclarativePluginHost:
return {"text": str(values.get("text", "")).upper()}
raise ExtensionError("PLUGIN_HANDLER_UNSUPPORTED", f"Unsupported handler: {handler}")
async def execute_command(
self,
handler: str,
arguments: dict[str, Any],
context: dict[str, Any],
settings: dict[str, Any],
resolve_secret: PluginSecretResolver,
) -> PluginCommandEffect:
"""执行宿主内置的白名单 Command handler,不导入 Plugin Python 代码。"""
if handler == "echo":
message = str(arguments.get("message", context.get("selection", "")))
if not message:
return PluginNoEffect()
return PluginNotificationEffect(
payload={"level": "info", "message": message},
)
if handler == "uppercase_selection":
text = str(arguments.get("text", context.get("selection", "")))
limit = int(settings.get("result_limit", 100))
return PluginNotificationEffect(
payload={"level": "success", "message": text[:limit].upper()},
)
raise ExtensionError(
"PLUGIN_HANDLER_UNSUPPORTED", f"Unsupported command handler: {handler}"
)
@dataclass(slots=True)
class _PluginRecord:
plugin: Plugin
tools: list[DeclarativeToolSpec]
commands: list[PluginCommandSpec]
settings_definition: PluginSettingsDefinition | None
package_path: Path
registered_tools: list[str]
registered_commands: list[str]
mcp_remote_names: dict[str, str]
mcp_command_schemas: dict[str, dict[str, Any]]
class PluginRuntime:
"""Plugin Manifest、生命周期及 Tool Contribution 注册。"""
def __init__(self, tools: ToolRegistry, host: DeclarativePluginHost | None = None) -> None:
def __init__(
self,
tools: ToolRegistry,
host: DeclarativePluginHost | None = None,
mcp_bridge: McpBridge | None = None,
credentials: EncryptedCredentialStore | None = None,
*,
allow_unsandboxed_mcp: bool = False,
) -> None:
self.registry = tools
self.host = host or DeclarativePluginHost()
self.mcp = mcp_bridge or McpBridge()
self.commands = CommandRegistry()
self.settings = PluginSettingsStore(credentials or EncryptedCredentialStore())
self.allow_unsandboxed_mcp = allow_unsandboxed_mcp
self._records: dict[str, _PluginRecord] = {}
self._lock = threading.RLock()
def install(self, package_path: str | Path) -> Plugin:
# 安装阶段只读取清单;MCP 子进程必须在权限授予后的 enable 阶段启动。
root = _package_dir(package_path)
raw = _read_yaml(root / "plugin.yaml")
if "id" in raw and "plugin_id" not in raw:
@@ -269,15 +339,19 @@ class PluginRuntime:
status_code=409,
)
specs = self._load_tools(root)
declared = set(manifest.contributes.tools)
actual = {spec.name for spec in specs}
if declared != actual:
raise ExtensionError(
"PLUGIN_CONTRIBUTION_INVALID",
"plugin.yaml tool contributions must exactly match tools.yaml",
details={"declared": sorted(declared), "actual": sorted(actual)},
)
_validate_backend(manifest)
specs = [] if manifest.backend.type == "mcp" else self._load_tools(root)
command_specs = self._load_commands(root)
settings_definition = self._load_settings(root)
if manifest.backend.type != "mcp":
declared = set(manifest.contributes.tools)
actual = {spec.name for spec in specs}
if declared != actual:
raise ExtensionError(
"PLUGIN_CONTRIBUTION_INVALID",
"plugin.yaml tool contributions must exactly match tools.yaml",
details={"declared": sorted(declared), "actual": sorted(actual)},
)
for spec in specs:
_validate_id("tool", spec.name)
_validate_tool_schema(spec)
@@ -287,6 +361,89 @@ class PluginRuntime:
f"Tool permission is not declared by Plugin: {spec.permission}",
details={"tool": spec.name, "permission": spec.permission},
)
declared_commands = set(manifest.contributes.commands)
actual_commands = {spec.command_id for spec in command_specs}
if (
declared_commands != actual_commands
or len(manifest.contributes.commands) != len(declared_commands)
or len(command_specs) != len(actual_commands)
):
raise ExtensionError(
"PLUGIN_CONTRIBUTION_INVALID",
"plugin.yaml command contributions must exactly match commands.yaml",
details={
"declared": sorted(declared_commands),
"actual": sorted(actual_commands),
},
)
for spec in command_specs:
validate_command_spec(manifest.plugin_id, spec)
if spec.permission and spec.permission not in manifest.permissions:
raise ExtensionError(
"PLUGIN_PERMISSION_UNDECLARED",
f"Command permission is not declared by Plugin: {spec.permission}",
details={"command": spec.command_id, "permission": spec.permission},
)
declared_sections = set(manifest.contributes.settings_sections)
actual_sections = (
{settings_definition.section_id} if settings_definition is not None else set()
)
if (
declared_sections != actual_sections
or len(manifest.contributes.settings_sections) != len(declared_sections)
):
raise ExtensionError(
"PLUGIN_CONTRIBUTION_INVALID",
"plugin.yaml settings contributions must exactly match settings.yaml",
details={
"declared": sorted(declared_sections),
"actual": sorted(actual_sections),
},
)
if settings_definition is not None:
validate_settings_definition(manifest.plugin_id, settings_definition)
secret_fields = (
{
field.key
for field in settings_definition.fields
if field.type == PluginSettingType.secret
}
if settings_definition is not None
else set()
)
for spec in command_specs:
unknown_secrets = sorted(set(spec.secrets) - secret_fields)
if unknown_secrets:
raise ExtensionError(
"PLUGIN_COMMAND_INVALID",
"Plugin command references undeclared Secret settings.",
details={
"command_id": spec.command_id,
"secrets": unknown_secrets,
},
)
if spec.secrets and "secrets.use" not in manifest.permissions:
raise ExtensionError(
"PLUGIN_PERMISSION_UNDECLARED",
"Commands using Secret settings require the secrets.use permission.",
details={"command_id": spec.command_id},
)
if spec.mcp_tool is not None:
_validate_id("MCP command target", spec.mcp_tool)
if manifest.backend.type != "mcp" or not spec.mcp_tool.startswith(
f"{manifest.plugin_id}."
):
raise ExtensionError(
"PLUGIN_COMMAND_INVALID",
"MCP Command target must use the current Plugin namespace.",
details={"command_id": spec.command_id},
)
if spec.mcp_tool in manifest.contributes.tools:
raise ExtensionError(
"PLUGIN_COMMAND_INVALID",
"MCP Command target cannot also be exposed as an Agent Tool.",
details={"command_id": spec.command_id},
)
record = _PluginRecord(
plugin=Plugin(
@@ -298,8 +455,13 @@ class PluginRuntime:
),
),
tools=specs,
commands=command_specs,
settings_definition=settings_definition,
package_path=root,
registered_tools=[],
registered_commands=[],
mcp_remote_names={},
mcp_command_schemas={},
)
self._records[manifest.plugin_id] = record
return record.plugin.model_copy(deep=True)
@@ -311,17 +473,14 @@ class PluginRuntime:
return self._record(plugin_id).plugin.model_copy(deep=True)
def enable(self, plugin_id: str) -> Plugin:
# Host 启动和 Tool 批量注册必须串行,避免并发 enable 产生重复进程或半注册状态。
with self._lock:
return self._enable(plugin_id)
def _enable(self, plugin_id: str) -> Plugin:
record = self._record(plugin_id)
if record.plugin.enabled:
return record.plugin.model_copy(deep=True)
if record.plugin.manifest.backend.type == "mcp":
record.plugin.status = PluginStatus.dependency_missing
raise ExtensionError(
"PLUGIN_HOST_UNAVAILABLE",
"MCP Plugin Host is reserved for the second development phase.",
status_code=501,
details={"plugin_id": plugin_id, "backend": "mcp"},
)
missing_grants = sorted(
set(record.plugin.manifest.permissions) - set(record.plugin.granted_permissions)
)
@@ -333,7 +492,20 @@ class PluginRuntime:
status_code=409,
details={"plugin_id": plugin_id, "permissions": missing_grants},
)
conflicts = [spec.name for spec in record.tools if self.registry.contains(spec.name)]
if (
record.plugin.manifest.backend.type == "mcp"
and not self.allow_unsandboxed_mcp
):
raise ExtensionError(
"MCP_TRUST_APPROVAL_REQUIRED",
"Unsandboxed MCP Hosts are disabled outside development mode.",
status_code=403,
details={"plugin_id": plugin_id},
)
if record.settings_definition is not None:
self.settings.runtime_values(plugin_id, record.settings_definition)
declared_tools = list(record.plugin.manifest.contributes.tools)
conflicts = [name for name in declared_tools if self.registry.contains(name)]
if conflicts:
raise ExtensionError(
"PLUGIN_TOOL_CONFLICT",
@@ -341,43 +513,229 @@ class PluginRuntime:
status_code=409,
details={"plugin_id": plugin_id, "tools": conflicts},
)
command_conflicts = [
spec.command_id for spec in record.commands if self.commands.contains(spec.command_id)
]
if command_conflicts:
raise ExtensionError(
"PLUGIN_COMMAND_CONFLICT",
"Plugin commands are already registered.",
status_code=409,
details={"plugin_id": plugin_id, "commands": command_conflicts},
)
record.plugin.status = PluginStatus.starting
try:
for spec in record.tools:
arguments_model = _arguments_model(spec)
if record.plugin.manifest.backend.type == "mcp":
discovered = self._start_mcp(record)
actual = {item.definition.name for item in discovered}
declared = set(declared_tools)
command_targets = {
spec.mcp_tool for spec in record.commands if spec.mcp_tool is not None
}
expected = declared | command_targets
if actual != expected:
raise ExtensionError(
"PLUGIN_CONTRIBUTION_INVALID",
"Discovered MCP tools must exactly match Tool and Command targets.",
details={"declared": sorted(expected), "actual": sorted(actual)},
)
for item in discovered:
if item.definition.name in declared:
self._register_mcp_tool(record, item)
else:
record.mcp_remote_names[item.definition.name] = item.remote_name
record.mcp_command_schemas[item.definition.name] = (
item.definition.parameters
)
for spec in (
command
for command in record.commands
if command.mcp_tool == item.definition.name
):
_validate_mcp_command_target_schema(
item.definition.parameters,
spec.command_id,
)
else:
for spec in record.tools:
arguments_model = _arguments_model(spec)
async def executor(
arguments: BaseModel,
context: ToolExecutionContext,
_handler: str = spec.handler,
) -> Any:
return await self.host.execute(_handler, arguments, context)
async def executor(
arguments: BaseModel,
context: ToolExecutionContext,
_handler: str = spec.handler,
) -> Any:
return await self.host.execute(_handler, arguments, context)
self.registry.register(
ToolDefinition(
name=spec.name,
description=spec.description,
parameters=spec.parameters,
permission=spec.permission,
source="plugin",
),
arguments_model,
executor,
)
record.registered_tools.append(spec.name)
self.registry.register(
ToolDefinition(
name=spec.name,
description=spec.description,
parameters=spec.parameters,
permission=spec.permission,
source="plugin",
),
arguments_model,
executor,
)
record.registered_tools.append(spec.name)
for spec in record.commands:
async def command_executor(
arguments: dict[str, Any],
context: dict[str, Any],
_spec: PluginCommandSpec = spec,
_record: _PluginRecord = record,
) -> PluginCommandEffect:
if (
not _record.plugin.enabled
or _record.plugin.status != PluginStatus.ready
):
raise ExtensionError(
"PLUGIN_COMMAND_NOT_FOUND",
"Plugin command is not available while its Plugin is inactive.",
status_code=404,
details={"command_id": _spec.command_id},
)
settings = (
self.settings.runtime_values(
_record.plugin.manifest.plugin_id,
_record.settings_definition,
)
if _record.settings_definition is not None
else {}
)
def resolve_secret(key: str) -> str | None:
if key not in _spec.secrets:
raise ExtensionError(
"PLUGIN_SECRET_ACCESS_DENIED",
"Command cannot access an undeclared Plugin Secret.",
status_code=403,
details={
"command_id": _spec.command_id,
"key": key,
},
)
if "secrets.use" not in _record.plugin.granted_permissions:
raise ExtensionError(
"PLUGIN_SECRET_ACCESS_DENIED",
"Plugin no longer has permission to access Secret settings.",
status_code=403,
details={"command_id": _spec.command_id, "key": key},
)
if _record.settings_definition is None:
return None
value = self.settings.resolve_secret(
_record.plugin.manifest.plugin_id,
_record.settings_definition,
key,
)
field = next(
item
for item in _record.settings_definition.fields
if item.key == key
)
if field.required and value is None:
raise ExtensionError(
"PLUGIN_SECRET_REQUIRED",
"A required Plugin Secret has not been configured.",
status_code=409,
details={"command_id": _spec.command_id, "key": key},
)
return value
if _spec.mcp_tool is not None:
remote_name = _record.mcp_remote_names[_spec.mcp_tool]
secret_values = {
key: value
for key in _spec.secrets
if (value := resolve_secret(key)) is not None
}
envelope = _mcp_command_envelope(
_spec,
arguments=arguments,
context=context,
settings=settings,
secrets=secret_values,
)
_validate_mcp_command_envelope(
_record.mcp_command_schemas[_spec.mcp_tool],
envelope,
_spec.command_id,
)
try:
effect = await self.mcp.call_tool(
_record.plugin.manifest.plugin_id,
remote_name,
envelope,
request_id=f"command:{uuid4().hex}",
)
except ToolExecutionError as exc:
raise ExtensionError(
exc.code,
"MCP Command target execution failed.",
status_code=502,
details={"command_id": _spec.command_id},
) from exc
try:
return TypeAdapter(PluginCommandEffect).validate_python(effect)
except ValidationError as exc:
raise ExtensionError(
"PLUGIN_COMMAND_RESULT_INVALID",
"MCP Command target returned an invalid effect.",
status_code=502,
details={"command_id": _spec.command_id},
) from exc
return await self.host.execute_command(
_spec.handler,
arguments,
context,
settings,
resolve_secret,
)
self.commands.register(plugin_id, spec, command_executor)
record.registered_commands.append(spec.command_id)
except Exception as exc:
# 注册过程必须具备回滚语义,防止半启用插件污染全局工具表。
for name in record.registered_tools:
self.registry.unregister(name)
record.registered_tools.clear()
for command_id in record.registered_commands:
self.commands.unregister(command_id)
record.registered_commands.clear()
record.mcp_remote_names.clear()
record.mcp_command_schemas.clear()
self.mcp.stop(plugin_id)
record.plugin.status = PluginStatus.error
record.plugin.error_message = str(exc)
raise
record.plugin.error_message = _safe_extension_message(exc)
if isinstance(exc, ExtensionError):
raise
if isinstance(exc, McpBridgeError):
raise ExtensionError(
exc.code,
exc.message,
status_code=exc.status_code,
details={"plugin_id": plugin_id},
) from exc
raise ExtensionError(
"PLUGIN_HOST_START_FAILED",
record.plugin.error_message,
status_code=503,
details={"plugin_id": plugin_id},
) from exc
record.plugin.enabled = True
record.plugin.status = PluginStatus.ready
record.plugin.error_message = None
return record.plugin.model_copy(deep=True)
def set_permissions(self, plugin_id: str, permissions: list[str]) -> Plugin:
with self._lock:
return self._set_permissions(plugin_id, permissions)
def _set_permissions(self, plugin_id: str, permissions: list[str]) -> Plugin:
record = self._record(plugin_id)
requested = set(permissions)
declared = set(record.plugin.manifest.permissions)
@@ -399,15 +757,174 @@ class PluginRuntime:
return record.plugin.model_copy(deep=True)
def disable(self, plugin_id: str) -> Plugin:
with self._lock:
return self._disable(plugin_id)
def _disable(self, plugin_id: str) -> Plugin:
record = self._record(plugin_id)
for name in record.registered_tools:
self.registry.unregister(name)
record.registered_tools.clear()
for command_id in record.registered_commands:
self.commands.unregister(command_id)
record.registered_commands.clear()
record.mcp_remote_names.clear()
record.mcp_command_schemas.clear()
if record.plugin.manifest.backend.type == "mcp":
self.mcp.stop(plugin_id)
record.plugin.enabled = False
record.plugin.status = PluginStatus.disabled
return record.plugin.model_copy(deep=True)
def get_host_status(self, plugin_id: str) -> PluginHostStatus:
record = self._record(plugin_id)
return self.mcp.status(plugin_id, record.plugin.manifest.backend)
def list_commands(
self, location: PluginCommandLocation | None = None
) -> list[PluginCommand]:
return self.commands.list(location)
async def execute_command(
self,
command_id: str,
arguments: dict[str, Any],
context: PluginCommandContext,
) -> PluginCommandResult:
return await self.commands.execute(command_id, arguments, context)
def get_settings(self, plugin_id: str) -> PluginSettingsSchema:
record = self._record(plugin_id)
definition = self._settings_definition(record)
return self.settings.get(plugin_id, definition)
def update_settings(
self, plugin_id: str, schema_version: int, values: dict[str, Any]
) -> PluginSettingsSchema:
record = self._record(plugin_id)
definition = self._settings_definition(record)
return self.settings.update(plugin_id, definition, schema_version, values)
def put_setting_secret(
self, plugin_id: str, key: str, secret: str
) -> PluginSecretStatus:
record = self._record(plugin_id)
definition = self._settings_definition(record)
return self.settings.put_secret(plugin_id, definition, key, secret)
def delete_setting_secret(self, plugin_id: str, key: str) -> PluginSecretStatus:
record = self._record(plugin_id)
definition = self._settings_definition(record)
return self.settings.delete_secret(plugin_id, definition, key)
def restart_host(self, plugin_id: str) -> PluginHostStatus:
with self._lock:
return self._restart_host(plugin_id)
def _restart_host(self, plugin_id: str) -> PluginHostStatus:
record = self._record(plugin_id)
if record.plugin.manifest.backend.type != "mcp":
raise ExtensionError(
"PLUGIN_HOST_UNAVAILABLE",
"Plugin does not use an MCP Host.",
status_code=409,
details={"plugin_id": plugin_id},
)
if record.plugin.status in {
PluginStatus.installed,
PluginStatus.disabled,
PluginStatus.permission_required,
}:
raise ExtensionError(
"PLUGIN_HOST_UNAVAILABLE",
"Disabled or inactive MCP Plugins must be started with Enable.",
status_code=409,
details={"plugin_id": plugin_id, "status": record.plugin.status.value},
)
for name in record.registered_tools:
self.registry.unregister(name)
record.registered_tools.clear()
for command_id in record.registered_commands:
self.commands.unregister(command_id)
record.registered_commands.clear()
record.mcp_remote_names.clear()
record.mcp_command_schemas.clear()
self.mcp.stop(plugin_id)
record.plugin.enabled = False
record.plugin.status = PluginStatus.installed
record.plugin.error_message = None
self.enable(plugin_id)
return self.get_host_status(plugin_id)
def shutdown(self) -> None:
"""关闭所有隔离 Host;用于 FastAPI lifespan 和测试清理。"""
with self._lock:
for plugin_id, record in list(self._records.items()):
if record.plugin.manifest.backend.type == "mcp":
self.mcp.stop(plugin_id)
def _start_mcp(self, record: _PluginRecord) -> list[McpDiscoveredTool]:
manifest = record.plugin.manifest
return self.mcp.start(
manifest.plugin_id,
manifest.backend,
record.package_path,
manifest.permissions,
self._handle_mcp_unavailable,
)
def _register_mcp_tool(
self, record: _PluginRecord, discovered: McpDiscoveredTool
) -> None:
definition = discovered.definition
arguments_model = _arguments_model_from_schema(
definition.name, definition.parameters
)
plugin_id = record.plugin.manifest.plugin_id
remote_name = discovered.remote_name
async def executor(
arguments: BaseModel,
context: ToolExecutionContext,
) -> Any:
return await self.mcp.call_tool(
plugin_id,
remote_name,
# 省略的可选字段不能被补成 null;显式传入的 null 仍由
# model_fields_set 保留并交给 MCP Server。
arguments.model_dump(exclude_unset=True),
request_id=context.tool_call_id or f"{context.run_id}:{definition.name}",
)
self.registry.register(definition, arguments_model, executor)
record.registered_tools.append(definition.name)
record.mcp_remote_names[definition.name] = remote_name
def _handle_mcp_unavailable(self, plugin_id: str, message: str) -> None:
with self._lock:
record = self._records.get(plugin_id)
if record is None:
return
for name in record.registered_tools:
self.registry.unregister(name)
record.registered_tools.clear()
for command_id in record.registered_commands:
self.commands.unregister(command_id)
record.registered_commands.clear()
record.mcp_remote_names.clear()
record.mcp_command_schemas.clear()
record.plugin.enabled = False
record.plugin.status = PluginStatus.error
record.plugin.error_message = message
def uninstall(self, plugin_id: str, dependent_skills: list[str] | None = None) -> None:
with self._lock:
self._uninstall(plugin_id, dependent_skills)
def _uninstall(
self, plugin_id: str, dependent_skills: list[str] | None = None
) -> None:
record = self._record(plugin_id)
if dependent_skills:
raise ExtensionError(
@@ -416,8 +933,14 @@ class PluginRuntime:
status_code=409,
details={"plugin_id": plugin_id, "skills": dependent_skills},
)
is_mcp = record.plugin.manifest.backend.type == "mcp"
if record.plugin.enabled:
self.disable(plugin_id)
if is_mcp:
# stop 只结束本次进程并保留状态供故障诊断;真正卸载时必须连同
# 历史状态一起遗忘,避免同 ID 重装继承旧协商信息。
self.mcp.remove(plugin_id)
self.settings.remove_plugin(plugin_id)
del self._records[plugin_id]
def _record(self, plugin_id: str) -> _PluginRecord:
@@ -439,6 +962,52 @@ class PluginRuntime:
except ValidationError as exc:
raise _manifest_error("plugin tool", exc) from exc
@staticmethod
def _load_commands(root: Path) -> list[PluginCommandSpec]:
path = root / "commands.yaml"
if not path.exists():
return []
raw = _read_yaml(path)
items = raw.get("commands", [])
if not isinstance(items, list):
raise ExtensionError(
"EXTENSION_MANIFEST_INVALID",
"Invalid plugin command manifest: commands must be an array.",
)
try:
return [
PluginCommandSpec.model_validate(item)
for item in items
]
except ValidationError as exc:
raise _manifest_error("plugin command", exc) from exc
@staticmethod
def _load_settings(root: Path) -> PluginSettingsDefinition | None:
path = root / "settings.yaml"
if not path.exists():
return None
raw = _read_yaml(path)
try:
return PluginSettingsDefinition.model_validate(raw)
except ValidationError as exc:
raise ExtensionError(
"PLUGIN_SETTINGS_SCHEMA_INVALID",
"Invalid Plugin settings schema.",
details={"errors": exc.errors(include_url=False)},
) from exc
@staticmethod
def _settings_definition(record: _PluginRecord) -> PluginSettingsDefinition:
if record.settings_definition is None:
raise ExtensionError(
"PLUGIN_SETTINGS_NOT_FOUND",
"Plugin does not contribute a Settings section.",
status_code=404,
details={"plugin_id": record.plugin.manifest.plugin_id},
)
return record.settings_definition
def _package_dir(package_path: str | Path) -> Path:
root = Path(package_path).expanduser().resolve()
@@ -494,34 +1063,85 @@ def _manifest_error(kind: str, exc: ValidationError) -> ExtensionError:
def _arguments_model(spec: DeclarativeToolSpec) -> type[BaseModel]:
schema = spec.parameters or {"type": "object", "properties": {}}
return _arguments_model_from_schema(spec.name, schema)
def _mcp_command_envelope(
spec: PluginCommandSpec,
*,
arguments: dict[str, Any],
context: dict[str, Any],
settings: dict[str, Any],
secrets: dict[str, str],
) -> dict[str, Any]:
return {
"_notesagent": {
"command_id": spec.command_id,
"arguments": arguments,
"context": context,
"settings": settings,
"secrets": secrets,
}
}
def _validate_mcp_command_envelope(
schema: dict[str, Any],
envelope: dict[str, Any],
command_id: str,
) -> None:
"""执行前用目标 Tool Schema 校验包含真实业务数据的宿主信封。"""
try:
Draft202012Validator(schema).validate(envelope)
except JsonSchemaValidationError as exc:
raise ExtensionError(
"PLUGIN_COMMAND_TARGET_SCHEMA_MISMATCH",
"MCP Command envelope does not match the target inputSchema.",
status_code=502,
details={"command_id": command_id, "path": list(exc.path)},
) from exc
def _validate_mcp_command_target_schema(
schema: dict[str, Any], command_id: str
) -> None:
"""启用时只检查稳定信封入口,避免用伪造业务值误判合法 Schema。"""
properties = schema.get("properties")
envelope_schema = (
properties.get("_notesagent") if isinstance(properties, dict) else None
)
if not isinstance(envelope_schema, dict) or envelope_schema.get("type") != "object":
raise ExtensionError(
"PLUGIN_CONTRIBUTION_INVALID",
"MCP Command target inputSchema must directly declare "
"_notesagent with type object.",
details={"command_id": command_id},
)
def _arguments_model_from_schema(
tool_name: str, schema: dict[str, Any]
) -> type[BaseModel]:
if schema.get("type", "object") != "object":
raise ExtensionError("PLUGIN_TOOL_SCHEMA_INVALID", "Tool parameters must be an object schema.")
properties = schema.get("properties", {})
required = set(schema.get("required", []))
fields: dict[str, tuple[Any, Any]] = {}
types = {
"string": str,
"number": float,
"integer": int,
"boolean": bool,
"array": list[Any],
"object": dict[str, Any],
}
for name, field_schema in properties.items():
annotation = types.get(field_schema.get("type"), Any)
fields[name] = (annotation, ... if name in required else None)
model_name = "PluginArgs_" + re.sub(r"\W+", "_", spec.name)
return create_model(model_name, __config__=ConfigDict(extra="forbid"), **fields)
model_name = "PluginArgs_" + re.sub(r"\W+", "_", tool_name)
# 完整 JSON Schema 已在 ToolRegistry 中先行校验。参数载体不重复声明字段,
# 从而完整保留 model_dump、连字符键、联合类型和动态属性等合法 JSON 键值。
return create_model(model_name, __config__=ConfigDict(extra="allow"))
def _validate_tool_schema(spec: DeclarativeToolSpec) -> None:
schema = spec.parameters or {"type": "object", "properties": {}}
try:
Draft202012Validator.check_schema(schema)
except SchemaError as exc:
reject_external_schema_references(schema)
except (SchemaReferenceError, SchemaError) as exc:
message = exc.message if isinstance(exc, SchemaError) else str(exc)
raise ExtensionError(
"PLUGIN_TOOL_SCHEMA_INVALID",
f"Invalid JSON Schema for tool {spec.name}: {exc.message}",
f"Invalid JSON Schema for tool {spec.name}: {message}",
details={"tool": spec.name},
) from exc
if schema.get("type", "object") != "object" or not isinstance(
@@ -532,3 +1152,30 @@ def _validate_tool_schema(spec: DeclarativeToolSpec) -> None:
"Tool parameters must be an object schema with object properties.",
details={"tool": spec.name},
)
def _validate_backend(manifest: PluginManifest) -> None:
backend = manifest.backend
if backend.type == "mcp":
if backend.transport != "stdio":
raise ExtensionError(
"MCP_CAPABILITY_UNSUPPORTED",
"Phase C MCP Plugins must use stdio transport.",
status_code=501,
)
if not backend.command or not backend.command.strip():
raise ExtensionError(
"EXTENSION_MANIFEST_INVALID",
"MCP stdio backend requires a command.",
)
elif backend.command is not None or backend.args:
raise ExtensionError(
"EXTENSION_MANIFEST_INVALID",
"Only MCP stdio backends may declare command or args.",
)
def _safe_extension_message(exc: Exception) -> str:
if isinstance(exc, (ExtensionError, McpBridgeError)):
return exc.message
return f"Plugin Host operation failed: {type(exc).__name__}."
+13
View File
@@ -1,19 +1,32 @@
from contextlib import asynccontextmanager
from fastapi import FastAPI
from fastapi.exceptions import RequestValidationError
from fastapi.middleware.cors import CORSMiddleware
from starlette.exceptions import HTTPException as StarletteHttpException
from app.config import get_settings
from app.container import container
from app.errors import ApiError, api_error_handler, http_error_handler, validation_error_handler
from app.routes import router as api_router
from app.schemas import HealthResponse, ServiceStatusResponse
settings = get_settings()
@asynccontextmanager
async def lifespan(_: FastAPI):
yield
# 第三方 MCP Server 必须跟随 AI Core 退出,不能遗留孤儿进程。
container.plugins.shutdown()
container.mcp_servers.shutdown()
app = FastAPI(
title=settings.name,
version=settings.version,
description="AI 笔记软件的本地 AI Core 与 Agent Core 服务。",
lifespan=lifespan,
)
app.add_middleware(
+163
View File
@@ -0,0 +1,163 @@
"""Native Anthropic Messages protocol with incrementally decoded content blocks."""
import json
from contextlib import aclosing
from app.contracts import MessageRole, ModelEventType, ModelRequest
from app.providers.base import ProviderError, ProviderToolCall, ProviderTurn
from app.providers.http_base import (
UsageTracker, check_error, decode_tool_arguments, invalid_response, list_value,
object_value, string_value, token_count, truncated_stream,
)
from app.providers.openai_compatible import OpenAICompatibleProvider
from app.providers.tool_names import mapped_tool_names
class AnthropicMessagesProvider(OpenAICompatibleProvider):
stream_path = "/messages"
def _headers(self) -> dict[str, str]:
headers = super()._headers()
authorization = headers.pop("Authorization", None)
if authorization:
headers["x-api-key"] = authorization.removeprefix("Bearer ")
headers["anthropic-version"] = "2023-06-01"
return headers
def _payload(self, request: ModelRequest, *, stream: bool) -> dict[str, object]:
systems = [request.system] if request.system else []
messages = []
for message in request.messages:
if message.role == MessageRole.system:
systems.append(message.content)
continue
if message.role == MessageRole.tool:
if not message.tool_call_id:
raise ProviderError("PROVIDER_INVALID_REQUEST", "Tool result requires a call identifier.")
role = "user"
content = [{"type": "tool_result", "tool_use_id": message.tool_call_id, "content": message.content}]
else:
role = message.role.value
content = [{"type": "text", "text": message.content}] if message.content else []
content += [{"type": "tool_use", "id": call.tool_call_id, "name": call.name,
"input": call.arguments} for call in message.tool_calls]
if not content:
continue
if messages and messages[-1]["role"] == role:
messages[-1]["content"].extend(content)
else:
messages.append({"role": role, "content": content})
payload: dict[str, object] = {"model": request.model, "messages": messages,
"max_tokens": request.max_tokens or 4096, "stream": stream}
if systems:
payload["system"] = "\n\n".join(systems)
if request.tools:
payload["tools"] = [{"name": tool.name, "description": tool.description,
"input_schema": tool.parameters} for tool in request.tools]
if request.temperature is not None:
payload["temperature"] = request.temperature
if request.response_format is not None:
format_ = request.response_format
if format_.get("type") != "json_schema":
raise ProviderError("PROVIDER_INVALID_REQUEST", "Messages requires a JSON schema response format.")
schema = object_value(format_.get("json_schema"))
payload["output_config"] = {"format": {"type": "json_schema", "schema": object_value(schema.get("schema"))}}
return payload
@mapped_tool_names
async def complete(self, request: ModelRequest) -> ProviderTurn:
data = await self._request("POST", self.stream_path, json=self._payload(request, stream=False))
texts = []
calls = []
for raw in list_value(data.get("content")):
block = object_value(raw)
if block.get("type") == "text":
texts.append(string_value(block.get("text")))
elif block.get("type") == "tool_use":
calls.append(ProviderToolCall(
tool_call_id=string_value(block.get("id"), nonempty=True),
name=string_value(block.get("name"), nonempty=True),
arguments=decode_tool_arguments(block.get("input")),
))
return ProviderTurn(text="".join(texts) or None, tool_calls=calls,
**UsageTracker(cache_tokens=True).update(data.get("usage") or {}))
async def _events(self, request: ModelRequest):
blocks: dict[int, dict] = {}
usage = UsageTracker(cache_tokens=True)
started = False
async with aclosing(self._stream_json(self._payload(request, stream=True))) as chunks:
async for data in chunks:
kind = string_value(data.get("type"), nonempty=True)
if kind == "message_start":
if started:
raise invalid_response()
started = True
message = object_value(data.get("message"))
check_error(message)
if message.get("usage") is not None:
yield ModelEventType.usage, usage.update(message["usage"])
elif kind == "content_block_start":
index = token_count(data.get("index"))
if not started or index in blocks:
raise invalid_response()
block = dict(object_value(data.get("content_block")))
blocks[index] = block
block["closed"] = False
if block.get("type") == "tool_use":
block["id"] = string_value(block.get("id"), nonempty=True)
block["name"] = string_value(block.get("name"), nonempty=True)
block["arguments"] = ""
block["input"] = object_value(block.get("input", {}))
yield ModelEventType.tool_call_start, {"tool_call_id": block["id"], "name": block["name"]}
elif block.get("type") == "text" and block.get("text"):
yield ModelEventType.text_delta, {"text": string_value(block["text"])}
elif block.get("type") == "thinking" and block.get("thinking"):
yield ModelEventType.thinking_delta, {"text": string_value(block["thinking"])}
elif kind == "content_block_delta":
block = blocks.get(token_count(data.get("index")))
if block is None or block["closed"]:
raise invalid_response()
delta = object_value(data.get("delta"))
delta_type = delta.get("type")
if delta_type == "text_delta":
if block.get("type") != "text":
raise invalid_response()
yield ModelEventType.text_delta, {"text": string_value(delta.get("text"))}
elif delta_type == "thinking_delta":
if block.get("type") != "thinking":
raise invalid_response()
yield ModelEventType.thinking_delta, {"text": string_value(delta.get("thinking"))}
elif delta_type == "input_json_delta" and block.get("type") == "tool_use":
fragment = string_value(delta.get("partial_json"))
block["arguments"] += fragment
yield ModelEventType.tool_call_delta, {"tool_call_id": block["id"], "arguments_delta": fragment}
# Signatures and future delta types have no representation in ModelEvent.
elif kind == "content_block_stop":
block = blocks.get(token_count(data.get("index")))
if block is None or block["closed"]:
raise invalid_response()
block["closed"] = True
if block.get("type") == "tool_use":
if block["arguments"]:
decode_tool_arguments(block["arguments"])
else:
yield ModelEventType.tool_call_delta, {
"tool_call_id": block["id"], "arguments_delta": json.dumps(block["input"]),
}
yield ModelEventType.tool_call_end, {"tool_call_id": block["id"]}
elif kind == "message_delta":
if not started:
raise invalid_response()
object_value(data.get("delta"))
if data.get("usage") is not None:
yield ModelEventType.usage, usage.update(data["usage"])
elif kind == "message_stop":
if not started:
raise invalid_response()
if any(not block["closed"] for block in blocks.values()):
raise truncated_stream()
return
elif kind == "[DONE]":
raise truncated_stream()
raise truncated_stream()
+232 -2
View File
@@ -1,17 +1,247 @@
"""Provider 凭据解析及本地加密存储。"""
import json
import os
import re
from typing import Protocol
import threading
from pathlib import Path
from typing import ClassVar, Protocol
from cryptography.fernet import Fernet, InvalidToken
from app.config import get_settings
_CREDENTIAL_ID = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$")
_PLUGIN_CREDENTIAL_PREFIX = "plugin."
_MCP_CREDENTIAL_PREFIX = "mcp."
class CredentialStoreError(RuntimeError):
pass
class CredentialResolver(Protocol):
def resolve(self, credential_id: str | None) -> str | None: ...
def validate_provider_credential_id(credential_id: str | None) -> None:
"""阻止 Provider 和通用凭据 API 跨入 Plugin 私有命名空间。"""
if credential_id and credential_id.casefold().startswith(_PLUGIN_CREDENTIAL_PREFIX):
raise CredentialStoreError(
"Credential namespace is reserved for Plugin settings."
)
if credential_id and credential_id.casefold().startswith(_MCP_CREDENTIAL_PREFIX):
raise CredentialStoreError("Credential namespace is reserved for MCP settings.")
class EnvironmentCredentialResolver:
"""解析由桌面 Host 注入 Sidecar 进程的临时凭证上下文。"""
_development_aliases: ClassVar[dict[str, str]] = {
"openai": "OPENAI_API_KEY",
"deepseek": "DEEPSEEK_API_KEY",
}
def resolve(self, credential_id: str | None) -> str | None:
if not credential_id:
return None
normalized = re.sub(r"[^A-Za-z0-9]", "_", credential_id).upper()
return os.getenv(f"AINOTE_CREDENTIAL_{normalized}")
injected = os.getenv(f"AINOTE_CREDENTIAL_{normalized}")
if injected:
return injected
alias = self._development_aliases.get(credential_id.lower())
return os.getenv(alias) if alias else None
class EncryptedCredentialStore:
"""将本地开发凭据作为 Fernet 密文存储,Provider 使用时按 ID 解密。"""
# TODO(security): 桌面 Host 接入后将主密钥迁移到系统钥匙串/凭据保险库。
def __init__(self) -> None:
self._lock = threading.RLock()
@staticmethod
def _validate_id(credential_id: str) -> None:
if not _CREDENTIAL_ID.fullmatch(credential_id):
raise CredentialStoreError("Credential ID contains unsupported characters.")
@staticmethod
def _paths() -> tuple[Path, Path]:
directory = get_settings().data_dir / "credentials"
return directory / "master.key", directory / "credentials.json"
@staticmethod
def _restrict(path: Path, mode: int) -> None:
try:
path.chmod(mode)
except OSError:
pass
def _fernet(self) -> Fernet:
key_path, _ = self._paths()
environment_key = os.getenv("APP_CREDENTIAL_MASTER_KEY")
if environment_key:
try:
return Fernet(environment_key.encode("ascii"))
except (ValueError, UnicodeEncodeError) as exc:
raise CredentialStoreError(
"APP_CREDENTIAL_MASTER_KEY is invalid."
) from exc
key_path.parent.mkdir(parents=True, exist_ok=True)
self._restrict(key_path.parent, 0o700)
if not key_path.exists():
# 先写临时文件再原子替换,避免异常退出留下半截主密钥。
temporary = key_path.with_suffix(".tmp")
temporary.write_bytes(Fernet.generate_key())
self._restrict(temporary, 0o600)
try:
temporary.replace(key_path)
except FileExistsError:
temporary.unlink(missing_ok=True)
self._restrict(key_path, 0o600)
try:
return Fernet(key_path.read_bytes().strip())
except (OSError, ValueError) as exc:
raise CredentialStoreError(
"Credential master key cannot be loaded."
) from exc
def _read_tokens(self) -> dict[str, str]:
_, store_path = self._paths()
if not store_path.exists():
return {}
try:
data = json.loads(store_path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError) as exc:
raise CredentialStoreError(
"Encrypted credential store cannot be loaded."
) from exc
if not isinstance(data, dict) or not all(
isinstance(key, str) and isinstance(value, str)
for key, value in data.items()
):
raise CredentialStoreError(
"Encrypted credential store has an invalid format."
)
return data
def _write_tokens(self, tokens: dict[str, str]) -> None:
_, store_path = self._paths()
temporary = store_path.with_suffix(".tmp")
try:
store_path.parent.mkdir(parents=True, exist_ok=True)
self._restrict(store_path.parent, 0o700)
temporary.write_text(
json.dumps(tokens, ensure_ascii=True, sort_keys=True),
encoding="utf-8",
)
self._restrict(temporary, 0o600)
# 凭据表同样使用原子替换,确保并发读取只会看到完整 JSON。
temporary.replace(store_path)
self._restrict(store_path, 0o600)
except OSError as exc:
try:
temporary.unlink(missing_ok=True)
except OSError:
pass
raise CredentialStoreError(
"Encrypted credential store cannot be written."
) from exc
def put(self, credential_id: str, secret: str) -> None:
self._validate_id(credential_id)
if not secret:
raise CredentialStoreError("Credential secret cannot be empty.")
with self._lock:
tokens = self._read_tokens()
token = self._fernet().encrypt(secret.encode("utf-8")).decode("ascii")
tokens[credential_id] = token
self._write_tokens(tokens)
def resolve(self, credential_id: str | None) -> str | None:
if not credential_id:
return None
self._validate_id(credential_id)
with self._lock:
token = self._read_tokens().get(credential_id)
if token is None:
return None
try:
return self._fernet().decrypt(token.encode("ascii")).decode("utf-8")
except (InvalidToken, UnicodeDecodeError) as exc:
raise CredentialStoreError("Credential cannot be decrypted.") from exc
def has(self, credential_id: str) -> bool:
self._validate_id(credential_id)
with self._lock:
return credential_id in self._read_tokens()
def delete(self, credential_id: str) -> bool:
self._validate_id(credential_id)
with self._lock:
tokens = self._read_tokens()
removed = tokens.pop(credential_id, None) is not None
if removed:
self._write_tokens(tokens)
return removed
def delete_many(self, credential_ids: list[str]) -> set[str]:
"""用一次原子替换删除多个凭据,避免插件卸载只删除部分 Secret。"""
for credential_id in credential_ids:
self._validate_id(credential_id)
with self._lock:
tokens = self._read_tokens()
removed = {
credential_id
for credential_id in credential_ids
if credential_id in tokens
}
if removed:
for credential_id in removed:
del tokens[credential_id]
self._write_tokens(tokens)
return removed
def move_many(self, replacements: dict[str, str]) -> None:
"""原子迁移凭据 ID,直接移动密文且不覆盖已经写入的新凭据。"""
for old_id, new_id in replacements.items():
self._validate_id(old_id)
self._validate_id(new_id)
with self._lock:
tokens = self._read_tokens()
changed = False
for old_id, new_id in replacements.items():
if old_id != new_id and old_id in tokens:
tokens.setdefault(new_id, tokens.pop(old_id))
changed = True
if changed:
self._write_tokens(tokens)
class ChainedCredentialResolver:
def __init__(self, *resolvers: CredentialResolver) -> None:
self._resolvers = resolvers
def resolve(self, credential_id: str | None) -> str | None:
# 顺序即优先级:调用方可让 Host 注入值覆盖本地开发凭据。
for resolver in self._resolvers:
value = resolver.resolve(credential_id)
if value:
return value
return None
class ProviderCredentialResolver:
"""Provider 专用防御层,避免配置绕过 HTTP 校验读取 Plugin Secret。"""
def __init__(self, delegate: CredentialResolver) -> None:
self._delegate = delegate
def resolve(self, credential_id: str | None) -> str | None:
validate_provider_credential_id(credential_id)
return self._delegate.resolve(credential_id)
+76 -3
View File
@@ -1,6 +1,6 @@
from app.contracts import ModelCapability, ProviderConfig, ProviderType
from app.contracts import ModelCapability, ProviderConfig, ProviderPreset, ProviderType
from app.providers.base import ModelProvider
from app.providers.credentials import CredentialResolver
from app.providers.credentials import CredentialResolver, ProviderCredentialResolver
from app.providers.ollama import OllamaProvider
from app.providers.openai_compatible import OpenAICompatibleProvider
@@ -11,9 +11,23 @@ class UnsupportedProviderError(ValueError):
class ProviderFactory:
def __init__(self, credentials: CredentialResolver) -> None:
self.credentials = credentials
# ProviderFactory 是所有可配置 Provider 的创建边界,在此统一禁止
# Provider 借用 Plugin Secret 引用,避免调用方漏包安全 Resolver。
self.credentials = ProviderCredentialResolver(credentials)
def build(self, config: ProviderConfig) -> ModelProvider:
if config.provider_type == ProviderType.openai_responses:
from app.providers.openai_responses import OpenAIResponsesProvider
return OpenAIResponsesProvider(
base_url=config.base_url or "https://api.openai.com/v1",
credential_id=config.credential_id, credentials=self.credentials,
)
if config.provider_type == ProviderType.anthropic_messages:
from app.providers.anthropic_messages import AnthropicMessagesProvider
return AnthropicMessagesProvider(
base_url=config.base_url or "https://api.anthropic.com/v1",
credential_id=config.credential_id, credentials=self.credentials,
)
if config.provider_type in {
ProviderType.openai_chat,
ProviderType.openai_compatible,
@@ -27,11 +41,70 @@ class ProviderFactory:
return OllamaProvider(config.base_url or "http://127.0.0.1:11434")
raise UnsupportedProviderError(config.provider_type.value)
@staticmethod
def presets() -> list[ProviderPreset]:
presets = [
ProviderPreset(
preset_id="openai",
name="OpenAI",
provider_type=ProviderType.openai_chat,
base_url="https://api.openai.com/v1",
default_credential_id="openai",
),
ProviderPreset(
preset_id="deepseek",
name="DeepSeek",
provider_type=ProviderType.openai_compatible,
base_url="https://api.deepseek.com",
default_credential_id="deepseek",
),
ProviderPreset(
preset_id="ollama",
name="Ollama",
provider_type=ProviderType.ollama,
base_url="http://127.0.0.1:11434",
requires_credential=False,
),
]
# General API endpoints. Coding-plan endpoints and keys are separate products.
domestic = [
("kimi", "Kimi / 月之暗面", "https://api.moonshot.cn/v1", [], "长上下文对话;模型以账号权限为准。"),
("qwen", "阿里云百炼", "https://dashscope.aliyuncs.com/compatible-mode/v1", [ModelCapability.embedding], "中国内地兼容接口;海外地域需修改地址。"),
("zhipu", "智谱 GLM", "https://open.bigmodel.cn/api/paas/v4", [ModelCapability.embedding], "通用 APICoding Plan 请使用其专用地址。"),
("volcengine", "火山方舟 / 豆包", "https://ark.cn-beijing.volces.com/api/v3", [ModelCapability.embedding], "按账号填写模型 ID 或推理接入点 ID。"),
("siliconflow", "硅基流动", "https://api.siliconflow.cn/v1", [ModelCapability.embedding, ModelCapability.transcription], "支持兼容 Embedding 和音频转写接口。"),
("baidu", "百度千帆", "https://qianfan.baidubce.com/v2", [ModelCapability.embedding], "使用千帆 API Key;模型列表取决于账号。"),
("hunyuan", "腾讯混元", "https://api.hunyuan.cloud.tencent.com/v1", [], "OpenAI 兼容对话接口。"),
("minimax", "MiniMax", "https://api.minimaxi.com/v1", [], "文本对话兼容接口;其他媒体协议需独立适配。"),
("stepfun", "阶跃星辰", "https://api.stepfun.com/v1", [], "通用 API;Step Plan 请使用其专用地址。"),
]
for preset_id, name, url, extra, description in domestic:
presets.append(ProviderPreset(
preset_id=preset_id, name=name, provider_type=ProviderType.openai_compatible,
base_url=url, default_credential_id=preset_id, logo_id=preset_id,
capabilities=[ModelCapability.chat, *extra], description=description,
))
presets.extend([
ProviderPreset(preset_id="openai-responses", name="OpenAI Responses", provider_type=ProviderType.openai_responses,
base_url="https://api.openai.com/v1", default_credential_id="openai", logo_id="openai"),
ProviderPreset(preset_id="anthropic", name="Anthropic / Claude", provider_type=ProviderType.anthropic_messages,
base_url="https://api.anthropic.com/v1", default_credential_id="anthropic", logo_id="anthropic"),
])
for preset in presets:
if preset.logo_id == "custom":
preset.logo_id = preset.preset_id
if not preset.capabilities:
preset.capabilities = [ModelCapability.chat]
presets[0].capabilities += [ModelCapability.embedding, ModelCapability.transcription]
return presets
@staticmethod
def capabilities(provider_type: ProviderType) -> list[ModelCapability]:
if provider_type in {
ProviderType.openai_chat,
ProviderType.openai_compatible,
ProviderType.openai_responses,
ProviderType.anthropic_messages,
}:
return [
ModelCapability.chat,
+230
View File
@@ -1,9 +1,13 @@
import json
from collections.abc import AsyncIterator
from contextlib import aclosing
from datetime import datetime, timezone
import httpx
from app.contracts import ModelEvent, ModelEventType, ModelRequest
from app.providers.base import ProviderError, ProviderTurn
from app.providers.tool_names import prepare_tool_names
class TurnStreamingMixin:
@@ -80,3 +84,229 @@ def decode_tool_arguments(value: object) -> dict[str, object]:
if not isinstance(decoded, dict):
raise ProviderError("PROVIDER_INVALID_RESPONSE", "Tool arguments must be an object.")
return decoded
def invalid_response() -> ProviderError:
return ProviderError("PROVIDER_INVALID_RESPONSE", "Provider returned an invalid response.")
def truncated_stream() -> ProviderError:
return ProviderError("PROVIDER_STREAM_TRUNCATED", "Provider stream ended before completion.")
def object_value(value: object) -> dict:
if not isinstance(value, dict):
raise invalid_response()
return value
def list_value(value: object) -> list:
if not isinstance(value, list):
raise invalid_response()
return value
def string_value(value: object, *, nonempty: bool = False) -> str:
if not isinstance(value, str) or (nonempty and not value):
raise invalid_response()
return value
def token_count(value: object) -> int:
if isinstance(value, bool) or not isinstance(value, int) or value < 0:
raise invalid_response()
return value
def remote_error(value: object) -> ProviderError:
# Never reflect upstream messages, URLs, request bodies or credentials.
error = value if isinstance(value, dict) else {}
code = error.get("code") or error.get("type")
mapping = {
"authentication_error": "PROVIDER_AUTH_FAILED",
"invalid_api_key": "PROVIDER_AUTH_FAILED",
"permission_error": "PROVIDER_AUTH_FAILED",
"rate_limit_error": "PROVIDER_RATE_LIMITED",
"rate_limit_exceeded": "PROVIDER_RATE_LIMITED",
"insufficient_quota": "PROVIDER_RATE_LIMITED",
"not_found_error": "MODEL_NOT_FOUND",
"model_not_found": "MODEL_NOT_FOUND",
"invalid_request_error": "PROVIDER_INVALID_REQUEST",
"context_length_exceeded": "PROVIDER_INVALID_REQUEST",
}
mapped = mapping.get(code, "PROVIDER_UNAVAILABLE") if isinstance(code, str) else "PROVIDER_UNAVAILABLE"
return ProviderError(mapped, "Provider could not complete the request.")
def check_error(data: dict) -> None:
if data.get("error") is not None or data.get("type") == "error":
raise remote_error(data.get("error") or data)
class UsageTracker:
"""Merge cumulative snapshots, including partial usage updates."""
def __init__(self, input_key: str = "input_tokens", output_key: str = "output_tokens",
*, cache_tokens: bool = False) -> None:
self.input_key = input_key
self.output_key = output_key
self.cache_tokens = cache_tokens
self.counts: dict[str, int] = {}
def update(self, value: object) -> dict[str, int]:
usage = object_value(value)
keys = [self.input_key, self.output_key]
if self.cache_tokens:
keys += ["cache_creation_input_tokens", "cache_read_input_tokens"]
for key in keys:
if key in usage:
self.counts[key] = max(self.counts.get(key, 0), token_count(usage[key]))
inputs = self.counts.get(self.input_key, 0)
if self.cache_tokens:
inputs += sum(self.counts.get(key, 0) for key in keys[2:])
return {"input_tokens": inputs, "output_tokens": self.counts.get(self.output_key, 0)}
class EventStreamingMixin:
async def stream(self, request: ModelRequest) -> AsyncIterator[ModelEvent]:
sequence = 0
status = "completed"
try:
request, originals = prepare_tool_names(request)
# Closing the public iterator must synchronously close every nested iterator.
async with aclosing(self._events(request)) as events:
async for kind, data in events:
if kind == ModelEventType.tool_call_start and "name" in data:
data = {**data, "name": originals.get(data["name"], data["name"])}
if kind == ModelEventType.usage:
data = {**data, "total_tokens": data["input_tokens"] + data["output_tokens"]}
yield ModelEvent(event=kind, data=data, sequence=sequence,
timestamp=datetime.now(timezone.utc))
sequence += 1
except ProviderError as exc:
status = "failed"
yield ModelEvent(event=ModelEventType.error, sequence=sequence,
data={"code": exc.code, "message": exc.message},
timestamp=datetime.now(timezone.utc))
sequence += 1
except (ValueError, TypeError, KeyError, IndexError, AttributeError, OverflowError):
status = "failed"
error = invalid_response()
yield ModelEvent(event=ModelEventType.error, sequence=sequence,
data={"code": error.code, "message": error.message},
timestamp=datetime.now(timezone.utc))
sequence += 1
# CancelledError and GeneratorExit deliberately propagate without a Done event.
yield ModelEvent(event=ModelEventType.done, sequence=sequence,
data={"status": status},
timestamp=datetime.now(timezone.utc))
async def sse_objects(response: httpx.Response) -> AsyncIterator[dict]:
"""Read SSE frames, accepting the adjacent data lines used by some gateways."""
parts: list[str] = []
event_name = ""
def decode() -> dict:
value = "\n".join(parts)
if value.strip() == "[DONE]":
return {"type": "[DONE]"}
try:
data = object_value(json.loads(value))
except (ValueError, TypeError) as exc:
raise invalid_response() from exc
if event_name and "type" not in data:
data["type"] = event_name
check_error(data)
return data
async for line in response.aiter_lines():
if not line:
if parts:
yield decode()
parts = []
event_name = ""
elif line.startswith(":"):
continue
elif line.startswith("event:"):
if parts:
yield decode()
parts = []
event_name = line[6:].strip()
elif line.startswith("data:"):
if parts:
# Legacy compatible endpoints sometimes omit blank separators.
try:
json.loads("\n".join(parts))
except ValueError:
pass
else:
yield decode()
parts = []
event_name = ""
parts.append(line[5:].removeprefix(" "))
if parts:
yield decode()
class HTTPProviderMixin:
stream_path = "/chat/completions"
stream_format = "sse"
def _headers(self) -> dict[str, str]:
return {"Content-Type": "application/json"}
@staticmethod
def _status_error(exc: httpx.HTTPStatusError) -> ProviderError:
status = exc.response.status_code
code = {400: "PROVIDER_INVALID_REQUEST", 401: "PROVIDER_AUTH_FAILED",
403: "PROVIDER_AUTH_FAILED", 404: "MODEL_NOT_FOUND",
408: "PROVIDER_TIMEOUT", 413: "PROVIDER_INVALID_REQUEST",
422: "PROVIDER_INVALID_REQUEST", 429: "PROVIDER_RATE_LIMITED"}.get(
status, "PROVIDER_UNAVAILABLE")
return ProviderError(code, f"Provider returned HTTP {status}.")
async def _request(self, method: str, path: str, **kwargs) -> dict:
headers = self._headers()
try:
async with httpx.AsyncClient(timeout=self.timeout_seconds, transport=self.transport) as client:
response = await client.request(method, f"{self.base_url}{path}", headers=headers, **kwargs)
response.raise_for_status()
data = object_value(response.json())
check_error(data)
return data
except httpx.TimeoutException as exc:
raise ProviderError("PROVIDER_TIMEOUT", "Provider request timed out.") from exc
except httpx.HTTPStatusError as exc:
raise self._status_error(exc) from exc
except httpx.HTTPError as exc:
raise ProviderError("PROVIDER_UNAVAILABLE", "Provider is unavailable.") from exc
except (ValueError, TypeError) as exc:
raise invalid_response() from exc
async def _stream_json(self, payload: dict[str, object]) -> AsyncIterator[dict]:
headers = self._headers()
headers["Accept"] = "text/event-stream" if self.stream_format == "sse" else "application/x-ndjson"
try:
async with httpx.AsyncClient(timeout=self.timeout_seconds, transport=self.transport) as client:
async with client.stream("POST", f"{self.base_url}{self.stream_path}",
headers=headers, json=payload) as response:
response.raise_for_status()
if self.stream_format == "sse":
async with aclosing(sse_objects(response)) as objects:
async for data in objects:
yield data
else:
async for line in response.aiter_lines():
if line.strip():
data = object_value(json.loads(line))
check_error(data)
yield data
except httpx.TimeoutException as exc:
raise ProviderError("PROVIDER_TIMEOUT", "Provider request timed out.") from exc
except httpx.HTTPStatusError as exc:
raise self._status_error(exc) from exc
except httpx.HTTPError as exc:
raise ProviderError("PROVIDER_UNAVAILABLE", "Provider is unavailable.") from exc
except (ValueError, TypeError) as exc:
raise invalid_response() from exc
+88 -178
View File
@@ -1,16 +1,22 @@
from uuid import uuid4
import json
from collections.abc import AsyncIterator
from datetime import datetime, timezone
from contextlib import aclosing
from uuid import uuid4
import httpx
from app.contracts import ModelCapability, ModelEvent, ModelEventType, ModelInfo, ModelRequest
from app.contracts import MessageRole, ModelCapability, ModelEventType, ModelInfo, ModelRequest
from app.providers.base import ProviderError, ProviderToolCall, ProviderTurn
from app.providers.http_base import TurnStreamingMixin, decode_tool_arguments
from app.providers.tool_names import mapped_tool_names
from app.providers.http_base import (
EventStreamingMixin, HTTPProviderMixin, UsageTracker, decode_tool_arguments,
invalid_response, list_value, object_value, string_value, truncated_stream,
)
class OllamaProvider(TurnStreamingMixin):
class OllamaProvider(EventStreamingMixin, HTTPProviderMixin):
stream_path = "/api/chat"
stream_format = "jsonl"
def __init__(
self,
base_url: str = "http://127.0.0.1:11434",
@@ -21,126 +27,55 @@ class OllamaProvider(TurnStreamingMixin):
self.timeout_seconds = timeout_seconds
self.transport = transport
@mapped_tool_names
async def complete(self, request: ModelRequest) -> ProviderTurn:
messages = []
if request.system:
messages.append({"role": "system", "content": request.system})
for message in request.messages:
item: dict[str, object] = {
"role": message.role.value,
"content": message.content,
}
if message.tool_calls:
item["tool_calls"] = [
{
"function": {
"name": call.name,
"arguments": call.arguments,
}
}
for call in message.tool_calls
]
messages.append(item)
payload: dict[str, object] = {
"model": request.model,
"messages": messages,
"stream": False,
}
if request.tools:
payload["tools"] = [
{
"type": "function",
"function": {
"name": tool.name,
"description": tool.description,
"parameters": tool.parameters,
},
}
for tool in request.tools
]
data = await self._request("POST", "/api/chat", json=payload)
message = data.get("message") or {}
tool_calls = []
for raw_call in message.get("tool_calls") or []:
function = raw_call.get("function") or {}
tool_calls.append(
ProviderToolCall(
tool_call_id=raw_call.get("id") or f"call_{uuid4().hex}",
name=function.get("name") or "",
arguments=decode_tool_arguments(function.get("arguments", {})),
)
)
return ProviderTurn(
text=message.get("content") or None,
tool_calls=tool_calls,
input_tokens=int(data.get("prompt_eval_count") or 0),
output_tokens=int(data.get("eval_count") or 0),
data = await self._request("POST", self.stream_path, json=self._chat_payload(request, stream=False))
message = object_value(data.get("message"))
calls = [self._tool_call(raw) for raw in list_value(message.get("tool_calls", []))]
content = message.get("content")
if content is not None:
content = string_value(content)
return ProviderTurn(text=content or None, tool_calls=calls,
**UsageTracker("prompt_eval_count", "eval_count").update(data))
@staticmethod
def _tool_call(raw: object) -> ProviderToolCall:
call = object_value(raw)
function = object_value(call.get("function"))
return ProviderToolCall(
tool_call_id=string_value(call.get("id") or f"call_{uuid4().hex}"),
name=string_value(function.get("name"), nonempty=True),
arguments=decode_tool_arguments(function.get("arguments", {})),
)
async def list_models(self) -> list[ModelInfo]:
data = await self._request("GET", "/api/tags")
return [
ModelInfo(
model=item["name"],
display_name=item.get("name", ""),
capabilities=[ModelCapability.chat, ModelCapability.streaming],
)
for item in data.get("models", [])
if isinstance(item, dict) and item.get("name")
]
async def stream(self, request: ModelRequest) -> AsyncIterator[ModelEvent]:
payload = self._chat_payload(request, stream=True)
sequence = 0
def event(kind: ModelEventType, data: dict | None = None) -> ModelEvent:
nonlocal sequence
item = ModelEvent(
event=kind, sequence=sequence, data=data or {},
timestamp=datetime.now(timezone.utc),
)
sequence += 1
return item
try:
async for data in self._stream_json(payload):
message = data.get("message") or {}
async def _events(self, request: ModelRequest):
usage = UsageTracker("prompt_eval_count", "eval_count")
async with aclosing(self._stream_json(self._chat_payload(request, stream=True))) as chunks:
async for data in chunks:
message = object_value(data.get("message", {}))
if message.get("thinking"):
yield event(ModelEventType.thinking_delta, {"text": message["thinking"]})
yield ModelEventType.thinking_delta, {"text": string_value(message["thinking"])}
if message.get("content"):
yield event(ModelEventType.text_delta, {"text": message["content"]})
for raw_call in message.get("tool_calls") or []:
function = raw_call.get("function") or {}
call_id = raw_call.get("id") or f"call_{uuid4().hex}"
yield event(
ModelEventType.tool_call_start,
{"tool_call_id": call_id, "name": function.get("name") or ""},
)
yield event(
ModelEventType.tool_call_delta,
{
"tool_call_id": call_id,
"arguments_delta": json.dumps(
function.get("arguments") or {}, ensure_ascii=False
),
},
)
yield event(ModelEventType.tool_call_end, {"tool_call_id": call_id})
if data.get("done"):
yield event(
ModelEventType.usage,
{
"input_tokens": int(data.get("prompt_eval_count") or 0),
"output_tokens": int(data.get("eval_count") or 0),
},
)
yield event(ModelEventType.done)
except ProviderError as exc:
yield event(ModelEventType.error, {"code": exc.code, "message": exc.message})
yield event(ModelEventType.done)
yield ModelEventType.text_delta, {"text": string_value(message["content"])}
for raw in list_value(message.get("tool_calls", [])):
call = self._tool_call(raw)
yield ModelEventType.tool_call_start, {"tool_call_id": call.tool_call_id, "name": call.name}
yield ModelEventType.tool_call_delta, {
"tool_call_id": call.tool_call_id,
"arguments_delta": json.dumps(call.arguments, ensure_ascii=False),
}
yield ModelEventType.tool_call_end, {"tool_call_id": call.tool_call_id}
if "done" in data and not isinstance(data["done"], bool):
raise invalid_response()
if "prompt_eval_count" in data or "eval_count" in data or data.get("done"):
yield ModelEventType.usage, usage.update(data)
if data.get("done") is True:
return
raise truncated_stream()
def _chat_payload(self, request: ModelRequest, *, stream: bool) -> dict[str, object]:
messages = []
names: dict[str, str] = {}
if request.system:
messages.append({"role": "system", "content": request.system})
for message in request.messages:
@@ -150,53 +85,49 @@ class OllamaProvider(TurnStreamingMixin):
{"function": {"name": call.name, "arguments": call.arguments}}
for call in message.tool_calls
]
names.update({call.tool_call_id: call.name for call in message.tool_calls})
if message.role == MessageRole.tool:
name = message.name or names.get(message.tool_call_id or "")
if name:
item["tool_name"] = name
messages.append(item)
payload: dict[str, object] = {
"model": request.model, "messages": messages, "stream": stream
"model": request.model, "messages": messages, "stream": stream,
}
if request.tools:
payload["tools"] = [
{
"type": "function",
"function": {
"name": tool.name,
"description": tool.description,
"parameters": tool.parameters,
},
}
for tool in request.tools
{"type": "function", "function": {
"name": tool.name, "description": tool.description, "parameters": tool.parameters,
}} for tool in request.tools
]
options = {}
if request.temperature is not None:
options["temperature"] = request.temperature
if request.max_tokens is not None:
options["num_predict"] = request.max_tokens
if options:
payload["options"] = options
if request.response_format:
format_ = request.response_format
if format_.get("type") == "json_object":
payload["format"] = "json"
elif format_.get("type") == "json_schema":
payload["format"] = object_value(object_value(format_.get("json_schema")).get("schema"))
else:
payload["format"] = format_
return payload
async def _stream_json(self, payload: dict[str, object]) -> AsyncIterator[dict]:
try:
async with httpx.AsyncClient(
timeout=self.timeout_seconds, transport=self.transport
) as client:
async with client.stream(
"POST", f"{self.base_url}/api/chat", json=payload
) as response:
response.raise_for_status()
async for line in response.aiter_lines():
if not line.strip():
continue
try:
data = json.loads(line)
except json.JSONDecodeError as exc:
raise ProviderError(
"PROVIDER_INVALID_RESPONSE", "Ollama returned invalid JSONL."
) from exc
if isinstance(data, dict):
yield data
except httpx.TimeoutException as exc:
raise ProviderError("PROVIDER_TIMEOUT", "Ollama request timed out.") from exc
except httpx.HTTPStatusError as exc:
raise ProviderError(
"MODEL_NOT_FOUND" if exc.response.status_code == 404 else "PROVIDER_UNAVAILABLE",
f"Ollama returned HTTP {exc.response.status_code}.",
) from exc
except httpx.HTTPError as exc:
raise ProviderError("PROVIDER_UNAVAILABLE", "Ollama is unavailable.") from exc
async def list_models(self) -> list[ModelInfo]:
data = await self._request("GET", "/api/tags")
return [
ModelInfo(
model=string_value(item["name"]), display_name=item["name"],
capabilities=([ModelCapability.embedding] if "embed" in item["name"].lower()
else [ModelCapability.chat, ModelCapability.streaming]),
)
for item in list_value(data.get("models"))
if isinstance(item, dict) and isinstance(item.get("name"), str) and item["name"]
]
async def test_connection(self, model: str | None = None) -> tuple[bool, str]:
try:
@@ -206,24 +137,3 @@ class OllamaProvider(TurnStreamingMixin):
if model and model not in {item.model for item in models}:
return False, f"Model is not installed: {model}"
return True, f"Connected; discovered {len(models)} local model(s)."
async def _request(self, method: str, path: str, **kwargs) -> dict:
try:
async with httpx.AsyncClient(
timeout=self.timeout_seconds, transport=self.transport
) as client:
response = await client.request(method, f"{self.base_url}{path}", **kwargs)
response.raise_for_status()
data = response.json()
except httpx.TimeoutException as exc:
raise ProviderError("PROVIDER_TIMEOUT", "Ollama request timed out.") from exc
except httpx.HTTPStatusError as exc:
raise ProviderError(
"MODEL_NOT_FOUND" if exc.response.status_code == 404 else "PROVIDER_UNAVAILABLE",
f"Ollama returned HTTP {exc.response.status_code}.",
) from exc
except (httpx.HTTPError, ValueError) as exc:
raise ProviderError("PROVIDER_UNAVAILABLE", "Ollama is unavailable.") from exc
if not isinstance(data, dict):
raise ProviderError("PROVIDER_INVALID_RESPONSE", "Ollama returned non-object JSON.")
return data
+108 -203
View File
@@ -1,24 +1,20 @@
import json
from collections.abc import AsyncIterator
from datetime import datetime, timezone
from contextlib import aclosing
from uuid import uuid4
import httpx
from app.contracts import (
MessageRole,
ModelCapability,
ModelEvent,
ModelEventType,
ModelInfo,
ModelRequest,
)
from app.contracts import MessageRole, ModelCapability, ModelEventType, ModelInfo, ModelRequest
from app.providers.base import ProviderError, ProviderToolCall, ProviderTurn
from app.providers.credentials import CredentialResolver
from app.providers.http_base import TurnStreamingMixin, decode_tool_arguments
from app.providers.credentials import CredentialResolver, CredentialStoreError
from app.providers.tool_names import mapped_tool_names
from app.providers.http_base import (
EventStreamingMixin, HTTPProviderMixin, UsageTracker, decode_tool_arguments,
invalid_response, list_value, object_value, string_value, token_count, truncated_stream,
)
class OpenAICompatibleProvider(TurnStreamingMixin):
class OpenAICompatibleProvider(EventStreamingMixin, HTTPProviderMixin):
def __init__(
self,
base_url: str,
@@ -33,50 +29,37 @@ class OpenAICompatibleProvider(TurnStreamingMixin):
self.timeout_seconds = timeout_seconds
self.transport = transport
@mapped_tool_names
async def complete(self, request: ModelRequest) -> ProviderTurn:
payload = self._payload(request, stream=False)
data = await self._request("POST", "/chat/completions", json=payload)
try:
message = data["choices"][0]["message"]
except (KeyError, IndexError, TypeError) as exc:
raise ProviderError("PROVIDER_INVALID_RESPONSE", "Missing completion message.") from exc
tool_calls = []
for raw_call in message.get("tool_calls") or []:
function = raw_call.get("function") or {}
tool_calls.append(
ProviderToolCall(
tool_call_id=raw_call.get("id") or f"call_{uuid4().hex}",
name=function.get("name") or "",
arguments=decode_tool_arguments(function.get("arguments", "{}")),
)
)
usage = data.get("usage") or {}
return ProviderTurn(
text=message.get("content"),
tool_calls=tool_calls,
input_tokens=int(usage.get("prompt_tokens") or 0),
output_tokens=int(usage.get("completion_tokens") or 0),
)
data = await self._request("POST", self.stream_path, json=self._payload(request, stream=False))
choices = list_value(data.get("choices"))
if not choices:
raise invalid_response()
message = object_value(object_value(choices[0]).get("message"))
calls = []
for raw in list_value(message.get("tool_calls", [])):
raw = object_value(raw)
function = object_value(raw.get("function"))
calls.append(ProviderToolCall(
tool_call_id=string_value(raw.get("id") or f"call_{uuid4().hex}"),
name=string_value(function.get("name"), nonempty=True),
arguments=decode_tool_arguments(function.get("arguments", "{}")),
))
text = message.get("content")
if text is not None:
text = string_value(text)
usage = UsageTracker("prompt_tokens", "completion_tokens").update(data.get("usage") or {})
return ProviderTurn(text=text, tool_calls=calls, **usage)
def _payload(self, request: ModelRequest, *, stream: bool) -> dict[str, object]:
payload: dict[str, object] = {
"model": request.model,
"messages": self._messages(request),
"stream": stream,
"model": request.model, "messages": self._messages(request), "stream": stream,
}
if request.tools:
payload["tools"] = [
{
"type": "function",
"function": {
"name": tool.name,
"description": tool.description,
"parameters": tool.parameters,
},
}
for tool in request.tools
{"type": "function", "function": {
"name": tool.name, "description": tool.description, "parameters": tool.parameters,
}} for tool in request.tools
]
if request.temperature is not None:
payload["temperature"] = request.temperature
@@ -84,124 +67,78 @@ class OpenAICompatibleProvider(TurnStreamingMixin):
payload["max_tokens"] = request.max_tokens
if request.response_format is not None:
payload["response_format"] = request.response_format
if stream:
payload["stream_options"] = {"include_usage": True}
return payload
async def stream(self, request: ModelRequest) -> AsyncIterator[ModelEvent]:
sequence = 0
open_calls: dict[int, str] = {}
def event(kind: ModelEventType, data: dict | None = None) -> ModelEvent:
nonlocal sequence
item = ModelEvent(
event=kind,
sequence=sequence,
data=data or {},
timestamp=datetime.now(timezone.utc),
)
sequence += 1
return item
try:
async for data in self._stream_json(self._payload(request, stream=True)):
usage = data.get("usage") or {}
if usage:
yield event(
ModelEventType.usage,
{
"input_tokens": int(usage.get("prompt_tokens") or 0),
"output_tokens": int(usage.get("completion_tokens") or 0),
},
)
choices = data.get("choices") or []
async def _events(self, request: ModelRequest):
calls: dict[int, dict] = {}
usage = UsageTracker("prompt_tokens", "completion_tokens")
finished = False
seen = False
async with aclosing(self._stream_json(self._payload(request, stream=True))) as chunks:
async for data in chunks:
if data.get("type") == "[DONE]":
if not seen:
raise invalid_response()
finished = True
break
if data.get("usage") is not None:
yield ModelEventType.usage, usage.update(data["usage"])
choices = list_value(data.get("choices", []))
if not choices:
continue
choice = choices[0]
delta = choice.get("delta") or {}
seen = True
choice = object_value(choices[0])
delta = object_value(choice.get("delta") or {})
if delta.get("reasoning_content"):
yield event(
ModelEventType.thinking_delta,
{"text": delta["reasoning_content"]},
)
yield ModelEventType.thinking_delta, {"text": string_value(delta["reasoning_content"])}
if delta.get("content"):
yield event(ModelEventType.text_delta, {"text": delta["content"]})
for raw_call in delta.get("tool_calls") or []:
index = int(raw_call.get("index") or 0)
function = raw_call.get("function") or {}
call_id = raw_call.get("id") or open_calls.get(index) or f"call_{uuid4().hex}"
if index not in open_calls:
open_calls[index] = call_id
yield event(
ModelEventType.tool_call_start,
{"tool_call_id": call_id, "name": function.get("name") or ""},
)
if function.get("arguments"):
yield event(
ModelEventType.tool_call_delta,
{
"tool_call_id": open_calls[index],
"arguments_delta": function["arguments"],
},
)
if choice.get("finish_reason") == "tool_calls":
for call_id in open_calls.values():
yield event(
ModelEventType.tool_call_end, {"tool_call_id": call_id}
)
open_calls.clear()
for call_id in open_calls.values():
yield event(ModelEventType.tool_call_end, {"tool_call_id": call_id})
yield event(ModelEventType.done)
except ProviderError as exc:
yield event(ModelEventType.error, {"code": exc.code, "message": exc.message})
yield event(ModelEventType.done)
async def _stream_json(self, payload: dict[str, object]) -> AsyncIterator[dict]:
headers = self._headers()
try:
async with httpx.AsyncClient(
timeout=self.timeout_seconds, transport=self.transport
) as client:
async with client.stream(
"POST", f"{self.base_url}/chat/completions", headers=headers, json=payload
) as response:
response.raise_for_status()
async for line in response.aiter_lines():
if not line.startswith("data:"):
continue
value = line[5:].strip()
if not value or value == "[DONE]":
continue
try:
data = json.loads(value)
except json.JSONDecodeError as exc:
raise ProviderError(
"PROVIDER_INVALID_RESPONSE", "Provider returned invalid SSE JSON."
) from exc
if isinstance(data, dict):
yield data
except httpx.TimeoutException as exc:
raise ProviderError("PROVIDER_TIMEOUT", "Provider request timed out.") from exc
except httpx.HTTPStatusError as exc:
raise self._status_error(exc) from exc
except httpx.HTTPError as exc:
raise ProviderError("PROVIDER_UNAVAILABLE", "Provider is unavailable.") from exc
yield ModelEventType.text_delta, {"text": string_value(delta["content"])}
for raw in list_value(delta.get("tool_calls", [])):
raw = object_value(raw)
index = token_count(raw.get("index", 0))
function = object_value(raw.get("function") or {})
call = calls.setdefault(index, {"id": "", "name": "", "arguments": ""})
if raw.get("id"):
call["id"] = string_value(raw["id"])
if function.get("name"):
call["name"] += string_value(function["name"])
fragment = string_value(function.get("arguments", ""))
call["arguments"] += fragment
if choice.get("finish_reason"):
finished = True
if not finished:
raise truncated_stream()
for call in calls.values():
if not call["name"]:
raise invalid_response()
decode_tool_arguments(call["arguments"] or "{}")
# A name can span multiple chunks; publish only the complete identity.
call["id"] = call["id"] or f"call_{uuid4().hex}"
yield ModelEventType.tool_call_start, {"tool_call_id": call["id"], "name": call["name"]}
yield ModelEventType.tool_call_delta, {"tool_call_id": call["id"], "arguments_delta": call["arguments"] or "{}"}
yield ModelEventType.tool_call_end, {"tool_call_id": call["id"]}
async def list_models(self) -> list[ModelInfo]:
data = await self._request("GET", "/models")
return [
ModelInfo(
model=item["id"],
display_name=item["id"],
capabilities=[
ModelCapability.chat,
ModelCapability.tool_calling,
ModelCapability.streaming,
],
)
for item in data.get("data", [])
if isinstance(item, dict) and item.get("id")
]
return [ModelInfo(model=string_value(item["id"]), display_name=item["id"],
capabilities=self._model_capabilities(string_value(item["id"])))
for item in list_value(data.get("data"))
if isinstance(item, dict) and item.get("id")]
@staticmethod
def _model_capabilities(model: str) -> list[ModelCapability]:
# /models does not advertise capabilities. Avoid known non-chat families;
# these are discovery hints, not a guarantee of support by a gateway.
name = model.lower()
if "embed" in name or name.startswith(("bge-", "bge/")):
return [ModelCapability.embedding]
if any(marker in name for marker in (
"whisper", "tts", "transcri", "audio", "realtime", "dall-e", "image", "moderation", "rerank",
)):
return []
return [ModelCapability.chat]
async def test_connection(self, model: str | None = None) -> tuple[bool, str]:
try:
@@ -217,62 +154,30 @@ class OpenAICompatibleProvider(TurnStreamingMixin):
if request.system:
result.append({"role": "system", "content": request.system})
for message in request.messages:
item: dict[str, object] = {
"role": message.role.value,
"content": message.content,
}
item: dict[str, object] = {"role": message.role.value, "content": message.content}
if message.name:
item["name"] = message.name
if message.role == MessageRole.tool and message.tool_call_id:
item["tool_call_id"] = message.tool_call_id
if message.tool_calls:
item["tool_calls"] = [
{
"id": call.tool_call_id,
"type": "function",
"function": {
"name": call.name,
"arguments": json.dumps(call.arguments),
},
}
for call in message.tool_calls
{"id": call.tool_call_id, "type": "function", "function": {
"name": call.name, "arguments": json.dumps(call.arguments),
}} for call in message.tool_calls
]
result.append(item)
return result
async def _request(self, method: str, path: str, **kwargs) -> dict:
headers = self._headers()
try:
async with httpx.AsyncClient(
timeout=self.timeout_seconds, transport=self.transport
) as client:
response = await client.request(
method, f"{self.base_url}{path}", headers=headers, **kwargs
)
response.raise_for_status()
data = response.json()
except httpx.TimeoutException as exc:
raise ProviderError("PROVIDER_TIMEOUT", "Provider request timed out.") from exc
except httpx.HTTPStatusError as exc:
raise self._status_error(exc) from exc
except (httpx.HTTPError, ValueError) as exc:
raise ProviderError("PROVIDER_UNAVAILABLE", "Provider is unavailable.") from exc
if not isinstance(data, dict):
raise ProviderError("PROVIDER_INVALID_RESPONSE", "Provider returned non-object JSON.")
return data
def _headers(self) -> dict[str, str]:
headers = {"Content-Type": "application/json"}
api_key = self.credentials.resolve(self.credential_id)
try:
api_key = self.credentials.resolve(self.credential_id)
except CredentialStoreError as exc:
raise ProviderError("PROVIDER_CREDENTIAL_UNAVAILABLE",
"Credential could not be decrypted by the AI Core.") from exc
if self.credential_id and not api_key:
raise ProviderError("PROVIDER_CREDENTIAL_MISSING",
"Credential is not available in the AI Core process.")
if api_key:
headers["Authorization"] = f"Bearer {api_key}"
return headers
@staticmethod
def _status_error(exc: httpx.HTTPStatusError) -> ProviderError:
code = {
401: "PROVIDER_AUTH_FAILED",
404: "MODEL_NOT_FOUND",
429: "PROVIDER_RATE_LIMITED",
}.get(exc.response.status_code, "PROVIDER_UNAVAILABLE")
return ProviderError(code, f"Provider returned HTTP {exc.response.status_code}.")
+168
View File
@@ -0,0 +1,168 @@
"""Native /responses adapter; stateless history uses function_call/output items."""
import json
from contextlib import aclosing
from app.contracts import MessageRole, ModelEventType, ModelRequest
from app.providers.base import ProviderError, ProviderToolCall, ProviderTurn
from app.providers.http_base import (
UsageTracker, check_error, decode_tool_arguments, invalid_response, list_value,
object_value, remote_error, string_value, token_count, truncated_stream,
)
from app.providers.openai_compatible import OpenAICompatibleProvider
from app.providers.tool_names import mapped_tool_names
class OpenAIResponsesProvider(OpenAICompatibleProvider):
stream_path = "/responses"
def _payload(self, request: ModelRequest, *, stream: bool) -> dict[str, object]:
inputs = []
for message in request.messages:
if message.role == MessageRole.tool:
if not message.tool_call_id:
raise ProviderError("PROVIDER_INVALID_REQUEST", "Tool result requires a call identifier.")
inputs.append({"type": "function_call_output", "call_id": message.tool_call_id,
"output": message.content})
continue
if message.content or not message.tool_calls:
inputs.append({"role": message.role.value, "content": message.content})
for call in message.tool_calls:
inputs.append({"type": "function_call", "call_id": call.tool_call_id,
"name": call.name, "arguments": json.dumps(call.arguments)})
payload: dict[str, object] = {"model": request.model, "input": inputs, "stream": stream}
if request.system:
payload["instructions"] = request.system
if request.tools:
payload["tools"] = [{"type": "function", "name": tool.name,
"description": tool.description, "parameters": tool.parameters}
for tool in request.tools]
if request.temperature is not None:
payload["temperature"] = request.temperature
if request.max_tokens is not None:
payload["max_output_tokens"] = request.max_tokens
if request.response_format is not None:
format_ = dict(request.response_format)
if format_.get("type") == "json_schema":
format_ = {"type": "json_schema", **object_value(format_.get("json_schema"))}
payload["text"] = {"format": format_}
return payload
@staticmethod
def _check_response(data: dict) -> None:
check_error(data)
status = data.get("status")
if status == "incomplete":
raise ProviderError("PROVIDER_INCOMPLETE_RESPONSE", "Provider response is incomplete.")
if status == "failed":
raise remote_error(data.get("error"))
if status is not None and status != "completed":
raise invalid_response()
@mapped_tool_names
async def complete(self, request: ModelRequest) -> ProviderTurn:
data = await self._request("POST", self.stream_path, json=self._payload(request, stream=False))
self._check_response(data)
texts = []
calls = []
for raw in list_value(data.get("output")):
item = object_value(raw)
if item.get("type") == "message":
for raw_part in list_value(item.get("content")):
part = object_value(raw_part)
if part.get("type") == "output_text":
texts.append(string_value(part.get("text")))
elif part.get("type") == "refusal":
texts.append(string_value(part.get("refusal")))
elif item.get("type") == "function_call":
calls.append(ProviderToolCall(
tool_call_id=string_value(item.get("call_id"), nonempty=True),
name=string_value(item.get("name"), nonempty=True),
arguments=decode_tool_arguments(item.get("arguments")),
))
return ProviderTurn(text="".join(texts) or None, tool_calls=calls,
**UsageTracker().update(data.get("usage") or {}))
async def _events(self, request: ModelRequest):
calls: dict[int, dict] = {}
usage = UsageTracker()
def finish_call(index: int, final: object = None):
call = calls[index]
if call["ended"]:
return []
events = []
if final is not None:
arguments = string_value(final)
if not arguments.startswith(call["arguments"]):
raise invalid_response()
remainder = arguments[len(call["arguments"]):]
if remainder:
events.append((ModelEventType.tool_call_delta,
{"tool_call_id": call["id"], "arguments_delta": remainder}))
call["arguments"] = arguments
decode_tool_arguments(call["arguments"])
call["ended"] = True
events.append((ModelEventType.tool_call_end, {"tool_call_id": call["id"]}))
return events
async with aclosing(self._stream_json(self._payload(request, stream=True))) as chunks:
async for data in chunks:
kind = string_value(data.get("type"), nonempty=True)
if kind in {"response.failed", "response.incomplete"}:
response = object_value(data.get("response"))
self._check_response({**response, "status": kind.split(".")[1]})
elif kind in {"response.output_text.delta", "response.refusal.delta"}:
yield ModelEventType.text_delta, {"text": string_value(data.get("delta"))}
elif kind in {"response.reasoning_summary_text.delta", "response.reasoning_text.delta"}:
yield ModelEventType.thinking_delta, {"text": string_value(data.get("delta"))}
elif kind in {"response.output_item.added", "response.output_item.done"}:
item = object_value(data.get("item"))
if item.get("type") != "function_call":
continue
index = token_count(data.get("output_index"))
call_id = string_value(item.get("call_id"), nonempty=True)
name = string_value(item.get("name"), nonempty=True)
if index not in calls:
calls[index] = {"id": call_id, "name": name, "arguments": "", "ended": False,
"item_id": item.get("id")}
yield ModelEventType.tool_call_start, {"tool_call_id": call_id, "name": name}
elif calls[index]["id"] != call_id or calls[index]["name"] != name:
raise invalid_response()
if kind == "response.output_item.done":
for event in finish_call(index, item.get("arguments")):
yield event
elif item.get("arguments"):
arguments = string_value(item["arguments"])
calls[index]["arguments"] += arguments
yield ModelEventType.tool_call_delta, {"tool_call_id": call_id, "arguments_delta": arguments}
elif kind in {"response.function_call_arguments.delta", "response.function_call_arguments.done"}:
index = token_count(data.get("output_index"))
call = calls.get(index)
if call is None or (data.get("item_id") and call["item_id"] != data["item_id"]):
raise invalid_response()
if kind.endswith(".done"):
for event in finish_call(index, data.get("arguments")):
yield event
else:
if call["ended"]:
raise invalid_response()
fragment = string_value(data.get("delta"))
call["arguments"] += fragment
yield ModelEventType.tool_call_delta, {"tool_call_id": call["id"], "arguments_delta": fragment}
elif kind == "response.completed":
response = object_value(data.get("response"))
self._check_response(response)
if any(not call["ended"] for call in calls.values()):
raise truncated_stream()
if response.get("usage") is not None:
yield ModelEventType.usage, usage.update(response["usage"])
return
elif kind == "[DONE]":
raise truncated_stream()
elif kind in {"response.created", "response.in_progress"}:
response = object_value(data.get("response"))
check_error(response)
if response.get("usage") is not None:
yield ModelEventType.usage, usage.update(response["usage"])
raise truncated_stream()
+52 -1
View File
@@ -1,5 +1,10 @@
from dataclasses import dataclass
from time import perf_counter
from pathlib import Path
from app.config import get_settings
from app.database.db import connect
from app.errors import ApiError
from app.contracts import ModelInfo, ProviderConfig, ProviderTestResponse
from app.providers.base import ModelProvider
@@ -16,20 +21,64 @@ class RegisteredProvider:
class ProviderRegistry:
def __init__(self) -> None:
def __init__(self, factory=None) -> None:
self._providers: dict[str, RegisteredProvider] = {}
self._factory = factory
self._loaded_path: Path | None = None
def _restore(self) -> None:
if self._factory is None or self._loaded_path == get_settings().db_path:
return
conn = connect()
try:
conn.execute("CREATE TABLE IF NOT EXISTS provider_configs (provider_id TEXT PRIMARY KEY, config_json TEXT NOT NULL)")
restored = {}
for row in conn.execute("SELECT config_json FROM provider_configs"):
config = ProviderConfig.model_validate_json(row["config_json"])
if config.provider_id == "mock":
raise ValueError("reserved provider")
restored[config.provider_id] = RegisteredProvider(config, self._factory.build(config))
if "mock" in self._providers:
restored["mock"] = self._providers["mock"]
self._providers = restored
self._loaded_path = get_settings().db_path
except (ValueError, TypeError) as exc:
raise ApiError(500, "PROVIDER_STORAGE_INVALID", "Saved provider configuration could not be loaded.") from exc
finally:
conn.close()
def _save(self, config: ProviderConfig) -> None:
if self._factory is None or config.provider_id == "mock":
return
conn = connect()
try:
conn.execute("INSERT OR REPLACE INTO provider_configs VALUES (?, ?)", (config.provider_id, config.model_dump_json()))
finally:
conn.close()
def register(self, config: ProviderConfig, adapter: ModelProvider) -> None:
if config.provider_id != "mock":
self._restore()
if config.provider_id in self._providers:
raise ValueError(f"Provider already registered: {config.provider_id}")
self._save(config)
self._providers[config.provider_id] = RegisteredProvider(config=config, adapter=adapter)
def unregister(self, provider_id: str) -> None:
self._restore()
if self._factory is not None:
conn = connect()
try:
conn.execute("DELETE FROM provider_configs WHERE provider_id = ?", (provider_id,))
finally:
conn.close()
self._providers.pop(provider_id, None)
def replace(self, config: ProviderConfig, adapter: ModelProvider) -> None:
self._restore()
if config.provider_id not in self._providers:
raise ProviderNotFoundError(config.provider_id)
self._save(config)
self._providers[config.provider_id] = RegisteredProvider(config=config, adapter=adapter)
def get(self, provider_id: str) -> RegisteredProvider:
@@ -39,12 +88,14 @@ class ProviderRegistry:
return provider
def get_any(self, provider_id: str) -> RegisteredProvider:
self._restore()
try:
return self._providers[provider_id]
except KeyError as exc:
raise ProviderNotFoundError(provider_id) from exc
def list_configs(self) -> list[ProviderConfig]:
self._restore()
return [item.config.model_copy(deep=True) for item in self._providers.values()]
async def list_models(self, provider_id: str) -> list[ModelInfo]:
+291
View File
@@ -0,0 +1,291 @@
"""Capability routing: validated remote results, then an explicit local backend.
Phase E supplies HTTP adapters and injectable local contracts. Hash embeddings are
still a development placeholder; speech models are installed in phase F.
"""
from __future__ import annotations
import hashlib
import json
import math
from dataclasses import dataclass
from pathlib import Path
from typing import Protocol
import httpx
from app.contracts import (
EmbeddingResult, LocalBackendStatus, ModelBinding, ModelRoutingConfig,
ModelRoutingResponse, ProviderType, SpeakerMatchResult,
)
from app.database.db import connect, transaction
from app.errors import ApiError
from app.providers.base import ProviderError
from app.providers.credentials import CredentialResolver, CredentialStoreError
from app.providers.registry import ProviderNotFoundError, ProviderRegistry
from app.retrieval.embedding import EmbeddingProvider, HashEmbeddingProvider
from app.retrieval.provenance import record_embedding
CAPABILITIES = ("embedding", "transcription", "speaker_matching")
HTTP_TYPES = {ProviderType.openai_chat, ProviderType.openai_compatible}
MAX_MEDIA_BYTES = 25 * 1024 * 1024
MAX_RESPONSE_BYTES = 16 * 1024 * 1024
class LocalSpeechBackend(Protocol):
available: bool
async def transcribe(self, source: Path, language: str | None) -> str: ...
async def match(self, source: Path, reference: Path) -> float: ...
class PendingSpeechBackend:
available = False
async def transcribe(self, source: Path, language: str | None) -> str:
raise ProviderError("LOCAL_MODEL_NOT_INSTALLED", "本地音频转写模型尚未安装,将在阶段 F 接入。")
async def match(self, source: Path, reference: Path) -> float:
raise ProviderError("LOCAL_MODEL_NOT_INSTALLED", "本地声纹模型尚未安装,将在阶段 F 接入。")
@dataclass(frozen=True)
class RoutedTranscript:
text: str
source: str
fallback_reason: str | None = None
def invalid_response() -> ProviderError:
return ProviderError("PROVIDER_INVALID_RESPONSE", "Model API returned an invalid result.")
def finite_number(value: object) -> bool:
if type(value) not in (int, float):
return False
try:
return math.isfinite(value)
except (OverflowError, ValueError):
return False
class ModelRoutingService:
def __init__(self, providers: ProviderRegistry, credentials: CredentialResolver, *,
local_embedding: EmbeddingProvider | None = None,
local_speech: LocalSpeechBackend | None = None,
transport: httpx.AsyncBaseTransport | None = None) -> None:
self.providers = providers
self.credentials = credentials
self.local_embedding = local_embedding or HashEmbeddingProvider()
self.local_speech = local_speech or PendingSpeechBackend()
self.transport = transport
@staticmethod
def _connection():
conn = connect()
conn.execute("CREATE TABLE IF NOT EXISTS model_routing (id INTEGER PRIMARY KEY CHECK(id=1), config_json TEXT NOT NULL)")
return conn
def configuration(self) -> ModelRoutingConfig:
conn = self._connection()
try:
row = conn.execute("SELECT config_json FROM model_routing WHERE id=1").fetchone()
return ModelRoutingConfig.model_validate_json(row[0]) if row else ModelRoutingConfig()
except ValueError as exc:
raise ApiError(500, "MODEL_ROUTING_STORAGE_INVALID", "Saved model routing could not be loaded.") from exc
finally:
conn.close()
def describe(self) -> ModelRoutingResponse:
return ModelRoutingResponse(config=self.configuration(), local_backends=[
LocalBackendStatus(capability="embedding", status="placeholder" if isinstance(self.local_embedding, HashEmbeddingProvider) else "ready",
message="当前为 hash-v1 确定性占位向量,真实本地语义模型尚未集成。" if isinstance(self.local_embedding, HashEmbeddingProvider) else "本地 Embedding 模型已就绪。"),
*[LocalBackendStatus(capability=capability, status="ready" if self.local_speech.available else "not_installed",
message="本地模型已就绪。" if self.local_speech.available else "阶段 F 接入本地模型;当前保留回退接口。")
for capability in ("transcription", "speaker_matching")],
])
def update(self, config: ModelRoutingConfig) -> ModelRoutingResponse:
for capability in CAPABILITIES:
binding = getattr(config, capability)
if binding:
try:
provider = self.providers.get_any(binding.provider_id).config
except ProviderNotFoundError as exc:
raise ApiError(422, "PROVIDER_NOT_FOUND", "请选择已保存的提供商。") from exc
if provider.provider_type not in HTTP_TYPES:
raise ApiError(422, "MODEL_ROUTING_PROTOCOL_UNSUPPORTED", "该能力当前需要 OpenAI Compatible HTTP 接口。")
conn = self._connection()
try:
with transaction(conn):
row = conn.execute("SELECT config_json FROM model_routing WHERE id=1").fetchone()
current = ModelRoutingConfig.model_validate_json(row[0]) if row else ModelRoutingConfig()
if current.version != config.version:
raise ApiError(409, "MODEL_ROUTING_VERSION_CONFLICT", "配置已更新,请重新加载后再保存。")
saved = config.model_copy(update={"version": config.version + 1})
conn.execute("INSERT OR REPLACE INTO model_routing VALUES (1, ?)", (saved.model_dump_json(),))
finally:
conn.close()
return self.describe()
def uses_provider(self, provider_id: str) -> bool:
config = self.configuration()
return any(binding and binding.provider_id == provider_id for binding in
(getattr(config, name) for name in CAPABILITIES))
def _remote(self, binding: ModelBinding) -> tuple[str, dict[str, str]]:
try:
provider = self.providers.get(binding.provider_id).config
except ProviderNotFoundError as exc:
raise ProviderError("PROVIDER_UNAVAILABLE", "Configured provider is unavailable.") from exc
if provider.provider_type not in HTTP_TYPES:
raise ProviderError("PROVIDER_CAPABILITY_UNSUPPORTED", "Provider does not support this HTTP capability.")
try:
key = self.credentials.resolve(provider.credential_id)
except CredentialStoreError as exc:
raise ProviderError("PROVIDER_CREDENTIAL_UNAVAILABLE", "Provider credential is unavailable.") from exc
if provider.credential_id and not key:
raise ProviderError("PROVIDER_CREDENTIAL_MISSING", "Provider credential is not configured.")
url = (provider.base_url or "https://api.openai.com/v1").rstrip("/") + binding.endpoint
return url, {"Authorization": f"Bearer {key}"} if key else {}
async def _request(self, binding: ModelBinding, *, remote: tuple[str, dict[str, str]] | None = None, **kwargs) -> tuple[dict, str]:
url, headers = remote or self._remote(binding)
try:
async with httpx.AsyncClient(timeout=30, transport=self.transport) as client:
async with client.stream("POST", url, headers=headers, **kwargs) as response:
response.raise_for_status()
body = bytearray()
async for chunk in response.aiter_bytes():
body.extend(chunk)
if len(body) > MAX_RESPONSE_BYTES:
raise invalid_response()
data = json.loads(body)
except httpx.TimeoutException as exc:
raise ProviderError("PROVIDER_TIMEOUT", "Model API timed out.") from exc
except httpx.HTTPStatusError as exc:
code = {401: "PROVIDER_AUTH_FAILED", 403: "PROVIDER_AUTH_FAILED", 404: "MODEL_NOT_FOUND", 429: "PROVIDER_RATE_LIMITED"}.get(exc.response.status_code, "PROVIDER_UNAVAILABLE")
raise ProviderError(code, f"Model API returned HTTP {exc.response.status_code}.") from exc
except (httpx.HTTPError, httpx.InvalidURL) as exc:
raise ProviderError("PROVIDER_UNAVAILABLE", "Model API is unavailable.") from exc
except (ValueError, UnicodeError) as exc:
raise invalid_response() from exc
if not isinstance(data, dict) or data.get("error"):
raise invalid_response()
return data, url
async def embed(self, texts: list[str]) -> EmbeddingResult:
config = self.configuration()
binding = config.embedding
record_embedding(route_version=config.version,
requested_route=binding.model_dump() if binding else None)
reason = None
if binding and texts:
try:
vectors = []
dimension = binding.dimensions
# Freeze the origin across batches, even if the user edits the provider.
remote = self._remote(binding)
for start in range(0, len(texts), 32):
batch = texts[start:start + 32]
payload = {"model": binding.model, "input": batch, "encoding_format": "float"}
if binding.dimensions is not None:
payload["dimensions"] = binding.dimensions
data, url = await self._request(binding, remote=remote, json=payload)
items = data.get("data")
if not isinstance(items, list) or len(items) != len(batch):
raise invalid_response()
indexed = {}
for item in items:
if not isinstance(item, dict):
raise invalid_response()
index, vector = item.get("index"), item.get("embedding")
if type(index) is not int or index in indexed or not 0 <= index < len(batch):
raise invalid_response()
if not isinstance(vector, list) or not 1 <= len(vector) <= 16384:
raise invalid_response()
if any(not finite_number(value) for value in vector):
raise invalid_response()
dimension = dimension or len(vector)
norm = math.hypot(*vector)
if len(vector) != dimension or not norm or not math.isfinite(norm):
raise invalid_response()
indexed[index] = [value / norm for value in vector]
vectors.extend(indexed[index] for index in range(len(batch)))
identity = json.dumps([url, binding.model, dimension], separators=(",", ":"))
return EmbeddingResult(vectors=vectors, source="api", dimensions=dimension,
model_id="api-" + hashlib.sha256(identity.encode()).hexdigest())
except ProviderError as exc:
reason = exc.code
vectors = await self.local_embedding.embed_documents(texts)
return EmbeddingResult(vectors=vectors, source="local", model_id=self.local_embedding.model_id,
dimensions=self.local_embedding.dim, fallback_reason=reason)
@staticmethod
def _media_file(path: Path):
try:
handle = path.open("rb")
except OSError as exc:
raise ApiError(404, "ATTACHMENT_NOT_FOUND", "Audio attachment was not found.") from exc
import os
if not 0 < os.fstat(handle.fileno()).st_size <= MAX_MEDIA_BYTES:
handle.close()
raise ApiError(413, "ATTACHMENT_TOO_LARGE", "Audio attachment must be between 1 byte and 25 MiB.")
return handle
async def transcribe(self, source: Path, language: str | None) -> RoutedTranscript:
binding = self.configuration().transcription
if binding is None:
with self._media_file(source):
pass
reason = None
if binding:
try:
fields = {"model": binding.model}
if language:
fields["language"] = language
with self._media_file(source) as handle:
data, _ = await self._request(binding, data=fields,
files={"file": (source.name, handle, "application/octet-stream")})
text = data.get("text")
if not isinstance(text, str) or not text.strip():
raise invalid_response()
return RoutedTranscript(text=text, source="api")
except ProviderError as exc:
reason = exc.code
try:
text = await self.local_speech.transcribe(source, language)
if not isinstance(text, str) or not text.strip():
raise ProviderError("LOCAL_MODEL_INVALID_RESPONSE", "Local transcription was empty.")
return RoutedTranscript(text=text, source="local", fallback_reason=reason)
except ProviderError as exc:
raise ApiError(503, exc.code, exc.message, {"fallback_reason": reason}) from exc
async def match_speakers(self, source: Path, reference: Path) -> SpeakerMatchResult:
binding = self.configuration().speaker_matching
if binding is None:
with self._media_file(source), self._media_file(reference):
pass
reason = None
if binding:
try:
# Explicit application contract, not an OpenAI-standard endpoint.
with self._media_file(source) as audio, self._media_file(reference) as sample:
data, _ = await self._request(binding, data={"model": binding.model}, files={
"file": (source.name, audio, "application/octet-stream"),
"reference_file": (reference.name, sample, "application/octet-stream"),
})
score = data.get("score")
if not finite_number(score) or not 0 <= score <= 1:
raise invalid_response()
return SpeakerMatchResult(score=score, source="api")
except ProviderError as exc:
reason = exc.code
try:
score = await self.local_speech.match(source, reference)
if not finite_number(score) or not 0 <= score <= 1:
raise ProviderError("LOCAL_MODEL_INVALID_RESPONSE", "Local speaker matching was invalid.")
return SpeakerMatchResult(score=score, source="local", fallback_reason=reason)
except ProviderError as exc:
raise ApiError(503, exc.code, exc.message, {"fallback_reason": reason}) from exc
+47
View File
@@ -0,0 +1,47 @@
"""Keep internal namespaced tools compatible with providers' 64-character names."""
import hashlib
import re
from functools import wraps
from app.contracts import MessageRole, ModelRequest
def prepare_tool_names(request: ModelRequest) -> tuple[ModelRequest, dict[str, str]]:
names = {tool.name for tool in request.tools}
for message in request.messages:
names.update(call.name for call in message.tool_calls)
if message.role == MessageRole.tool and message.name:
names.add(message.name)
mapping = {name: name for name in names if re.fullmatch(r"[A-Za-z0-9_-]{1,64}", name)}
used = set(mapping)
for name in sorted(names - mapping.keys()):
salt = 0
while True:
alias = "tool_" + hashlib.sha256(f"{name}:{salt}".encode()).hexdigest()[:56]
if alias not in used:
break
salt += 1
mapping[name] = alias
used.add(alias)
if all(name == alias for name, alias in mapping.items()):
return request, {}
wire = request.model_copy(deep=True)
for tool in wire.tools:
tool.name = mapping[tool.name]
for message in wire.messages:
for call in message.tool_calls:
call.name = mapping[call.name]
if message.role == MessageRole.tool and message.name:
message.name = mapping[message.name]
return wire, {alias: name for name, alias in mapping.items()}
def mapped_tool_names(complete):
@wraps(complete)
async def wrapped(self, request: ModelRequest):
wire, originals = prepare_tool_names(request)
turn = await complete(self, wire)
for call in turn.tool_calls:
call.name = originals.get(call.name, call.name)
return turn
return wrapped
+141 -17
View File
@@ -63,6 +63,14 @@ class FtsHit:
bm25: float
@dataclass(frozen=True, slots=True)
class NoteLocation:
note_id: str
title: str
file_path: str
folder: str
def replace_note_metadata(
*,
conn: sqlite3.Connection,
@@ -219,11 +227,61 @@ def fts_search(match: str, limit: int = 100) -> list[FtsHit]:
conn.close()
def fts_search_page(
def list_note_locations(*, conn: sqlite3.Connection | None = None) -> list[NoteLocation]:
"""返回 Workspace 构树和目录事务所需的最小笔记位置集合。"""
owns = conn is None
conn = conn or connect()
try:
rows = conn.execute(
"SELECT note_id, title, file_path, folder FROM notes ORDER BY file_path"
).fetchall()
return [
NoteLocation(
note_id=row["note_id"],
title=row["title"],
file_path=row["file_path"],
folder=row["folder"],
)
for row in rows
]
finally:
if owns:
conn.close()
def update_note_location(
*,
conn: sqlite3.Connection,
note_id: str,
title: str,
file_path: str,
folder: str,
updated_at: datetime,
) -> None:
"""更新文件位置和展示标题;Block/FTS/向量内容不变,无需重新生成。"""
cursor = conn.execute(
"""
UPDATE notes
SET title = ?, file_path = ?, folder = ?, updated_at = ?
WHERE note_id = ?
""",
(title, file_path, folder, _iso(updated_at), note_id),
)
if cursor.rowcount != 1:
raise LookupError(note_id)
_FTS_FROM = """
FROM blocks_fts
JOIN blocks AS b ON b.block_id = blocks_fts.block_id
JOIN notes AS n ON n.note_id = b.note_id
"""
def _fts_where(
match: str,
limit: int,
offset: int,
folders: list[str],
note_ids: list[str],
tags: list[str],
@@ -231,8 +289,11 @@ def fts_search_page(
created_to: datetime | None,
updated_from: datetime | None,
updated_to: datetime | None,
) -> tuple[list[FtsHit], int]:
"""执行带元数据过滤的 FTS 精确分页,并返回过滤后的完整命中数。"""
) -> tuple[str, list[object]]:
"""构建 FTS 过滤 WHERE 子句(不含 WHERE 关键字),返回 (where_sql, params)。
fts_search_page fts_score_bounds 共用保证计数与取数口径一致
"""
where = ["blocks_fts MATCH ?"]
params: list[object] = [match]
@@ -263,22 +324,44 @@ def fts_search_page(
where.append(f"julianday({column}) <= julianday(?)")
params.append(_iso(upper))
from_sql = """
FROM blocks_fts
JOIN blocks AS b ON b.block_id = blocks_fts.block_id
JOIN notes AS n ON n.note_id = b.note_id
return " AND ".join(where), params
def fts_search_page(
*,
match: str,
limit: int,
offset: int,
folders: list[str],
note_ids: list[str],
tags: list[str],
created_from: datetime | None,
created_to: datetime | None,
updated_from: datetime | None,
updated_to: datetime | None,
bm25_max: float | None = None,
) -> tuple[list[FtsHit], int]:
"""执行带元数据过滤的 FTS 精确分页,并返回过滤后的完整命中数。
bm25_max 非空时按 bm25 截止值过滤用于阈值过滤的精确分页计数与取数同口径
"""
where_sql = " AND ".join(where)
where_sql, params = _fts_where(
match, folders, note_ids, tags,
created_from, created_to, updated_from, updated_to,
)
if bm25_max is not None:
where_sql += " AND bm25(blocks_fts) <= ?"
params.append(bm25_max)
conn = connect()
try:
total = conn.execute(
f"SELECT COUNT(*) {from_sql} WHERE {where_sql}", params
f"SELECT COUNT(*) {_FTS_FROM} WHERE {where_sql}", params
).fetchone()[0]
rows = conn.execute(
f"""
SELECT blocks_fts.block_id, blocks_fts.note_id, bm25(blocks_fts) AS rank
{from_sql}
{_FTS_FROM}
WHERE {where_sql}
ORDER BY rank
LIMIT ? OFFSET ?
@@ -294,6 +377,45 @@ def fts_search_page(
conn.close()
def fts_score_bounds(
*,
match: str,
folders: list[str],
note_ids: list[str],
tags: list[str],
created_from: datetime | None,
created_to: datetime | None,
updated_from: datetime | None,
updated_to: datetime | None,
) -> tuple[float, float] | None:
"""返回 metadata 过滤后的 FTS 命中集里 bm25 的 (min, max),无命中时返回 None。
用于阈值过滤min-max 归一化是 bm25 的线性函数据此可把阈值换算为 bm25 截止值
"""
where_sql, params = _fts_where(
match, folders, note_ids, tags,
created_from, created_to, updated_from, updated_to,
)
conn = connect()
try:
# bm25() 不能作为聚合函数参数,也不能用在被聚合的子查询里;改用 ORDER BY 取首尾两行
lo_row = conn.execute(
f"SELECT bm25(blocks_fts) AS rank {_FTS_FROM} WHERE {where_sql}"
" ORDER BY rank ASC LIMIT 1",
params,
).fetchone()
if lo_row is None or lo_row["rank"] is None:
return None
hi_row = conn.execute(
f"SELECT bm25(blocks_fts) AS rank {_FTS_FROM} WHERE {where_sql}"
" ORDER BY rank DESC LIMIT 1",
params,
).fetchone()
return (float(lo_row["rank"]), float(hi_row["rank"]))
finally:
conn.close()
def get_block_hits(block_ids: list[str]) -> list[BlockHit]:
if not block_ids:
return []
@@ -338,16 +460,18 @@ def get_index_meta() -> dict[str, str]:
conn.close()
def clear_all() -> None:
"""清空元数据、Block 与 FTS5(重建索引用,向量由 VectorStore.clear 处理)。"""
conn = connect()
def clear_all(*, conn: sqlite3.Connection | None = None) -> None:
"""Clear rebuildable metadata using the caller's transaction when provided."""
owns = conn is None
conn = conn or connect()
try:
with transaction(conn):
with transaction(conn) if owns else nullcontext():
conn.execute("DELETE FROM blocks_fts")
conn.execute("DELETE FROM blocks")
conn.execute("DELETE FROM notes")
finally:
conn.close()
if owns:
conn.close()
def stats() -> dict[str, int]:
+2
View File
@@ -19,6 +19,7 @@ class EmbeddingProvider(Protocol):
"""统一 Embedding 接口(与文档一致)。"""
model_id: str
version: str
dim: int
async def embed_documents(self, texts: list[str]) -> list[list[float]]: ...
@@ -33,6 +34,7 @@ class HashEmbeddingProvider:
"""
model_id = "hash-v1"
version = "1"
dim = EMBEDDING_DIM
async def embed_documents(self, texts: list[str]) -> list[list[float]]:
+90 -19
View File
@@ -22,6 +22,8 @@ from app.repository import BlockHit
from app.retrieval.embedding import EmbeddingProvider, HashEmbeddingProvider
from app.retrieval.hybrid import normalize_scores, rrf_fuse
from app.retrieval.reranker import LexicalReranker, RankedCandidate, RerankerProvider
from app.retrieval import routed_vectors
from app.retrieval.provenance import record_embedding
from app.retrieval.vectorstore import SqliteVecStore, VectorStore
from app.textutils import make_snippet, match_query
@@ -39,10 +41,15 @@ class RetrievalEngine:
embedding: EmbeddingProvider,
reranker: RerankerProvider,
vector_store: VectorStore,
*,
route_embeddings: bool = False,
) -> None:
self.embedding = embedding
self.reranker = reranker
self.vector_store = vector_store
# Only the production instance opts in. Replaced test dependencies must
# remain authoritative, including monkeypatches on the singleton.
self._routed_defaults = (embedding, vector_store) if route_embeddings else None
async def search(self, request: SearchRequest) -> SearchResponse:
if request.mode == SearchMode.fts:
@@ -56,7 +63,7 @@ class RetrievalEngine:
# 候选池至少覆盖本次请求的 offset+limit,保证分页能取到目标页;设上限防内存失控
window = min(request.offset + request.limit, MAX_CANDIDATE_POOL)
pool_size = max(CANDIDATE_POOL, window)
# 带过滤时放大召回;FTS 则一次性取全量命中(≤FTS_FETCH_LIMIT)避免截断漏召回
# 带过滤时放大召回,缓解「先截断候选池再过滤」造成的漏召回
recall = min(pool_size * OVERSCAN_FACTOR, MAX_CANDIDATE_POOL) if has_filters else pool_size
# 1. 按模式收集候选(FTS 与 Vector 各产出「按相关性降序」的 block_id 列表)
@@ -74,8 +81,19 @@ class RetrievalEngine:
fts_scores = {h.block_id: -h.bm25 for h in fts_hits}
if request.mode in (SearchMode.vector, SearchMode.hybrid):
query_vec = await self.embedding.embed_query(request.query)
vec_hits = await self.vector_store.search(query_vec, top_k=recall)
record_embedding(source="unavailable")
vec_hits = None
if (
self._routed_defaults is not None
and self.embedding is self._routed_defaults[0]
and self.vector_store is self._routed_defaults[1]
):
vec_hits = await routed_vectors.search_remote(request.query, top_k=recall)
if vec_hits is None:
query_vec = await self.embedding.embed_query(request.query)
vec_hits = await self.vector_store.search(query_vec, top_k=recall)
record_embedding(source="local", model_id=self.embedding.model_id,
dimensions=self.embedding.dim, version=self.embedding.version)
vec_ranked = [v.id for v in vec_hits]
vec_scores = {v.id: v.score for v in vec_hits}
@@ -84,7 +102,7 @@ class RetrievalEngine:
elif request.mode == SearchMode.vector:
candidate_scores = vec_scores
else: # hybridRRF 融合
candidate_scores = rrf_fuse([fts_ranked, vec_ranked])
candidate_scores = rrf_fuse([fts_ranked, vec_ranked], k=request.rrf_k)
if not candidate_scores:
return self._empty(request)
@@ -97,14 +115,23 @@ class RetrievalEngine:
if not filtered:
return self._empty(request)
# 4. 排序 / 精排
# 4. 排序 / 精排:hybrid 先按融合分预排序,再对前 rerank_candidates 个候选做精排,
# 剩余候选按融合分排在精排结果之后;rerank=False 时跳过精排直接按融合分排序。
if request.mode == SearchMode.hybrid:
candidates = [
RankedCandidate(block_id=h.block_id, score=candidate_scores[h.block_id], text=h.content)
for h in filtered
]
ranked = await self.reranker.rerank(request.query, candidates)
ordered = [(c.block_id, c.score) for c in ranked]
pre_sorted = sorted(filtered, key=lambda h: -candidate_scores[h.block_id])
if request.rerank:
limit = request.rerank_candidates
pool = pre_sorted if limit is None else pre_sorted[:limit]
rest = [] if limit is None else pre_sorted[limit:]
candidates = [
RankedCandidate(block_id=h.block_id, score=candidate_scores[h.block_id], text=h.content)
for h in pool
]
ranked = await self.reranker.rerank(request.query, candidates)
ordered = [(c.block_id, c.score) for c in ranked]
ordered += [(h.block_id, candidate_scores[h.block_id]) for h in rest]
else:
ordered = [(h.block_id, candidate_scores[h.block_id]) for h in pre_sorted]
else:
ordered = sorted(
((h.block_id, candidate_scores[h.block_id]) for h in filtered),
@@ -112,8 +139,10 @@ class RetrievalEngine:
)
ordered = normalize_scores(ordered)
# score_threshold:归一化后过滤低分结果(默认 0 不过滤)
ordered = [(bid, score) for bid, score in ordered if score >= request.score_threshold]
# 5. 分页:total = 过滤后候选集大小。fts 已取全量(≤FTS_FETCH_LIMIT)故为真实命中数;
# 5. 分页:total = 过滤后候选集大小。fts 走数据库精确分页,total 为真实命中数;
# vector/hybrid 为 KNN 候选集,无全局 total。
total = len(ordered)
page = ordered[request.offset : request.offset + request.limit]
@@ -126,11 +155,41 @@ class RetrievalEngine:
)
def _search_fts(self, request: SearchRequest) -> SearchResponse:
"""FTS 专用路径:过滤、COUNT 与分页全部在 SQLite 中完成。"""
"""FTS 专用路径:在数据库侧完成过滤、计数与分页,不取全量后再截断。
阈值过滤时min-max 归一化是 bm25 的线性函数据此把 score_threshold 换算为
bm25 截止值bm25_max使过滤计数与分页口径一致无阈值时走数据库原生分页
total 始终为过滤后的真实命中数不再受固定截断影响
"""
match = match_query(request.query)
if not match:
return self._empty(request)
bounds = repository.fts_score_bounds(
match=match,
folders=request.folders,
note_ids=request.note_ids,
tags=request.tags,
created_from=request.created_from,
created_to=request.created_to,
updated_from=request.updated_from,
updated_to=request.updated_to,
)
if bounds is None:
return self._empty(request)
lo, hi = bounds
span = hi - lo
bm25_max: float | None = None
if request.score_threshold > 0:
if span == 0:
# 全部命中 bm25 相同,归一化后皆为 1.0;阈值超过 1.0 时无命中
if request.score_threshold > 1.0:
return self._empty(request)
else:
# norm = (hi - bm25) / spannorm >= threshold ⟺ bm25 <= hi - threshold * span
bm25_max = hi - request.score_threshold * span
fts_hits, total = repository.fts_search_page(
match=match,
limit=request.limit,
@@ -142,19 +201,29 @@ class RetrievalEngine:
created_to=request.created_to,
updated_from=request.updated_from,
updated_to=request.updated_to,
bm25_max=bm25_max,
)
if not fts_hits:
# 本页无结果:offset 越过末页时 total 仍为真实命中数(>0),需保留而非归零
return SearchResponse(
query=request.query,
mode=request.mode,
items=[],
page=PageMeta(total=total, limit=request.limit, offset=request.offset),
)
hits = {h.block_id: h for h in repository.get_block_hits([hit.block_id for hit in fts_hits])}
ordered = normalize_scores(
[(hit.block_id, -hit.bm25) for hit in fts_hits if hit.block_id in hits]
)
items = [self._build_result(hits[block_id], request, score) for block_id, score in ordered]
# 分数按全局 bm25 上下界归一化(与取全量后 normalize_scores 等价),保证跨页一致
span = hi - lo
if span == 0:
ordered = [(hit.block_id, 1.0) for hit in fts_hits]
else:
ordered = [(hit.block_id, round((hi - hit.bm25) / span, 6)) for hit in fts_hits]
hits = {h.block_id: h for h in repository.get_block_hits([bid for bid, _ in ordered])}
items = [
self._build_result(hits[block_id], request, score)
for block_id, score in ordered
if block_id in hits
]
return SearchResponse(
query=request.query,
mode=request.mode,
@@ -217,4 +286,6 @@ def _utc(dt: datetime) -> datetime:
# 默认引擎实例:轻量实现跑通链路,后续可替换真实模型实现
engine = RetrievalEngine(HashEmbeddingProvider(), LexicalReranker(), SqliteVecStore())
engine = RetrievalEngine(
HashEmbeddingProvider(), LexicalReranker(), SqliteVecStore(), route_embeddings=True,
)
+21
View File
@@ -0,0 +1,21 @@
"""Task-local observations of the embedding path actually used by a search."""
from contextlib import contextmanager
from contextvars import ContextVar
_observation: ContextVar[dict | None] = ContextVar("embedding_observation", default=None)
@contextmanager
def capture_embedding():
result = {"source": "not_used"}
token = _observation.set(result)
try:
yield result
finally:
_observation.reset(token)
def record_embedding(**fields) -> None:
result = _observation.get()
if result is not None:
result.update(fields)
+2
View File
@@ -24,6 +24,7 @@ class RerankerProvider(Protocol):
"""统一 Reranker 接口:输入候选块,输出按相关性重排后的候选块。"""
model_id: str
version: str
async def rerank(self, query: str, candidates: list[RankedCandidate]) -> list[RankedCandidate]: ...
@@ -32,6 +33,7 @@ class LexicalReranker:
"""轻量精排:query 与块正文的词面重叠度,与归一化后的原始分数加权求和。"""
model_id = "lexical-v1"
version = "1"
def __init__(self, lexical_weight: float = 0.5) -> None:
self.lexical_weight = lexical_weight
+201
View File
@@ -0,0 +1,201 @@
"""Optional API embeddings, isolated from the stable hash/sqlite-vec index.
The runtime's model_id is the authoritative space ID (including provider URL,
endpoint, model and dimensions); equal dimensions alone never imply compatibility.
This phase uses a lazy, rebuildable SQLite side table instead of a schema migration.
Search scans only current blocks in one database snapshot and requires complete
coverage. Cosine ranking costs O(blocks * dimensions) with an O(top_k) heap; this
small-vault implementation should become a per-space ANN index at larger scale.
"""
from __future__ import annotations
import heapq
import json
import logging
import math
import sqlite3
from dataclasses import dataclass
from typing import Protocol
from app.database.db import connect, transaction
from app.retrieval.vectorstore import VectorHit
from app.retrieval.provenance import record_embedding
logger = logging.getLogger(__name__)
class EmbeddingResult(Protocol):
vectors: list[list[float]]
source: str
model_id: str
dimensions: int
fallback_reason: str | None
class EmbeddingRuntime(Protocol):
async def embed(self, texts: list[str]) -> EmbeddingResult: ...
@dataclass(frozen=True)
class RemoteEmbeddings:
space_id: str
dimensions: int
vectors: list[list[float]]
def get_model_routing() -> EmbeddingRuntime | None:
"""Lazy integration hook; tests can inject a runtime without any network I/O."""
from app.container import container
return getattr(container, "model_routing", None)
def _unit_vector(vector: list[float], dimensions: int) -> list[float]:
if len(vector) != dimensions:
raise ValueError("embedding dimension mismatch")
if any(isinstance(value, bool) or not isinstance(value, (int, float)) for value in vector):
raise ValueError("embedding must be numeric")
if not all(math.isfinite(value) for value in vector):
raise ValueError("embedding must be finite")
scale = max(abs(value) for value in vector)
if scale == 0:
raise ValueError("embedding must be nonzero")
# Scaling first avoids overflow/underflow for finite but extreme API values.
scaled = [value / scale for value in vector]
norm = math.sqrt(math.fsum(value * value for value in scaled))
return [value / norm for value in scaled]
async def embed_remote(texts: list[str]) -> RemoteEmbeddings | None:
"""Return validated API vectors, or None to use the caller's local baseline.
Do not use the runtime's local result: the caller may have injected its own
embedding/store pair. Exception deliberately excludes cancellation.
"""
if not texts:
return None
try:
runtime = get_model_routing()
if runtime is None:
return None
result = await runtime.embed(texts)
if result.source != "api":
record_embedding(fallback_reason=result.fallback_reason)
return None
if not isinstance(result.model_id, str) or not result.model_id or result.model_id == "hash-v1":
raise ValueError("API embedding needs a distinct space ID")
if type(result.dimensions) is not int or result.dimensions <= 0:
raise ValueError("invalid embedding dimensions")
if len(result.vectors) != len(texts):
raise ValueError("embedding count mismatch")
return RemoteEmbeddings(
space_id=result.model_id,
dimensions=result.dimensions,
vectors=[_unit_vector(vector, result.dimensions) for vector in result.vectors],
)
except Exception as exc:
# Avoid logging provider exceptions containing credentials or note text.
record_embedding(fallback_reason="REMOTE_EMBEDDING_UNAVAILABLE")
logger.warning("Remote embedding unavailable (%s); using local index", type(exc).__name__)
return None
def _ensure_table(conn: sqlite3.Connection) -> None:
conn.execute("""
CREATE TABLE IF NOT EXISTS routed_block_vectors (
space_id TEXT NOT NULL,
block_id TEXT NOT NULL REFERENCES blocks(block_id) ON DELETE CASCADE,
dimensions INTEGER NOT NULL CHECK (dimensions > 0),
vector TEXT NOT NULL,
PRIMARY KEY (space_id, block_id)
)
""")
conn.execute("""
CREATE INDEX IF NOT EXISTS routed_block_vectors_block_id
ON routed_block_vectors(block_id)
""")
def store_remote(
conn: sqlite3.Connection, block_ids: list[str], batch: RemoteEmbeddings | None,
) -> None:
"""Best-effort side-index write inside the caller's metadata transaction.
A savepoint prevents partial remote batches and isolates storage failures from
note saving. Replacing/deleting blocks cascades all old spaces automatically.
"""
if batch is None:
return
try:
conn.execute("SAVEPOINT routed_vectors_write")
try:
if len(block_ids) != len(batch.vectors):
raise ValueError("block/vector count mismatch")
_ensure_table(conn)
conn.executemany(
"""INSERT INTO routed_block_vectors (space_id, block_id, dimensions, vector)
VALUES (?, ?, ?, ?)
ON CONFLICT (space_id, block_id) DO UPDATE SET
dimensions = excluded.dimensions, vector = excluded.vector""",
[
(batch.space_id, block_id, batch.dimensions, json.dumps(vector, allow_nan=False))
for block_id, vector in zip(block_ids, batch.vectors)
],
)
except BaseException:
conn.execute("ROLLBACK TO routed_vectors_write")
raise
finally:
conn.execute("RELEASE routed_vectors_write")
except Exception as exc:
logger.warning("Remote vector storage unavailable (%s); local index retained", type(exc).__name__)
async def search_remote(query: str, *, top_k: int) -> list[VectorHit] | None:
"""None means fallback, including any missing/invalid current-block vector.
Read coverage and vectors together so concurrent note updates cannot produce
an apparently complete subset. Never fill missing remote hits with local hits.
"""
batch = await embed_remote([query])
if batch is None:
return None
record_embedding(attempted_space={"model_id": batch.space_id, "dimensions": batch.dimensions})
try:
conn = connect()
try:
with transaction(conn):
exists = conn.execute(
"SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = 'routed_block_vectors'"
).fetchone()
if exists is None:
record_embedding(fallback_reason="REMOTE_INDEX_MISSING")
return None
rows = conn.execute(
"""SELECT b.block_id, r.vector
FROM blocks AS b
LEFT JOIN routed_block_vectors AS r
ON r.block_id = b.block_id AND r.space_id = ? AND r.dimensions = ?
ORDER BY b.block_id""",
(batch.space_id, batch.dimensions),
)
def hits():
for row in rows:
if row["vector"] is None:
raise ValueError("remote space has incomplete block coverage")
vector = _unit_vector(json.loads(row["vector"]), batch.dimensions)
score = math.fsum(a * b for a, b in zip(batch.vectors[0], vector))
yield VectorHit(id=row["block_id"], score=max(0.0, min(1.0, score)))
result = heapq.nlargest(top_k, hits(), key=lambda hit: hit.score)
record_embedding(source="api", model_id=batch.space_id,
dimensions=batch.dimensions, fallback_reason=None)
return result
finally:
conn.close()
except Exception as exc:
record_embedding(fallback_reason="REMOTE_INDEX_UNAVAILABLE")
logger.debug("Remote vector search unavailable (%s); using local index", type(exc).__name__)
return None
+13 -3
View File
@@ -35,6 +35,7 @@ class VectorStore(Protocol):
async def upsert(self, records: list[VectorRecord]) -> None: ...
async def delete(self, ids: list[str]) -> None: ...
async def search(self, vector: list[float], *, top_k: int) -> list[VectorHit]: ...
async def count(self) -> int: ...
class SqliteVecStore:
@@ -85,10 +86,19 @@ class SqliteVecStore:
finally:
conn.close()
async def clear(self) -> None:
conn = connect()
async def clear(self, *, conn: sqlite3.Connection | None = None) -> None:
owns = conn is None
conn = conn or connect()
try:
with transaction(conn):
with transaction(conn) if owns else nullcontext():
conn.execute("DELETE FROM vec_blocks")
finally:
if owns:
conn.close()
async def count(self) -> int:
conn = connect()
try:
return conn.execute("SELECT COUNT(*) FROM vec_blocks").fetchone()[0]
finally:
conn.close()
+816 -40
View File
File diff suppressed because it is too large Load Diff
+60
View File
@@ -0,0 +1,60 @@
"""共享 JSON Schema 安全约束。"""
from __future__ import annotations
from typing import Any
from urllib.parse import urljoin
from referencing import Registry
from referencing.exceptions import Unresolvable
from referencing.jsonschema import DRAFT202012
_SCHEMA_BASE_URI = "https://notesagent.invalid/local-schema"
class SchemaReferenceError(ValueError):
"""Schema 引用不符合宿主的离线、文档内解析约束。"""
class ExternalSchemaReferenceError(SchemaReferenceError):
def __init__(self, keyword: str, reference: Any) -> None:
super().__init__(f"External JSON Schema reference is not allowed: {reference!r}")
self.keyword = keyword
self.reference = reference
class UnresolvableLocalSchemaReferenceError(SchemaReferenceError):
def __init__(self, reference: str) -> None:
super().__init__(f"Local JSON Schema reference cannot be resolved: {reference!r}")
self.reference = reference
def reject_external_schema_references(schema: Any) -> None:
"""只允许可解析的文档内 Fragment,并按 JSON Schema Resource 作用域解析。"""
root = DRAFT202012.create_resource(schema)
root_uri = urljoin(_SCHEMA_BASE_URI, root.id() or "")
registry = Registry().with_resource(_SCHEMA_BASE_URI, root).crawl()
resolver = registry.resolver(root_uri)
_validate_resource_references(root, resolver)
def _validate_resource_references(resource, resolver: Any) -> None:
contents = resource.contents
if isinstance(contents, dict):
for keyword in ("$ref", "$dynamicRef"):
if keyword not in contents:
continue
reference = contents[keyword]
if not isinstance(reference, str) or not reference.startswith("#"):
raise ExternalSchemaReferenceError(keyword, reference)
try:
resolver.lookup(reference)
except Unresolvable as exc:
raise UnresolvableLocalSchemaReferenceError(reference) from exc
for subresource in resource.subresources():
_validate_resource_references(
subresource,
resolver.in_subresource(subresource),
)
+24 -25
View File
@@ -6,7 +6,6 @@ MVP 阶段重建是同步的(数据量小),完成后直接返回 completed
from __future__ import annotations
import shutil
from datetime import datetime, timezone
from pathlib import Path
from uuid import uuid4
@@ -16,8 +15,8 @@ from app.config import get_settings
from app.contracts import IndexJob, IndexRebuildRequest, IndexStatus
from app.errors import ApiError
from app.knowledge.parser import parse_note
from app.services.note_service import index_note
from app.services import task_service
from app.services.note_service import index_note, prepare_note_index
from app.database.db import connect, transaction
from app.services.coordination import serialized_vault_mutation
from app.retrieval.vectorstore import SqliteVecStore
@@ -74,18 +73,7 @@ async def rebuild(request: IndexRebuildRequest) -> IndexJob:
{"scope": request.scope, "note_ids": request.note_ids},
)
# 先扫描到内存(失败不会清旧索引),再快照旧库用于失败回滚
docs = _scan_vault()
settings = get_settings()
database_existed = settings.db_path.exists()
task_note_links = task_service.note_links() if database_existed else {}
backup_path = (
settings.db_path.with_name(f"{settings.db_path.name}.{job_id}.bak")
if database_existed
else None
)
if backup_path is not None:
shutil.copy2(settings.db_path, backup_path)
_active_job_id = job_id
_last_error = None
@@ -94,21 +82,34 @@ async def rebuild(request: IndexRebuildRequest) -> IndexJob:
created_at=datetime.now(timezone.utc),
))
try:
repository.clear_all()
await vector_store.clear()
prepared_notes = []
for rel, folder, markdown, created, updated in docs:
parsed = parse_note(
markdown=markdown, file_path=rel, folder=folder, tags=None,
created_at=created, updated_at=updated,
)
await index_note(parsed)
task_service.restore_note_links(task_note_links)
prepared_notes.append((parsed, await prepare_note_index(parsed)))
# All network/model awaits precede the transaction. The concrete SQLite
# methods below complete synchronously despite their async interfaces.
conn = connect()
try:
with transaction(conn):
task_note_links = dict(conn.execute(
"SELECT task_id, note_id FROM tasks WHERE note_id IS NOT NULL"
).fetchall())
repository.clear_all(conn=conn)
await vector_store.clear(conn=conn)
for parsed, prepared in prepared_notes:
await index_note(parsed, prepared=prepared, conn=conn)
for task_id, note_id in task_note_links.items():
conn.execute(
"UPDATE tasks SET note_id = ? WHERE task_id = ? "
"AND EXISTS (SELECT 1 FROM notes WHERE note_id = ?)",
(note_id, task_id, note_id),
)
finally:
conn.close()
except BaseException as exc:
# 重建失败:恢复旧索引,避免留下半成品;记录 failed 任务后向上抛
if backup_path is not None and backup_path.exists():
shutil.copy2(backup_path, settings.db_path)
elif not database_existed:
settings.db_path.unlink(missing_ok=True)
_remember_job(IndexJob(
job_id=job_id, status="failed", scope=request.scope,
created_at=datetime.now(timezone.utc),
@@ -117,8 +118,6 @@ async def rebuild(request: IndexRebuildRequest) -> IndexJob:
raise
finally:
_active_job_id = None
if backup_path is not None:
backup_path.unlink(missing_ok=True)
job = IndexJob(job_id=job_id, status="completed", scope=request.scope, created_at=datetime.now(timezone.utc))
_remember_job(job)
+99 -57
View File
@@ -6,88 +6,55 @@ Markdown 文件是笔记正文的持久化载体(Vault),SQLite/FTS5/向量
from __future__ import annotations
import re
import sqlite3
from contextlib import nullcontext
from datetime import datetime, timezone
from pathlib import Path
from uuid import uuid4
from app import repository
from app.config import get_settings
from app.contracts import Note, NoteBlock, NoteSummary
from app.database.db import connect, transaction
from app.errors import ApiError
from app.knowledge.parser import ParsedNote, parse_note
from app.retrieval.embedding import HashEmbeddingProvider
from app.retrieval import routed_vectors
from app.retrieval.vectorstore import SqliteVecStore, VectorRecord
from app.services.coordination import serialized_vault_mutation
from app.services.vault_paths import (
normalize_entry_name,
normalize_folder,
resolve_in_vault,
safe_note_filename,
)
# 轻量实现实例(无状态,可直接复用);接入真实模型后替换为对应 Provider
embedding = HashEmbeddingProvider()
vector_store = SqliteVecStore()
def _vault() -> Path:
return get_settings().vault_path
def _safe_name(title: str) -> str:
name = re.sub(r'[\\/:*?"<>|]', "_", title).strip()
return name or "untitled"
def _normalize_folder(folder: str | None) -> str:
"""清洗 folder 为安全的相对目录,拒绝 `..`/`.`/绝对路径/盘符/空字节,防路径逃逸。"""
if not folder:
return ""
if "\x00" in folder:
raise ApiError(400, "INVALID_PATH", "folder must not contain NUL bytes", {"folder": folder})
segments: list[str] = []
for part in re.split(r"[\\/]+", folder):
if part == "":
continue
if part in (".", ".."):
raise ApiError(400, "INVALID_PATH", "folder must not contain '.' or '..'", {"folder": folder})
if ":" in part:
raise ApiError(400, "INVALID_PATH", "folder must be a relative path", {"folder": folder})
segments.append(part)
return "/".join(segments)
def _rel_path(folder: str | None, title: str) -> tuple[str, str]:
"""由 folder + title 生成安全的相对路径,返回 (rel_path, 清洗后的 folder)。"""
clean_folder = _normalize_folder(folder)
name = _safe_name(title)
if not name.endswith(".md"):
name += ".md"
clean_folder = normalize_folder(folder)
name = safe_note_filename(title)
rel = f"{clean_folder}/{name}" if clean_folder else name
return rel, clean_folder
def _abs_path(rel_path: str) -> Path:
"""把相对路径解析为 Vault 内的绝对路径;越界即报 400,杜绝路径逃逸。"""
if not rel_path or "\x00" in rel_path:
raise ApiError(400, "INVALID_PATH", "invalid file path", {"file_path": rel_path})
root = _vault().resolve()
candidate = (_vault() / rel_path).resolve()
if not candidate.is_relative_to(root):
raise ApiError(400, "INVALID_PATH", "path escapes vault", {"file_path": rel_path})
return candidate
def _read_markdown(rel_path: str) -> str:
path = _abs_path(rel_path)
path = resolve_in_vault(rel_path)
return path.read_text(encoding="utf-8") if path.exists() else ""
def _write_markdown(rel_path: str, markdown: str) -> None:
path = _abs_path(rel_path)
path = resolve_in_vault(rel_path)
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(markdown, encoding="utf-8")
def _create_markdown(rel_path: str, markdown: str) -> None:
"""排他创建 Markdown;目标已存在时返回资源冲突,不覆盖用户文件。"""
path = _abs_path(rel_path)
path = resolve_in_vault(rel_path)
path.parent.mkdir(parents=True, exist_ok=True)
try:
with path.open("x", encoding="utf-8") as handle:
@@ -102,22 +69,39 @@ def _create_markdown(rel_path: str, markdown: str) -> None:
def _delete_markdown(rel_path: str) -> None:
path = _abs_path(rel_path)
path = resolve_in_vault(rel_path)
if path.exists():
path.unlink()
async def index_note(parsed: ParsedNote) -> None:
PreparedIndex = tuple[list[list[float]], routed_vectors.RemoteEmbeddings | None]
async def prepare_note_index(parsed: ParsedNote) -> PreparedIndex:
"""Compute vectors before opening a write transaction (including API I/O)."""
texts = [block.content for block in parsed.blocks]
vectors = await embedding.embed_documents(texts)
remote = await routed_vectors.embed_remote(texts)
return vectors, remote
async def index_note(
parsed: ParsedNote, *, prepared: PreparedIndex | None = None,
conn: sqlite3.Connection | None = None,
) -> None:
"""把解析结果写入元数据 + FTS5 + 向量(三层可重建索引),单事务保证原子性。
元数据与向量在同一连接同一事务内提交避免新元数据已提交向量写入失败
半提交状态替换元数据时拿到旧 block_id清理已删除/内容变化的旧向量只为新增
block 写向量内容未变的 block 其向量仍有效无需重复写入
"""
vectors = await embedding.embed_documents([block.content for block in parsed.blocks])
conn = connect()
if conn is not None and prepared is None:
raise ValueError("Prepare embeddings before supplying a write connection")
vectors, remote = prepared if prepared is not None else await prepare_note_index(parsed)
owns = conn is None
conn = conn or connect()
try:
with transaction(conn):
with transaction(conn) if owns else nullcontext():
old_block_ids = repository.replace_note_metadata(
conn=conn,
note_id=parsed.note_id,
@@ -141,12 +125,14 @@ async def index_note(parsed: ParsedNote) -> None:
if block.block_id in missing_ids
]
await vector_store.upsert(records, conn=conn)
routed_vectors.store_remote(conn, [block.block_id for block in parsed.blocks], remote)
repository.set_index_meta(
{"embedding_model": embedding.model_id, "embedding_dim": str(embedding.dim)},
conn=conn,
)
finally:
conn.close()
if owns:
conn.close()
@serialized_vault_mutation
@@ -222,7 +208,7 @@ async def move_note(note_id: str, *, folder: str) -> Note:
if record is None:
raise ApiError(404, "RESOURCE_NOT_FOUND", "note not found", {"note_id": note_id})
clean_folder = _normalize_folder(folder)
clean_folder = normalize_folder(folder)
filename = Path(record.file_path).name
new_rel_path = f"{clean_folder}/{filename}" if clean_folder else filename
if new_rel_path == record.file_path:
@@ -230,8 +216,8 @@ async def move_note(note_id: str, *, folder: str) -> Note:
assert note is not None
return note
source = _abs_path(record.file_path)
target = _abs_path(new_rel_path)
source = resolve_in_vault(record.file_path)
target = resolve_in_vault(new_rel_path)
if not source.is_file():
raise ApiError(
409, "NOTE_FILE_MISSING", "note file is missing from the Vault",
@@ -267,13 +253,69 @@ async def move_note(note_id: str, *, folder: str) -> Note:
)
@serialized_vault_mutation
async def rename_note(note_id: str, *, file_name: str) -> Note:
"""重命名 Markdown 文件并保留 note_id、Block 与向量身份。"""
record = repository.get_note_record(note_id)
if record is None:
raise ApiError(404, "RESOURCE_NOT_FOUND", "note not found", {"note_id": note_id})
normalized = normalize_entry_name(file_name, markdown=True)
source = resolve_in_vault(record.file_path)
folder = normalize_folder(record.folder)
new_file_path = f"{folder}/{normalized}" if folder else normalized
target = resolve_in_vault(new_file_path)
if new_file_path == record.file_path:
note = await get_note(note_id)
assert note is not None
return note
if not source.is_file():
raise ApiError(
409,
"NOTE_FILE_MISSING",
"note file is missing from the Vault",
{"note_id": note_id, "file_path": record.file_path},
)
if target.exists():
raise ApiError(
409,
"RESOURCE_CONFLICT",
"a note already exists with the requested file name",
{"note_id": note_id, "file_path": new_file_path},
)
source.replace(target)
now = datetime.now(timezone.utc)
conn = connect()
try:
with transaction(conn):
repository.update_note_location(
conn=conn,
note_id=note_id,
title=Path(normalized).stem,
file_path=new_file_path,
folder=folder,
updated_at=now,
)
except BaseException:
target.replace(source)
raise
finally:
conn.close()
note = await get_note(note_id)
assert note is not None
return note
@serialized_vault_mutation
async def delete_note(note_id: str) -> bool:
record = repository.get_note_record(note_id)
if record is None:
return False
path = _abs_path(record.file_path)
path = resolve_in_vault(record.file_path)
tombstone = path.with_name(f".{path.name}.{uuid4().hex}.deleting") if path.exists() else None
if tombstone is not None:
path.replace(tombstone)
+37 -12
View File
@@ -1,34 +1,59 @@
"""转写作业:API 优先,本地模型回退;保留已有 Host 文本入口。"""
from __future__ import annotations
from collections import OrderedDict
from datetime import datetime, timezone
from pathlib import Path
from uuid import uuid4
from app.contracts import TranscriptionJob
from app.errors import ApiError
from app.services.attachment_service import attachment_path
_jobs: OrderedDict[str, TranscriptionJob] = OrderedDict()
MAX_JOBS = 100
def create_transcription(attachment_id: str, language: str | None = None) -> TranscriptionJob:
del language # 预生成 transcript 暂不需要语言识别。
async def create_transcription(attachment_id: str, language: str | None = None, *, diarization: bool = False) -> TranscriptionJob:
from app.container import container
source = attachment_path(attachment_id)
transcript = source if source.suffix.lower() in {".txt", ".md"} else Path(f"{source}.txt")
job = TranscriptionJob(
job_id=f"transcription_{uuid4().hex}",
attachment_id=attachment_id,
status="completed" if transcript.is_file() else "failed",
text=transcript.read_text(encoding="utf-8") if transcript.is_file() else None,
error_code=None if transcript.is_file() else "TRANSCRIPTION_BACKEND_UNAVAILABLE",
error_message=(
None
if transcript.is_file()
else "No host-generated transcript is available; local speech models are phase two."
),
status="processing",
created_at=datetime.now(timezone.utc),
)
try:
if diarization:
# Speaker verification and diarization are different capabilities.
raise ApiError(501, "DIARIZATION_NOT_IMPLEMENTED", "说话人分离将在阶段 F 接入,当前不能忽略 diarization 请求。")
transcript = source if source.suffix.lower() in {".txt", ".md"} else attachment_path(f"{attachment_id}.txt")
# A saved transcript remains an explicit import path, never faked ASR.
if transcript.is_file() and (source == transcript or container.model_routing.configuration().transcription is None):
with transcript.open("rb") as handle:
content = handle.read(1024 * 1024 + 1)
if len(content) > 1024 * 1024:
raise ApiError(413, "TRANSCRIPT_TOO_LARGE", "Transcript exceeds 1 MiB.")
job.text = content.decode("utf-8")
if not job.text.strip():
raise ApiError(422, "TRANSCRIPT_EMPTY", "Transcript is empty.")
job.source = "sidecar"
else:
result = await container.model_routing.transcribe(source, language)
job.text = result.text
job.source = result.source
job.fallback_reason = result.fallback_reason
job.status = "completed"
except ApiError as exc:
job.status = "failed"
job.error_code = exc.code
job.error_message = exc.message
job.fallback_reason = exc.details.get("fallback_reason")
except (OSError, UnicodeError):
job.status = "failed"
job.error_code = "TRANSCRIPT_UNREADABLE"
job.error_message = "Transcript could not be read."
_jobs[job.job_id] = job
while len(_jobs) > MAX_JOBS:
_jobs.popitem(last=False)
+78
View File
@@ -0,0 +1,78 @@
"""Vault 相对路径校验;所有文件操作必须先经过本模块。"""
from __future__ import annotations
import re
from pathlib import Path
from app.config import get_settings
from app.errors import ApiError
_INVALID_FILE_CHARS = re.compile(r'[\\/:*?"<>|]')
def normalize_folder(folder: str | None) -> str:
"""返回使用 `/` 的安全相对目录;根目录表示为空字符串。"""
if not folder or folder in {"/", "\\"}:
return ""
if "\x00" in folder:
raise ApiError(400, "INVALID_PATH", "folder must not contain NUL bytes")
segments: list[str] = []
for part in re.split(r"[\\/]+", folder):
if not part:
continue
if part in {".", ".."} or ":" in part:
raise ApiError(
400,
"INVALID_PATH",
"folder must be a relative path without '.' or '..' segments",
{"folder": folder},
)
segments.append(part)
return "/".join(segments)
def normalize_entry_name(name: str, *, markdown: bool = False) -> str:
"""校验单个目录项名称;不静默接受路径分隔符或保留段。"""
value = name.strip()
if not value or value in {".", ".."} or "\x00" in value:
raise ApiError(400, "INVALID_PATH", "entry name is invalid", {"name": name})
if _INVALID_FILE_CHARS.search(value):
raise ApiError(
400,
"INVALID_PATH",
"entry name contains unsupported characters",
{"name": name},
)
if markdown and not value.lower().endswith(".md"):
value += ".md"
return value
def safe_note_filename(title: str) -> str:
"""为创建笔记保留原有的宽松清洗行为。"""
value = _INVALID_FILE_CHARS.sub("_", title).strip() or "untitled"
return value if value.lower().endswith(".md") else f"{value}.md"
def resolve_in_vault(relative_path: str) -> Path:
"""把相对路径解析到当前 Vault,并拒绝符号链接/`..` 导致的越界。"""
if not relative_path or "\x00" in relative_path:
raise ApiError(
400, "INVALID_PATH", "invalid Vault-relative path", {"path": relative_path}
)
root = get_settings().vault_path.resolve()
candidate = (root / relative_path.replace("\\", "/").lstrip("/")).resolve()
if not candidate.is_relative_to(root):
raise ApiError(
400, "INVALID_PATH", "path escapes Vault", {"path": relative_path}
)
return candidate
def relative_to_vault(path: Path) -> str:
return path.resolve().relative_to(get_settings().vault_path.resolve()).as_posix()
+249
View File
@@ -0,0 +1,249 @@
"""Web 联调 Workspace:把单一配置 Vault 映射为前端可用的真实文件树。"""
from __future__ import annotations
import hashlib
import shutil
from datetime import datetime, timezone
from pathlib import Path
from uuid import uuid4
from app import repository
from app.config import get_settings
from app.contracts import (
IndexRebuildRequest,
OperationResponse,
WorkspaceEntry,
WorkspaceInfo,
WorkspaceSnapshot,
)
from app.database.db import connect, transaction
from app.errors import ApiError
from app.retrieval.vectorstore import SqliteVecStore
from app.services import index_service
from app.services.coordination import serialized_vault_mutation
from app.services.vault_paths import normalize_entry_name, normalize_folder, resolve_in_vault
vector_store = SqliteVecStore()
def _entry_id(kind: str, path: str) -> str:
digest = hashlib.sha256(f"{kind}:{path}".encode("utf-8")).hexdigest()[:16]
return f"{kind}_{digest}"
def _disk_markdown_paths() -> set[str]:
root = get_settings().vault_path
if not root.exists():
return set()
resolved_root = root.resolve()
paths: set[str] = set()
for path in root.rglob("*.md"):
if path.is_symlink():
continue
resolved = path.resolve()
if resolved.is_file() and resolved.is_relative_to(resolved_root):
paths.add(resolved.relative_to(resolved_root).as_posix())
return paths
def get_workspace_info() -> WorkspaceInfo:
root = get_settings().vault_path.resolve()
disk_paths = _disk_markdown_paths()
indexed_paths = {item.file_path for item in repository.list_note_locations()}
return WorkspaceInfo(
name=root.name or "Vault",
path=str(root),
file_count=len(disk_paths),
indexed_note_count=len(indexed_paths),
requires_refresh=disk_paths != indexed_paths,
)
def _tree(directory: Path, locations: dict[str, repository.NoteLocation]) -> list[WorkspaceEntry]:
if not directory.exists():
return []
root = get_settings().vault_path.resolve()
entries: list[WorkspaceEntry] = []
children = sorted(
directory.iterdir(), key=lambda item: (not item.is_dir(), item.name.casefold())
)
for child in children:
if child.name.startswith(".") or child.is_symlink():
continue
resolved = child.resolve()
if not resolved.is_relative_to(root):
continue
relative = resolved.relative_to(root).as_posix()
public_path = f"/{relative}"
if resolved.is_dir():
entries.append(
WorkspaceEntry(
entry_id=_entry_id("folder", relative),
name=child.name,
path=public_path,
type="folder",
children=_tree(resolved, locations),
)
)
elif resolved.is_file() and child.suffix.lower() == ".md":
location = locations.get(relative)
entries.append(
WorkspaceEntry(
entry_id=location.note_id if location else _entry_id("file", relative),
note_id=location.note_id if location else None,
name=child.name,
path=public_path,
type="file",
)
)
return entries
def get_workspace_tree() -> list[WorkspaceEntry]:
locations = {item.file_path: item for item in repository.list_note_locations()}
return _tree(get_settings().vault_path.resolve(), locations)
async def open_workspace(requested_path: str | None) -> WorkspaceSnapshot:
"""打开当前配置 Vault;发现未索引文件时先执行一次安全全量刷新。"""
root = get_settings().vault_path.resolve()
if requested_path and Path(requested_path).resolve() != root:
raise ApiError(
409,
"WORKSPACE_PATH_MISMATCH",
"Web development mode can only open the backend configured Vault.",
{"configured_path": str(root)},
)
root.mkdir(parents=True, exist_ok=True)
info = get_workspace_info()
if info.requires_refresh:
await index_service.rebuild(IndexRebuildRequest())
info = get_workspace_info()
return WorkspaceSnapshot(workspace=info, items=get_workspace_tree())
@serialized_vault_mutation
async def create_folder(parent: str, name: str) -> WorkspaceEntry:
clean_parent = normalize_folder(parent)
clean_name = normalize_entry_name(name)
relative = f"{clean_parent}/{clean_name}" if clean_parent else clean_name
target = resolve_in_vault(relative)
if not clean_parent:
get_settings().vault_path.mkdir(parents=True, exist_ok=True)
if target.exists():
raise ApiError(
409, "RESOURCE_CONFLICT", "folder already exists", {"path": relative}
)
if not target.parent.is_dir():
raise ApiError(
404,
"RESOURCE_NOT_FOUND",
"parent folder not found",
{"parent": clean_parent},
)
target.mkdir(parents=False)
return WorkspaceEntry(
entry_id=_entry_id("folder", relative),
name=clean_name,
path=f"/{relative}",
type="folder",
)
@serialized_vault_mutation
async def rename_folder(path: str, new_name: str) -> WorkspaceEntry:
old_folder = normalize_folder(path)
if not old_folder:
raise ApiError(400, "INVALID_PATH", "the Vault root cannot be renamed")
clean_name = normalize_entry_name(new_name)
parent = Path(old_folder).parent.as_posix()
parent = "" if parent == "." else parent
new_folder = f"{parent}/{clean_name}" if parent else clean_name
source = resolve_in_vault(old_folder)
target = resolve_in_vault(new_folder)
if not source.is_dir() or source.is_symlink():
raise ApiError(404, "RESOURCE_NOT_FOUND", "folder not found", {"path": path})
if target.exists():
raise ApiError(
409, "RESOURCE_CONFLICT", "target folder already exists", {"path": new_folder}
)
affected = [
item
for item in repository.list_note_locations()
if item.folder == old_folder or item.folder.startswith(f"{old_folder}/")
]
source.replace(target)
conn = connect()
now = datetime.now(timezone.utc)
try:
with transaction(conn):
for item in affected:
file_suffix = item.file_path[len(old_folder) :].lstrip("/")
folder_suffix = item.folder[len(old_folder) :].lstrip("/")
repository.update_note_location(
conn=conn,
note_id=item.note_id,
title=item.title,
file_path=f"{new_folder}/{file_suffix}",
folder=(
f"{new_folder}/{folder_suffix}" if folder_suffix else new_folder
),
updated_at=now,
)
except BaseException:
target.replace(source)
raise
finally:
conn.close()
return WorkspaceEntry(
entry_id=_entry_id("folder", new_folder),
name=clean_name,
path=f"/{new_folder}",
type="folder",
children=_tree(target, {item.file_path: item for item in repository.list_note_locations()}),
)
@serialized_vault_mutation
async def delete_folder(path: str) -> OperationResponse:
folder = normalize_folder(path)
if not folder:
raise ApiError(400, "INVALID_PATH", "the Vault root cannot be deleted")
source = resolve_in_vault(folder)
if not source.is_dir() or source.is_symlink():
raise ApiError(404, "RESOURCE_NOT_FOUND", "folder not found", {"path": path})
affected = [
item
for item in repository.list_note_locations()
if item.folder == folder or item.folder.startswith(f"{folder}/")
]
tombstone = source.with_name(f".{source.name}.{uuid4().hex}.deleting")
source.replace(tombstone)
conn = connect()
try:
with transaction(conn):
block_ids: list[str] = []
for item in affected:
block_ids.extend(repository.delete_note(item.note_id, conn=conn))
await vector_store.delete(block_ids, conn=conn)
except BaseException:
tombstone.replace(source)
raise
finally:
conn.close()
try:
shutil.rmtree(tombstone)
except OSError:
# 已提交的删除不回滚;隐藏 tombstone 可由后续维护任务清理。
pass
return OperationResponse(
status="completed",
resource_id=_entry_id("folder", folder),
message=f"deleted folder and {len(affected)} indexed notes",
)
+48
View File
@@ -0,0 +1,48 @@
{
"dataset_id": "rag-core-v1",
"kind": "rag",
"version": "1.0.0",
"description": "基础中文笔记检索集(对应 backend/data/vault 内置语料,重建索引后即可复现)",
"cases": [
{
"case_id": "rag-vector-sim",
"query": "向量数据库如何进行相似度检索",
"expected_note_ids": ["note_c1454740a0e55ef5"],
"expected_block_ids": ["blk_07c4c6bce0ec4d12", "blk_605fb3593809f224"],
"citation_required": true,
"tags": ["向量数据库", "检索"]
},
{
"case_id": "rag-python-func",
"query": "Python 如何定义函数",
"expected_note_ids": ["note_424c3742c6f0e555"],
"expected_block_ids": ["blk_45d48cae2fed40fe", "blk_0768d9c25c2ecf07"],
"citation_required": true,
"tags": ["python"]
},
{
"case_id": "rag-citation",
"query": "搜索结果如何定位到原文位置",
"expected_note_ids": ["note_0c619caa30b1614c"],
"expected_block_ids": ["blk_3f6fcead71c25fc6", "blk_9af7b12e9ce909fc"],
"citation_required": true,
"tags": ["RAG"]
},
{
"case_id": "rag-hybrid",
"query": "混合检索怎么融合全文和向量",
"expected_note_ids": ["note_c1454740a0e55ef5"],
"expected_block_ids": ["blk_82b45418dba9f720"],
"citation_required": true,
"tags": ["检索"]
},
{
"case_id": "rag-tech-stack",
"query": "这个项目用什么后端和检索技术",
"expected_note_ids": ["note_3327e6cf18f3701f"],
"expected_block_ids": ["blk_feb2a9c42e7d31ad"],
"citation_required": false,
"tags": ["项目"]
}
]
}
@@ -0,0 +1,20 @@
commands:
- command_id: mcp-fixture.notify
title: MCP 通知
description: 通过隔离 MCP Host 返回宿主白名单通知 effect。
icon: bolt
locations:
- command_palette
when:
- editor.has_selection
context:
- selection
secrets:
- api_key
mcp_tool: mcp-fixture.command
parameters:
type: object
properties:
message:
type: string
additionalProperties: false
@@ -0,0 +1,26 @@
id: mcp-fixture
name: MCP Fixture
version: 1.0.0
description: 阶段 C/D 离线联调 Fixture,覆盖 MCP Tool、Command 与错误边界。
permissions:
- notes.read
- secrets.use
contributes:
tools:
- mcp-fixture.echo
- mcp-fixture.fail
- mcp-fixture.sleep
- mcp-fixture.large
- mcp-fixture.environment
- mcp-fixture.exit
commands:
- mcp-fixture.notify
settings_sections:
- mcp-fixture.general
backend:
type: mcp
transport: stdio
command: python
args: [server.py]
startup_timeout_seconds: 5
tool_timeout_seconds: 1
@@ -0,0 +1,248 @@
"""确定性的 MCP stdio 测试 Server;仅使用标准库,不依赖产品代码。"""
from __future__ import annotations
import json
import os
import sys
import threading
import time
from typing import Any
WRITE_LOCK = threading.Lock()
CANCELLED: dict[int, threading.Event] = {}
MODE = sys.argv[1] if len(sys.argv) > 1 else "normal"
def send(message: dict[str, Any]) -> None:
with WRITE_LOCK:
sys.stdout.write(json.dumps(message, ensure_ascii=False, separators=(",", ":")) + "\n")
sys.stdout.flush()
def respond(request_id: int, result: dict[str, Any]) -> None:
send({"jsonrpc": "2.0", "id": request_id, "result": result})
def tool(name: str, description: str, properties: dict[str, Any] | None = None) -> dict[str, Any]:
return {
"name": name,
"description": description,
"inputSchema": {
"type": "object",
"properties": properties or {},
"required": list(properties or {}),
"additionalProperties": False,
},
}
TOOLS = {
"echo": {
**tool(
"echo",
"Return the provided text.",
{
"text": {"type": "string"},
"suffix": {"type": ["string", "null"]},
},
),
"_meta": {"notesagent/permission": "notes.read"},
},
"fail": tool("fail", "Return an MCP business error."),
"sleep": tool("sleep", "Wait until completed or cancelled.", {"seconds": {"type": "number"}}),
"large": tool("large", "Return a result larger than the host limit."),
"environment": tool("environment", "Report whether host secrets leaked into the process."),
"exit": tool("exit", "Terminate the fixture process."),
"command": tool(
"command",
"Execute a NotesAgent Plugin Command envelope.",
{"_notesagent": {"type": "object"}},
),
}
# suffix 是可选字段,用于验证 Host 不会把缺省值擅自补成 null。
TOOLS["echo"]["inputSchema"]["required"] = ["text"]
def call_tool(request_id: int, params: dict[str, Any]) -> None:
name = params.get("name")
arguments = params.get("arguments") or {}
if name == "command":
envelope = arguments.get("_notesagent") or {}
command_arguments = envelope.get("arguments") or {}
context = envelope.get("context") or {}
settings = envelope.get("settings") or {}
secrets = envelope.get("secrets") or {}
if not isinstance(secrets.get("api_key"), str):
respond(
request_id,
{
"content": [{"type": "text", "text": "declared secret missing"}],
"isError": True,
},
)
return
message = command_arguments.get("message") or context.get("selection") or ""
message = f"{settings.get('message_prefix', '')}{message}"
respond(
request_id,
{
"content": [{"type": "text", "text": "command completed"}],
"structuredContent": {
"type": "notification",
"payload": {
"level": "success",
"message": str(message),
},
},
"isError": False,
},
)
return
if name == "echo":
text = str(arguments.get("text", ""))
structured_content = {"echo": text}
if "suffix" in arguments:
structured_content["suffix"] = arguments["suffix"]
respond(
request_id,
{
"content": [{"type": "text", "text": text}],
"structuredContent": structured_content,
"isError": False,
},
)
return
if name == "fail":
respond(
request_id,
{
"content": [{"type": "text", "text": "fixture failure"}],
"isError": True,
},
)
return
if name == "large":
respond(
request_id,
{
"content": [{"type": "text", "text": "x" * 300_000}],
"isError": False,
},
)
return
if name == "environment":
respond(
request_id,
{
"content": [{"type": "text", "text": "environment checked"}],
"structuredContent": {
"has_openai_key": "OPENAI_API_KEY" in os.environ,
"has_app_db_path": "APP_DB_PATH" in os.environ,
},
"isError": False,
},
)
return
if name == "exit":
os._exit(17)
if name == "sleep":
cancelled = CANCELLED.setdefault(request_id, threading.Event())
seconds = max(0.0, min(float(arguments.get("seconds", 0)), 30.0))
if cancelled.wait(seconds):
respond(
request_id,
{
"content": [{"type": "text", "text": "cancelled"}],
"isError": True,
},
)
else:
respond(
request_id,
{
"content": [{"type": "text", "text": "completed"}],
"structuredContent": {"slept": seconds},
"isError": False,
},
)
CANCELLED.pop(request_id, None)
return
send(
{
"jsonrpc": "2.0",
"id": request_id,
"error": {"code": -32602, "message": f"Unknown tool: {name}"},
}
)
def main() -> None:
for line in sys.stdin:
message = json.loads(line)
method = message.get("method")
request_id = message.get("id")
params = message.get("params") or {}
if method == "initialize" and isinstance(request_id, int):
if MODE == "invalid-result":
send({"jsonrpc": "2.0", "id": request_id, "result": None})
continue
if MODE == "oversized-stdout":
# 不带换行,验证 Host 在读取完整内容前执行硬上限。
sys.stdout.write("x" * (2 * 1024 * 1024 + 1))
sys.stdout.flush()
time.sleep(10)
return
respond(
request_id,
{
"protocolVersion": params.get("protocolVersion"),
"capabilities": (
{} if MODE == "no-tools" else {"tools": {"listChanged": False}}
),
"serverInfo": {"name": "notesagent-mcp-fixture", "version": "1.0.0"},
},
)
elif method == "tools/list" and isinstance(request_id, int):
if MODE == "invalid-schema":
respond(
request_id,
{
"tools": [
{
"name": "broken",
"description": "invalid schema",
"inputSchema": {"type": "string"},
}
]
},
)
elif params.get("cursor") == "page-2":
respond(
request_id,
{
"tools": [
TOOLS["large"],
TOOLS["environment"],
TOOLS["exit"],
TOOLS["command"],
]
},
)
else:
respond(
request_id,
{"tools": [TOOLS["echo"], TOOLS["fail"], TOOLS["sleep"]], "nextCursor": "page-2"},
)
elif method == "tools/call" and isinstance(request_id, int):
threading.Thread(target=call_tool, args=(request_id, params), daemon=True).start()
elif method == "notifications/cancelled":
cancelled_id = params.get("requestId")
if isinstance(cancelled_id, int):
CANCELLED.setdefault(cancelled_id, threading.Event()).set()
elif method == "ping" and isinstance(request_id, int):
respond(request_id, {})
if __name__ == "__main__":
main()
@@ -0,0 +1,11 @@
section_id: mcp-fixture.general
schema_version: 1
fields:
- key: message_prefix
label: Message Prefix
type: string
default: ""
- key: api_key
label: Fixture API Key
type: secret
required: true
@@ -0,0 +1,19 @@
commands:
- command_id: text-tools.uppercase-selection
title: 转为大写
description: 将当前选区或传入文本转换为大写并显示通知。
icon: edit
locations:
- command_palette
- context_menu
when:
- editor.has_selection
context:
- selection
handler: uppercase_selection
parameters:
type: object
properties:
text:
type: string
additionalProperties: false
@@ -6,6 +6,10 @@ permissions: []
contributes:
tools:
- text.uppercase
commands:
- text-tools.uppercase-selection
settings_sections:
- text-tools.general
backend:
type: internal_rpc
transport: none
@@ -0,0 +1,31 @@
section_id: text-tools.general
schema_version: 1
fields:
- key: result_limit
label: 结果字符数
description: Command 通知中最多保留的字符数。
type: number
required: true
default: 100
minimum: 1
maximum: 1000
- key: label_prefix
label: 标签前缀
type: string
default: ""
- key: output_style
label: 输出样式
type: select
default: notification
options:
- notification
- compact
- key: enabled_hint
label: 显示提示
type: boolean
default: true
- key: api_key
label: API Key
description: Secret 示例字段;普通 Settings API 永不返回明文。
type: secret
required: false
+3
View File
@@ -5,10 +5,13 @@ description = "Notes Agent 的 FastAPI 基础壳子"
readme = "README.md"
requires-python = ">=3.11"
dependencies = [
"cryptography>=45,<52",
"fastapi>=0.116,<1.0",
"httpx>=0.28,<1.0",
"jsonschema>=4.25,<5.0",
"mistune>=3.0,<4.0",
"pyyaml>=6.0,<7.0",
"referencing>=0.36,<1.0",
"sqlite-vec>=0.1.9",
"uvicorn[standard]>=0.35,<1.0",
]
+200
View File
@@ -1,10 +1,18 @@
import asyncio
from datetime import datetime, timezone
import pytest
from app.agent.trace_repository import AgentTraceRepository
from app.agent.permissions import PermissionMode
from app.agent.tools import ToolExecutionContext
from app.container import build_container
from app.database.db import connect
from app.errors import ApiError
from app.routes import agent_events
from app.contracts import (
AgentEventType,
AgentRun,
AgentRunCreateRequest,
AgentRunStatus,
ToolCall,
@@ -108,8 +116,200 @@ def test_permission_confirmation_resumes_agent() -> None:
created.run_id, request_id, "allow_once"
)
completed = await container.agent.wait(created.run_id)
events = [event async for event in container.agent.events(created.run_id)]
assert completed.status == AgentRunStatus.completed
assert completed.tool_results[0].success is True
assert AgentEventType.permission_resolved in {event.event for event in events}
run(scenario())
def test_agent_trace_persists_and_replays_from_sequence() -> None:
async def scenario() -> None:
first = build_container()
created = await first.agent.create_run(
AgentRunCreateRequest(
input="persistent trace",
provider_id="mock",
model="mock-1",
metadata={"suite": "agent-benchmark-v1"},
)
)
completed = await first.agent.wait(created.run_id)
restarted = build_container()
restored = restarted.agent.get_run(created.run_id)
first_page = restarted.agent.get_trace(
created.run_id, after_sequence=-1, limit=2
)
second_page = restarted.agent.get_trace(
created.run_id,
after_sequence=first_page.next_sequence,
limit=100,
)
replay = [
event
async for event in restarted.agent.events(
created.run_id, after_sequence=first_page.next_sequence
)
]
assert completed.status == restored.status == AgentRunStatus.completed
assert first_page.has_more is True
assert [item.sequence for item in first_page.items] == [0, 1]
assert second_page.items[0].sequence == 2
assert replay == second_page.items
assert first_page.summary.model_calls == 1
assert first_page.summary.token_usage == completed.token_usage
assert first_page.config_snapshot["metadata"] == {
"suite": "agent-benchmark-v1"
}
assert second_page.items[-1].event == AgentEventType.run_completed
run(scenario())
def test_interrupted_persisted_run_is_closed_after_restart() -> None:
now = datetime.now(timezone.utc)
request = AgentRunCreateRequest(
input="interrupted",
provider_id="mock",
model="mock-1",
)
persisted = AgentRun(
run_id="run_interrupted",
status=AgentRunStatus.running,
input=request.input,
provider_id=request.provider_id,
model=request.model,
max_steps=request.max_steps,
created_at=now,
updated_at=now,
)
AgentTraceRepository().create_run(persisted, request, {"model": "mock-1"})
restarted = build_container()
recovered = restarted.agent.get_run(persisted.run_id)
events = run(
_collect_events(restarted.agent.events(persisted.run_id, after_sequence=-1))
)
assert recovered.status == AgentRunStatus.failed
assert recovered.error_code == "AGENT_PROCESS_RESTARTED"
assert events[-1].event == AgentEventType.run_failed
assert events[-1].sequence == 0
def test_trace_redacts_secrets_and_truncates_large_values() -> None:
async def scenario() -> None:
container = build_container()
secret = "sk-should-not-be-stored"
created = await container.agent.create_run(
AgentRunCreateRequest(
input=f'/tool system.echo {{"text":"{"x" * 4200}","api_key":"{secret}"}}',
provider_id="mock",
model="mock-1",
allowed_tools=["system.echo"],
metadata={"authorization": secret},
)
)
await container.agent.wait(created.run_id)
trace = container.agent.get_trace(
created.run_id, after_sequence=-1, limit=100
)
tool_call = next(
item for item in trace.items if item.event == AgentEventType.tool_call
)
assert tool_call.data["arguments"]["api_key"] == "[REDACTED]"
assert str(tool_call.data["arguments"]["text"]).endswith("...[TRUNCATED]")
assert trace.config_snapshot["metadata"]["authorization"] == "[REDACTED]"
assert secret not in trace.model_dump_json()
conn = connect()
try:
stored_row = conn.execute(
"""
SELECT run_json, request_json, config_snapshot_json
FROM agent_runs WHERE run_id = ?
""",
(created.run_id,),
).fetchone()
stored = "\n".join(str(value) for value in stored_row)
finally:
conn.close()
assert secret not in stored
run(scenario())
def test_persisted_agent_run_preserves_long_input_and_output() -> None:
"""审计事件可以限长,但重启后读取的 AgentRun 不能丢失正文。"""
now = datetime.now(timezone.utc)
long_input = "输入" * 2_500
long_output = "输出" * 2_500
request = AgentRunCreateRequest(
input=long_input,
provider_id="mock",
model="mock-1",
)
persisted = AgentRun(
run_id="run_long_content",
status=AgentRunStatus.completed,
input=long_input,
output=long_output,
provider_id=request.provider_id,
model=request.model,
max_steps=request.max_steps,
created_at=now,
updated_at=now,
)
repository = AgentTraceRepository()
repository.create_run(persisted, request, {"model": request.model})
restored = repository.get_run(persisted.run_id)
assert restored is not None
assert restored.input == long_input
assert restored.output == long_output
async def _collect_events(iterator):
return [event async for event in iterator]
def test_agent_sse_uses_last_event_id_and_emits_event_ids(monkeypatch) -> None:
async def scenario() -> None:
test_container = build_container()
monkeypatch.setattr("app.routes.container", test_container)
created = await test_container.agent.create_run(
AgentRunCreateRequest(
input="resume sse",
provider_id="mock",
model="mock-1",
)
)
await test_container.agent.wait(created.run_id)
response = await agent_events(
created.run_id, after_sequence=None, last_event_id="1"
)
chunks = [chunk async for chunk in response.body_iterator]
body = "".join(
chunk.decode("utf-8") if isinstance(chunk, bytes) else chunk
for chunk in chunks
)
assert "id: 0\n" not in body
assert "id: 1\n" not in body
assert "id: 2\n" in body
assert "event: RunCompleted" in body
with pytest.raises(ApiError) as error:
await agent_events(
created.run_id, after_sequence=None, last_event_id="invalid"
)
assert error.value.code == "TRACE_CURSOR_INVALID"
run(scenario())
+271 -14
View File
@@ -1,19 +1,12 @@
import asyncio
import threading
from types import SimpleNamespace
import pytest
from app.main import health, service_status
from app.routes import get_index_status, list_notes, list_plugins, list_providers, list_skills
from app.routes import (
create_provider,
create_task,
delete_provider,
delete_task,
get_provider,
get_task,
list_tasks,
update_provider,
update_task,
)
from app.contracts import (
McpServerSecretStatus,
McpServerSecretWriteRequest,
ProviderCreateRequest,
ProviderType,
ProviderUpdateRequest,
@@ -21,6 +14,61 @@ from app.contracts import (
TaskStatus,
TaskUpdateRequest,
)
from app.main import health, service_status
from app.routes import (
create_provider,
create_task,
delete_provider,
delete_task,
get_index_status,
get_provider,
get_task,
list_notes,
list_plugins,
list_provider_presets,
list_providers,
list_skills,
list_tasks,
update_provider,
update_task,
)
def test_mcp_secret_routes_offload_blocking_lifecycle_work(monkeypatch) -> None:
from app import routes
caller_thread = threading.get_ident()
worker_threads: list[int] = []
class FakeMcpRegistry:
def put_secret(self, server_id, key, secret, *, kind):
worker_threads.append(threading.get_ident())
return McpServerSecretStatus(key=key, configured=True)
def delete_secret(self, server_id, key, *, kind):
worker_threads.append(threading.get_ident())
return McpServerSecretStatus(key=key, configured=False)
monkeypatch.setattr(
routes,
"container",
SimpleNamespace(mcp_servers=FakeMcpRegistry()),
)
written = asyncio.run(
routes.put_mcp_server_secret(
"server-1",
"TOKEN",
McpServerSecretWriteRequest(secret="hidden"),
kind="environment",
)
)
deleted = asyncio.run(
routes.delete_mcp_server_secret("server-1", "TOKEN", kind="environment")
)
assert written.configured is True
assert deleted.configured is False
assert worker_threads and all(item != caller_thread for item in worker_threads)
def test_health() -> None:
@@ -29,6 +77,172 @@ def test_health() -> None:
assert response.model_dump() == {"status": "ok"}
def test_mcp_create_and_trust_are_not_executed_on_event_loop(monkeypatch) -> None:
from app import routes
from app.contracts import McpServerCreateRequest, McpServerTrustRequest
caller = threading.get_ident()
workers = []
class Registry:
def create(self, request):
workers.append(threading.get_ident())
return "created"
def trust(self, server_id, digest):
workers.append(threading.get_ident())
return "trusted"
monkeypatch.setattr(routes, "container", SimpleNamespace(mcp_servers=Registry()))
assert (
asyncio.run(
routes.create_mcp_server(McpServerCreateRequest(name="test", command="uvx"))
)
== "created"
)
assert (
asyncio.run(
routes.trust_mcp_server(
"test", McpServerTrustRequest(command_digest="a" * 64)
)
)
== "trusted"
)
assert len(workers) == 2
assert all(worker != caller for worker in workers)
def test_mcp_split_config_and_secret_requests_persist_without_plaintext(
monkeypatch,
) -> None:
from fastapi.testclient import TestClient
from app import routes
from app.agent.tools import ToolRegistry
from app.config import get_settings
from app.extensions.mcp_registry import McpServerRegistry
from app.main import app
from app.providers.credentials import EncryptedCredentialStore
service = McpServerRegistry(
ToolRegistry(),
EncryptedCredentialStore(),
get_settings().data_dir,
allow_process_launch=True,
)
monkeypatch.setattr(routes, "container", SimpleNamespace(mcp_servers=service))
client = TestClient(app)
config = {
"name": "MiniMax configuration test",
"command": "uvx",
"environment": {"MINIMAX_API_HOST": "https://api.minimaxi.com"},
"secret_environment_keys": ["MINIMAX_API_KEY"],
"startup_timeout_seconds": 120,
"tool_timeout_seconds": 300,
}
# Reproduce the old frontend payload. The backend still enforces separation.
invalid = client.post(
"/api/mcp/servers",
json={
**config,
"environment": {
**config["environment"],
"MINIMAX_API_KEY": "synthetic-only",
},
},
)
assert invalid.status_code == 422
assert invalid.json()["error"]["code"] == "MCP_ENVIRONMENT_INVALID"
created = client.post("/api/mcp/servers", json=config)
assert created.status_code == 201
server_id = created.json()["server_id"]
saved = client.put(
f"/api/mcp/servers/{server_id}/secrets/MINIMAX_API_KEY",
json={"secret": "synthetic-only"},
)
assert saved.status_code == 200
current = client.get(f"/api/mcp/servers/{server_id}")
assert current.json()["secret_environment"] == {"MINIMAX_API_KEY": True}
assert "synthetic-only" not in current.text
assert "synthetic-only" not in service._path.read_text(encoding="utf-8")
_, credentials_path = service.credentials._paths()
assert "synthetic-only" not in credentials_path.read_text(encoding="utf-8")
assert not current.json()["enabled"] # Saving never starts a third-party process.
client.close()
@pytest.mark.parametrize("operation", ["create", "trust"])
def test_mcp_lifecycle_lock_contention_keeps_event_loop_responsive(
monkeypatch,
operation,
) -> None:
from app import routes
from app.agent.tools import ToolRegistry
from app.config import get_settings
from app.contracts import McpServerCreateRequest, McpServerTrustRequest
from app.extensions.mcp_registry import McpServerRegistry
from app.providers.credentials import EncryptedCredentialStore
service = McpServerRegistry(
ToolRegistry(),
EncryptedCredentialStore(),
get_settings().data_dir,
allow_process_launch=True,
)
request = McpServerCreateRequest(
name="Lock contention fixture", command="not-executed"
)
server = service.create(request)
monkeypatch.setattr(routes, "container", SimpleNamespace(mcp_servers=service))
entered = threading.Event()
locked = threading.Event()
release = threading.Event()
original = getattr(service, operation)
def observed(*args):
entered.set()
return original(*args)
def hold_lifecycle_lock():
with service._lifecycle_lock:
locked.set()
release.wait(timeout=5)
monkeypatch.setattr(service, operation, observed)
holder = threading.Thread(target=hold_lifecycle_lock, daemon=True)
holder.start()
# An independent watchdog lets the test fail rather than hang if a regression
# blocks the event loop itself (an asyncio timeout alone cannot catch that).
watchdog = threading.Timer(5, release.set)
watchdog.start()
async def exercise():
pending = asyncio.create_task(
routes.create_mcp_server(request)
if operation == "create"
else routes.trust_mcp_server(
server.server_id,
McpServerTrustRequest(command_digest=server.command_digest),
)
)
try:
assert await asyncio.to_thread(entered.wait, 2)
assert not pending.done()
assert not release.is_set()
assert (await health()).status == "ok"
finally:
release.set()
await pending
try:
assert locked.wait(timeout=2)
asyncio.run(exercise())
finally:
release.set()
watchdog.cancel()
holder.join(timeout=2)
def test_service_status() -> None:
response = asyncio.run(service_status())
@@ -45,7 +259,9 @@ def test_core_collections_are_typed() -> None:
assert notes.items == []
assert notes.page.limit == 20
assert [skill.manifest.skill_id for skill in skills.items] == ["knowledge-assistant"]
assert [skill.manifest.skill_id for skill in skills.items] == [
"knowledge-assistant"
]
assert skills.items[0].status == "ready"
assert [plugin.manifest.plugin_id for plugin in plugins.items] == ["text-tools"]
assert plugins.items[0].status == "ready"
@@ -53,6 +269,30 @@ def test_core_collections_are_typed() -> None:
assert index.status == "idle"
def test_provider_presets_include_openai_and_deepseek() -> None:
presets = asyncio.run(list_provider_presets())
by_id = {item.preset_id: item for item in presets.items}
assert by_id["openai"].base_url == "https://api.openai.com/v1"
assert by_id["deepseek"].base_url == "https://api.deepseek.com"
assert by_id["deepseek"].provider_type == ProviderType.openai_compatible
assert by_id["deepseek"].default_credential_id == "deepseek"
def test_provider_presets_static_route_precedes_provider_id_route() -> None:
from app.routes import router
get_paths = [
route.path
for route in router.routes
if "GET" in getattr(route, "methods", set())
]
assert get_paths.index("/api/providers/presets") < get_paths.index(
"/api/providers/{provider_id}"
)
def test_openapi_contains_documented_frontend_interfaces() -> None:
from app.main import app
@@ -64,12 +304,29 @@ def test_openapi_contains_documented_frontend_interfaces() -> None:
"/api/agent/runs",
"/api/agent/runs/{run_id}/cancel",
"/api/agent/runs/{run_id}/events",
"/api/agent/runs/{run_id}/trace",
"/api/skills",
"/api/plugins",
"/api/plugins/install",
"/api/plugins/{plugin_id}/host",
"/api/plugins/{plugin_id}/host/restart",
"/api/plugin-contributions/commands",
"/api/plugin-contributions/commands/{command_id}/execute",
"/api/plugins/{plugin_id}/settings",
"/api/plugins/{plugin_id}/settings/{key}/secret",
"/api/plugins/{plugin_id}/enable",
"/api/plugins/{plugin_id}/disable",
"/api/mcp/servers",
"/api/mcp/servers/{server_id}",
"/api/mcp/servers/{server_id}/tools",
"/api/mcp/servers/{server_id}/trust",
"/api/mcp/servers/{server_id}/test",
"/api/mcp/servers/{server_id}/enable",
"/api/mcp/servers/{server_id}/disable",
"/api/mcp/servers/{server_id}/secrets/{key}",
"/api/providers/test",
"/api/providers/presets",
"/api/credentials/{credential_id}",
"/api/index/rebuild",
}
+567
View File
@@ -0,0 +1,567 @@
"""Benchmark 服务的单元与端到端测试。
沿用 conftest 的隔离机制APP_DATA_DIR / DB / Vault 都指向临时目录benchmark
数据集也落在临时目录settings.benchmark_datasets_path不读写真实数据
运行采用创建即 queued + 后台 Task 执行的异步模型测试通过 _run 在同一事件循环内
创建并等待后台任务结束得到终态 BenchmarkRun 后再断言
"""
from __future__ import annotations
import asyncio
import json
import pytest
from pydantic import ValidationError
from app.benchmarks import datasets, metrics as m, service
from app.config import get_settings
from app.contracts import (
BenchmarkKind,
BenchmarkRun,
BenchmarkStatus,
RAGRunRequest,
SearchMode,
)
from app.errors import ApiError
def _write_dataset(dataset_id: str, cases: list[dict], *, kind: str = "rag") -> None:
directory = get_settings().benchmark_datasets_path
directory.mkdir(parents=True, exist_ok=True)
payload = {
"dataset_id": dataset_id,
"kind": kind,
"version": "1.0.0",
"description": "test dataset",
"cases": cases,
}
(directory / f"{dataset_id}.json").write_text(
json.dumps(payload, ensure_ascii=False), encoding="utf-8"
)
def _write_raw(dataset_id: str, raw: dict) -> None:
directory = get_settings().benchmark_datasets_path
directory.mkdir(parents=True, exist_ok=True)
(directory / f"{dataset_id}.json").write_text(
json.dumps(raw, ensure_ascii=False), encoding="utf-8"
)
def _run(request: RAGRunRequest):
"""创建运行并在同一事件循环内等待后台任务结束,返回终态 BenchmarkRun。"""
from app.contracts import BenchmarkRun
async def _execute() -> BenchmarkRun:
run = await service.create_rag_run(request)
return await service.wait_for_run(run.run_id)
return asyncio.run(_execute())
# --------------------------------------------------------------------------- #
# 指标纯函数
# --------------------------------------------------------------------------- #
def test_hit_at_k_and_recall() -> None:
retrieved = ["a", "b", "c"]
expected = {"b", "z"}
assert m.hit_at_k(retrieved, expected, 1) is False
assert m.hit_at_k(retrieved, expected, 2) is True
assert m.recall_at_k(retrieved, expected, 5) == 0.5 # 只召回 b
def test_recall_at_k_dedups_duplicate_notes() -> None:
# 同一 Note 经多个 Block 重复出现,去重后 Recall 不应超过 1
assert m.recall_at_k(["note-a", "note-a"], {"note-a"}, 2) == 1.0
assert m.recall_at_k(["note-a", "note-a", "note-b"], {"note-a"}, 3) == 1.0
def test_reciprocal_rank_and_citation_hit() -> None:
assert m.reciprocal_rank(["x", "a", "b"], {"b"}) == 1 / 3
assert m.reciprocal_rank(["x"], {"b"}) == 0.0
assert m.citation_hit(["blk_1"], {"blk_1"}) is True
assert m.citation_hit(["blk_2"], {"blk_1"}) is False
assert m.citation_hit([], {"blk_1"}) is False
def test_percentile() -> None:
assert m.percentile([1.0, 2.0, 3.0, 4.0], 50.0) == 2.5
assert m.percentile([], 50.0) == 0.0
assert m.percentile([7.0], 95.0) == 7.0
# --------------------------------------------------------------------------- #
# Dataset 注册与校验
# --------------------------------------------------------------------------- #
def test_list_datasets_empty_by_default() -> None:
assert datasets.list_datasets(BenchmarkKind.rag) == []
def test_load_missing_dataset_raises() -> None:
with pytest.raises(ApiError) as exc:
datasets.load_dataset("does-not-exist", BenchmarkKind.rag)
assert exc.value.status_code == 404
assert exc.value.code == "BENCHMARK_DATASET_NOT_FOUND"
def test_dataset_without_expected_ids_is_invalid() -> None:
_write_dataset("bad-v1", [{"case_id": "x", "query": "q", "citation_required": False}])
with pytest.raises(ApiError) as exc:
datasets.load_dataset("bad-v1", BenchmarkKind.rag)
assert exc.value.code == "BENCHMARK_DATASET_INVALID"
def test_dataset_kind_mismatch_is_invalid() -> None:
_write_dataset("agent-v1", [{"case_id": "x", "query": "q", "expected_note_ids": ["n"]}], kind="agent")
with pytest.raises(ApiError) as exc:
datasets.load_dataset("agent-v1", BenchmarkKind.rag)
assert exc.value.code == "BENCHMARK_DATASET_INVALID"
def test_citation_required_requires_expected_block_ids() -> None:
# citation_required=true 却没有 expected_block_ids,无法计算 Citation Hit Rate,应拒绝
_write_dataset(
"cit-req-v1",
[{"case_id": "x", "query": "q", "expected_note_ids": ["n"], "citation_required": True}],
)
with pytest.raises(ApiError) as exc:
datasets.load_dataset("cit-req-v1", BenchmarkKind.rag)
assert exc.value.code == "BENCHMARK_DATASET_INVALID"
def test_list_datasets_skips_corrupted_structure() -> None:
# 合法 JSON 但字段结构错误(cases: 42),列表接口应隔离该文件而非整体 500
_write_raw("bad-structure", {"dataset_id": "bad-structure", "kind": "rag", "cases": 42})
_write_dataset("good-v1", [{"case_id": "x", "query": "q", "expected_note_ids": ["n"]}])
infos = datasets.list_datasets(BenchmarkKind.rag)
ids = {info.dataset_id for info in infos}
assert "good-v1" in ids
assert "bad-structure" not in ids
# --------------------------------------------------------------------------- #
# 请求校验(空 / 重复 modes)
# --------------------------------------------------------------------------- #
def test_empty_modes_rejected() -> None:
with pytest.raises(ValidationError):
RAGRunRequest(dataset_id="x", modes=[])
def test_duplicate_modes_rejected() -> None:
with pytest.raises(ValidationError):
RAGRunRequest(dataset_id="x", modes=[SearchMode.fts, SearchMode.fts])
# --------------------------------------------------------------------------- #
# RAG Benchmark 端到端
# --------------------------------------------------------------------------- #
def _single_note_case() -> tuple[str, str, dict]:
from app.services import note_service
note = asyncio.run(
note_service.create_note(
title="向量库",
markdown="向量数据库用于存储高维向量并支持近似最近邻检索。",
folder="",
tags=["向量"],
)
)
case = {
"case_id": "c1",
"query": "向量数据库相似度检索",
"expected_note_ids": [note.note_id],
"expected_block_ids": [note.blocks[0].block_id],
"citation_required": True,
"tags": ["向量"],
}
return note.note_id, note.blocks[0].block_id, case
def test_rag_benchmark_end_to_end() -> None:
_, _, case = _single_note_case()
_write_dataset("e2e-v1", [case])
run = _run(RAGRunRequest(dataset_id="e2e-v1", modes=[SearchMode.fts]))
assert run.status.value == "completed"
assert run.dataset_hash.startswith("sha256:")
assert run.metrics is not None
fts = run.metrics["fts"]
assert fts["hit_at_1"] == 1.0
assert fts["recall_at_k"] == 1.0
assert fts["mrr"] == 1.0
assert fts["citation_hit_rate"] == 1.0
assert fts["p50_latency_ms"] >= 0.0
assert fts["p95_latency_ms"] >= fts["p50_latency_ms"]
def test_rag_benchmark_all_modes_produce_metrics() -> None:
_, _, case = _single_note_case()
_write_dataset("e2e-modes-v1", [case])
run = _run(RAGRunRequest(dataset_id="e2e-modes-v1"))
assert run.status.value == "completed"
for mode in ("fts", "vector", "hybrid"):
assert mode in run.metrics
for key in ("hit_at_1", "hit_at_5", "recall_at_k", "mrr", "citation_hit_rate"):
assert 0.0 <= run.metrics[mode][key] <= 1.0
def test_config_snapshot_records_index_and_models() -> None:
_, _, case = _single_note_case()
_write_dataset("snapshot-v1", [case])
run = _run(RAGRunRequest(dataset_id="snapshot-v1", modes=[SearchMode.fts]))
snapshot = run.config_snapshot
assert snapshot["index_meta"] is not None
assert snapshot["embedding"]["policy"] == "per_case"
assert snapshot["local_embedding"]["version"]
assert snapshot["local_embedding"]["dim"]
assert snapshot["reranker"]["version"]
assert snapshot["retrieval"]["rrf_k"] == 60
def test_benchmark_report_and_events() -> None:
_, _, case = _single_note_case()
_write_dataset("report-v1", [case])
run = _run(RAGRunRequest(dataset_id="report-v1", modes=[SearchMode.fts]))
report = service.get_report(run.run_id)
events = service.get_events(run.run_id)
assert report is not None
assert report.run_id == run.run_id
assert len(report.cases) == 1
assert report.cases[0].case_id == "c1"
assert report.cases[0].hit_at_1 is True
assert events, "运行应产生事件"
assert events[0].event.value == "RunStarted"
assert events[-1].event.value == "RunCompleted"
def test_cancel_completed_run_keeps_status() -> None:
_, _, case = _single_note_case()
_write_dataset("cancel-v1", [case])
run = _run(RAGRunRequest(dataset_id="cancel-v1", modes=[SearchMode.fts]))
assert run.status.value == "completed"
cancelled = service.cancel_run(run.run_id)
assert cancelled.status.value == "completed" # 已结束,不再变 cancelled
def test_cancel_queued_run_marks_cancelled() -> None:
_, _, case = _single_note_case()
_write_dataset("cancel-queued-v1", [case])
async def _scenario():
run = await service.create_rag_run(
RAGRunRequest(dataset_id="cancel-queued-v1", modes=[SearchMode.fts])
)
service.cancel_run(run.run_id)
return await service.wait_for_run(run.run_id)
run = asyncio.run(_scenario())
assert run.status.value == "cancelled"
# --------------------------------------------------------------------------- #
# 指标聚合:Citation Hit Rate 只统计 citation_required 样本
# --------------------------------------------------------------------------- #
def test_citation_hit_rate_only_counts_citation_required() -> None:
from app.benchmarks import rag as rag_module
from app.contracts import RAGCaseResult
cases = [
RAGCaseResult(
case_id="a", mode=SearchMode.fts, repeat=0, latency_ms=1.0,
citation_hit=True, citation_applicable=True,
),
RAGCaseResult(
case_id="b", mode=SearchMode.fts, repeat=0, latency_ms=1.0,
citation_hit=False, citation_applicable=False,
),
]
metrics = rag_module._aggregate(cases, SearchMode.fts)
# 只有 citation_applicablecitation_required=true)的样本计入分母
assert metrics.citation_hit_rate == 1.0
# --------------------------------------------------------------------------- #
# 路由接入
# --------------------------------------------------------------------------- #
def test_benchmark_routes_wired() -> None:
from app import routes
_, _, case = _single_note_case()
_write_dataset("route-v1", [case])
async def _scenario():
listed = await routes.list_benchmark_datasets(BenchmarkKind.rag)
assert any(item.dataset_id == "route-v1" for item in listed.items)
run = await routes.create_rag_benchmark(
RAGRunRequest(dataset_id="route-v1", modes=[SearchMode.fts])
)
assert run.status.value == "queued"
return await service.wait_for_run(run.run_id)
run = asyncio.run(_scenario())
assert run.status.value == "completed"
got = asyncio.run(routes.get_benchmark_run(run.run_id))
assert got.run_id == run.run_id
report = asyncio.run(routes.get_benchmark_report(run.run_id))
assert report.cases[0].case_id == "c1"
def test_benchmark_run_not_found_raises() -> None:
from app import routes
with pytest.raises(ApiError) as exc:
asyncio.run(routes.get_benchmark_run("benchmark_missing"))
assert exc.value.code == "BENCHMARK_RUN_NOT_FOUND"
# --------------------------------------------------------------------------- #
# 审阅回归:索引兼容 / 容量 / 失败样本 / 取消事件 / 数据集隔离
# --------------------------------------------------------------------------- #
def test_create_rag_run_requires_built_index() -> None:
# 空索引(无已索引 block)会让所有模式得到全 0 指标,应在创建时拒绝而非跑出误导结果
_write_dataset("empty-index-v1", [{"case_id": "x", "query": "q", "expected_note_ids": ["n"]}])
with pytest.raises(ApiError) as exc:
asyncio.run(
service.create_rag_run(
RAGRunRequest(dataset_id="empty-index-v1", modes=[SearchMode.fts])
)
)
assert exc.value.status_code == 409
assert exc.value.code == "BENCHMARK_INDEX_INCOMPATIBLE"
def test_capacity_exceeded_when_all_runs_active(monkeypatch) -> None:
# 满容量且全为活动(非终态)run 时,无法淘汰,应拒绝创建而非删掉正在运行的 run
_, _, case = _single_note_case()
_write_dataset("capacity-v1", [case])
monkeypatch.setattr(service, "MAX_RUNS", 1)
fake_id = "benchmark_fake_active"
service._runs[fake_id] = BenchmarkRun(
run_id=fake_id,
kind=BenchmarkKind.rag,
dataset_id="capacity-v1",
dataset_hash="sha256:fake",
status=BenchmarkStatus.queued,
created_at=service._now(),
)
try:
with pytest.raises(ApiError) as exc:
asyncio.run(
service.create_rag_run(
RAGRunRequest(dataset_id="capacity-v1", modes=[SearchMode.fts])
)
)
assert exc.value.status_code == 429
assert exc.value.code == "BENCHMARK_CAPACITY_EXCEEDED"
finally:
service._runs.pop(fake_id, None)
def test_failed_samples_counted_as_zero_in_aggregate() -> None:
from app.benchmarks import rag as rag_module
from app.contracts import RAGCaseResult
cases = [
RAGCaseResult(
case_id="ok", mode=SearchMode.fts, repeat=0, latency_ms=10.0,
hit_at_1=True, recall=1.0, reciprocal_rank=1.0,
citation_hit=True, citation_applicable=True,
),
RAGCaseResult(
case_id="boom", mode=SearchMode.fts, repeat=0, latency_ms=0.0,
error="RAG case evaluation failed.",
error_code="BENCHMARK_CASE_EVALUATION_FAILED",
),
]
metrics = rag_module._aggregate(cases, SearchMode.fts)
assert metrics.total_cases == 2
assert metrics.successful_cases == 1
assert metrics.failed_cases == 1
assert metrics.failure_rate == 0.5
# 失败样本按零分计入质量指标分母,汇总不虚高
assert metrics.hit_at_1 == 0.5
assert metrics.recall_at_k == 0.5
# 延迟只统计成功样本
assert metrics.p50_latency_ms == 10.0
def test_cancel_emits_run_cancelled_event() -> None:
_, _, case = _single_note_case()
_write_dataset("cancel-event-v1", [case])
async def _scenario():
run = await service.create_rag_run(
RAGRunRequest(dataset_id="cancel-event-v1", modes=[SearchMode.fts])
)
service.cancel_run(run.run_id)
return await service.wait_for_run(run.run_id)
run = asyncio.run(_scenario())
assert run.status.value == "cancelled"
events = service.get_events(run.run_id)
assert events[-1].event.value == "RunCancelled"
def test_load_dataset_ignores_corrupted_unrelated_files() -> None:
# 无关文件损坏(非法 JSON / 顶层非对象)不应阻断目标数据集加载
directory = get_settings().benchmark_datasets_path
directory.mkdir(parents=True, exist_ok=True)
(directory / "broken.json").write_text("{ not valid json", encoding="utf-8")
(directory / "array.json").write_text('["a", "b"]', encoding="utf-8")
_write_dataset("ok-v1", [{"case_id": "x", "query": "q", "expected_note_ids": ["n"]}])
dataset = datasets.load_dataset("ok-v1", BenchmarkKind.rag)
assert dataset.dataset_id == "ok-v1"
assert len(dataset.cases) == 1
def test_load_dataset_top_level_must_be_object() -> None:
_write_raw("array-top", ["a", "b"])
with pytest.raises(ApiError) as exc:
datasets.load_dataset("array-top", BenchmarkKind.rag)
assert exc.value.code == "BENCHMARK_DATASET_INVALID"
# --------------------------------------------------------------------------- #
# 审阅回归:运行中取消 / 仅块标注 / SSE 终止事件
# --------------------------------------------------------------------------- #
def test_cancel_running_benchmark_stops_early() -> None:
"""运行中取消应在样本边界及时生效,而非跑完全部样本(审阅 P1)。"""
from app.benchmarks import service
from app.services import note_service
note = asyncio.run(
note_service.create_note(
title="取消回归", markdown="向量数据库用于存储高维向量。", folder="", tags=["向量"]
)
)
cases = [
{
"case_id": f"c{i}",
"query": "向量数据库",
"expected_note_ids": [note.note_id],
"expected_block_ids": [note.blocks[0].block_id],
"citation_required": True,
}
for i in range(50)
]
_write_dataset("cancel-running-v1", cases)
async def _scenario():
run = await service.create_rag_run(
RAGRunRequest(dataset_id="cancel-running-v1", modes=[SearchMode.fts])
)
async def _cancel_after_start():
# 取消通过事件循环调度(独立 Task),而非同步直调,才能复现事件循环饥饿
while service.get_run(run.run_id).status == BenchmarkStatus.queued:
await asyncio.sleep(0)
service.cancel_run(run.run_id)
cancel_task = asyncio.create_task(_cancel_after_start())
finished = await service.wait_for_run(run.run_id)
await cancel_task
return finished
run = asyncio.run(_scenario())
assert run.status.value == "cancelled"
completed = sum(
1 for e in service.get_events(run.run_id) if e.event.value == "CaseCompleted"
)
assert completed < 50 # 未跑完全部样本,证明取消在样本边界生效
def test_block_only_annotation_resolves_note_and_scores() -> None:
"""仅标注 expected_block_ids 的样本应按块反查笔记评分,而非零分(审阅 P2)。"""
from app.services import note_service
note = asyncio.run(
note_service.create_note(
title="仅块标注", markdown="向量数据库存储高维向量。", folder="", tags=["向量"]
)
)
_write_dataset("block-only-v1", [{
"case_id": "c1",
"query": "向量数据库",
"expected_block_ids": [note.blocks[0].block_id],
"citation_required": False,
}])
run = _run(RAGRunRequest(dataset_id="block-only-v1", modes=[SearchMode.fts]))
assert run.status.value == "completed"
fts = run.metrics["fts"]
assert fts["hit_at_1"] == 1.0
assert fts["recall_at_k"] == 1.0
assert fts["mrr"] == 1.0
def test_sse_stream_ends_on_terminal_event_in_replay() -> None:
"""历史回放期间遇到终止事件时流应立即结束,而非进入实时队列永久等待(审阅 P2)。"""
from app import routes
from app.benchmarks import service
from app.contracts import BenchmarkEvent, BenchmarkEventType
run_id = "benchmark_sse_replay"
now = service._now()
# 模拟「回放期间运行完成」:run 仍为 runningsubscribe 返回非空队列),
# 但历史事件里已含 RunCompleted 终止事件。
service._runs[run_id] = BenchmarkRun(
run_id=run_id,
kind=BenchmarkKind.rag,
dataset_id="d",
dataset_hash="sha256:x",
status=BenchmarkStatus.running,
created_at=now,
)
service._events[run_id] = [
BenchmarkEvent(
event=BenchmarkEventType.run_started, run_id=run_id, sequence=0,
data={}, timestamp=now,
),
BenchmarkEvent(
event=BenchmarkEventType.run_completed, run_id=run_id, sequence=1,
data={}, timestamp=now,
),
]
try:
# 直调路由函数时 FastAPI 不解析 Query/Header 默认值,需显式传 None 覆盖 Header 哨兵
response = asyncio.run(
routes.benchmark_events(run_id, after_sequence=-1, last_event_id=None)
)
async def _collect() -> list[str]:
out: list[str] = []
async for chunk in response.body_iterator:
out.append(chunk)
return out
# 加超时防止回归(旧实现会永久挂起)
chunks = asyncio.run(asyncio.wait_for(_collect(), timeout=5))
finally:
service._forget(run_id)
events = [
line for chunk in chunks for line in chunk.splitlines() if line.startswith("event: ")
]
assert events == ["event: RunStarted", "event: RunCompleted"]
+132
View File
@@ -0,0 +1,132 @@
import asyncio
from pathlib import Path
import httpx
import pytest
from app.config import get_settings
from app.contracts import CredentialWriteRequest
from app.errors import ApiError
from app.providers.credentials import (
ChainedCredentialResolver,
CredentialStoreError,
EncryptedCredentialStore,
EnvironmentCredentialResolver,
)
from app.providers.factory import ProviderFactory
from app.providers.openai_compatible import OpenAICompatibleProvider
from app.routes import delete_credential, get_credential_status, put_credential
def test_encrypted_credential_store_round_trip_without_plaintext_on_disk() -> None:
store = EncryptedCredentialStore()
secret = "sk-test-sensitive-value"
store.put("deepseek", secret)
store_path = get_settings().data_dir / "credentials" / "credentials.json"
key_path = get_settings().data_dir / "credentials" / "master.key"
assert store_path.exists()
assert key_path.exists()
assert secret not in store_path.read_text(encoding="utf-8")
assert secret not in key_path.read_text(encoding="ascii")
assert store.resolve("deepseek") == secret
assert store.has("deepseek") is True
assert store.delete("deepseek") is True
assert store.resolve("deepseek") is None
def test_encrypted_credential_store_deletes_multiple_credentials_atomically() -> None:
store = EncryptedCredentialStore()
store.put("plugin.first", "first")
store.put("plugin.second", "second")
store.put("openai", "keep")
removed = store.delete_many(["plugin.first", "plugin.second"])
assert removed == {"plugin.first", "plugin.second"}
assert store.resolve("plugin.first") is None
assert store.resolve("plugin.second") is None
assert store.resolve("openai") == "keep"
def test_credential_write_os_error_uses_stable_store_error(monkeypatch) -> None:
store = EncryptedCredentialStore()
store.put("existing", "value")
def fail_replace(_path: Path, _target: Path) -> Path:
raise OSError("injected replace failure")
monkeypatch.setattr(Path, "replace", fail_replace)
with pytest.raises(CredentialStoreError, match="cannot be written"):
store.put("new", "value")
def test_credential_api_never_returns_secret() -> None:
written = asyncio.run(
put_credential(
"deepseek",
CredentialWriteRequest(api_key="sk-test-sensitive-value"),
)
)
status = asyncio.run(get_credential_status("deepseek"))
assert written.model_dump() == {"credential_id": "deepseek", "configured": True}
assert status.configured is True
assert "sk-test-sensitive-value" not in written.model_dump_json()
def test_provider_reads_decrypted_api_key_from_encrypted_store() -> None:
store = EncryptedCredentialStore()
store.put("deepseek", "sk-test-sensitive-value")
def handler(request: httpx.Request) -> httpx.Response:
assert request.headers["Authorization"] == "Bearer sk-test-sensitive-value"
return httpx.Response(200, json={"data": [{"id": "deepseek-chat"}]})
provider = OpenAICompatibleProvider(
base_url="https://api.deepseek.test",
credential_id="deepseek",
credentials=store,
transport=httpx.MockTransport(handler),
)
models = asyncio.run(provider.list_models())
assert [model.model for model in models] == ["deepseek-chat"]
def test_saved_credential_takes_precedence_over_environment_fallback(monkeypatch) -> None:
monkeypatch.setenv("DEEPSEEK_API_KEY", "environment-key")
store = EncryptedCredentialStore()
store.put("deepseek", "saved-key")
resolver = ChainedCredentialResolver(store, EnvironmentCredentialResolver())
assert resolver.resolve("deepseek") == "saved-key"
def test_public_credential_api_rejects_plugin_namespace() -> None:
operations = [
get_credential_status("plugin.text-tools.api_key"),
put_credential(
"plugin.text-tools.api_key",
CredentialWriteRequest(api_key="must-not-write"),
),
delete_credential("plugin.text-tools.api_key"),
]
for operation in operations:
with pytest.raises(ApiError) as exc:
asyncio.run(operation)
assert exc.value.code == "CREDENTIAL_NAMESPACE_RESERVED"
assert EncryptedCredentialStore().resolve("plugin.text-tools.api_key") is None
def test_provider_resolver_cannot_read_plugin_secret() -> None:
store = EncryptedCredentialStore()
store.put("plugin.text-tools.api_key", "private-plugin-secret")
resolver = ProviderFactory(store).credentials
with pytest.raises(CredentialStoreError, match="reserved for Plugin settings"):
resolver.resolve("plugin.text-tools.api_key")
+303
View File
@@ -0,0 +1,303 @@
"""Export Service 的单元与端到端测试。
沿用 conftest 隔离机制APP_DATA_DIR / DB / Vault / exports 目录都落在临时目录
不读写真实数据导出采用创建即 queued + 后台 Task 执行的异步模型测试在同一
事件循环内创建并等待后台任务结束得到终态 ExportJob 后再断言
"""
from __future__ import annotations
import asyncio
from datetime import datetime, timedelta, timezone
import pytest
from pydantic import ValidationError
from app.config import get_settings
from app.contracts import (
ExportFormat,
ExportOptions,
ExportRequest,
ExportSource,
ExportSourceType,
ExportStatus,
)
from app.errors import ApiError
from app.export import service as export_service
from app.export.exporters.html import HtmlExporter
from app.export.markdown import parse_document
MD = """# 进程调度
一些 **加粗** *斜体*[链接](https://a.b) `code`
- 项目一
- 项目二
```python
print(1)
```
```mermaid
graph LR
```
```function_plot
y = x
```
| a | b |
|---|---|
| 1 | 2 |
行内 $x^2$ 与块级
$$
y = mx + b
$$
"""
@pytest.fixture(autouse=True)
def _reset_export_state():
"""清空内存注册表,避免跨用例的任务/取消标志互相污染。"""
export_service._jobs.clear()
export_service._tasks.clear()
export_service._cancel_flags.clear()
yield
export_service._jobs.clear()
export_service._tasks.clear()
export_service._cancel_flags.clear()
def _create_and_wait(request: ExportRequest) -> object:
"""创建导出并在同一事件循环内等待后台任务结束,返回终态 ExportJob。"""
async def _execute():
job = await export_service.create_export(request)
return await export_service.wait_for_export(job.job_id)
return asyncio.run(_execute())
# --------------------------------------------------------------------------- #
# markdown → Document AST
# --------------------------------------------------------------------------- #
def _types(nodes) -> list[str]:
return [n.type for n in nodes]
def test_parse_document_heading_and_inline() -> None:
doc = parse_document("# 标题\n\n一段 **加粗** 和 [链接](https://a.b)。")
assert doc.type == "document"
heading = doc.children[0]
assert heading.type == "heading"
assert heading.attributes["level"] == 1
para = doc.children[1]
assert para.type == "paragraph"
kinds = _types(para.children)
assert "text" in kinds
assert "strong" in kinds
assert "link" in kinds
link = next(c for c in para.children if c.type == "link")
assert link.attributes["href"] == "https://a.b"
def test_parse_document_list_and_code_fencing() -> None:
doc = parse_document("- a\n- b\n\n```mermaid\ngraph LR\n```\n\n```function_plot\ny=x\n```\n\n```python\nx\n```")
kinds = [c.type for c in doc.children]
assert kinds[0] == "list"
assert kinds[1] == "mermaid"
assert kinds[2] == "function_plot"
assert kinds[3] == "code_block"
code = doc.children[3]
assert code.attributes["language"] == "python"
assert code.text == "x"
def test_parse_document_table_and_math() -> None:
doc = parse_document("| a | b |\n|---|---|\n| 1 | 2 |\n\n$x^2$\n\n$$\ny=mx\n$$")
table = doc.children[0]
assert table.type == "table"
assert table.children[0].type == "table_row"
assert table.children[0].children[0].attributes["head"] is True
# 表格后是「行内数学所在段落」与「块级数学」
kinds = [c.type for c in doc.children[1:]]
assert "paragraph" in kinds
assert "math_block" in kinds
# --------------------------------------------------------------------------- #
# HtmlExporter
# --------------------------------------------------------------------------- #
async def _render(markdown: str, *, title: str = "") -> str:
doc = parse_document(markdown)
doc.attributes["title"] = title
result = await HtmlExporter().export(doc, ExportOptions())
return result.content.decode("utf-8")
def test_html_exporter_renders_basic_nodes_and_escapes() -> None:
html = asyncio.run(_render("# 标题\n\n**加粗** [链接](https://a.b) 与 <b>原始</b>。"))
assert "<h1>标题</h1>" in html
assert "<strong>加粗</strong>" in html
assert '<a href="https://a.b">链接</a>' in html
# 原始 HTML 必须被转义,不能注入文档
assert "&lt;b&gt;原始&lt;/b&gt;" in html
assert "<b>原始</b>" not in html
def test_html_exporter_marks_mermaid_and_function_plot() -> None:
result = asyncio.run(HtmlExporter().export(parse_document("```mermaid\ngraph LR\n```"), ExportOptions()))
html = result.content.decode("utf-8")
assert '<pre class="mermaid">graph LR</pre>' in html
assert any("mermaid" in w for w in result.warnings)
def test_html_exporter_include_title_and_metadata() -> None:
doc = parse_document("正文")
doc.attributes["title"] = "操作系统复习"
doc.attributes["metadata"] = {"tags": ["os", "复习"]}
opts = ExportOptions(include_title=True, include_metadata=True)
result = asyncio.run(HtmlExporter().export(doc, opts))
html = result.content.decode("utf-8")
assert '<h1 class="title">操作系统复习</h1>' in html
assert "os, 复习" in html
# --------------------------------------------------------------------------- #
# ExportService
# --------------------------------------------------------------------------- #
def _markdown_request(markdown: str, *, format: ExportFormat = ExportFormat.html) -> ExportRequest:
return ExportRequest(
source=ExportSource(type=ExportSourceType.markdown, markdown=markdown),
format=format,
)
def test_export_markdown_source_completes_and_writes_file() -> None:
finished = _create_and_wait(_markdown_request(MD))
assert finished.status == ExportStatus.completed
assert finished.file is not None
assert finished.file.mime_type == "text/html"
assert finished.file.size > 0
assert len(finished.file.sha256) == 64
path = get_settings().exports_path / f"{finished.job_id}.html"
assert path.exists()
content = path.read_text(encoding="utf-8")
assert "进程调度" in content
def test_export_note_source_resolves_title_and_metadata() -> None:
from app.services import note_service
async def _go():
note = await note_service.create_note(
title="操作系统复习", markdown="# 进程调度\n\n内容。", folder="导出", tags=["os"]
)
request = ExportRequest(
source=ExportSource(type=ExportSourceType.note, note_id=note.note_id),
format=ExportFormat.html,
options=ExportOptions(include_metadata=True),
)
job = await export_service.create_export(request)
return await export_service.wait_for_export(job.job_id)
finished = asyncio.run(_go())
assert finished.status == ExportStatus.completed
assert finished.file is not None
assert finished.file.file_name == "操作系统复习.html"
content = (get_settings().exports_path / f"{finished.job_id}.html").read_text(encoding="utf-8")
assert "操作系统复习" in content
assert "进程调度" in content
def test_export_pdf_unsupported() -> None:
with pytest.raises(ApiError) as exc:
asyncio.run(
export_service.create_export(_markdown_request("# x", format=ExportFormat.pdf))
)
assert exc.value.status_code == 400
assert exc.value.code == "EXPORT_FORMAT_UNSUPPORTED"
def test_export_unknown_note_404() -> None:
request = ExportRequest(
source=ExportSource(type=ExportSourceType.note, note_id="note_missing"),
format=ExportFormat.html,
)
with pytest.raises(ApiError) as exc:
asyncio.run(export_service.create_export(request))
assert exc.value.status_code == 404
assert exc.value.code == "EXPORT_SOURCE_NOT_FOUND"
def test_export_empty_markdown_invalid() -> None:
with pytest.raises(ApiError) as exc:
asyncio.run(export_service.create_export(_markdown_request(" ")))
assert exc.value.status_code == 400
assert exc.value.code == "EXPORT_OPTIONS_INVALID"
def test_export_cancel_queued_job() -> None:
async def _go():
job = await export_service.create_export(_markdown_request("# x"))
cancelled = export_service.cancel_export(job.job_id)
assert cancelled is not None
return await export_service.wait_for_export(job.job_id)
finished = asyncio.run(_go())
assert finished.status == ExportStatus.cancelled
assert finished.file is None
def test_export_file_expired_410() -> None:
async def _go():
job = await export_service.create_export(_markdown_request("# x"))
finished = await export_service.wait_for_export(job.job_id)
past = datetime.now(timezone.utc) - timedelta(hours=1)
export_service._jobs[job.job_id] = finished.model_copy(
update={"file": finished.file.model_copy(update={"expires_at": past})}
)
return job.job_id
job_id = asyncio.run(_go())
with pytest.raises(ApiError) as exc:
export_service.get_export_file(job_id)
assert exc.value.status_code == 410
assert exc.value.code == "EXPORT_FILE_EXPIRED"
def test_export_list_and_get() -> None:
finished = _create_and_wait(_markdown_request("# 列表测试"))
items, total = export_service.list_exports(limit=50, offset=0)
assert total == 1
assert items[0].job_id == finished.job_id
got = export_service.get_export(finished.job_id)
assert got is not None and got.status == ExportStatus.completed
assert export_service.get_export("export_missing") is None
# --------------------------------------------------------------------------- #
# 契约校验
# --------------------------------------------------------------------------- #
def test_export_source_requires_matching_field() -> None:
with pytest.raises(ValidationError):
ExportSource(type=ExportSourceType.note, note_id=None)
with pytest.raises(ValidationError):
ExportSource(type=ExportSourceType.markdown, markdown=None)
+478 -1
View File
@@ -1,4 +1,7 @@
import asyncio
import shutil
import threading
import time
import pytest
@@ -8,18 +11,39 @@ from app.container import build_container
from app.contracts import (
AgentRunCreateRequest,
AgentRunStatus,
PluginCommandContext,
SkillStatus,
ToolCall,
)
from app.extensions import ExtensionError
from app.extensions.mcp import McpStdioClient
from app.extensions.runtime import (
_arguments_model_from_schema,
_validate_mcp_command_target_schema,
)
from app.services import note_service
from app.config import get_settings
from app.config import BACKEND_DIR, get_settings
MCP_FIXTURE = BACKEND_DIR / "extensions" / "fixtures" / "mcp-echo"
def run(coroutine):
return asyncio.run(coroutine)
@pytest.fixture
def mcp_container():
container = build_container()
installed = container.plugins.install(MCP_FIXTURE)
assert installed.status == "permission_required"
container.plugins.set_permissions("mcp-fixture", ["notes.read", "secrets.use"])
try:
yield container
finally:
container.plugins.shutdown()
def test_bundled_plugin_registers_tool_and_skill_is_ready() -> None:
async def scenario() -> None:
container = build_container()
@@ -297,3 +321,456 @@ def test_attachment_and_transcription_tools_use_host_storage() -> None:
assert transcription.output["text"] == "会议转写内容"
run(scenario())
def test_mcp_stdio_host_discovers_namespaced_tools_and_maps_results(
mcp_container, monkeypatch
) -> None:
async def scenario() -> None:
monkeypatch.setenv("OPENAI_API_KEY", "must-not-enter-plugin-host")
enabled = mcp_container.plugins.enable("mcp-fixture")
status = mcp_container.plugins.get_host_status("mcp-fixture")
definition = mcp_container.tools.get("mcp-fixture.echo").definition
result = await mcp_container.tools.execute(
ToolCall(
tool_call_id="call_mcp_echo",
name="mcp-fixture.echo",
arguments={"text": "hello mcp"},
),
ToolExecutionContext(
run_id="run_mcp_fixture", tool_call_id="call_mcp_echo"
),
)
assert enabled.status == "ready" and enabled.enabled is True
assert status.status == "ready"
environment = await mcp_container.tools.execute(
ToolCall(
tool_call_id="call_mcp_environment",
name="mcp-fixture.environment",
arguments={},
),
ToolExecutionContext(run_id="run_mcp_fixture"),
)
assert status.tools_count == 7
assert status.protocol_version == "2025-11-25"
assert status.server_name == "notesagent-mcp-fixture"
assert definition.permission == "notes.read"
assert result.success is True
assert result.output == {"echo": "hello mcp"}
explicit_null = await mcp_container.tools.execute(
ToolCall(
tool_call_id="call_mcp_explicit_null",
name="mcp-fixture.echo",
arguments={"text": "null stays explicit", "suffix": None},
),
ToolExecutionContext(run_id="run_mcp_fixture"),
)
assert explicit_null.success is True
assert explicit_null.output == {
"echo": "null stays explicit",
"suffix": None,
}
assert environment.success is True
assert environment.output == {
"has_openai_key": False,
"has_app_db_path": False,
}
disabled = mcp_container.plugins.disable("mcp-fixture")
assert disabled.status == "disabled"
assert mcp_container.plugins.get_host_status("mcp-fixture").status == "stopped"
assert not mcp_container.tools.contains("mcp-fixture.echo")
with pytest.raises(ExtensionError) as exc:
mcp_container.plugins.restart_host("mcp-fixture")
assert exc.value.code == "PLUGIN_HOST_UNAVAILABLE"
assert mcp_container.plugins.get("mcp-fixture").status == "disabled"
assert not mcp_container.tools.contains("mcp-fixture.echo")
mcp_container.plugins.uninstall("mcp-fixture")
reinstalled = mcp_container.plugins.install(MCP_FIXTURE)
fresh_status = mcp_container.plugins.get_host_status("mcp-fixture")
assert reinstalled.status == "permission_required"
assert fresh_status.status == "stopped"
assert fresh_status.started_at is None
assert fresh_status.protocol_version is None
assert fresh_status.server_name is None
run(scenario())
def test_mcp_command_target_receives_scoped_context_and_declared_secret(
mcp_container,
) -> None:
async def scenario() -> None:
mcp_container.plugins.enable("mcp-fixture")
assert not mcp_container.tools.contains("mcp-fixture.command")
with pytest.raises(ExtensionError) as missing:
await mcp_container.plugins.execute_command(
"mcp-fixture.notify",
{},
PluginCommandContext(selection="来自选区"),
)
assert missing.value.code == "PLUGIN_SECRET_REQUIRED"
mcp_container.plugins.put_setting_secret(
"mcp-fixture", "api_key", "mcp-command-secret"
)
mcp_container.plugins.update_settings(
"mcp-fixture", 1, {"message_prefix": "Fixture: "}
)
result = await mcp_container.plugins.execute_command(
"mcp-fixture.notify",
{},
PluginCommandContext(
note_id="must-not-enter-envelope",
selection="来自选区",
),
)
assert result.effect.type == "notification"
assert result.effect.payload.model_dump() == {
"level": "success",
"message": "Fixture: 来自选区",
}
assert "mcp-command-secret" not in repr(
mcp_container.plugins.commands.audit_events()
)
run(scenario())
def test_mcp_command_target_rejects_incompatible_envelope_schema(tmp_path) -> None:
package = tmp_path / "mcp-bad-command"
shutil.copytree(MCP_FIXTURE, package)
for filename in ("plugin.yaml", "commands.yaml", "settings.yaml"):
path = package / filename
path.write_text(
path.read_text(encoding="utf-8").replace(
"mcp-fixture", "mcp-bad-command"
),
encoding="utf-8",
)
server_path = package / "server.py"
server_path.write_text(
server_path.read_text(encoding="utf-8").replace(
'{"_notesagent": {"type": "object"}}',
'{"unexpected": {"type": "string"}}',
),
encoding="utf-8",
)
container = build_container()
container.plugins.install(package)
container.plugins.set_permissions(
"mcp-bad-command", ["notes.read", "secrets.use"]
)
try:
with pytest.raises(ExtensionError) as exc:
container.plugins.enable("mcp-bad-command")
assert exc.value.code == "PLUGIN_CONTRIBUTION_INVALID"
assert container.plugins.get("mcp-bad-command").status == "error"
finally:
container.plugins.shutdown()
def test_mcp_command_target_enable_check_only_requires_protocol_marker() -> None:
# `not`/`oneOf` 等完整语义由实际调用前的官方 Validator 处理;启用检查
# 只确认不可被引用或组合隐藏的稳定宿主入口,避免维护不完整的求解器。
_validate_mcp_command_target_schema(
{
"type": "object",
"properties": {
"_notesagent": {
"type": "object",
"not": {"type": "object"},
}
},
},
"marker.run",
)
invalid_markers = [
{
"$defs": {"envelope": {"type": "object"}},
"properties": {"_notesagent": {"$ref": "#/$defs/envelope"}},
},
{
"allOf": [
{"properties": {"_notesagent": {"type": "object"}}},
]
},
]
for schema in invalid_markers:
with pytest.raises(ExtensionError) as exc:
_validate_mcp_command_target_schema(schema, "marker.run")
assert exc.value.code == "PLUGIN_CONTRIBUTION_INVALID"
def test_mcp_command_validates_actual_envelope_before_call(tmp_path) -> None:
package = tmp_path / "mcp-runtime-schema"
shutil.copytree(MCP_FIXTURE, package)
for filename in ("plugin.yaml", "commands.yaml", "settings.yaml"):
path = package / filename
path.write_text(
path.read_text(encoding="utf-8").replace(
"mcp-fixture", "mcp-runtime-schema"
),
encoding="utf-8",
)
server_path = package / "server.py"
server_path.write_text(
server_path.read_text(encoding="utf-8").replace(
'{"_notesagent": {"type": "object"}}',
'{"_notesagent": {"type": "object", "properties": '
'{"arguments": {"type": "object", "maxProperties": 0}, '
'"context": {"type": "object", "properties": '
'{"selection": {"type": "string"}}, "required": ["selection"]}}, '
'"required": ["arguments", "context"]}}',
),
encoding="utf-8",
)
container = build_container()
container.plugins.install(package)
container.plugins.set_permissions(
"mcp-runtime-schema", ["notes.read", "secrets.use"]
)
try:
# context.selection 是 Command 的 when/context 契约保证的真实字段;
# 启用期结构检查不得因没有伪造该业务值而拒绝目标 Schema。
container.plugins.enable("mcp-runtime-schema")
container.plugins.put_setting_secret(
"mcp-runtime-schema", "api_key", "configured"
)
with pytest.raises(ExtensionError) as exc:
run(
container.plugins.execute_command(
"mcp-runtime-schema.notify",
{"message": "must be rejected locally"},
PluginCommandContext(selection="visible"),
)
)
assert exc.value.code == "PLUGIN_COMMAND_TARGET_SCHEMA_MISMATCH"
finally:
container.plugins.shutdown()
def test_agent_calls_mcp_tool_through_registry_and_writes_trace(mcp_container) -> None:
async def scenario() -> None:
mcp_container.plugins.enable("mcp-fixture")
created = await mcp_container.agent.create_run(
AgentRunCreateRequest(
input='/tool mcp-fixture.echo {"text":"agent mcp"}',
provider_id="mock",
model="mock-1",
allowed_tools=["mcp-fixture.echo"],
)
)
completed = await mcp_container.agent.wait(created.run_id)
trace = mcp_container.agent.get_trace(
created.run_id, after_sequence=-1, limit=100
)
assert completed.status == AgentRunStatus.completed
assert completed.tool_results[0].success is True
assert completed.tool_results[0].output == {"echo": "agent mcp"}
assert any(
item.event == "ToolCall" and item.data.get("name") == "mcp-fixture.echo"
for item in trace.items
)
run(scenario())
def test_mcp_business_error_size_limit_and_timeout_are_structured(mcp_container) -> None:
async def scenario() -> None:
mcp_container.plugins.enable("mcp-fixture")
context = ToolExecutionContext(run_id="run_mcp_errors")
failed = await mcp_container.tools.execute(
ToolCall(tool_call_id="call_fail", name="mcp-fixture.fail", arguments={}),
context,
)
oversized = await mcp_container.tools.execute(
ToolCall(tool_call_id="call_large", name="mcp-fixture.large", arguments={}),
context,
)
timed_out = await mcp_container.tools.execute(
ToolCall(
tool_call_id="call_sleep",
name="mcp-fixture.sleep",
arguments={"seconds": 5},
),
ToolExecutionContext(
run_id="run_mcp_errors", tool_call_id="call_sleep"
),
)
recovered = await mcp_container.tools.execute(
ToolCall(
tool_call_id="call_after_timeout",
name="mcp-fixture.echo",
arguments={"text": "still ready"},
),
context,
)
assert failed.success is False
assert failed.error_code == "MCP_TOOL_CALL_FAILED"
assert failed.error_message == "fixture failure"
assert oversized.success is False
assert oversized.error_code == "MCP_TOOL_RESULT_TOO_LARGE"
assert timed_out.success is False
assert timed_out.error_code == "MCP_TOOL_CALL_FAILED"
assert recovered.success is True
assert mcp_container.plugins.get_host_status("mcp-fixture").status == "ready"
run(scenario())
def test_mcp_cancel_releases_blocking_response_thread(
mcp_container, monkeypatch
) -> None:
async def scenario() -> None:
mcp_container.plugins.enable("mcp-fixture")
released = threading.Event()
original_wait = McpStdioClient.wait_response
def tracked_wait(self, *args, **kwargs):
try:
return original_wait(self, *args, **kwargs)
finally:
released.set()
monkeypatch.setattr(McpStdioClient, "wait_response", tracked_wait)
task = asyncio.create_task(
mcp_container.plugins.mcp.call_tool(
"mcp-fixture",
"sleep",
{"seconds": 5},
request_id="call_cancel_release",
)
)
await asyncio.sleep(0.05)
task.cancel()
with pytest.raises(asyncio.CancelledError):
await task
deadline = time.monotonic() + 0.5
while not released.is_set() and time.monotonic() < deadline:
await asyncio.sleep(0.01)
assert released.is_set(), "cancelled MCP wait must not occupy a worker until timeout"
run(scenario())
def test_mcp_argument_model_preserves_json_schema_additional_properties() -> None:
arguments_model = _arguments_model_from_schema(
"mcp-fixture.dynamic",
{
"type": "object",
"properties": {"model_dump": {"type": "string"}},
"required": ["model_dump"],
"additionalProperties": {"type": "string"},
},
)
arguments = arguments_model.model_validate(
{"model_dump": "method name remains data", "dynamic-key": "value"}
)
assert arguments.model_dump() == {
"model_dump": "method name remains data",
"dynamic-key": "value",
}
def test_production_rejects_unsandboxed_mcp_host(monkeypatch) -> None:
monkeypatch.setenv("APP_ENVIRONMENT", "production")
get_settings.cache_clear()
container = build_container()
installed = container.plugins.install(MCP_FIXTURE)
assert installed.status == "permission_required"
container.plugins.set_permissions("mcp-fixture", ["notes.read", "secrets.use"])
try:
with pytest.raises(ExtensionError) as exc:
container.plugins.enable("mcp-fixture")
assert exc.value.code == "MCP_TRUST_APPROVAL_REQUIRED"
assert container.plugins.get_host_status("mcp-fixture").status == "stopped"
assert not container.tools.contains("mcp-fixture.echo")
finally:
container.plugins.shutdown()
get_settings.cache_clear()
def test_mcp_abnormal_exit_unregisters_tools_and_restart_recovers(mcp_container) -> None:
async def scenario() -> None:
mcp_container.plugins.enable("mcp-fixture")
crashed = await mcp_container.tools.execute(
ToolCall(tool_call_id="call_exit", name="mcp-fixture.exit", arguments={}),
ToolExecutionContext(run_id="run_mcp_exit", tool_call_id="call_exit"),
)
deadline = time.monotonic() + 2
while mcp_container.tools.contains("mcp-fixture.echo") and time.monotonic() < deadline:
await asyncio.sleep(0.02)
plugin = mcp_container.plugins.get("mcp-fixture")
status = mcp_container.plugins.get_host_status("mcp-fixture")
assert crashed.success is False
assert crashed.error_code == "PLUGIN_HOST_UNAVAILABLE"
assert plugin.status == "error" and plugin.enabled is False
assert status.status == "unhealthy"
assert not mcp_container.tools.contains("mcp-fixture.echo")
restarted = mcp_container.plugins.restart_host("mcp-fixture")
assert restarted.status == "ready"
assert restarted.tools_count == 7
assert mcp_container.tools.contains("mcp-fixture.echo")
run(scenario())
@pytest.mark.parametrize(
("mode", "contributions", "expected_code"),
[
("no-tools", "[]", "MCP_CAPABILITY_UNSUPPORTED"),
("invalid-schema", "[mcp-invalid.broken]", "MCP_TOOL_SCHEMA_INVALID"),
("invalid-result", "[]", "MCP_INITIALIZE_FAILED"),
("oversized-stdout", "[]", "PLUGIN_HOST_UNAVAILABLE"),
],
)
def test_mcp_rejects_invalid_initialization_and_discovery(
tmp_path, mode, contributions, expected_code
) -> None:
package = tmp_path / f"mcp-{mode}"
package.mkdir()
shutil.copyfile(MCP_FIXTURE / "server.py", package / "server.py")
(package / "plugin.yaml").write_text(
f"""
id: mcp-invalid
name: Invalid MCP Fixture
version: 1.0.0
contributes:
tools: {contributions}
backend:
type: mcp
transport: stdio
command: python
args: [server.py, {mode}]
startup_timeout_seconds: 5
tool_timeout_seconds: 1
""".strip(),
encoding="utf-8",
)
container = build_container()
container.plugins.install(package)
try:
with pytest.raises(ExtensionError) as exc:
container.plugins.enable("mcp-invalid")
assert exc.value.code == expected_code
assert container.plugins.get("mcp-invalid").status == "error"
assert container.plugins.get_host_status("mcp-invalid").status == "error"
assert not container.tools.contains("mcp-invalid.broken")
finally:
container.plugins.shutdown()
+769
View File
@@ -0,0 +1,769 @@
import asyncio
import hashlib
import json
import sys
import threading
import time
from concurrent.futures import ThreadPoolExecutor
import httpx
import pytest
from app.agent.tools import ToolExecutionContext, ToolRegistry
from app.config import BACKEND_DIR, get_settings
from app.contracts import McpServerCreateRequest, McpServerUpdateRequest, ToolCall
from app.extensions.mcp import McpLegacySseClient
from app.extensions.mcp_registry import McpRegistryError, McpServerRegistry
from app.providers.credentials import CredentialStoreError, EncryptedCredentialStore
SERVER = BACKEND_DIR / "extensions" / "fixtures" / "mcp-echo" / "server.py"
def request(**overrides) -> McpServerCreateRequest:
values = {
"name": "Echo MCP",
"command": sys.executable,
"args": [str(SERVER)],
"permissions": ["notes.read", "secrets.use"],
"secret_environment_keys": ["TEST_MCP_SECRET"],
}
values.update(overrides)
return McpServerCreateRequest(**values)
def registry(*, launch: bool = True) -> McpServerRegistry:
return McpServerRegistry(
ToolRegistry(),
EncryptedCredentialStore(),
get_settings().data_dir,
allow_process_launch=launch,
)
def test_registry_requires_current_trust_and_never_returns_secret() -> None:
service = registry()
created = service.create(request())
assert created.trusted is False
assert created.secret_environment == {"TEST_MCP_SECRET": False}
service.put_secret(created.server_id, "TEST_MCP_SECRET", "do-not-return")
configured = service.get(created.server_id)
assert configured.secret_environment == {"TEST_MCP_SECRET": True}
assert "do-not-return" not in configured.model_dump_json()
with pytest.raises(McpRegistryError, match="approve"):
service.test(created.server_id)
service.trust(created.server_id, created.command_digest)
tested = service.test(created.server_id)
assert tested.status == "stopped"
assert tested.last_test_succeeded is True
assert tested.tools_count > 0
service.shutdown()
def test_secret_change_disables_server_and_requires_a_new_connection_test() -> None:
service = registry()
created = service.create(request())
service.put_secret(created.server_id, "TEST_MCP_SECRET", "first")
service.trust(created.server_id, created.command_digest)
service.test(created.server_id)
service.enable(created.server_id)
service.put_secret(created.server_id, "TEST_MCP_SECRET", "second")
current = service.get(created.server_id)
assert current.enabled is False
assert current.last_test_succeeded is None
assert not any(
item.name.startswith(f"mcp.{created.server_id}.")
for item in service.tools.definitions()
)
with pytest.raises(McpRegistryError) as error:
service.enable(created.server_id)
assert error.value.code == "MCP_CONNECTION_TEST_REQUIRED"
service.shutdown()
def test_update_disables_server_and_revokes_command_trust() -> None:
service = registry()
created = service.create(request(secret_environment_keys=[]))
service.trust(created.server_id, created.command_digest)
service.test(created.server_id)
enabled = service.enable(created.server_id)
assert enabled.enabled is True
assert any(
item.name.startswith(f"mcp.{created.server_id}.")
for item in service.tools.definitions()
)
updated = service.update(
created.server_id,
McpServerUpdateRequest(
**request(name="Changed", secret_environment_keys=[]).model_dump(),
version=enabled.version,
),
)
assert updated.enabled is False
assert updated.trusted is False
assert not any(
item.name.startswith(f"mcp.{created.server_id}.")
for item in service.tools.definitions()
)
service.shutdown()
def test_update_remains_retryable_when_removed_secret_cleanup_fails(
monkeypatch,
) -> None:
service = registry()
created = service.create(request())
service.put_secret(created.server_id, "TEST_MCP_SECRET", "keep-until-retry")
def fail_delete_many(_secret_ids: list[str]) -> set[str]:
raise CredentialStoreError("credential store unavailable")
monkeypatch.setattr(service.credentials, "delete_many", fail_delete_many)
with pytest.raises(McpRegistryError) as error:
service.update(
created.server_id,
McpServerUpdateRequest(
**request(secret_environment_keys=[]).model_dump(),
version=created.version,
),
)
current = service.get(created.server_id)
assert error.value.code == "MCP_SECRET_STORE_ERROR"
assert current.version == created.version
assert current.secret_environment == {"TEST_MCP_SECRET": True}
service.shutdown()
def test_delete_keeps_server_retryable_when_secret_cleanup_fails(monkeypatch) -> None:
service = registry()
created = service.create(request())
service.put_secret(created.server_id, "TEST_MCP_SECRET", "keep-until-retry")
def fail_delete_many(_secret_ids: list[str]) -> set[str]:
raise CredentialStoreError("credential store unavailable")
monkeypatch.setattr(service.credentials, "delete_many", fail_delete_many)
with pytest.raises(McpRegistryError) as error:
service.delete(created.server_id)
current = service.get(created.server_id)
assert error.value.code == "MCP_SECRET_STORE_ERROR"
assert current.server_id == created.server_id
assert current.secret_environment == {"TEST_MCP_SECRET": True}
service.shutdown()
def test_unavailable_server_removes_bridge_host(monkeypatch) -> None:
service = registry()
created = service.create(request(secret_environment_keys=[]))
with service._lock:
service._records[created.server_id] = {
**service._records[created.server_id],
"enabled": True,
}
removed: list[str] = []
monkeypatch.setattr(service.bridge, "remove", removed.append)
generation = object()
service._generations[created.server_id] = generation
service._unavailable(created.server_id, generation, "connection lost")
current = service.get(created.server_id)
assert removed == [f"mcp.{created.server_id}"]
assert current.enabled is False
assert current.status == "unhealthy"
service.shutdown()
def test_old_failure_callback_cannot_stop_replacement_host(monkeypatch) -> None:
service = registry()
callbacks = []
original_start = service.bridge.start
def capture_callback(*args, **kwargs):
callbacks.append(args[4])
return original_start(*args, **kwargs)
monkeypatch.setattr(service.bridge, "start", capture_callback)
created = service.create(request(secret_environment_keys=[]))
service.trust(created.server_id, created.command_digest)
callback_thread = None
try:
service.test(created.server_id)
service.enable(created.server_id)
old_callback = callbacks[-1]
callback_started = threading.Event()
callback_finished = threading.Event()
def delayed_failure():
callback_started.set()
old_callback(f"mcp.{created.server_id}", "delayed old failure")
callback_finished.set()
# Queue the old callback while a replacement owns the lifecycle lock.
with service._lifecycle_lock:
callback_thread = threading.Thread(target=delayed_failure, daemon=True)
callback_thread.start()
assert callback_started.wait(timeout=2)
service.disable(created.server_id)
service.enable(created.server_id)
assert callback_finished.wait(timeout=2)
assert service.get(created.server_id).enabled is True
assert service.get(created.server_id).status == "ready"
assert service.tools.definitions()
callbacks[-1](f"mcp.{created.server_id}", "current failure")
assert service.get(created.server_id).enabled is False
assert service.get(created.server_id).status == "unhealthy"
finally:
service.shutdown()
if callback_thread is not None:
callback_thread.join(timeout=2)
def test_header_case_only_rename_preserves_secret() -> None:
service = registry()
config = {
"name": "HTTP",
"transport": "streamable_http",
"url": "https://example.test/mcp",
"secret_header_keys": ["Authorization"],
}
created = service.create(McpServerCreateRequest(**config))
service.put_secret(created.server_id, "Authorization", "synthetic", kind="header")
config["secret_header_keys"] = ["authorization"]
updated = service.update(
created.server_id, McpServerUpdateRequest(**config, version=created.version)
)
assert updated.secret_headers == {"authorization": True}
assert (
service.credentials.resolve(
service._secret_id(created.server_id, "authorization", "header")
)
== "synthetic"
)
def test_environment_secrets_are_case_sensitive_and_delete_independently() -> None:
service = registry()
created = service.create(request(secret_environment_keys=["TOKEN", "token"]))
service.put_secret(created.server_id, "TOKEN", "upper")
service.put_secret(created.server_id, "token", "lower")
assert (
service.credentials.resolve(service._secret_id(created.server_id, "TOKEN"))
== "upper"
)
assert (
service.credentials.resolve(service._secret_id(created.server_id, "token"))
== "lower"
)
service.delete_secret(created.server_id, "TOKEN")
assert service.get(created.server_id).secret_environment == {
"TOKEN": False,
"token": True,
}
def test_legacy_environment_credential_migration_is_idempotent() -> None:
service = registry()
created = service.create(request(secret_environment_keys=["TOKEN"]))
suffix = hashlib.sha256(b"environment\0token").hexdigest()[:20]
legacy_id = f"mcp.{created.server_id}.{suffix}"
service.credentials.put(legacy_id, "legacy-value")
service._records[created.server_id]["secret_environment_version"] = 1
service._write()
migrated = registry()
assert migrated.get(created.server_id).secret_environment == {"TOKEN": True}
assert (
migrated.credentials.resolve(migrated._secret_id(created.server_id, "TOKEN"))
== "legacy-value"
)
assert not migrated.credentials.has(legacy_id)
migrated.put_secret(created.server_id, "TOKEN", "new-value")
assert (
registry().credentials.resolve(migrated._secret_id(created.server_id, "TOKEN"))
== "new-value"
)
def test_ambiguous_legacy_credentials_are_not_assigned_to_two_variables() -> None:
service = registry()
created = service.create(request(secret_environment_keys=["TOKEN", "token"]))
suffix = hashlib.sha256(b"environment\0token").hexdigest()[:20]
legacy_id = f"mcp.{created.server_id}.{suffix}"
service.credentials.put(legacy_id, "cannot-reconstruct-originals")
service._records[created.server_id]["secret_environment_version"] = 1
service._write()
migrated = registry()
current = migrated.get(created.server_id)
assert current.secret_environment == {"TOKEN": False, "token": False}
assert current.enabled is False
assert current.last_test_succeeded is None
assert migrated.credentials.has(
legacy_id
) # Keep the original ciphertext recoverable.
migrated.put_secret(created.server_id, "TOKEN", "upper")
migrated.put_secret(created.server_id, "token", "lower")
assert registry().get(created.server_id).secret_environment == {
"TOKEN": True,
"token": True,
}
migrated.delete(created.server_id)
assert not migrated.credentials.has(legacy_id)
def test_credential_id_migration_keeps_new_values_and_is_atomic(monkeypatch) -> None:
credentials = EncryptedCredentialStore()
credentials.put("mcp.old", "old-value")
credentials.put("mcp.new", "new-value")
original_write = credentials._write_tokens
def fail_write(_tokens):
raise CredentialStoreError("synthetic failure")
monkeypatch.setattr(credentials, "_write_tokens", fail_write)
with pytest.raises(CredentialStoreError):
credentials.move_many({"mcp.old": "mcp.new"})
assert credentials.resolve("mcp.old") == "old-value"
assert credentials.resolve("mcp.new") == "new-value"
monkeypatch.setattr(credentials, "_write_tokens", original_write)
credentials.move_many({"mcp.old": "mcp.new"})
assert credentials.resolve("mcp.old") is None
assert credentials.resolve("mcp.new") == "new-value"
def test_ambiguous_legacy_secret_is_not_resurrected_after_removing_a_key() -> None:
service = registry()
created = service.create(request(secret_environment_keys=["TOKEN", "token"]))
legacy_id = service._legacy_environment_secret_id(created.server_id, "TOKEN")
service.credentials.put(legacy_id, "ambiguous-old-value")
service._records[created.server_id]["secret_environment_version"] = 1
service._write()
migrated = registry()
migrated.update(
created.server_id,
McpServerUpdateRequest(
**request(secret_environment_keys=["token"]).model_dump(),
version=created.version,
),
)
assert registry().get(created.server_id).secret_environment == {"token": False}
def test_production_rejects_process_launch_even_after_approval() -> None:
service = registry(launch=False)
created = service.create(request(secret_environment_keys=[]))
service.trust(created.server_id, created.command_digest)
with pytest.raises(McpRegistryError) as error:
service.enable(created.server_id)
assert error.value.code == "MCP_SANDBOX_REQUIRED"
@pytest.mark.parametrize("startup,tool", [(120, 300), (1.5, 2.5)])
def test_server_timeouts_survive_bridge_adaptation_and_reload(startup, tool) -> None:
service = registry()
created = service.create(
request(
secret_environment_keys=[],
startup_timeout_seconds=startup,
tool_timeout_seconds=tool,
)
)
service.trust(created.server_id, created.command_digest)
try:
tested = service.test(created.server_id)
assert tested.last_test_succeeded is True
assert tested.startup_timeout_seconds == startup
assert tested.tool_timeout_seconds == tool
restored = registry().get(created.server_id)
assert restored.startup_timeout_seconds == startup
assert restored.tool_timeout_seconds == tool
finally:
service.shutdown()
def test_enable_requires_successful_test_and_update_checks_version() -> None:
service = registry()
created = service.create(request(secret_environment_keys=[]))
service.trust(created.server_id, created.command_digest)
with pytest.raises(McpRegistryError) as error:
service.enable(created.server_id)
assert error.value.code == "MCP_CONNECTION_TEST_REQUIRED"
with pytest.raises(McpRegistryError) as error:
service.update(
created.server_id,
McpServerUpdateRequest(
**request(secret_environment_keys=[]).model_dump(), version=99
),
)
assert error.value.code == "MCP_SERVER_VERSION_CONFLICT"
def test_http_transport_rejects_invalid_cross_transport_fields() -> None:
service = registry()
with pytest.raises(McpRegistryError) as error:
service.create(
request(
transport="streamable_http",
url="https://example.invalid/mcp",
secret_environment_keys=[],
)
)
assert error.value.code == "MCP_CONFIG_INVALID"
def test_registry_rejects_corrupt_persisted_json(tmp_path) -> None:
path = tmp_path / "mcp"
path.mkdir()
(path / "servers.json").write_text("{broken", encoding="utf-8")
with pytest.raises(McpRegistryError) as error:
McpServerRegistry(
ToolRegistry(),
EncryptedCredentialStore(),
tmp_path,
allow_process_launch=True,
)
assert error.value.code == "MCP_REGISTRY_INVALID"
def test_registry_rejects_structurally_invalid_record(tmp_path) -> None:
path = tmp_path / "mcp"
path.mkdir()
(path / "servers.json").write_text(
json.dumps({"server-1": {"name": "Broken", "transport": "stdio"}}),
encoding="utf-8",
)
with pytest.raises(McpRegistryError) as error:
McpServerRegistry(
ToolRegistry(),
EncryptedCredentialStore(),
tmp_path,
allow_process_launch=True,
)
assert error.value.code == "MCP_REGISTRY_INVALID"
def test_registry_rejects_create_before_exceeding_persisted_limit(
monkeypatch,
) -> None:
service = registry()
service.create(request(name="Only server"))
monkeypatch.setattr("app.extensions.mcp_registry._MAX_MCP_SERVERS", 1)
with pytest.raises(McpRegistryError) as error:
service.create(request(name="One too many"))
assert error.value.code == "MCP_SERVER_LIMIT_REACHED"
assert len(service.list()) == 1
service.shutdown()
def test_stdio_command_is_not_parsed_as_a_shell_string() -> None:
service = registry()
created = service.create(
request(
command=f'"{sys.executable}" "{SERVER}"',
args=[],
secret_environment_keys=[],
)
)
service.trust(created.server_id, created.command_digest)
with pytest.raises(McpRegistryError) as error:
service.test(created.server_id)
assert error.value.code == "PLUGIN_HOST_START_FAILED"
assert service.get(created.server_id).last_test_succeeded is False
service.shutdown()
def test_enabled_server_is_restored_from_persisted_registry() -> None:
first = registry()
created = first.create(request(secret_environment_keys=[]))
first.trust(created.server_id, created.command_digest)
first.test(created.server_id)
first.enable(created.server_id)
first.shutdown()
restored = registry()
restored.restore_enabled()
current = restored.get(created.server_id)
assert current.enabled is True
assert current.status == "ready"
assert any(
item.name.startswith(f"mcp.{created.server_id}.")
for item in restored.tools.definitions()
)
restored.shutdown()
def test_lifecycle_operations_are_serialized_and_tool_names_are_isolated() -> None:
service = registry()
servers = [
service.create(request(name=f"Echo {index}", secret_environment_keys=[]))
for index in range(2)
]
for server in servers:
service.trust(server.server_id, server.command_digest)
service.test(server.server_id)
with ThreadPoolExecutor(max_workers=4) as pool:
enabled = list(
pool.map(lambda item: service.enable(item.server_id), servers * 2)
)
assert all(item.enabled for item in enabled)
names = [
item.name for item in service.tools.definitions() if item.source == "mcp_server"
]
assert len(names) == len(set(names))
assert all(
any(name.startswith(f"mcp.{item.server_id}.") for name in names)
for item in servers
)
with ThreadPoolExecutor(max_workers=4) as pool:
list(pool.map(lambda item: service.disable(item.server_id), servers * 2))
assert not any(item.source == "mcp_server" for item in service.tools.definitions())
service.shutdown()
def _http_result(request_id: int, result: dict) -> httpx.Response:
return httpx.Response(
200,
headers={"content-type": "application/json"},
json={"jsonrpc": "2.0", "id": request_id, "result": result},
)
def test_streamable_http_supports_session_headers_secrets_and_tool_summary(
monkeypatch,
) -> None:
requests: list[httpx.Request] = []
request_timeouts: dict[str, float] = {}
def handler(request_value: httpx.Request) -> httpx.Response:
requests.append(request_value)
if request_value.method == "GET":
return httpx.Response(405)
if request_value.method == "DELETE":
return httpx.Response(405)
payload = json.loads(request_value.content)
timeout = request_value.extensions.get("timeout", {}).get("read")
if isinstance(timeout, (int, float)):
request_timeouts[payload.get("method", "notification")] = float(timeout)
if payload.get("method") == "initialize":
response = _http_result(
payload["id"],
{
"protocolVersion": "2025-11-25",
"capabilities": {"tools": {}},
"serverInfo": {"name": "HTTP Fixture", "version": "1"},
},
)
response.headers["MCP-Session-Id"] = "session-test"
return response
if payload.get("method") == "tools/list":
return _http_result(
payload["id"],
{
"tools": [
{
"name": "echo",
"description": "Echo over HTTP",
"inputSchema": {"type": "object", "properties": {}},
}
]
},
)
if payload.get("method") == "tools/call":
return _http_result(
payload["id"], {"structuredContent": {"transport": "http"}}
)
return httpx.Response(202)
real_client = httpx.Client
monkeypatch.setattr(
"app.extensions.mcp.httpx.Client",
lambda **kwargs: real_client(transport=httpx.MockTransport(handler), **kwargs),
)
service = registry()
created = service.create(
McpServerCreateRequest(
name="Remote MCP",
transport="streamable_http",
url="https://mcp.example.test/mcp",
headers={"X-Client": "NotesAgent"},
secret_header_keys=["Authorization"],
)
)
service.put_secret(
created.server_id, "Authorization", "Bearer hidden", kind="header"
)
service.trust(created.server_id, created.command_digest)
tested = service.test(created.server_id)
assert tested.last_test_succeeded is True
assert tested.secret_headers == {"Authorization": True}
assert "Bearer hidden" not in tested.model_dump_json()
assert service.list_tools(created.server_id)[0].remote_name == "echo"
assert any(
request.headers.get("mcp-session-id") == "session-test" for request in requests
)
assert any(
request.headers.get("mcp-protocol-version") == "2025-11-25"
for request in requests
)
assert all(
request.headers.get("authorization") == "Bearer hidden" for request in requests
)
assert request_timeouts["initialize"] == 15
assert request_timeouts["notifications/initialized"] == 15
assert request_timeouts["tools/list"] == 15
enabled = service.enable(created.server_id)
tool_name = service.list_tools(created.server_id)[0].name
result = asyncio.run(
service.tools.execute(
ToolCall(tool_call_id="call-1", name=tool_name, arguments={}),
ToolExecutionContext(run_id="run-1"),
)
)
assert enabled.enabled is True
assert result.success is True
assert result.output == {"transport": "http"}
assert request_timeouts["tools/call"] == 30
service.disable(created.server_id)
service.shutdown()
class _LegacyEventStream(httpx.SyncByteStream):
def __init__(self) -> None:
self.closed = threading.Event()
def __iter__(self):
yield b"event: endpoint\ndata: /messages\n\n"
time.sleep(0.1)
initialize = {
"jsonrpc": "2.0",
"id": 1,
"result": {
"protocolVersion": "2024-11-05",
"capabilities": {"tools": {}},
"serverInfo": {"name": "Legacy Fixture"},
},
}
yield f"data: {json.dumps(initialize)}\n\n".encode()
time.sleep(0.1)
tools = {
"jsonrpc": "2.0",
"id": 2,
"result": {"tools": []},
}
yield f"data: {json.dumps(tools)}\n\n".encode()
self.closed.wait()
def close(self) -> None:
self.closed.set()
def test_legacy_sse_uses_same_origin_endpoint(monkeypatch) -> None:
posted_urls: list[str] = []
event_stream = _LegacyEventStream()
def handler(request_value: httpx.Request) -> httpx.Response:
if request_value.method == "GET":
return httpx.Response(
200,
headers={"content-type": "text/event-stream"},
stream=event_stream,
)
posted_urls.append(str(request_value.url))
return httpx.Response(202)
real_client = httpx.Client
monkeypatch.setattr(
"app.extensions.mcp.httpx.Client",
lambda **kwargs: real_client(transport=httpx.MockTransport(handler), **kwargs),
)
service = registry()
created = service.create(
McpServerCreateRequest(
name="Legacy MCP",
transport="sse",
url="https://legacy.example.test/sse",
)
)
service.trust(created.server_id, created.command_digest)
tested = service.test(created.server_id)
assert tested.last_test_succeeded is True
assert posted_urls and all(
url == "https://legacy.example.test/messages" for url in posted_urls
)
service.shutdown()
event_stream.close()
class _EndingLegacyEventStream(httpx.SyncByteStream):
def __iter__(self):
yield b"event: endpoint\ndata: /messages\n\n"
def test_legacy_sse_eof_marks_client_unavailable(monkeypatch) -> None:
def handler(_request_value: httpx.Request) -> httpx.Response:
return httpx.Response(
200,
headers={"content-type": "text/event-stream"},
stream=_EndingLegacyEventStream(),
)
real_client = httpx.Client
monkeypatch.setattr(
"app.extensions.mcp.httpx.Client",
lambda **kwargs: real_client(transport=httpx.MockTransport(handler), **kwargs),
)
broken = threading.Event()
client = McpLegacySseClient(
"https://legacy.example.test/sse",
headers={},
startup_timeout_seconds=1,
on_seen=lambda: None,
on_broken=lambda _message: broken.set(),
on_tools_changed=lambda: None,
)
client.start()
assert broken.wait(timeout=1)
client.stop()
class _CrossOriginLegacyEventStream(httpx.SyncByteStream):
def __iter__(self):
yield b"event: endpoint\ndata: https://attacker.example/messages\n\n"
def test_legacy_sse_rejects_cross_origin_message_endpoint(monkeypatch) -> None:
def handler(request_value: httpx.Request) -> httpx.Response:
assert request_value.method == "GET"
return httpx.Response(
200,
headers={"content-type": "text/event-stream"},
stream=_CrossOriginLegacyEventStream(),
)
real_client = httpx.Client
monkeypatch.setattr(
"app.extensions.mcp.httpx.Client",
lambda **kwargs: real_client(transport=httpx.MockTransport(handler), **kwargs),
)
service = registry()
created = service.create(
McpServerCreateRequest(
name="Unsafe legacy MCP",
transport="sse",
url="https://legacy.example.test/sse",
)
)
service.trust(created.server_id, created.command_digest)
with pytest.raises(McpRegistryError) as error:
service.test(created.server_id)
assert error.value.code == "MCP_HTTP_RESPONSE_INVALID"
service.shutdown()
+83
View File
@@ -0,0 +1,83 @@
import json
from contextlib import closing
import httpx
import pytest
from app.extensions import mcp
class ChunkStream(httpx.SyncByteStream):
def __init__(self, chunks):
self.chunks = chunks
self.bytes_read = 0
def __iter__(self):
for chunk in self.chunks:
self.bytes_read += len(chunk)
yield chunk
def test_sse_rejects_unterminated_line_before_reading_entire_stream(monkeypatch):
monkeypatch.setattr(mcp, "MAX_MCP_MESSAGE_BYTES", 1024)
stream = ChunkStream([b"x" * 256] * 256)
with (
closing(httpx.Response(200, stream=stream)) as response,
pytest.raises(mcp.McpBridgeError, match="too large"),
):
list(mcp._iter_sse(response))
assert stream.bytes_read == 1280
def test_sse_limits_combined_event_before_partial_line_is_complete(monkeypatch):
monkeypatch.setattr(mcp, "MAX_MCP_MESSAGE_BYTES", 32)
stream = ChunkStream(
[b"data: 123456789\n", b"data: 123456789\n", b"x", b"not-read"]
)
with (
closing(httpx.Response(200, stream=stream)) as response,
pytest.raises(mcp.McpBridgeError, match="too large"),
):
list(mcp._iter_sse(response))
assert stream.bytes_read == 33
@pytest.mark.parametrize("separator", [b"\n", b"\r", b"\r\n"])
@pytest.mark.parametrize("chunk_size", [1, 2, 7, 1024])
def test_sse_preserves_utf8_and_line_endings_across_chunks(separator, chunk_size):
payload = json.dumps(
{"jsonrpc": "2.0", "id": 1, "result": {"text": "中文"}}, ensure_ascii=False
)
wire = b"\xef\xbb\xbf" + separator.join(
[
b": heartbeat",
b"event: message",
b"id: replay-1",
("data: " + payload).encode(),
b"",
b"",
]
)
stream = ChunkStream(
[wire[index : index + chunk_size] for index in range(0, len(wire), chunk_size)]
)
with closing(httpx.Response(200, stream=stream)) as response:
assert list(mcp._iter_sse(response)) == [("message", "replay-1", payload)]
def test_sse_event_limit_resets_between_events(monkeypatch):
monkeypatch.setattr(mcp, "MAX_MCP_MESSAGE_BYTES", 16)
with closing(
httpx.Response(200, stream=ChunkStream([b"data: one\n\ndata: two\r\r"]))
) as response:
assert list(mcp._iter_sse(response)) == [
("message", None, "one"),
("message", None, "two"),
]
def test_sse_preserves_multiline_data_and_final_unterminated_line():
with closing(
httpx.Response(200, stream=ChunkStream([b"data: first\ndata: last"]))
) as response:
assert list(mcp._iter_sse(response)) == [("message", None, "first\nlast")]
+663
View File
@@ -0,0 +1,663 @@
"""Offline model-routing contracts, HTTP validation, media lifetimes and persistence.
All HTTP uses MockTransport (or the in-process API). Credentials, models and
attachments are fakes, and conftest redirects all storage to temporary paths.
"""
from __future__ import annotations
import asyncio
import hashlib
import json
from email import policy
from email.parser import BytesParser
from types import SimpleNamespace
import httpx
import pytest
from fastapi.testclient import TestClient
from app.contracts import ModelBinding, ModelRoutingConfig, ProviderConfig, ProviderType
from app.errors import ApiError
from app.providers import MockProvider
from app.providers.credentials import CredentialStoreError
from app.providers.registry import ProviderRegistry
from app.providers.routing import ModelRoutingService, PendingSpeechBackend
from app.retrieval.embedding import HashEmbeddingProvider
def run(awaitable):
return asyncio.run(awaitable)
def response(data, status=200):
# Raw JSON intentionally permits NaN/Infinity to exercise hostile API output.
return httpx.Response(status, content=json.dumps(data).encode(), headers={"content-type": "application/json"})
class FakeCredentials:
def __init__(self):
self.value = "unit-test-placeholder"
self.error = None
self.calls = []
def resolve(self, credential_id):
self.calls.append(credential_id)
if self.error:
raise self.error
return self.value if credential_id else None
class FakeEmbedding:
model_id = "fake-local-model"
dim = 3
def __init__(self):
self.calls = []
self.error = None
async def embed_documents(self, texts):
self.calls.append(list(texts))
if self.error:
raise self.error
return [[0.6, 0.8, 0.0] for _ in texts]
class FakeSpeech:
available = True
def __init__(self):
self.calls = []
self.text = "local transcript"
self.score = 0.25
self.error = None
async def transcribe(self, source, language):
self.calls.append(("transcribe", source, language))
if self.error:
raise self.error
return self.text
async def match(self, source, reference):
self.calls.append(("match", source, reference))
if self.error:
raise self.error
return self.score
@pytest.fixture(autouse=True)
def no_real_http(monkeypatch):
async def reject_async(*args, **kwargs):
pytest.fail("Real HTTP transport is forbidden in model-routing tests")
def reject_sync(*args, **kwargs):
pytest.fail("Real HTTP transport is forbidden in model-routing tests")
monkeypatch.setattr(httpx.AsyncHTTPTransport, "handle_async_request", reject_async)
monkeypatch.setattr(httpx.HTTPTransport, "handle_request", reject_sync)
@pytest.fixture
def rig():
requests = []
def unexpected(request):
pytest.fail(f"Unexpected model HTTP request: {request.url}")
state = SimpleNamespace(handler=unexpected)
async def dispatch(request):
requests.append(request)
result = state.handler(request)
return await result if hasattr(result, "__await__") else result
providers = ProviderRegistry()
config = ProviderConfig(
provider_id="test-provider", provider_type=ProviderType.openai_compatible,
name="Fake provider", base_url="https://models.invalid/v1/", credential_id="test-credential",
)
providers.register(config, MockProvider())
credentials, embedding, speech = FakeCredentials(), FakeEmbedding(), FakeSpeech()
service = ModelRoutingService(
providers, credentials, local_embedding=embedding, local_speech=speech,
transport=httpx.MockTransport(dispatch),
)
return SimpleNamespace(
service=service, providers=providers, credentials=credentials,
embedding=embedding, speech=speech, requests=requests, http=state,
)
def bind(rig, capability="embedding", **overrides):
endpoints = {
"embedding": "/embeddings", "transcription": "/audio/transcriptions",
"speaker_matching": "/audio/speaker-matches",
}
binding = ModelBinding(**{
"provider_id": "test-provider", "model": "test-model",
"endpoint": endpoints[capability], **overrides,
})
current = rig.service.configuration()
return rig.service.update(current.model_copy(update={capability: binding}))
def assert_local(rig, result, texts, reason):
assert result.source == "local"
assert result.model_id == rig.embedding.model_id
assert result.dimensions == 3
assert result.vectors == [[0.6, 0.8, 0.0] for _ in texts]
assert result.fallback_reason == reason
assert rig.embedding.calls == [texts]
def test_embedding_observation_keeps_request_binding_when_config_changes(rig):
from app.retrieval.provenance import capture_embedding
initial = bind(rig, model="original-model")
def handler(request):
assert json.loads(request.content)["model"] == "original-model"
bind(rig, model="next-model")
return response({"data": [{"index": 0, "embedding": [1, 0, 0]}]})
rig.http.handler = handler
with capture_embedding() as observation:
result = run(rig.service.embed(["query"]))
assert result.source == "api"
assert observation["route_version"] == initial.config.version
assert observation["requested_route"]["model"] == "original-model"
assert observation["requested_route"]["provider_id"] == "test-provider"
assert rig.service.configuration().embedding.model == "next-model"
assert rig.credentials.value not in json.dumps(observation)
assert "credential_id" not in json.dumps(observation)
@pytest.fixture
def audio(tmp_path):
source, reference = tmp_path / "audio.wav", tmp_path / "reference.wav"
source.write_bytes(b"fake-audio-content")
reference.write_bytes(b"fake-reference-content")
return source, reference
def media_call(rig, capability, audio):
if capability == "transcription":
return rig.service.transcribe(audio[0], "zh")
return rig.service.match_speakers(*audio)
def track_media_handles(rig, monkeypatch):
handles = []
original = rig.service._media_file
def tracked(path):
handle = original(path)
handles.append(handle)
return handle
monkeypatch.setattr(rig.service, "_media_file", tracked)
return handles
def test_absent_binding_uses_hash_without_network(rig):
rig.service.local_embedding = HashEmbeddingProvider()
texts = ["hello retrieval", "向量检索"]
result = run(rig.service.embed(texts))
assert result.source == "local"
assert result.model_id == "hash-v1"
assert result.dimensions == 128
assert result.vectors == run(HashEmbeddingProvider().embed_documents(texts))
assert result.fallback_reason is None
assert rig.requests == rig.credentials.calls == []
statuses = {item.capability: item.status for item in rig.service.describe().local_backends}
assert statuses == {"embedding": "placeholder", "transcription": "ready", "speaker_matching": "ready"}
def test_empty_embedding_input_does_not_call_remote(rig):
bind(rig)
result = run(rig.service.embed([]))
assert result.vectors == [] and result.source == "local"
assert rig.requests == []
def test_remote_embedding_restores_batch_order_normalizes_and_sends_auth(rig):
bind(rig, dimensions=2)
texts = [str(index) for index in range(35)]
def handler(request):
assert request.method == "POST"
assert str(request.url) == "https://models.invalid/v1/embeddings"
assert request.headers["authorization"] == "Bearer unit-test-placeholder"
payload = json.loads(request.content)
assert payload["model"] == "test-model"
assert payload["dimensions"] == 2
assert payload["encoding_format"] == "float"
return response({"data": [
{"index": index, "embedding": [float(int(text) + 1), 1.0]}
for index, text in reversed(list(enumerate(payload["input"])))
]})
rig.http.handler = handler
result = run(rig.service.embed(texts))
assert result.source == "api" and result.fallback_reason is None
assert result.dimensions == 2 and len(result.vectors) == 35
for index, vector in enumerate(result.vectors):
assert sum(value * value for value in vector) == pytest.approx(1.0)
assert vector[0] / vector[1] == pytest.approx(index + 1)
assert [json.loads(req.content)["input"] for req in rig.requests] == [texts[:32], texts[32:]]
assert rig.embedding.calls == []
def test_space_id_is_stable_and_includes_full_url_model_and_inferred_dimensions(rig):
dimensions = 2
def handler(request):
assert "dimensions" not in json.loads(request.content)
return response({"data": [{"index": 0, "embedding": [1.0] * dimensions}]})
rig.http.handler = handler
bind(rig, model=" trimmed-model ")
def check(url, model, dimension):
result = run(rig.service.embed(["hello"]))
digest = hashlib.sha256(json.dumps([url, model, dimension], separators=(",", ":")).encode()).hexdigest()
assert result.model_id == "api-" + digest
assert result.source == "api"
return result.model_id
first = check("https://models.invalid/v1/embeddings", "trimmed-model", 2)
assert first == check("https://models.invalid/v1/embeddings", "trimmed-model", 2)
config = rig.providers.get_any("test-provider").config.model_copy(update={"base_url": "https://models.invalid/v1"})
rig.providers.replace(config, MockProvider())
assert first == check("https://models.invalid/v1/embeddings", "trimmed-model", 2)
bind(rig, model="trimmed-model", endpoint="/custom/embeddings")
endpoint_id = check("https://models.invalid/v1/custom/embeddings", "trimmed-model", 2)
bind(rig, model="another-model", endpoint="/custom/embeddings")
model_id = check("https://models.invalid/v1/custom/embeddings", "another-model", 2)
dimensions = 3
dim_id = check("https://models.invalid/v1/custom/embeddings", "another-model", 3)
config = config.model_copy(update={"base_url": "https://other.invalid/v1"})
rig.providers.replace(config, MockProvider())
provider_id = check("https://other.invalid/v1/custom/embeddings", "another-model", 3)
assert len({first, endpoint_id, model_id, dim_id, provider_id}) == 5
@pytest.mark.parametrize("data", [
{"data": []},
{"data": [{"index": 0, "embedding": [1, 0]}]},
{"data": [{"index": 0, "embedding": [1, 0]}, {"index": 0, "embedding": [0, 1]}]},
{"data": [{"index": 0, "embedding": [1, 0]}, {"index": 2, "embedding": [0, 1]}]},
{"data": [{"index": False, "embedding": [1, 0]}, {"index": 1, "embedding": [0, 1]}]},
{"data": [{"index": 0, "embedding": [1, 0]}, {"index": 1, "embedding": [0, 1, 0]}]},
{"data": [{"index": 0, "embedding": [1, 0]}, {"index": 1, "embedding": [float("nan"), 1]}]},
{"data": [{"index": 0, "embedding": [1, 0]}, {"index": 1, "embedding": [float("inf"), 1]}]},
{"data": [{"index": 0, "embedding": [1, 0]}, {"index": 1, "embedding": [True, 1]}]},
{"data": [{"index": 0, "embedding": [1, 0]}, {"index": 1, "embedding": [0, 0]}]},
{"data": [{"index": 0, "embedding": [1, 0]}, {"index": 1, "embedding": []}]},
{"data": [{"index": 0, "embedding": [1, 0]}, {"index": 1, "embedding": ["1", 0]}]},
{"data": [None, None]},
{"error": {"message": "in-band failure"}, "data": []},
[],
], ids=["empty", "count", "duplicate-index", "out-of-range-index", "bool-index", "dimensions", "nan", "infinity", "bool", "zero", "empty-vector", "string", "invalid-items", "in-band-error", "non-object"])
def test_invalid_remote_embeddings_fall_back_as_a_whole(rig, data):
bind(rig)
rig.http.handler = lambda request: response(data)
texts = ["first", "second"]
assert_local(rig, run(rig.service.embed(texts)), texts, "PROVIDER_INVALID_RESPONSE")
def test_explicit_embedding_dimension_mismatch_falls_back(rig):
bind(rig, dimensions=3)
rig.http.handler = lambda request: response({"data": [{"index": 0, "embedding": [1, 0]}]})
assert_local(rig, run(rig.service.embed(["text"])), ["text"], "PROVIDER_INVALID_RESPONSE")
def test_later_batch_dimension_mismatch_discards_earlier_remote_vectors(rig):
bind(rig)
def handler(request):
batch = json.loads(request.content)["input"]
dimension = 2 if len(rig.requests) == 1 else 3
return response({"data": [{"index": i, "embedding": [1] * dimension} for i in range(len(batch))]})
rig.http.handler = handler
texts = [str(i) for i in range(33)]
assert_local(rig, run(rig.service.embed(texts)), texts, "PROVIDER_INVALID_RESPONSE")
assert len(rig.requests) == 2
@pytest.mark.parametrize("failure, reason", [
(401, "PROVIDER_AUTH_FAILED"), (403, "PROVIDER_AUTH_FAILED"),
(404, "MODEL_NOT_FOUND"), (429, "PROVIDER_RATE_LIMITED"), (500, "PROVIDER_UNAVAILABLE"),
("timeout", "PROVIDER_TIMEOUT"), ("connect", "PROVIDER_UNAVAILABLE"),
("json", "PROVIDER_INVALID_RESPONSE"),
])
def test_embedding_http_failures_use_injected_local(rig, failure, reason):
bind(rig)
def handler(request):
if failure == "timeout":
raise httpx.ReadTimeout("simulated timeout", request=request)
if failure == "connect":
raise httpx.ConnectError("simulated connection failure", request=request)
if failure == "json":
return httpx.Response(200, content=b"not JSON")
return response({"error": "failed"}, failure)
rig.http.handler = handler
assert_local(rig, run(rig.service.embed(["text"])), ["text"], reason)
@pytest.mark.parametrize("failure, reason", [
("missing-key", "PROVIDER_CREDENTIAL_MISSING"),
("unreadable-key", "PROVIDER_CREDENTIAL_UNAVAILABLE"),
("disabled-provider", "PROVIDER_UNAVAILABLE"),
])
def test_unavailable_remote_configuration_falls_back_without_http(rig, failure, reason):
bind(rig)
if failure == "missing-key":
rig.credentials.value = None
elif failure == "unreadable-key":
rig.credentials.error = CredentialStoreError("fake unavailable store")
else:
config = rig.providers.get_any("test-provider").config.model_copy(update={"enabled": False})
rig.providers.replace(config, MockProvider())
assert_local(rig, run(rig.service.embed(["text"])), ["text"], reason)
assert rig.requests == []
@pytest.mark.parametrize("capability", ["transcription", "speaker_matching"])
def test_media_success_sends_expected_multipart_and_closes_files(rig, audio, monkeypatch, capability):
bind(rig, capability)
handles = track_media_handles(rig, monkeypatch)
def handler(request):
assert request.headers["authorization"] == "Bearer unit-test-placeholder"
assert str(request.url).endswith("/audio/transcriptions" if capability == "transcription" else "/audio/speaker-matches")
message = BytesParser(policy=policy.default).parsebytes(
b"Content-Type: " + request.headers["content-type"].encode() + b"\r\nMIME-Version: 1.0\r\n\r\n" + request.content,
)
parts = {part.get_param("name", header="content-disposition"): part for part in message.iter_parts()}
assert parts["model"].get_payload(decode=True) == b"test-model"
assert parts["file"].get_filename() == audio[0].name
assert parts["file"].get_payload(decode=True) == audio[0].read_bytes()
if capability == "transcription":
assert set(parts) == {"model", "language", "file"}
assert parts["language"].get_payload(decode=True) == b"zh"
return response({"text": "remote transcript"})
assert set(parts) == {"model", "file", "reference_file"}
assert parts["reference_file"].get_filename() == audio[1].name
assert parts["reference_file"].get_payload(decode=True) == audio[1].read_bytes()
return response({"score": 0.875})
rig.http.handler = handler
result = run(media_call(rig, capability, audio))
assert result.source == "api" and result.fallback_reason is None
assert result.text == "remote transcript" if capability == "transcription" else result.score == 0.875
assert len(handles) == (1 if capability == "transcription" else 2)
assert all(handle.closed for handle in handles)
assert rig.speech.calls == []
@pytest.mark.parametrize("capability, data", [
("transcription", {}), ("transcription", {"text": " "}), ("transcription", {"text": False}),
("transcription", {"error": "in-band", "text": "must not use"}),
("speaker_matching", {}), ("speaker_matching", {"score": -0.1}),
("speaker_matching", {"score": 1.1}), ("speaker_matching", {"score": True}),
("speaker_matching", {"score": float("nan")}), ("speaker_matching", {"score": "0.5"}),
("speaker_matching", {"error": "in-band", "score": 0.9}),
])
def test_invalid_remote_media_falls_back_to_injected_local(rig, audio, monkeypatch, capability, data):
bind(rig, capability)
handles = track_media_handles(rig, monkeypatch)
rig.http.handler = lambda request: response(data)
result = run(media_call(rig, capability, audio))
assert result.source == "local" and result.fallback_reason == "PROVIDER_INVALID_RESPONSE"
assert result.text == "local transcript" if capability == "transcription" else result.score == 0.25
assert rig.speech.calls == [
("transcribe", audio[0], "zh") if capability == "transcription" else ("match", *audio)
]
assert handles and all(handle.closed for handle in handles)
@pytest.mark.parametrize("capability", ["transcription", "speaker_matching"])
@pytest.mark.parametrize("configured", [False, True])
def test_pending_local_backend_has_explicit_503_and_fallback_details(rig, audio, capability, configured):
rig.service.local_speech = PendingSpeechBackend()
if configured:
bind(rig, capability)
rig.http.handler = lambda request: response({"error": "unauthorized"}, 401)
with pytest.raises(ApiError) as caught:
run(media_call(rig, capability, audio))
assert caught.value.status_code == 503
assert caught.value.code == "LOCAL_MODEL_NOT_INSTALLED"
assert caught.value.details == {"fallback_reason": "PROVIDER_AUTH_FAILED" if configured else None}
statuses = {item.capability: item.status for item in rig.service.describe().local_backends}
assert statuses["transcription"] == statuses["speaker_matching"] == "not_installed"
assert len(rig.requests) == int(configured)
@pytest.mark.parametrize("capability", ["transcription", "speaker_matching"])
def test_invalid_local_speech_returns_explicit_503(rig, audio, capability):
rig.speech.text = ""
rig.speech.score = True
with pytest.raises(ApiError) as caught:
run(media_call(rig, capability, audio))
assert (caught.value.status_code, caught.value.code) == (503, "LOCAL_MODEL_INVALID_RESPONSE")
assert caught.value.details == {"fallback_reason": None}
@pytest.mark.parametrize("capability", ["embedding", "transcription", "speaker_matching"])
@pytest.mark.parametrize("stage", ["remote", "local"])
def test_cancellation_propagates_and_upload_handles_close(rig, audio, monkeypatch, capability, stage):
bind(rig, capability)
handles = track_media_handles(rig, monkeypatch)
async def cancelled(request):
raise asyncio.CancelledError()
if stage == "remote":
rig.http.handler = cancelled
else:
rig.http.handler = lambda request: response({"error": "fallback"}, 500)
rig.embedding.error = rig.speech.error = asyncio.CancelledError()
operation = rig.service.embed(["text"]) if capability == "embedding" else media_call(rig, capability, audio)
with pytest.raises(asyncio.CancelledError):
run(operation)
assert len(handles) == {"embedding": 0, "transcription": 1, "speaker_matching": 2}[capability]
assert all(handle.closed for handle in handles)
if stage == "remote":
assert rig.embedding.calls == rig.speech.calls == []
def test_missing_reference_closes_already_open_source(rig, audio, monkeypatch):
bind(rig, "speaker_matching")
handles = track_media_handles(rig, monkeypatch)
audio[1].unlink()
with pytest.raises(ApiError) as caught:
run(rig.service.match_speakers(*audio))
assert caught.value.status_code == 404
assert len(handles) == 1 and handles[0].closed
assert rig.requests == []
def test_config_optimistic_conflict_preserves_saved_bindings(rig):
assert rig.service.configuration().version == 0
saved = bind(rig).config
assert saved.version == 1
with pytest.raises(ApiError) as caught:
rig.service.update(ModelRoutingConfig(version=0))
assert (caught.value.status_code, caught.value.code) == (409, "MODEL_ROUTING_VERSION_CONFLICT")
assert rig.service.configuration() == saved
assert rig.service.uses_provider("test-provider")
assert not rig.service.uses_provider("not-a-provider")
cleared = rig.service.update(ModelRoutingConfig(version=1)).config
assert cleared.version == 2 and cleared.embedding is None
assert not rig.service.uses_provider("test-provider")
@pytest.mark.parametrize("capability", ["embedding", "transcription", "speaker_matching"])
@pytest.mark.parametrize("provider_id, code", [
("missing", "PROVIDER_NOT_FOUND"), ("unsupported", "MODEL_ROUTING_PROTOCOL_UNSUPPORTED"),
])
def test_config_references_require_existing_supported_providers(rig, capability, provider_id, code):
rig.providers.register(
ProviderConfig(provider_id="unsupported", provider_type=ProviderType.ollama, name="unsupported"), MockProvider(),
)
with pytest.raises(ApiError) as caught:
bind(rig, capability, provider_id=provider_id)
assert (caught.value.status_code, caught.value.code) == (422, code)
assert rig.service.configuration() == ModelRoutingConfig()
assert rig.requests == []
@pytest.fixture
def api(monkeypatch, no_real_http, _isolate_data_dir):
# Import the production container only after temporary storage is configured.
from app import container as container_module, routes
from app.main import app
containers = []
def restart():
container = container_module.build_container()
container.model_routing.credentials = FakeCredentials()
def unexpected(request):
pytest.fail(f"Unexpected API-side provider HTTP: {request.url}")
container.model_routing.transport = httpx.MockTransport(unexpected)
monkeypatch.setattr(container_module, "container", container)
monkeypatch.setattr(routes, "container", container)
containers.append(container)
return container
container = restart()
client = TestClient(app)
yield SimpleNamespace(client=client, container=container, restart=restart)
client.close()
for container in containers:
container.plugins.shutdown()
container.mcp_servers.shutdown()
def create_api_provider(api):
result = api.client.post("/api/providers", json={
"provider_type": "openai_compatible", "name": "Persisted fake",
"base_url": "https://persist.invalid/v1", "default_model": "fake-model",
})
assert result.status_code == 200, result.text
return result.json()
def test_api_config_conflict_reference_delete_and_restart_persistence(api):
provider = create_api_provider(api)
provider_id = provider["provider_id"]
assert api.client.get("/api/model-routing").json()["config"]["version"] == 0
config = {"version": 0, "embedding": {"provider_id": provider_id, "model": "embed-model", "endpoint": "/embeddings"}}
saved = api.client.put("/api/model-routing", json=config)
assert saved.status_code == 200
assert saved.json()["config"]["version"] == 1
conflict = api.client.put("/api/model-routing", json=config)
assert conflict.status_code == 409
assert conflict.json()["error"]["code"] == "MODEL_ROUTING_VERSION_CONFLICT"
blocked = api.client.delete(f"/api/providers/{provider_id}")
assert blocked.status_code == 409 and blocked.json()["error"]["code"] == "PROVIDER_IN_USE"
restarted = api.restart()
assert restarted.providers.get_any(provider_id).config.model_dump(mode="json") == provider
assert api.client.get("/api/model-routing").json()["config"] == saved.json()["config"]
assert {item["provider_id"] for item in api.client.get("/api/providers").json()["items"]} == {"mock", provider_id}
cleared = api.client.put("/api/model-routing", json={"version": 1})
assert cleared.status_code == 200
assert api.client.delete(f"/api/providers/{provider_id}").status_code == 200
api.restart()
assert api.client.get(f"/api/providers/{provider_id}").status_code == 404
assert api.client.get("/api/model-routing").json()["config"]["version"] == 2
def test_api_provider_type_patch_rebuilds_adapter_and_persists(api):
from app.providers.anthropic_messages import AnthropicMessagesProvider
provider = create_api_provider(api)
provider_id = provider["provider_id"]
changed = api.client.patch(f"/api/providers/{provider_id}", json={
"provider_type": "anthropic_messages", "base_url": "https://anthropic.invalid/v1",
})
assert changed.status_code == 200, changed.text
assert changed.json()["provider_type"] == "anthropic_messages"
assert changed.json()["name"] == provider["name"]
assert isinstance(api.container.providers.get_any(provider_id).adapter, AnthropicMessagesProvider)
restarted = api.restart()
assert isinstance(restarted.providers.get_any(provider_id).adapter, AnthropicMessagesProvider)
assert api.client.get(f"/api/providers/{provider_id}").json() == changed.json()
for invalid_type in (None, "mock", "nonexistent-type"):
rejected = api.client.patch(f"/api/providers/{provider_id}", json={"provider_type": invalid_type})
assert rejected.status_code == 422
assert api.client.get(f"/api/providers/{provider_id}").json() == changed.json()
@pytest.mark.parametrize("endpoint", ["https://elsewhere.invalid/embed", "//elsewhere.invalid/embed", "relative", "/../embed", "/embed?key=test"])
def test_api_config_rejects_non_provider_endpoint_paths(api, endpoint):
provider = create_api_provider(api)
result = api.client.put("/api/model-routing", json={
"embedding": {"provider_id": provider["provider_id"], "model": "embed", "endpoint": endpoint},
})
assert result.status_code == 422
assert api.client.get("/api/model-routing").json()["config"]["version"] == 0
def test_api_embedding_reports_remote_and_fallback_sources(api):
provider = create_api_provider(api)
assert api.client.put("/api/model-routing", json={
"embedding": {"provider_id": provider["provider_id"], "model": "embed", "endpoint": "/embeddings"},
}).status_code == 200
api.container.model_routing.transport = httpx.MockTransport(
lambda request: response({"data": [{"index": 0, "embedding": [3, 4]}]}),
)
result = api.client.post("/api/models/embeddings", json={"texts": ["hello"]})
assert result.status_code == 200
assert result.json()["source"] == "api" and result.json()["vectors"][0] == pytest.approx([0.6, 0.8])
api.container.model_routing.transport = httpx.MockTransport(lambda request: response({"error": "denied"}, 401))
result = api.client.post("/api/models/embeddings", json={"texts": ["hello"]})
assert result.status_code == 200
assert result.json()["source"] == "local" and result.json()["model_id"] == "hash-v1"
assert result.json()["fallback_reason"] == "PROVIDER_AUTH_FAILED"
assert api.client.post("/api/models/embeddings", json={"texts": []}).status_code == 422
def test_api_speech_failure_reports_reason_in_503_and_transcription_job(api):
from app.services.attachment_service import attachment_path
source, reference = attachment_path("audio.wav"), attachment_path("reference.wav")
source.parent.mkdir(parents=True, exist_ok=True)
source.write_bytes(b"test audio")
reference.write_bytes(b"test reference")
provider = create_api_provider(api)
assert api.client.put("/api/model-routing", json={
"transcription": {"provider_id": provider["provider_id"], "model": "asr", "endpoint": "/audio/transcriptions"},
"speaker_matching": {"provider_id": provider["provider_id"], "model": "voice", "endpoint": "/audio/speaker-matches"},
}).status_code == 200
api.container.model_routing.transport = httpx.MockTransport(lambda request: response({"error": "offline"}, 500))
match = api.client.post("/api/media/speaker-matches", json={"attachment_id": source.name, "reference_attachment_id": reference.name})
assert match.status_code == 503
assert match.json()["error"]["code"] == "LOCAL_MODEL_NOT_INSTALLED"
assert match.json()["error"]["details"] == {"fallback_reason": "PROVIDER_UNAVAILABLE"}
transcript = api.client.post("/api/media/transcriptions", json={"attachment_id": source.name, "language": "zh"})
assert transcript.status_code == 202
job = transcript.json()
assert job["status"] == "failed" and job["error_code"] == "LOCAL_MODEL_NOT_INSTALLED"
assert job["fallback_reason"] == "PROVIDER_UNAVAILABLE"
assert api.client.get(f"/api/media/transcriptions/{job['job_id']}").json() == job
@pytest.mark.parametrize("capability", ["embedding", "speaker_matching"])
def test_out_of_float_range_json_number_is_invalid_remote_and_falls_back(rig, audio, capability):
"""JSON integers may be finite but too large to convert to a Python float."""
bind(rig, capability)
data = {"data": [{"index": 0, "embedding": [10 ** 400, 1]}]} if capability == "embedding" else {"score": 10 ** 400}
rig.http.handler = lambda request: response(data)
if capability == "embedding":
assert_local(rig, run(rig.service.embed(["text"])), ["text"], "PROVIDER_INVALID_RESPONSE")
else:
result = run(media_call(rig, capability, audio))
assert result.source == "local" and result.score == rig.speech.score
assert result.fallback_reason == "PROVIDER_INVALID_RESPONSE"
+653
View File
@@ -0,0 +1,653 @@
import asyncio
import json
from pathlib import Path
import pytest
from pydantic import TypeAdapter, ValidationError
from app.agent import ToolRegistry
from app.config import BACKEND_DIR, get_settings
from app.container import build_container
from app.contracts import (
PluginCommandContext,
PluginCommandEffect,
PluginNoEffect,
PluginSettingType,
)
from app.extensions import ExtensionError, PluginRuntime
from app.extensions.contributions import _secret_reference
from app.extensions.runtime import DeclarativePluginHost
from app.providers.credentials import CredentialStoreError
TEXT_TOOLS = BACKEND_DIR / "extensions" / "plugins" / "text-tools"
def run(coroutine):
return asyncio.run(coroutine)
def test_command_list_filter_and_lifecycle() -> None:
container = build_container()
commands = container.plugins.list_commands()
palette = container.plugins.list_commands(location="command_palette")
assert [item.command_id for item in commands] == ["text-tools.uppercase-selection"]
assert palette[0].plugin_id == "text-tools"
assert palette[0].icon == "edit"
assert palette[0].when == ["editor.has_selection"]
container.plugins.disable("text-tools")
assert container.plugins.list_commands() == []
with pytest.raises(ExtensionError) as exc:
run(
container.plugins.execute_command(
"text-tools.uppercase-selection",
{},
PluginCommandContext(selection="hello"),
)
)
assert exc.value.code == "PLUGIN_COMMAND_NOT_FOUND"
container.plugins.enable("text-tools")
assert len(container.plugins.list_commands()) == 1
def test_command_executes_with_scoped_context_and_settings() -> None:
container = build_container()
container.plugins.update_settings("text-tools", 1, {"result_limit": 4})
result = run(
container.plugins.execute_command(
"text-tools.uppercase-selection",
{},
PluginCommandContext(
vault_id="default",
note_id="note_private",
file_path="private.md",
selection="abcdef",
),
)
)
assert result.status == "completed"
assert result.effect.type == "notification"
assert result.effect.payload.model_dump() == {
"level": "success",
"message": "ABCD",
}
def test_echo_command_returns_none_for_empty_message() -> None:
host = DeclarativePluginHost()
empty = run(host.execute_command("echo", {}, {}, {}, lambda _: None))
populated = run(
host.execute_command("echo", {"message": "hello"}, {}, {}, lambda _: None)
)
assert isinstance(empty, PluginNoEffect)
assert populated.type == "notification"
assert populated.payload.message == "hello"
@pytest.mark.parametrize(
("effect_type", "payload"),
[
("none", {"unexpected": True}),
("notification", {"level": "debug", "message": "invalid"}),
("navigate", {"route": "https://example.com"}),
("refresh", {"scope": "everything"}),
("job", {"job_id": "invalid job id"}),
],
)
def test_command_effect_rejects_untrusted_payloads(effect_type, payload) -> None:
with pytest.raises(ValidationError):
TypeAdapter(PluginCommandEffect).validate_python(
{"type": effect_type, "payload": payload}
)
def test_command_rejects_missing_context_and_invalid_arguments() -> None:
container = build_container()
with pytest.raises(ExtensionError) as context_error:
run(
container.plugins.execute_command(
"text-tools.uppercase-selection", {}, PluginCommandContext()
)
)
assert context_error.value.code == "PLUGIN_COMMAND_CONTEXT_INVALID"
with pytest.raises(ExtensionError) as argument_error:
run(
container.plugins.execute_command(
"text-tools.uppercase-selection",
{"unknown": True},
PluginCommandContext(selection="hello"),
)
)
assert argument_error.value.code == "PLUGIN_COMMAND_ARGUMENT_INVALID"
audit = container.plugins.commands.audit_events()
assert [event.error_code for event in audit[-2:]] == [
"PLUGIN_COMMAND_CONTEXT_INVALID",
"PLUGIN_COMMAND_ARGUMENT_INVALID",
]
# 审计事件不得携带参数、正文选区或返回 effect。
assert "hello" not in repr(audit)
def test_command_only_receives_declared_context() -> None:
class CapturingHost(DeclarativePluginHost):
def __init__(self) -> None:
self.context = None
async def execute_command(
self, handler, arguments, context, settings, resolve_secret
):
self.context = context
return PluginNoEffect()
host = CapturingHost()
runtime = PluginRuntime(ToolRegistry(), host=host)
runtime.install(TEXT_TOOLS)
runtime.enable("text-tools")
run(
runtime.execute_command(
"text-tools.uppercase-selection",
{},
PluginCommandContext(
vault_id="default", note_id="note_private", selection="visible"
),
)
)
assert host.context == {"selection": "visible"}
def test_command_resolves_only_declared_plugin_secrets(tmp_path: Path) -> None:
class SecretHost(DeclarativePluginHost):
def __init__(self) -> None:
self.secret = None
self.denied_code = None
async def execute_command(
self, handler, arguments, context, settings, resolve_secret
):
self.secret = resolve_secret("api_key")
try:
resolve_secret("undeclared")
except ExtensionError as exc:
self.denied_code = exc.code
return PluginNoEffect()
host = SecretHost()
runtime = PluginRuntime(ToolRegistry(), host=host)
package = tmp_path / "secret-command"
package.mkdir()
(package / "plugin.yaml").write_text(
"""
id: secret-command
name: Secret Command
version: 1.0.0
permissions: [secrets.use]
contributes:
commands: [secret-command.run]
settings_sections: [secret-command.general]
backend:
type: internal_rpc
transport: none
""".strip(),
encoding="utf-8",
)
(package / "commands.yaml").write_text(
"""
commands:
- command_id: secret-command.run
title: Secret Command
locations: [command_palette]
secrets: [api_key]
handler: echo
""".strip(),
encoding="utf-8",
)
(package / "settings.yaml").write_text(
"""
section_id: secret-command.general
schema_version: 1
fields:
- key: api_key
label: API Key
type: secret
""".strip(),
encoding="utf-8",
)
runtime.install(package)
runtime.set_permissions("secret-command", ["secrets.use"])
runtime.enable("secret-command")
runtime.put_setting_secret("secret-command", "api_key", "runtime-only-secret")
run(
runtime.execute_command(
"secret-command.run",
{},
PluginCommandContext(selection="visible"),
)
)
assert host.secret == "runtime-only-secret"
assert host.denied_code == "PLUGIN_SECRET_ACCESS_DENIED"
assert "runtime-only-secret" not in repr(runtime.commands.audit_events())
def test_settings_schema_contains_defaults_and_hides_secret() -> None:
container = build_container()
schema = container.plugins.get_settings("text-tools")
by_key = {field.key: field for field in schema.fields}
assert schema.schema_version == 1
assert schema.values == {
"result_limit": 100,
"label_prefix": "",
"output_style": "notification",
"enabled_hint": True,
}
assert "api_key" not in schema.values
assert schema.secrets["api_key"].configured is False
assert by_key["api_key"].type == PluginSettingType.secret
def test_settings_update_validates_version_type_bounds_and_secret_boundary() -> None:
container = build_container()
updated = container.plugins.update_settings(
"text-tools", 1, {"result_limit": 20, "output_style": "compact"}
)
assert updated.values["result_limit"] == 20
assert updated.values["output_style"] == "compact"
cases = [
(2, {}, "PLUGIN_SETTINGS_VERSION_CONFLICT"),
(1, {"result_limit": 0}, "PLUGIN_SETTINGS_FIELD_INVALID"),
(1, {"enabled_hint": "yes"}, "PLUGIN_SETTINGS_FIELD_INVALID"),
(1, {"output_style": "unknown"}, "PLUGIN_SETTINGS_FIELD_INVALID"),
(1, {"api_key": "plaintext"}, "PLUGIN_SETTINGS_FIELD_INVALID"),
(1, {"unknown": True}, "PLUGIN_SETTINGS_FIELD_INVALID"),
]
for version, values, code in cases:
with pytest.raises(ExtensionError) as exc:
container.plugins.update_settings("text-tools", version, values)
assert exc.value.code == code
def test_required_plain_setting_blocks_enable_until_configured(tmp_path: Path) -> None:
package = tmp_path / "required-setting"
package.mkdir()
(package / "plugin.yaml").write_text(
"""
id: required-setting
name: Required Setting
version: 1.0.0
contributes:
commands: [required-setting.run]
settings_sections: [required-setting.general]
backend:
type: internal_rpc
transport: none
""".strip(),
encoding="utf-8",
)
(package / "commands.yaml").write_text(
"""
commands:
- command_id: required-setting.run
title: Required Setting
locations: [command_palette]
handler: echo
""".strip(),
encoding="utf-8",
)
(package / "settings.yaml").write_text(
"""
section_id: required-setting.general
schema_version: 1
fields:
- key: endpoint
label: Endpoint
type: string
required: true
""".strip(),
encoding="utf-8",
)
runtime = PluginRuntime(ToolRegistry())
runtime.install(package)
with pytest.raises(ExtensionError) as exc:
runtime.enable("required-setting")
assert exc.value.code == "PLUGIN_SETTINGS_REQUIRED"
assert runtime.get("required-setting").status == "installed"
runtime.update_settings("required-setting", 1, {"endpoint": "local"})
assert runtime.enable("required-setting").status == "ready"
def test_secret_roundtrip_never_enters_plain_settings_storage() -> None:
container = build_container()
plaintext = "stage-d-secret-value"
status = container.plugins.put_setting_secret("text-tools", "api_key", plaintext)
schema = container.plugins.get_settings("text-tools")
settings_path = get_settings().data_dir / "plugins" / "settings.json"
credentials_path = get_settings().data_dir / "credentials" / "credentials.json"
assert status.configured is True
assert schema.secrets["api_key"].configured is True
assert "api_key" not in schema.values
assert plaintext not in settings_path.read_text(encoding="utf-8")
assert plaintext not in credentials_path.read_text(encoding="utf-8")
stored_settings = json.loads(settings_path.read_text(encoding="utf-8"))
reference = stored_settings["text-tools"]["secret_refs"]["api_key"]
assert reference.startswith("plugin.")
assert len(reference) == 71
assert "text-tools" not in reference and "api_key" not in reference
assert container.credentials.resolve(reference) == plaintext
deleted = container.plugins.delete_setting_secret("text-tools", "api_key")
assert deleted.configured is False
assert container.credentials.resolve(reference) is None
def test_uninstall_removes_plugin_settings_and_secret_namespace() -> None:
container = build_container()
container.plugins.update_settings("text-tools", 1, {"result_limit": 12})
container.plugins.put_setting_secret("text-tools", "api_key", "temporary")
settings_path = get_settings().data_dir / "plugins" / "settings.json"
reference = json.loads(settings_path.read_text(encoding="utf-8"))[
"text-tools"
]["secret_refs"]["api_key"]
container.plugins.uninstall("text-tools")
stored = json.loads(settings_path.read_text(encoding="utf-8"))
assert "text-tools" not in stored
assert container.credentials.resolve(reference) is None
def test_plugin_secret_reference_has_fixed_credential_safe_length() -> None:
reference = _secret_reference("p" * 512, "k" * 128)
assert reference.startswith("plugin.")
assert len(reference) <= 128
def test_tampered_secret_reference_cannot_cross_credential_namespace() -> None:
container = build_container()
container.credentials.put("openai", "provider-private-secret")
settings_path = get_settings().data_dir / "plugins" / "settings.json"
settings_path.parent.mkdir(parents=True, exist_ok=True)
settings_path.write_text(
json.dumps(
{
"text-tools": {
"schema_version": 1,
"values": {},
"secret_refs": {"api_key": "openai"},
}
}
),
encoding="utf-8",
)
with pytest.raises(ExtensionError) as read_error:
container.plugins.get_settings("text-tools")
with pytest.raises(ExtensionError) as uninstall_error:
container.plugins.uninstall("text-tools")
assert read_error.value.code == "PLUGIN_STORAGE_ERROR"
assert uninstall_error.value.code == "PLUGIN_STORAGE_ERROR"
assert container.credentials.resolve("openai") == "provider-private-secret"
def test_secret_delete_restores_reference_when_credential_delete_fails(
monkeypatch,
) -> None:
container = build_container()
container.plugins.put_setting_secret("text-tools", "api_key", "keep-me")
settings_path = get_settings().data_dir / "plugins" / "settings.json"
original = settings_path.read_text(encoding="utf-8")
reference = _secret_reference("text-tools", "api_key")
def fail_delete(_credential_id: str) -> bool:
raise CredentialStoreError("injected delete failure")
monkeypatch.setattr(container.credentials, "delete", fail_delete)
with pytest.raises(ExtensionError) as exc:
container.plugins.delete_setting_secret("text-tools", "api_key")
assert exc.value.code == "PLUGIN_SECRET_STORE_ERROR"
assert settings_path.read_text(encoding="utf-8") == original
assert container.credentials.resolve(reference) == "keep-me"
def test_uninstall_restores_settings_when_atomic_secret_delete_fails(
monkeypatch,
) -> None:
container = build_container()
container.plugins.update_settings("text-tools", 1, {"result_limit": 12})
container.plugins.put_setting_secret("text-tools", "api_key", "keep-me")
settings_path = get_settings().data_dir / "plugins" / "settings.json"
original = settings_path.read_text(encoding="utf-8")
reference = _secret_reference("text-tools", "api_key")
def fail_delete_many(_credential_ids: list[str]) -> set[str]:
raise CredentialStoreError("injected batch delete failure")
monkeypatch.setattr(container.credentials, "delete_many", fail_delete_many)
with pytest.raises(ExtensionError) as exc:
container.plugins.uninstall("text-tools")
assert exc.value.code == "PLUGIN_SECRET_STORE_ERROR"
assert settings_path.read_text(encoding="utf-8") == original
assert container.credentials.resolve(reference) == "keep-me"
assert container.plugins.get("text-tools").manifest.plugin_id == "text-tools"
def test_invalid_command_and_settings_manifest_are_rejected(tmp_path: Path) -> None:
invalid_command = tmp_path / "invalid-command"
invalid_command.mkdir()
(invalid_command / "plugin.yaml").write_text(
"""
id: invalid-command
name: Invalid Command
version: 1.0.0
contributes:
commands: [other.run]
""".strip(),
encoding="utf-8",
)
(invalid_command / "commands.yaml").write_text(
"""
commands:
- command_id: other.run
title: Invalid
locations: [command_palette]
handler: echo
""".strip(),
encoding="utf-8",
)
invalid_settings = tmp_path / "invalid-settings"
invalid_settings.mkdir()
(invalid_settings / "plugin.yaml").write_text(
"""
id: invalid-settings
name: Invalid Settings
version: 1.0.0
contributes:
settings_sections: [invalid-settings.general]
""".strip(),
encoding="utf-8",
)
(invalid_settings / "settings.yaml").write_text(
"""
section_id: invalid-settings.general
schema_version: 1
fields:
- key: token
label: Token
type: secret
default: leaked-default
""".strip(),
encoding="utf-8",
)
runtime = PluginRuntime(ToolRegistry())
with pytest.raises(ExtensionError) as command_error:
runtime.install(invalid_command)
assert command_error.value.code == "PLUGIN_COMMAND_INVALID"
with pytest.raises(ExtensionError) as settings_error:
runtime.install(invalid_settings)
assert settings_error.value.code == "PLUGIN_SETTINGS_SCHEMA_INVALID"
@pytest.mark.parametrize("bound", [".nan", ".inf", "-.inf"])
def test_non_finite_setting_bounds_are_rejected(tmp_path: Path, bound: str) -> None:
package = tmp_path / f"invalid-bound-{bound.replace('.', 'dot').replace('-', 'neg')}"
package.mkdir()
(package / "plugin.yaml").write_text(
"""
id: invalid-bound
name: Invalid Bound
version: 1.0.0
contributes:
settings_sections: [invalid-bound.general]
backend:
type: none
transport: none
""".strip(),
encoding="utf-8",
)
(package / "settings.yaml").write_text(
f"""
section_id: invalid-bound.general
schema_version: 1
fields:
- key: limit
label: Limit
type: number
minimum: {bound}
""".strip(),
encoding="utf-8",
)
with pytest.raises(ExtensionError) as exc:
PluginRuntime(ToolRegistry()).install(package)
assert exc.value.code == "PLUGIN_SETTINGS_SCHEMA_INVALID"
assert "must be finite" in exc.value.message
def test_null_command_list_returns_stable_manifest_error(tmp_path: Path) -> None:
package = tmp_path / "null-commands"
package.mkdir()
(package / "plugin.yaml").write_text(
"""
id: null-commands
name: Null Commands
version: 1.0.0
contributes:
commands: []
backend:
type: internal_rpc
transport: none
""".strip(),
encoding="utf-8",
)
(package / "commands.yaml").write_text("commands:\n", encoding="utf-8")
with pytest.raises(ExtensionError) as exc:
PluginRuntime(ToolRegistry()).install(package)
assert exc.value.code == "EXTENSION_MANIFEST_INVALID"
def test_external_command_schema_reference_is_rejected(tmp_path: Path) -> None:
package = tmp_path / "external-ref"
package.mkdir()
(package / "plugin.yaml").write_text(
"""
id: external-ref
name: External Ref
version: 1.0.0
contributes:
commands: [external-ref.run]
backend:
type: internal_rpc
transport: none
""".strip(),
encoding="utf-8",
)
(package / "commands.yaml").write_text(
"""
commands:
- command_id: external-ref.run
title: External Ref
locations: [command_palette]
handler: echo
parameters:
$ref: file:///host/private-schema.json
""".strip(),
encoding="utf-8",
)
with pytest.raises(ExtensionError) as exc:
PluginRuntime(ToolRegistry()).install(package)
assert exc.value.code == "PLUGIN_COMMAND_INVALID"
assert "External JSON Schema reference" in exc.value.message
def test_settings_missing_and_secret_field_errors_are_stable() -> None:
container = build_container()
with pytest.raises(ExtensionError) as missing:
container.plugins.get_settings("does-not-exist")
assert missing.value.code == "PLUGIN_NOT_FOUND"
with pytest.raises(ExtensionError) as field:
container.plugins.put_setting_secret("text-tools", "result_limit", "secret")
assert field.value.code == "PLUGIN_SECRET_FIELD_NOT_FOUND"
with pytest.raises(ExtensionError) as empty:
container.plugins.put_setting_secret("text-tools", "api_key", "")
assert empty.value.code == "PLUGIN_SECRET_VALUE_INVALID"
def test_corrupted_plugin_settings_namespace_returns_stable_error() -> None:
container = build_container()
settings_path = get_settings().data_dir / "plugins" / "settings.json"
settings_path.parent.mkdir(parents=True, exist_ok=True)
settings_path.write_text('{"text-tools": []}', encoding="utf-8")
with pytest.raises(ExtensionError) as exc:
container.plugins.get_settings("text-tools")
assert exc.value.code == "PLUGIN_STORAGE_ERROR"
with pytest.raises(ExtensionError) as secret_exc:
container.plugins.put_setting_secret("text-tools", "api_key", "must-not-orphan")
assert secret_exc.value.code == "PLUGIN_STORAGE_ERROR"
credentials_path = get_settings().data_dir / "credentials" / "credentials.json"
credential_ids = (
json.loads(credentials_path.read_text(encoding="utf-8")).keys()
if credentials_path.exists()
else []
)
assert not any(item.startswith("plugin.") for item in credential_ids)
+50 -1
View File
@@ -12,6 +12,8 @@ from app.contracts import (
ToolDefinition,
)
from app.providers.ollama import OllamaProvider
from app.providers.base import ProviderError
from app.providers.credentials import EnvironmentCredentialResolver
from app.providers.openai_compatible import OpenAICompatibleProvider
@@ -82,7 +84,7 @@ def test_openai_compatible_maps_tool_call_and_credentials() -> None:
)
)
assert captured["tools"][0]["function"]["name"] == "math.add"
assert captured["tools"][0]["function"]["name"].startswith("tool_")
assert turn.tool_calls[0].name == "math.add"
assert turn.tool_calls[0].arguments == {"left": 1, "right": 2}
assert turn.input_tokens == 8
@@ -134,6 +136,53 @@ def test_openai_compatible_preserves_tool_call_context() -> None:
assert turn.text == "done"
def test_openai_compatible_fetches_and_maps_model_list() -> None:
def handler(request: httpx.Request) -> httpx.Response:
assert request.method == "GET"
assert request.url.path == "/v1/models"
assert request.headers["Authorization"] == "Bearer secret-test-key"
return httpx.Response(
200,
json={"data": [{"id": "model-b"}, {"id": "model-a"}]},
)
provider = OpenAICompatibleProvider(
base_url="https://provider.test/v1",
credential_id="provider-test",
credentials=StaticCredentials(),
transport=httpx.MockTransport(handler),
)
models = run(provider.list_models())
assert [item.model for item in models] == ["model-b", "model-a"]
def test_environment_credentials_support_deepseek_development_alias(monkeypatch) -> None:
monkeypatch.setenv("DEEPSEEK_API_KEY", "secret-test-key")
assert EnvironmentCredentialResolver().resolve("deepseek") == "secret-test-key"
def test_openai_compatible_rejects_missing_named_credential_before_request() -> None:
class EmptyCredentials:
def resolve(self, credential_id: str | None) -> str | None:
return None
provider = OpenAICompatibleProvider(
base_url="https://provider.test/v1",
credential_id="deepseek",
credentials=EmptyCredentials(),
)
try:
run(provider.list_models())
except ProviderError as error:
assert error.code == "PROVIDER_CREDENTIAL_MISSING"
else:
raise AssertionError("Missing credential should fail before the provider request")
def test_ollama_maps_models_and_completion() -> None:
def handler(request: httpx.Request) -> httpx.Response:
if request.url.path == "/api/tags":
+610
View File
@@ -0,0 +1,610 @@
"""Wire-level provider tests: no credentials, SDKs, clocks, or network services."""
import asyncio
import json
import httpx
import pytest
from app.contracts import Message, MessageRole, ModelCapability, ModelEventType as E, ModelRequest, ToolCall, ToolDefinition
from app.providers.anthropic_messages import AnthropicMessagesProvider
from app.providers.base import ProviderError
from app.providers.ollama import OllamaProvider
from app.providers.openai_compatible import OpenAICompatibleProvider
from app.providers.openai_responses import OpenAIResponsesProvider
NATIVE = ["responses", "anthropic"]
PROTOCOLS = [*NATIVE, "compatible", "ollama"]
SECRET = "test-only-sensitive-upstream-body"
class Credentials:
def resolve(self, credential_id):
return SECRET if credential_id else None
class Bytes(httpx.AsyncByteStream):
def __init__(self, body: bytes, *, fragment: int = 17):
self.body = body
self.fragment = fragment
self.closed = False
async def __aiter__(self):
for offset in range(0, len(self.body), self.fragment):
yield self.body[offset:offset + self.fragment]
async def aclose(self):
self.closed = True
class GatedBytes(Bytes):
def __init__(self, body):
super().__init__(body)
self.waiting = asyncio.Event()
self.release = asyncio.Event()
async def __aiter__(self):
yield self.body
self.waiting.set()
await self.release.wait()
def provider(protocol, handler, *, credential_id="test"):
transport = httpx.MockTransport(handler)
if protocol == "ollama":
return OllamaProvider("https://provider.test", transport=transport)
cls = {"responses": OpenAIResponsesProvider, "anthropic": AnthropicMessagesProvider,
"compatible": OpenAICompatibleProvider}[protocol]
return cls("https://provider.test/v1/", credential_id, Credentials(), transport=transport)
def request(*, history=False):
messages = [Message(role=MessageRole.user, content="查笔记")]
if history:
messages += [
Message(role=MessageRole.system, content="Additional rules"),
Message(role=MessageRole.assistant, content="Checking", tool_calls=[
ToolCall(tool_call_id="old_1", name="lookup", arguments={"query": "a"}),
ToolCall(tool_call_id="old_2", name="lookup", arguments={"query": "b"}),
]),
Message(role=MessageRole.tool, tool_call_id="old_1", content='{"found":1}'),
Message(role=MessageRole.tool, tool_call_id="old_2", content='{"found":2}'),
]
return ModelRequest(
provider_id="test", model="model", system="System rules", messages=messages,
tools=[ToolDefinition(name="lookup", description="Find notes", parameters={"type": "object"})],
max_tokens=512, temperature=0,
)
async def collect(iterator):
return [event async for event in iterator]
@pytest.mark.parametrize("name", ["lookup", "notes.search"])
def test_compatible_split_tool_name_preserves_identity(name):
from app.providers.tool_names import prepare_tool_names
req = request()
req.tools[0].name = name
wire, _ = prepare_tool_names(req)
alias = wire.tools[0].name
def handler(_):
return httpx.Response(200, content=sse(
{"choices": [{"delta": {"tool_calls": [{"index": 0, "id": "call_1",
"function": {"name": alias[:3], "arguments": ""}}]}}]},
{"choices": [{"delta": {"tool_calls": [{"index": 0,
"function": {"name": alias[3:], "arguments": '{"query":"x"}'}}]},
"finish_reason": "tool_calls"}]},
{"type": "[DONE]"},
))
events = asyncio.run(collect(provider("compatible", handler).stream(req)))
assert [e.data["name"] for e in events if e.event == E.tool_call_start] == [name]
assert json.loads("".join(e.data["arguments_delta"] for e in events
if e.event == E.tool_call_delta)) == {"query": "x"}
assert events[-1].data["status"] == "completed"
def sse(*events):
return "".join(
f"event: {event.get('type', 'message')}\r\ndata: {json.dumps(event, ensure_ascii=False)}\r\n\r\n"
for event in events
).encode()
def wire(protocol, *events):
if protocol == "ollama":
return ("\n".join(json.dumps(event, ensure_ascii=False) for event in events) + "\n").encode()
return sse(*events)
def start(protocol):
if protocol == "responses":
return [{"type": "response.output_text.delta", "delta": "你好"}]
if protocol == "anthropic":
return [{"type": "message_start", "message": {"usage": {"input_tokens": 7, "output_tokens": 0}}},
{"type": "content_block_start", "index": 0, "content_block": {"type": "text", "text": ""}},
{"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": "你好"}}]
if protocol == "compatible":
return [{"choices": [{"delta": {"content": "你好"}}]}]
return [{"message": {"content": "你好"}, "done": False}]
def terminal(protocol):
if protocol == "responses":
return [{"type": "response.completed", "response": {"status": "completed", "usage": {"input_tokens": 7, "output_tokens": 2}}}]
if protocol == "anthropic":
return [{"type": "content_block_stop", "index": 0},
{"type": "message_delta", "delta": {"stop_reason": "end_turn"}, "usage": {"output_tokens": 2}},
{"type": "message_stop"}]
if protocol == "compatible":
return [{"choices": [{"delta": {}, "finish_reason": "stop"}], "usage": {"prompt_tokens": 7, "completion_tokens": 2}}]
return [{"message": {}, "done": True, "prompt_eval_count": 7, "eval_count": 2}]
def assert_events(events):
assert events[-1].event == E.done
assert events[-1].data["status"] == ("failed" if any(event.event == E.error for event in events) else "completed")
assert sum(event.event == E.done for event in events) == 1
assert [event.sequence for event in events] == list(range(len(events)))
assert all(event.timestamp.tzinfo is not None for event in events)
def assert_error(events, code):
assert_events(events)
assert events[-2].event == E.error
assert events[-2].data["code"] == code
assert SECRET not in str(events[-2].data)
@pytest.mark.parametrize("protocol", NATIVE)
def test_native_completion_and_history(protocol):
captured = {}
def handler(req):
captured.update(json.loads(req.content))
assert req.url.path == ("/v1/responses" if protocol == "responses" else "/v1/messages")
if protocol == "responses":
assert req.headers["authorization"] == f"Bearer {SECRET}"
body = {"status": "completed", "output": [
{"type": "reasoning", "summary": [{"type": "summary_text", "text": "thinking"}]},
{"type": "message", "content": [{"type": "output_text", "text": "完成"}]},
{"type": "function_call", "call_id": "next", "name": "lookup", "arguments": '{"query":"c"}'},
], "usage": {"input_tokens": 10, "output_tokens": 3}}
else:
assert "authorization" not in req.headers
assert req.headers["x-api-key"] == SECRET
assert req.headers["anthropic-version"] == "2023-06-01"
body = {"type": "message", "content": [
{"type": "thinking", "thinking": "thinking", "signature": "sig"},
{"type": "text", "text": "完成"},
{"type": "tool_use", "id": "next", "name": "lookup", "input": {"query": "c"}},
], "usage": {"input_tokens": 5, "cache_creation_input_tokens": 2, "cache_read_input_tokens": 3, "output_tokens": 3}}
return httpx.Response(200, json=body)
turn = asyncio.run(provider(protocol, handler).complete(request(history=True)))
assert turn.text == "完成"
assert (turn.input_tokens, turn.output_tokens) == (10, 3)
assert turn.tool_calls[0].tool_call_id == "next"
assert turn.tool_calls[0].arguments == {"query": "c"}
assert captured["stream"] is False
assert captured["temperature"] == 0
if protocol == "responses":
assert captured["instructions"] == "System rules"
assert captured["max_output_tokens"] == 512
assert captured["tools"][0]["parameters"] == {"type": "object"}
calls = [item for item in captured["input"] if item.get("type") == "function_call"]
outputs = [item for item in captured["input"] if item.get("type") == "function_call_output"]
assert [call["call_id"] for call in calls] == ["old_1", "old_2"]
assert json.loads(calls[1]["arguments"]) == {"query": "b"}
assert outputs == [{"type": "function_call_output", "call_id": "old_1", "output": '{"found":1}'},
{"type": "function_call_output", "call_id": "old_2", "output": '{"found":2}'}]
assert {"role": "system", "content": "Additional rules"} in captured["input"]
else:
assert captured["system"] == "System rules\n\nAdditional rules"
assert captured["max_tokens"] == 512
assert captured["tools"][0]["input_schema"] == {"type": "object"}
assert captured["messages"][1]["content"][2] == {
"type": "tool_use", "id": "old_2", "name": "lookup", "input": {"query": "b"},
}
assert captured["messages"][-1] == {"role": "user", "content": [
{"type": "tool_result", "tool_use_id": "old_1", "content": '{"found":1}'},
{"type": "tool_result", "tool_use_id": "old_2", "content": '{"found":2}'},
]}
def responses_tool_events():
events = [
{"type": "response.created", "response": {"usage": {"input_tokens": 10, "output_tokens": 0}}},
{"type": "response.reasoning_summary_text.delta", "delta": "计划"},
{"type": "response.output_text.delta", "delta": ""},
{"type": "response.output_text.delta", "delta": ""},
]
for index in (2, 3):
events.append({"type": "response.output_item.added", "output_index": index, "item": {
"id": f"item_{index}", "type": "function_call", "call_id": f"call_{index}", "name": "lookup", "arguments": "",
}})
for index, fragment in [(2, '{"query":'), (3, '{}'), (2, '"笔记"}')]:
events.append({"type": "response.function_call_arguments.delta", "output_index": index,
"item_id": f"item_{index}", "delta": fragment})
for index, arguments in [(3, '{}'), (2, '{"query":"笔记"}')]:
events += [
{"type": "response.function_call_arguments.done", "output_index": index, "item_id": f"item_{index}", "arguments": arguments},
{"type": "response.output_item.done", "output_index": index, "item": {
"id": f"item_{index}", "type": "function_call", "call_id": f"call_{index}", "name": "lookup", "arguments": arguments,
}},
]
events += [{"type": "future.event"}, {"type": "response.completed", "response": {
"status": "completed", "usage": {"input_tokens": 10, "output_tokens": 9},
}}]
return events
def anthropic_tool_events():
events = [
{"type": "message_start", "message": {"usage": {
"input_tokens": 5, "cache_read_input_tokens": 3, "cache_creation_input_tokens": 2, "output_tokens": 1,
}}},
{"type": "ping"},
{"type": "content_block_start", "index": 0, "content_block": {"type": "thinking", "thinking": ""}},
{"type": "content_block_delta", "index": 0, "delta": {"type": "thinking_delta", "thinking": "计划"}},
{"type": "content_block_delta", "index": 0, "delta": {"type": "signature_delta", "signature": "sig"}},
{"type": "content_block_stop", "index": 0},
{"type": "content_block_start", "index": 1, "content_block": {"type": "text", "text": ""}},
{"type": "content_block_delta", "index": 1, "delta": {"type": "text_delta", "text": ""}},
{"type": "content_block_stop", "index": 1},
]
for index, fragments in [(2, ['{"query":', '"笔记"}']), (3, [])]:
events.append({"type": "content_block_start", "index": index, "content_block": {
"type": "tool_use", "id": f"call_{index}", "name": "lookup", "input": {},
}})
for fragment in fragments:
events.append({"type": "content_block_delta", "index": index,
"delta": {"type": "input_json_delta", "partial_json": fragment}})
events.append({"type": "content_block_stop", "index": index})
events += [
{"type": "message_delta", "delta": {"stop_reason": "tool_use"}, "usage": {"output_tokens": 4}},
{"type": "future.event"},
{"type": "message_delta", "delta": {}, "usage": {"output_tokens": 9}},
{"type": "message_stop"},
]
return events
@pytest.mark.parametrize("protocol", NATIVE)
def test_native_stream_tools_reasoning_usage_and_fragmented_utf8(protocol):
frames = responses_tool_events() if protocol == "responses" else anthropic_tool_events()
body = Bytes(b": comment\r\n\r\n" + sse(*frames) + b"data: malformed after completion\n\n", fragment=1)
def handler(req):
payload = json.loads(req.content)
assert payload["stream"] is True
assert payload["tools"]
assert (payload.get("input") or payload.get("messages"))
return httpx.Response(200, stream=body)
events = asyncio.run(collect(provider(protocol, handler).stream(request(history=True))))
assert_events(events)
assert not any(event.event == E.error for event in events)
assert [event.data["text"] for event in events if event.event == E.text_delta] == ["", ""]
assert [event.data["text"] for event in events if event.event == E.thinking_delta] == ["计划"]
assert [event.data["tool_call_id"] for event in events if event.event == E.tool_call_start] == ["call_2", "call_3"]
assert sorted(event.data["tool_call_id"] for event in events if event.event == E.tool_call_end) == ["call_2", "call_3"]
for call_id, expected in [("call_2", {"query": "笔记"}), ("call_3", {})]:
arguments = "".join(event.data["arguments_delta"] for event in events
if event.event == E.tool_call_delta and event.data["tool_call_id"] == call_id)
assert json.loads(arguments) == expected
usages = [event.data for event in events if event.event == E.usage]
assert usages[-1] == {"input_tokens": 10, "output_tokens": 9, "total_tokens": 19}
assert all(usage["input_tokens"] == 10 for usage in usages)
if protocol == "anthropic":
assert [usage["output_tokens"] for usage in usages] == [1, 4, 9]
assert body.closed
@pytest.mark.parametrize("protocol", PROTOCOLS)
def test_stream_terminal_usage_and_closure(protocol):
body = Bytes(wire(protocol, *start(protocol), *terminal(protocol)))
events = asyncio.run(collect(provider(protocol, lambda _: httpx.Response(200, stream=body)).stream(request())))
assert_events(events)
assert not any(event.event == E.error for event in events)
assert [event.data["text"] for event in events if event.event == E.text_delta] == ["你好"]
assert [event.data for event in events if event.event == E.usage][-1] == {"input_tokens": 7, "output_tokens": 2, "total_tokens": 9}
assert body.closed
@pytest.mark.parametrize("protocol", PROTOCOLS)
@pytest.mark.parametrize("empty", [False, True])
def test_truncated_stream(protocol, empty):
body = Bytes(b"" if empty else wire(protocol, *start(protocol)))
events = asyncio.run(collect(provider(protocol, lambda _: httpx.Response(200, stream=body)).stream(request())))
assert_error(events, "PROVIDER_STREAM_TRUNCATED")
assert body.closed
@pytest.mark.parametrize("protocol", PROTOCOLS)
@pytest.mark.parametrize("bad", [b"not-json", b"[]", b"null", b'{"usage":'])
def test_malformed_stream_is_sanitized(protocol, bad):
suffix = bad + b"\n" if protocol == "ollama" else b"data: " + bad + b"\n\n"
body = Bytes(wire(protocol, *start(protocol)) + suffix)
events = asyncio.run(collect(provider(protocol, lambda _: httpx.Response(200, stream=body)).stream(request())))
assert_error(events, "PROVIDER_INVALID_RESPONSE")
assert body.closed
@pytest.mark.parametrize("protocol", PROTOCOLS)
@pytest.mark.parametrize("error_type,code", [("rate_limit_error", "PROVIDER_RATE_LIMITED"),
("authentication_error", "PROVIDER_AUTH_FAILED"),
("overloaded_error", "PROVIDER_UNAVAILABLE")])
def test_in_band_error_after_partial_output(protocol, error_type, code):
body = Bytes(wire(protocol, *start(protocol), {"type": "error", "error": {"type": error_type, "message": SECRET}}))
events = asyncio.run(collect(provider(protocol, lambda _: httpx.Response(200, stream=body)).stream(request())))
assert any(event.event == E.text_delta for event in events)
assert_error(events, code)
assert body.closed
@pytest.mark.parametrize("protocol", PROTOCOLS)
@pytest.mark.parametrize("status,code", [(400, "PROVIDER_INVALID_REQUEST"), (401, "PROVIDER_AUTH_FAILED"),
(403, "PROVIDER_AUTH_FAILED"), (404, "MODEL_NOT_FOUND"),
(429, "PROVIDER_RATE_LIMITED"), (500, "PROVIDER_UNAVAILABLE")])
def test_http_errors_completion_and_stream(protocol, status, code):
adapter = provider(protocol, lambda _: httpx.Response(status, text=SECRET))
with pytest.raises(ProviderError) as exc:
asyncio.run(adapter.complete(request()))
assert exc.value.code == code
assert SECRET not in str(exc.value)
assert_error(asyncio.run(collect(adapter.stream(request()))), code)
@pytest.mark.parametrize("protocol", PROTOCOLS)
@pytest.mark.parametrize("body,code", [(b"broken", "PROVIDER_INVALID_RESPONSE"),
(b"[]", "PROVIDER_INVALID_RESPONSE"),
(b"{}", "PROVIDER_INVALID_RESPONSE"),
(json.dumps({"error": {"code": "invalid_api_key", "message": SECRET}}).encode(), "PROVIDER_AUTH_FAILED")])
def test_bad_completion(protocol, body, code):
adapter = provider(protocol, lambda _: httpx.Response(200, content=body))
with pytest.raises(ProviderError) as exc:
asyncio.run(adapter.complete(request()))
assert exc.value.code == code
assert SECRET not in str(exc.value)
@pytest.mark.parametrize("protocol", PROTOCOLS)
@pytest.mark.parametrize("error,code", [(httpx.ReadTimeout, "PROVIDER_TIMEOUT"),
(httpx.ConnectError, "PROVIDER_UNAVAILABLE")])
def test_transport_error_mapping(protocol, error, code):
def handler(req):
raise error(SECRET, request=req)
adapter = provider(protocol, handler)
with pytest.raises(ProviderError) as exc:
asyncio.run(adapter.complete(request()))
assert exc.value.code == code
assert SECRET not in str(exc.value)
assert_error(asyncio.run(collect(adapter.stream(request()))), code)
@pytest.mark.parametrize("protocol", PROTOCOLS)
@pytest.mark.parametrize("cancel", [True, False])
def test_incremental_delivery_cancellation_and_explicit_close(protocol, cancel):
async def scenario():
body = GatedBytes(wire(protocol, *start(protocol)))
adapter = provider(protocol, lambda _: httpx.Response(200, stream=body))
iterator = adapter.stream(request())
seen = []
while True:
event = await asyncio.wait_for(anext(iterator), timeout=1)
seen.append(event)
if event.event == E.text_delta:
break
# The first token arrives while the response is still open and blocked.
assert seen[-1].data["text"] == "你好"
assert not body.closed
if cancel:
pending = asyncio.create_task(anext(iterator))
await asyncio.wait_for(body.waiting.wait(), timeout=1)
pending.cancel()
with pytest.raises(asyncio.CancelledError):
await pending
else:
await iterator.aclose()
assert body.closed
assert not any(event.event in {E.error, E.done} for event in seen)
asyncio.run(scenario())
@pytest.mark.parametrize("protocol", NATIVE)
def test_cancellation_before_response_headers(protocol):
async def scenario():
entered = asyncio.Event()
closed = asyncio.Event()
async def handler(req):
entered.set()
try:
await asyncio.Event().wait()
finally:
closed.set()
adapter = provider(protocol, handler)
pending = asyncio.create_task(adapter.complete(request()))
await asyncio.wait_for(entered.wait(), timeout=1)
pending.cancel()
with pytest.raises(asyncio.CancelledError):
await pending
assert closed.is_set()
asyncio.run(scenario())
@pytest.mark.parametrize("protocol", NATIVE)
def test_native_discovery_does_not_claim_non_chat_capabilities(protocol):
def handler(req):
assert req.url.path == "/v1/models"
return httpx.Response(200, json={"data": [{"id": name} for name in ["chat-model", "text-embedding-3-small", "whisper-1", "gpt-audio"]]})
models = asyncio.run(provider(protocol, handler).list_models())
assert ModelCapability.chat in models[0].capabilities
assert models[1].capabilities == [ModelCapability.embedding]
assert all(ModelCapability.chat not in model.capabilities for model in models[1:])
@pytest.mark.parametrize("protocol", NATIVE)
def test_native_structured_format_mapping(protocol):
adapter = provider(protocol, lambda _: pytest.fail("No network expected"))
req = request()
req.response_format = {"type": "json_schema", "json_schema": {
"name": "answer", "strict": True, "schema": {"type": "object", "properties": {}},
}}
payload = adapter._payload(req, stream=False)
format_ = payload["text"]["format"] if protocol == "responses" else payload["output_config"]["format"]
assert format_["type"] == "json_schema"
assert format_["schema"] == {"type": "object", "properties": {}}
if protocol == "responses":
assert format_["name"] == "answer"
assert format_["strict"] is True
@pytest.mark.parametrize("protocol", NATIVE)
def test_invalid_tool_arguments_and_unclosed_tool(protocol):
frames = responses_tool_events() if protocol == "responses" else anthropic_tool_events()
# A syntactically valid terminal cannot rescue an unfinished tool block.
index = next(i for i, frame in enumerate(frames)
if frame["type"] in {"response.function_call_arguments.delta", "content_block_delta"}
and (frame.get("output_index") == 2 or frame.get("index") == 2))
partial = frames[:index + 1]
final = frames[-1]
events = asyncio.run(collect(provider(protocol, lambda _: httpx.Response(200, content=sse(*partial, final))).stream(request())))
assert_error(events, "PROVIDER_STREAM_TRUNCATED")
assert not any(event.event == E.tool_call_end for event in events)
for frame in frames:
if frame["type"] == "response.function_call_arguments.done":
frame["arguments"] = "[]"
break
if frame["type"] == "content_block_delta" and frame.get("index") == 2:
frame["delta"]["partial_json"] = "malformed"
break
events = asyncio.run(collect(provider(protocol, lambda _: httpx.Response(200, content=sse(*frames))).stream(request())))
assert_error(events, "PROVIDER_INVALID_RESPONSE")
@pytest.mark.parametrize("kind,code", [("response.failed", "PROVIDER_UNAVAILABLE"),
("response.incomplete", "PROVIDER_INCOMPLETE_RESPONSE")])
def test_responses_failed_and_incomplete(kind, code):
frame = {"type": kind, "response": {"status": kind.split(".")[1], "incomplete_details": {"reason": SECRET}}}
events = asyncio.run(collect(provider("responses", lambda _: httpx.Response(200, content=sse(*start("responses"), frame))).stream(request())))
assert_error(events, code)
def test_sse_multiline_data_and_event_name_without_json_type():
body = (b': keepalive\n\nevent: response.output_text.delta\ndata: {\ndata: "delta": "hello"\ndata: }\n\n'
+ sse({"type": "response.completed", "response": {"status": "completed"}}))
events = asyncio.run(collect(provider("responses", lambda _: httpx.Response(200, content=body)).stream(request())))
assert_events(events)
assert [event.data["text"] for event in events if event.event == E.text_delta] == ["hello"]
assert not any(event.event == E.error for event in events)
def test_ollama_history_options_and_in_band_string_error():
captured = {}
def handler(req):
captured.update(json.loads(req.content))
return httpx.Response(200, json={"error": SECRET})
with pytest.raises(ProviderError) as exc:
asyncio.run(provider("ollama", handler).complete(request(history=True)))
assert exc.value.code == "PROVIDER_UNAVAILABLE"
assert SECRET not in str(exc.value)
assert captured["messages"][-1]["tool_name"] == "lookup"
assert captured["options"] == {"temperature": 0.0, "num_predict": 512}
@pytest.mark.parametrize("protocol", PROTOCOLS)
@pytest.mark.parametrize("streaming", [False, True])
def test_namespaced_tools_roundtrip_without_changing_internal_request(protocol, streaming):
import re
model_request = request(history=True)
original_name = "mcp.my-server.search.notes"
model_request.tools[0].name = original_name
for message in model_request.messages:
for call in message.tool_calls:
call.name = original_name
before = model_request.model_dump()
def handler(req):
payload = json.loads(req.content)
definition = payload["tools"][0]
name = (definition.get("function") or definition)["name"]
assert name != original_name and re.fullmatch(r"[a-zA-Z0-9_-]{1,64}", name)
assert original_name not in req.content.decode()
if protocol == "responses":
item = {"type": "function_call", "id": "item1", "call_id": "call1", "name": name, "arguments": "{}"}
body = {"status": "completed", "output": [item]}
events = [
{"type": "response.output_item.done", "output_index": 0, "item": item},
{"type": "response.completed", "response": {"status": "completed"}},
]
elif protocol == "anthropic":
item = {"type": "tool_use", "id": "call1", "name": name, "input": {}}
body = {"content": [item]}
events = [
{"type": "message_start", "message": {}},
{"type": "content_block_start", "index": 0, "content_block": item},
{"type": "content_block_stop", "index": 0},
{"type": "message_stop"},
]
elif protocol == "compatible":
item = {"id": "call1", "function": {"name": name, "arguments": "{}"}}
body = {"choices": [{"message": {"tool_calls": [item]}}]}
events = [{"choices": [{"delta": {"tool_calls": [{"index": 0, **item}]}, "finish_reason": "tool_calls"}]}]
else:
item = {"function": {"name": name, "arguments": {}}}
body = {"message": {"tool_calls": [item]}, "done": True}
events = [body]
return httpx.Response(200, content=wire(protocol, *events)) if streaming else httpx.Response(200, json=body)
adapter = provider(protocol, handler)
if streaming:
events = asyncio.run(collect(adapter.stream(model_request)))
assert_events(events)
assert [event.data["name"] for event in events if event.event == E.tool_call_start] == [original_name]
else:
assert asyncio.run(adapter.complete(model_request)).tool_calls[0].name == original_name
assert model_request.model_dump() == before
def test_chat_route_closes_upstream_and_sanitizes_unexpected_errors(monkeypatch):
from types import SimpleNamespace
from datetime import datetime, timezone
from app import routes
from app.contracts import ChatRequest, ModelEvent
closed = []
class Adapter:
async def stream(self, request):
try:
yield ModelEvent(event=E.text_delta, sequence=0, data={"text": "first"}, timestamp=datetime.now(timezone.utc))
raise RuntimeError(SECRET)
finally:
closed.append(True)
monkeypatch.setattr(routes, "provider_or_404", lambda _: SimpleNamespace(adapter=Adapter()))
async def scenario():
response = await routes.chat(ChatRequest(provider_id="test", model="test", messages=[]))
iterator = response.body_iterator
await anext(iterator)
await iterator.aclose()
assert len(closed) == 1
response = await routes.chat(ChatRequest(provider_id="test", model="test", messages=[]))
items = [json.loads(chunk.split("data: ")[1].strip()) async for chunk in response.body_iterator]
assert [item["sequence"] for item in items] == [0, 1, 2]
assert items[-1]["data"]["status"] == "failed"
assert SECRET not in str(items)
assert len(closed) == 2
asyncio.run(scenario())
+82 -4
View File
@@ -436,6 +436,83 @@ def test_fts_pagination_is_not_truncated_at_one_thousand(vault) -> None:
assert len(response.items) == 10
def test_fts_score_threshold_filters_before_total(vault) -> None:
"""score_threshold 先于计数与分页生效:total 反映过滤后数量,与 items 一致。
高阈值过滤掉全部结果时 total==0 items 为空杜绝空页但 total>0
不一致审阅 P2-7
"""
from app.retrieval.engine import engine
from app.services import note_service
# 10 个 block,含「目标」次数递增,bm25 分数各异,min-max 归一化后分数落在 [0,1]
markdown = "\n\n".join(f"{'目标' * i} 分隔内容" for i in range(1, 11))
asyncio.run(
note_service.create_note(title="阈值过滤", markdown=markdown, folder="", tags=[])
)
all_hits = asyncio.run(
engine.search(
SearchRequest(query="目标", mode=SearchMode.fts, limit=20, score_threshold=0.0)
)
)
filtered = asyncio.run(
engine.search(
SearchRequest(query="目标", mode=SearchMode.fts, limit=20, score_threshold=0.5)
)
)
none = asyncio.run(
engine.search(
SearchRequest(query="目标", mode=SearchMode.fts, limit=20, score_threshold=2.0)
)
)
assert all_hits.page.total >= 10
assert 0 < filtered.page.total < all_hits.page.total # 阈值过滤掉部分而非全部
assert filtered.page.total == len(filtered.items)
assert none.page.total == 0
assert none.items == []
def test_fts_offset_beyond_end_reports_real_total(vault) -> None:
"""offset 越过末页时 items 为空,但 total 仍为真实命中数而非归零。"""
from app.retrieval.engine import engine
from app.services import note_service
asyncio.run(
note_service.create_note(title="越界分页", markdown="检索 检索 检索 检索", folder="", tags=[])
)
resp = asyncio.run(
engine.search(SearchRequest(query="检索", mode=SearchMode.fts, limit=10, offset=100))
)
assert resp.page.total >= 1
assert resp.items == []
def test_fts_not_truncated_at_five_thousand(vault) -> None:
"""FTS 结果不再被 5000 条上限截断:>5000 命中时 total 为真实计数,末页仍可访问。"""
from app.retrieval.engine import engine
from app.services import note_service
markdown = "\n\n".join(f"共同词 q{i}" for i in range(5010))
asyncio.run(
note_service.create_note(title="五千条分页", markdown=markdown, folder="", tags=[])
)
first = asyncio.run(
engine.search(SearchRequest(query="共同词", mode=SearchMode.fts, limit=10, offset=0))
)
assert first.page.total == 5010
assert len(first.items) == 10
last = asyncio.run(
engine.search(SearchRequest(query="共同词", mode=SearchMode.fts, limit=10, offset=5005))
)
assert last.page.total == 5010
assert len(last.items) == 5
# --------------------------------------------------------------------------- #
# 审阅回归:PATCH tags 语义 / 向量-块一致性 / 过滤漏召回 / rebuild 语义与回滚
# --------------------------------------------------------------------------- #
@@ -563,9 +640,10 @@ def test_rebuild_failure_restores_old_index(vault, monkeypatch) -> None:
assert repository.stats() == before # 旧索引已恢复,无半成品
def test_first_rebuild_failure_removes_partial_database(vault, monkeypatch) -> None:
def test_first_rebuild_failure_leaves_no_partial_index(vault, monkeypatch) -> None:
"""首次启动没有旧库时,失败也不能留下已经写入的部分索引。"""
from app.services import index_service
from app import repository
_write_vault(
vault,
@@ -574,17 +652,17 @@ def test_first_rebuild_failure_removes_partial_database(vault, monkeypatch) -> N
real_index = index_service.index_note
calls = {"count": 0}
async def fail_on_second(parsed):
async def fail_on_second(parsed, **kwargs):
calls["count"] += 1
if calls["count"] == 2:
raise RuntimeError("injected first-rebuild failure")
await real_index(parsed)
await real_index(parsed, **kwargs)
monkeypatch.setattr(index_service, "index_note", fail_on_second)
with pytest.raises(RuntimeError):
asyncio.run(index_service.rebuild(IndexRebuildRequest(scope="all")))
assert not get_settings().db_path.exists()
assert repository.stats() == {"notes": 0, "blocks": 0}
def test_rebuild_preserves_task_note_links(vault) -> None:
+466
View File
@@ -0,0 +1,466 @@
"""Phase E route integration: deterministic runtimes, isolated DBs, no network."""
from __future__ import annotations
import asyncio
import json
from dataclasses import dataclass, field
from types import SimpleNamespace
import pytest
from app import repository
from app.config import get_settings
from app.contracts import IndexRebuildRequest, SearchMode, SearchRequest
from app.database.db import connect, transaction
from app.retrieval import routed_vectors
from app.retrieval.embedding import HashEmbeddingProvider
from app.retrieval.engine import RetrievalEngine, engine
from app.retrieval.reranker import LexicalReranker
from app.retrieval.vectorstore import SqliteVecStore, VectorHit
from app.services import index_service, note_service
@dataclass
class FakeRuntime:
model_id: str = "space-a"
dimensions: int = 3 # Deliberately differs from sqlite-vec's fixed 128.
source: str = "api"
error: BaseException | None = None
calls: list[list[str]] = field(default_factory=list)
result_override: object | None = None
async def embed(self, texts):
self.calls.append(list(texts))
if self.error is not None:
raise self.error
if self.result_override is not None:
return self.result_override
vectors = []
for text in texts:
# The API associates "apple" with banana; hash retrieval picks apple.
first = text == "apple orchard"
if self.model_id == "space-b":
first = not first
vectors.append(([1.0, 0.0] if first else [0.0, 1.0]) + [0.0] * (self.dimensions - 2))
return SimpleNamespace(
vectors=vectors, source=self.source, model_id=self.model_id,
dimensions=self.dimensions, fallback_reason=None,
)
@pytest.fixture
def runtime(monkeypatch):
runtime = FakeRuntime()
monkeypatch.setattr(routed_vectors, "get_model_routing", lambda: runtime)
return runtime
async def seed():
apple = await note_service.create_note(
title="Apple", markdown="apple orchard", folder=None, tags=[],
)
banana = await note_service.create_note(
title="Banana", markdown="banana grove", folder=None, tags=[],
)
return apple, banana
@pytest.mark.parametrize("outcome", ["api", "api_failure", "missing_space"])
def test_benchmark_reports_actual_embedding_and_fallback(runtime, outcome):
from app.benchmarks import service
from app.contracts import RAGRunRequest
async def scenario():
apple, banana = await seed()
if outcome == "api_failure":
runtime.result_override = SimpleNamespace(source="local", fallback_reason="PROVIDER_TIMEOUT")
elif outcome == "missing_space":
runtime.model_id = "space-without-index"
directory = get_settings().benchmark_datasets_path
directory.mkdir(parents=True, exist_ok=True)
(directory / "routing.json").write_text(json.dumps({
"dataset_id": "routing", "kind": "rag", "version": "1",
"cases": [{"case_id": "query", "query": "apple", "expected_note_ids": [banana.note_id]}],
}), encoding="utf-8")
run = await service.create_rag_run(RAGRunRequest(
dataset_id="routing", modes=[SearchMode.fts, SearchMode.vector],
))
await service.wait_for_run(run.run_id)
report = service.get_report(run.run_id)
assert report.config_snapshot["embedding"]["policy"] == "per_case"
fts, vector = report.cases
assert fts.embedding == {"source": "not_used"}
if outcome == "api":
assert vector.embedding["source"] == "api"
assert vector.embedding["model_id"] == "space-a"
assert vector.embedding["dimensions"] == 3
assert vector.retrieved_note_ids[0] == banana.note_id
else:
assert vector.embedding["source"] == "local"
assert vector.embedding["model_id"] == "hash-v1"
assert vector.embedding["dimensions"] == 128
assert vector.retrieved_note_ids[0] == apple.note_id
if outcome == "api_failure":
assert vector.embedding["fallback_reason"] == "PROVIDER_TIMEOUT"
if outcome == "missing_space":
assert vector.embedding["fallback_reason"] == "REMOTE_INDEX_UNAVAILABLE"
assert vector.embedding["attempted_space"]["model_id"] == "space-without-index"
events = service.get_events(run.run_id)
case_events = [e for e in events if e.event.value == "CaseCompleted"]
assert case_events[-1].data["embedding"] == vector.embedding
asyncio.run(scenario())
def test_embedding_observations_are_isolated_between_concurrent_searches(runtime, monkeypatch):
from app.retrieval.provenance import capture_embedding
async def scenario():
await seed()
original = runtime.embed
async def embed(texts):
await asyncio.sleep(0)
if texts == ["offline"]:
raise RuntimeError("private upstream details")
return await original(texts)
monkeypatch.setattr(runtime, "embed", embed)
async def query(text):
with capture_embedding() as observation:
await engine.search(SearchRequest(query=text, mode=SearchMode.vector))
return observation
remote, local, another = await asyncio.gather(query("apple"), query("offline"), query("apple"))
assert remote["source"] == another["source"] == "api"
assert local["source"] == "local"
assert local["fallback_reason"] == "REMOTE_EMBEDDING_UNAVAILABLE"
assert "fallback_reason" not in remote or remote["fallback_reason"] is None
assert "private upstream" not in json.dumps(local)
asyncio.run(scenario())
@pytest.mark.parametrize("failure", ["cancel", "write"])
def test_rebuild_failure_preserves_concurrent_configuration_and_all_indexes(runtime, monkeypatch, failure):
from app.container import container
from app.contracts import ModelRoutingConfig, ProviderConfig, ProviderType
from app.services import task_service
async def scenario():
apple, _ = await seed()
task = task_service.create_task(title="before", note_id=apple.note_id)
before = {table: [tuple(row) for row in rows(f"SELECT * FROM {table}")]
for table in ("notes", "blocks", "blocks_fts", "vec_blocks", "index_meta", "routed_block_vectors")}
container.model_routing.update(ModelRoutingConfig())
entered, release = asyncio.Event(), asyncio.Event()
original_embed = runtime.embed
async def pending_embed(texts):
entered.set()
await release.wait()
return await original_embed(texts)
monkeypatch.setattr(runtime, "embed", pending_embed)
original_index = index_service.index_note
writes = 0
async def fail_write(parsed, **kwargs):
nonlocal writes
await original_index(parsed, **kwargs)
writes += 1
if writes == 2:
raise RuntimeError("injected write failure")
if failure == "write":
monkeypatch.setattr(index_service, "index_note", fail_write)
rebuilding = asyncio.create_task(index_service.rebuild(IndexRebuildRequest()))
await asyncio.wait_for(entered.wait(), timeout=5)
saved = container.model_routing.update(container.model_routing.configuration())
config = ProviderConfig(provider_id="concurrent", provider_type=ProviderType.openai_compatible,
name="saved during rebuild", base_url="https://unused.invalid/v1")
container.providers.register(config, container.provider_factory.build(config))
task_service.update_task(task.task_id, {"title": "saved during rebuild"})
# Preparation keeps the old searchable index intact while API I/O is pending.
assert repository.stats()["notes"] == 2
if failure == "cancel":
rebuilding.cancel()
expected = asyncio.CancelledError
else:
release.set()
expected = RuntimeError
with pytest.raises(expected):
await rebuilding
assert container.model_routing.configuration().version == saved.config.version
assert rows("SELECT provider_id FROM provider_configs")[-1][0] == "concurrent"
restored = task_service.get_task(task.task_id)
assert restored.title == "saved during rebuild"
assert restored.note_id == apple.note_id
for table, values in before.items():
assert [tuple(row) for row in rows(f"SELECT * FROM {table}")] == values
asyncio.run(scenario())
def local_engine():
return RetrievalEngine(HashEmbeddingProvider(), LexicalReranker(), SqliteVecStore())
def request(mode=SearchMode.vector):
return SearchRequest(query="apple", mode=mode, limit=10)
def rows(sql, parameters=()):
conn = connect()
try:
return conn.execute(sql, parameters).fetchall()
finally:
conn.close()
def test_api_index_and_query_use_matching_space_and_keep_local_metadata(runtime):
async def scenario():
apple, banana = await seed()
result = await engine.search(request())
assert result.items[0].note_id == banana.note_id
baseline = await local_engine().search(request())
assert baseline.items[0].note_id == apple.note_id
assert rows("SELECT DISTINCT space_id, dimensions FROM routed_block_vectors")[0][:] == ("space-a", 3)
assert rows("SELECT COUNT(*) FROM routed_block_vectors")[0][0] == len(apple.blocks) + len(banana.blocks)
meta = repository.get_index_meta()
assert meta["embedding_model"] == "hash-v1"
assert meta["embedding_dim"] == "128"
assert len(runtime.calls) == 3
asyncio.run(scenario())
@pytest.mark.parametrize("failure", ["exception", "local", "missing", "dimension", "corrupt"])
def test_query_falls_back_to_exact_local_results(runtime, failure):
async def scenario():
await seed()
if failure == "exception":
runtime.error = RuntimeError("offline")
elif failure == "local":
runtime.source = "local"
elif failure == "missing":
rows("DELETE FROM routed_block_vectors WHERE block_id = (SELECT MIN(block_id) FROM blocks)")
elif failure == "dimension":
runtime.dimensions = 4
else:
rows("UPDATE routed_block_vectors SET vector = ?", ("[0, 0, 0]",))
actual = await engine.search(request())
baseline = await local_engine().search(request())
assert actual == baseline
asyncio.run(scenario())
def test_same_dimension_model_switch_never_combines_partial_spaces(runtime):
async def scenario():
apple, banana = await seed()
baseline = await local_engine().search(request())
runtime.model_id = "space-b"
assert await engine.search(request()) == baseline
await note_service.update_note(apple.note_id, markdown="apple orchard")
assert {row[0] for row in rows("SELECT DISTINCT space_id FROM routed_block_vectors")} == {"space-a", "space-b"}
assert await routed_vectors.search_remote("apple", top_k=10) is None
assert await engine.search(request()) == baseline
runtime.model_id = "space-a"
assert await engine.search(request()) == baseline
runtime.model_id = "space-b"
await note_service.update_note(banana.note_id, markdown="banana grove")
hits = await routed_vectors.search_remote("apple", top_k=10)
assert hits is not None and hits[0].id == banana.blocks[0].block_id
assert (await engine.search(request())).items[0].note_id == banana.note_id
asyncio.run(scenario())
def test_complete_spaces_coexist_but_only_requested_space_is_ranked(runtime):
async def scenario():
apple, banana = await seed()
conn = connect()
try:
with transaction(conn):
routed_vectors.store_remote(
conn, [apple.blocks[0].block_id, banana.blocks[0].block_id],
routed_vectors.RemoteEmbeddings("space-b", 3, [[1, 0, 0], [0, 1, 0]]),
)
finally:
conn.close()
assert (await engine.search(request())).items[0].note_id == banana.note_id
runtime.model_id = "space-b"
result = await engine.search(request())
assert len(result.items) == 2
assert result.items[0].note_id == apple.note_id
asyncio.run(scenario())
def test_failed_note_embedding_preserves_save_and_forces_coverage_fallback(runtime):
async def scenario():
apple, banana = await seed()
runtime.error = RuntimeError("offline")
await note_service.update_note(banana.note_id, markdown="banana changed")
assert (await note_service.get_note(banana.note_id)).markdown == "banana changed"
assert rows("SELECT COUNT(*) FROM routed_block_vectors")[0][0] == len(apple.blocks)
runtime.error = None
assert await engine.search(request()) == await local_engine().search(request())
asyncio.run(scenario())
@pytest.mark.parametrize("vectors, dimensions, space", [
([], 3, "space-a"),
([[1, 0]], 3, "space-a"),
([[0, 0, 0]], 3, "space-a"),
([[float("nan"), 0, 0]], 3, "space-a"),
([[float("inf"), 0, 0]], 3, "space-a"),
([[True, 0, 0]], 3, "space-a"),
([[1, 0, 0]], 0, "space-a"),
([[1, 0, 0]], 3, "hash-v1"),
])
def test_invalid_remote_batch_does_not_break_note_saving(runtime, vectors, dimensions, space):
runtime.result_override = SimpleNamespace(
source="api", vectors=vectors, dimensions=dimensions, model_id=space,
)
async def scenario():
note = await note_service.create_note(title="Apple", markdown="apple orchard", folder=None, tags=[])
assert (await local_engine().search(request())).items[0].note_id == note.note_id
assert await routed_vectors.search_remote("apple", top_k=10) is None
asyncio.run(scenario())
def test_remote_storage_failure_rolls_back_batch_but_keeps_local_index(runtime):
async def scenario():
await seed()
rows("""CREATE TRIGGER reject_remote_vector BEFORE INSERT ON routed_block_vectors
WHEN (SELECT content FROM blocks WHERE block_id = NEW.block_id) = 'second'
BEGIN SELECT RAISE(ABORT, 'simulated storage failure'); END""")
note = await note_service.create_note(
title="Multi", markdown="first\n\nsecond", folder=None, tags=[],
)
assert len(note.blocks) == 2
assert rows(
"SELECT COUNT(*) FROM routed_block_vectors r JOIN blocks b USING(block_id) WHERE b.note_id = ?",
(note.note_id,),
)[0][0] == 0
assert rows("SELECT COUNT(*) FROM vec_blocks")[0][0] == rows("SELECT COUNT(*) FROM blocks")[0][0]
assert (get_settings().vault_path / note.file_path).exists()
asyncio.run(scenario())
def test_rebuild_and_delete_clear_old_remote_rows_through_foreign_keys(runtime):
async def scenario():
apple, _ = await seed()
await note_service.delete_note(apple.note_id)
assert rows("SELECT COUNT(*) FROM routed_block_vectors")[0][0] == 1
runtime.source = "local"
job = await index_service.rebuild(IndexRebuildRequest())
assert job.status == "completed"
assert rows("SELECT COUNT(*) FROM routed_block_vectors")[0][0] == 0
assert rows("SELECT COUNT(*) FROM vec_blocks")[0][0] == 1
runtime.source = "api"
runtime.model_id = "space-b"
await index_service.rebuild(IndexRebuildRequest())
assert [row[0] for row in rows("SELECT space_id FROM routed_block_vectors")] == ["space-b"]
asyncio.run(scenario())
@pytest.mark.parametrize("operation", ["save", "query", "rebuild"])
def test_cancellation_propagates_and_mutations_roll_back(runtime, operation):
async def scenario():
apple, _ = await seed()
before = [tuple(row) for row in rows("SELECT * FROM routed_block_vectors ORDER BY block_id")]
runtime.error = asyncio.CancelledError()
with pytest.raises(asyncio.CancelledError):
if operation == "query":
await engine.search(request())
elif operation == "rebuild":
await index_service.rebuild(IndexRebuildRequest())
else:
await note_service.update_note(apple.note_id, markdown="changed")
assert (await note_service.get_note(apple.note_id)).markdown == "apple orchard"
assert [tuple(row) for row in rows("SELECT * FROM routed_block_vectors ORDER BY block_id")] == before
asyncio.run(scenario())
@pytest.mark.parametrize("injected", ["embedding", "vector_store", "constructor"])
def test_injected_engine_dependencies_are_respected(runtime, monkeypatch, injected):
async def scenario():
apple, _ = await seed()
target = engine
if injected == "constructor":
target = local_engine()
elif injected == "embedding":
monkeypatch.setattr(engine, "embedding", HashEmbeddingProvider())
else:
class FakeStore:
async def search(self, vector, *, top_k):
assert len(vector) == 128
return [VectorHit(id=apple.blocks[0].block_id, score=1.0)]
monkeypatch.setattr(engine, "vector_store", FakeStore())
runtime.calls.clear()
assert (await target.search(request())).items[0].note_id == apple.note_id
assert runtime.calls == []
asyncio.run(scenario())
def test_fts_skips_routing_and_hybrid_uses_routed_vector_channel(runtime, monkeypatch):
async def scenario():
_, banana = await seed()
runtime.calls.clear()
await engine.search(request(SearchMode.fts))
assert runtime.calls == []
# Empty lexical channel isolates the vector contribution to hybrid fusion.
monkeypatch.setattr(repository, "fts_search", lambda *_: [])
class PreserveOrder:
async def rerank(self, query, candidates):
return sorted(candidates, key=lambda candidate: -candidate.score)
monkeypatch.setattr(engine, "reranker", PreserveOrder())
result = await engine.search(request(SearchMode.hybrid))
assert result.items[0].note_id == banana.note_id
assert runtime.calls == [["apple"]]
asyncio.run(scenario())
def test_arbitrary_dimensions_and_extreme_finite_values(runtime):
dimensions = 257
runtime.result_override = SimpleNamespace(
source="api", model_id="space-wide", dimensions=dimensions,
vectors=[[1e308, 1e308] + [0.0] * (dimensions - 2)],
)
async def scenario():
note = await note_service.create_note(title="Apple", markdown="apple orchard", folder=None, tags=[])
hits = await routed_vectors.search_remote("apple", top_k=1)
assert hits is not None and hits[0].id == note.blocks[0].block_id
assert hits[0].score == pytest.approx(1.0)
vector = json.loads(rows("SELECT vector FROM routed_block_vectors")[0][0])
assert len(vector) == dimensions
asyncio.run(scenario())
def test_missing_runtime_uses_unchanged_local_retrieval(runtime, monkeypatch):
monkeypatch.setattr(routed_vectors, "get_model_routing", lambda: None)
async def scenario():
await seed()
assert await engine.search(request()) == await local_engine().search(request())
assert runtime.calls == []
asyncio.run(scenario())
+65
View File
@@ -0,0 +1,65 @@
import pytest
from app.schema_security import (
ExternalSchemaReferenceError,
UnresolvableLocalSchemaReferenceError,
reject_external_schema_references,
)
@pytest.mark.parametrize(
"schema",
[
{"$ref": "file:///host/private-schema.json"},
{"properties": {"value": {"$ref": "https://schema.invalid/value.json"}}},
{"allOf": [{"$dynamicRef": "https://schema.invalid/dynamic"}]},
],
)
def test_external_json_schema_references_are_rejected(schema) -> None:
with pytest.raises(ExternalSchemaReferenceError):
reject_external_schema_references(schema)
def test_local_json_schema_fragment_reference_is_allowed() -> None:
reject_external_schema_references(
{
"$defs": {"value": {"type": "string"}},
"properties": {"value": {"$ref": "#/$defs/value"}},
}
)
@pytest.mark.parametrize("reference", ["#/$defs/missing", "#missing-anchor"])
def test_unresolvable_local_schema_reference_is_rejected(reference: str) -> None:
with pytest.raises(UnresolvableLocalSchemaReferenceError):
reject_external_schema_references({"type": "object", "$ref": reference})
def test_root_reference_cannot_use_anchor_from_nested_schema_resource() -> None:
schema = {
"$defs": {
"nested": {
"$id": "nested",
"$anchor": "inside",
"type": "string",
}
},
"properties": {"value": {"$ref": "#inside"}},
}
with pytest.raises(UnresolvableLocalSchemaReferenceError):
reject_external_schema_references(schema)
def test_nested_schema_resource_can_resolve_its_own_anchor() -> None:
schema = {
"$defs": {
"nested": {
"$id": "nested",
"$anchor": "inside",
"allOf": [{"$ref": "#inside"}],
}
}
}
reject_external_schema_references(schema)
+116
View File
@@ -0,0 +1,116 @@
import asyncio
import pytest
from app.config import get_settings
from app.contracts import (
FolderCreateRequest,
FolderDeleteRequest,
FolderRenameRequest,
NoteCreateRequest,
NoteRenameRequest,
WorkspaceOpenRequest,
)
from app.errors import ApiError
from app.routes import (
create_note,
create_workspace_folder,
delete_workspace_folder,
get_note,
get_workspace_tree,
open_workspace,
rename_note,
rename_workspace_folder,
)
def test_open_workspace_indexes_real_markdown_and_returns_tree() -> None:
vault = get_settings().vault_path
note_path = vault / "课程" / "操作系统.md"
note_path.parent.mkdir(parents=True)
note_path.write_text("# 操作系统\n\n进程调度。\n", encoding="utf-8")
snapshot = asyncio.run(open_workspace(WorkspaceOpenRequest()))
assert snapshot.workspace.path == str(vault.resolve())
assert snapshot.workspace.requires_refresh is False
assert snapshot.workspace.file_count == snapshot.workspace.indexed_note_count == 1
folder = snapshot.items[0]
assert folder.path == "/课程"
assert folder.children[0].path == "/课程/操作系统.md"
assert folder.children[0].note_id is not None
def test_open_workspace_rejects_unconfigured_path() -> None:
with pytest.raises(ApiError) as error:
asyncio.run(open_workspace(WorkspaceOpenRequest(path="C:/another-vault")))
assert error.value.code == "WORKSPACE_PATH_MISMATCH"
def test_note_rename_preserves_identity_and_content() -> None:
created = asyncio.run(
create_note(
NoteCreateRequest(
title="旧名称", markdown="# 标题不变\n\n真实正文。\n", folder="课程"
)
)
)
renamed = asyncio.run(
rename_note(created.note_id, NoteRenameRequest(file_name="新名称.md"))
)
assert renamed.note_id == created.note_id
assert renamed.file_path == "课程/新名称.md"
assert renamed.title == "新名称"
assert renamed.markdown == "# 标题不变\n\n真实正文。\n"
assert not (get_settings().vault_path / "课程" / "旧名称.md").exists()
def test_folder_lifecycle_updates_database_and_vectors() -> None:
folder = asyncio.run(
create_workspace_folder(FolderCreateRequest(parent="/", name="课程"))
)
created = asyncio.run(
create_note(
NoteCreateRequest(title="网络", markdown="# 网络\n\nTCP。\n", folder="课程")
)
)
renamed_folder = asyncio.run(
rename_workspace_folder(
FolderRenameRequest(path=folder.path, new_name="计算机课程")
)
)
moved_note = asyncio.run(get_note(created.note_id))
assert renamed_folder.path == "/计算机课程"
assert moved_note.note_id == created.note_id
assert moved_note.file_path == "计算机课程/网络.md"
assert asyncio.run(get_workspace_tree())[0].children[0].note_id == created.note_id
response = asyncio.run(
delete_workspace_folder(FolderDeleteRequest(path=renamed_folder.path))
)
assert response.status == "completed"
with pytest.raises(ApiError) as error:
asyncio.run(get_note(created.note_id))
assert error.value.code == "RESOURCE_NOT_FOUND"
assert asyncio.run(get_workspace_tree()) == []
def test_workspace_openapi_paths_are_published() -> None:
from app.main import app
paths = app.openapi()["paths"]
assert {
"/api/workspace",
"/api/workspace/open",
"/api/workspace/tree",
"/api/workspace/folders",
"/api/workspace/folders/rename",
"/api/workspace/folders/delete",
"/api/notes/{note_id}/rename",
} <= paths.keys()
+178
View File
@@ -51,6 +51,104 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/0b/a7/71ac2cff56fec219ed242bb11b8efb69fcc4bec75db06fb7bfe35de520e6/certifi-2026.7.22-py3-none-any.whl", hash = "sha256:62f22742b58a1a33014a2b6b706588a8d7e2a88ae7bd1a6ebe8c992928483775", size = 136983, upload-time = "2026-07-22T03:35:11.276Z" },
]
[[package]]
name = "cffi"
version = "2.1.1"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "pycparser", marker = "implementation_name != 'PyPy'" },
]
sdist = { url = "https://files.pythonhosted.org/packages/9e/ef/008a1939e372c06329a3fce4279c02f328488f3526744906eeec3da7ad5f/cffi-2.1.1.tar.gz", hash = "sha256:dd31f52ea1086513bb9df30f8fcee9b8918323ae067a3d5b78bc826a000712be", size = 530807, upload-time = "2026-08-03T21:21:18.939Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/70/d2/16d99a0c4948febc0ebd133a13b2f688ff7f8cb04da971e1128872ce0c03/cffi-2.1.1-cp311-cp311-macosx_10_15_x86_64.whl", hash = "sha256:c8d2c9fd1f2d16f780d15127abb050d13d1a76c03a4bd87d7e4980e45e511e12", size = 183838, upload-time = "2026-08-03T21:19:29.637Z" },
{ url = "https://files.pythonhosted.org/packages/cd/95/31b535a9f0220ae9f357de4a08d57ce89cb417653c2fd9f075f50822a388/cffi-2.1.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:398aff33cee2767e3e781d2554c54bd0dff386bb437581e0d8011fde1a942ec1", size = 184168, upload-time = "2026-08-03T21:19:30.764Z" },
{ url = "https://files.pythonhosted.org/packages/ad/5a/4707a0dc1f203f5dde5a907b0d4e3c25d71120241048bd5bc6f1bb9d4e71/cffi-2.1.1-cp311-cp311-manylinux1_i686.manylinux2014_i686.manylinux_2_17_i686.manylinux_2_5_i686.whl", hash = "sha256:154852545011f779917b11c78db2358d095da62a9a172b78ad0a583ee5adc0d0", size = 211805, upload-time = "2026-08-03T21:19:31.867Z" },
{ url = "https://files.pythonhosted.org/packages/ad/66/c19feabb28485b6e0bbaaafa90837a1ef5d302e90f2178bd33f17a49879b/cffi-2.1.1-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:3311ed60d36f83378794e1009ac6258bafbf81f7888b4caa7b35a521e3f95813", size = 218716, upload-time = "2026-08-03T21:19:32.896Z" },
{ url = "https://files.pythonhosted.org/packages/a7/92/500760486c8baab49a7a8a58ba7fc3355ec3974b454b8a09e528efde9e1d/cffi-2.1.1-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:6e192623c49c94421616a5778fba35cf0d5a8d000650c1967ef4448ee5cdd990", size = 205569, upload-time = "2026-08-03T21:19:34.142Z" },
{ url = "https://files.pythonhosted.org/packages/a5/a7/a67c733254d6e7373f7822f8082d8d6beade791e0cf12a7611f376fa61c7/cffi-2.1.1-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:a6e721d4b0e45d5b65e87534470e67b18dcd092c83f68fba09f152b9cbc061af", size = 204907, upload-time = "2026-08-03T21:19:35.174Z" },
{ url = "https://files.pythonhosted.org/packages/f7/a4/4399daaf8f7dfee9d7c3327fdb0426ee041cc63edc358b93911ceb2bfc7a/cffi-2.1.1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:34e261f78cb6ceaaa36f42f2613f4380d94d9c759a9c73c769ee6e0247364632", size = 217807, upload-time = "2026-08-03T21:19:36.286Z" },
{ url = "https://files.pythonhosted.org/packages/28/f7/dabe6da2466ecbd82dc62e7342dc6b1065dad990c06f00f0ede9ebf2a0ed/cffi-2.1.1-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:7225e4514edb64eb6740324353e0da0711954fd8d7da4576755b1c6e09b697cd", size = 221252, upload-time = "2026-08-03T21:19:37.416Z" },
{ url = "https://files.pythonhosted.org/packages/ce/87/616202d8e51342c07d2534c510111c4cc37201775ce8f60802c9335d1edd/cffi-2.1.1-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:df913725b79db7bcf03448f36b7bf8815363417d5b58deecf9305e3e30f0f21a", size = 214214, upload-time = "2026-08-03T21:19:38.507Z" },
{ url = "https://files.pythonhosted.org/packages/b4/c6/ab025d75d2c26c19b087c0124e75ee31cb65032f4fe345d356d8c507ab97/cffi-2.1.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:f5cfbc5fe74540d335175b656c725d74d90e3730c626d92575eea35029d9afaa", size = 219408, upload-time = "2026-08-03T21:19:39.809Z" },
{ url = "https://files.pythonhosted.org/packages/db/e2/7e8109f65445bdc673a7b54f02c677de462db75674220fd1335efc8eb598/cffi-2.1.1-cp311-cp311-win32.whl", hash = "sha256:f8ec5e643a9a937f64e1999eb9f75d072263751912dc5cd06d3c85f8f44be7c3", size = 174470, upload-time = "2026-08-03T21:19:41.246Z" },
{ url = "https://files.pythonhosted.org/packages/73/c0/77ba02423c2f7d7091143c45cd49e0e6575c4c1967394bb542bd923a9b74/cffi-2.1.1-cp311-cp311-win_amd64.whl", hash = "sha256:42f6930c31dc7f50732c9ae793c2786c7b6b044195967bbdde40bb9be81c4cc0", size = 185096, upload-time = "2026-08-03T21:19:42.615Z" },
{ url = "https://files.pythonhosted.org/packages/7c/47/9f1f85f9672ceda4984dc6c4f8824e8558992a2972c3d3c81fb8eb28d4ba/cffi-2.1.1-cp311-cp311-win_arm64.whl", hash = "sha256:c7659f22557c5a0bc4855cd635f55edec690cc008a40768527762cb9fb263455", size = 179941, upload-time = "2026-08-03T21:19:43.747Z" },
{ url = "https://files.pythonhosted.org/packages/10/69/43965eccfdead3b9220015fd1320e117be8c6ed01a62ffab76eeb752f5d5/cffi-2.1.1-cp312-cp312-macosx_10_15_x86_64.whl", hash = "sha256:c8c69575568085ba0b1b10c0249d779a214aea6f6522e949a0fc9fb0fcb449d0", size = 184821, upload-time = "2026-08-03T21:19:44.887Z" },
{ url = "https://files.pythonhosted.org/packages/54/7d/16e5a096677b5e313ca80cd5e5170efa3ea44624a82bb111925522da64b1/cffi-2.1.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:f81b3b8f3d4e343550fa4baa0e479bba9f2d29ce9c2e9b51d1ce1718d7442fcf", size = 184719, upload-time = "2026-08-03T21:19:46.129Z" },
{ url = "https://files.pythonhosted.org/packages/56/e6/8941622732edec876dd17d0453dce07317ae96db34f2ec1436c9d3785986/cffi-2.1.1-cp312-cp312-manylinux1_i686.manylinux2014_i686.manylinux_2_17_i686.manylinux_2_5_i686.whl", hash = "sha256:811bd1e21d32de12efca32393a0ab3f5133b54fce9bd44b8bd77ab07da14bf6a", size = 214799, upload-time = "2026-08-03T21:19:47.218Z" },
{ url = "https://files.pythonhosted.org/packages/44/de/f98430906df1545ffde0d543dd124a7a439bc2cd32b36b9c53f805df7333/cffi-2.1.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:68e62fe11f30d5ca8289242866f0a5291402d8529ca2178ab8afc5c9694ae890", size = 222389, upload-time = "2026-08-03T21:19:48.331Z" },
{ url = "https://files.pythonhosted.org/packages/6a/5b/717f1526b9957b34456313c31645c5b82b8fb5c3fe9e4752999be7128bfc/cffi-2.1.1-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:4a7c934f7360e8cd64fe9efadcbd10c7c6364f531e432b9a4bf5ccbc9e0e8b50", size = 210249, upload-time = "2026-08-03T21:19:49.543Z" },
{ url = "https://files.pythonhosted.org/packages/64/b3/f8aa4f3e34986c7e4ec45072d1b1b9dd295b6b18007b45518d79726dd725/cffi-2.1.1-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:3143d81e29e1e20a9ce10901ec369012947876596f75a222235965f2b7ae832e", size = 208775, upload-time = "2026-08-03T21:19:50.918Z" },
{ url = "https://files.pythonhosted.org/packages/b1/db/dceb9dd5b231e1da801793f8acc9f3c52a7e1afe40bb1aae37e02b0faad5/cffi-2.1.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:c1453022f490d2459a11819d83ad1d586e9ff65a12ac3e705ffebd46d3685dcf", size = 221822, upload-time = "2026-08-03T21:19:52.054Z" },
{ url = "https://files.pythonhosted.org/packages/a0/d2/6cd24ae3be000a634109c247d1475d62e5616d0dc78c82770942ec384248/cffi-2.1.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:208f941bb9d18e768138677f0a6d2ce01f590df56043dda1df1535ac57c88517", size = 225232, upload-time = "2026-08-03T21:19:53.109Z" },
{ url = "https://files.pythonhosted.org/packages/cb/52/3fa190537004dd7f0ab860a6dc7c0175b8667f68d1e618a46f5498d30250/cffi-2.1.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:210019b6c7cf07f081b4c54635c8cf744377001350e29cc0f81c4377b4797735", size = 223597, upload-time = "2026-08-03T21:19:54.515Z" },
{ url = "https://files.pythonhosted.org/packages/80/fb/0bb75b7039588c074b37ae99f40d9bfddf990ecb2fbc346ebccd2e56b9be/cffi-2.1.1-cp312-cp312-win32.whl", hash = "sha256:046bfc24911b37851ee1b51aab8bffe713d89c68c6a057b09484ce9fd5f69b4e", size = 175292, upload-time = "2026-08-03T21:19:55.566Z" },
{ url = "https://files.pythonhosted.org/packages/d9/79/615cc094e2fb508cade7de88d3b4f6c4ec2bab695c97bce9153dc65aadf5/cffi-2.1.1-cp312-cp312-win_amd64.whl", hash = "sha256:f53e442b08449d42821fa4a4fba000095af9f62742a500f978a9f557ec44339a", size = 185919, upload-time = "2026-08-03T21:19:56.89Z" },
{ url = "https://files.pythonhosted.org/packages/70/c6/d0ea84713fe46b243a436a18fcd47d639732747e21635c8a27191b06dc30/cffi-2.1.1-cp312-cp312-win_arm64.whl", hash = "sha256:7bde5e4cc5c10140859842b9d383af292b22639a4dffb725314baf45968cef80", size = 180093, upload-time = "2026-08-03T21:19:58.155Z" },
{ url = "https://files.pythonhosted.org/packages/9d/f4/035513d4117049066b4779dc3b7c0c0fdad175fa13731c9f4003f1cd1478/cffi-2.1.1-cp313-cp313-ios_13_0_arm64_iphoneos.whl", hash = "sha256:b5bdfd1c873d4e093aabc0ca84c4ca6dbc4f752afb5c86f146d9742580c9da2e", size = 194248, upload-time = "2026-08-03T21:19:59.399Z" },
{ url = "https://files.pythonhosted.org/packages/76/af/2aeb4dbb5fc41a04161ae9ff1518de7cec08e164f44a8ce6a4cf7fd2cd1d/cffi-2.1.1-cp313-cp313-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:31348097ff5bbe827ccc41795d4dd099d9f0625e7def00ee653c137a490c2a6c", size = 196908, upload-time = "2026-08-03T21:20:00.746Z" },
{ url = "https://files.pythonhosted.org/packages/a7/46/2e5fdde8555706dd98139a910ca11be02809f3f605ce956f655d0214e100/cffi-2.1.1-cp313-cp313-macosx_10_15_x86_64.whl", hash = "sha256:9d2055050ea716bd38b7f7f1579c275386646b4894c155a3e2f3cd62ed41b7c6", size = 184805, upload-time = "2026-08-03T21:20:02.02Z" },
{ url = "https://files.pythonhosted.org/packages/55/41/4c7042f317b9217502988f0873af87e16ad606dc20f84e546e3e6ce9764c/cffi-2.1.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:19ee6127ee34de7d83ce3d371ebc5ed91addbdcc39f9ab15ce4eb35a4e534971", size = 184764, upload-time = "2026-08-03T21:20:03.141Z" },
{ url = "https://files.pythonhosted.org/packages/43/1f/1c3d90d91811c8f86ced9ed637956c54bfe5b79ca98fe976d7f8c8979f6b/cffi-2.1.1-cp313-cp313-manylinux1_i686.manylinux2014_i686.manylinux_2_17_i686.manylinux_2_5_i686.whl", hash = "sha256:6a8dddef476fab96d066d578fc88526767b836ab5ab21754e1d5bf3879c31c7c", size = 214722, upload-time = "2026-08-03T21:20:04.377Z" },
{ url = "https://files.pythonhosted.org/packages/37/6f/3b5ce4c3b2192d250f04908f2bfd91ef34552ec8f7716a5d4abdb8d67bb2/cffi-2.1.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:f16c709686a78c727bbbf059f92b0bf41c6fc60deec706d2dc19f529175a6125", size = 222369, upload-time = "2026-08-03T21:20:05.544Z" },
{ url = "https://files.pythonhosted.org/packages/02/10/4b3c75dde3d9663c9e02ba05c2668b954f671d4bbe346413ca8c696b295a/cffi-2.1.1-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:fcd22650c908d7b7da162bbfaab594a1227a15d1643a98c68b122ac642fa2264", size = 210175, upload-time = "2026-08-03T21:20:06.75Z" },
{ url = "https://files.pythonhosted.org/packages/df/62/14f74b9543e605d17701dc797b815958b8bb70b7624ce1b832ddad48ed6c/cffi-2.1.1-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:aa9511c62d14da7aacc9b4bf51f3f697a621e83b2d6919008243c3aad168eea3", size = 208670, upload-time = "2026-08-03T21:20:08.04Z" },
{ url = "https://files.pythonhosted.org/packages/95/95/86342356ff5953b3fb06f7ef7c5bee212d45e770abc7218d451b9148313c/cffi-2.1.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:a931079504ecc49efed7744c476a5c343a92fabf66dec2db95edb1b2fdc770e2", size = 221824, upload-time = "2026-08-03T21:20:09.274Z" },
{ url = "https://files.pythonhosted.org/packages/eb/ff/7b3429ff53aafe931ed8a5fc69f481bbef7ba6de87ddcbb63d08f483f613/cffi-2.1.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:a2d7755bef5a12ed488f4ef1f1b69ee9191d7396083b755a5d2295f6edb4768b", size = 225148, upload-time = "2026-08-03T21:20:10.7Z" },
{ url = "https://files.pythonhosted.org/packages/34/34/a95870b9221e09cf4f2ce3178b1a210abdfe63a1bd357da940418d7b8d15/cffi-2.1.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:e0bcb7e0f677f543555d2adff3bf19c05f66cdb4796e5ff602442ab2fe3c4ef7", size = 223564, upload-time = "2026-08-03T21:20:12.165Z" },
{ url = "https://files.pythonhosted.org/packages/70/ea/839b50531021a647fb5e929f72cf97bc1ff702b5472166164b5b6e76b851/cffi-2.1.1-cp313-cp313-win32.whl", hash = "sha256:334644fbac4eff73d985a17a91226df55d0f394160c4cfb880e084c8f7161cac", size = 175263, upload-time = "2026-08-03T21:20:13.559Z" },
{ url = "https://files.pythonhosted.org/packages/60/a6/8b149b2c3f2e11aaa1618ef64500b45f50f22c57a977a4dff1aff1f91042/cffi-2.1.1-cp313-cp313-win_amd64.whl", hash = "sha256:1aa5645c30469b09530c4ebca77ebf8f17618293c58f8549cb1a543a50236e7d", size = 185688, upload-time = "2026-08-03T21:20:14.69Z" },
{ url = "https://files.pythonhosted.org/packages/01/9a/11f687cb39d6a3504060d5242f04f48c735afb4d3d533958a20594890cb2/cffi-2.1.1-cp313-cp313-win_arm64.whl", hash = "sha256:63bbfd5ded17c4840ac07cd8f1c21ba9d9708141f840b324f422f41b207e3973", size = 180078, upload-time = "2026-08-03T21:20:15.917Z" },
{ url = "https://files.pythonhosted.org/packages/d3/7b/d6bbf82b8b96e7391438898c42f5bd96dd02030fd5b64937d248220003e2/cffi-2.1.1-cp314-cp314-ios_13_0_arm64_iphoneos.whl", hash = "sha256:7dbb61fe3a7699468030f71bbe5f8a0e326a151daa91beb11a6fc1f980c55e1c", size = 194064, upload-time = "2026-08-03T21:20:17.148Z" },
{ url = "https://files.pythonhosted.org/packages/94/e6/bcc91b283be94735e268487a054004f0aa19947b6348fa367db53230abc8/cffi-2.1.1-cp314-cp314-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:f24fb43132a4c6b4cb4eb029492919b2db645be6808d738f244fd146c03c32cb", size = 196720, upload-time = "2026-08-03T21:20:18.268Z" },
{ url = "https://files.pythonhosted.org/packages/d9/99/c4b0c17cacdc9c3b8f280026286a9826d6a208c0f047591a3c3ce99b91fd/cffi-2.1.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:d28630f5854ab07ab1fd4aba756de52326c82e6be15d414b12793f1975048b54", size = 184964, upload-time = "2026-08-03T21:20:19.708Z" },
{ url = "https://files.pythonhosted.org/packages/b3/a9/9db617d05d7367c1ad0ab00b3aa6e6f9281edd689b4ee9ea0e5a84e89c97/cffi-2.1.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:661c298b4821edebead0c91edd2b00374d67ad7c5a1f7a91d4442633b79d6a72", size = 184962, upload-time = "2026-08-03T21:20:20.833Z" },
{ url = "https://files.pythonhosted.org/packages/67/b8/b42132ca113dc567d37684437b46ca1dafc885902b02a110a02d5b511857/cffi-2.1.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:58acb8ab8e295e6c5ea12f888cbb13cf21511ef2a3303a23f4325c29d17fe5c1", size = 222328, upload-time = "2026-08-03T21:20:22.118Z" },
{ url = "https://files.pythonhosted.org/packages/80/10/c5c0cbf0a657aecf59ef511409734230bf556f05a0d6c9eed7aa5c0a0166/cffi-2.1.1-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:456a61fa52d579ebf9df2e9552ead5129855dbaff6c1e5a9b1bc408809bdc062", size = 209985, upload-time = "2026-08-03T21:20:23.401Z" },
{ url = "https://files.pythonhosted.org/packages/d5/6c/bfa0b87b03b9238148beca990292843c9396ba069b54496596594173de7b/cffi-2.1.1-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:a4f00aa42f75d6e4595e8866e748cc1705adc0cddfeb2ca86d0d03993d63ba03", size = 208530, upload-time = "2026-08-03T21:20:24.628Z" },
{ url = "https://files.pythonhosted.org/packages/e9/02/4e7d553a7ac4b4238b38b3c1b80d486e9d4436f8d2acbf87a0997fe3f402/cffi-2.1.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:b0431303acaea1089ad4b3e9ce4e6518193def1118d4073ca848635ee4ea2e96", size = 221525, upload-time = "2026-08-03T21:20:25.758Z" },
{ url = "https://files.pythonhosted.org/packages/82/1d/a4aaf9babd75acb4d5f223bff71533bee748dd770a382619a798960ee9ba/cffi-2.1.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:64faea20f4e2613363a1a9b9c7dd73058f3ecd00133a511e72ad7c511658f527", size = 225053, upload-time = "2026-08-03T21:20:26.985Z" },
{ url = "https://files.pythonhosted.org/packages/81/10/5dc0e7bdd18e22107054288283380fc97a06ae3f1656a106908d666a3c88/cffi-2.1.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:5c58fe613dc5e5336357eff555824a314d8e43282600435c8d1cb6a7a2fedd13", size = 223213, upload-time = "2026-08-03T21:20:28.277Z" },
{ url = "https://files.pythonhosted.org/packages/0b/e9/d0061c364cde06ee43168a0d076ac1da512cbc380d44767b844ba34fe2b6/cffi-2.1.1-cp314-cp314-win32.whl", hash = "sha256:1a18a57b58cfb21fc28d72e876acf10eaed67a1ed96226f92af4df681d571c4c", size = 177682, upload-time = "2026-08-03T21:20:44.288Z" },
{ url = "https://files.pythonhosted.org/packages/a7/06/1c3e01e3ba14c39f6d10bfbac52753b7e22259e38088e5cfe1d704918690/cffi-2.1.1-cp314-cp314-win_amd64.whl", hash = "sha256:3222ba5d678f80a030e6afbcc33dc1ae5cb45facabb61cee2c7016b8432fde48", size = 187949, upload-time = "2026-08-03T21:20:45.623Z" },
{ url = "https://files.pythonhosted.org/packages/87/5b/da4e39efe18eeb89cf580ea9cfc66b6a7c3eadb808fc0cc1d3a295cb5a5d/cffi-2.1.1-cp314-cp314-win_arm64.whl", hash = "sha256:ab36d55f9ed2d067327667c2fea18dda018eb628dd6347aa01dda6cf1f5d3836", size = 182947, upload-time = "2026-08-03T21:20:46.955Z" },
{ url = "https://files.pythonhosted.org/packages/23/59/40338bf421c5accea1d45158170c87006ef1cd371b05c077e76476949728/cffi-2.1.1-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:7750c6449dff7864bb9bb27ddfb0267756189201a3afc911d82b3caacd70dfc3", size = 188504, upload-time = "2026-08-03T21:20:29.495Z" },
{ url = "https://files.pythonhosted.org/packages/7d/47/5ecf1023850036e674c77ec4de86182d309ae344e39e7cba984b7df5d647/cffi-2.1.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:0beceaabe56af686895136a2de78db54ecd8e4046b236b8fd6d6cb61389e9bf2", size = 188259, upload-time = "2026-08-03T21:20:31.291Z" },
{ url = "https://files.pythonhosted.org/packages/2a/9c/92934c3bea9f785b23eba304538c0b4d37a2a96d2431eb3a1bc87a11aa19/cffi-2.1.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:49cbc70e6542d4ccccb936558d1064a8012541e78f821f955cff24e357776c94", size = 223864, upload-time = "2026-08-03T21:20:32.571Z" },
{ url = "https://files.pythonhosted.org/packages/4d/45/ba4c93527bc38616a8bd36488acb69a2212d60486794f0c1f318949bbb76/cffi-2.1.1-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:e2d65b31f36619cda3999b78b2aa9632e76b78448e7a56fc4240824200e7c4fc", size = 211538, upload-time = "2026-08-03T21:20:33.808Z" },
{ url = "https://files.pythonhosted.org/packages/80/e9/b6ef565e452acb932fb0cb5443f44a78efbd1233e566f02b5a83855e9115/cffi-2.1.1-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:28907ab9bfb6aa13184cfc17c6b8e1023c5ab6fd7076d8c20a35e59fe04f8f29", size = 210688, upload-time = "2026-08-03T21:20:34.974Z" },
{ url = "https://files.pythonhosted.org/packages/9a/95/eff5f0cee78d2eabc7eebffec40d3fc1876b5f3c95582e018bb4b99601f2/cffi-2.1.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:51b31d1c98274844cfd7838ce00bfc27c7423a4dc00fc0772fc3331c2cc90676", size = 223803, upload-time = "2026-08-03T21:20:36.564Z" },
{ url = "https://files.pythonhosted.org/packages/fa/01/579d39fb8bef00a335a23d83757b44feb24cd6345a2c451b64cb67b9c362/cffi-2.1.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:5e7cecbaadb83884793e05828cee59b210b24583b9c7425d0ba6a754fe22eb4e", size = 226763, upload-time = "2026-08-03T21:20:37.816Z" },
{ url = "https://files.pythonhosted.org/packages/8d/b0/0b44f47c60b01b57b6e2bbd92343f13a85a1d93bc46ccf6e47e244acd99c/cffi-2.1.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:25792eac27877609e7bb06d42ff88278a6624fff2ba9bbb523c09616b117e80f", size = 225688, upload-time = "2026-08-03T21:20:38.959Z" },
{ url = "https://files.pythonhosted.org/packages/eb/d2/3b7176cb570a1d3e27faf67b72f591af508036e0d8b2be2ef9af9e8c84bb/cffi-2.1.1-cp314-cp314t-win32.whl", hash = "sha256:8ef53b2de9bcb9197d31854256575d59dbac0cba72ac627bb291ef5eceb74be4", size = 182868, upload-time = "2026-08-03T21:20:40.388Z" },
{ url = "https://files.pythonhosted.org/packages/56/78/31f00c1bcd97c9bbf55f1bfdf5bc809a5de8887473e90bb9960dca825e80/cffi-2.1.1-cp314-cp314t-win_amd64.whl", hash = "sha256:616f097f2fe415bc92a247f02e11f634e1f9e9a83d327e3c915c15089c87869e", size = 194104, upload-time = "2026-08-03T21:20:41.725Z" },
{ url = "https://files.pythonhosted.org/packages/7b/1b/58496f2ed0a35de575250c02a43ab3cc2c04d494a88fed31c1cabc0fd176/cffi-2.1.1-cp314-cp314t-win_arm64.whl", hash = "sha256:ad2c86c495b899d862ea0f4b42891b8713a3bd45dd4105c7fd51c2a72f39f3a5", size = 186402, upload-time = "2026-08-03T21:20:43.042Z" },
{ url = "https://files.pythonhosted.org/packages/c1/8f/9ebe220eab48a093d1a5a5e339ab0dc7316eef3bb04d63c42f0251b61f50/cffi-2.1.1-cp315-cp315-ios_13_0_arm64_iphoneos.whl", hash = "sha256:dddad92b554513a31f272570678ba307fb9f618f05e3d4a5eacafff9eae03e1d", size = 194043, upload-time = "2026-08-03T21:20:48.179Z" },
{ url = "https://files.pythonhosted.org/packages/ff/69/844bad3ece306c4782c2ecb93597035b6690d48704b803914c199da1e8b3/cffi-2.1.1-cp315-cp315-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:da0e573f9f97159390c89d9f1a9e41908b66d408cc5b58d08cf3847d844c531b", size = 196737, upload-time = "2026-08-03T21:20:49.457Z" },
{ url = "https://files.pythonhosted.org/packages/1b/8a/af668013284634733f02d683458a0728739c7d6ddb5e14cb0c20832266fe/cffi-2.1.1-cp315-cp315-macosx_10_15_x86_64.whl", hash = "sha256:fb92203a88b3d3053034db775110081c49d28be6551923805e039924093761e4", size = 184933, upload-time = "2026-08-03T21:20:50.639Z" },
{ url = "https://files.pythonhosted.org/packages/0c/75/2f5207ff6d1a613133b23a5203cc0c2a628313b5eb3974d7956ae3c57950/cffi-2.1.1-cp315-cp315-macosx_11_0_arm64.whl", hash = "sha256:2ae64be792b8966f2c69538199728b290e34726562896df1e5dc8ffd8d8188e8", size = 185002, upload-time = "2026-08-03T21:20:52.173Z" },
{ url = "https://files.pythonhosted.org/packages/e2/31/9e1313b0a6e30e91b3b3d3fff51ae99c857c07738e3afcce1f7334e1b7ab/cffi-2.1.1-cp315-cp315-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:507a24c282e0f42f8ed737cf048572cbf580468da5555764a8331735e9c736b6", size = 222271, upload-time = "2026-08-03T21:20:53.462Z" },
{ url = "https://files.pythonhosted.org/packages/50/e3/f6234a833e6e08c7007003074723c406559eecf9b48dfc97471e5a8eb7a0/cffi-2.1.1-cp315-cp315-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:246fa40ce8645a614ff682e0b70f37134e460eaf93a775e0cbe3cca585a67a80", size = 209919, upload-time = "2026-08-03T21:20:54.783Z" },
{ url = "https://files.pythonhosted.org/packages/0d/fc/5f74e293fced6edb51af3a46c4ccf6c23c9943774ecb375ddbd522c76add/cffi-2.1.1-cp315-cp315-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:471cee653ae88de62096552e6d24ccb4a5adb8c8c9f10b5054d0122c15bf2779", size = 208529, upload-time = "2026-08-03T21:20:56.066Z" },
{ url = "https://files.pythonhosted.org/packages/44/16/29e6d01b388bef055ecd6ca8244b3f4d336bd09e92d5d892187b9601084e/cffi-2.1.1-cp315-cp315-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:aeae0e330c9f6acd681f647d46cefd30c29f93e3392882e792e82080c9691399", size = 221630, upload-time = "2026-08-03T21:20:57.336Z" },
{ url = "https://files.pythonhosted.org/packages/a4/18/fa7f1f6857d5eb88a4ca99ffcbfb7c387a287ccc154c64a73e86314745d7/cffi-2.1.1-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:42a494cee34437f05546455144f2b5d9ac09b1face62bcfce597d2e521066688", size = 225134, upload-time = "2026-08-03T21:20:58.675Z" },
{ url = "https://files.pythonhosted.org/packages/e0/9f/e8e3dfa04a1b4c241f8c91faacad872b4d4efd051d49764ad4e2fd4b9fea/cffi-2.1.1-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:cc572dace3f60ef98d7b12ff411d20f5362feb31a0439eab0085bbfd349982d7", size = 223197, upload-time = "2026-08-03T21:20:59.968Z" },
{ url = "https://files.pythonhosted.org/packages/f8/7e/8debeb04f1ab9fe2a6963964cd6f1aaf7192627b83926586a6a4e089c9fa/cffi-2.1.1-cp315-cp315-win32.whl", hash = "sha256:4f42141fc14250de6dde5ee7ea4432be017252d91f19c5ad043c084cea629cac", size = 177683, upload-time = "2026-08-03T21:21:14.901Z" },
{ url = "https://files.pythonhosted.org/packages/e0/31/5158704cc474ab65c1647932e88be78dc0873f47130e253be38bcaf13d01/cffi-2.1.1-cp315-cp315-win_amd64.whl", hash = "sha256:e6e8cff14d6fb0be70a09c0bdc58096f501952d04624ebf867e0e56da2df8960", size = 187897, upload-time = "2026-08-03T21:21:16.108Z" },
{ url = "https://files.pythonhosted.org/packages/cc/4b/b3a2da8570c704ffc0f9762cdc3ec0f02c8573798e0b5cf7f11c82bbb70f/cffi-2.1.1-cp315-cp315-win_arm64.whl", hash = "sha256:27350daa11d4f10c540e6e89dada4c54feb7256ad03e9a4dc075ebad7ba360d1", size = 182935, upload-time = "2026-08-03T21:21:17.271Z" },
{ url = "https://files.pythonhosted.org/packages/d0/ef/5443574510a1207e6f6bc38ba6e1f1de36cb48fef07b2728bb896a21f430/cffi-2.1.1-cp315-cp315t-macosx_10_15_x86_64.whl", hash = "sha256:c26608d2222fb1e94487e4a387d85f13eb55d5ed725cb25a0c589ac4ee60e7bc", size = 188464, upload-time = "2026-08-03T21:21:01.163Z" },
{ url = "https://files.pythonhosted.org/packages/7e/ae/a56fa8c4686ad50e148fcbc8d3ae0d03915ff5c30d795058988c24118cef/cffi-2.1.1-cp315-cp315t-macosx_11_0_arm64.whl", hash = "sha256:4be96343e422f2dfcd12ab5c9f5aebe03f82f737c6bffeca6830b3875cb44aab", size = 188262, upload-time = "2026-08-03T21:21:02.382Z" },
{ url = "https://files.pythonhosted.org/packages/53/b2/6187f46f2912276a3ae284076109cc5c8680482f11f766ccf26db4a86427/cffi-2.1.1-cp315-cp315t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:937c0052c05a31ca1daf18de3158eed4dbfcb9cc107adbea227728d647be701e", size = 223779, upload-time = "2026-08-03T21:21:03.553Z" },
{ url = "https://files.pythonhosted.org/packages/8a/f6/c3ad28bd19f77047a03084424fbd4cbe997303267c14423737324be0385d/cffi-2.1.1-cp315-cp315t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:df423d40ee8654634421812bc3b196da3f9bd7d32929da813f8394c4348a5358", size = 211520, upload-time = "2026-08-03T21:21:04.863Z" },
{ url = "https://files.pythonhosted.org/packages/a0/cd/ccac9013a5bd9fd764de118674ab9c805b5ca10c19270d90ee273f8b2240/cffi-2.1.1-cp315-cp315t-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:a730a083190634c65cca36ba5f489531576ebd79bcd5c8e172130f6453127231", size = 210673, upload-time = "2026-08-03T21:21:06.223Z" },
{ url = "https://files.pythonhosted.org/packages/52/86/2976131c639aead931c5bee5aba67e4b09fbeb8018b6f282f70803f923a7/cffi-2.1.1-cp315-cp315t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:363e05fa78e15116c3c32c210ee36884fd6b9afa6d440e47112c3bd511d64cb6", size = 223835, upload-time = "2026-08-03T21:21:07.539Z" },
{ url = "https://files.pythonhosted.org/packages/ac/0c/33a7aeab2f9c76918c52e084beb39c570db3588133412929e8ec06fab90b/cffi-2.1.1-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:770de9db11e84213beec501cfcaa013b019820ca881e03344dea5844f7876d94", size = 226705, upload-time = "2026-08-03T21:21:08.774Z" },
{ url = "https://files.pythonhosted.org/packages/e3/26/2cde30fdde421130bfc18f70395731a6e6b2053c6a1978a5258ff04e72fa/cffi-2.1.1-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:7da0c5eff80f0197f3b3d1232ec5a682a9325f4ae9016a78f5f5ca35f9ced1f5", size = 225539, upload-time = "2026-08-03T21:21:09.911Z" },
{ url = "https://files.pythonhosted.org/packages/6d/cd/a361394c94b2129d604bb846f624a8e88255a3ee33129c434a00d715e64f/cffi-2.1.1-cp315-cp315t-win32.whl", hash = "sha256:06c72bb76605a4b0cd0aad6930b69d4baf7dd5d806cfc409b824191099700e66", size = 182707, upload-time = "2026-08-03T21:21:11.226Z" },
{ url = "https://files.pythonhosted.org/packages/9b/b5/ba2b299993c26577d529b6ae29841f9e15b9fcf004d65f423f4fcf94ade9/cffi-2.1.1-cp315-cp315t-win_amd64.whl", hash = "sha256:d9c275eaacd24aa73f94ffd6de08fc3f932424d8b6c376f4bed7cde376fe7bc3", size = 193772, upload-time = "2026-08-03T21:21:12.39Z" },
{ url = "https://files.pythonhosted.org/packages/aa/29/35e016098c814cd93de9cd320c66b5bfba14dc6ecedd3cb518fa7c408c69/cffi-2.1.1-cp315-cp315t-win_arm64.whl", hash = "sha256:d18e5ac0f2f03f4f518d3e23db0f0cad7faa1da8620e9c09461d443bbf6e6692", size = 186360, upload-time = "2026-08-03T21:21:13.636Z" },
]
[[package]]
name = "click"
version = "8.5.0"
@@ -69,6 +167,62 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335, upload-time = "2022-10-25T02:36:20.889Z" },
]
[[package]]
name = "cryptography"
version = "50.0.1"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "cffi", marker = "platform_python_implementation != 'PyPy'" },
]
sdist = { url = "https://files.pythonhosted.org/packages/bb/ad/5d6702db60b1e40b41ef513b6967ff5848f307d50f8449baf1634f5908f1/cryptography-50.0.1.tar.gz", hash = "sha256:5dd9bda1c12b4162f6ff568eeb5e0ff956c28d14406e875cfe8a63a2d414ff20", size = 880381, upload-time = "2026-08-25T19:45:45.499Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/ba/19/797e2aaac9df6a66f1550f49979dc1b1e39ecd2077501c30efa81e8d5d67/cryptography-50.0.1-cp311-abi3-macosx_11_0_arm64.whl", hash = "sha256:b8f852c65863251b9e3a1b8c150ce21e59b522dbb6a7d4bc80e680d38388e986", size = 4010153, upload-time = "2026-08-25T19:44:03.155Z" },
{ url = "https://files.pythonhosted.org/packages/90/34/9ce9a62ed9dc82ca9fd6a34445b6904af56e5f38b3eae2ed32e49c36053d/cryptography-50.0.1-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:53e279950892dc102c6b4e52af03ae5ea92fac572a1ddab78ca73a997f62b69f", size = 4723133, upload-time = "2026-08-25T19:44:05.461Z" },
{ url = "https://files.pythonhosted.org/packages/57/26/e6d4fc8512a51a5f9ee7bfdbfb853bce1197087df40c9ad993ad370b846f/cryptography-50.0.1-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:ff838d62ec1bfce4f9ba7fa16f4a7b554cd8d0c299e6be37502161a660c84eef", size = 4712478, upload-time = "2026-08-25T19:44:07.375Z" },
{ url = "https://files.pythonhosted.org/packages/e6/de/d3cdc2815697aae84126cbd6a030ca7b6b452e28a88b501b836bd3aa7a86/cryptography-50.0.1-cp311-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:e74591e283fe6eb956416c929eb58262a719fe0311fd9054c62c3350ed8760d8", size = 4730726, upload-time = "2026-08-25T19:44:09.294Z" },
{ url = "https://files.pythonhosted.org/packages/55/32/38c0d344b98c06d34b5df8946565a9c0d6dbf32c8e0730a7f05f0a3c6cab/cryptography-50.0.1-cp311-abi3-manylinux_2_28_ppc64le.whl", hash = "sha256:5fe002589592ed749ce77fe0695fcbd3500dd61d7d6db5858a7544c612fa8e45", size = 5353524, upload-time = "2026-08-25T19:44:11.96Z" },
{ url = "https://files.pythonhosted.org/packages/e1/1b/82f0f0d8858d4432be1af790477edf62aef90324041aa07c57e57bef1af7/cryptography-50.0.1-cp311-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:51593d180cf6d179bde5c5d065bed81386b1f381656ae7d042b7ffc87a9895ad", size = 4746720, upload-time = "2026-08-25T19:44:14.051Z" },
{ url = "https://files.pythonhosted.org/packages/29/ba/042ca458b8c64348c768284b5d23e69b92ed53d057ab779fee628564676d/cryptography-50.0.1-cp311-abi3-manylinux_2_31_armv7l.whl", hash = "sha256:359e62deae718bce96170e223fdcb6357e4fbd3bb7a3a75f4430763532560e49", size = 4361866, upload-time = "2026-08-25T19:44:16.167Z" },
{ url = "https://files.pythonhosted.org/packages/39/3b/e96c1ef71edef71057c7e3c3d982ce8fda554e0c52d0cc19c18845cde3eb/cryptography-50.0.1-cp311-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:e2ca8fd1b6b4b82a1c4cb02841d0837e3c12336c2e24b520ab8ab3b969733d8f", size = 4730028, upload-time = "2026-08-25T19:44:18.085Z" },
{ url = "https://files.pythonhosted.org/packages/e3/38/45abd72ef63f2e7d0754a6cacf97bd8b69512ace7f6130d24c39ece65da2/cryptography-50.0.1-cp311-abi3-manylinux_2_34_ppc64le.whl", hash = "sha256:76de83fbd91ac49c0feaaa983d0748fd7a53176afac5fb3bf7478d244f0eb527", size = 5308405, upload-time = "2026-08-25T19:44:20.197Z" },
{ url = "https://files.pythonhosted.org/packages/85/66/6ccca4722987ddedaa7fc9c3f4708af7431f5535666c174350830888c6b7/cryptography-50.0.1-cp311-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:51afcfceb15597cf2635068e4ac9a56b2abde622edde17f37d85fd7b5306497a", size = 4746230, upload-time = "2026-08-25T19:44:22.376Z" },
{ url = "https://files.pythonhosted.org/packages/13/0e/b1f92e013228111413f2e6743948b80bc24dfd3c1b87ba98ceea16f5df89/cryptography-50.0.1-cp311-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:be224a65493ec5b74a158ff22a5522ce4a5ca1e543c647a3a4730d4a09e5f959", size = 4862596, upload-time = "2026-08-25T19:44:24.472Z" },
{ url = "https://files.pythonhosted.org/packages/7e/22/c3654cccc856e9d682817b04ac3ee79731cb09ca6f95996a95c904de2883/cryptography-50.0.1-cp311-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:9ebcdd5519be9b652a46f507817a74591774fc3d6923ac364e4dfa64e36b291b", size = 5014082, upload-time = "2026-08-25T19:44:26.709Z" },
{ url = "https://files.pythonhosted.org/packages/42/8b/cb12b1b60c91b074ca6bf0fdd59aa8f10d8bc5f73af8faece86ef0421b37/cryptography-50.0.1-cp311-abi3-win_amd64.whl", hash = "sha256:aed8db4f6d71c51efb89530e12d9464e7bf2923d46c3205dc794a2a93f8c0648", size = 3842826, upload-time = "2026-08-25T19:44:28.784Z" },
{ url = "https://files.pythonhosted.org/packages/5b/f0/424cb557d99aa86ac55da5e2add02e2882e44047b6264f93ade1b975a993/cryptography-50.0.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:30a125032e5642a21ff816e021152bd4e7e94f03eff3f4b7fca41cd22bc3110f", size = 3973525, upload-time = "2026-08-25T19:44:30.7Z" },
{ url = "https://files.pythonhosted.org/packages/4d/72/3a2711d967977ab5fc80b782837c7e8d1ac7445e764c20c381a265c57ef3/cryptography-50.0.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:a0b1a59e3a089064a0ec309e9428c8e3ae4e161419d20ac33600767e83fc658a", size = 4708817, upload-time = "2026-08-25T19:44:32.773Z" },
{ url = "https://files.pythonhosted.org/packages/b4/f2/bb1f56e10815b789df0b409a69fa4992ff3d3fef9c72747f4a6b26fed38e/cryptography-50.0.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:8921d58f426793c5f1b47f0b59575780de9a095214958d0eb37d909593db8367", size = 4697300, upload-time = "2026-08-25T19:44:35.144Z" },
{ url = "https://files.pythonhosted.org/packages/08/bd/ed5396be499ffcf8807a585bfe38b71a1fbdd1c342b4f9b6d0ef5162a946/cryptography-50.0.1-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:a8f40ea47330e71b594a7e246898f93177c259490c63183dbaf9e571d71ed9a5", size = 4716039, upload-time = "2026-08-25T19:44:37.192Z" },
{ url = "https://files.pythonhosted.org/packages/f6/6e/1cf405c5c8e8df7545378048e954792f00b7f2367af8863ce8b8f3e10607/cryptography-50.0.1-cp314-cp314t-manylinux_2_28_ppc64le.whl", hash = "sha256:a255449073358275b64b67d3f595f268bbef70e72b6edb65e0c70c735bf739c9", size = 5332388, upload-time = "2026-08-25T19:44:39.16Z" },
{ url = "https://files.pythonhosted.org/packages/47/92/b4317e8c32c4f47b062f5398bd79106b220a124546f42be83bf32b761e2a/cryptography-50.0.1-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:8df2de9102026855887e4587084f6eabd80ed0f345b8ad8a7ac27ab9bf4723e0", size = 4730293, upload-time = "2026-08-25T19:44:41.298Z" },
{ url = "https://files.pythonhosted.org/packages/39/0d/a1e7633e2c744d0f2983320a27e924ef2264c79c56e1a58d5fb0a1cfd413/cryptography-50.0.1-cp314-cp314t-manylinux_2_31_armv7l.whl", hash = "sha256:ac02b07824d4d1001bd4367599f839c19cb171924c796e52c23508ac14c2c0cc", size = 4346031, upload-time = "2026-08-25T19:44:43.245Z" },
{ url = "https://files.pythonhosted.org/packages/88/dd/b215616f9bab3fc18510c78a4e5c9f362d77838503c363dc747c7d4f5c6f/cryptography-50.0.1-cp314-cp314t-manylinux_2_34_aarch64.whl", hash = "sha256:cbf74a81765ee67413503ca6e26dcc4f6f5a519822436cc0a1b97aab6c1b8a17", size = 4715344, upload-time = "2026-08-25T19:44:45.291Z" },
{ url = "https://files.pythonhosted.org/packages/b1/1b/ec3ebd31741d0e963612c4fe43caa39341b9b1e031e469820e42e4c83918/cryptography-50.0.1-cp314-cp314t-manylinux_2_34_ppc64le.whl", hash = "sha256:16c5ecd954b3330ebfb6605eca4fd952da8bef376551d5cc264534e3770a9ee6", size = 5287201, upload-time = "2026-08-25T19:44:47.297Z" },
{ url = "https://files.pythonhosted.org/packages/1a/01/0127d11a762b31a9ee0221894f540318761783f3fdc4bc5d057698caebd5/cryptography-50.0.1-cp314-cp314t-manylinux_2_34_x86_64.whl", hash = "sha256:79bf008d1f9af6071c797ad133e39915dfee7614f18f18f4db9072eb715064a3", size = 4730023, upload-time = "2026-08-25T19:44:49.435Z" },
{ url = "https://files.pythonhosted.org/packages/9e/b9/e7425ebfb599241a0c1d7000f1b466c3062da66c19d9525031315dff7213/cryptography-50.0.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:330fbb252391c596f1ae42c5754449dc924e6ad012dca8efe0d703f9f2d12ec6", size = 4847362, upload-time = "2026-08-25T19:44:51.94Z" },
{ url = "https://files.pythonhosted.org/packages/2d/fd/60d0ddf4defa12e482c9d5e0f554384d6e8ab25341fd15f060028fd92e6a/cryptography-50.0.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:42be3bb70596b3abe4ac097b75be223e8b3ab614a0e5de068e3dcc54d71d6149", size = 4999247, upload-time = "2026-08-25T19:44:53.876Z" },
{ url = "https://files.pythonhosted.org/packages/4d/56/bc4f2b209e766c93372cfcd59b781a0b2b59700f62a969580415b699c2b2/cryptography-50.0.1-cp314-cp314t-win_amd64.whl", hash = "sha256:f74455bb086a85d5e81246412602aaa97ed095e504cd40dd261ef50be42205bf", size = 3825806, upload-time = "2026-08-25T19:44:56.209Z" },
{ url = "https://files.pythonhosted.org/packages/84/a9/ee16a903f13755e914d1eecc482fe64d1f10761c3960e5d8fa6837377aff/cryptography-50.0.1-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:ca83d00d9e69cd5eb63f2e69c3a5a59e0cecae5ae14c6ae0b35830fe3b37bad0", size = 4035307, upload-time = "2026-08-25T19:44:58.305Z" },
{ url = "https://files.pythonhosted.org/packages/5e/a5/9ec7e81e8526c0d7a387d73386b2daed3f39e10d81a85930bd1b6bfba65c/cryptography-50.0.1-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:05ba322c4da95b262a212c345af888ef2c37c88c0509756ea00a0e6d68850f23", size = 4751900, upload-time = "2026-08-25T19:45:00.401Z" },
{ url = "https://files.pythonhosted.org/packages/7e/3c/0e77bd5ffcf078e9dd27d3074aad6c030d9b10d0bf69329d573c927a188c/cryptography-50.0.1-cp39-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:e22dfed744bd4002e909464cb23d2f0b05c6f3113a79ef2e9864a53db737c733", size = 4738357, upload-time = "2026-08-25T19:45:02.786Z" },
{ url = "https://files.pythonhosted.org/packages/27/3a/3c5f80daa4dcd47323c7af8a2fcb90de27a33564d4fcac69846c0972691a/cryptography-50.0.1-cp39-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:4c4188f7c0cf655be5c06342b817ed0f9595b69ffa2b12026e5353eed29dea88", size = 4758474, upload-time = "2026-08-25T19:45:04.889Z" },
{ url = "https://files.pythonhosted.org/packages/6e/2b/214cf0cf93db9628c3c20c896b229f327f6fb1b20e4b3743d8ad3f00af8b/cryptography-50.0.1-cp39-abi3-manylinux_2_28_ppc64le.whl", hash = "sha256:2ebbfb0f1fed745e91796e3e1080a1440423fdae8ece1b995a1d80883a409054", size = 5375862, upload-time = "2026-08-25T19:45:07.163Z" },
{ url = "https://files.pythonhosted.org/packages/d6/51/3f9701867a46b6c1740c9b52fc4d3bed6cbdcfedcc9b6e64305c07f39cff/cryptography-50.0.1-cp39-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:407fe2b6db00939c05c0e945e9914238f2f0a430974839429dafc82b1ee6bee5", size = 4772942, upload-time = "2026-08-25T19:45:09.396Z" },
{ url = "https://files.pythonhosted.org/packages/0d/5c/13ea642e08e2544d0f5396122055f4820cfacb3203562197b5967125ea97/cryptography-50.0.1-cp39-abi3-manylinux_2_31_armv7l.whl", hash = "sha256:2b34d76a652ea2b6faf777c35df230c5637842cd904e04f16230c3f9f03e4361", size = 4383347, upload-time = "2026-08-25T19:45:11.659Z" },
{ url = "https://files.pythonhosted.org/packages/84/d5/7d1fe1cb93f91c428093ff234e128c89ba8ea61a6f26aab406081f9b996e/cryptography-50.0.1-cp39-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:01f41478cf33fc605a6a089cd56d28b45c6c0b45a1928b61797f2621a04bac71", size = 4758050, upload-time = "2026-08-25T19:45:13.745Z" },
{ url = "https://files.pythonhosted.org/packages/dd/04/557fc5ead96a829e0bc812a3b9dc4a52a2f27e4f7f5950da7ff27653a805/cryptography-50.0.1-cp39-abi3-manylinux_2_34_ppc64le.whl", hash = "sha256:fc3ed7ebd2a8c96f5b166de0ab9b624996bef3b07bbeb19364dfb78222c22c80", size = 5332955, upload-time = "2026-08-25T19:45:16.193Z" },
{ url = "https://files.pythonhosted.org/packages/8c/eb/5d7124083e8d8cda8f5b348f544b71ad6f707ad63193758ef4d8e569da02/cryptography-50.0.1-cp39-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:9dde0a357190eb3b1da1bb9ab750e9c85cba82ca5977aa0836cbb94e92611239", size = 4772694, upload-time = "2026-08-25T19:45:18.315Z" },
{ url = "https://files.pythonhosted.org/packages/63/8e/f1f955e0921dd2b6d22eae7e8d24a4c4b638d10735ffbf6a71f99eb0fcb8/cryptography-50.0.1-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:fd3718b960d0b5dd213cdf03f3bcb7000e69dda0de8b956061947ff6bcff5558", size = 4888413, upload-time = "2026-08-25T19:45:20.4Z" },
{ url = "https://files.pythonhosted.org/packages/1f/ab/89e2b798d2c3925f82e2bb72d5979f3d2f6da2dd22ef4a8cd8b70d920039/cryptography-50.0.1-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:2a93d05e34d5f67fba6f891fe85d929999baa7195e853923ea6d7576c9e68c5e", size = 5044355, upload-time = "2026-08-25T19:45:22.353Z" },
{ url = "https://files.pythonhosted.org/packages/99/89/87ef49ffe383ef4e147d27b7bf2088fb0b54ea409dd87b5a89442e5828a5/cryptography-50.0.1-cp39-abi3-win_amd64.whl", hash = "sha256:55d16b1ef3ee0958d893a977b19777887e546c9954ea81b200c3301a864013f2", size = 3875429, upload-time = "2026-08-25T19:45:24.418Z" },
{ url = "https://files.pythonhosted.org/packages/c7/27/8d207af749c453ee17ea087340b3f2b4adef75aadd1d277b1b129bdda84e/cryptography-50.0.1-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:9cb3cb952cf5a8abd50c782a98a89d71699715e802fe349704b47f2425b42a94", size = 3974350, upload-time = "2026-08-25T19:45:26.551Z" },
{ url = "https://files.pythonhosted.org/packages/14/9a/6d3a4d7852e22d657438b7bf51f66102c7d71c0e1fafeec652281d0403e5/cryptography-50.0.1-pp311-pypy311_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:5fe939deeb161024a6be98229c953b6591fef1f41214497a78fe793a244c017f", size = 4698675, upload-time = "2026-08-25T19:45:28.658Z" },
{ url = "https://files.pythonhosted.org/packages/73/35/5c3717edf9e68a0550ce04e28eab493fe545eccd81742af03f6a75fe260b/cryptography-50.0.1-pp311-pypy311_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:fb4b9672d389c738b175c4166e78310f8a70358886aacd9173ee03a85ffdc671", size = 4707410, upload-time = "2026-08-25T19:45:30.816Z" },
{ url = "https://files.pythonhosted.org/packages/1d/e0/e786934472e3ac4ecdecc7b129a0ca1a2a40dffdafcf2c3ea9d4397f8def/cryptography-50.0.1-pp311-pypy311_pp73-manylinux_2_34_aarch64.whl", hash = "sha256:d63ae8f6481fec907ac0f588eee8a90aefde112c633131fe540e5711ddbb5a4e", size = 4698378, upload-time = "2026-08-25T19:45:33.043Z" },
{ url = "https://files.pythonhosted.org/packages/51/cf/5b3f53a0b74d122f023476ede40ba5d3e70d5cf475f73b899740d26a4fb2/cryptography-50.0.1-pp311-pypy311_pp73-manylinux_2_34_x86_64.whl", hash = "sha256:804728ce710890870f3aaa344b2e161172d258d768ac139d02cfd9092d0d94e6", size = 4706889, upload-time = "2026-08-25T19:45:35.086Z" },
{ url = "https://files.pythonhosted.org/packages/71/44/711e61f7d014be825ef79b285b047292d1bf893732ac1bc030a351fb517f/cryptography-50.0.1-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:693c99b49bd37d0d096e4334c10232c77248c415b98d35236094cdf96d57258b", size = 3824006, upload-time = "2026-08-25T19:45:37.281Z" },
]
[[package]]
name = "fastapi"
version = "0.141.1"
@@ -210,15 +364,27 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/41/45/1a4ed80516f02155c51f51e8cedb3c1902296743db0bbc66608a0db2814f/jsonschema_specifications-2025.9.1-py3-none-any.whl", hash = "sha256:98802fee3a11ee76ecaca44429fda8a41bff98b00a0f2838151b113f210cc6fe", size = 18437, upload-time = "2025-09-08T01:34:57.871Z" },
]
[[package]]
name = "mistune"
version = "3.3.4"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/7b/92/328a294a6de83bacb95bed01f04e0eaff4e3616ee359fc821a5dfc539b02/mistune-3.3.4.tar.gz", hash = "sha256:58b5c96d6fcb61190dfe5fae498d2b2065f99cf61e9649418fd54cf1ada86dfe", size = 121426, upload-time = "2026-07-22T05:22:30.89Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/77/e4/288365afae98953bc01de09f686f40d8ee84578135aa7767d5d4e60b5278/mistune-3.3.4-py3-none-any.whl", hash = "sha256:ee015381e955e370962968befe1d729ab60fafb6a715ac6751763fbce38c8d4a", size = 66862, upload-time = "2026-07-22T05:22:29.419Z" },
]
[[package]]
name = "notes-agent-backend"
version = "0.1.0"
source = { virtual = "." }
dependencies = [
{ name = "cryptography" },
{ name = "fastapi" },
{ name = "httpx" },
{ name = "jsonschema" },
{ name = "mistune" },
{ name = "pyyaml" },
{ name = "referencing" },
{ name = "sqlite-vec" },
{ name = "uvicorn", extra = ["standard"] },
]
@@ -230,10 +396,13 @@ dev = [
[package.metadata]
requires-dist = [
{ name = "cryptography", specifier = ">=45,<52" },
{ name = "fastapi", specifier = ">=0.116,<1.0" },
{ name = "httpx", specifier = ">=0.28,<1.0" },
{ name = "jsonschema", specifier = ">=4.25,<5.0" },
{ name = "mistune", specifier = ">=3.0,<4.0" },
{ name = "pyyaml", specifier = ">=6.0,<7.0" },
{ name = "referencing", specifier = ">=0.36,<1.0" },
{ name = "sqlite-vec", specifier = ">=0.1.9" },
{ name = "uvicorn", extras = ["standard"], specifier = ">=0.35,<1.0" },
]
@@ -259,6 +428,15 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl", hash = "sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746", size = 20538, upload-time = "2025-05-15T12:30:06.134Z" },
]
[[package]]
name = "pycparser"
version = "3.0"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/1b/7d/92392ff7815c21062bea51aa7b87d45576f649f16458d78b7cf94b9ab2e6/pycparser-3.0.tar.gz", hash = "sha256:600f49d217304a5902ac3c37e1281c9fe94e4d0489de643a9504c5cdfdfc6b29", size = 103492, upload-time = "2026-01-21T14:26:51.89Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/0c/c3/44f3fbbfa403ea2a7c779186dc20772604442dde72947e7d01069cbe98e3/pycparser-3.0-py3-none-any.whl", hash = "sha256:b727414169a36b7d524c1c3e31839a521725078d7b2ff038656844266160a992", size = 48172, upload-time = "2026-01-21T14:26:50.693Z" },
]
[[package]]
name = "pydantic"
version = "2.13.4"
+75
View File
@@ -0,0 +1,75 @@
# NotesAgent 文档索引
本目录集中保存团队开发期间需要长期维护的架构、接口、实现、协作和问题复盘文档。文档按用途分类,避免设计约束、开发记录与故障复盘混放。
## 目录分类
| 目录 | 内容 | 适用场景 |
| --- | --- | --- |
| [`architecture/`](architecture/) | 技术栈、阶段目标与团队分工 | 确认整体边界、模块依赖和阶段范围 |
| [`contracts/`](contracts/) | 前后端接口契约与页面需求 | 开发前对齐 DTO、路由、事件和交互 |
| [`development/`](development/) | 各模块的实现说明 | 阅读现有代码、联调和扩展功能 |
| [`guides/`](guides/) | Git、测试、注释和 CI/CD 规范 | 日常开发、提交、审阅和发布 |
| [`retrospectives/`](retrospectives/) | 审阅发现的问题与修复复盘 | 排查同类问题、撰写总结或博客 |
## architecture:架构与分工
- [AI 笔记软件技术栈说明](architecture/AI笔记软件技术栈说明-团队版-v2.3.md)
- [第一阶段分工表](architecture/第一阶段分工表.md)
- [第二阶段团队分工表](architecture/第二阶段团队分工表.md)
## contracts:契约与需求
- [后端接口契约](contracts/后端接口契约-开发版.md)
- [第二阶段接口契约](contracts/第二阶段接口契约-开发版.md)
- [前端页面需求说明](contracts/前端页面需求说明-开发版.md)
运行中的后端以 `/openapi.json` 为机器可读事实来源。接口契约用于描述设计意图、联调约束和实现状态;两者不一致时,应先确认代码行为,再在同一个 PR 中同步修正文档或实现。
## development:开发说明
- [AI Core 与 Agent Core 开发说明](development/AI-Core与Agent-Core开发说明.md)
- [Knowledge 与 Retrieval Core 开发说明](development/Knowledge与Retrieval-Core开发说明.md)
- [Benchmark 开发说明](development/Benchmark开发说明.md)
- [Export 开发说明](development/Export开发说明.md)
- [模型提供商与模型发现开发说明](development/模型提供商与模型发现开发说明.md)
- [MCP Bridge 与 Plugin Host 开发说明](development/MCP-Bridge与Plugin-Host开发说明.md)
- [独立 MCP Server 配置中心开发说明](development/独立MCP-Server配置中心开发说明.md)
- [Plugin Command 与 Settings 开发说明](development/Plugin-Command与Settings开发说明.md)
- [前端壳子与接口层开发说明](development/前端壳子与接口层开发说明.md)
- [前端写作体验优化开发说明](development/前端写作体验优化开发说明.md)
- [前端视觉与轻量动效优化开发说明](development/前端视觉与轻量动效优化开发说明.md)
## guides:团队协作规范
- [Git 使用细则](guides/Git使用细则-团队开发版.md)
- [CI/CD 细则](guides/CI-CD细则-团队开发版.md)
- [代码注释与 TODO 约定](guides/代码注释与TODO约定.md)
- [第一阶段测试验证操作手册](guides/第一阶段测试验证操作手册.md)
## retrospectives:问题与修复复盘
- [后端全面审阅问题与修复复盘](retrospectives/后端全面审阅问题与修复复盘.md)
- [Agent Core 第二阶段问题与修复复盘](retrospectives/Agent-Core第二阶段问题与修复复盘.md)
- [Knowledge 与 Retrieval Core 问题与修复复盘](retrospectives/Knowledge与Retrieval-Core问题与修复复盘.md)
- [Plugin Command 与 Settings 问题与修复复盘](retrospectives/Plugin-Command与Settings问题与修复复盘.md)
- [前端合并审阅问题与修复复盘](retrospectives/前端合并审阅问题与修复复盘.md)
## 推荐阅读顺序
新成员或新阶段开始时,建议按以下顺序阅读:
1. 技术栈说明和当前阶段分工表;
2. 所负责功能对应的接口契约;
3. 对应模块的开发说明;
4. Git、CI/CD、测试及注释规范;
5. 与当前任务相关的问题复盘。
## 维护规则
- 新文档先判断用途,再放入对应分类目录,不在 `docs/` 根目录继续堆放业务文档。
- 移动或重命名文档时,同步修正仓库内全部链接,并执行本地链接检查。
- 接口、数据结构或事件格式发生变化时,同一个 PR 内同步更新契约和相关开发说明。
- 问题复盘至少写清原因、后果、解决思路、实际方案和验证结果。
- `.local-plans/` 只保存个人或阶段性的本地计划,不属于正式团队文档,不应提交到远程仓库。
- 文档中的“计划实现”和“已经实现”必须明确区分;实现状态以代码、测试和运行时契约为准。
@@ -1,9 +1,12 @@
# AI 笔记软件技术栈说明
> 文档性质:团队技术基线
> 基线版本:v2.3
> 适用范围:桌面客户端、本地知识库、RAG、Agent、Skill、多模型接入、多模态处理与可选云同步
> 目标读者:前端、Rust 桌面端、Python AI Core、算法、测试与后续接手项目的开发成员
> 实施状态更新:2026-09-04。本文同时包含目标架构、当前实现和第二阶段接口基线。第一阶段已完成 Vue Web 联调前端、FastAPI、Knowledge/Retrieval、Agent/Tool/Permission、Skill/Plugin 声明式运行时、Mock/OpenAI-Compatible/Ollama Provider、DeepSeek/OpenAI 预设、模型发现及开发阶段 Fernet 凭据存储。Web Workspace 已通过 FastAPI 接入后端配置的真实单 Vault;第二阶段 Agent Trace 持久化、分页快照、可恢复 SSE、stdio MCP Bridge、隔离 Plugin Host、Plugin Command 与 Plugin Settings/Secret Contract 已完成。阶段 E 已完成 Responses/Anthropic 协议、国内 logo 预设、Provider 配置恢复和 Embedding/转写/声纹 API 路由;本地语音模型仍为阶段 F 接口预留。RAG Benchmark 检索评测(Dataset 加载、异步运行、SSE 进度、指标聚合与报告)已完成,Agent Benchmark 暂缓。后续继续接入真实音频处理、文档导出、主题包、Trace 可视化、Mermaid 和函数图像。Tauri/Rust Host、Stronghold、原生多 Vault 文件系统和 Sync Server 仍未实现。
---
## 1. 文档目的
@@ -34,8 +37,10 @@
| 桌面容器 | Tauri 2 + Rust | 桌面窗口、系统 API、本地文件访问、Sidecar 管理、安全边界 |
| 前端 | Vue 3 + TypeScript + Vite | 工作区、编辑器、AI 对话、搜索、设置、扩展管理等用户界面 |
| 状态管理 | Pinia | 管理工作区、编辑器、搜索、会话、Agent、Skill、主题与模型状态 |
| UI 基础 | Reka UI / Headless Components + Design Token | 通用交互组件和主题化能力 |
| UI 基础 | 当前公共 Vue 组件 + Element Plus 图标 + Design Token;目标按需引入 Reka UI | 通用交互组件、无障碍交互和主题化能力 |
| Markdown 编辑器 | Milkdown + CodeMirror 6 | 可视化 Markdown 编辑与源码编辑 |
| 图表渲染 | Mermaid + 受控 SVG 输出 | Markdown 流程图、时序图等图表的预览与静态导出 |
| 函数图像 | `FunctionPlot` 结构化模型 + Renderer Adapter | 二维函数解析、交互预览与导出静态图 |
| 本地核心服务 | Python + FastAPI + Pydantic v2 | RAG、Agent、Skill、模型访问、多模态、索引和本地 API |
| Python 打包 | PyInstaller / Nuitka | 将 Python AI Core 打包为 Tauri Sidecar |
| 笔记存储 | Markdown + Assets | 保存用户正文和附件 |
@@ -47,14 +52,16 @@
| Agent | 自研 Agent Runtime | 模型推理、工具选择、工具调用、结果回灌、运行控制 |
| Skill | 自研声明式 Skill Runtime | 复用提示词、工具集合、权限和检索配置 |
| Plugin | 自研 Plugin Runtime + Plugin Manifest + MCP Bridge | 扩展程序能力、Tool、外部服务集成和受控 UI Contribution |
| Theme | Theme Manifest + Design Token + 受限 CSS | 本地主题包导入、预览、启停与社区格式兼容 |
| LLM | 自研 Provider Adapter | 统一不同模型服务商的输入、输出、Streaming 与 Tool Calling |
| 模型协议 | OpenAI Responses / Chat Completions compatible / Anthropic Messages / Ollama | 用户自定义模型接入 |
| ASR | faster-whisper | 音频转写 |
| 说话人分离 | pyannote.audio | 课堂、会议等多人音频中的说话人区分 |
| 情感识别 | emotion2vec | 可选音频分析能力 |
| 密钥存储 | Tauri Stronghold | 保存模型 API Key 和同步凭证 |
| 文档导出 | Document AST + Exporter Adapter | Markdown 到 HTML、PDF、DOCX,并保留图表、公式和代码块 |
| 密钥存储 | 当前 Fernet 开发存储;目标 Tauri Stronghold | Web 联调期避免明文落盘,桌面集成后保存模型 API Key 和同步凭证 |
| 云同步 | 独立 Sync ServerFastAPI + PostgreSQL + S3/MinIO | 可选自托管,多设备 Vault 同步、版本管理和设备管理 |
| 测试 | pytest + 自建 RAG / Agent Dataset | 单元、接口、检索、Agent 和模型适配器测试 |
| 测试 | pytest + Vitest + 版本化 RAG / Agent Dataset | 后端、前端组件、接口、检索质量、Agent 行为和模型适配器测试 |
表中的技术选型构成当前开发基线。新增依赖时需要明确其所属层、调用方、运行位置和替换成本,避免同一功能出现多套并行实现。
@@ -390,6 +397,22 @@ Reka UI / Headless Components 提供 Dialog、Popover、Menu、Tabs、Select、T
业务组件中优先引用 Design Token。主题包负责覆盖 Token 和允许开放的组件样式。主题加载器需要限制资源路径,避免主题 CSS 引用 Vault 外的任意本地文件。
### 5.5 Theme Package 与社区格式
第二阶段统一 Theme Package 的本地分发格式:
```text
my-theme/
├── theme.yaml
├── theme.css
├── preview.png
└── README.md
```
`theme.yaml` 至少声明 `id``name``version``author``min_app_version`。导入流程固定为 Manifest 校验、CSS 安全检查、隔离预览、安装、启用;停用或卸载后必须恢复内置 Design Token,不残留全局样式。
主题 CSS 只能使用开放的 Token 和宿主允许的稳定选择器,不允许脚本、远程资源、任意本地路径、`@import` 或覆盖安全相关界面。第二阶段只定义本地导入和社区包兼容格式,不把联网 Theme Marketplace 作为客户端依赖。
---
## 6. Markdown 编辑与知识结构
@@ -470,6 +493,32 @@ Save Markdown
外部编辑器修改文件时也走相同流程。索引任务写入 `index_jobs`,前端可以展示待处理、处理中和失败状态。
### 6.5 Mermaid Code Block
Mermaid 使用标准 fenced code block 持久化,Markdown 文件仍是唯一正文来源:
````markdown
```mermaid
flowchart LR
A[Markdown] --> B[Renderer]
```
````
前端识别代码块后调用 Mermaid Renderer 生成 SVG。渲染必须支持亮暗主题、编辑后重新计算、错误占位、缩放查看和销毁旧实例。生成的 SVG 经过净化后才能插入 DOM;导出链路使用同一份 Mermaid 源码生成稳定的 SVG 或图片,不能抓取编辑器界面截图。
### 6.6 函数图像 Block
函数图像采用独立 fenced block,并在实现稳定后冻结语法:
````markdown
```function-plot
y = x^2
y = sin(x)
```
````
解析层将文本转换为与渲染库无关的 `FunctionPlot`,至少包含表达式、定义域、显示范围、坐标轴和渲染配置。表达式解析不得使用 `eval` 或执行任意 JavaScript。交互预览和导出共享同一数据模型;HTML 可以保留交互渲染,PDF/DOCX 使用 SVG 或栅格化静态图。
---
## 7. SQLite 与数据归属
@@ -496,7 +545,7 @@ index_jobs
sync_state
```
其中 `notes``blocks` 保存 Markdown 的结构化投影;FTS5 建立全文索引;sqlite-vec 保存 Block 向量;`agent_runs``tool_calls` 保存 Agent TraceProvider 表保存非敏感模型配置。
其中 `notes``blocks` 保存 Markdown 的结构化投影;FTS5 建立全文索引;sqlite-vec 保存 Block 向量;当前实现以 `agent_runs``agent_events` 保存 Agent TraceProvider 表保存非敏感模型配置。
API Key、同步 Token 等机密数据不进入 SQLite,通过 `credential_id` 与 Stronghold 中的实际密钥关联。
@@ -626,7 +675,9 @@ class EmbeddingProvider(Protocol):
async def embed_query(self, query: str) -> list[float]: ...
```
默认配置使用本地 BGE-M3 类模型。模型名称和具体推理实现由配置决定,索引记录需要保存 embedding model id 和向量维度。用户更换模型后,由索引服务识别维度或模型变化并提示重新建立向量索引
目标默认配置使用本地 BGE-M3 类模型。当前第一阶段实现是 128 维 `HashEmbeddingProvider`,只用于离线跑通向量存储、索引更新和 Hybrid 链路,不代表真实语义召回质量。第二阶段接入真实 Embedding 时继续实现相同接口,上层 Retrieval Core 不依赖具体模型运行时
索引记录需要保存 embedding model id、模型版本、向量维度和归一化方式。用户更换模型或任一索引兼容字段变化后,索引服务必须将旧向量标记为不可用并要求重建,禁止把不同模型生成的向量写入同一索引空间。
### 9.5 RRF 与 Reranker
@@ -722,7 +773,7 @@ Agent Run 至少提供以下限制:
- 网络访问权限;
- 并发 Tool 数量。
Agent 运行过程中产生的每一步写入 `agent_runs``tool_calls`,用户可以在 Agent Trace 中查看工具名称、参数摘要、耗时、执行结果和权限状态。
Agent 运行过程中产生的每一步写入 `agent_runs``agent_events`,用户可以在 Agent Trace 中查看工具名称、参数摘要、耗时、执行结果和权限状态。SSE 与 Benchmark 均从同一事件事实读取,不维护旁路数据。
### 10.3 Tool Registry
@@ -769,6 +820,25 @@ audio.transcribe
Tool Executor 对参数再次进行 Pydantic 校验。文件修改类 Tool 调用 Knowledge Core,不允许 Tool 自行读取或修改 SQLite 表。
### 10.4 Agent Trace Contract
第二阶段的 Trace 不从前端临时状态反推,而由 Agent Runtime 产生可回放事件。一个 Trace 至少覆盖:
```text
Run Started / Completed / Failed / Cancelled
Model Call Started / Completed
Thinking / Text Delta
Tool Call / Tool Result
Permission Required / Resolved
Citation
Usage
Error
```
每个事件携带 `run_id`、单调递增的 `sequence`、时间戳和结构化 `data`。Tool 事件额外记录调用 ID、参数摘要、耗时、权限结果、输出摘要和错误;Provider 事件记录 Provider、模型、Usage 和耗时,但不得记录 API Key、完整敏感正文或未经净化的第三方响应。
前端先按 `run_id + sequence` 回放和去重,再构建 Trace Tree。Benchmark 复用同一事件流统计工具选择、步骤、延迟和 Token Usage,不另建一套只供测试使用的 Agent 执行协议。持久化层接入后,SSE 使用 `Last-Event-ID` 或等价游标恢复中断连接。
---
## 11. Skill Runtime
@@ -914,7 +984,7 @@ Manifest 进入安装流程前使用 Pydantic Schema 校验。Plugin ID、版本
### 12.3 Plugin Contribution
第一阶段允许 Plugin 声明以下 Contribution
Plugin Manifest 可以声明以下 Contribution
```text
Tool
@@ -927,6 +997,10 @@ Settings Section
其中 Tool 面向 Agent;Command 面向命令面板和快捷操作;Importer / Exporter 用于文件格式扩展;Sidebar Panel 和 Settings Section 为前端提供受控扩展位置。
第一阶段只落地声明式 Tool Contribution。第二阶段新增 Command 和 Settings ContributionCommand 由后端注册为稳定 ID、标题、参数和执行目标,前端只消费 Contribution ContractSettings Schema 首批只允许 `string``number``boolean``select``secret reference`,由宿主动态生成表单。
Secret Setting 只保存 Credential ID,明文通过 Secret API 写入凭据存储,不进入 Manifest、Plugin Storage、Pinia 或 Agent Trace。Importer、Exporter 和 Sidebar Panel 保留现有 Manifest 扩展位,未完成宿主实现前不得标记为可用。
Contribution 由宿主应用决定挂载位置。Plugin 不直接修改应用路由、Pinia Store 或核心数据库 Schema。
### 12.4 Plugin Host
@@ -966,10 +1040,18 @@ Plugin Host 负责:
内置 Plugin 可以使用相同的 Plugin Interface 注册能力,减少内置功能和社区扩展之间的接口差异。
Python 包形式的 MCP Server 推荐使用固定版本的 `uvx --isolated --from <package>==<version> <command>` 启动,以隔离依赖并避免污染 AI Core 环境;包内脚本和非 Python Server 仍可使用受控 `command + args``uvx` 的虚拟环境不是安全沙箱,不能限制文件、网络、子进程或系统调用。
面向社区或不可信 Plugin 开放前,Tauri/Rust Host 必须增加平台级沙箱、完整进程树回收、包来源/签名校验,并在首次安装或命令变化时向用户完整展示 executable 和参数、要求明确同意。当前 Python Host 的独立进程、环境裁剪和 Permission 只用于可信开发联调,不能替代这些生产安全门槛。
在该门槛完成前,后端仅允许 `APP_ENVIRONMENT=development` 启动未沙箱化 MCP Host;生产环境统一返回 `MCP_TRUST_APPROVAL_REQUIRED`。Python `uvx` Server 在开发模式首次运行可能联网解析依赖,生产版本必须在安装/更新阶段预取并验证固定版本,正常运行阶段只使用已经准备好的环境。
### 12.5 MCP Bridge
MCP Bridge 用于接入具有 MCP Server 接口的插件或外部工具服务。
当前 Plugin Runtime 已实现本地 stdio Host:在授权后的启用阶段启动独立 Server 进程,完成 `initialize`、capability negotiation、分页 `tools/list``tools/call`、取消、超时、异常退出和 Host Restart。独立 MCP Server Registry 另行支持 stdio、Streamable HTTP 与旧 HTTP+SSE 兼容,包括 Session、协议 Header、认证 Header Secret、测试门禁和 Tool 动态映射。实现接受 `2025-11-25``2025-06-18``2025-03-26``2024-11-05` 协议版本;Plugin Manifest 的 Streamable HTTP、Resource、Prompt、Sampling 与操作系统级沙箱仍属于后续范围。
MCP Tool 进入系统后的调用路径为:
```text
@@ -988,6 +1070,20 @@ Tool Registry 仍使用项目自己的 `ToolDefinition` 和 `ToolResult`。MCP B
MCP 能力首先用于 Tool 和 Resource 类扩展。需要复杂 UI 的插件通过 Frontend Extension Slot 单独处理。
当前 stdio MCP Bridge 已覆盖以下协议边界:
```text
Server Process / Connection Lifecycle
initialize 与 capability negotiation
tools/list 与 ToolDefinition 映射
tools/call 与 ToolResult 映射
超时、取消和进程退出
协议错误与业务错误转换
健康检查与 Tool 注销
```
首个宿主实现支持本地 `stdio` 传输;其他传输在兼容性测试后增加。外部 Server 的 Tool 名称进入项目注册表前添加 Plugin 命名空间,并校验 JSON Schema、权限、重复 ID 及其与 Manifest Contribution 的一致性。MCP 调用复用项目自己的 Permission、超时、Agent Trace、日志净化和结果大小限制;子进程环境按白名单裁剪,不传入 Provider Key、Vault 或数据库路径。
### 12.6 Frontend Extension Slot
前端预留受控扩展点:
@@ -1064,6 +1160,8 @@ Plugin Storage API 负责访问该目录。插件不能通过自身目录拼接
需要保存密钥的 Plugin 通过 Secret API 请求独立 Credential ID,由 Stronghold 保存实际值。
第二阶段 Plugin 安装记录、启停状态、授权、配置 Schema 版本和 Contribution 元数据需要持久化。应用启动时先恢复元数据,再启动已启用的 Host;恢复失败的 Plugin 保持隔离并标记为 `error`,不能留下已注册但没有可用执行后端的 Tool 或 Command。
---
## 13. Provider Adapter 与模型接入
@@ -1144,9 +1242,11 @@ Done
前端 Streaming UI 只认识这些事件类型。
第二阶段 Provider 兼容性不以 Adapter 数量为目标,而以同一组行为测试为准:普通对话、Streaming、Tool Calling、Reasoning Event、取消、Usage 和错误映射。优先验证 OpenAI Responses、OpenAI Chat Completions、OpenAI-Compatible、Anthropic Messages 与 Ollama;不支持的 Capability 必须在请求前拒绝,不能静默丢弃 Tool 或附件。
### 13.4 Provider 配置与 API Key
普通 Provider 配置保存在 SQLite
目标桌面架构将普通 Provider 配置保存在 SQLite。当前 Web 联调版由内存 `ProviderRegistry` 持有,AI Core 重启后清空
```text
provider_id
@@ -1157,18 +1257,19 @@ credential_id
enabled
```
Stronghold 保存 `credential_id` 对应的实际 API Key。
当前开发版使用 Fernet 加密文件保存 `credential_id` 对应的实际 API Key,并允许环境变量回退;Tauri 集成后由 Stronghold 替换该存储实现
设置页面执行“测试连接”时:
```text
读取 Provider Config
Rust Host 读取 Secret
→ 构造临时 Credential Context
Credential Resolver 按 ID 读取开发密文或环境变量
→ AI Core 调用 Provider
→ 返回连接测试结果
```
当前设置页已经提供 OpenAI、DeepSeek 与 Ollama 预设,保存后通过 `/api/providers/{provider_id}/models` 自动发现模型。Credential API 只返回配置状态,不提供任何明文读取接口。
日志中不记录完整 API Key。请求异常信息在进入前端前过滤 Authorization Header 和密钥片段。
---
@@ -1201,6 +1302,23 @@ text
内容结构化模块将 Transcript 整理为 Markdown,同时保留音频时间信息,后续 RAG 引用可以跳回音频片段。
音频管线使用后台 Job,不在 HTTP 请求中长时间同步阻塞。转写结果统一为:
```text
job_id
attachment_id
language
segments[]
speaker
start_time
end_time
text
status
error
```
`pyannote.audio``faster-whisper` 通过独立 Adapter 加载,模型下载、设备选择、精度、批量大小和缓存目录由配置管理。缺少说话人模型时可以只返回时间戳转写,但必须明确标记 diarization 不可用;模型失败不能生成伪造的 completed 结果。
### 14.2 OCR
OCR 作为 Media Pipeline 的输入适配能力,用于图片笔记、白板照片、PPT 截图和扫描资料。OCR 输出进入附件文本索引,也可以由用户选择生成 Markdown。
@@ -1211,6 +1329,39 @@ OCR 引擎在当前技术栈中尚未固定,调用接口先定义为 `OCRProvi
emotion2vec 作为音频扩展分析模块。输出可以附加到音频段元数据,不参与核心 RAG 索引和 Agent 启动流程。
### 14.4 Document AST 与多格式导出
第二阶段建立统一导出链路:
```text
Markdown
→ Markdown AST
→ Document AST
→ DocumentExporter
├── HtmlExporter
├── PdfExporter
└── DocxExporter
```
`Document AST` 是导出器共享的中间表示,覆盖标题、段落、列表、表格、图片、引用、代码块、数学公式、Mermaid 和函数图像。Exporter 不直接解析编辑器 DOM,避免不同界面状态产生不同输出。
统一接口返回文件、MIME、警告和失败节点:
```python
class DocumentExporter(Protocol):
async def export(
self,
document: Document,
options: ExportOptions,
) -> ExportResult: ...
```
HTML 导出保留结构化语义和受控样式;PDF 与 DOCX 在不支持交互内容时使用静态 SVG 或图片。具体底层库在 PoC 后冻结,但必须封装在 Exporter Adapter 内,不允许导出库的数据结构渗透到 Knowledge Core。
### 14.5 可视化内容的统一静态输出
Mermaid Renderer 和 Function Plot Renderer 除前端预览外,都必须提供可重复的静态输出接口。导出器只消费 SVG、PNG 或带尺寸信息的资源引用,不调用 Vue 组件。渲染结果按源码 Hash、主题和渲染器版本缓存;源码、主题或版本变化时缓存失效。
---
## 15. 安全与权限
@@ -1716,6 +1867,8 @@ POST /api/index/rebuild
GET /health
```
第一阶段已实现路径和第二阶段冻结草案分别见 `../contracts/后端接口契约-开发版.md``../contracts/第二阶段接口契约-开发版.md`。规划接口完成前不得在前端假定其可用;完成后以 OpenAPI、Pydantic Contract 和 TypeScript Wire DTO 的一致结果为准。
HTTP 返回统一错误结构:
```json
@@ -1790,6 +1943,8 @@ ainote/
│ │ ├── skills/
│ │ ├── providers/
│ │ ├── media/
│ │ ├── extensions/
│ │ ├── export/
│ │ └── database/
│ └── tests/
@@ -1810,7 +1965,11 @@ ainote/
├── benchmarks/
│ ├── rag/
│ │ ├── datasets/
│ │ └── reports/
│ └── agent/
│ ├── datasets/
│ └── reports/
├── infra/
│ └── cloud/
@@ -1829,6 +1988,8 @@ ainote/
- VectorStore 具体实现只出现在 Retrieval 基础设施层。
- 内置 Skill 放在 `skills/builtin`,不硬编码在 Agent Runtime。
- Plugin Runtime、Plugin Host 和 MCP Bridge 位于 Extension Core,对第三方插件暴露的稳定接口放在 `packages/plugin-sdk`
- Mermaid 与函数图像的 Markdown 源码解析归编辑/文档模型,交互渲染归前端 Renderer,静态渲染契约由 Export Service 复用。
- Document AST 和 Exporter Adapter 位于 Export Service,导出器不得读取 Vue 组件 DOM 或 Provider 内部状态。
- 内置 Plugin 放在 `plugins/builtin`,通过与第三方 Plugin 相同的 Contribution 接口注册。
- Benchmark 数据和运行脚本放在 `benchmarks`
@@ -1838,36 +1999,34 @@ ainote/
### 19.1 基础环境
团队开发机需要准备:
当前 Web 联调开发机需要准备:
```text
Node.js
pnpm
Rust toolchain
Tauri CLI
Python 3.x
uv / Poetry(团队确定一种)
Node.js 22+
pnpm 10+
Python 3.11+
uv
SQLite
Git
```
Python 环境需要支持 faster-whisper、pyannote.audio、Embedding 和 Reranker 所需依赖。涉及 CUDA 的开发成员可以安装 GPU 版本,基础功能仍需提供 CPU 可运行路径
Rust Toolchain 与 Tauri CLI 只在桌面容器阶段安装。当前轻量 Embedding/Reranker 不要求 CUDA;接入 faster-whisper、pyannote.audio 或真实本地模型时再按所选运行时增加 CPU/GPU 依赖
### 19.2 本地开发
开发模式下分别启动 AI Core 和 Tauri
当前开发模式下分别启动 FastAPI 和 Vite
```text
Terminal A
services/ai-core
→ start FastAPI dev server
cd backend
uv run uvicorn app.main:app --reload --host 127.0.0.1 --port 8000
Terminal B
apps/desktop
pnpm tauri dev
cd frontend
pnpm dev
```
开发配置允许桌面端连接固定开发端口。正式构建时改为 Sidecar 随机端口模式。
Vite 将 `/api``/health` 代理到固定开发端口。正式桌面构建时改为 Tauri Sidecar 随机端口和临时访问令牌模式。
### 19.3 配置
@@ -1901,13 +2060,15 @@ Python 使用 pytest。重点覆盖:
- Plugin Manifest
- Plugin Lifecycle
- Plugin Tool Registration
- MCP Bridge
- MCP 生命周期、Tool 映射、取消与异常退出
- Provider Adapter
- 权限判断;
- API 错误转换。
前端测试重点覆盖 Store、Service 和关键交互流程。Rust 侧覆盖路径处理、Sidecar 启停和 Secret 访问封装。
第二阶段还需要增加 Theme Manifest/CSS 安全校验、Mermaid 错误与主题切换、Function Plot 表达式拒绝策略、Document AST 快照和 HTML/PDF/DOCX 导出回归测试。多模态测试使用小型固定音频 Fixture,模型级测试与不下载模型的快速单元测试分组执行。
### 20.2 RAG Benchmark
RAG Dataset 结构:
@@ -1932,15 +2093,20 @@ Hybrid + Reranker
核心指标:
```text
Hit@1
Hit@5
Recall@K
Hit@K
MRR
Citation Precision
Citation Recall
Latency
Citation Hit Rate
P50 Latency
P95 Latency
total_cases
successful_cases
failed_cases
failure_rate
```
Benchmark 参数、Embedding 模型、Reranker、数据集版本和运行环境需要一起记录,保证不同实验结果可以复现。
失败样本按零分计入质量指标分母,报告同时输出样本构成字段标明实际分母。Benchmark 参数、Embedding 模型、Reranker、数据集版本和运行环境需要一起记录,保证不同实验结果可以复现。
### 20.3 Agent Benchmark
@@ -1952,12 +2118,13 @@ Agent Dataset 保存任务目标、允许工具、期望工具序列和结果条
Task Success Rate
Tool Selection Accuracy
Tool Argument Accuracy
Invalid Tool Call Rate
Average Steps
Average Latency
Token Usage
```
Agent Benchmark 同时记录模型 Provider模型版本。
Agent Benchmark 同时记录模型 Provider模型版本、Skill、可用工具集合、运行配置和 Dataset 版本。指标直接从 Agent Trace Contract 计算,测试框架不得绕过权限或使用另一套 Tool 执行器
---
@@ -2076,6 +2243,47 @@ Audio
→ RAG Index
```
### 21.8 MCP Tool 接入
```text
Enable Plugin
→ Start MCP Server
→ MCP initialize / tools/list
→ Validate Schema and Permission
→ Namespace Tool ID
→ Register Tool Registry
→ Agent Tool Call
→ MCP tools/call
→ Normalize ToolResult / Error
→ Agent Trace
```
### 21.9 Mermaid 与函数图像
```text
Markdown Fenced Block
→ Parse Source
→ Mermaid / FunctionPlot Model
→ Renderer Adapter
├── Interactive Preview
└── Sanitized SVG / Image
→ Export Service
```
### 21.10 多格式文档导出
```text
Markdown
→ Markdown AST
→ Document AST
→ Resolve Assets / Formula / Diagram / Plot
→ Select Exporter
├── HTML
├── PDF
└── DOCX
→ ExportResult + Warnings
```
---
## 22. 异常处理与恢复
@@ -2105,7 +2313,7 @@ SQLite 索引损坏或版本不一致时,应用允许重建 `.ainote/app.db`
## 23. 当前版本实施边界
第一阶段开发围绕能够形成完整桌面知识工作流的能力展开
第一阶段围绕可联调的本地知识工作流建立基础接口
```text
Markdown Workspace
@@ -2123,21 +2331,42 @@ Markdown Workspace
→ Plugin Manifest / Tool Contribution
```
第一阶段 Plugin Runtime 需要完成安装、启用、停用、权限Tool 注册和至少一个示例 Plugin,建立 Skill 调用 Plugin Tool 的完整链路。
第一阶段 Plugin Runtime 完成安装、启用、停用、权限和声明式 Tool 注册,建立 Skill 调用 Plugin Tool 的基础链路。Command、Settings 和 MCP 执行不计入第一阶段完成项。
第二阶段接入:
截至 2026-09-03,上述第一阶段后端链路和 Web 联调前端均已完成;第二阶段的 Workspace 去 Mock 联调、Agent Trace 持久化/恢复接口、stdio MCP Bridge / Plugin Host,以及 Plugin Command/Settings 前后端闭环也已完成。Plugin 详情页现已提供 Host 状态、重启、动态设置、Secret 管理和命令执行,全局命令面板可加载 Plugin Command。当前验证基线为后端 136 项测试、前端 32 项测试、TypeScript 类型检查及生产构建通过。向量链路当前使用 `HashEmbeddingProvider` 验证工程正确性,真实 Embedding 召回质量不属于该测试结论。
第二阶段在既有 Contract 上接入:
```text
faster-whisper
pyannote.audio
主题导入与社区格式
MCP Bridge
Plugin Command / Settings Contribution
更多 Provider
Agent Trace 可视化
RAG / Agent Benchmark
Multimodal
├── faster-whisper
└── pyannote.audio
Extension / Model
├── MCP Bridgestdio 首版已实现)
├── Plugin Command Contribution
├── Plugin Settings Contribution
└── Provider Streaming / Tool Calling / Error Mapping 增强
Quality
├── RAG Benchmark
├── Agent Benchmark
└── Retrieval 参数调优
Content Output
├── Markdown → HTML(已实现)/ PDF / DOCX(暂缓)
├── Mermaid 编辑、预览与静态导出
└── Function Plot 解析、预览与静态导出
Frontend Extension
├── Theme Package 导入与社区包格式
├── Agent Trace 可视化
├── Plugin Command UI
└── Plugin Settings UI
```
上述列表描述第二阶段技术范围,其中 stdio MCP Bridge、Plugin Command Contribution、Plugin Settings Contribution 后端 Contract 与 Markdown → HTML 导出已实现,其余能力以各自开发说明的状态为准。每项功能必须继续经过现有 Service、Contract、Permission 和 Adapter 边界,不因 Demo 需要在 Vue 组件、Router 或 Agent Runtime 中直接绑定第三方协议。
第三阶段处理:
```text
@@ -2151,7 +2380,7 @@ Plugin 安装清单同步
Skill 分发
Plugin 分发与社区仓库
Plugin Sidebar Panel 等前端扩展点
Theme 社区
联网 Theme Marketplace
OCR 深度集成
更多多模态能力
扩展协议
@@ -2159,7 +2388,7 @@ OCR 深度集成
Sync Server 按独立服务开发和部署,不进入桌面客户端核心启动依赖。第一版同步完成文件级 Revision、多设备增量同步和冲突保留后,再评估端到端加密与实时协同编辑。
阶段划分用于安排开发顺序。模块接口在第一阶段完成时确定基础版本,后续功能通过现有接口扩展
阶段划分用于限定交付范围。人员分工、任务顺序和协作安排以阶段分工表为准,不在技术栈说明中重复维护;本文只维护技术选型、模块边界和跨模块 Contract
---
@@ -2188,10 +2417,12 @@ Sync Server 按独立服务开发和部署,不进入桌面客户端核心启
## 25. 当前技术基线摘要
目桌面端采用 Tauri 2、Rust、Vue 3 和 TypeScript。用户笔记以 Markdown 和 Assets 保存在本地 Vault,SQLite 管理元数据、全文索引、向量索引 Agent Trace。
桌面端采用 Tauri 2、Rust、Vue 3 和 TypeScript;当前可运行形态是 Vue/Vite Web 前端加 FastAPI。用户笔记以 Markdown 和 Assets 保存在本地 VaultSQLite 管理笔记元数据、全文索引、向量索引、任务及 Agent TraceProvider/Extension Registry 当前仍为内存实现
Python AI Core 作为 Tauri Sidecar 运行,FastAPI 提供本地接口。Knowledge Core 管理笔记结构;Retrieval Core 通过 FTS5、Embedding、sqlite-vec、RRF 和 Reranker 提供混合检索;Agent Runtime 使用 Tool Registry 操作知识库和任务;Skill Runtime 将提示词、工具、权限和检索参数组装为可复用 Agent 配置Plugin Runtime 通过 Plugin Manifest、Plugin Host 和 MCP Bridge 扩展 Tool、Command、导入导出和受控 UI ContributionPlugin 注册的 Tool 可以被 Agent 与 Skill 共同使用;Provider Adapter 对接 OpenAI、OpenAI-Compatible、Anthropic 和 Ollama 等模型服务
Python AI Core 未来作为 Tauri Sidecar 运行,当前由开发命令独立启动,FastAPI 提供本地接口。Knowledge Core 管理笔记结构;Retrieval Core 当前通过 FTS5、`HashEmbeddingProvider`、sqlite-vec、RRF 和轻量 Reranker 跑通混合检索,真实 Embedding 与正式 Benchmark 仍待第二阶段后续接入Agent Runtime 使用 Tool Registry 操作知识库和任务,并已持久化可供前端可视化与 Benchmark 共用的 Agent Trace ContractSkill Runtime 将提示词、工具、权限和检索参数组装为可复用 Agent 配置。
多模态处理使用 faster-whisper 和 pyannote.audio 完成音频转写和说话人分离,emotion2vec 作为扩展分析能力。API Key 和同步凭证存放在 Tauri Stronghold。多设备同步由独立 Sync Server 提供,采用 FastAPI、PostgreSQL 和 S3/MinIO,可由用户自托管。客户端在没有 Sync Server 时保持完整本地功能;连接服务器后同步 Markdown、Assets 和必要配置,各设备自行维护 FTS5、Embedding 和 Vector Index
当前 Plugin Runtime 支持 Manifest、生命周期、声明式白名单 Tool Contribution、Plugin Command 与 Plugin Settings/Secret,并已通过 stdio MCP Bridge 接入独立进程 Tool、专用 MCP Command Target、Host 状态与重启接口。Provider Adapter 当前实现 Mock、OpenAI Chat/OpenAI-Compatible 与 OllamaOpenAI Responses、Anthropic Messages 等协议仍待第二阶段后续完善。多模态目标方案使用 faster-whisper、pyannote.audio 和可选 emotion2vec;当前只读取 Host 预生成 transcript
第二阶段内容输出以 Document AST、Exporter Adapter、Mermaid Renderer 和 Function Plot Renderer 为共同边界,支持 HTML、PDF、DOCX 与静态图导出。Theme Package 使用 Manifest、Design Token 和受限 CSS 实现本地导入;联网主题市场不属于本阶段核心依赖。API Key 在 Web 联调期由 Fernet 开发存储加密保存,桌面版迁移到 Tauri Stronghold。多设备同步的目标方案为独立、可自托管的 Sync Server,目前尚未实现;本地核心功能不依赖 Sync Server。
该技术基线用于指导当前比赛版本的代码组织、接口设计、模块协作、测试和交付。
@@ -1,5 +1,17 @@
# 第一阶段分工表
> 状态更新:2026-09-02。本文保留第一阶段原始职责划分和交付口径。第一阶段后端目标与前端 Web 联调页面均已完成;第二阶段此后又完成 Agent Trace 持久化与 SSE 恢复、独立 stdio MCP Plugin Host 和 Plugin Command/Settings 后端 Contract。Tauri Host、Stronghold、原生多 Vault 文件系统、真实音频模型和 Sync Server 仍未实现。
## 当前交付状态
| 领域 | 当前状态 | 说明 |
| --- | --- | --- |
| Desktop Frontend / UI | Web 联调版已完成 | 全部业务路由、Markdown 写作/源码编辑、Chat、Search、Agent、扩展管理、设置页和 Provider 配置已落地 |
| Knowledge / Retrieval Core | 第一阶段已完成 | Markdown Block、SQLite/FTS5、sqlite-vec、Hybrid/RRF/Reranker、Citation 与事务修复均已覆盖测试 |
| AI / Agent Core | 第一阶段已完成 | Streaming、Agent Loop、Tool、Permission、Trace、Skill/Plugin 与 Knowledge/Retrieval 调用链已落地 |
| Model Core | 第一阶段已完成 | Mock、OpenAI Chat/OpenAI-Compatible、DeepSeek 预设、Ollama、模型发现和加密凭据链路可用 |
| Desktop Host / Sync | 后续阶段 | Tauri、Stronghold、系统文件访问、Sidecar 生命周期和云同步尚未实现 |
## 总分工表
| 成员 | 主要职责 | 第一阶段负责模块 | 具体工作内容 | 主要交付物 |
@@ -0,0 +1,821 @@
# 第二阶段团队分工表
## 一、阶段目标
第二阶段延续第一阶段已经形成的模块边界,重点推进多模态输入、MCP 与 Plugin 扩展、更多 Provider、RAG / Agent Benchmark、多格式导出、主题社区格式、Agent Trace 可视化、Mermaid 渲染和函数图像绘制。
第二阶段继续保持第一阶段的模块 ownership:
- 范涵宇:Agent Core、Extension Core、Model Core、Multimodal、Plugin Command / Settings UI、整体架构与代码审阅。
- 杨星萱:Knowledge Core、Retrieval Core、Benchmark、文档导出、函数图像绘制。
- 吉海燕:Frontend、Theme、Agent Trace、Mermaid 渲染。
---
## 二、总分工表
| 成员 | 主要负责方向 | 第二阶段任务 | 配合事项 |
| --- | --- | --- | --- |
| 范涵宇 | Agent Core / Extension Core / Model Core / Multimodal / 总体架构 | faster-whisper、pyannote.audio、MCP Bridge、Plugin Command / Settings 前后端闭环、更多 Provider、整体集成、代码审阅与统筹 | 为杨星萱的 Agent Benchmark 提供 Agent Trace、Tool Call 等测试接口 |
| 杨星萱 | Knowledge Core / Retrieval Core / Benchmark / Export / 数学内容渲染 | RAG Benchmark、Agent Benchmark 基础设施、Markdown → HTML / PDF / DOCX、函数图像绘制与渲染支持、Retrieval 调优 | 与范涵宇确认 Agent Benchmark 事件和测试数据结构;与吉海燕联调函数图像在编辑器和预览区中的展示 |
| 吉海燕 | Frontend / Theme / Visualization | Theme Import、Theme Manifest、社区主题格式、Agent Trace 可视化、Mermaid 渲染支持 | 与范涵宇联调 AgentEvent;与杨星萱联调函数图像及导出预览 |
---
## 三、范涵宇
### 3.1 faster-whisper
负责接入 faster-whisper,完成真实音频转写。
目标链路:
```text
Audio
faster-whisper
Timestamped Transcript
Knowledge Core
```
输出至少包含:
```text
text
start_time
end_time
language
```
### 3.2 pyannote.audio
负责说话人分离,并与 faster-whisper 组合。
```text
Audio
pyannote.audio
Speaker Segments
faster-whisper
Timestamped + Speaker Transcript
```
输出至少包含:
```text
speaker
start_time
end_time
text
```
### 3.3 MCP Bridge
负责实现 MCP Bridge,使外部 MCP Server 可以进入现有 Plugin / Tool 体系。
```text
External MCP Server
MCP Bridge
Plugin Runtime
Tool Registry
Agent Runtime
```
Agent Runtime 继续使用项目内部的:
```text
ToolDefinition
ToolCall
ToolResult
```
MCP Bridge 负责协议转换。
### 3.4 Plugin Command Contribution
负责 Plugin Runtime 中 Command Contribution 的解析、注册和执行接口。
```text
Plugin Manifest
Plugin Runtime
Command Registry
Frontend Contract
```
吉海燕负责 Command Palette 等前端展示。
### 3.5 Plugin Settings Contribution
负责 Plugin Manifest 中 Settings Schema 的解析和后端配置接口。
首批支持:
```text
string
number
boolean
select
secret reference
```
范涵宇负责 Manifest Parsing、Settings Schema、Plugin Storage、Secret Reference 和 Runtime Contract。
### 3.6 更多 Provider
继续完善 Provider Adapter。
第二阶段优先验证:
```text
Normal Chat
Streaming
Tool Calling
Reasoning Event
Cancellation
Usage
Error Mapping
```
主要协议目标:
```text
OpenAI Responses
OpenAI Chat Completions
OpenAI-Compatible
Anthropic Messages
Ollama
```
Provider 数量不作为主要验收指标,优先保证协议适配稳定。
### 3.7 审阅与统筹
持续负责:
- 公共 Contract 审阅;
- Agent / Skill / Plugin / Provider 接口审阅;
- 跨模块 PR Review
- 第二阶段整体架构一致性检查;
- Demo 链路集成;
- 公共错误类型和事件格式统一;
- 第二阶段版本合并与发布前检查。
---
## 四、杨星萱
### 4.1 RAG Benchmark
建立正式 RAG Benchmark。
比较:
```text
FTS5
Vector Search
Hybrid Retrieval
Hybrid + RRF
Hybrid + RRF + Reranker
```
核心指标:
```text
Hit@1
Hit@5
Recall@K
MRR
Citation Hit Rate
P50 Latency
P95 Latency
```
Dataset 至少记录:
```text
query
expected_note_id
expected_block_id
expected_citation
tags
```
### 4.2 Agent Benchmark
负责 Agent Benchmark 测试框架、Dataset、指标统计和报告生成。
范涵宇提供:
```text
Agent Run
Agent Event
Tool Call
Tool Result
Trace
```
主要指标:
```text
Task Success Rate
Tool Selection Accuracy
Tool Argument Accuracy
Invalid Tool Call Rate
Average Steps
Average Latency
Token Usage
```
示例:
```yaml
id: agent-os-review-001
prompt: >
找出操作系统笔记中关于死锁的内容,
生成总结并创建三个复习任务。
expected_tools:
- rag.search
- notes.read
- tasks.create
expected_conditions:
citation_required: true
tasks_created: 3
```
### 4.3 多格式文档导出
负责建立统一 Export Service
```text
Document AST
DocumentExporter
├── HtmlExporter
├── PdfExporter
└── DocxExporter
```
统一接口示意:
```python
class DocumentExporter(Protocol):
async def export(
self,
document: Document,
options: ExportOptions,
) -> ExportResult:
...
```
第二阶段完成:
```text
Markdown → HTML
Markdown → PDF
Markdown → DOCX
```
导出时尽量保持:
- 标题;
- 段落;
- 列表;
- 表格;
- 图片;
- 引用;
- 代码块;
- 数学公式;
- Mermaid
- 函数图像。
Exporter 接口保留未来由 Plugin 增加 EPUB、LaTeX 等格式的扩展空间。
### 4.4 函数图像绘制与渲染支持
负责函数图像相关的数据解析、表达、绘制和渲染支持。
目标示例:
```text
y = x^2
y = sin(x)
y = 2x + 1
```
内部建议抽象:
```text
FunctionPlot
├── expressions
├── domain
├── range
├── axis config
└── render config
```
处理链:
```text
Markdown / Structured Block
Function Expression Parser
Function Plot Model
Renderer
Editor / Preview
```
需要考虑:
- 二维函数;
- 多函数同图;
- 定义域;
- 坐标轴;
- 缩放;
- 图像刷新;
- Markdown 中的持久化格式;
- HTML / PDF / DOCX 导出时的静态渲染。
可设计独立 fenced block,例如:
````markdown
```function-plot
y = x^2
y = sin(x)
```
````
具体语法在实现阶段冻结。
函数图像的数据结构、解析和渲染逻辑由杨星萱负责;吉海燕配合完成前端容器、布局和交互。
### 4.5 Retrieval 调优
根据 RAG Benchmark 数据继续调整:
```text
Block Chunking
FTS Query
Vector Top-K
RRF Parameters
Reranker Candidate Count
Score Threshold
Metadata Filter
```
所有调优保留 Benchmark 前后对比数据。
---
## 五、前端展示层(吉海燕;5.4—5.5 由范涵宇负责)
### 5.1 Theme Import
负责 Theme Package 导入。
建议结构:
```text
my-theme/
├── theme.yaml
├── theme.css
├── preview.png
└── README.md
```
流程:
```text
Import
Manifest Validation
Preview
Install
Enable
Disable
Uninstall
```
### 5.2 Theme Community Format
负责制定和实现社区主题格式。
示例:
```yaml
id: example-theme
name: Example Theme
version: 1.0.0
author: example
min_app_version: 0.2.0
```
主题继续基于:
```text
Design Token
CSS Variables
Theme Manifest
```
第二阶段优先完成本地导入和社区包格式,不要求立即实现联网 Theme Marketplace。
### 5.3 Agent Trace 可视化
负责将 Agent Runtime 执行过程可视化。
建议结构:
```text
User Request
├── Model Call
├── rag.search
│ ├── arguments
│ ├── duration
│ └── result
├── notes.read
├── Model Call
└── Completed
```
Trace Node 可展开查看:
```text
Arguments
Result
Duration
Permission
Usage
Error
```
需要支持:
- Agent Run 状态;
- Tool Call 顺序;
- Tool 调用耗时;
- Tool Result 摘要;
- Provider 调用;
- Token Usage
- 错误状态;
- 用户取消;
- Citation 跳转。
### 5.4 Plugin Command 前端(范涵宇)
负责 Command Contribution 在前端呈现。
主要挂载:
```text
Command Palette
Context Menu
Toolbar Action
```
前端使用 Plugin Contribution Contract,不直接解析插件后端 Manifest。
### 5.5 Plugin Settings 前端(范涵宇)
根据范涵宇提供的 Plugin Settings Schema 动态生成设置表单。
支持:
```text
Input
Number
Switch
Select
Secret Input
```
Secret 类型通过 Secret API 提交,不写入 Pinia 持久化或普通配置文件。
### 5.6 Mermaid 渲染支持
负责在 Markdown 编辑器、预览区域和相关展示界面加入 Mermaid 渲染。
目标 Markdown
````markdown
```mermaid
flowchart LR
A[Markdown] --> B[Renderer]
B --> C[Diagram]
```
````
处理链:
```text
Markdown
Mermaid Code Block
Mermaid Renderer
SVG
Editor / Preview
```
需要支持:
- Flowchart
- Sequence Diagram
- Class Diagram
- State Diagram
- ER Diagram
- Gantt 等常用 Mermaid 图;
- Light / Dark Theme
- 编辑后重新渲染;
- 渲染错误提示;
- SVG 缩放与查看;
- 导出时的静态图处理。
Mermaid 渲染需要与 Theme Design Token 联动。
与杨星萱负责的 Export Service 联调时,前端或渲染层需要提供 Mermaid → SVG / Image 的稳定输出,使 HTML、PDF 和 DOCX 导出能够保留图表。
---
## 六、跨成员协作表
| 协作事项 | 主负责人 | 配合人员 |
| --- | --- | --- |
| MCP Bridge → Agent Tool | 范涵宇 | 杨星萱 |
| Plugin Command Runtime | 范涵宇 | 吉海燕 |
| Plugin Command UI | 范涵宇 | 吉海燕 |
| Plugin Settings Runtime | 范涵宇 | 吉海燕 |
| Plugin Settings UI | 范涵宇 | 吉海燕 |
| Agent Benchmark Framework | 杨星萱 | 范涵宇 |
| Agent Trace Event Contract | 范涵宇 | 吉海燕、杨星萱 |
| Agent Trace Visualization | 吉海燕 | 范涵宇 |
| RAG Benchmark | 杨星萱 | 范涵宇 |
| Markdown Export | 杨星萱 | 吉海燕 |
| Mermaid Editor Rendering | 吉海燕 | 杨星萱 |
| Mermaid Export Rendering | 杨星萱 | 吉海燕 |
| Function Plot Core / Rendering | 杨星萱 | 吉海燕 |
| Function Plot Frontend Integration | 吉海燕 | 杨星萱 |
| Theme Community Format | 吉海燕 | 范涵宇 |
| Provider Streaming UI | 范涵宇 | 吉海燕 |
| Multimodal → Knowledge Core | 范涵宇 | 杨星萱 |
| 第二阶段整体 Demo | 范涵宇 | 吉海燕、杨星萱 |
| 核心代码 Review | 范涵宇 | 对应模块负责人 |
---
## 七、第二阶段优先级
### P0
#### 范涵宇
```text
MCP Bridge
faster-whisper
pyannote.audio
```
#### 杨星萱
```text
RAG Benchmark
Agent Benchmark Infrastructure
Markdown → HTML
函数图像基础渲染
```
#### 吉海燕
```text
Agent Trace Visualization
Theme Manifest / Theme Import
Mermaid Rendering
```
### P1
#### 范涵宇
```text
Plugin Command Contribution
Plugin Settings Contribution
Provider Adapter 完善
```
#### 杨星萱
```text
Markdown → PDF
Markdown → DOCX
Retrieval 调优
函数图像导出适配
```
#### 吉海燕
```text
Theme Community Format
Plugin Command UI
Plugin Settings UI
Mermaid Theme Adaptation
```
### P2
#### 范涵宇
```text
更多 Provider
MCP 兼容性增强
Multimodal Pipeline 优化
```
#### 杨星萱
```text
复杂格式导出
Benchmark 报告自动生成
函数图像高级配置
```
#### 吉海燕
```text
Theme 浏览与管理体验
Agent Trace 高级筛选
Mermaid 高级交互
```
---
## 八、第二阶段共同验收 Demo
建议使用“课堂学习全过程”作为第二阶段总 Demo:
```text
导入课堂录音
pyannote.audio
说话人分离
faster-whisper
带时间戳 Transcript
生成 Markdown
Knowledge Core 建立索引
用户询问课程内容
Agent 调用 RAG
返回 Citation
Agent Trace 可视化
```
随后展示:
```text
选择 Skill
调用 Agent
通过 MCP Bridge 使用外部 Tool
生成整理后的 Markdown
```
Markdown 中展示:
```text
普通文本
数学公式
Mermaid 图
函数图像
```
最后:
```text
Markdown
├── Export HTML
├── Export PDF
└── Export DOCX
```
并切换一个导入的 Community Theme。
---
## 九、Definition of Done
### 范涵宇
- [ ] faster-whisper 能完成真实音频转写;
- [ ] pyannote.audio 能生成说话人分段;
- [ ] 两者能组合生成带时间戳和 Speaker 的 Transcript
- [x] MCP Server 能通过 MCP Bridge 注册 Tool
- [x] Agent 能调用 MCP Tool
- [x] Plugin Command Contribution 后端可注册;
- [x] Plugin Settings Contribution 后端可解析;
- [x] Plugin Command 可以显示并从前端执行;
- [x] Plugin Settings 可以动态生成设置项并独立提交 Secret;
- [ ] Provider Adapter 的 Streaming / Tool Calling / Error Mapping 稳定;
- [ ] 完成跨模块接口审阅和第二阶段集成。
### 杨星萱
- [ ] RAG Benchmark Dataset 可以稳定运行;
- [ ] 能输出 Hit@K、Recall@K、MRR、Latency 等指标;
- [ ] Agent Benchmark Framework 可以执行标准 Case
- [ ] Markdown 可以导出 HTML
- [ ] Markdown 可以导出 PDF
- [ ] Markdown 可以导出 DOCX
- [ ] Mermaid 在导出链路中可以保留为静态图;
- [ ] 函数图像能够由结构化表达生成;
- [ ] 函数图像能够进入预览和导出链路;
- [ ] Retrieval 调优结果有 Benchmark 数据支撑。
### 吉海燕
- [ ] Theme Package 可以导入;
- [ ] Theme Manifest 可以校验;
- [ ] Theme 可以启用、停用和卸载;
- [ ] Agent Trace 可以展示完整 Tool Call 顺序;
- [ ] Trace Node 可以查看参数、结果、耗时和错误;
- [ ] Markdown Mermaid Code Block 可以渲染;
- [ ] Mermaid 支持主题切换;
- [ ] Mermaid 渲染错误可以明确展示;
- [ ] Mermaid 图能够提供给 Export Service。
---
## 十、分工摘要
```text
范涵宇
├── faster-whisper
├── pyannote.audio
├── MCP Bridge
├── Plugin Command Runtime
├── Plugin Settings Runtime
├── Plugin Command UI
├── Plugin Settings UI
├── Provider Adapter
├── Code Review
└── Integration / Coordination
杨星萱
├── RAG Benchmark
├── Agent Benchmark Infrastructure
├── Retrieval Optimization
├── Export Service
│ ├── Markdown → HTML
│ ├── Markdown → PDF
│ └── Markdown → DOCX
└── Function Plot
├── Expression / Data Model
├── Rendering
└── Export Integration
吉海燕
├── Theme Import
├── Theme Community Format
├── Agent Trace Visualization
└── Mermaid
├── Markdown Rendering
├── Theme Adaptation
└── Export Rendering Interface
```
@@ -2,7 +2,9 @@
> 文档用途:供团队在第一阶段进行页面设计、Vue 开发、前后端联调和验收。
> 文档性质:开发需求基线,不是最终视觉规范或产品宣传文档。
> 依据:`第一阶段分工表.md``AI笔记软件技术栈说明-团队版-v2.2.md``后端接口契约-开发版.md`
> 依据:`../architecture/第一阶段分工表.md``../architecture/AI笔记软件技术栈说明-团队版-v2.3.md``后端接口契约-开发版.md`
> 实现状态:更新至 2026-08-31。全部已注册业务路由均已有真实页面;Markdown 写作/源码模式、Search、Chat、智能体执行轨迹、扩展管理、设置、Provider 预设、模型发现和开发阶段加密凭据输入均已落地。Web Workspace 已连接 FastAPI 管理的真实单 VaultTauri 原生目录选择和多 Vault 尚未接入。
## 1. 第一阶段目标
@@ -34,7 +36,7 @@
| --- | --- |
| 框架 | Vue 3、Composition API、TypeScript、Vite |
| 状态管理 | Pinia,只保存跨组件或跨页面状态 |
| UI 基础 | Reka UI / Headless Components |
| UI 基础 | 当前为项目公共组件、CSS Design Token 与 Element Plus 图标;复杂无障碍 Headless 组件后续按需引入 Reka UI |
| 编辑器 | Milkdown 为默认编辑模式,CodeMirror 6 为源码模式 |
| 桌面容器 | Tauri 2;文件、密钥、Sidecar 和系统能力通过 Rust Command |
| 本地 AI API | FastAPI;普通请求使用 HTTP JSON,流式数据使用 SSE |
@@ -611,18 +613,22 @@ Settings 使用分区导航,不把全部配置堆在一个表单中。
- 测试连接并显示耗时;
- 根据 Capability 标记是否支持 Chat、Tool Calling、Vision、Streaming 等。
API Key 输入后立即交给 Rust Stronghold,前端仅保存 `credential_id`。页面回显只能显示“已配置/未配置”,不得回显完整密钥。
当前 Web 联调阶段,API Key 通过 Credential API 交给 FastAPI,由 Fernet 加密保存;前端仅在提交期间持有明文,Provider 和 Store 只保留 `credential_id`。页面只能回显“已配置/未配置”,不得回显完整密钥。Tauri 集成后由 Stronghold 替换后端开发存储实现,接口边界保持不变。
接口:
```text
GET /api/providers
GET /api/providers/presets
POST /api/providers
GET /api/providers/{provider_id}
PATCH /api/providers/{provider_id}
DELETE /api/providers/{provider_id}
GET /api/providers/{provider_id}/models
POST /api/providers/test
GET /api/credentials/{credential_id}
PUT /api/credentials/{credential_id}
DELETE /api/credentials/{credential_id}
```
### 14.4 Index 与 Models
@@ -710,10 +716,10 @@ SecretService
### 16.3 WorkspaceService
- 统一封装 Tauri 文件命令;
- 统一封装 FastAPI Workspace/Note API,并为 Tauri 文件命令保留适配边界
- 规范化路径;
- 处理文件锁、自动保存和冲突;
- Web 开发模式提供可替换的 Mock 实现
- Web 开发模式连接后端配置的单一 Vault,禁止失败后回退 Mock
- 不把任意本地路径直接暴露给 Plugin UI。
## 17. 公共组件
@@ -1,6 +1,6 @@
# 后端接口契约(开发版)
> 本文档记录当前前后端联调使用的接口壳子。业务服务尚未实现,最终字段以 FastAPI 运行时生成的 OpenAPI 为准
> 更新日期:2026-09-02。本文档记录当前前后端联调使用的已实现接口;机器可读字段、校验规则和响应模型以 FastAPI 运行时生成的 OpenAPI 为准。第二阶段尚未实现的规划接口见 `第二阶段接口契约-开发版.md`,不要将规划路径视为当前服务能力
## 契约入口
@@ -30,8 +30,22 @@
| PATCH | `/api/notes/{note_id}` | 更新笔记 |
| DELETE | `/api/notes/{note_id}` | 删除笔记 |
| POST | `/api/notes/{note_id}/move` | 移动笔记 |
| POST | `/api/notes/{note_id}/rename` | 重命名笔记文件并保留 Note/Block 身份 |
| POST | `/api/search` | FTS、Vector 或 Hybrid 检索 |
### Workspace
Web 联调阶段只暴露后端通过 `APP_VAULT_PATH` 配置的单一 Vault,不接受浏览器传入任意本地目录。桌面多 Vault 与目录选择仍由后续 Tauri Host 提供。
| 方法 | 路径 | 用途 |
| --- | --- | --- |
| GET | `/api/workspace` | 获取当前 Vault、文件数和索引同步状态 |
| POST | `/api/workspace/open` | 打开配置的 Vault;磁盘路径集变化时重建索引 |
| GET | `/api/workspace/tree` | 获取真实 Markdown 文件和目录树 |
| POST | `/api/workspace/folders` | 新建目录 |
| POST | `/api/workspace/folders/rename` | 重命名目录并同步 Note 路径 |
| POST | `/api/workspace/folders/delete` | 删除目录及其 Note、Block、FTS 和向量记录 |
### Chat、Agent 与 Tool
| 方法 | 路径 | 用途 |
@@ -41,7 +55,8 @@
| POST | `/api/agent/runs` | 创建 Agent Run |
| GET | `/api/agent/runs/{run_id}` | 获取 Agent Run 状态与 Trace 摘要 |
| POST | `/api/agent/runs/{run_id}/cancel` | 取消 Agent Run |
| GET | `/api/agent/runs/{run_id}/events` | 订阅 AgentEvent SSE |
| GET | `/api/agent/runs/{run_id}/events` | 订阅 AgentEvent SSE,支持 `Last-Event-ID` / `after_sequence` 恢复 |
| GET | `/api/agent/runs/{run_id}/trace` | 分页读取持久化 Trace、摘要和运行配置快照 |
| POST | `/api/agent/runs/{run_id}/permissions/{request_id}` | 响应 Tool 权限确认 |
| GET | `/api/tools` | 获取已注册 Tool Definition |
@@ -61,6 +76,8 @@
| POST | `/api/plugins/{plugin_id}/enable` | 启用 Plugin |
| POST | `/api/plugins/{plugin_id}/disable` | 停用 Plugin |
| PUT | `/api/plugins/{plugin_id}/permissions` | 设置 Plugin 已授权权限 |
| GET | `/api/plugins/{plugin_id}/host` | 获取隔离 MCP Host 状态、工具数和协商信息 |
| POST | `/api/plugins/{plugin_id}/host/restart` | 重启 MCP Host 并重新发现、校验和注册 Tool |
| DELETE | `/api/plugins/{plugin_id}` | 卸载 Plugin |
### Provider
@@ -68,14 +85,18 @@
| 方法 | 路径 | 用途 |
| --- | --- | --- |
| GET | `/api/providers` | 获取 Provider 配置列表 |
| GET | `/api/providers/presets` | 获取 OpenAI、DeepSeek 与 Ollama 配置预设 |
| POST | `/api/providers` | 新建 Provider 配置 |
| GET | `/api/providers/{provider_id}` | 获取 Provider 配置 |
| PATCH | `/api/providers/{provider_id}` | 更新 Provider 配置 |
| DELETE | `/api/providers/{provider_id}` | 删除 Provider 配置 |
| GET | `/api/providers/{provider_id}/models` | 获取模型及 Capability 列表 |
| POST | `/api/providers/test` | 测试 Provider 连接 |
| GET | `/api/credentials/{credential_id}` | 查询凭据是否已配置,不返回明文 |
| PUT | `/api/credentials/{credential_id}` | 加密保存开发阶段 API Key |
| DELETE | `/api/credentials/{credential_id}` | 删除已保存凭据 |
Provider Contract 只传递 `credential_id` 或临时 `credential_context_id`,不通过普通 JSON 接口传递明文 API Key
Provider Contract 只传递 `credential_id` 或临时 `credential_context_id`。当前前后端开发阶段通过独立的 `PUT /api/credentials/{credential_id}` 接收 API Key,并立即加密落盘;该接口只返回配置状态,不返回密钥。Provider CRUD、模型列表和测试接口均不携带明文 API Key。Tauri 集成后由 Stronghold 接管存储实现
### Tasks、Media 与 Index
@@ -99,8 +120,8 @@ Provider Contract 只传递 `credential_id` 或临时 `credential_context_id`
```json
{
"error": {
"code": "NOT_IMPLEMENTED",
"message": "The contract is available, but its business service is not implemented.",
"code": "RESOURCE_NOT_FOUND",
"message": "Resource was not found.",
"details": {}
}
}
@@ -155,10 +176,17 @@ RunCancelled
## 当前实现状态
更新至 2026-09-02:后端 136 项回归测试通过;第二阶段 Plugin Command 与 Plugin Settings/Secret 接口已实现,详细 DTO 和边界见《第二阶段接口契约-开发版》第 7 节。
- Chat、Agent Run、Agent Events、Tool 列表、Provider 配置生命周期、模型列表和连接测试已经接入 AI Core。
- Provider Adapter 当前包含 Mock、真正增量 SSE 的 OpenAI-Compatible Chat Completions,以及 Ollama JSONL Streaming
- Agent Run/Event 已持久化到 SQLiteSSE 帧携带 sequence `id`,断线后可以回放缺失事件。Trace API 与 Benchmark 共用同一事件事实,并在入库前执行 Secret 脱敏和结果限长
- Provider Adapter 当前包含 Mock、增量 SSE 的 OpenAI-Compatible Chat Completions、OpenAI Responses、Anthropic Messages,以及 Ollama JSONL Streaming。阶段 E 增加 `/api/model-routing``/api/models/embeddings``/api/media/speaker-matches`;具体请求和阶段边界见第二阶段契约 §8.5。
- Notes、Search、Index、Skills、Plugins、Tasks 和 Provider 生命周期均已接入业务服务。
- Workspace 已接入后端配置的真实 Vault;文件树、笔记读写、文件/目录新建、重命名和删除不再使用前端 Mock Fallback。
- Note Move 保留 `note_id`;Citation 的字符偏移统一使用 UTF-16 code unit,供浏览器编辑器直接定位。
- Plugin 启用前必须通过权限接口记录授权,未知权限默认拒绝。
- 本地 stdio MCP Server 已通过独立子进程接入 Plugin RuntimeAgent 只消费内部 Tool Contract。Host 支持 initialize、分页发现、调用、超时取消、状态查询、重启和异常退出后的 Tool 注销。
- Attachment Tool 读取 Host 管理的 `attachments` 目录;音频接口读取 Host 生成的转写文本,真实本地语音模型在第二阶段接入。
- 接入业务模块时保持当前路径和 Contract,不在 Router 中直接实现数据库、Provider 或 Agent 逻辑。
第二阶段开发保持本文件中已有路径兼容,并按 `第二阶段接口契约-开发版.md` 增加子资源、可选字段和事件。接口完成后先更新 OpenAPI 与本文件,再将第二阶段文档中的状态改为已实现。
File diff suppressed because it is too large Load Diff
@@ -2,6 +2,8 @@
> 本文档用于团队开发和模块联调,记录当前已经落地的核心边界与使用方式。
> 更新日期:2026-09-02。第一阶段 AI Core、Agent Core、Extension Core 和 Model Core 主链路已经完成;第二阶段 Agent Trace 持久化、可恢复 SSE、stdio MCP Bridge、隔离 Plugin Host 以及 Plugin Command/Settings 已落地,后端当前回归基线为 136 项测试通过。
## 当前实现
当前已经建立第一条可运行链路:
@@ -26,16 +28,19 @@ backend/app/
│ └── mock.py 离线开发 Provider
├── agent/
│ ├── runtime.py Agent Loop、限制、取消、Trace 和 SSE
│ ├── trace_repository.py Run/Event SQLite 持久化、分页、摘要与脱敏
│ ├── tools.py Tool 注册、参数校验、隔离执行和结果转换
│ ├── permissions.py 权限策略、确认请求和会话授权
│ └── builtin_tools.py 无副作用的内置开发 Tool
├── extensions/
── runtime.py Skill/Plugin Manifest、生命周期、依赖与 Tool Contribution
── runtime.py Skill/Plugin Manifest、生命周期、依赖与 Tool Contribution
│ └── mcp.py stdio JSON-RPC、MCP 生命周期、发现、调用与 Host 隔离
└── container.py AI Core 依赖组装
backend/extensions/
├── skills/knowledge-assistant/ 内置知识库 Skill
── plugins/text-tools/ 内置示例 Plugin
── plugins/text-tools/ 内置声明式 Plugin
└── fixtures/mcp-echo/ 离线 MCP Server 联调 Fixture
```
Router 只负责 HTTP/SSE 与错误转换,不实现 Agent、Tool 或 Provider 业务逻辑。
@@ -51,9 +56,10 @@ Router 只负责 HTTP/SSE 与错误转换,不实现 Agent、Tool 或 Provider
- Permission
- Step、Timeout、Token Budget、取消;
- Tool 并发上限与 run 级网络权限;
- 内存 Trace 与 SSE
- SQLite Trace、分页快照与可恢复 SSE
- Skill Manifest、Prompt、Tool/Permission/模型能力解析;
- Plugin Manifest、生命周期和 Tool Contribution
- stdio MCP Bridge、隔离进程生命周期、Tool 映射与 Host 健康状态;
- Skill 调用内置 Tool 与 Plugin Tool
- 公共 Contract 和 API 接入。
@@ -62,7 +68,7 @@ Router 只负责 HTTP/SSE 与错误转换,不实现 Agent、Tool 或 Provider
- Note、NoteBlock、Markdown Parser:由 Knowledge Core 提供;
- FTS5、Vector、RRF、Reranker、Citation:由 Retrieval Core 提供;
- 文件系统和 API Key 明文读取:由 Rust Host 提供;
- MCP Plugin Host、Frontend Extension Slot:按技术基线放在第二阶段实现
- Frontend Extension Slot 与 Plugin Command/Settings UI:后端 Contract 与前端 Service 已完成,页面由前端后续联调
## Provider
@@ -125,20 +131,26 @@ ollama
}
```
凭证 ID `openai-main` 对应 Sidecar 进程中的临时环境变量 `AINOTE_CREDENTIAL_OPENAI_MAIN`环境变量由 Rust Host 从 Stronghold 读取后注入,不写入 Provider Config、日志或前端 Store
凭证 ID `openai-main` 可以对应开发环境变量 `AINOTE_CREDENTIAL_OPENAI_MAIN`当前 Web 联调版也允许设置页通过 Credential API 提交 API Key,由 `EncryptedCredentialStore` 使用 Fernet 加密保存;前端 Store、Provider Config、日志和读取响应都不保存或返回明文。未来接入 Tauri 后,由 Rust Host 从 Stronghold 注入或替换存储实现
Provider 配置生命周期接口已经可用:
```text
GET /api/providers
GET /api/providers/presets
POST /api/providers
GET /api/providers/{provider_id}
PATCH /api/providers/{provider_id}
DELETE /api/providers/{provider_id}
GET /api/providers/{provider_id}/models
POST /api/providers/test
GET /api/credentials/{credential_id}
PUT /api/credentials/{credential_id}
DELETE /api/credentials/{credential_id}
```
设置页现已提供 OpenAI、DeepSeek 和 Ollama 预设,并在保存后自动获取、排序和去重模型列表。模型发现会区分凭据缺失、鉴权失败、限流、超时和上游不可用等错误。
## Agent Run
创建普通 Agent Run
@@ -163,10 +175,13 @@ POST /api/agent/runs
```text
GET /api/agent/runs/{run_id}
GET /api/agent/runs/{run_id}/events
GET /api/agent/runs/{run_id}/trace?after_sequence=-1&limit=200
POST /api/agent/runs/{run_id}/cancel
```
当前 Run 与 Trace 保存在内存中,AI Core 重启后清空。Runtime 最多保留 200 个 Run,每个 Run 最多保留 2000 个事件,并限制单轮 Tool Call 数量,避免长时间运行时无界增长。后续数据库层接入时替换 Repository,不改变 API Contract
Run 与 AgentEvent 已写入 SQLite`run_id + sequence` 是幂等键。SSE 每帧包含 `id: sequence`;客户端可以通过 `Last-Event-ID` 请求头或 `after_sequence` 查询参数恢复缺失事件。Trace API 返回平铺事件、下一游标、分页状态、模型/工具调用统计、耗时、Token Usage 和创建 Run 时的配置快照,不负责生成前端树形布局
运行时内存仍只保留最近 2000 个事件用于实时订阅,完整 Trace 以 SQLite 为准。AI Core 重启后,已经终止的 Run 可以继续查询和回放;重启前未终止的 Run 会收束为 `AGENT_PROCESS_RESTARTED`,避免永久停在 `running`。API Key、Authorization、Password、Secret、常见 `sk-`/Bearer 值在入库前脱敏。Event、Request 和配置快照中的超长字符串与集合会截断;作为查询事实来源的 `AgentRun` 只脱敏、不限长,保证重启前后 input/output 内容一致。
## Tool Calling
@@ -290,7 +305,7 @@ DELETE /api/skills/{skill_id}
### Plugin Runtime
第一阶段 Plugin Runtime 完成 Manifest 校验、安装、启用、停用、卸载和 Tool Contribution。第三方代码不会直接 import 到 AI Core;当前 Declarative Plugin Host 只执行宿主实现的白名单 handler,MCP Host 留到第二阶段
第一阶段 Plugin Runtime 完成 Manifest 校验、安装、启用、停用、卸载和声明式 Tool Contribution。阶段 C 增加 stdio MCP Bridge第三方代码不会直接 import 到 AI Core,而由独立子进程运行,通过换行分隔 JSON-RPC 完成 initialize、Tool 发现和调用
启用 Plugin 时将 Tool 注册到统一 Tool Registry,并标记 `source=plugin`;停用或异常时注销 Tool。启用中的 Skill 依赖某 Plugin Tool 时,Plugin 不能直接卸载。
@@ -303,11 +318,15 @@ GET /api/plugins/{plugin_id}
POST /api/plugins/{plugin_id}/enable
POST /api/plugins/{plugin_id}/disable
PUT /api/plugins/{plugin_id}/permissions
GET /api/plugins/{plugin_id}/host
POST /api/plugins/{plugin_id}/host/restart
DELETE /api/plugins/{plugin_id}
```
Plugin Manifest 中的权限只是声明,不代表已经授权。带权限的 Plugin 安装后进入 `permission_required`,Host 必须通过权限接口记录用户授权,之后才能启用。JSON Schema 在安装阶段校验,Tool 调用时再次校验实际参数。
MCP Tool 进入 Registry 前统一增加 `<plugin_id>.<remote_name>` 命名空间。Server 声明的 `notesagent/permission` 必须属于已知权限并出现在 Plugin Manifest;发现集合还必须与 Manifest Contribution 完全一致。启用失败会回滚全部 Tool 并关闭子进程,异常退出会把 Plugin 标记为 `error` 并立即注销对应 Tool。详细实现和 Fixture 操作见 [MCP Bridge 与 Plugin Host 开发说明](MCP-Bridge与Plugin-Host开发说明.md)。
内置示例 `text-tools` 注册 `text.uppercase`。内置 `knowledge-assistant` Skill 同时声明 `notes.search``text.uppercase`,用于验证完整链路:
```text
@@ -321,11 +340,13 @@ Skill Manifest
## 当前限制与下一步
- 已实现 Mock、OpenAI-Compatible Chat Completions 与 Ollama AdapterOpenAI Responses 和 Anthropic Messages 尚未实现
前端智能体页面已经完成中文联调:运行状态、Agent Event、内置 Tool、Permission 和常用事件详情字段均通过集中标签映射展示中文;`notes.search` 等技术 ID 继续保留,便于与后端 Trace、日志和接口契约对应
- 已实现 Mock、OpenAI-Compatible Chat Completions、Ollama、OpenAI Responses 和 Anthropic Messages Adapter;阶段 E 同时完成国内预设、持久化配置和能力模型路由,详见 [模型提供商与模型发现开发说明](模型提供商与模型发现开发说明.md)。
- Provider 配置暂存内存,后续通过 Repository 接入 SQLitePATCH 已支持用显式 `null` 清空 base URL、默认模型和凭据引用。
- Run/Trace 暂存内存;下一步抽象 Repository 接入 SQLite。
- Permission 已有核心等待/恢复机制,前端确认 UI 尚未联调
- Run/Trace 已通过 Repository 接入 SQLite;后续增加按保留策略归档和 Benchmark 引用保护
- Permission 已有核心等待/恢复机制,前端确认 UI 已完成联调和中文展示
- Task 已持久化到 SQLiteAttachment Tool 读取 Host 管理目录中的 UTF-8 文件。
- `audio.transcribe` 当前消费 Host 预生成的 transcriptfaster-whisper 与说话人分离仍按技术基线在第二阶段接入。
- `audio.transcribe` 当前消费 Host 预生成的 transcriptfaster-whisper 与说话人分离仍第二阶段后续接入。
- Extension 安装记录暂存内存;后续接入持久化 Registry 与版本升级流程。
- 当前 Plugin Host 支持内置声明式白名单 handler;MCP Bridge、独立进程健康检查与 UI Contribution 在第二阶段实现
- 当前 Plugin Host 支持内置声明式 handler、本地 stdio MCP Server 以及 Plugin Command/Settings;独立 MCP Server Registry 另行支持 stdio、Streamable HTTP 与旧 SSE 兼容。OS 级沙箱与 UI Contribution 留在后续阶段
+78
View File
@@ -0,0 +1,78 @@
# Benchmark 开发说明
> 所属模块:Knowledge / Retrieval Core(后端,负责人 yxx)。RAG Benchmark 已交付;Agent Benchmark 暂缓,待 Agent Runtime 完成后在同一契约下补齐。
## 定位
Benchmark Service 用受控 Dataset 对检索引擎做可复现评测:创建即返回 queued、后台 asyncio.Task 执行、SSE 实时推送进度、结束后产出结构化报告。CLI、测试与前端报告页复用同一 Service,不各自实现指标。
## 接口
| 方法 | 路径 | 用途 |
| --- | --- | --- |
| GET | `/api/benchmarks/datasets?kind=rag` | 枚举受控目录下的 Dataset 元信息 |
| POST | `/api/benchmarks/rag/runs` | 创建 RAG Benchmark202 |
| GET | `/api/benchmarks/runs?kind=&status=&limit=&offset=` | 分页获取运行记录 |
| GET | `/api/benchmarks/runs/{run_id}` | 状态与指标摘要 |
| GET | `/api/benchmarks/runs/{run_id}/events` | SSE 进度与 Case 结果 |
| POST | `/api/benchmarks/runs/{run_id}/cancel` | 取消运行 |
| GET | `/api/benchmarks/runs/{run_id}/report` | 结构化完整报告 |
Agent Benchmark 的 `/api/benchmarks/agent/runs` 未暴露(暂缓),不在 OpenAPI 注册占位接口。
## Dataset
Dataset 来自 `settings.benchmark_datasets_path`(默认 `backend/data/benchmarks`),API 不接受调用方提交任意路径。按文件名 stem 精确匹配 `{dataset_id}.json`,与请求无关文件的损坏(JSON 语法错误、UTF-8 解码错误、顶层非对象)不会阻断加载;只有目标文件本身损坏才返回 `BENCHMARK_DATASET_INVALID`
RAG Case 结构:`case_id``query``expected_note_ids``expected_block_ids``citation_required``tags``citation_required=true` 时必须声明 `expected_block_ids`,否则无法计算 Citation Hit Rate。
## 运行生命周期
`queued → running → completed | failed | cancelled`
- 创建时校验索引兼容性:索引非空、Embedding model/dim 与当前引擎一致、vector/hybrid 时向量索引非空;不满足返回 `BENCHMARK_INDEX_INCOMPATIBLE`(409),避免把环境/索引错误误判为检索质量差。
- 内存注册表上限 `MAX_RUNS=100`,超限只淘汰终态 run;满容量且全为活动 run 时返回 `BENCHMARK_CAPACITY_EXCEEDED`429)。
- 失败/取消只向公开响应暴露项目错误码与安全消息,详细异常进入日志,不通过 HTTP/SSE 返回。
## 指标
RAG 按 (mode, case, repeat) 逐样本计算,再按 mode 聚合:
- 质量:`hit_at_1``hit_at_5``recall_at_k``mrr``citation_hit_rate`
- 延迟:`p50_latency_ms``p95_latency_ms`(仅统计成功样本);
- 样本构成:`total_cases``successful_cases``failed_cases``failure_rate`
失败样本按零分计入质量指标分母,报告据此可知实际分母,避免把执行失败误判为检索质量差。
## 事件与 SSE
事件流:`RunStarted → CaseCompleted* → RunCompleted | RunFailed | RunCancelled`
`GET /api/benchmarks/runs/{run_id}/events` 支持 `Last-Event-ID``?after_sequence=` 游标恢复(复用 Agent SSE 的解析逻辑),`RunCompleted` / `RunFailed` / `RunCancelled` 为终止事件,收到后断流。
## 错误码
```text
BENCHMARK_DATASET_NOT_FOUND
BENCHMARK_DATASET_INVALID
BENCHMARK_INDEX_INCOMPATIBLE
BENCHMARK_CAPACITY_EXCEEDED
BENCHMARK_RUN_NOT_FOUND
BENCHMARK_RUN_FAILED
BENCHMARK_CASE_EVALUATION_FAILED
```
## 配置快照
报告与运行记录保存 `config_snapshot`dataset hash/version、modes、retrieval 参数、Reranker、索引元数据、App 版本与环境、Python 版本。`local_embedding` 记录本地基线 model/version/dim`embedding.policy = per_case` 表示实际来源以逐样本结果为准,不能把本地基线当作本次使用的模型。
每个 `RAGCaseResult.embedding`(同时出现在报告 cases 和 CaseCompleted SSE 中)记录 `source`api/local/not_used/unavailable)、实际 `model_id` 空间标识、`dimensions`、本地 `version``fallback_reason`。远程路由还记录请求时的 `route_version``requested_route`provider_id/model/endpoint/dimensions,不含凭据)、成功生成查询向量后的 `attempted_space`。FTS 标记 not_used;调用失败而未完成向量检索时标记 unavailable。API 不可用或远程索引缺失时,实际模型仍记录最终使用的本地基线。配置允许在样本间改变,逐样本记录对应实际调用;汇总指标可能包含多种空间,比较实验时需检查 cases。记录使用任务局部上下文隔离,并发评测不会相互覆盖。
## 测试
```powershell
cd backend
uv run pytest -q
```
`tests/test_benchmark.py` 覆盖数据集注册与校验、指标纯函数、端到端运行、取消、索引兼容、容量与失败样本聚合;`tests/test_retrieval.py` 覆盖 FTS 阈值与分页 total 一致性。
+85
View File
@@ -0,0 +1,85 @@
# Export 开发说明
> 所属模块:Export Service(后端,负责人 yxx)。本次交付「多格式文档导出」第一步:Markdown → HTML 的完整生命周期;PDF/DOCX 与函数图像静态渲染在后续 PR 补齐。契约对应 [第二阶段接口契约 §10](../contracts/第二阶段接口契约-开发版.md)。
## 定位
Export Service 把笔记或未保存的 Markdown 文本渲染为可下载的 HTML 文件。采用与 Benchmark 一致的「创建即返回 queued、后台 asyncio.Task 执行」的内存模型,产物带 24h 过期时间,过期后不可下载。导出是轮询式(无 SSE 事件流),客户端通过 `GET /api/exports/{job_id}` 轮询状态,完成后走 `GET /api/exports/{job_id}/file` 下载。
## 模块布局
```text
backend/app/export/
├── __init__.py 包说明
├── document.py Document AST 内部协议 + DocumentExporter Protocol + ExportResult
├── markdown.py mistune 'ast' renderer → Document AST
├── exporters/
│ ├── __init__.py
│ └── html.py HtmlExporterDocument AST → 完整 HTML5
└── service.py ExportService(注册表 + 后台渲染 + 取消 + 产物生命周期)
```
HTTP DTO`ExportStatus` / `ExportFormat` / `ExportSource` / `ExportOptions` / `ExportJob` 等)放在 [app/contracts.py](../../backend/app/contracts.py),与 Benchmark DTO 同层;`DocumentNode` / `ExportResult` 属导出器内部协议,放在 `export/document.py`,不进入 HTTP 契约。
## 接口
| 方法 | 路径 | 用途 |
| --- | --- | --- |
| POST | `/api/exports` | 创建导出任务(202 |
| GET | `/api/exports?status=&format=&limit=&offset=` | 分页获取任务 |
| GET | `/api/exports/{job_id}` | 查询任务状态 |
| GET | `/api/exports/{job_id}/file` | 下载已完成产物 |
| POST | `/api/exports/{job_id}/cancel` | 取消任务 |
`source.type` 支持 `note`(引用已建索引笔记)与 `markdown`(未保存预览,字段为 `source.markdown`,上限 200 000 字符)。当前仅 `format=html` 实现,`pdf`/`docx` 返回 `EXPORT_FORMAT_UNSUPPORTED`
## Markdown → Document AST
解析用 [mistune](https://github.com/lepture/mistune) 的内置 `renderer="ast"`(非自写 `BaseRenderer`),因为 mistune 的行内渲染按字符串拼接、无法承载结构化子节点;ast renderer 直接给出带 `children`/`attrs`/`raw` 的 token 树,`_AstMapper` 只做 token → `DocumentNode` 的搬运,不掺入任何 HTML。插件启用 `table``math``url``task_lists`
fenced code 按语言分流:`mermaid``mermaid` 节点、`function_plot`/`functionplot``function_plot` 节点,其余 → `code_block``attributes.language`)。`node_id` 按遍历顺序 `node_{seq:03d}` 生成,仅渲染内部使用,无需跨请求稳定。
## HtmlExporter
递归渲染 Document AST 为完整 HTML5 文档(`<!doctype html>` + `<head>` 内嵌基础 CSS + `<body>`),标题/正文/元信息文本一律 `html.escape``mermaid``function_plot` 无法静态表达,渲染为占位 `<pre class="mermaid">`/`<pre class="function-plot">` 并记 warning,不静默丢失;`code_theme` 仅作为代码容器 class,不引入 JS 高亮库。无法表示的节点统一 `warnings.append(...)` 跳过。
## 运行生命周期
`queued → running → completed | failed | cancelled`
- 创建时校验:`format` 非 html → `EXPORT_FORMAT_UNSUPPORTED``note` 源不存在 → `EXPORT_SOURCE_NOT_FOUND`404);`markdown` 源为空或超上限 → `EXPORT_OPTIONS_INVALID`
- 内存注册表上限 `MAX_JOBS=100`,超限只淘汰终态任务;满容量且全为活动任务时返回 `EXPORT_CAPACITY_EXCEEDED`429)。
- 后台渲染在解析前后各让出一次执行权,使「创建后立即取消」的 queued 任务能及时进入 cancelled。
- 失败只向公开响应暴露项目错误码与安全消息,详细异常进入日志。
## 产物生命周期
产物写入 `settings.exports_path`(默认 `backend/data/exports/`,可通过 `APP_EXPORTS_PATH` 覆盖,已加入 `.gitignore`),文件名为 `{job_id}.html`,下载 `Content-Disposition``_safe_download_name` 清洗标题得到。`ExportFile` 记录 `sha256``size``expires_at``completed_at + 24h`),过期返回 `EXPORT_FILE_EXPIRED`410)。
## 错误码
```text
EXPORT_SOURCE_NOT_FOUND 404
EXPORT_FORMAT_UNSUPPORTED 400
EXPORT_OPTIONS_INVALID 400
EXPORT_RENDER_FAILED 500
EXPORT_UNSUPPORTED_CONTENT 422
EXPORT_JOB_NOT_FOUND 404
EXPORT_FILE_EXPIRED 410
EXPORT_CAPACITY_EXCEEDED 429
```
## 测试
```powershell
cd backend
uv run pytest -q
```
`tests/test_export.py` 覆盖 Markdown 解析(标题/行内/列表/代码分流/表格/数学)、HTML 渲染(标签 + 转义 + warning)、Service 端到端(note 源与 markdown 源、pdf 拒绝、未知 note、取消、list/get、过期 410)与 `ExportSource` 契约校验。
## 范围外(后续 PR
- PDF / DOCX 导出(`python-docx` 等底层库在 PoC 后冻结,封装在 Exporter Adapter 内)。
- 函数图像绘制(FunctionPlot 结构化模型 + 白名单表达式解析器 + SVG 静态渲染,契约 §10.4/§12)。
- 代码语法高亮(当前仅 CSS class 占位)。
@@ -3,6 +3,8 @@
> 本文档用于团队开发和模块联调,记录 Knowledge Core / Retrieval Core 已经落地的
> 模块边界、数据模型、接口与使用方式,对应分工表中的杨星萱。
> 更新日期:2026-09-02。第一阶段 Knowledge/Retrieval 主链路已经完成,并已接入 Agent Tool Registry;完整后端回归基线为 136 项测试通过。
## 当前实现
当前已经建立第一条可运行的检索链路:
@@ -72,7 +74,7 @@ block_id = "blk_" + sha256(note_id | heading_path | content)[:16]
citation_id = "cit_" + block_id
```
> 注意:MVP 阶段 `note_id` 由相对路径派生,移动文件会改变 ID;后续 `move` 流程会保留原 ID。
> 注意:`note_id` 是稳定业务 ID;移动接口会保留原 ID,文件路径不能代替业务实体 ID。
每个 Block 记录 `heading_path`(章节路径)、`start_offset` / `end_offset`(相对原文的
字符偏移,用于前端跳转高亮)、`content_hash``token_count`
@@ -130,7 +132,7 @@ POST /api/notes
GET /api/notes/{note_id}
PATCH /api/notes/{note_id}
DELETE /api/notes/{note_id}
POST /api/notes/{note_id}/move 501,待定语义
POST /api/notes/{note_id}/move 已实现,保留 note_id
```
创建笔记:
@@ -196,7 +198,7 @@ cd backend
uv run pytest -q
```
当前后端完整测试共 62 个用例通过(单元 + 端到端)。测试通过 `tests/conftest.py` 的 autouse fixture 把
当前后端完整测试共 218 个用例通过(单元 + 端到端)。测试通过 `tests/conftest.py` 的 autouse fixture 把
数据目录/DB/Vault 重定向到临时目录,不读写真实 `backend/data`,任何本机状态下结果确定。
## 配置
@@ -230,4 +232,6 @@ rag.search
- Embedding / Reranker 为轻量实现,后续替换为真实模型(接口不变)。
- 小语料下 hybrid 检索召回偏宽(向量 Top-K 覆盖全部 block),可加相关性阈值收紧。
- 重建为同步 + 全量,后续接入增量索引与异步任务队列。
- 检索 Benchmark 建立
- RAG Benchmark 建立`POST /api/benchmarks/rag/runs` 创建即返回 queued、后台 Task 执行,
通过 SSE 实时推送进度,报告含逐 Case 结果与 `total_cases` / `successful_cases` / `failed_cases` / `failure_rate`
- Agent Benchmark 暂缓,待 Agent Runtime 完成后交付。
@@ -0,0 +1,283 @@
# MCP Bridge 与 Plugin Host 开发说明
> 更新日期:2026-09-02。本文记录第二阶段阶段 C 已实现的本地 stdio MCP Bridge、隔离 Plugin Host、Tool Contract 转换和离线测试方式。阶段 D 的 Plugin Command 与 Settings 已在其独立开发说明中落地。
## 1. 目标与实现状态
阶段 C 的目标是让外部 MCP Server 进入既有 Plugin、Tool、Permission、Agent 和 Trace 链路,同时避免 Agent Runtime、前端或 Benchmark 直接依赖 MCP 原始消息。
当前链路:
```text
Plugin Manifest
→ Plugin Runtime
→ 独立 stdio MCP Server 进程
→ initialize / capability negotiation
→ tools/list 分页发现与校验
→ NotesAgent ToolDefinition
→ Tool Registry / Permission Manager
→ Agent Runtime / Agent Trace
```
已经实现:
- 本地 stdio 子进程启动、关闭和异常退出检测;
- UTF-8、换行分隔的 JSON-RPC 2.0 消息;
- initialize、协议版本与 tools capability 协商;
- `notifications/initialized`
- 分页 `tools/list`
- `tools/call`、业务错误与 JSON-RPC 错误转换;
- 超时和 `notifications/cancelled`
- Tool 命名空间、JSON Schema、权限和 Manifest 集合校验;
- Host 状态查询、重启和异常后的 Tool 自动注销;
- stderr 隔离、环境变量裁剪、消息及结果大小限制;
- 无网络、无密钥的确定性 MCP Fixture。
实现依据为 MCP 官方 [Lifecycle 2025-11-25](https://modelcontextprotocol.io/specification/2025-11-25/basic/lifecycle)、[Transports 2025-11-25](https://modelcontextprotocol.io/specification/2025-11-25/basic/transports) 和 [Tools 2025-11-25](https://modelcontextprotocol.io/specification/2025-11-25/server/tools)。
## 2. 代码位置
```text
backend/app/extensions/mcp.py
stdio 进程、JSON-RPC、MCP 生命周期、发现、调用和 Host 状态
backend/app/extensions/runtime.py
Plugin Manifest、权限、MCP Tool 批量注册/回滚和生命周期集成
backend/app/agent/tools.py
内部 Tool 参数校验、结构化执行错误和线程安全 Registry
backend/extensions/fixtures/mcp-echo/
确定性 stdio MCP Server 与 Plugin Manifest
```
## 3. Plugin Manifest
MCP Plugin 的后端配置示例:
```yaml
id: example-mcp
name: Example MCP
version: 1.0.0
permissions:
- notes.read
contributes:
tools:
- example-mcp.search
backend:
type: mcp
transport: stdio
command: uvx
args: [--isolated, --from, example-mcp==1.2.3, example-mcp]
startup_timeout_seconds: 60
tool_timeout_seconds: 30
```
约束:
- 阶段 C 只接受 `type: mcp``transport: stdio`
- 命令和参数通过数组直接传给 `subprocess.Popen`,不经过 Shell
- Python 包形式的 MCP Server 推荐使用 `uvx --isolated --from <package>==<version> <command>`,固定版本并与 NotesAgent 项目环境隔离;
- Plugin 包内自带且不需要第三方依赖的 Python 脚本可以使用 `python server.py`Node、Rust 等 Server 继续使用各自受控启动器,因此 Host 不强制所有 MCP 都经过 `uvx`
- PATH 中的 executable 使用名称,例如 `uvx``python``node`
- manifest 中带目录的 executable 必须解析到 Plugin 包内部;
- `contributes.tools` 使用 `<plugin_id>.<remote_name>`
- 安装阶段只读 Manifest,不启动第三方进程;
- 完成用户授权后,`enable` 才启动 Host。
## 4. 生命周期
### 4.1 启动
启用 MCP Plugin 时依次执行:
1. 检查 Plugin 声明权限是否全部获得授权;
2. 检查 Manifest 声明的 Tool ID 是否与现有 Registry 冲突;
3. 启动独立 stdio Server
4. 发送 `initialize`
5. 校验协商版本和 `tools` capability
6. 发送 `notifications/initialized`
7. 分页读取 `tools/list`
8. 校验全部 Tool
9. 确认发现集合与 Manifest 完全一致;
10. 将完整集合注册到 Tool Registry
11. Plugin 和 Host 进入 `ready`
任何步骤失败都会注销本轮已注册 Tool、关闭子进程并把 Plugin 标记为 `error`,不会留下半启用状态。
### 4.2 停止与异常退出
停用、卸载或应用关闭时,先注销 Tool,再关闭 stdin,等待 Server 正常退出。超时后依次 terminate 和 kill。
Server 异常退出、stdout 出现非 JSON-RPC 内容或发送超大协议消息时:
- 未完成请求返回 `PLUGIN_HOST_UNAVAILABLE`
- Host 进入 `unhealthy`
- Plugin 进入 `error`
- 对应 Tool 从 Registry 中立即注销;
- 用户可以调用 Host Restart 接口重新协商和发现。
Server 发送 `notifications/tools/list_changed` 时不会直接信任新集合。当前实现先把 Host 标记为不健康并注销旧 Tool,要求通过 Restart 重新执行完整发现与校验。
## 5. Tool Contract 转换
MCP Tool
```json
{
"name": "search",
"description": "Search notes",
"inputSchema": { "type": "object", "properties": {} },
"_meta": { "notesagent/permission": "notes.read" }
}
```
进入系统后转换为:
```json
{
"name": "example-mcp.search",
"description": "Search notes",
"parameters": { "type": "object", "properties": {} },
"permission": "notes.read",
"source": "plugin"
}
```
转换规则:
- 远端名称必须能转换为合法且稳定的项目 Tool ID;
- `inputSchema` 必须是有效的 object JSON Schema
- `additionalProperties``patternProperties` 等动态字段先由完整 JSON Schema 校验,Pydantic 参数载体不会再次误拒绝合法字段;
- `_meta.notesagent/permission` 必须属于项目已知权限;
- Tool 权限必须同时出现在 Plugin Manifest 中;
- Agent 仍通过 Tool Registry 执行参数校验、Permission、超时和 Trace
- MCP `structuredContent` 存在时映射为内部 output;否则保留为受控 `content` 数组;
- MCP `isError: true` 映射为 `MCP_TOOL_CALL_FAILED`
- 结果超过 256 KiB 映射为 `MCP_TOOL_RESULT_TOO_LARGE`
## 6. 隔离与安全边界
当前隔离是“独立进程 + 协议边界”,不是完整的操作系统沙箱。
`uvx` 解决的是 Python 工具依赖隔离:它等价于 `uv tool run`,在 uv 缓存中使用可丢弃的独立虚拟环境。它不会限制 Server 读取用户文件、访问网络、创建子进程或调用系统 API,因此不能代替安全沙箱。当前开发模式下,首次 `enable` 尚未缓存的包可能访问包索引,因此示例使用 60 秒启动上限;生产实现不得依赖该行为,必须在用户确认后的安装/更新阶段预取和验证固定版本,运行阶段只启动已准备好的环境。
已经执行的保护:
- 第三方模块不 import 到 AI Core
- 子进程 `cwd` 固定为 Plugin 包目录;
- 不使用 Shell 拼接命令;
- 不把 Provider API Key、`APP_DB_PATH`、Vault 路径和其他宿主环境变量传入子进程;
- stderr 与 JSON-RPC stdout 分离,stderr 不进入 API 和 Agent Trace
- stdout 只能发送合法 MCP JSON-RPC
- stdout 在读取完整行前即应用有界读取,单条协议消息上限 2 MiB;stderr 也按固定大小分块读取;
- 单次 Tool Result 上限 256 KiB
- MCP Tool 不绕过 Permission Manager 和 Agent Tool Timeout。
- 调用被 Agent 取消时,同时通知 Server 并唤醒本地 pending Queue,阻塞线程不会继续占用线程池直至远端超时。
当前尚未提供容器、受限系统账户、seccomp、Windows AppContainer 或 macOS Sandbox,因此 Plugin 进程仍具有当前操作系统用户授予的一般文件访问能力。正式社区插件分发或“一键安装”前必须完成以下安全门槛:
- 由 Tauri/Rust Host 统一启动进程并提供平台级文件、网络、子进程和资源配额限制;
- 安装/更新时完整展示 executable 与全部参数,明确警告并要求用户主动确认;
- 固定包来源和版本,增加包哈希/签名与可信发布者校验;
- 默认禁止访问 Vault、凭据和宿主环境,只通过声明 Permission 与受控 Host API 授权;
- 关闭 Host 时终止完整进程树,不只结束直接子进程。
在这些门槛完成前,当前 MCP Host 只适用于内置 Fixture、团队可信插件和开发联调;不得把它描述为可以安全执行任意社区代码。上述安装确认要求遵循 MCP [SEP-1024](https://modelcontextprotocol.io/seps/1024-mcp-client-security-requirements-for-local-server-)`uvx` 行为依据 uv 官方 [Using tools](https://docs.astral.sh/uv/guides/tools/) 文档。
后端通过 `APP_ENVIRONMENT` 强制该边界:只有 `development` 可以启动当前未沙箱化的 MCP Host;其他环境返回 `403 MCP_TRUST_APPROVAL_REQUIRED`,且不会创建进程或注册 Tool。后续 Tauri/Rust Host 提供沙箱与绑定完整命令摘要的可信许可后,再替换此临时门禁。
## 7. Host API
```http
GET /api/plugins/{plugin_id}/host
POST /api/plugins/{plugin_id}/host/restart
```
状态响应包含:
```text
plugin_id
backend_type / transport
status
tools_count
started_at / last_seen_at
protocol_version
server_name / server_version
error
```
状态值:
```text
stopped
starting
ready
unhealthy
error
```
Restart 返回 `202 OperationResponse`。接口返回前已完成本地 Host 重启和 Tool 重新发现;`message` 中给出最终 Host 状态。Restart 只用于运行中或异常 Host;用户主动停用、尚未启用或等待授权的 Plugin 返回 `409 PLUGIN_HOST_UNAVAILABLE`,必须通过 Enable 明确启动。
## 8. 离线 Fixture
Fixture 位于:
```text
backend/extensions/fixtures/mcp-echo
```
它提供:
- `mcp-fixture.echo`:返回 structuredContent
- `mcp-fixture.fail`:返回 `isError: true`
- `mcp-fixture.sleep`:验证超时和取消;
- `mcp-fixture.large`:验证结果大小上限;
- `mcp-fixture.environment`:验证宿主 Secret/路径没有进入子进程;
- `mcp-fixture.exit`:验证异常退出、Tool 注销和 Restart。
- `mcp-fixture.command`:作为 Plugin Command 专用 MCP Target,验证 Context 裁剪、Secret 传递和与 Agent Tool 的隔离。
Fixture 的 `tools/list` 使用两页响应,用于覆盖分页发现。测试还会启动缺少 tools capability、返回无效 Schema/initialize result,以及输出超长无换行 stdout 的变体。
## 9. 验证
```powershell
cd backend
uv run python -m compileall -q app
uv run pytest
cd ../frontend
pnpm test
pnpm type-check
pnpm build
```
阶段 C 新增测试覆盖:
- initialize、版本和 capability negotiation
- 分页 `tools/list` 与命名空间映射;
- Permission、JSON Schema 与 Contribution 集合;
- Tool 成功、业务错误、结果过大和超时;
- Agent 取消后 pending 等待线程及时释放;
- `additionalProperties` 动态参数保持 JSON Schema 语义;
- Agent Runtime 调用 MCP Tool 并写入正式 Trace
- Secret/Vault 环境隔离;
- Server 异常退出、Tool 注销和 Host Restart
- 缺少 capability、无效 initialize result、无效 MCP Schema 和超长无换行 stdout
- disabled Plugin 不会被 Host Restart 隐式重新启用;
- OpenAPI 发布 Host 状态和重启路径。
## 10. 当前边界与后续阶段
阶段 C 不包含:
- Streamable HTTP MCP transport
- Resources、Prompts、Sampling、Elicitation 和 MCP Tasks
- Plugin Command 与 Settings Contribution
- Secret Reference 注入;
- Plugin Registry 持久化、签名与社区来源校验;
- 操作系统级沙箱;
- 一键安装前的完整命令展示与确认 UI;
- Tool 列表热更新的无中断替换。
阶段 D 已在当前 Plugin Runtime 上增加 Command、Settings、Secret Contract 和命名空间 Storage,且未修改 Agent 使用内部 Tool Contract 的原则。实现细节见《Plugin Command 与 Settings 开发说明》。
@@ -0,0 +1,117 @@
# Plugin Command 与 Settings 开发说明
> 更新日期:2026-09-03。本文记录第二阶段阶段 D 已实现的 Plugin Command Contribution、Plugin Settings Contribution、Secret 边界及前端闭环。当前回归基线为后端 136 项测试、前端 32 项测试,TypeScript 类型检查和生产构建通过。
## 1. 阶段目标
阶段 D 在阶段 C 的 Plugin Runtime 与隔离 MCP Host 上补齐两类宿主贡献:
- Command:插件声明命令,宿主负责注册、展示、校验、执行和返回白名单 effect;
- Settings:插件声明设置 Schema,宿主负责动态表单 Contract、非敏感值持久化和 Secret 加密引用;
- FrontendPlugin 详情页提供 Host 状态、动态设置、Secret 管理与命令执行,全局命令面板加载 command_palette Contribution。
第三方代码不会导入 FastAPI 进程。操作系统级安全沙箱仍按规划在第三阶段桌面基础集成完成后、Tauri/Rust 沙箱正式构建前处理。
## 2. 包内声明
Plugin 在 `plugin.yaml``contributes.commands``contributes.settings_sections` 声明贡献标识,并分别提供 `commands.yaml``settings.yaml`。安装时宿主要求声明集合与文件内容完全一致,拒绝重复项、越过 Plugin 命名空间的 ID、未声明权限和无效 Schema。
`commands.yaml` 的首批字段包括:
- `command_id`、标题、描述、宿主图标;
- `locations``command_palette``context_menu``toolbar`
- `when` 与允许传入执行器的 Context 字段;
- 参数 JSON Schema、可选权限、执行目标和超时;
- 执行目标必须在宿主白名单 `handler` 与当前插件命名空间的 `mcp_tool` 之间二选一。
- 可选 `secrets` 字段:只声明当前 Command 允许按需读取的 Secret Setting Key,不暴露给前端 DTO。
`settings.yaml` 采用递增 `schema_version`,首批字段类型固定为 `string``number``boolean``select``secret`。宿主会校验默认值、必填项、有限数值边界、Select 选项,以及 Secret 不得携带默认明文;`NaN` 与正负无穷不能用作上下界。
仓库内 `text-tools` 是联调 Fixture,覆盖 Command 和五种 Settings 字段类型。
## 3. Command 运行链路
`CommandRegistry` 只发布处于启用状态的 Plugin Command。Plugin 禁用、Host 不可用或重启时,Command 与 Tool 使用同样的注销/重新注册生命周期,避免前端看到实际不可执行的命令。
执行顺序如下:
1. 查找已注册 Command
2. 使用 Draft 2020-12 JSON Schema 校验 arguments
3. 根据 `when` 检查必要上下文;
4. 仅向执行器传递声明过的 Context 字段;
5. 在超时范围内调用宿主受控 handler,或调用独立的 MCP Command Target
6. 按 effect 类型校验专属 payload、可序列化性和 64 KiB 大小上限;
7. 返回统一 `PluginCommandResult`
首批 effect 为 `none``notification``navigate``refresh``job`。后端分别限制通知级别与消息、宿主路由名、刷新范围和 Job IDPydantic 与 TypeScript 均使用同一判别语义,前端不得把 effect 当作任意代码执行。
Command 执行器通过受控 Resolver 按需读取 `commands.yaml` 已声明且确实属于当前 Plugin Schema 的 Secret;使用 Secret 的 Plugin 还必须声明并获授 `secrets.use` 权限。读取未声明字段返回 `PLUGIN_SECRET_ACCESS_DENIED`,必填 Secret 未配置则返回 `PLUGIN_SECRET_REQUIRED`。Secret 不会并入普通 Settings 字典。Command 审计使用 500 条有界内存队列,仅保留 `command_id``plugin_id`、成功/失败状态、耗时、错误码和时间。arguments、正文选区、文件路径、effect 与 Secret 均不进入审计事件。
MCP Command Target 是专用执行目标,不注册进 Agent `ToolRegistry`,因此模型无法绕过 Command 权限与 Context 裁剪直接调用。宿主通过 `_notesagent` 保留包装传入 `command_id`、已校验 arguments、已裁剪 Context、已校验的非敏感 Settings 和声明过的 Secret。启用阶段只检查目标 `inputSchema` 在顶层 `properties` 中直接声明 `_notesagent: { type: object }`,不自行求解 JSON Schema,也不用空对象伪造业务数据;引用和组合约束可以放在 `_notesagent` 对象内部。执行阶段再用保留的完整 Schema 和官方 Draft 2020-12 Validator 校验真实信封。MCP Server 必须返回结构化的白名单 effect。远程原始错误不直接透传给 HTTP 调用方。插件仍不能把模块路径或 Shell 字符串作为执行器。
Command 与 Tool 的 JSON Schema 只允许当前文档内的 Fragment 引用(`#...`);宿主在注册前递归拒绝 `$ref` / `$dynamicRef` 指向的文件、HTTP 或其他外部资源,避免 Schema 校验触发未授权 I/O。文档内引用使用 Draft 2020-12 Resource Resolver 预检,嵌套 `$id` 创建的新资源及其 Anchor 按各自作用域解析,无法解析的引用在注册阶段返回稳定错误。
## 4. Settings 与 Secret 边界
普通 Settings 以 Plugin 为命名空间持久化到:
```text
APP_DATA_DIR/plugins/settings.json
```
该文件只包含:
- 当前 Schema 版本;
- 非敏感字段值;
- Secret 的确定性定长引用,格式为 `plugin.<sha256(plugin_id\\0setting_key)>`
Secret 写入必须调用专用端点。后端通过 `SecretStr` 接收明文,再交给现有 `EncryptedCredentialStore`;普通 Settings API 只返回 `{ configured: true|false }`,不会返回 Secret 值。`plugin.*` 是保留命名空间,通用凭据 API、Provider 配置、Provider 临时测试凭据和 Provider Resolver 均不得访问,防止覆盖、删除或外发 Plugin Secret。卸载 Plugin 时同时删除普通设置命名空间和对应加密凭据。
没有默认值的 `required` 普通字段必须在启用 Plugin 前配置。Enable 和每次 Command Execute 都会重新检查有效设置;缺失时返回 `PLUGIN_SETTINGS_REQUIRED`,不启动 MCP Host,也不调用 Command handler。
读取持久化引用时,宿主会重新计算并核对 `plugin.<sha256(...)>`,引用不匹配即按损坏存储拒绝处理,不能借由篡改 `settings.json` 读取或删除 Provider 等其他命名空间的凭据。删除单个 Secret 或卸载 Plugin 时先原子更新 Settings 引用,再删除加密凭据;底层删除失败会恢复原引用。多 Secret 卸载使用一次凭据表原子替换,避免分批删除部分删除。
开发阶段凭据文件由本机 Fernet Key 加密。桌面端落地后,应由 Tauri Host 将同一引用语义迁移到 Stronghold 或系统 KeychainHTTP Contract 无需因此改变。
## 5. HTTP 与前端 Service
后端已实现:
```text
GET /api/plugin-contributions/commands?location=command_palette
POST /api/plugin-contributions/commands/{command_id}/execute
GET /api/plugins/{plugin_id}/settings
PUT /api/plugins/{plugin_id}/settings
PUT /api/plugins/{plugin_id}/settings/{key}/secret
DELETE /api/plugins/{plugin_id}/settings/{key}/secret
```
前端 `pluginService` 提供对应方法及 Wire DTOPlugin 详情页据此展示 MCP Host 状态和重启入口、动态生成五类设置字段、独立写入或删除 Secret,并执行带参数的 Plugin Command。全局命令面板打开时获取 `command_palette` 命令;需要必填参数的命令会引导用户进入详情页填写。调用方必须使用服务层,不自行拼接路径。
Secret 明文仅存在于当前密码输入框绑定的组件内存,提交后立即清空;不得写入 Pinia、LocalStorage、普通 Settings 请求或调试日志。前端不会读取 Secret 明文,只展示后端返回的 `configured` 状态。
## 6. 主要错误边界
- Command 未注册、冲突、参数或 Context 无效;
- 执行超时、执行器异常、effect 无效或过大;
- Settings Schema 无效、版本冲突、字段类型/边界错误或运行时必填值缺失;
- Secret 字段不存在、空 Secret、凭据存储异常;
- Settings JSON 根结构或 Plugin 命名空间损坏。
以上错误统一转换为 `ExtensionError` 和稳定业务错误码,HTTP 层不暴露内部堆栈、Secret 或插件返回的原始异常。
## 7. 验证
```powershell
cd backend
uv run pytest
cd ../frontend
pnpm test -- --run
pnpm type-check
pnpm build
```
阶段 D 测试覆盖注册/注销生命周期、位置过滤、参数与 Context 校验、上下文裁剪、设置影响命令执行、声明式 Secret Resolver 与越权拒绝、真实 MCP Command Target 与 Agent Tool 隔离、必填 Secret 传递、外部 Schema 引用拒绝、定长 Secret Reference、篡改引用的跨命名空间阻断、Secret 删除与卸载失败回滚、Provider/通用凭据命名空间隔离、五类设置字段、Schema 版本冲突、Secret 密文与清理、损坏存储、空 Command 列表等无效贡献文件、OpenAPI 路径、前端 Service 请求格式、Host 状态展示和动态 Secret 表单。
生产构建仍会报告现有大 Chunk 警告,不影响构建成功;该问题属于前端按路由和 Markdown 依赖拆包的后续性能任务。
@@ -0,0 +1,124 @@
# 前端写作体验优化开发说明
> 更新日期:2026-09-02。本文所述优化均已进入 `main`;当前前端完整回归基线为 29 项测试通过,TypeScript 检查和 Vite 生产构建通过。
## 1. 本次目标
本次优化聚焦笔记写作主流程,不调整后端接口:
- 将界面中的装饰性 Emoji 统一替换为 Element Plus 图标;
- 将“写作”模式由 Markdown 源码与预览双栏改为单一可视化编辑区;
- 为写作区增加标题、加粗、斜体、有序列表、无序列表工具栏;
- 写作页代码块默认展开为可编辑状态;只读 Markdown 区域使用 Shiki 提供亮暗主题高亮。
## 2. 实现说明
### 2.1 图标体系
新增 `AppIcon.vue` 作为轻量图标出口,页面直接传入 `@element-plus/icons-vue` 组件。侧边栏、文件树、Vault 入口、主题按钮、空状态及扩展列表不再使用 Emoji 表达操作含义。
这样处理后,图标尺寸、颜色和主题状态都由 CSS 统一控制,也避免不同系统 Emoji 字体造成的显示差异。
### 2.2 可视化 Markdown 编辑器
写作模式使用 Milkdown Crepe 渲染 Markdown 文档,磁盘中仍保存标准 Markdown 文本。编辑器监听 Markdown 更新并写回 Pinia 状态,继续复用原有自动保存逻辑。
“源码”模式保留为独立模式,便于需要精确编辑 Markdown 的用户使用;写作模式中不再同时展示 Markdown 源码。
编辑器按当前文件路径重新挂载,保证切换文件、切换源码模式后,展示内容与 Store 中的最新 Markdown 一致。
### 2.3 Markdown 工具栏
写作区顶部提供以下基础格式操作:
- H1 至 H6 标题下拉选择,标题默认使用粗体显示;
- 加粗;
- 斜体;
- 有序列表;
- 无序列表;
- 12 px 至 32 px 字号选择;
- 行内代码与代码块;
- 行内公式与公式块;
- 链接插入。
标题、加粗、斜体和列表工具调用 Milkdown Command 修改当前选区或块级结构,因此能正确处理光标、选区和嵌套列表。工具栏使用常见的 `H``B``I``1.``•` 排版符号,减少图标语义歧义。
标准 Markdown 没有字号语法。字号功能仅在用户已选择文本时生效,并将内容写为兼容 Markdown 的内联 HTML
```markdown
<span style="font-size: 18px">选中的文本</span>
```
Milkdown 自定义插件在写作模式中隐藏 HTML 标记,并通过 ProseMirror Decoration 显示实际字号;切换到源码模式时可以直接看到并修改上述 Markdown 内容。
字号栏同时提供预设下拉框和 `896 px` 数值输入框。输入数值后按 Enter 或点击“应用”即可写入当前选区。标题下拉框提供“正文”选项,用于将标题恢复为普通段落;正文显式使用正常字重,只有 H1 至 H6 默认加粗。
选中文本后出现的 Crepe 浮动格式栏使用应用正文前景色、实色描边和悬浮强调色,避免亮暗主题下图标对比度不足。
浮动栏由 Crepe Tooltip Provider 挂载,不保证位于 Vue scoped 样式容器内部,因此对比度规则使用全局 `.milkdown-toolbar` 选择器,并通过主题变量适配亮暗模式。顶部格式按钮统一在 `pointerdown` 阶段阻止默认焦点迁移并执行命令,确保点击工具栏时不会丢失编辑器选区。
有序列表与无序列表使用相同尺寸、相同线条结构的经典列表符号,仅通过左侧的数字或圆点区分类型。亮色主题下,表格边框使用更高对比度的文本辅助色,列表序号、圆点及任务图标也改用辅助文本色并增加字重。
编辑器左侧加号打开的块菜单已完成中文本地化:
- “文本”分组包含正文、H1 至 H6、引用和分割线;
- “列表”分组包含无序列表、有序列表和任务列表;
- “插入”分组包含图片、代码块、表格和公式块。
代码语言搜索、复制操作、链接编辑及公式确认浮层也统一使用中文文案。
### 2.4 代码块编辑与 Shiki 高亮
代码高亮使用 Shiki 的 JavaScript 正则引擎,并只注册第一阶段常用语言:Markdown、HTML、CSS、JavaScript、TypeScript、JSON、Python、Shell 和 SQL。未知语言回退为 Markdown 语法展示,不阻塞整篇内容渲染。
Shiki 同时生成 `github-light``github-dark` 两套 CSS 变量。主题页提供“跟随主题 / GitHub Light / GitHub Dark”选项,通过根节点 `data-code-theme` 切换对应变量,无需重新执行高亮。偏好写入 `editor-appearance`,内置主题和后续主题包也可通过 `ThemeConfig.code_theme` 指定默认代码主题。
代码主题选择器下方使用真实的 `MarkdownContent` 和 Shiki 渲染 TypeScript 示例,选项变化后立即展示对应 GitHub 高亮效果。该预览只存在于主题设置页,不会恢复写作页代码块的额外预览面板。
代码块容器使用 GitHub 风格的背景、边框、6px 圆角、16px 内边距和等宽字体;相关颜色由 `--color-code-*` Token 控制,方便主题商店覆盖。
高亮结果同时生成 `github-light``github-dark` 颜色变量。根节点的 `data-theme` 变化后由 CSS 选择对应颜色,因此切换主题无需重新解析整篇 Markdown。
写作编辑器中的普通代码块进入文档后直接展开 CodeMirror 编辑区,不再先显示 Shiki 预览,也不再提供“编辑代码/查看高亮”切换,减少一次多余操作。公式块仍由 Milkdown 的 LaTeX 功能负责编辑和渲染。
Shiki 仅应用于:
- AI 对话中的 Markdown 代码块。
- Search、智能体等复用 `MarkdownContent` 的只读 Markdown 代码块。
Markdown HTML 仍在写入 DOM 前经过 DOMPurify 清理。
## 3. 新增依赖
- `@element-plus/icons-vue`:统一界面图标;
- `@milkdown/crepe``@milkdown/kit`:可视化 Markdown 编辑器及命令;
- `shiki``@shikijs/langs``@shikijs/themes``@shikijs/engine-javascript`:代码高亮和按需语言注册。
## 4. 验证记录
`frontend` 目录执行:
```bash
pnpm build
pnpm test
```
验证结果:TypeScript 类型检查与 Vite 生产构建均通过。当前前端完整回归测试共 29 项;其中写作与文件切换相关回归覆盖:
- 顶部工具栏对选区应用加粗;
- 浮动工具栏对选区应用斜体;
- 自定义字号输入写入 Markdown;
- 标题恢复为普通正文;
- 连续切换文件后渲染新文件内容;
- 从文件树连续点击时,活动路径与编辑器内容同步切换;
- 欢迎笔记的异步初始化不会覆盖用户刚点击的文件。
文件切换失效包含两层原因。第一层是旧实现先更新 `currentFilePath`、后等待文件内容,导致编辑器使用新路径和旧内容提前重建;现在改为文件读取成功后一次性提交路径和内容。第二层是工作区欢迎笔记的异步初始化结束后会无条件设为活动文件,可能覆盖用户在此期间的真实点击;现在点击文件时立即同步工作区活动路径,默认初始化仅在用户尚未选择文件且欢迎笔记确实加载成功时提交。文件读取失败时则恢复点击前的活动文件。
本地内置浏览器测试运行时因环境资源路径缺失未能启动,因此本次没有把自动化交互测试列为已通过项。合并前建议人工检查一次工具栏选区操作、文件切换同步,以及跟随主题、GitHub Light、GitHub Dark 三种代码块设置下的显示效果。
## 5. 后续建议
- 根据真实文档规模评估 Milkdown 与只读 Markdown 高亮模块的懒加载拆包;
- 为工具栏补充撤销、重做、引用、行内代码和链接;
- 增加编辑器选区命令与文件切换的组件测试。
@@ -0,0 +1,209 @@
# 前端壳子与接口层开发说明
> 更新日期:2026-09-02
> 适用范围:Vue 3 + TypeScript 页面、Workspace、公共 Service、FastAPI 接口适配和 SSE。
> 文档用途:帮助团队理解当前前端可用能力、模块边界、启动方式和后续页面开发入口。
## 1. 当前实现状态
当前前端已经形成一条可安装、可类型检查、可生产构建和可联调的基础链路:
```text
Vue Router
→ App Shell
→ Pinia Store
→ Service / FastAPI DTO Adapter
→ HTTP 或 SSE
→ FastAPI
```
当前已经落地的页面和公共界面包括:
- Vault 入口页;
- 应用标题栏、主侧边栏、辅助侧边栏和状态栏;
- Workspace 文件树;
- Markdown 写作/源码模式、手动保存和自动保存状态;
- 文件打开、新建、删除和重命名交互壳子;
- Search 查询、筛选、结果列表和 Citation 定位;
- Chat 会话、Provider/Model/Skill 选择和 SSE 输出;
- Agent Run 创建、Trace、取消和权限确认;
- Task 筛选、创建、编辑、状态切换和删除;
- Skill、Plugin 生命周期管理;
- Theme 预览、切换和编辑器 Token 覆盖;
- Settings 的通用、编辑器、Provider、索引、权限和 AI Core 诊断分区;
- 可收起主导航、功能型二级侧栏、状态栏和 `Ctrl+P` 命令面板。
- Milkdown 可视化写作、CodeMirror 源码/代码块编辑、Markdown 格式栏和 Shiki 只读代码高亮;
- OpenAI、DeepSeek、Ollama 预设、自动模型发现和开发阶段加密 API Key 输入;
- 智能体页面、运行状态、事件、工具和权限详情的中文展示。
原统一占位页已经删除,所有已注册业务路由均指向真实页面。Web Workspace 已通过 FastAPI 连接后端配置的单一真实 Vault,不再回退 Mock 数据;Tauri 多 Vault、原生目录选择、Stronghold 和桌面窗口能力仍在桌面容器阶段接入,不影响页面与 Store 的调用边界。
## 2. 目录与职责
```text
frontend/src/
├── components/common/ App Shell、导航、命令面板与扩展公共组件
├── contracts/index.ts UI View Model 与 FastAPI Wire DTO
├── features/ 按页面领域拆分的业务组件
├── features/editor/ 编辑器头部与写作/源码编辑区
├── features/vault/ Vault 入口
├── features/workspace/ Workspace 与递归文件树
├── router/index.ts 页面路由和 Vault Guard
├── services/ HTTP、SSE、DTO 映射和模块 API
├── stores/ Pinia 状态
└── styles/tokens.css Design Token
```
职责约定:
- Component 不直接拼接后端 URL
- Store 负责页面状态和业务操作编排;
- Service 负责 HTTP/SSE 调用以及 Wire DTO 到 View Model 的转换;
- `contracts/index.ts` 同时保留界面模型和以 `Api` 开头的 FastAPI DTO,两者不能混用;
- OpenAPI `/openapi.json` 是后端 Wire Contract 的最终依据。
## 3. 路由与页面壳子
已注册路由:
```text
/
/workspace
/search
/chat
/agent/runs/:runId?
/tasks
/extensions/skills
/extensions/plugins
/themes
/settings
```
除 Vault 入口外,其余路由需要先打开 Vault。全部路由均使用懒加载真实页面组件,既保持首屏包体可控,也避免占位页面掩盖缺失实现。
## 4. 页面实现边界
| 页面 | 当前可用能力 | 主要 Store / Service |
| --- | --- | --- |
| Workspace | 文件树、新建、重命名、删除、打开、编辑、保存、模式切换 | `workspaceStore``editorStore``workspaceService` |
| Search | FTS/Vector/Hybrid、文件夹与标签筛选、结果定位 | `searchStore``searchService` |
| Chat | 会话选择、模型配置、RAG、Skill、SSE、Citation | `chatStore``providerStore``chatService` |
| Agent | Run 配置、Tool 选择、Trace SSE、权限确认、取消 | `agentStore``agentService` |
| Tasks | 状态筛选、CRUD、完成与恢复 | `taskStore``taskService` |
| Skills | 列表、详情、安装、启停、卸载 | `skillStore``skillService` |
| Plugins | 列表、权限确认、安装、启停、卸载 | `pluginStore``pluginService` |
| Themes | 主题预览、应用、字体与行高覆盖、恢复默认 | `themeStore` |
| Settings | 通用、编辑器、Provider、索引、权限、诊断 | `settingsStore``providerStore`、相关 Service |
## 5. Workspace 与编辑器
Workspace 当前由以下组件构成:
```text
WorkspaceView
├── EditorHeader
└── EditorPane
SecondarySidebar
└── FileTreePanel
└── FileTreeNode(递归)
```
文件树把右键目标保存在 `contextTarget`,重命名和删除始终作用于实际被右键的节点,不再依赖当前编辑文件。根目录使用 `/` 表示,新增根级文件时直接写入 Store 顶层数组。
当前 `workspaceService` 是 FastAPI Workspace Adapter。打开 Vault 时只允许后端 `APP_VAULT_PATH` 配置的目录,随后通过 Workspace/Note API 读取真实文件树和 Markdown,并完成文件、目录的新建、重命名、移动、保存和删除。接口错误直接进入统一错误链路,不再用 Mock Fallback 掩盖连接或契约失败。
浏览器不能获得任意本地文件系统权限,因此 Web 模式不提供目录选择和多 Vault 管理。进入桌面端阶段后,由 Tauri Host 实现同一 Service 边界下的原生适配器,组件和 Store 无需感知底层传输变化。
## 6. HTTP 接口层
公共请求由 `apiClient.ts` 处理:
- 支持 GET、POST、PUT、PATCH 和 DELETE
- 使用 `VITE_API_BASE_URL`,并兼容旧的 `VITE_API_BASE`
- 自动附加 `X-Request-Id`
- 将后端统一错误体转换为 `ApiErrorClass`
- 204 响应返回 `undefined`
Service 已适配当前 FastAPI Contract
| 模块 | 主要适配内容 |
| --- | --- |
| Notes | `folder``markdown`、直接 Note 响应和 `{items, page}` |
| Search | 数组筛选字段、`items/page` 响应和 Search View Model 映射 |
| Chat | `provider_id``model``messages` 和 ModelEvent SSE |
| Agent | `input`、秒级 Timeout 字段、Run DTO 和 Permission Decision |
| Skill / Plugin | 嵌套 `manifest`、安装 `package_path` 和 Plugin Permission PUT |
| Provider | Provider Type、Capability 数组、模型列表包装和 Test 响应 |
| Task | `due_at`、分页响应和当前后端支持字段 |
| Index | `all/notes/vectors` Scope、Job 与状态 DTO |
| System | `/health``/api/status` 的真实响应字段 |
界面模型中存在的展示字段不能直接发送给后端。例如 Task View Model 的 `priority``source` 当前只是界面层字段,Service 创建与更新请求不会把它们发送给不支持这些字段的 FastAPI Contract。
## 7. SSE
`SseClient` 同时服务于 Chat 和 Agent Event
- 使用与普通 HTTP 相同的 API Base URL
- 支持 POST Chat Stream 和 GET Agent Event Stream
- 使用 `TextDecoder` 处理 UTF-8 增量字节;
- 在网络分片之间保留 `event` 和多行 `data` 状态;
- 以空行作为单个 SSE Event 的结束标志;
- 识别 `Done``RunCompleted``RunFailed``RunCancelled`
- 支持 AbortController 主动取消。
Chat Store 已从定时器模拟输出切换为真实 `/api/chat` SSE。默认离线联调配置为:
```text
provider_id = mock
model = mock-1
```
## 8. 环境和启动
```powershell
cd frontend
pnpm install --frozen-lockfile
pnpm dev
```
联调前在另一个终端启动后端:
```powershell
cd backend
uv run uvicorn app.main:app --reload --host 127.0.0.1 --port 8000
```
生产构建:
```powershell
cd frontend
pnpm build
```
## 9. 当前验证基线
```text
pnpm build passed
pnpm test 29 passed
uv run pytest 136 passed
preview smoke HTTP 200
git diff --check passed
```
当前前端使用 Vitest 执行 Store、Workspace API Adapter、SSE 恢复游标、Plugin Command/Settings Service、文件树、编辑器组件、智能体标签、轻量动效约束、Markdown 对比度 Token、scoped CSS 选择器约束和 Shiki GitHub 双主题测试;`pnpm build` 同时执行 `vue-tsc -b` 与 Vite 生产构建。后端测试出现过 `.pytest_cache` 无法写入的 Windows 权限警告,不影响 136 项测试结果,也不涉及产品代码。
Vite 当前会提示 Chat 与 Workspace 的部分异步 Chunk 超过 500 kB,这是 Milkdown、CodeMirror、KaTeX 和 Shiki 等编辑/渲染依赖带来的性能优化项,不影响构建成功或功能正确性;进入桌面打包前应通过手动分包或更细粒度动态加载继续优化。
浏览器可视化冒烟在本次执行环境中因浏览器运行资源缺失未能启动;HTTP 冒烟已确认前端入口、后端健康检查与 OpenAPI 均能访问。进入合并验收前,仍建议团队在本机打开各路由完成一次人工视觉检查。
## 10. 后续开发要求
- 新页面文件与路由修改必须在同一提交中出现;
- 新增或修改接口时同步更新 FastAPI DTO、Service 映射和接口文档;
- 不允许用 `as any` 或错误返回类型掩盖 Contract 差异;
- SSE 相关变更需要覆盖跨 Chunk、CRLF、多行 data、终态事件和取消;
- Workspace 接入 Tauri 后,需要增加路径规范化、写入失败恢复和外部修改冲突测试;
- 页面新增交互必须经过键盘、空状态、加载状态、错误状态和窄窗口检查;
- Workspace 的 Milkdown 写作模式与 CodeMirror 源码模式共享同一 Markdown 数据源;后续修改编辑器时不得改变 Store/Service 边界,并必须保留文件切换、自动保存和选区格式化回归测试。
@@ -0,0 +1,113 @@
# 前端视觉与轻量动效优化开发说明
> 更新日期:2026-08-30
> 适用范围:全局 Design Token、App Shell、功能页、卡片、表单、弹窗和轻量交互动效
## 1. 目标
本轮优化不改变页面功能和前后端契约,主要解决原界面层级偏平、组件间距不统一、交互反馈不足的问题,同时为后续主题商店 CSS 注入保留稳定边界。
设计原则:
- 颜色、圆角、阴影、间距和速度继续使用 CSS 变量;
- 页面与弹窗动画只改变 `opacity``transform`
- 不使用背景模糊、连续粒子、视差、复杂 SVG 或大范围布局动画;
- 不使用 `transition: all`,只声明需要变化的属性;
- 尊重系统 `prefers-reduced-motion` 设置;
- 主题只需覆盖现有 Token,不需要了解组件内部动画实现。
## 2. 全局视觉基线
更新 `tokens.css`
- 调整四级圆角和阴影,使卡片、弹窗与导航层次更清楚;
- 调整标题栏、状态栏和双侧栏尺寸;
- 统一更短的运动时间与缓动曲线;
- 增加键盘 `focus-visible` 焦点环;
- 为 checkbox、radio 和 range 使用主题强调色;
- 窄窗口下收缩辅助侧栏与展开导航宽度;
- 在减少动态效果模式下,把动画和过渡缩短到近似即时完成。
更新 `features.css`
- 功能页增加受控内容宽度、响应式留白和低强度主题渐变;
- 卡片增加边框、阴影与最多 2px 的悬浮位移;
- 按钮、输入框、Badge、空状态和通知统一交互反馈;
- 设置页导航改为分段式卡片导航;
- 弹窗与页面增加一次性淡入和轻微位移动画。
## 3. 页面与布局优化
### 3.1 App Shell
- 主侧栏增加明确的 Active 标记、悬浮反馈和宽度过渡;
- 辅助侧栏统一为次级 Surface,并改善标题与标签层级;
- 标题栏应用名改为轻量胶囊标识;
- 状态栏强化状态点和窄窗口降级;
- 命令面板增加轻量入场、圆角、阴影和列表反馈。
### 3.2 业务页面
- Search 结果卡片增加左侧强调线和统一内容宽度;
- Chat 增加消息容器、头像层级、Citation 悬浮反馈和 Composer 顶部阴影;
- Agent Tool 选择卡增加选中状态,Trace 使用轻量时间线;
- Task 完成按钮增加主题化状态反馈;
- Settings 行在悬浮时提供背景提示;
- Workspace 空状态与 Vault 入口增加清晰的层级和一次性入场动画。
## 4. 动效性能边界
允许的常规动效:
```text
opacity
transform: translate / scale / rotate
background-color
border-color
color
box-shadow
```
默认禁止:
```text
transition: all
backdrop-filter / filter 模糊
持续改变 width / height / margin / padding 的动画
无限循环的装饰动画
全屏高频渐变或粒子动画
```
状态栏 Spinner 和 AI Core 检查状态点属于有明确状态含义的循环动画,并会被 `prefers-reduced-motion` 全局规则降级。
## 5. 主题商店接入约定
自定义主题优先覆盖以下 Token
- `--color-background-*`
- `--color-surface-*`
- `--color-text-*`
- `--color-accent-*`
- `--color-border-*`
- `--color-code-*`
- `--shadow-*`
- `--radius-*`
- `--motion-*`
主题 CSS 不应给通配选择器增加动画,不应重新启用高成本滤镜,也不应覆盖 `prefers-reduced-motion` 的降级规则。若主题需要完全静态的界面,可把三个 `--motion-*` Token 设置为接近 0ms。
## 6. 验证
新增 `styles/motion.spec.ts`,防止全局样式重新引入 `transition: all`、高成本模糊滤镜或布局型页面入场动画,并约束 Markdown 表格与列表使用独立的高对比度主题变量。
当前验证结果:
```text
pnpm test 9 files / 23 tests passed
pnpm build passed
git diff --check passed
```
生产构建仍有已有的大 Chunk 警告,主要来自 Milkdown、CodeMirror、KaTeX 和 Shiki;本轮样式及动效未增加 JavaScript 动画库或运行时依赖。
本轮已完成样式静态检查、自动化测试和生产构建。由于本机内置浏览器运行资源路径缺失,亮色/暗色主题的人工页面巡检需在 PR 验收环境补做。
@@ -0,0 +1,107 @@
# 模型提供商、协议适配与模型路由开发说明
> 更新日期:2026-09-04。阶段 E 实现记录。本地小模型的实际安装与多模态队列属于阶段 F;本阶段保留并测试可注入的本地后端接口。
## 1. 设置与凭据
设置 → 模型提供商 → 新增 Provider 提供可搜索的 logo 预设网格,包含 DeepSeek、Kimi、阿里云百炼、智谱 GLM、火山方舟、硅基流动、百度千帆、腾讯混元、MiniMax、阶跃星辰,以及 OpenAI Chat / Responses、Anthropic 和 Ollama。图标打包到前端,使用时不请求第三方图片服务;来源和许可见前端 assets/providers 目录。
预设返回 `preset_id``logo_id``name``provider_type``base_url``requires_credential``description``capabilities`。能力标签表示预设接入范围,不保证该账号的每个模型支持全部能力。厂商专用媒体协议、Coding Plan 和海外地域需要使用对应地址,不能仅凭厂商名称推断协议兼容。
预设和自定义服务都可以直接输入 API Key。每个新配置分配独立 Credential ID,避免同厂商多账号相互覆盖。明文只留在密码输入框和专用请求中,提交、失败、切换预设及关闭时清空;密钥不进入 Pinia、localStorage、Provider 配置响应或模型路由。
凭据继续使用独立的 `PUT /api/credentials/{credential_id}` 和 Fernet 开发存储。`plugin.*``mcp.*` 是保留命名空间。桌面端阶段仍需要把主密钥管理迁移到 Stronghold。保存密钥与保存 Provider 是两个请求,Provider 保存失败时可能留下未引用的加密凭据,可通过凭据删除接口清理。
Provider 配置和 Credential ID 写入 SQLite `provider_configs`,重启后恢复。Mock 为内置 Provider,不能编辑或删除。PATCH 已支持变更 `provider_type` 并重新创建 Adapter;Base URL 限制为不带用户信息、查询或 fragment 的 HTTP(S) 地址。
## 2. 协议适配
支持的协议是 OpenAI Chat Completions、OpenAI-Compatible、OpenAI Responses、Anthropic Messages 和 Ollama。Agent、Chat、Skill 仍只依赖内部 `ModelRequest` / `ModelEvent` / `ProviderTurn`,不直接解释厂商协议。
Adapter 负责消息及 Tool 历史转换、增量文本、可用的 reasoning delta、工具参数片段、usage、终止与统一错误。外部错误正文不原样返回;HTTP 鉴权、限流、超时、无效数据、流中断分别映射为内部错误。取消继续传播并关闭上游连接,不触发第二次本地推理。
`GET /api/providers/{provider_id}/models` 用于发现模型。模型列表不等于每个模型的能力承诺;部分厂商或代理不提供 `/models` 时,允许直接手动输入模型 ID。连接测试验证模型发现接口,不代表每一种媒体模型已完成真实推理验收。
## 3. 三类模型路由
接口:
| 方法 | 路径 | 用途 |
| --- | --- | --- |
| GET | `/api/model-routing` | 读取配置和本地后端状态 |
| PUT | `/api/model-routing` | 带版本更新三类模型绑定 |
| POST | `/api/models/embeddings` | 文本向量,返回来源和回退原因 |
| POST | `/api/media/transcriptions` | 附件转写作业 |
| GET | `/api/media/transcriptions/{job_id}` | 获取转写作业 |
| POST | `/api/media/speaker-matches` | 两个音频附件的声纹相似度 |
设置 → 索引与模型分别选择 Embedding、音频转文本和声纹匹配。三种绑定互相独立,可使用不同提供商、模型、密钥和 API 路径。
GET / PUT 响应:
```json
{
"config": {
"version": 1,
"embedding": {
"provider_id": "provider_example",
"model": "your-embedding-model",
"endpoint": "/embeddings",
"dimensions": null
},
"transcription": null,
"speaker_matching": null
},
"local_backends": [
{"capability": "embedding", "status": "placeholder", "message": "当前为 hash-v1 占位向量"},
{"capability": "transcription", "status": "not_installed", "message": "阶段 F 接入"},
{"capability": "speaker_matching", "status": "not_installed", "message": "阶段 F 接入"}
]
}
```
PUT body 只提交 `config` 的内容。`version` 为读取时的版本,成功递增;并发更新返回 `MODEL_ROUTING_VERSION_CONFLICT`。绑定为空表示使用本地后端。删除仍被路由引用的 Provider 返回 `PROVIDER_IN_USE`,须先解除绑定。
本阶段三类远程路由使用 `openai_chat` / `openai_compatible` 的 Bearer HTTP 配置,endpoint 只能是该提供商下的路径。Responses、Anthropic 和 Ollama 原生协议不冒充上述媒体协议;Ollama 用户需要另建兼容 HTTP 配置才能用于当前远程 Embedding 接口。
调用规则:无绑定 → 本地接口;有绑定 → API → 校验结果 → 失败或无效时调用本地接口。Provider 停用、密钥缺失、鉴权失败、限流、网络超时及无效结果均可回退;用户取消不会回退。附件不存在、大小非法等输入错误直接返回,不把用户输入错误当成模型故障。
## 4. Embedding 与索引一致性
请求使用 `model``input``encoding_format: float`;只有明确配置维度时才发送 `dimensions`。按最多 32 条分批请求,全部批次有效才使用 API 结果。校验返回数量、连续唯一 index、维度一致性、有限数值、非零范数,并 L2 归一化。维度可为 1–16384,不截断、补零或混用不同模型的向量。
返回 `vectors``source``model_id``dimensions``fallback_reason`。远程空间 ID 由完整 API URL、模型和实际维度生成;即使维度相同,不同模型的空间也不同。
笔记索引始终保留现有 hash/sqlite-vec 本地基线,远程向量写入独立 `routed_block_vectors` 表。远程查询只搜索对应空间,并要求覆盖全部当前 Block。API 失败、索引缺失、不完整或损坏时使用完整本地索引。切换模型、URL、维度后应在设置中重建全部索引。旧空间与当前文本不会混合打分,删除笔记或重建索引会通过外键清理远程向量。
当前远程侧索引采用 SQLite JSON 向量和精确余弦扫描,复杂度 O(Block 数量 × 维度),适用于当前小型 Vault;后续大规模索引需替换为按空间隔离的 ANN。网络等待发生在数据库写事务之前,当前仍会增加保存或重建延迟,异步索引队列尚未接入。全量重建先在内存中准备全部向量,再使用一个 SQLite 事务更新元数据、FTS、本地与远程向量及任务关联;取消或失败只回滚索引事务,不再覆盖整库文件。准备阶段保留旧索引可查询,代价是内存同时容纳本次重建的向量。
OpenAI Compatible 流中,工具名称可能分片返回。适配器在本轮输出结束后发送完整工具名及已缓冲参数,避免把名称片段当作工具 ID;文本与推理内容仍逐片发送。
无 API 时使用的 `HashEmbeddingProvider` 是确定性特征哈希占位实现,**不是已集成的小型语义模型**。真实本地 Embedding 可实现既有 `EmbeddingProvider` 接口注入。
## 5. 音频与声纹边界
转写默认请求 `/audio/transcriptions`multipart 字段 `model`、可选 `language``file`,响应必须包含非空字符串 `text`。已有纯文本附件和 Host 旁路 `.txt` 导入保留,来源标记 `sidecar`,不伪称 ASR。转写作业新增 `source``fallback_reason`;回退失败的作业记录 `LOCAL_MODEL_NOT_INSTALLED` 等明确错误。作业目前同步执行、限量保存在内存中,不是持久化异步队列。
声纹匹配使用**本项目自定义 HTTP 契约**,默认 `/audio/speaker-matches`multipart 字段 `model``file``reference_file`;响应为 `{"score": 0.85}`,score 必须为有限的 0–1 数值。公共入口只接受 `attachment_id``reference_attachment_id`,不接收任意文件路径。此接口用于一对一声纹比对,不等同于 pyannote 说话人分离,也不声称任意国内厂商原生支持该路径。
媒体文件限制 1 字节至 25 MiB,API 响应限制 16 MiB,单次请求超时 30 秒。文件从后端受控附件目录读取,使用结束或取消时关闭句柄。
`LocalSpeechBackend` 提供 `transcribe``match` 接口。阶段 E 默认 `PendingSpeechBackend` 明确报告未安装;阶段 F 接入 faster-whisper、pyannote.audio 及模型资源后替换。当前 `diarization=true` 明确返回失败作业 `DIARIZATION_NOT_IMPLEMENTED`,不会静默忽略。视频解码、TTS、视频生成及厂商专用异步媒体协议不在本次交付内。
## 6. 官方协议依据与验证
国内通用地址核对依据:[阿里云百炼兼容接口](https://help.aliyun.com/zh/model-studio/compatibility-of-openai-with-dashscope)、[百度千帆兼容接口](https://cloud.baidu.com/doc/qianfan/s/Hmh4suq26)、[腾讯混元兼容接口](https://cloud.tencent.com/document/product/1729/111007)、[MiniMax 文本接口](https://platform.minimaxi.com/docs/guides/text-generation)、[阶跃星辰通用与套餐地址区别](https://platform.stepfun.com/docs/zh/step-plan/overview)、[火山方舟 API](https://www.volcengine.com/docs/82379/1795150)、[智谱开放接口](https://docs.bigmodel.cn/api-reference/文件-api/文件列表)。模型 ID 以账号实际开通列表为准,不写死“最新模型”。
流式事件依据:[OpenAI Responses streaming](https://platform.openai.com/docs/api-reference/responses-streaming)、[Anthropic streaming](https://platform.claude.com/docs/en/build-with-claude/streaming)。音频请求依据:[SiliconFlow transcription](https://docs.siliconflow.com/en/api-reference/audio/create-audio-transcriptions)。
自动化验证使用虚构凭据、本地附件、httpx.MockTransport 和可注入本地模型,覆盖流式 Tool/Usage/取消、错误映射、回退、索引空间隔离、版本冲突、重启恢复和界面凭据行为。没有使用真实 API Key 或向厂商发送推理请求。审阅修复并同步主分支后验证:后端全量 447 项、前端 76 项测试通过,Vue/TypeScript 类型检查和生产构建通过,浅色/深色预设页面与路由保存经过浏览器检查,git diff --check 通过。后端仅保留既有 Starlette 测试客户端弃用提示,前端保留既有大 bundle 提示。
```powershell
cd backend
uv run pytest -q -p no:cacheprovider
cd ../frontend
pnpm test
pnpm build
```
@@ -0,0 +1,141 @@
# 独立 MCP Server 配置中心开发说明
> 更新日期:2026-09-03。本文记录第二阶段 C.1 的完整实现;独立 MCP Server Registry 与 Plugin 自带 MCP Host 是两个并列入口。
## 1. 已实现范围
- 独立 Server 的创建、读取、版本化编辑、删除和 Tool 摘要查询;
- `stdio`、Streamable HTTP 和旧版 HTTP+SSE 三种 Transport
- stdio 可执行文件、参数、普通/加密环境变量,以及 HTTP URL、普通/加密 Header
- 配置摘要确认、连接测试、启停、异常状态与最近一次测试结果;
- MCP initialize、`tools/list``tools/call`、取消与动态 Tool 注册,名称为 `mcp.{server_id}.{tool}`
- Streamable HTTP Session、协议版本 Header、JSON/SSE POST 响应、可选 GET 事件流和 `Last-Event-ID` 重连;
- 旧 HTTP+SSE 的 endpoint 事件与消息 POST,并强制消息地址和配置地址同源;
- 前端表单/JSON 双模式、三种模板、高风险变更确认及请求期 Secret 输入。
Streamable HTTP 按 MCP 当前规范实现;SSE 仅用于兼容旧 Server,不应作为新部署首选。
## 2. 配置、版本与 Secret
普通配置原子写入 `APP_DATA_DIR/mcp/servers.json`。更新请求必须携带读取到的 `version`;版本过期返回 `409 MCP_SERVER_VERSION_CONFLICT`,避免多个页面互相覆盖。改变 Transport、命令、URL、Header、环境变量或权限后,旧授权和测试结果立即失效。
`backend/data/mcp/` 是本机运行数据,包含连接配置、授权状态和第三方进程工作目录,不属于团队共享配置。`.gitignore` 忽略整个目录以及 `server.json``servers.json` 文件名;不得强制添加到 Git。提交前检查暂存文件清单,不要将本地密钥、连接配置或运行数据推送到远程。
Secret 使用带类型和键名哈希的 `mcp.*` 内部 ID 写入 Fernet 凭据存储。查询 API 只返回环境变量或 Header 是否配置,不返回明文。删除 Server 或移除 Secret 键会同步清理密文。前端密码框提交后立即清空;用户主动粘贴到 JSON 的密钥仅在当前编辑会话中暂存,解析后从 JSON 中移除,不写入 localStorage、普通配置或日志。
环境变量的凭据 ID 使用区分大小写的 v2 名称规则,Header ID 保持大小写不敏感。更新配置按凭据 ID 的差集删除密文,因此 `Authorization` 改为 `authorization` 不会丢失认证信息。启动时对无歧义的旧环境变量凭据原子迁移密文,不覆盖新 ID 已有的值;若旧配置把 `TOKEN``token` 合并存到了同一个 ID,无法推断原来的两个值,会保留旧密文、停用连接并要求重新录入和测试。删除服务器时也会清理这些保留的旧密文。
### 2.1 JSON 导入与 API Key 填写
前端支持 NotesAgent 完整/精简配置、单个 `command / args / env` 配置,以及只含一个服务器的 `mcpServers` 包装。JSON 与表单之间切换会补齐数组、对象及超时默认值,并校验字段类型。批量导入暂不支持;后端配置接口仍只接收 NotesAgent DTO,兼容转换发生在前端。
可以先声明 `secret_environment_keys`,保存后在服务器卡片的密码框填写密钥;也可以把密钥放进 JSON 的 `environment` 或通用配置的 `env`。前端会将已声明的敏感变量,以及名称含 API Key、Token、Secret、Password、Authorization、Cookie、Credential 的常见字段拆出:普通配置请求只包含键名,密钥另经 Secret API 加密保存。其他敏感字段必须显式声明,不能只依赖名称识别;命令与参数中不要携带密钥。
例如 MiniMax 的输入结构如下,占位值需在自己的本地页面替换,不要把真实密钥贴进聊天或提交到 Git:
```json
{
"name": "MiniMax Coding Plan",
"command": "uvx",
"args": ["--index-url", "https://pypi.tuna.tsinghua.edu.cn/simple", "--with", "mcp<2", "minimax-coding-plan-mcp", "-y"],
"environment": {
"MINIMAX_API_HOST": "https://api.minimaxi.com",
"MINIMAX_API_KEY": "<在本地填入新密钥>"
},
"secret_environment_keys": ["MINIMAX_API_KEY"],
"startup_timeout_seconds": 120,
"tool_timeout_seconds": 300
}
```
旧版前端将 `environment.MINIMAX_API_KEY``secret_environment_keys` 原样一起发送,触发后端“普通与敏感变量不可同名”的校验。这是配置保存失败,不是模型服务返回的鉴权失败。现在在前端拆分两类请求,后端仍保留互斥校验。
导入兼容规则:`env` 转为 `environment``timeout` 作为启动超时;`sse_read_timeout` 作为工具等待上限,不保留其原客户端 SSE 读取超时语义。启动超时范围为 1–120 秒,工具超时为 1–300 秒。URL 必须是纯地址,不能使用 Markdown 链接,JSON 中不能包含 `\_` 这样的非法转义。
另一个已修复的失败原因是运行时适配层复用了 `PluginBackend` 的整数超时与 60 秒启动上限,导致合法的 120 秒或小数超时配置在保存返回、读取或测试时失败。独立 Server 现在使用专门的 Bridge 适配模型,保留自己的浮点超时范围,不改变 Plugin 清单原有约束。已有的 120 秒记录可直接读取,无需删库重建。
配置保存成功但后续 Secret 写入失败时,窗口保留服务器 ID、新版本和未写入的密钥。点击保存会更新同一服务器并重试,不重复创建记录;取消会清除未保存密钥,已经保存的服务器和凭据不会回滚。错误信息显示在配置窗口内。保存配置不会自动运行第三方进程,仍需确认、测试和启用。
暂存的 Header Secret 与已保存凭据使用一致的大小写规则:将 `Authorization` 改为 `authorization` 不会丢弃尚未保存的值,提交时采用当前声明名。重新输入同一 Header 的值会覆盖旧草稿;真正删除声明才清除草稿。环境变量仍区分大小写,不会把 `TOKEN` 的草稿转交给 `token`
跨 Registry 与凭据存储的删除以“失败后可重试”为顺序约束:先原子清理密文,再提交新版本或删除 Registry 记录。凭据存储失败时保留原版本和 Server 记录,避免出现返回 500 但配置已提交、版本无法重试或密文失去清理入口的状态。
写入或删除 Secret 会先停用正在运行的连接、注销动态 Tool,并撤销当前配置的测试通过状态;必须使用新凭据重新测试后才能启用。这样页面展示的凭据状态不会与运行中进程实际持有的旧凭据不一致。
## 3. 启用与运行时规则
一次连接按以下顺序执行:
1. 用户检查服务端生成的连接摘要并确认当前摘要;
2. 后端临时连接,完成 initialize 和 `tools/list` 后关闭连接;
3. 只有当前摘要测试成功,启用操作才会启动长期连接并注册 Tool;
4. 停用、删除、超时或异常退出会注销 Tool 并关闭连接。
运行期连接异常或 Tool 列表变化时,Registry 会在同一生命周期临界区内标记不可用、注销 Tool,并从 Bridge 移除 Host。每次启动分配独立的连接代次;失败回调取得锁后先核对代次,旧连接延迟到达的回调不能停用新连接。停用、测试结束及关闭服务时撤销对应代次。
所有独立 MCP API 都通过工作线程执行,包括新增、确认授权和读取接口。虽然部分操作不直接访问网络,但仍可能等待正在测试或启动的连接持有的锁,不能在 FastAPI 事件循环上同步等待。
HTTP Header 中 `Host``Content-Type``MCP-Session-Id` 等协议保留项不可由配置覆盖。URL 不允许内嵌凭据或 Fragment。旧 SSE 返回的 POST endpoint 必须与初始 URL 同源,防止认证 Header 被转发到其他站点。
启动与 Tool 请求超时会同时应用于业务等待和底层 HTTP 请求;旧 SSE 的 endpoint 等待也使用启动超时。非主动结束的旧 SSE 事件流视为 Host 不可用,宿主随后注销 Tool。注册表加载时逐条校验 Server ID、配置字段、Transport 组合和摘要格式,损坏记录统一返回 `MCP_REGISTRY_INVALID`
两种 HTTP Transport 共用有界 SSE 行解析器:按响应字节块检查未完成行及当前事件的累计大小,再扩展缓冲区,不依赖 `iter_lines()` 先缓存完整行。持续无换行输入也会及时触发上限;解析兼容跨块 UTF-8、首行 BOM、LF/CR/CRLF、多行 data 和事件间计数重置。`tests/test_mcp_sse_limits.py` 覆盖这些边界,防止仅在完整行生成后检查大小。
stdio 命令不经过 Shell,管道、重定向和命令拼接不会被解释。Windows 使用新进程组并通过 `taskkill /T` 回收子树;POSIX 使用独立 session/process group 并向进程组发信号。Python 阶段仍无法提供文件、网络、系统调用或操作系统版本差异下的绝对隔离保证。
`uvx` 模板使用 `--isolated`、明确的 `--from` 和固定包版本。它只能隔离依赖,不能替代安全沙箱。非开发环境仍拒绝启动 stdio Server,并返回 `403 MCP_SANDBOX_REQUIRED`;远程 HTTP Transport 不创建本机子进程,但仍要求摘要确认和成功测试。第三阶段前的 C.5 将冻结 Tauri/Rust 沙箱设计。
## 4. 接口
```text
GET /api/mcp/servers
POST /api/mcp/servers
GET /api/mcp/servers/{server_id}
PUT /api/mcp/servers/{server_id}
DELETE /api/mcp/servers/{server_id}
GET /api/mcp/servers/{server_id}/tools
POST /api/mcp/servers/{server_id}/trust
POST /api/mcp/servers/{server_id}/test
POST /api/mcp/servers/{server_id}/enable
POST /api/mcp/servers/{server_id}/disable
PUT /api/mcp/servers/{server_id}/secrets/{key}?kind=environment|header
DELETE /api/mcp/servers/{server_id}/secrets/{key}?kind=environment|header
```
完整字段、状态和错误码见《第二阶段接口契约-开发版》。协议实现参考 MCP 官方的 [Transports](https://modelcontextprotocol.io/specification/2025-11-25/basic/transports) 与 [Lifecycle](https://modelcontextprotocol.io/specification/2025-11-25/basic/lifecycle)。
## 5. 验证
```powershell
cd backend
uv run pytest -q tests/test_mcp_registry.py tests/test_extension_core.py
cd ../frontend
npm run type-check
npm test
npm run build
```
后端测试使用无需网络或真实密钥的 stdio Fixture,以及 `httpx.MockTransport` 驱动的确定性 HTTP/SSE Server Fixture。覆盖摘要授权、Secret 不回显、版本冲突、生产门禁、重启恢复、Streamable HTTP Session/Header/工具调用及旧 SSE 同源校验;新增覆盖旧回调隔离、路由线程卸载、凭据大小写差异与旧密文迁移。前端覆盖模板切换、JSON 默认值与格式兼容、明文拆分、模式切换、部分保存失败重试、取消清理、测试失败和删除确认。此处的 MiniMax 配置转换测试使用假密钥,不等同于真实 MiniMax 网络调用验证。
## 6. 后续边界
### 本轮 P1/P2 修复验收
| 审阅问题 | 修复方式 | 回归验证 |
| --- | --- | --- |
| P1:新增或授权等待生命周期锁时阻塞事件循环 | 独立 MCP 路由统一交给工作线程 | `test_mcp_lifecycle_lock_contention_keeps_event_loop_responsive`:分别阻塞 create/trust,在锁释放前仍能执行健康检查 |
| P2:旧失败回调误停新连接 | 回调在锁内核对连接代次 | `test_old_failure_callback_cannot_stop_replacement_host`:旧回调排队期间重启连接,释放锁后新连接仍可用,当前代次的失败仍正确停用 |
| P2:精简 JSON 切换模式或编辑保存报错 | 运行时校验、默认值补全、统一转换 | `configuration.spec.ts``McpServersView.spec.ts`:精简 JSON、编辑版本、模式切换和 Secret 保存重试 |
| P2:Header 大小写改名删除凭据 | 按规范化凭据 ID 而非原始键名计算差集 | `test_header_case_only_rename_preserves_secret` |
| P2:大小写不同的环境变量覆盖同一凭据 | 区分大小写的 v2 ID,带迁移标记的旧密文迁移 | `test_environment_secrets_are_case_sensitive_and_delete_independently` 及 legacy migration 测试,包含删除后不复活旧密钥 |
| P1:SSE 无换行输入在大小校验前无限缓冲 | 在行拼接前校验字节数与事件累计大小 | `test_mcp_sse_limits.py`,包括小块持续输入和跨块换行 |
| P2:Header 大小写改名丢失未保存密钥 | 草稿使用规范化名称匹配,并重新绑定当前声明名 | `configuration.spec.ts` 与页面保存回归测试 |
这些修复不放宽 stdio 的 JSON-RPC 校验。第三方程序向 stdout 打印普通日志造成的握手失败,应由服务端调整输出或使用不打印日志的启动入口处理。
### 后续工作
- 增加真实第三方 Server 的兼容矩阵;确定性 Fixture 只能证明宿主协议行为,不能代表所有实现兼容;
- C.5 在第二阶段开发与测试完成后、第三阶段桌面端实现前冻结沙箱 Contract;
- 第三阶段将 stdio 进程创建和 Secret 托管迁移至 Tauri/Rust Host 与 Stronghold/系统 Keychain。
+166
View File
@@ -0,0 +1,166 @@
# CI/CD 细则(团队开发版)
> 本文档规定 NotesAgent 在 Gitea 上的持续集成、构建产物、发布和回滚要求。当前仓库尚未提交 Gitea Actions 工作流,因此本文首先作为落地流水线时的统一规范;流水线启用前,Pull Request 仍须人工执行同等检查。
> 更新日期:2026-09-01。当前阶段的 CD 指“生成可验证的候选构建与发布产物”,不包含把后端自动部署到公网环境。
## 1. 目标与原则
CI/CD 用于尽早发现依赖锁文件失效、类型错误、测试回归、前后端契约不一致和生产构建失败。流水线应遵守以下原则:
- 以 Gitea 为唯一远程和流水线入口;
- `main` 始终保持可安装、可测试、可构建;
- 安装依赖时使用锁文件,避免流水线与开发机解析出不同版本;
- 未通过必需检查的提交不得合入 `main`
- 外部模型、真实 API Key 和用户本地数据不得成为基础 CI 的前置条件;
- 缓存只用于加速,不得影响构建结果;删除缓存后流水线仍应成功;
- 测试、构建和发布步骤使用最小权限,敏感信息不得写入日志或产物。
## 2. 运行环境基线
| 组件 | CI 要求 | 说明 |
| --- | --- | --- |
| Python | 3.12 | 项目最低支持 3.11,CI 使用团队推荐版本 |
| uv | 当前稳定版,并在日志中输出版本 | 按 `backend/uv.lock` 安装后端依赖 |
| Node.js | 22 LTS | 满足前端环境要求并保持 Runner 兼容性 |
| pnpm | 10 | 按 `frontend/pnpm-lock.yaml` 安装前端依赖 |
| 操作系统 | Linux Runner 为基础门禁 | 桌面端启用后再增加 Windows、macOS 构建矩阵 |
Runner 镜像或 Action 的大版本必须固定。升级 Python、Node.js、uv、pnpm 或基础 Action 时,应使用独立的 `chore/` 分支,并完整运行前后端检查。
## 3. 触发规则
| 事件 | 必须执行 | 用途 |
| --- | --- | --- |
| Pull Request 指向 `main` | 文档检查、后端测试、前端测试、类型检查、生产构建 | 合并门禁 |
| 推送到 `main` | 全量检查、集成冒烟、保存候选构建 | 验证合并结果 |
| 推送功能分支 | 至少执行受影响模块的检查 | 尽早反馈;不得替代 PR 全量门禁 |
| 推送 `v*` 标签 | 全量检查、构建、校验和、发布候选产物 | 正式发布入口 |
| 手动触发 | 可选择全量回归或重新生成候选产物 | 发布前复核和故障恢复 |
纯文档变更可以跳过前后端耗时任务,但必须执行文档链接检查和 `git diff --check`。只有可靠的路径检测结果才能判定为纯文档变更;锁文件、工作流、构建配置和接口契约变更一律按代码变更处理。
## 4. Pull Request 必需检查
建议将以下 Job 名称固定为 Gitea 分支保护所要求的状态检查:
| Job | 必需命令或行为 | 通过标准 |
| --- | --- | --- |
| `docs-check` | `git diff --check`,检查仓库内 Markdown 相对链接 | 无空白错误、无失效本地链接 |
| `backend-test` | `uv sync --frozen`、编译检查、`uv run pytest` | 依赖锁有效且测试全部通过 |
| `frontend-test` | `pnpm install --frozen-lockfile``pnpm test` | 依赖锁有效且测试全部通过 |
| `frontend-typecheck` | `pnpm type-check` | 无 TypeScript/Vue 类型错误 |
| `frontend-build` | `pnpm build` | Vite 生产构建成功 |
| `integration-smoke` | 启动 FastAPI,验证健康检查和关键本地链路 | 服务可启动,响应与契约符合预期 |
后端 Job 的基准命令:
```bash
cd backend
uv sync --frozen
uv run python -m compileall -q app
uv run pytest
```
前端 Job 的基准命令:
```bash
cd frontend
pnpm install --frozen-lockfile
pnpm test
pnpm type-check
pnpm build
```
`integration-smoke` 应使用 Mock Provider、临时数据库和临时附件目录,不访问 OpenAI、DeepSeek 或其他外部服务。测试结束后必须关闭服务并清理临时数据。
## 5. 路径与模块检查规则
- 修改 `backend/**``backend/uv.lock` 或后端配置时,必须运行 `backend-test``integration-smoke`
- 修改 `frontend/**``frontend/pnpm-lock.yaml` 或前端配置时,必须运行全部前端 Job。
- 修改 `docs/contracts/**`、FastAPI 路由、DTO、SSE 事件或前端 Service 类型时,必须同时运行前后端全量检查。
- 修改 `.gitea/**`、根目录工程配置或依赖版本时,必须运行所有 Job。
- 修改 `docs/**` 以外且无法明确归类的文件时,默认运行所有 Job。
路径过滤只用于减少无关重复任务,不得造成关键检查缺失。若无法可靠判断影响范围,应执行全量流水线。
## 6. 凭据与敏感信息
- 基础 CI 不配置真实模型 API Key,Provider 相关测试统一使用 Mock 或请求桩。
- 确需发布签名或访问受保护服务时,凭据只保存在 Gitea Actions Secrets 中,不写入仓库、工作流参数、缓存或构建产物。
- 来自外部分支或不受信任 Pull Request 的任务不得读取发布凭据。
- Secret 名称表达用途和环境,例如 `RELEASE_SIGNING_KEY`;禁止使用含义模糊的 `KEY1``TOKEN2`
- 日志中禁止输出请求头、完整 Token、API Key、用户笔记内容和本地凭据存储内容。
- 生产凭据与测试凭据分离,并遵循最小权限、定期轮换和可撤销原则。
前端构建时注入的变量会进入静态资源,不能用于保存秘密。只有明确可公开的配置才允许使用 Vite 客户端环境变量。
## 7. 缓存与产物
可以缓存 uv 下载缓存和 pnpm Store,缓存键至少包含操作系统、运行时版本和对应锁文件哈希。不得缓存:
- `backend/.venv/`
- `frontend/node_modules/`
- `backend/data/`、测试数据库和用户附件;
- `.env`、API Key、本地凭据库或签名材料。
普通 PR 不上传可执行发布包,只保留必要的测试报告和前端构建日志。`main` 或版本标签的候选产物应记录提交 SHA,生成 SHA-256 校验和,并设置明确的保留期限;非正式候选产物建议保留 14 天。
## 8. 分支保护与合并门禁
Gitea 中的 `main` 应启用以下保护:
- 禁止普通成员直接推送和强制推送;
- 要求 Pull Request 审阅通过;
- 要求第 4 节列出的适用状态检查成功;
- Head 更新后使旧审阅和旧检查失效,必须针对最新提交重新检查;
- 对话和审阅意见处理完成后才允许合并;
- 优先使用 squash 或 rebase 保持主线清晰,具体方式遵循 [Git 使用细则](Git使用细则-团队开发版.md)。
临时绕过门禁只允许用于明确的仓库级故障。绕过者需要记录原因、影响、补验计划,并在恢复后立即补跑全部检查。
## 9. 发布流程
当前阶段按以下顺序生成发布候选:
1. 从已通过全部检查的 `main` 提交确定发布 SHA
2. 更新版本号、变更说明和必要文档;
3. 创建形如 `v0.2.0` 的语义化版本标签;
4. 标签流水线重新执行全部测试和生产构建;
5. 对产物执行本地启动或安装冒烟测试;
6. 生成校验和,并把版本、提交 SHA、构建环境和已知限制写入发布说明;
7. 人工确认后在 Gitea 发布页面公开产物。
Tauri 桌面端接入后,发布流水线再增加 Windows、macOS 和 Linux 构建矩阵、平台签名及安装包验证。在签名、更新通道和回滚方案准备完成前,不启用面向用户的自动更新。
## 10. 回滚与热修复
- 尚未公开的候选产物直接标记为失败,不覆盖同一版本的已有产物;修复后递增预发布编号或版本号。
- 已发布版本出现问题时,优先停止分发并回退到最近一个已验证版本。
- 代码修复从 `main` 创建 `hotfix/<模块>-<问题>` 分支,通过完整门禁后合并并发布补丁版本。
- 禁止重写已公开版本标签或用新文件替换旧版本同名产物。
- 回滚或热修复完成后,在 `docs/retrospectives/` 记录原因、影响、处置过程和防复发措施。
## 11. 流水线失败处理
1. 先确认失败是否可在本地使用相同锁文件和命令复现;
2. 判断是代码、测试、依赖、Runner 还是外部基础设施问题;
3. 代码或测试问题由当前 PR 修复,不通过重跑掩盖不稳定测试;
4. Runner 或 Gitea 故障应记录日志和时间,恢复后针对同一 Head 重新执行;
5. 连续出现的偶发失败必须作为缺陷处理,明确负责人并增加稳定性修复;
6. 修复流水线本身时,不得顺便降低测试范围或绕过既有门禁。
## 12. 落地清单
首次创建 `.gitea/workflows/` 时,应逐项确认:
- [ ] 工作流只使用 Gitea Runner 支持且来源可信的 Action
- [ ] Python、Node.js、uv 和 pnpm 版本符合本规范;
- [ ] 后端和前端依赖均以 frozen 模式安装;
- [ ] 必需 Job 名称与 `main` 分支保护一致;
- [ ] Mock 测试不依赖外部模型服务和真实凭据;
- [ ] 缓存键包含锁文件哈希,缓存内容不含用户数据或秘密;
- [ ] PR、`main`、版本标签和手动触发行为分别验证;
- [ ] 失败任务能返回非零退出码,后续发布步骤不会继续;
- [ ] 候选产物包含提交 SHA、校验和和保留期限;
- [ ] 团队成员能够按本文档在本地复现全部门禁。

Some files were not shown because too many files have changed in this diff Show More