Compare commits
41
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
902e784df3 | ||
|
|
0ed604d432 | ||
|
|
b97e4c14be | ||
|
|
04ed7901bb | ||
|
|
90041ad424 | ||
|
|
ab70678f3f | ||
|
|
483cb4ec8c | ||
|
|
82ad81fdf0 | ||
|
|
3ef97a472e | ||
|
|
7e823d4fd3 | ||
|
|
28122a0e2d | ||
|
|
e7518a4fea | ||
|
|
bb273ec724 | ||
|
|
ccf1900996 | ||
|
|
ceb15445ed | ||
|
|
27b4435bba | ||
|
|
751667d2c8 | ||
|
|
55ad897c56 | ||
|
|
ca55923559 | ||
|
|
d0efc90155 | ||
|
|
e48baccb6c | ||
|
|
8aa3435d8b | ||
|
|
5e5cba2698 | ||
|
|
1c07d9ed16 | ||
|
|
3ce5df4188 | ||
|
|
97c9ea2cb4 | ||
|
|
cf4e65bb9d | ||
|
|
41b3ae0e41 | ||
|
|
3f25ffab83 | ||
|
|
411eadb431 | ||
|
|
1c9b351507 | ||
|
|
bb015058c0 | ||
|
|
60c1e9b45a | ||
|
|
a8b2160ac1 | ||
|
|
38359c1874 | ||
|
|
138a72fef6 | ||
|
|
097477ca2c | ||
|
|
4e16a635aa | ||
|
|
654dc0511a | ||
|
|
6eb1d12580 | ||
|
|
e673fba5b3 |
@@ -8,3 +8,6 @@ dist/
|
||||
.vscode/
|
||||
.idea/
|
||||
*.log
|
||||
.ruff_cache/
|
||||
.coverage
|
||||
htmlcov/
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
# 那拉提文旅智能问答:前后端基础壳子
|
||||
# 那拉提文旅智能问答
|
||||
|
||||
基础工程骨架:Vue 3 + TypeScript + Vite 前端;FastAPI + Pydantic + SQLAlchemy 后端;PostgreSQL + pgvector 数据库配置。
|
||||
Vue 3 + TypeScript + Vite 前端;FastAPI + Pydantic + SQLAlchemy 后端;PostgreSQL + pgvector。当前后端提供知识管理、来源、文档、人工审核、关键词/向量/RRF 检索、文本与向量发布、历史记录及认证/角色权限,问答仍为占位接口。
|
||||
|
||||
## 启动数据库
|
||||
需要 Docker Desktop,在项目根目录运行:
|
||||
需要可用的 Docker Engine(Windows 可使用 Docker Desktop),在项目根目录运行:
|
||||
```bash
|
||||
docker compose up -d db
|
||||
```
|
||||
@@ -15,11 +15,27 @@ cd backend
|
||||
py -3.12 -m venv .venv
|
||||
.\.venv\Scripts\Activate.ps1
|
||||
pip install -r requirements.txt
|
||||
Copy-Item .env.example .env
|
||||
if (!(Test-Path .env)) { Copy-Item .env.example .env }
|
||||
alembic upgrade head
|
||||
uvicorn app.main:app --reload
|
||||
```
|
||||
API 文档:http://127.0.0.1:8000/docs
|
||||
|
||||
Linux / macOS 可使用以下等价命令(需要 Python 3.11+;`.env` 已存在时不要覆盖):
|
||||
```bash
|
||||
cd backend
|
||||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
pip install -r requirements.txt
|
||||
test -f .env || cp .env.example .env
|
||||
alembic upgrade head
|
||||
uvicorn app.main:app --reload --host 127.0.0.1
|
||||
```
|
||||
|
||||
配置始终从 `backend/.env` 读取,环境变量优先。必须先迁移再启动;应用启动只检查数据库版本,不再执行 `create_all`。旧版 `knowledge_items` 会被非破坏性接管;首次迁移前仍须备份。迁移会保留原条目及导入快照,不推测缺失的采集时间。旧版 `verified` 条目因缺少完整审核记录,会转为 `pending`,原状态保存在导入日志中。
|
||||
|
||||
认证迁移为 `0004_auth_sessions`。知识管理、来源、文档及检索历史现要求管理员登录;迁移不创建默认账号。运行 `python -m app.manage create-admin --username local_admin`,通过隐藏输入设置独立密码。默认 Cookie 要求 HTTPS;本机 HTTP 联调须在未入库的 `.env` 显式设置 `AUTH_COOKIE_SECURE=false`(仅 development/test 允许)。完整来源、CSRF、字段和权限说明见 [认证接口契约](docs/AUTH_API.md)。当前数据库最新迁移为 `0006_vectors`;文本构建/发布见 [INDEX_API.md](docs/INDEX_API.md),本地模型及向量发布见 [VECTOR_API.md](docs/VECTOR_API.md)。
|
||||
|
||||
## 启动前端(另开终端)
|
||||
```powershell
|
||||
cd frontend
|
||||
@@ -29,19 +45,87 @@ npm run dev
|
||||
Vite 通常使用 http://localhost:5173。
|
||||
|
||||
## 已有接口
|
||||
- 注册/登录/退出、当前会话、资料修改、改密与管理员用户列表,详见 [认证接口契约](docs/AUTH_API.md)
|
||||
- GET `/api/health`
|
||||
- GET `/api/ready`:检查数据库连接
|
||||
- POST `/api/chat`:占位接口,尚未接入检索和 LLM
|
||||
- GET `/api/knowledge`
|
||||
- POST `/api/knowledge`
|
||||
- GET `/api/knowledge/page`、GET/PATCH/DELETE `/api/knowledge/{id}`
|
||||
- POST `/api/knowledge/{id}/review`、GET `/api/knowledge/{id}/history`
|
||||
- GET/POST `/api/sources`、GET/PATCH/DELETE `/api/sources/{id}`、GET `/api/sources/{id}/history`
|
||||
- GET/POST `/api/documents`、GET/PATCH/DELETE `/api/documents/{id}`、GET `/api/documents/{id}/history`
|
||||
- POST `/api/retrieval`:已核验知识的关键词检索
|
||||
- GET `/api/retrieval/runs`、GET `/api/retrieval/runs/{run_id}`:检索历史和原始结果
|
||||
- `/api/indexes`:管理员构建/查询切分快照、发布/查询当前版本、查询 chunks 和发布历史,详见 [接口契约](docs/INDEX_API.md)
|
||||
|
||||
### 阶段 2 接口约定
|
||||
|
||||
- 所有日期均为带时区的 ISO 8601。输入拒绝未知字段,标题/正文不能全为空白,网址只接受 HTTP(S);可选网址空字符串按未填写处理。
|
||||
- `GET /api/knowledge` 保留数组响应,支持 `q`、`category`、`verification_status`、`limit`(默认 100,最大 200)和 `offset`。新页面应使用 `/api/knowledge/page`,避免把首批条目数量当作总数。
|
||||
- 分页响应为 `{items, total, page, page_size}`;`page` 从 1 开始,`page_size` 默认 20、最大 100。来源、文档、知识分页和各类历史接口均支持分页。
|
||||
- 来源分页支持 `q`;文档分页支持 `q`、`source_id`、`category`;知识分页支持 `q`、`category`、`verification_status`、`source_id`、`document_id`。`q` 为字面文本子串搜索,尚非论文中的关键词检索基线。
|
||||
- 创建知识沿用 `title/category/content/source_url`,新增可选 `source_id/document_id/collected_at`;状态固定为 `pending`。关联文档时自动解析其来源,填写的来源或网址若冲突会报错。直接来源链接的旧调用仍受支持。
|
||||
- 来源创建字段:`name`、可选 `source_url`、`notes`。文档创建字段:`source_id/title/content/collected_at`,可选 `category/notes`。文档来源归属固定,换来源需新建文档。
|
||||
- PATCH 请求必须携带读取时获得的 `expected_version`,并至少提交一个修改字段;DELETE 使用查询参数 `expected_version`。版本过期返回 409,防止覆盖他人修改。
|
||||
- 审核请求字段见下方示例。`verification_status` 只允许 `pending/verified/rejected`;核验人和说明必填。通过审核前必须有来源网址及真实采集时间。这里只记录人工结论,不会自动核实网页。
|
||||
- 修改条目、关联来源或关联文档都会将相关知识转为 `pending`,清除当前审核字段并记录失效原因。历史快照保留当时的条目、来源和文档。
|
||||
- 删除采用软删除;被有效文档或知识引用的来源、被有效知识引用的文档不能删除(409)。已删除记录不出现在列表,但历史仍可查看。
|
||||
- 错误响应使用 `detail`:参数错误 422、记录不存在 404、版本/引用冲突 409、数据库暂不可用 503。
|
||||
|
||||
审核请求示例(仅为占位测试数据):
|
||||
```json
|
||||
{
|
||||
"expected_version": 1,
|
||||
"verification_status": "verified",
|
||||
"reviewer": "test-reviewer",
|
||||
"verification_notes": "示例审核说明,实际使用须填写真实核验依据"
|
||||
}
|
||||
```
|
||||
|
||||
响应新增 `version/updated_at/is_deleted` 及关联、采集、审核字段;完整类型以 `/openapi.json` 和 `/docs` 为准。前端已接入登录/注册、Cookie 会话恢复、个人中心、账户设置、角色路由守卫、角色菜单、知识录入和来源登记/查询;文档、审核、完整分页和浏览器端 PostgreSQL 联调仍需按接口验收。
|
||||
|
||||
### 关键词检索基线
|
||||
|
||||
先安装更新后的依赖并执行 `alembic upgrade head`,再启动后端。检索请求示例:
|
||||
```json
|
||||
{"query": "那拉提旅游资源", "mode": "keyword", "top_k": 5, "category": "旅游资源"}
|
||||
```
|
||||
|
||||
`query` 为 1–500 字符;`top_k` 默认 5、最大 20;`category` 可省略。当前只接受 `keyword` 模式。纯标点、空白或仅含停用词的查询返回 422。
|
||||
|
||||
- 仅使用 `verified` 且未删除、采集/审核字段齐全的知识。关联来源/文档须仍有效,来源网址须与登记来源一致;`pending/rejected` 不参与分词、IDF 统计或结果排名。
|
||||
- 使用固定 `jieba==0.42.1` 搜索分词,关闭 HMM,加入项目词“那拉提”,采用字符级 NFKC 和大小写折叠、小型固定停用词表。标题词频权重 2,正文权重 1;BM25 参数 `k1=1.2/b=0.75`,IDF 为 `ln(1+(N-df+0.5)/(df+0.5))`。重复查询词计一次,同分按知识 ID 升序。
|
||||
- 返回 `run_id/status/query_terms/corpus_size/corpus_fingerprint/total_hits/results/elapsed_ms/config` 等字段。命中包含知识及关联文档/来源的 ID 与版本、标题、来源网址、分数、命中词和最长 240 字符的原文片段;`snippet_start/end` 是原文字符偏移,结束位置不包含在片段内。分数仅用于排序,不是事实正确率或置信度。独立网址的旧条目可没有文档 ID 或来源名称。
|
||||
- 关键词基线的检索单元仍是已审核知识条目;该接口的显示片段独立于新增的持久化 chunks,尚无向量索引。没有匹配证据时返回 `status=no_evidence` 和空结果,当前不会调用模型生成答案。
|
||||
- 每次成功检索(含零命中)保存不可变结果快照、实际检索耗时、语料指纹和算法参数。`elapsed_ms` 包含读取、分词、排序和片段生成,不包含记录写入和网络时间。历史记录用于复核,不能当作仍然有效的当前知识;依据变更后的新检索会重新应用审核条件。
|
||||
- `GET /api/retrieval/runs` 使用标准分页参数;`GET /api/retrieval/runs/{run_id}` 读取当时结果。当前知识修改、失效或删除不会改写历史结果。检索记录保存在数据库,运行日志和实际查询不要提交到 Git。
|
||||
- 本基线按请求计算语料词频,面向小规模资料集:筛选后的语料最多 5000 条、标题与正文合计最多 200 万字符。超过容量明确返回 503,不截断语料或伪造空结果;可缩小类别范围,大规模索引留到后续实现。
|
||||
|
||||
前端 `src/api.ts` 已提供带类型的检索及历史调用;混合检索证据展示已接入问答页。chunk 关键词、精确向量与 RRF 的独立调用及比较口径见 [混合检索契约](docs/HYBRID_API.md)。LLM 问答与真实领域评测仍待后续阶段。
|
||||
|
||||
前端包括问答、知识库管理、来源与更新、实验评估四个页面。实验评估页面没有虚构数据;所有页面明确标记当前功能状态。
|
||||
|
||||
注意:当前没有实现 Embedding、RAG、网络搜索、后台更新任务、用户认证或生产级数据库迁移。下一步建议先验证前后端联通,再按模块实现。`.env` 不要提交到 Git。
|
||||
注意:当前已实现本地 Embedding、文本/向量快照、精确向量及 RRF 混合检索;`/api/chat` 仍是占位接口,DeepSeek RAG 调用暂缓,网络搜索、后台更新任务和真实领域评测尚未实现。管理接口由服务端认证与角色校验;审核表单的核验人仍是人工输入字段。服务保持仅本机访问,完整生产部署、Docker 数据库和浏览器端 Cookie 联调尚未验收。`.env` 不要提交到 Git。
|
||||
|
||||
### 后端测试和迁移回滚
|
||||
|
||||
在 `backend` 目录安装 `requirements-dev.txt` 后执行:
|
||||
```bash
|
||||
ruff check .
|
||||
ruff format --check .
|
||||
pytest -q
|
||||
```
|
||||
|
||||
测试需要真实 PostgreSQL + pgvector:设置 `TEST_DATABASE_URL`(`postgresql+psycopg://...`),或另行 `pip install pgserver` 使用可选的本机临时实例。`pgserver` 仅用于测试,不是应用运行依赖;外部测试数据库账号需有建 Schema 权限且已启用 vector 扩展。每个测试使用独立 Schema,不会清空业务表。测试使用的占位正文和审核人不代表真实资料。
|
||||
|
||||
迁移命令为 `alembic upgrade head`,可重复执行。仅撤回向量快照时,备份后配合旧后端执行 `alembic downgrade 0005_chunk_indexes`,保留文本、账号、知识及检索历史;新模式历史的旧代码兼容边界见 [HYBRID_API.md](docs/HYBRID_API.md)。继续撤回切分/索引版本时,备份后执行 `alembic downgrade 0004_auth_sessions`,保留账号、会话、知识和检索记录。继续撤回认证时,备份后执行 `alembic downgrade 0003_retrieval_runs`,删除用户/会话/预算表,保留知识和检索历史。继续执行 `alembic downgrade 0002_catalog` 还会删除检索历史。需要撤回阶段 2 时,先备份,再执行 `alembic downgrade 0001_baseline`;它保留基础知识字段,但会删除新增来源、文档、审核字段与所有历史记录,软删除的基础条目也会重新可见。降级须配合对应旧版后端,当前应用会拒绝旧 Schema。恢复完整数据应使用备份。不要在有业务数据的数据库执行 `downgrade base`。
|
||||
|
||||
## 团队协作文档
|
||||
|
||||
- [项目章程](docs/PROJECT_CHARTER.md)
|
||||
- [开发计划](docs/DEVELOPMENT_PLAN.md)
|
||||
- [阶段 1 详细开发计划](docs/PHASE_1_DEVELOPMENT_PLAN.md)
|
||||
- [共同开发规范](docs/CONTRIBUTING.md)
|
||||
- [系统架构](docs/ARCHITECTURE.md)
|
||||
- [任务分工](docs/TASKS.md)
|
||||
|
||||
@@ -3,3 +3,13 @@ APP_ENV=development
|
||||
API_PREFIX=/api
|
||||
DATABASE_URL=postgresql+psycopg://nalati:change-this-password@127.0.0.1:5432/nalati_rag
|
||||
CORS_ORIGINS=http://localhost:5173,http://127.0.0.1:5173
|
||||
AUTH_ALLOWED_ORIGINS=http://localhost:5173,http://127.0.0.1:5173,http://localhost:8000,http://127.0.0.1:8000
|
||||
# 默认要求 HTTPS;仅本机 HTTP 联调可显式改为 false。
|
||||
AUTH_COOKIE_SECURE=true
|
||||
AUTH_SESSION_SECONDS=28800
|
||||
# 可选 CPU 模型;先在独立环境准备固定修订缓存,再显式启用。API 永不下载权重。
|
||||
EMBEDDING_ENABLED=false
|
||||
EMBEDDING_MODEL=small
|
||||
EMBEDDING_CACHE_DIR=
|
||||
EMBEDDING_BATCH_SIZE=8
|
||||
EMBEDDING_THREADS=2
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
[alembic]
|
||||
script_location = %(here)s/migrations
|
||||
prepend_sys_path = %(here)s
|
||||
path_separator = os
|
||||
@@ -0,0 +1,122 @@
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Request, Response
|
||||
from sqlalchemy import delete, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.db.session import get_db
|
||||
from app.models.auth import AuthSession, User
|
||||
from app.schemas.auth import (
|
||||
LoginInput,
|
||||
PasswordChangeInput,
|
||||
ProfileInput,
|
||||
RegisterInput,
|
||||
SessionRead,
|
||||
UserRead,
|
||||
)
|
||||
from app.services.auth import (
|
||||
PASSWORD_HASHER,
|
||||
CurrentSession,
|
||||
check_password,
|
||||
clear_cookie,
|
||||
dummy_hash,
|
||||
issue_session,
|
||||
limit_auth_requests,
|
||||
require_admin,
|
||||
require_origin,
|
||||
require_session,
|
||||
)
|
||||
|
||||
router = APIRouter(tags=["auth"])
|
||||
|
||||
|
||||
@router.post("/auth/register", response_model=UserRead, status_code=201)
|
||||
def register(
|
||||
payload: RegisterInput,
|
||||
request: Request,
|
||||
db: Session = Depends(get_db),
|
||||
origin=Depends(require_origin),
|
||||
):
|
||||
"""Registration never accepts role, ID, password hash or credential version fields."""
|
||||
limit_auth_requests(db, request)
|
||||
user = User(
|
||||
username=payload.username,
|
||||
password_hash=PASSWORD_HASHER.hash(payload.password.get_secret_value()),
|
||||
nickname=payload.nickname,
|
||||
email=str(payload.email) if payload.email else None,
|
||||
role="user",
|
||||
)
|
||||
db.add(user)
|
||||
db.commit()
|
||||
return user
|
||||
|
||||
|
||||
@router.post("/auth/login", response_model=SessionRead)
|
||||
def login(
|
||||
payload: LoginInput,
|
||||
request: Request,
|
||||
response: Response,
|
||||
db: Session = Depends(get_db),
|
||||
origin=Depends(require_origin),
|
||||
):
|
||||
limit_auth_requests(db, request, payload.username)
|
||||
user = db.scalar(select(User).where(User.username == payload.username).with_for_update())
|
||||
encoded = user.password_hash if user else dummy_hash()
|
||||
if not check_password(encoded, payload.password.get_secret_value()) or user is None:
|
||||
raise HTTPException(401, "用户名或密码错误")
|
||||
if PASSWORD_HASHER.check_needs_rehash(user.password_hash):
|
||||
user.password_hash = PASSWORD_HASHER.hash(payload.password.get_secret_value())
|
||||
return issue_session(db, user, request, response)
|
||||
|
||||
|
||||
@router.get("/auth/me", response_model=SessionRead)
|
||||
def me(current: CurrentSession = Depends(require_session)):
|
||||
return current.response()
|
||||
|
||||
|
||||
@router.post("/auth/logout", status_code=204)
|
||||
def logout(
|
||||
response: Response,
|
||||
current: CurrentSession = Depends(require_session),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
db.delete(current.session)
|
||||
db.commit()
|
||||
clear_cookie(response)
|
||||
|
||||
|
||||
@router.patch("/users/me", response_model=UserRead)
|
||||
def update_profile(
|
||||
payload: ProfileInput,
|
||||
current: CurrentSession = Depends(require_session),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
if not payload.model_fields_set:
|
||||
raise HTTPException(422, "请至少填写一个可修改字段")
|
||||
for key, value in payload.model_dump(exclude_unset=True, mode="json").items():
|
||||
setattr(current.user, key, value)
|
||||
db.commit()
|
||||
return current.user
|
||||
|
||||
|
||||
@router.post("/auth/change-password", status_code=204)
|
||||
def change_password(
|
||||
payload: PasswordChangeInput,
|
||||
response: Response,
|
||||
current: CurrentSession = Depends(require_session),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
if not check_password(current.user.password_hash, payload.current_password.get_secret_value()):
|
||||
raise HTTPException(400, "当前密码不正确")
|
||||
current.user.password_hash = PASSWORD_HASHER.hash(payload.new_password.get_secret_value())
|
||||
current.user.credential_version += 1
|
||||
db.execute(delete(AuthSession).where(AuthSession.user_id == current.user.id))
|
||||
db.commit()
|
||||
clear_cookie(response)
|
||||
|
||||
|
||||
@router.get("/admin/users", response_model=list[UserRead], dependencies=[Depends(require_admin)])
|
||||
def list_users(
|
||||
limit: int = Query(100, ge=1, le=100),
|
||||
offset: int = Query(0, ge=0),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
return list(db.scalars(select(User).order_by(User.id).offset(offset).limit(limit)))
|
||||
@@ -1,7 +1,15 @@
|
||||
from fastapi import APIRouter
|
||||
|
||||
from app.schemas.chat import ChatRequest, ChatResponse
|
||||
|
||||
router = APIRouter(tags=["chat"])
|
||||
|
||||
|
||||
@router.post("/chat", response_model=ChatResponse)
|
||||
def chat(payload: ChatRequest) -> ChatResponse:
|
||||
# TODO: retrieval -> context assembly -> LLM -> evidence/source checks
|
||||
return ChatResponse(answer=f"已收到问题:{payload.question}。当前为基础壳子,尚未接入知识检索或大语言模型。", status="stub", sources=[])
|
||||
return ChatResponse(
|
||||
answer=f"已收到问题:{payload.question}。当前为基础壳子,尚未接入知识检索或大语言模型。",
|
||||
status="stub",
|
||||
sources=[],
|
||||
)
|
||||
|
||||
@@ -0,0 +1,105 @@
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Response
|
||||
from sqlalchemy import or_, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.db.session import get_db
|
||||
from app.models import Document, KnowledgeItem, Source
|
||||
from app.schemas.catalog import AuditRead, DocumentCreate, DocumentRead, DocumentUpdate, Page
|
||||
from app.services.auth import require_admin
|
||||
from app.services.catalog import (
|
||||
ensure_version,
|
||||
get_item,
|
||||
history,
|
||||
invalidate_knowledge,
|
||||
page_result,
|
||||
record_event,
|
||||
update_values,
|
||||
)
|
||||
from app.services.indexing import catalog_write_guard
|
||||
|
||||
router = APIRouter(
|
||||
prefix="/documents",
|
||||
tags=["documents"],
|
||||
dependencies=[Depends(require_admin), Depends(catalog_write_guard)],
|
||||
)
|
||||
|
||||
|
||||
@router.get("", response_model=Page[DocumentRead])
|
||||
def list_documents(
|
||||
q: str = Query("", max_length=200),
|
||||
source_id: int | None = Query(None, gt=0),
|
||||
category: str | None = Query(None, max_length=80),
|
||||
page: int = Query(1, ge=1),
|
||||
page_size: int = Query(20, ge=1, le=100),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
statement = select(Document).where(Document.is_deleted.is_(False))
|
||||
if source_id is not None:
|
||||
statement = statement.where(Document.source_id == source_id)
|
||||
if category is not None:
|
||||
statement = statement.where(Document.category == category)
|
||||
if q:
|
||||
statement = statement.where(
|
||||
or_(
|
||||
Document.title.icontains(q, autoescape=True),
|
||||
Document.content.icontains(q, autoescape=True),
|
||||
)
|
||||
)
|
||||
return page_result(db, statement.order_by(Document.id.desc()), page, page_size)
|
||||
|
||||
|
||||
@router.post("", response_model=DocumentRead, status_code=201)
|
||||
def create_document(payload: DocumentCreate, db: Session = Depends(get_db)):
|
||||
get_item(db, Source, payload.source_id, lock=True)
|
||||
item = Document(**payload.model_dump())
|
||||
db.add(item)
|
||||
record_event(db, item, "create")
|
||||
db.commit()
|
||||
return item
|
||||
|
||||
|
||||
@router.get("/{item_id}", response_model=DocumentRead)
|
||||
def read_document(item_id: int, db: Session = Depends(get_db)):
|
||||
return get_item(db, Document, item_id)
|
||||
|
||||
|
||||
@router.patch("/{item_id}", response_model=DocumentRead)
|
||||
def update_document(item_id: int, payload: DocumentUpdate, db: Session = Depends(get_db)):
|
||||
item = get_item(db, Document, item_id, lock=True)
|
||||
ensure_version(item, payload.expected_version)
|
||||
update_values(item, payload)
|
||||
record_event(db, item, "update")
|
||||
invalidate_knowledge(db, KnowledgeItem.document_id == item.id)
|
||||
db.commit()
|
||||
return item
|
||||
|
||||
|
||||
@router.delete("/{item_id}", status_code=204)
|
||||
def delete_document(
|
||||
item_id: int, expected_version: int = Query(..., ge=1), db: Session = Depends(get_db)
|
||||
):
|
||||
item = get_item(db, Document, item_id, lock=True)
|
||||
ensure_version(item, expected_version)
|
||||
if (
|
||||
db.scalar(
|
||||
select(KnowledgeItem.id)
|
||||
.where(KnowledgeItem.document_id == item_id, KnowledgeItem.is_deleted.is_(False))
|
||||
.limit(1)
|
||||
)
|
||||
is not None
|
||||
):
|
||||
raise HTTPException(409, "文档仍被知识条目引用,不能删除")
|
||||
item.is_deleted = True
|
||||
record_event(db, item, "delete")
|
||||
db.commit()
|
||||
return Response(status_code=204)
|
||||
|
||||
|
||||
@router.get("/{item_id}/history", response_model=Page[AuditRead])
|
||||
def document_history(
|
||||
item_id: int,
|
||||
page: int = Query(1, ge=1),
|
||||
page_size: int = Query(20, ge=1, le=100),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
return history(db, Document, item_id, page, page_size)
|
||||
@@ -1,5 +1,18 @@
|
||||
from fastapi import APIRouter
|
||||
from fastapi import APIRouter, Depends
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.db.session import get_db
|
||||
|
||||
router = APIRouter(tags=["health"])
|
||||
|
||||
|
||||
@router.get("/health")
|
||||
def health():
|
||||
return {"status": "ok", "service": "nalati-rag-api"}
|
||||
|
||||
|
||||
@router.get("/ready")
|
||||
def ready(db: Session = Depends(get_db)):
|
||||
db.execute(text("SELECT 1"))
|
||||
return {"status": "ok", "database": "connected"}
|
||||
|
||||
@@ -0,0 +1,91 @@
|
||||
from uuid import UUID
|
||||
|
||||
from fastapi import APIRouter, Depends, Query
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.db.session import get_db
|
||||
from app.models.indexing import Chunk, IndexPublication, IndexVersion
|
||||
from app.schemas.catalog import Page
|
||||
from app.schemas.indexing import (
|
||||
ActiveIndex,
|
||||
BuildIndex,
|
||||
ChunkRead,
|
||||
IndexRead,
|
||||
PublicationRead,
|
||||
PublishIndex,
|
||||
)
|
||||
from app.services.auth import CurrentSession, require_admin
|
||||
from app.services.catalog import page_result
|
||||
from app.services.indexing import active_index, build_index, publish_index, version_or_404
|
||||
|
||||
router = APIRouter(prefix="/indexes", tags=["indexes"], dependencies=[Depends(require_admin)])
|
||||
|
||||
|
||||
@router.post("", response_model=IndexRead, status_code=201)
|
||||
def build(
|
||||
payload: BuildIndex,
|
||||
current: CurrentSession = Depends(require_admin),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
return build_index(db, payload, current.user.id)
|
||||
|
||||
|
||||
@router.get("", response_model=Page[IndexRead])
|
||||
def list_versions(
|
||||
page: int = Query(1, ge=1),
|
||||
page_size: int = Query(20, ge=1, le=100),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
return page_result(
|
||||
db,
|
||||
select(IndexVersion).order_by(IndexVersion.created_at.desc(), IndexVersion.id),
|
||||
page,
|
||||
page_size,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/active", response_model=ActiveIndex)
|
||||
def active(db: Session = Depends(get_db)):
|
||||
return active_index(db)
|
||||
|
||||
|
||||
@router.get("/publications", response_model=Page[PublicationRead])
|
||||
def publications(
|
||||
page: int = Query(1, ge=1),
|
||||
page_size: int = Query(20, ge=1, le=100),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
return page_result(
|
||||
db, select(IndexPublication).order_by(IndexPublication.generation.desc()), page, page_size
|
||||
)
|
||||
|
||||
|
||||
@router.get("/{version_id}", response_model=IndexRead)
|
||||
def read_version(version_id: UUID, db: Session = Depends(get_db)):
|
||||
return version_or_404(db, str(version_id))
|
||||
|
||||
|
||||
@router.post("/{version_id}/publish", response_model=ActiveIndex)
|
||||
def publish(
|
||||
version_id: UUID,
|
||||
payload: PublishIndex,
|
||||
current: CurrentSession = Depends(require_admin),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
return publish_index(db, version_id, payload.expected_generation, current.user.id)
|
||||
|
||||
|
||||
@router.get("/{version_id}/chunks", response_model=Page[ChunkRead])
|
||||
def chunks(
|
||||
version_id: UUID,
|
||||
knowledge_id: int | None = Query(None, gt=0),
|
||||
page: int = Query(1, ge=1),
|
||||
page_size: int = Query(20, ge=1, le=100),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
version_or_404(db, str(version_id))
|
||||
statement = select(Chunk).where(Chunk.index_version_id == str(version_id))
|
||||
if knowledge_id is not None:
|
||||
statement = statement.where(Chunk.knowledge_id == knowledge_id)
|
||||
return page_result(db, statement.order_by(Chunk.knowledge_id, Chunk.ordinal), page, page_size)
|
||||
@@ -1,15 +1,157 @@
|
||||
from fastapi import APIRouter, Depends
|
||||
from sqlalchemy import select
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Response
|
||||
from sqlalchemy import or_, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.db.session import get_db
|
||||
from app.models.knowledge import KnowledgeItem
|
||||
from app.schemas.knowledge import KnowledgeCreate, KnowledgeRead
|
||||
router = APIRouter(tags=["knowledge"])
|
||||
@router.get("/knowledge", response_model=list[KnowledgeRead])
|
||||
def list_knowledge(db: Session = Depends(get_db)):
|
||||
return list(db.scalars(select(KnowledgeItem).order_by(KnowledgeItem.id.desc())).all())
|
||||
@router.post("/knowledge", response_model=KnowledgeRead, status_code=201)
|
||||
from app.models import KnowledgeItem
|
||||
from app.models.catalog import utc_now
|
||||
from app.schemas.catalog import AuditRead, Page, Status
|
||||
from app.schemas.knowledge import KnowledgeCreate, KnowledgeRead, KnowledgeReview, KnowledgeUpdate
|
||||
from app.services.auth import require_admin
|
||||
from app.services.catalog import (
|
||||
ensure_version,
|
||||
get_item,
|
||||
history,
|
||||
page_result,
|
||||
record_event,
|
||||
reset_review,
|
||||
resolve_knowledge_links,
|
||||
update_values,
|
||||
)
|
||||
from app.services.indexing import catalog_write_guard
|
||||
|
||||
router = APIRouter(
|
||||
prefix="/knowledge",
|
||||
tags=["knowledge"],
|
||||
dependencies=[Depends(require_admin), Depends(catalog_write_guard)],
|
||||
)
|
||||
|
||||
|
||||
def filtered_query(q="", category=None, verification_status=None, source_id=None, document_id=None):
|
||||
statement = select(KnowledgeItem).where(KnowledgeItem.is_deleted.is_(False))
|
||||
for column, value in (
|
||||
(KnowledgeItem.category, category),
|
||||
(KnowledgeItem.verification_status, verification_status),
|
||||
(KnowledgeItem.source_id, source_id),
|
||||
(KnowledgeItem.document_id, document_id),
|
||||
):
|
||||
if value is not None:
|
||||
statement = statement.where(column == value)
|
||||
if q:
|
||||
statement = statement.where(
|
||||
or_(
|
||||
KnowledgeItem.title.icontains(q, autoescape=True),
|
||||
KnowledgeItem.content.icontains(q, autoescape=True),
|
||||
)
|
||||
)
|
||||
return statement.order_by(KnowledgeItem.id.desc())
|
||||
|
||||
|
||||
@router.get("", response_model=list[KnowledgeRead])
|
||||
def list_knowledge(
|
||||
q: str = Query("", max_length=200),
|
||||
category: str | None = Query(None, max_length=80),
|
||||
verification_status: Status | None = None,
|
||||
limit: int = Query(100, ge=1, le=200),
|
||||
offset: int = Query(0, ge=0),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""保留数组响应;完整分页与关联筛选使用 /knowledge/page。"""
|
||||
return list(
|
||||
db.scalars(filtered_query(q, category, verification_status).offset(offset).limit(limit))
|
||||
)
|
||||
|
||||
|
||||
@router.get("/page", response_model=Page[KnowledgeRead])
|
||||
def paged_knowledge(
|
||||
q: str = Query("", max_length=200),
|
||||
category: str | None = Query(None, max_length=80),
|
||||
verification_status: Status | None = None,
|
||||
source_id: int | None = Query(None, gt=0),
|
||||
document_id: int | None = Query(None, gt=0),
|
||||
page: int = Query(1, ge=1),
|
||||
page_size: int = Query(20, ge=1, le=100),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
return page_result(
|
||||
db,
|
||||
filtered_query(q, category, verification_status, source_id, document_id),
|
||||
page,
|
||||
page_size,
|
||||
)
|
||||
|
||||
|
||||
@router.post("", response_model=KnowledgeRead, status_code=201)
|
||||
def create_knowledge(payload: KnowledgeCreate, db: Session = Depends(get_db)):
|
||||
item = KnowledgeItem(title=payload.title, category=payload.category, content=payload.content, source_url=str(payload.source_url) if payload.source_url else None, verification_status="pending")
|
||||
db.add(item); db.commit(); db.refresh(item)
|
||||
item = KnowledgeItem(
|
||||
**payload.model_dump(exclude={"source_url"}),
|
||||
source_url=str(payload.source_url) if payload.source_url else None,
|
||||
verification_status="pending",
|
||||
)
|
||||
resolve_knowledge_links(db, item)
|
||||
db.add(item)
|
||||
record_event(db, item, "create")
|
||||
db.commit()
|
||||
return item
|
||||
|
||||
|
||||
@router.get("/{item_id}", response_model=KnowledgeRead)
|
||||
def read_knowledge(item_id: int, db: Session = Depends(get_db)):
|
||||
return get_item(db, KnowledgeItem, item_id)
|
||||
|
||||
|
||||
@router.patch("/{item_id}", response_model=KnowledgeRead)
|
||||
def update_knowledge(item_id: int, payload: KnowledgeUpdate, db: Session = Depends(get_db)):
|
||||
item = get_item(db, KnowledgeItem, item_id)
|
||||
ensure_version(item, payload.expected_version)
|
||||
update_values(item, payload)
|
||||
if payload.document_id is not None and "source_id" not in payload.model_fields_set:
|
||||
item.source_id = None
|
||||
if {
|
||||
"source_id",
|
||||
"document_id",
|
||||
} & payload.model_fields_set and "source_url" not in payload.model_fields_set:
|
||||
item.source_url = None
|
||||
resolve_knowledge_links(db, item)
|
||||
reset_review(item)
|
||||
record_event(db, item, "update")
|
||||
db.commit()
|
||||
return item
|
||||
|
||||
|
||||
@router.post("/{item_id}/review", response_model=KnowledgeRead)
|
||||
def review_knowledge(item_id: int, payload: KnowledgeReview, db: Session = Depends(get_db)):
|
||||
item = get_item(db, KnowledgeItem, item_id)
|
||||
ensure_version(item, payload.expected_version)
|
||||
resolve_knowledge_links(db, item)
|
||||
if payload.verification_status == "verified" and (not item.source_url or not item.collected_at):
|
||||
raise HTTPException(422, "核验通过前必须补全来源网址和真实采集时间")
|
||||
item.verification_status = payload.verification_status
|
||||
item.reviewer = payload.reviewer
|
||||
item.reviewed_at = utc_now()
|
||||
item.verification_notes = payload.verification_notes
|
||||
record_event(db, item, "review", actor=payload.reviewer, notes=payload.verification_notes)
|
||||
db.commit()
|
||||
return item
|
||||
|
||||
|
||||
@router.delete("/{item_id}", status_code=204)
|
||||
def delete_knowledge(
|
||||
item_id: int, expected_version: int = Query(..., ge=1), db: Session = Depends(get_db)
|
||||
):
|
||||
item = get_item(db, KnowledgeItem, item_id)
|
||||
ensure_version(item, expected_version)
|
||||
item.is_deleted = True
|
||||
record_event(db, item, "delete")
|
||||
db.commit()
|
||||
return Response(status_code=204)
|
||||
|
||||
|
||||
@router.get("/{item_id}/history", response_model=Page[AuditRead])
|
||||
def knowledge_history(
|
||||
item_id: int,
|
||||
page: int = Query(1, ge=1),
|
||||
page_size: int = Query(20, ge=1, le=100),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
return history(db, KnowledgeItem, item_id, page, page_size)
|
||||
|
||||
@@ -0,0 +1,81 @@
|
||||
from uuid import UUID
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.db.session import get_db
|
||||
from app.models import RetrievalRun
|
||||
from app.schemas.catalog import Page
|
||||
from app.schemas.hybrid import (
|
||||
ChunkKeywordRequest,
|
||||
ChunkKeywordResponse,
|
||||
HybridRequest,
|
||||
HybridResponse,
|
||||
)
|
||||
from app.schemas.retrieval import RetrievalRequest, RetrievalResponse
|
||||
from app.schemas.vectors import VectorRetrievalResponse
|
||||
from app.services.auth import require_admin
|
||||
from app.services.embedding_runtime import get_embedding_provider
|
||||
from app.services.hybrid import retrieve_chunks, retrieve_hybrid
|
||||
from app.services.retrieval import retrieve
|
||||
|
||||
router = APIRouter(prefix="/retrieval", tags=["retrieval"])
|
||||
|
||||
RunResponse = RetrievalResponse | VectorRetrievalResponse | ChunkKeywordResponse | HybridResponse
|
||||
|
||||
|
||||
@router.post("", response_model=RetrievalResponse)
|
||||
def keyword_retrieval(payload: RetrievalRequest, db: Session = Depends(get_db)):
|
||||
"""只检索已核验知识;保存原始检索结果供复核,不调用大模型。"""
|
||||
return retrieve(db, payload)
|
||||
|
||||
|
||||
@router.post("/chunks", response_model=ChunkKeywordResponse)
|
||||
def chunk_keyword_retrieval(payload: ChunkKeywordRequest, db: Session = Depends(get_db)):
|
||||
return retrieve_chunks(db, payload)
|
||||
|
||||
|
||||
@router.post("/hybrid", response_model=HybridResponse)
|
||||
def hybrid_retrieval(
|
||||
payload: HybridRequest,
|
||||
db: Session = Depends(get_db),
|
||||
provider=Depends(get_embedding_provider),
|
||||
):
|
||||
return retrieve_hybrid(db, payload, provider)
|
||||
|
||||
|
||||
@router.get(
|
||||
"/runs",
|
||||
response_model=Page[RunResponse],
|
||||
dependencies=[Depends(require_admin)],
|
||||
)
|
||||
def list_runs(
|
||||
page: int = Query(1, ge=1),
|
||||
page_size: int = Query(20, ge=1, le=100),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
rows = db.scalars(
|
||||
select(RetrievalRun)
|
||||
.order_by(RetrievalRun.created_at.desc(), RetrievalRun.id)
|
||||
.offset((page - 1) * page_size)
|
||||
.limit(page_size)
|
||||
)
|
||||
return {
|
||||
"items": [row.response for row in rows],
|
||||
"total": db.scalar(select(func.count()).select_from(RetrievalRun)),
|
||||
"page": page,
|
||||
"page_size": page_size,
|
||||
}
|
||||
|
||||
|
||||
@router.get(
|
||||
"/runs/{run_id}",
|
||||
response_model=RunResponse,
|
||||
dependencies=[Depends(require_admin)],
|
||||
)
|
||||
def read_run(run_id: UUID, db: Session = Depends(get_db)):
|
||||
row = db.get(RetrievalRun, str(run_id))
|
||||
if row is None:
|
||||
raise HTTPException(404, "检索记录不存在")
|
||||
return row.response
|
||||
@@ -0,0 +1,101 @@
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Response
|
||||
from sqlalchemy import or_, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.db.session import get_db
|
||||
from app.models import Document, KnowledgeItem, Source
|
||||
from app.schemas.catalog import AuditRead, Page, SourceCreate, SourceRead, SourceUpdate
|
||||
from app.services.auth import require_admin
|
||||
from app.services.catalog import (
|
||||
ensure_version,
|
||||
get_item,
|
||||
history,
|
||||
invalidate_knowledge,
|
||||
page_result,
|
||||
record_event,
|
||||
update_values,
|
||||
)
|
||||
from app.services.indexing import catalog_write_guard
|
||||
|
||||
router = APIRouter(
|
||||
prefix="/sources",
|
||||
tags=["sources"],
|
||||
dependencies=[Depends(require_admin), Depends(catalog_write_guard)],
|
||||
)
|
||||
|
||||
|
||||
@router.get("", response_model=Page[SourceRead])
|
||||
def list_sources(
|
||||
q: str = Query("", max_length=200),
|
||||
page: int = Query(1, ge=1),
|
||||
page_size: int = Query(20, ge=1, le=100),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
statement = select(Source).where(Source.is_deleted.is_(False))
|
||||
if q:
|
||||
statement = statement.where(
|
||||
or_(
|
||||
Source.name.icontains(q, autoescape=True),
|
||||
Source.notes.icontains(q, autoescape=True),
|
||||
)
|
||||
)
|
||||
return page_result(db, statement.order_by(Source.id.desc()), page, page_size)
|
||||
|
||||
|
||||
@router.post("", response_model=SourceRead, status_code=201)
|
||||
def create_source(payload: SourceCreate, db: Session = Depends(get_db)):
|
||||
item = Source(**payload.model_dump(mode="json"))
|
||||
db.add(item)
|
||||
record_event(db, item, "create")
|
||||
db.commit()
|
||||
return item
|
||||
|
||||
|
||||
@router.get("/{item_id}", response_model=SourceRead)
|
||||
def read_source(item_id: int, db: Session = Depends(get_db)):
|
||||
return get_item(db, Source, item_id)
|
||||
|
||||
|
||||
@router.patch("/{item_id}", response_model=SourceRead)
|
||||
def update_source(item_id: int, payload: SourceUpdate, db: Session = Depends(get_db)):
|
||||
item = get_item(db, Source, item_id, lock=True)
|
||||
ensure_version(item, payload.expected_version)
|
||||
update_values(item, payload)
|
||||
record_event(db, item, "update")
|
||||
invalidate_knowledge(
|
||||
db, KnowledgeItem.source_id == item.id, source_url=item.source_url, update_url=True
|
||||
)
|
||||
db.commit()
|
||||
return item
|
||||
|
||||
|
||||
@router.delete("/{item_id}", status_code=204)
|
||||
def delete_source(
|
||||
item_id: int, expected_version: int = Query(..., ge=1), db: Session = Depends(get_db)
|
||||
):
|
||||
item = get_item(db, Source, item_id, lock=True)
|
||||
ensure_version(item, expected_version)
|
||||
for model in (Document, KnowledgeItem):
|
||||
if (
|
||||
db.scalar(
|
||||
select(model.id)
|
||||
.where(model.source_id == item_id, model.is_deleted.is_(False))
|
||||
.limit(1)
|
||||
)
|
||||
is not None
|
||||
):
|
||||
raise HTTPException(409, "来源仍被文档或知识条目引用,不能删除")
|
||||
item.is_deleted = True
|
||||
record_event(db, item, "delete")
|
||||
db.commit()
|
||||
return Response(status_code=204)
|
||||
|
||||
|
||||
@router.get("/{item_id}/history", response_model=Page[AuditRead])
|
||||
def source_history(
|
||||
item_id: int,
|
||||
page: int = Query(1, ge=1),
|
||||
page_size: int = Query(20, ge=1, le=100),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
return history(db, Source, item_id, page, page_size)
|
||||
@@ -0,0 +1,106 @@
|
||||
from uuid import UUID
|
||||
|
||||
from fastapi import APIRouter, Depends, Query
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.db.session import get_db
|
||||
from app.models.vectors import EmbeddingSpace, VectorPublication, VectorVersion
|
||||
from app.schemas.catalog import Page
|
||||
from app.schemas.indexing import PublicationRead, PublishIndex
|
||||
from app.schemas.vectors import (
|
||||
ActiveVectors,
|
||||
BuildVectors,
|
||||
SpaceRead,
|
||||
VectorRead,
|
||||
VectorRequest,
|
||||
VectorRetrievalResponse,
|
||||
)
|
||||
from app.services.auth import CurrentSession, require_admin
|
||||
from app.services.catalog import page_result
|
||||
from app.services.embedding_runtime import get_embedding_provider
|
||||
from app.services.vectors import (
|
||||
active_vectors,
|
||||
build_vectors,
|
||||
publish_vectors,
|
||||
retrieve_vectors,
|
||||
vector_or_404,
|
||||
)
|
||||
|
||||
router = APIRouter(
|
||||
prefix="/vector-indexes", tags=["vector-indexes"], dependencies=[Depends(require_admin)]
|
||||
)
|
||||
retrieval_router = APIRouter(tags=["retrieval"])
|
||||
|
||||
|
||||
@router.post("", response_model=VectorRead, status_code=201)
|
||||
def build(
|
||||
payload: BuildVectors,
|
||||
current: CurrentSession = Depends(require_admin),
|
||||
db: Session = Depends(get_db),
|
||||
provider=Depends(get_embedding_provider),
|
||||
):
|
||||
return build_vectors(db, payload.text_index_version_id, current.user.id, provider)
|
||||
|
||||
|
||||
@router.get("", response_model=Page[VectorRead])
|
||||
def versions(
|
||||
page: int = Query(1, ge=1),
|
||||
page_size: int = Query(20, ge=1, le=100),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
return page_result(
|
||||
db,
|
||||
select(VectorVersion).order_by(VectorVersion.created_at.desc(), VectorVersion.id),
|
||||
page,
|
||||
page_size,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/active", response_model=ActiveVectors)
|
||||
def active(db: Session = Depends(get_db)):
|
||||
return active_vectors(db)
|
||||
|
||||
|
||||
@router.get("/spaces", response_model=Page[SpaceRead])
|
||||
def spaces(
|
||||
page: int = Query(1, ge=1),
|
||||
page_size: int = Query(20, ge=1, le=100),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
return page_result(db, select(EmbeddingSpace).order_by(EmbeddingSpace.id), page, page_size)
|
||||
|
||||
|
||||
@router.get("/publications", response_model=Page[PublicationRead])
|
||||
def publications(
|
||||
page: int = Query(1, ge=1),
|
||||
page_size: int = Query(20, ge=1, le=100),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
return page_result(
|
||||
db, select(VectorPublication).order_by(VectorPublication.generation.desc()), page, page_size
|
||||
)
|
||||
|
||||
|
||||
@router.get("/{version_id}", response_model=VectorRead)
|
||||
def read_version(version_id: UUID, db: Session = Depends(get_db)):
|
||||
return vector_or_404(db, str(version_id))
|
||||
|
||||
|
||||
@router.post("/{version_id}/publish", response_model=ActiveVectors)
|
||||
def publish(
|
||||
version_id: UUID,
|
||||
payload: PublishIndex,
|
||||
current: CurrentSession = Depends(require_admin),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
return publish_vectors(db, version_id, payload.expected_generation, current.user.id)
|
||||
|
||||
|
||||
@retrieval_router.post("/retrieval/vector", response_model=VectorRetrievalResponse)
|
||||
def search(
|
||||
payload: VectorRequest,
|
||||
db: Session = Depends(get_db),
|
||||
provider=Depends(get_embedding_provider),
|
||||
):
|
||||
return retrieve_vectors(db, payload, provider)
|
||||
@@ -1,14 +1,64 @@
|
||||
from pathlib import Path
|
||||
from typing import Literal
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from pydantic import Field, field_validator, model_validator
|
||||
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||
|
||||
|
||||
class Settings(BaseSettings):
|
||||
app_name: str = "Nalati Tourism QA API"
|
||||
app_env: str = "development"
|
||||
api_prefix: str = "/api"
|
||||
database_url: str = "postgresql+psycopg://nalati:change-this-password@127.0.0.1:5432/nalati_rag"
|
||||
cors_origins: str = "http://localhost:5173,http://127.0.0.1:5173"
|
||||
model_config = SettingsConfigDict(env_file=".env", extra="ignore")
|
||||
auth_allowed_origins: str = (
|
||||
"http://localhost:5173,http://127.0.0.1:5173,http://localhost:8000,http://127.0.0.1:8000"
|
||||
)
|
||||
auth_cookie_secure: bool = True
|
||||
auth_session_seconds: int = Field(28800, ge=300, le=604800)
|
||||
embedding_enabled: bool = False
|
||||
embedding_model: Literal["small", "base"] = "small"
|
||||
embedding_cache_dir: Path | None = None
|
||||
embedding_batch_size: int = Field(8, ge=1, le=32)
|
||||
embedding_threads: int = Field(2, ge=1, le=4)
|
||||
model_config = SettingsConfigDict(
|
||||
env_file=Path(__file__).resolve().parents[2] / ".env", extra="ignore"
|
||||
)
|
||||
|
||||
@property
|
||||
def cors_origin_list(self) -> list[str]:
|
||||
return [x.strip() for x in self.cors_origins.split(",") if x.strip()]
|
||||
|
||||
@property
|
||||
def auth_origin_list(self) -> list[str]:
|
||||
return [x.strip() for x in self.auth_allowed_origins.split(",") if x.strip()]
|
||||
|
||||
@field_validator("embedding_cache_dir", mode="before")
|
||||
@classmethod
|
||||
def empty_embedding_cache_uses_default(cls, value):
|
||||
return None if isinstance(value, str) and not value.strip() else value
|
||||
|
||||
@model_validator(mode="after")
|
||||
def validate_auth_settings(self):
|
||||
if not self.auth_cookie_secure and self.app_env not in {"development", "test"}:
|
||||
raise ValueError("非开发/测试环境必须使用 Secure Cookie 与 HTTPS")
|
||||
if not self.auth_origin_list:
|
||||
raise ValueError("必须显式设置认证允许的来源")
|
||||
for origin in [*self.auth_origin_list, *self.cors_origin_list]:
|
||||
parsed = urlsplit(origin)
|
||||
if (
|
||||
parsed.scheme not in {"http", "https"}
|
||||
or not parsed.hostname
|
||||
or parsed.netloc != parsed.hostname + (f":{parsed.port}" if parsed.port else "")
|
||||
or parsed.path
|
||||
or parsed.query
|
||||
or parsed.fragment
|
||||
or "*" in origin
|
||||
or (self.app_env not in {"development", "test"} and parsed.scheme != "https")
|
||||
):
|
||||
raise ValueError("认证和跨域来源须为明确的 HTTP(S) Origin,不允许通配符或路径")
|
||||
return self
|
||||
|
||||
|
||||
settings = Settings()
|
||||
|
||||
@@ -1,16 +1,25 @@
|
||||
from sqlalchemy import create_engine
|
||||
from sqlalchemy.orm import DeclarativeBase, sessionmaker
|
||||
|
||||
from app.core.config import settings
|
||||
|
||||
|
||||
class Base(DeclarativeBase):
|
||||
pass
|
||||
|
||||
engine = create_engine(settings.database_url, pool_pre_ping=True)
|
||||
|
||||
engine = create_engine(
|
||||
settings.database_url, pool_pre_ping=True, connect_args={"connect_timeout": 5}
|
||||
)
|
||||
SessionLocal = sessionmaker(bind=engine, autoflush=False, autocommit=False)
|
||||
|
||||
|
||||
def get_db():
|
||||
db = SessionLocal()
|
||||
try:
|
||||
yield db
|
||||
except Exception:
|
||||
db.rollback()
|
||||
raise
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
+103
-18
@@ -1,25 +1,110 @@
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
from fastapi import FastAPI
|
||||
from fastapi.exceptions import RequestValidationError
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
from fastapi.responses import JSONResponse
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.exc import IntegrityError, OperationalError, ProgrammingError
|
||||
from sqlalchemy.orm.exc import StaleDataError
|
||||
|
||||
from app.api.routes import (
|
||||
auth,
|
||||
chat,
|
||||
documents,
|
||||
health,
|
||||
indexes,
|
||||
knowledge,
|
||||
retrieval,
|
||||
sources,
|
||||
vectors,
|
||||
)
|
||||
from app.core.config import settings
|
||||
from app.db.session import Base, engine
|
||||
from app.models import KnowledgeItem # noqa: F401
|
||||
from app.api.routes import health, chat, knowledge
|
||||
from app.db.session import engine
|
||||
from app.services.auth import dummy_hash
|
||||
from app.services.retrieval import tokenizer
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
# Convenience for local bootstrap. Use Alembic migrations before production.
|
||||
with engine.begin() as connection:
|
||||
connection.execute(text("CREATE EXTENSION IF NOT EXISTS vector"))
|
||||
Base.metadata.create_all(bind=engine)
|
||||
yield
|
||||
|
||||
app = FastAPI(title=settings.app_name, version="0.1.0", lifespan=lifespan)
|
||||
app.add_middleware(CORSMiddleware, allow_origins=settings.cors_origin_list, allow_credentials=True, allow_methods=["*"], allow_headers=["*"])
|
||||
app.include_router(health.router, prefix=settings.api_prefix)
|
||||
app.include_router(chat.router, prefix=settings.api_prefix)
|
||||
app.include_router(knowledge.router, prefix=settings.api_prefix)
|
||||
@app.get("/")
|
||||
def root():
|
||||
return {"message": "Nalati Tourism QA API", "docs": "/docs"}
|
||||
def create_app(db_engine=engine) -> FastAPI:
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
try:
|
||||
with db_engine.connect() as connection:
|
||||
revision = connection.scalar(text("SELECT version_num FROM alembic_version"))
|
||||
if revision != "0006_vectors":
|
||||
raise RuntimeError(
|
||||
"数据库版本不匹配,请先在 backend 目录执行 alembic upgrade head"
|
||||
)
|
||||
except ProgrammingError as exc:
|
||||
raise RuntimeError(
|
||||
"数据库尚未初始化,请先在 backend 目录执行 alembic upgrade head"
|
||||
) from exc
|
||||
tokenizer()
|
||||
dummy_hash()
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
db_engine.dispose()
|
||||
|
||||
app = FastAPI(title=settings.app_name, version="0.6.0", lifespan=lifespan)
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=settings.cors_origin_list,
|
||||
allow_credentials=True,
|
||||
allow_methods=["*"],
|
||||
allow_headers=["*"],
|
||||
)
|
||||
for router in (
|
||||
health.router,
|
||||
auth.router,
|
||||
indexes.router,
|
||||
vectors.router,
|
||||
vectors.retrieval_router,
|
||||
chat.router,
|
||||
knowledge.router,
|
||||
sources.router,
|
||||
documents.router,
|
||||
retrieval.router,
|
||||
):
|
||||
app.include_router(router, prefix=settings.api_prefix)
|
||||
|
||||
@app.middleware("http")
|
||||
async def no_cache_api(request, call_next):
|
||||
response = await call_next(request)
|
||||
if request.url.path.startswith(settings.api_prefix + "/"):
|
||||
response.headers["Cache-Control"] = "no-store"
|
||||
return response
|
||||
|
||||
@app.exception_handler(RequestValidationError)
|
||||
async def validation_handler(request, exc):
|
||||
# FastAPI's default includes raw input (including passwords) in error responses.
|
||||
return JSONResponse(
|
||||
status_code=422,
|
||||
content={
|
||||
"detail": [
|
||||
{"loc": list(error["loc"]), "msg": error["msg"], "type": error["type"]}
|
||||
for error in exc.errors()
|
||||
]
|
||||
},
|
||||
)
|
||||
|
||||
@app.exception_handler(StaleDataError)
|
||||
async def stale_data_handler(request, exc):
|
||||
return JSONResponse(status_code=409, content={"detail": "记录已更新,请刷新后重试"})
|
||||
|
||||
@app.exception_handler(IntegrityError)
|
||||
async def integrity_handler(request, exc):
|
||||
return JSONResponse(status_code=409, content={"detail": "数据关联或约束冲突"})
|
||||
|
||||
@app.exception_handler(OperationalError)
|
||||
async def database_unavailable_handler(request, exc):
|
||||
return JSONResponse(status_code=503, content={"detail": "数据库暂不可用"})
|
||||
|
||||
@app.get("/")
|
||||
def root():
|
||||
return {"message": "Nalati Tourism QA API", "docs": "/docs"}
|
||||
|
||||
return app
|
||||
|
||||
|
||||
app = create_app()
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
"""Local-only administrator provisioning. Passwords are read from the terminal, never argv."""
|
||||
|
||||
import argparse
|
||||
import getpass
|
||||
|
||||
from pydantic import ValidationError
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
|
||||
from app.db.session import SessionLocal
|
||||
from app.models.auth import User
|
||||
from app.schemas.auth import RegisterInput
|
||||
from app.services.auth import PASSWORD_HASHER
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="本机管理员初始化(须先完成数据库迁移)")
|
||||
parser.add_argument("command", choices=["create-admin"])
|
||||
parser.add_argument("--username", required=True)
|
||||
args = parser.parse_args()
|
||||
password = getpass.getpass("管理员密码(至少 15 字符):")
|
||||
if password != getpass.getpass("再次输入密码:"):
|
||||
parser.exit(1, "两次密码不一致\n")
|
||||
try:
|
||||
payload = RegisterInput(username=args.username, password=password)
|
||||
except ValidationError:
|
||||
parser.exit(1, "用户名须为 3–32 位字母开头的小写字母/数字/下划线;密码须为 15–128 字符\n")
|
||||
with SessionLocal() as db:
|
||||
if db.scalar(select(User.id).where(User.username == payload.username)) is not None:
|
||||
parser.exit(1, "用户已存在,未修改原有用户或权限\n")
|
||||
db.add(
|
||||
User(
|
||||
username=payload.username,
|
||||
password_hash=PASSWORD_HASHER.hash(password),
|
||||
role="admin",
|
||||
)
|
||||
)
|
||||
try:
|
||||
db.commit()
|
||||
except IntegrityError:
|
||||
db.rollback()
|
||||
parser.exit(1, "用户已存在,未修改原有用户或权限\n")
|
||||
print("管理员已创建;请通过登录接口建立会话。")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1 +1,32 @@
|
||||
from app.models.auth import AuthRateLimit, AuthSession, User
|
||||
from app.models.catalog import AuditEvent, Document, Source
|
||||
from app.models.indexing import Chunk, IndexPublication, IndexState, IndexVersion
|
||||
from app.models.knowledge import KnowledgeItem
|
||||
from app.models.retrieval import RetrievalRun
|
||||
from app.models.vectors import (
|
||||
ChunkVector,
|
||||
EmbeddingSpace,
|
||||
VectorPublication,
|
||||
VectorState,
|
||||
VectorVersion,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"AuditEvent",
|
||||
"AuthRateLimit",
|
||||
"AuthSession",
|
||||
"Chunk",
|
||||
"ChunkVector",
|
||||
"Document",
|
||||
"EmbeddingSpace",
|
||||
"IndexPublication",
|
||||
"IndexState",
|
||||
"IndexVersion",
|
||||
"KnowledgeItem",
|
||||
"RetrievalRun",
|
||||
"Source",
|
||||
"User",
|
||||
"VectorPublication",
|
||||
"VectorState",
|
||||
"VectorVersion",
|
||||
]
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
"""Users, opaque sessions and shared authentication request budgets."""
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import CheckConstraint, DateTime, ForeignKey, Integer, String, func
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from app.db.session import Base
|
||||
|
||||
|
||||
class User(Base):
|
||||
__tablename__ = "users"
|
||||
__table_args__ = (CheckConstraint("role IN ('user', 'admin')", name="ck_users_role"),)
|
||||
|
||||
id: Mapped[int] = mapped_column(primary_key=True)
|
||||
username: Mapped[str] = mapped_column(String(32), unique=True, nullable=False)
|
||||
password_hash: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||
nickname: Mapped[str | None] = mapped_column(String(80))
|
||||
email: Mapped[str | None] = mapped_column(String(254))
|
||||
role: Mapped[str] = mapped_column(String(10), default="user", nullable=False)
|
||||
credential_version: Mapped[int] = mapped_column(Integer, default=1, nullable=False)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
|
||||
|
||||
|
||||
class AuthSession(Base):
|
||||
__tablename__ = "auth_sessions"
|
||||
|
||||
token_hash: Mapped[str] = mapped_column(String(64), primary_key=True)
|
||||
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), index=True)
|
||||
credential_version: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
|
||||
expires_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, index=True
|
||||
)
|
||||
|
||||
|
||||
class AuthRateLimit(Base):
|
||||
__tablename__ = "auth_rate_limits"
|
||||
|
||||
key: Mapped[str] = mapped_column(String(64), primary_key=True)
|
||||
window_start: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||
attempts: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
@@ -0,0 +1,54 @@
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy import JSON, Boolean, DateTime, ForeignKey, Integer, String, Text, func
|
||||
from sqlalchemy.orm import Mapped, declared_attr, mapped_column
|
||||
|
||||
from app.db.session import Base
|
||||
|
||||
|
||||
def utc_now() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
class Versioned:
|
||||
version: Mapped[int] = mapped_column(Integer, nullable=False, default=1)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
|
||||
updated_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), default=utc_now, onupdate=utc_now
|
||||
)
|
||||
is_deleted: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
|
||||
|
||||
@declared_attr.directive
|
||||
def __mapper_args__(cls):
|
||||
return {"version_id_col": cls.version}
|
||||
|
||||
|
||||
class Source(Versioned, Base):
|
||||
__tablename__ = "sources"
|
||||
id: Mapped[int] = mapped_column(primary_key=True)
|
||||
name: Mapped[str] = mapped_column(String(250), nullable=False)
|
||||
source_url: Mapped[str | None] = mapped_column(String(1000))
|
||||
notes: Mapped[str] = mapped_column(Text, nullable=False, default="")
|
||||
|
||||
|
||||
class Document(Versioned, Base):
|
||||
__tablename__ = "documents"
|
||||
id: Mapped[int] = mapped_column(primary_key=True)
|
||||
source_id: Mapped[int] = mapped_column(ForeignKey("sources.id"), index=True)
|
||||
title: Mapped[str] = mapped_column(String(250), nullable=False)
|
||||
category: Mapped[str] = mapped_column(String(80), nullable=False)
|
||||
content: Mapped[str] = mapped_column(Text, nullable=False)
|
||||
collected_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||
notes: Mapped[str] = mapped_column(Text, nullable=False, default="")
|
||||
|
||||
|
||||
class AuditEvent(Base):
|
||||
__tablename__ = "audit_events"
|
||||
id: Mapped[int] = mapped_column(primary_key=True)
|
||||
entity_type: Mapped[str] = mapped_column(String(30), nullable=False, index=True)
|
||||
entity_id: Mapped[int] = mapped_column(Integer, nullable=False, index=True)
|
||||
action: Mapped[str] = mapped_column(String(30), nullable=False)
|
||||
actor: Mapped[str | None] = mapped_column(String(100))
|
||||
notes: Mapped[str] = mapped_column(Text, nullable=False, default="")
|
||||
snapshot: Mapped[dict] = mapped_column(JSON, nullable=False)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
|
||||
@@ -0,0 +1,75 @@
|
||||
"""Immutable chunk snapshots and atomic publication metadata (no vector index yet)."""
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import (
|
||||
JSON,
|
||||
CheckConstraint,
|
||||
DateTime,
|
||||
ForeignKey,
|
||||
Integer,
|
||||
String,
|
||||
Text,
|
||||
UniqueConstraint,
|
||||
func,
|
||||
)
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from app.db.session import Base
|
||||
|
||||
|
||||
class IndexVersion(Base):
|
||||
__tablename__ = "index_versions"
|
||||
__table_args__ = (
|
||||
CheckConstraint(
|
||||
"status IN ('ready', 'published', 'superseded')", name="ck_index_version_status"
|
||||
),
|
||||
)
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True)
|
||||
status: Mapped[str] = mapped_column(String(20), nullable=False, default="ready")
|
||||
config: Mapped[dict] = mapped_column(JSON, nullable=False)
|
||||
corpus_fingerprint: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
catalog_revision: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
item_count: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
chunk_count: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
created_by: Mapped[int] = mapped_column(ForeignKey("users.id"), nullable=False)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
|
||||
published_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
|
||||
|
||||
class IndexState(Base):
|
||||
__tablename__ = "index_state"
|
||||
__table_args__ = (CheckConstraint("id = 1", name="ck_index_state_singleton"),)
|
||||
id: Mapped[int] = mapped_column(primary_key=True)
|
||||
active_version_id: Mapped[str | None] = mapped_column(ForeignKey("index_versions.id"))
|
||||
generation: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
catalog_revision: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
|
||||
|
||||
class Chunk(Base):
|
||||
__tablename__ = "chunks"
|
||||
__table_args__ = (
|
||||
UniqueConstraint("index_version_id", "knowledge_id", "ordinal", name="uq_chunk_position"),
|
||||
UniqueConstraint("id", "index_version_id", name="uq_chunk_text_scope"),
|
||||
CheckConstraint("start_offset >= 0 AND end_offset > start_offset", name="ck_chunk_offsets"),
|
||||
)
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True)
|
||||
index_version_id: Mapped[str] = mapped_column(ForeignKey("index_versions.id"), index=True)
|
||||
knowledge_id: Mapped[int] = mapped_column(ForeignKey("knowledge_items.id"), nullable=False)
|
||||
ordinal: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
start_offset: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
end_offset: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
text: Mapped[str] = mapped_column(Text, nullable=False)
|
||||
text_sha256: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
provenance: Mapped[dict] = mapped_column(JSON, nullable=False)
|
||||
|
||||
|
||||
class IndexPublication(Base):
|
||||
__tablename__ = "index_publications"
|
||||
generation: Mapped[int] = mapped_column(primary_key=True)
|
||||
previous_version_id: Mapped[str | None] = mapped_column(ForeignKey("index_versions.id"))
|
||||
version_id: Mapped[str] = mapped_column(ForeignKey("index_versions.id"), nullable=False)
|
||||
actor_id: Mapped[int] = mapped_column(ForeignKey("users.id"), nullable=False)
|
||||
published_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), server_default=func.now()
|
||||
)
|
||||
@@ -1,14 +1,29 @@
|
||||
from datetime import datetime
|
||||
from sqlalchemy import DateTime, Integer, String, Text, func
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
from app.db.session import Base
|
||||
|
||||
class KnowledgeItem(Base):
|
||||
from sqlalchemy import CheckConstraint, DateTime, ForeignKey, Integer, String, Text
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from app.db.session import Base
|
||||
from app.models.catalog import Versioned
|
||||
|
||||
|
||||
class KnowledgeItem(Versioned, Base):
|
||||
__tablename__ = "knowledge_items"
|
||||
__table_args__ = (
|
||||
CheckConstraint(
|
||||
"verification_status IN ('pending', 'verified', 'rejected')",
|
||||
name="ck_knowledge_verification_status",
|
||||
),
|
||||
)
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True, index=True)
|
||||
title: Mapped[str] = mapped_column(String(250), nullable=False, index=True)
|
||||
category: Mapped[str] = mapped_column(String(80), nullable=False, default="未分类")
|
||||
content: Mapped[str] = mapped_column(Text, nullable=False)
|
||||
source_url: Mapped[str | None] = mapped_column(String(1000), nullable=True)
|
||||
verification_status: Mapped[str] = mapped_column(String(30), nullable=False, default="pending")
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
|
||||
source_id: Mapped[int | None] = mapped_column(ForeignKey("sources.id"), index=True)
|
||||
document_id: Mapped[int | None] = mapped_column(ForeignKey("documents.id"), index=True)
|
||||
collected_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
reviewer: Mapped[str | None] = mapped_column(String(100))
|
||||
reviewed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
verification_notes: Mapped[str | None] = mapped_column(Text)
|
||||
|
||||
@@ -0,0 +1,18 @@
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import JSON, DateTime, String, Text, func
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from app.db.session import Base
|
||||
|
||||
|
||||
class RetrievalRun(Base):
|
||||
__tablename__ = "retrieval_runs"
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True)
|
||||
query: Mapped[str] = mapped_column(Text, nullable=False)
|
||||
mode: Mapped[str] = mapped_column(String(20), nullable=False)
|
||||
corpus_fingerprint: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
response: Mapped[dict] = mapped_column(JSON, nullable=False)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), server_default=func.now(), index=True
|
||||
)
|
||||
@@ -0,0 +1,112 @@
|
||||
"""Versioned vector spaces, complete snapshots and independent publication history."""
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
from pgvector.sqlalchemy import Vector
|
||||
from sqlalchemy import (
|
||||
JSON,
|
||||
CheckConstraint,
|
||||
DateTime,
|
||||
ForeignKey,
|
||||
ForeignKeyConstraint,
|
||||
Integer,
|
||||
String,
|
||||
UniqueConstraint,
|
||||
func,
|
||||
)
|
||||
from sqlalchemy.dialects.postgresql.base import ischema_names
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from app.db.session import Base
|
||||
|
||||
# PostgreSQL reflects a qualified name when public is intentionally absent from search_path.
|
||||
# Retain real type comparison during Alembic inspection in isolated schemas.
|
||||
ischema_names.setdefault("public.vector", Vector)
|
||||
|
||||
|
||||
class EmbeddingSpace(Base):
|
||||
__tablename__ = "embedding_spaces"
|
||||
__table_args__ = (
|
||||
UniqueConstraint("id", "dimension", name="uq_embedding_space_dimension"),
|
||||
CheckConstraint("dimension BETWEEN 1 AND 2000", name="ck_embedding_space_dimension"),
|
||||
)
|
||||
id: Mapped[str] = mapped_column(String(64), primary_key=True)
|
||||
dimension: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
spec: Mapped[dict] = mapped_column(JSON, nullable=False)
|
||||
|
||||
|
||||
class VectorVersion(Base):
|
||||
__tablename__ = "vector_versions"
|
||||
__table_args__ = (
|
||||
ForeignKeyConstraint(
|
||||
["space_id", "dimension"],
|
||||
["embedding_spaces.id", "embedding_spaces.dimension"],
|
||||
name="fk_vector_version_space",
|
||||
),
|
||||
UniqueConstraint(
|
||||
"id", "text_index_version_id", "space_id", "dimension", name="uq_vector_version_scope"
|
||||
),
|
||||
CheckConstraint(
|
||||
"status IN ('ready', 'published', 'superseded')", name="ck_vector_version_status"
|
||||
),
|
||||
CheckConstraint("chunk_count BETWEEN 0 AND 2000", name="ck_vector_version_count"),
|
||||
)
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True)
|
||||
text_index_version_id: Mapped[str] = mapped_column(ForeignKey("index_versions.id"))
|
||||
space_id: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
dimension: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
chunk_count: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
status: Mapped[str] = mapped_column(String(20), nullable=False, default="ready")
|
||||
created_by: Mapped[int] = mapped_column(ForeignKey("users.id"))
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
|
||||
published_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
|
||||
|
||||
class ChunkVector(Base):
|
||||
__tablename__ = "chunk_vectors"
|
||||
__table_args__ = (
|
||||
ForeignKeyConstraint(
|
||||
["version_id", "text_index_version_id", "space_id", "dimension"],
|
||||
[
|
||||
"vector_versions.id",
|
||||
"vector_versions.text_index_version_id",
|
||||
"vector_versions.space_id",
|
||||
"vector_versions.dimension",
|
||||
],
|
||||
name="fk_chunk_vector_scope",
|
||||
),
|
||||
ForeignKeyConstraint(
|
||||
["chunk_id", "text_index_version_id"],
|
||||
["chunks.id", "chunks.index_version_id"],
|
||||
name="fk_chunk_vector_text",
|
||||
),
|
||||
CheckConstraint("public.vector_dims(embedding) = dimension", name="ck_vector_dimension"),
|
||||
CheckConstraint(
|
||||
"public.vector_norm(embedding) BETWEEN 0.9999 AND 1.0001", name="ck_vector_unit_norm"
|
||||
),
|
||||
)
|
||||
version_id: Mapped[str] = mapped_column(String(36), primary_key=True)
|
||||
chunk_id: Mapped[str] = mapped_column(String(36), primary_key=True)
|
||||
text_index_version_id: Mapped[str] = mapped_column(String(36), nullable=False)
|
||||
space_id: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
dimension: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
embedding = mapped_column(Vector(), nullable=False)
|
||||
|
||||
|
||||
class VectorState(Base):
|
||||
__tablename__ = "vector_state"
|
||||
__table_args__ = (CheckConstraint("id = 1", name="ck_vector_state_singleton"),)
|
||||
id: Mapped[int] = mapped_column(primary_key=True)
|
||||
active_version_id: Mapped[str | None] = mapped_column(ForeignKey("vector_versions.id"))
|
||||
generation: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
|
||||
|
||||
class VectorPublication(Base):
|
||||
__tablename__ = "vector_publications"
|
||||
generation: Mapped[int] = mapped_column(primary_key=True)
|
||||
previous_version_id: Mapped[str | None] = mapped_column(ForeignKey("vector_versions.id"))
|
||||
version_id: Mapped[str] = mapped_column(ForeignKey("vector_versions.id"), nullable=False)
|
||||
actor_id: Mapped[int] = mapped_column(ForeignKey("users.id"), nullable=False)
|
||||
published_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), server_default=func.now()
|
||||
)
|
||||
@@ -0,0 +1,67 @@
|
||||
from datetime import datetime
|
||||
from typing import Annotated, Literal
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, EmailStr, Field, SecretStr, field_validator
|
||||
|
||||
Password = Annotated[SecretStr, Field(min_length=15, max_length=128)]
|
||||
|
||||
|
||||
class AuthInput(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
|
||||
class ProfileInput(AuthInput):
|
||||
nickname: str | None = Field(None, min_length=1, max_length=80)
|
||||
email: EmailStr | None = Field(None, max_length=254)
|
||||
|
||||
@field_validator("nickname", "email", mode="before")
|
||||
@classmethod
|
||||
def optional_text(cls, value):
|
||||
return (value.strip() or None) if isinstance(value, str) else value
|
||||
|
||||
@field_validator("nickname")
|
||||
@classmethod
|
||||
def no_control_characters(cls, value):
|
||||
if value and any(ord(char) < 32 or ord(char) == 127 for char in value):
|
||||
raise ValueError("昵称不能包含控制字符")
|
||||
return value
|
||||
|
||||
|
||||
class RegisterInput(ProfileInput):
|
||||
username: str = Field(min_length=3, max_length=32, pattern=r"^[a-z][a-z0-9_]{2,31}$")
|
||||
password: Password
|
||||
|
||||
@field_validator("username", mode="before")
|
||||
@classmethod
|
||||
def canonical_username(cls, value):
|
||||
return value.strip().lower() if isinstance(value, str) else value
|
||||
|
||||
|
||||
class LoginInput(AuthInput):
|
||||
username: str = Field(min_length=3, max_length=32, pattern=r"^[a-z][a-z0-9_]{2,31}$")
|
||||
password: SecretStr = Field(min_length=1, max_length=128)
|
||||
|
||||
@field_validator("username", mode="before")
|
||||
@classmethod
|
||||
def canonical_username(cls, value):
|
||||
return value.strip().lower() if isinstance(value, str) else value
|
||||
|
||||
|
||||
class PasswordChangeInput(AuthInput):
|
||||
current_password: SecretStr = Field(min_length=1, max_length=128)
|
||||
new_password: Password
|
||||
|
||||
|
||||
class UserRead(BaseModel):
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
id: int
|
||||
username: str
|
||||
nickname: str | None
|
||||
email: str | None
|
||||
role: Literal["user", "admin"]
|
||||
created_at: datetime
|
||||
|
||||
|
||||
class SessionRead(UserRead):
|
||||
csrf_token: str
|
||||
@@ -0,0 +1,124 @@
|
||||
from datetime import datetime
|
||||
from typing import Annotated, Generic, Literal, TypeVar
|
||||
|
||||
from pydantic import (
|
||||
AwareDatetime,
|
||||
BaseModel,
|
||||
ConfigDict,
|
||||
Field,
|
||||
HttpUrl,
|
||||
field_validator,
|
||||
model_validator,
|
||||
)
|
||||
|
||||
Title = Annotated[str, Field(min_length=1, max_length=250)]
|
||||
Category = Annotated[str, Field(min_length=1, max_length=80)]
|
||||
Body = Annotated[str, Field(min_length=1, max_length=100_000)]
|
||||
Notes = Annotated[str, Field(max_length=10_000)]
|
||||
Status = Literal["pending", "verified", "rejected"]
|
||||
PositiveId = Annotated[int, Field(gt=0)]
|
||||
|
||||
|
||||
class Input(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid", str_strip_whitespace=True)
|
||||
|
||||
@field_validator("source_url", mode="before", check_fields=False)
|
||||
@classmethod
|
||||
def empty_url_is_absent(cls, value):
|
||||
return None if isinstance(value, str) and not value.strip() else value
|
||||
|
||||
|
||||
class Patch(Input):
|
||||
expected_version: PositiveId
|
||||
|
||||
@model_validator(mode="after")
|
||||
def require_changes(self):
|
||||
if not self.model_fields_set - {"expected_version"}:
|
||||
raise ValueError("至少提供一个需要更新的字段")
|
||||
return self
|
||||
|
||||
@field_validator("*", mode="before")
|
||||
@classmethod
|
||||
def disallow_required_null(cls, value, info):
|
||||
if (
|
||||
info.field_name
|
||||
in {"name", "title", "category", "content", "notes", "collected_at", "expected_version"}
|
||||
and value is None
|
||||
):
|
||||
raise ValueError("该字段不能为 null")
|
||||
return value
|
||||
|
||||
|
||||
class Read(BaseModel):
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
id: int
|
||||
version: int
|
||||
created_at: datetime
|
||||
updated_at: datetime
|
||||
is_deleted: bool
|
||||
|
||||
|
||||
class SourceCreate(Input):
|
||||
name: Title
|
||||
source_url: HttpUrl | None = Field(default=None, max_length=1000)
|
||||
notes: Notes = ""
|
||||
|
||||
|
||||
class SourceUpdate(Patch):
|
||||
name: Title | None = None
|
||||
source_url: HttpUrl | None = Field(default=None, max_length=1000)
|
||||
notes: Notes | None = None
|
||||
|
||||
|
||||
class SourceRead(Read):
|
||||
name: str
|
||||
source_url: str | None
|
||||
notes: str
|
||||
|
||||
|
||||
class DocumentCreate(Input):
|
||||
source_id: PositiveId
|
||||
title: Title
|
||||
category: Category = "未分类"
|
||||
content: Body
|
||||
collected_at: AwareDatetime
|
||||
notes: Notes = ""
|
||||
|
||||
|
||||
class DocumentUpdate(Patch):
|
||||
title: Title | None = None
|
||||
category: Category | None = None
|
||||
content: Body | None = None
|
||||
collected_at: AwareDatetime | None = None
|
||||
notes: Notes | None = None
|
||||
|
||||
|
||||
class DocumentRead(Read):
|
||||
source_id: int
|
||||
title: str
|
||||
category: str
|
||||
content: str
|
||||
collected_at: datetime
|
||||
notes: str
|
||||
|
||||
|
||||
T = TypeVar("T")
|
||||
|
||||
|
||||
class Page(BaseModel, Generic[T]):
|
||||
items: list[T]
|
||||
total: int
|
||||
page: int
|
||||
page_size: int
|
||||
|
||||
|
||||
class AuditRead(BaseModel):
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
id: int
|
||||
entity_type: str
|
||||
entity_id: int
|
||||
action: str
|
||||
actor: str | None
|
||||
notes: str
|
||||
snapshot: dict
|
||||
created_at: datetime
|
||||
@@ -1,12 +1,15 @@
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class ChatRequest(BaseModel):
|
||||
question: str = Field(min_length=1, max_length=2000)
|
||||
|
||||
|
||||
class SourcePreview(BaseModel):
|
||||
title: str
|
||||
source_url: str | None = None
|
||||
|
||||
|
||||
class ChatResponse(BaseModel):
|
||||
answer: str
|
||||
status: str
|
||||
|
||||
@@ -0,0 +1,95 @@
|
||||
"""Chunk-level lexical and hybrid contracts; the item-level baseline stays distinct."""
|
||||
|
||||
from datetime import datetime
|
||||
from typing import Literal
|
||||
from uuid import UUID
|
||||
|
||||
from pydantic import BaseModel, Field, model_validator
|
||||
|
||||
from app.schemas.catalog import Category, Input
|
||||
from app.schemas.indexing import IndexConfig
|
||||
from app.schemas.retrieval import RetrievalConfig
|
||||
from app.schemas.vectors import SpaceRead, VectorHit
|
||||
|
||||
|
||||
class ChunkKeywordRequest(Input):
|
||||
query: str = Field(min_length=1, max_length=500)
|
||||
mode: Literal["keyword_chunks"] = "keyword_chunks"
|
||||
top_k: int = Field(5, ge=1, le=20)
|
||||
category: Category | None = None
|
||||
|
||||
|
||||
class HybridRequest(ChunkKeywordRequest):
|
||||
mode: Literal["hybrid"] = "hybrid"
|
||||
candidate_k: int = Field(50, ge=1, le=200)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def sufficient_candidates(self):
|
||||
if self.candidate_k < self.top_k:
|
||||
raise ValueError("candidate_k 不得小于 top_k")
|
||||
return self
|
||||
|
||||
|
||||
class ChunkRetrievalConfig(BaseModel):
|
||||
ranking_unit: Literal["chunk"] = "chunk"
|
||||
lexical: RetrievalConfig
|
||||
chunking: IndexConfig
|
||||
tie_break: Literal["knowledge_id,chunk_ordinal,chunk_id"] = (
|
||||
"knowledge_id,chunk_ordinal,chunk_id"
|
||||
)
|
||||
|
||||
|
||||
class HybridConfig(ChunkRetrievalConfig):
|
||||
scorer: Literal["rrf-v1"] = "rrf-v1"
|
||||
rrf_k: Literal[60] = 60
|
||||
weights: tuple[Literal[1], Literal[1]] = (1, 1)
|
||||
candidate_k: int
|
||||
vector_scorer: Literal["pgvector-cosine-exact-v1"] = "pgvector-cosine-exact-v1"
|
||||
space: SpaceRead
|
||||
|
||||
|
||||
class RankedCandidate(BaseModel):
|
||||
chunk_id: UUID
|
||||
rank: int
|
||||
score: float
|
||||
|
||||
|
||||
class HybridHit(VectorHit):
|
||||
lexical_rank: int | None
|
||||
lexical_score: float | None
|
||||
vector_rank: int | None
|
||||
vector_score: float | None
|
||||
lexical_contribution: float
|
||||
vector_contribution: float
|
||||
|
||||
|
||||
class ChunkKeywordResponse(BaseModel):
|
||||
run_id: UUID
|
||||
query: str
|
||||
mode: Literal["keyword_chunks"] = "keyword_chunks"
|
||||
status: Literal["ok", "no_evidence"]
|
||||
top_k: int
|
||||
category: str | None
|
||||
query_terms: list[str]
|
||||
text_index_version_id: UUID
|
||||
text_generation: int
|
||||
catalog_revision: int
|
||||
corpus_fingerprint: str
|
||||
corpus_size: int
|
||||
total_hits: int
|
||||
results: list[VectorHit]
|
||||
elapsed_ms: float
|
||||
created_at: datetime
|
||||
config: ChunkRetrievalConfig
|
||||
|
||||
|
||||
class HybridResponse(ChunkKeywordResponse):
|
||||
mode: Literal["hybrid"] = "hybrid"
|
||||
vector_version_id: UUID
|
||||
generation: int
|
||||
lexical_total_hits: int
|
||||
vector_total_hits: int
|
||||
lexical_candidates: list[RankedCandidate]
|
||||
vector_candidates: list[RankedCandidate]
|
||||
results: list[HybridHit]
|
||||
config: HybridConfig
|
||||
@@ -0,0 +1,89 @@
|
||||
from datetime import datetime, timezone
|
||||
from typing import Literal
|
||||
from uuid import UUID
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
|
||||
|
||||
from app.schemas.catalog import Input
|
||||
|
||||
|
||||
class BuildIndex(Input):
|
||||
chunk_size: int = Field(500, ge=100, le=2000)
|
||||
overlap: int = Field(80, ge=0, le=1000)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def bounded_overlap(self):
|
||||
if self.overlap > self.chunk_size // 2:
|
||||
raise ValueError("重叠不得超过切分长度的一半")
|
||||
return self
|
||||
|
||||
|
||||
class PublishIndex(Input):
|
||||
expected_generation: int = Field(ge=0)
|
||||
|
||||
|
||||
class IndexConfig(BuildIndex):
|
||||
splitter: Literal["unicode-window-v1"]
|
||||
|
||||
|
||||
class ChunkProvenance(BaseModel):
|
||||
knowledge_id: int
|
||||
knowledge_version: int
|
||||
document_id: int | None
|
||||
document_version: int | None
|
||||
source_id: int | None
|
||||
source_version: int | None
|
||||
source_name: str | None
|
||||
source_url: str
|
||||
title: str
|
||||
category: str
|
||||
|
||||
|
||||
class TimestampRead(BaseModel):
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
@field_validator("created_at", "published_at", check_fields=False)
|
||||
@classmethod
|
||||
def utc_timestamp(cls, value):
|
||||
return value.astimezone(timezone.utc) if value is not None else None
|
||||
|
||||
|
||||
class IndexRead(TimestampRead):
|
||||
id: UUID
|
||||
status: Literal["ready", "published", "superseded"]
|
||||
config: IndexConfig
|
||||
corpus_fingerprint: str
|
||||
catalog_revision: int
|
||||
item_count: int
|
||||
chunk_count: int
|
||||
created_by: int
|
||||
created_at: datetime
|
||||
published_at: datetime | None
|
||||
|
||||
|
||||
class ActiveIndex(BaseModel):
|
||||
generation: int
|
||||
catalog_revision: int
|
||||
is_current: bool
|
||||
active_version: IndexRead | None
|
||||
|
||||
|
||||
class ChunkRead(BaseModel):
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
id: UUID
|
||||
index_version_id: UUID
|
||||
knowledge_id: int
|
||||
ordinal: int
|
||||
start_offset: int
|
||||
end_offset: int
|
||||
text: str
|
||||
text_sha256: str
|
||||
provenance: ChunkProvenance
|
||||
|
||||
|
||||
class PublicationRead(TimestampRead):
|
||||
generation: int
|
||||
previous_version_id: UUID | None
|
||||
version_id: UUID
|
||||
actor_id: int
|
||||
published_at: datetime
|
||||
@@ -1,18 +1,46 @@
|
||||
from datetime import datetime
|
||||
from pydantic import BaseModel, ConfigDict, Field, HttpUrl
|
||||
|
||||
class KnowledgeCreate(BaseModel):
|
||||
title: str = Field(min_length=1, max_length=250)
|
||||
category: str = Field(default="未分类", max_length=80)
|
||||
content: str = Field(min_length=1)
|
||||
source_url: HttpUrl | None = None
|
||||
from pydantic import AwareDatetime, Field, HttpUrl
|
||||
|
||||
class KnowledgeRead(BaseModel):
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
id: int
|
||||
from app.schemas.catalog import Body, Category, Input, Patch, PositiveId, Read, Status, Title
|
||||
|
||||
|
||||
class KnowledgeCreate(Input):
|
||||
title: Title
|
||||
category: Category = "未分类"
|
||||
content: Body
|
||||
source_url: HttpUrl | None = Field(default=None, max_length=1000)
|
||||
source_id: PositiveId | None = None
|
||||
document_id: PositiveId | None = None
|
||||
collected_at: AwareDatetime | None = None
|
||||
|
||||
|
||||
class KnowledgeUpdate(Patch):
|
||||
title: Title | None = None
|
||||
category: Category | None = None
|
||||
content: Body | None = None
|
||||
source_url: HttpUrl | None = Field(default=None, max_length=1000)
|
||||
source_id: PositiveId | None = None
|
||||
document_id: PositiveId | None = None
|
||||
collected_at: AwareDatetime | None = None
|
||||
|
||||
|
||||
class KnowledgeReview(Input):
|
||||
expected_version: PositiveId
|
||||
verification_status: Status
|
||||
reviewer: str = Field(min_length=1, max_length=100)
|
||||
verification_notes: str = Field(min_length=1, max_length=10_000)
|
||||
|
||||
|
||||
class KnowledgeRead(Read):
|
||||
title: str
|
||||
category: str
|
||||
content: str
|
||||
source_url: str | None
|
||||
verification_status: str
|
||||
created_at: datetime
|
||||
verification_status: Status
|
||||
source_id: int | None
|
||||
document_id: int | None
|
||||
collected_at: datetime | None
|
||||
reviewer: str | None
|
||||
reviewed_at: datetime | None
|
||||
verification_notes: str | None
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
from datetime import datetime
|
||||
from typing import Literal
|
||||
from uuid import UUID
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from app.schemas.catalog import Category, Input
|
||||
|
||||
|
||||
class RetrievalRequest(Input):
|
||||
query: str = Field(min_length=1, max_length=500)
|
||||
mode: Literal["keyword"] = "keyword"
|
||||
top_k: int = Field(default=5, ge=1, le=20)
|
||||
category: Category | None = None
|
||||
|
||||
|
||||
class RetrievalHit(BaseModel):
|
||||
rank: int
|
||||
knowledge_id: int
|
||||
knowledge_version: int
|
||||
document_id: int | None
|
||||
document_version: int | None
|
||||
source_id: int | None
|
||||
source_version: int | None
|
||||
source_name: str | None
|
||||
source_url: str
|
||||
title: str
|
||||
category: str
|
||||
snippet: str
|
||||
snippet_start: int
|
||||
snippet_end: int
|
||||
score: float
|
||||
matched_terms: list[str]
|
||||
|
||||
|
||||
class RetrievalConfig(BaseModel):
|
||||
tokenizer: str
|
||||
scorer: str
|
||||
k1: float
|
||||
b: float
|
||||
title_weight: int
|
||||
stopwords_version: str
|
||||
|
||||
|
||||
class RetrievalResponse(BaseModel):
|
||||
run_id: UUID
|
||||
query: str
|
||||
mode: Literal["keyword"]
|
||||
status: Literal["ok", "no_evidence"]
|
||||
top_k: int
|
||||
category: str | None
|
||||
query_terms: list[str]
|
||||
corpus_size: int
|
||||
corpus_fingerprint: str
|
||||
total_hits: int
|
||||
results: list[RetrievalHit]
|
||||
elapsed_ms: float
|
||||
created_at: datetime
|
||||
config: RetrievalConfig
|
||||
@@ -0,0 +1,88 @@
|
||||
from datetime import datetime
|
||||
from typing import Literal
|
||||
from uuid import UUID
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
|
||||
from app.schemas.catalog import Category, Input
|
||||
from app.schemas.indexing import TimestampRead
|
||||
from app.schemas.retrieval import RetrievalHit
|
||||
|
||||
|
||||
class EmbeddingSpecRead(BaseModel):
|
||||
model: str
|
||||
revision: str
|
||||
dimension: int
|
||||
token_limit: int
|
||||
query_instruction: str
|
||||
implementation: str
|
||||
checkpoint_format: str
|
||||
|
||||
|
||||
class SpaceRead(BaseModel):
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
id: str
|
||||
dimension: int
|
||||
spec: EmbeddingSpecRead
|
||||
|
||||
|
||||
class BuildVectors(Input):
|
||||
text_index_version_id: UUID
|
||||
|
||||
|
||||
class VectorRead(TimestampRead):
|
||||
id: UUID
|
||||
text_index_version_id: UUID
|
||||
space_id: str
|
||||
dimension: int
|
||||
chunk_count: int
|
||||
status: Literal["ready", "published", "superseded"]
|
||||
created_by: int
|
||||
created_at: datetime
|
||||
published_at: datetime | None
|
||||
|
||||
|
||||
class ActiveVectors(BaseModel):
|
||||
generation: int
|
||||
text_generation: int
|
||||
catalog_revision: int
|
||||
is_current: bool
|
||||
active_version: VectorRead | None
|
||||
|
||||
|
||||
class VectorRequest(Input):
|
||||
query: str = Field(min_length=1, max_length=500)
|
||||
mode: Literal["vector"] = "vector"
|
||||
top_k: int = Field(5, ge=1, le=20)
|
||||
category: Category | None = None
|
||||
|
||||
|
||||
class VectorHit(RetrievalHit):
|
||||
chunk_id: UUID
|
||||
chunk_ordinal: int
|
||||
|
||||
|
||||
class VectorRetrievalConfig(BaseModel):
|
||||
scorer: Literal["pgvector-cosine-exact-v1"] = "pgvector-cosine-exact-v1"
|
||||
ranking_unit: Literal["chunk"] = "chunk"
|
||||
space: SpaceRead
|
||||
|
||||
|
||||
class VectorRetrievalResponse(BaseModel):
|
||||
run_id: UUID
|
||||
query: str
|
||||
mode: Literal["vector"] = "vector"
|
||||
status: Literal["ok", "no_evidence"]
|
||||
top_k: int
|
||||
category: str | None
|
||||
vector_version_id: UUID
|
||||
text_index_version_id: UUID
|
||||
generation: int
|
||||
text_generation: int
|
||||
corpus_fingerprint: str
|
||||
corpus_size: int
|
||||
total_hits: int
|
||||
results: list[VectorHit]
|
||||
elapsed_ms: float
|
||||
created_at: datetime
|
||||
config: VectorRetrievalConfig
|
||||
@@ -0,0 +1,161 @@
|
||||
"""Database-backed sessions; no bearer secrets are stored in the database."""
|
||||
|
||||
import hashlib
|
||||
import hmac
|
||||
import re
|
||||
import secrets
|
||||
from dataclasses import dataclass
|
||||
from datetime import timedelta
|
||||
from functools import lru_cache
|
||||
|
||||
from argon2 import PasswordHasher
|
||||
from argon2.exceptions import InvalidHashError, VerificationError
|
||||
from fastapi import Depends, HTTPException, Request, Response
|
||||
from sqlalchemy import case, delete, select
|
||||
from sqlalchemy.dialects.postgresql import insert
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.core.config import settings
|
||||
from app.db.session import get_db
|
||||
from app.models.auth import AuthRateLimit, AuthSession, User
|
||||
from app.models.catalog import utc_now
|
||||
from app.schemas.auth import SessionRead, UserRead
|
||||
|
||||
COOKIE_NAME = "nalati_session"
|
||||
UNSAFE_METHODS = {"POST", "PUT", "PATCH", "DELETE"}
|
||||
PASSWORD_HASHER = PasswordHasher(time_cost=2, memory_cost=19456, parallelism=1)
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def dummy_hash():
|
||||
return PASSWORD_HASHER.hash(secrets.token_urlsafe(32))
|
||||
|
||||
|
||||
def check_password(encoded: str, password: str) -> bool:
|
||||
try:
|
||||
return PASSWORD_HASHER.verify(encoded, password)
|
||||
except (VerificationError, InvalidHashError):
|
||||
return False
|
||||
|
||||
|
||||
def token_hash(raw: str) -> str:
|
||||
return hashlib.sha256(raw.encode()).hexdigest()
|
||||
|
||||
|
||||
def csrf_token(raw: str) -> str:
|
||||
# Domain-separated derivative; knowing the CSRF token does not reveal the cookie.
|
||||
return hmac.new(raw.encode(), b"nalati-csrf-v1", hashlib.sha256).hexdigest()
|
||||
|
||||
|
||||
def require_origin(request: Request):
|
||||
if request.headers.get("origin") not in settings.auth_origin_list:
|
||||
raise HTTPException(403, "请求来源不受信任")
|
||||
|
||||
|
||||
def limit_auth_requests(db: Session, request: Request, username: str | None = None):
|
||||
"""Atomic, shared across workers and restarts; never trust forwarded client headers here."""
|
||||
now = utc_now()
|
||||
window = timedelta(minutes=5)
|
||||
buckets = [("ip:" + (request.client.host if request.client else "unknown"), 30)]
|
||||
if username is not None:
|
||||
buckets.append(("username:" + username, 10))
|
||||
blocked = False
|
||||
for identity, maximum in buckets:
|
||||
statement = insert(AuthRateLimit).values(
|
||||
key=token_hash(identity), window_start=now, attempts=1
|
||||
)
|
||||
expired = AuthRateLimit.window_start <= now - window
|
||||
statement = statement.on_conflict_do_update(
|
||||
index_elements=[AuthRateLimit.key],
|
||||
set_={
|
||||
"window_start": case((expired, now), else_=AuthRateLimit.window_start),
|
||||
"attempts": case((expired, 1), else_=AuthRateLimit.attempts + 1),
|
||||
},
|
||||
).returning(AuthRateLimit.attempts)
|
||||
blocked |= db.scalar(statement) > maximum
|
||||
db.execute(delete(AuthRateLimit).where(AuthRateLimit.window_start < now - timedelta(days=1)))
|
||||
# Persist even failed authentication; no password or raw address is logged.
|
||||
db.commit()
|
||||
if blocked:
|
||||
raise HTTPException(429, "认证请求过于频繁,请稍后重试", headers={"Retry-After": "300"})
|
||||
|
||||
|
||||
@dataclass
|
||||
class CurrentSession:
|
||||
user: User
|
||||
session: AuthSession
|
||||
raw_token: str
|
||||
|
||||
def response(self):
|
||||
return SessionRead(
|
||||
**UserRead.model_validate(self.user).model_dump(),
|
||||
csrf_token=csrf_token(self.raw_token),
|
||||
)
|
||||
|
||||
|
||||
def require_session(request: Request, db: Session = Depends(get_db)) -> CurrentSession:
|
||||
raw = request.cookies.get(COOKIE_NAME, "")
|
||||
if not re.fullmatch(r"[A-Za-z0-9_-]{43}", raw):
|
||||
raise HTTPException(401, "请先登录")
|
||||
session = db.get(AuthSession, token_hash(raw))
|
||||
if session is None:
|
||||
raise HTTPException(401, "登录已失效,请重新登录")
|
||||
# Lock the user before checking the current session again, serializing password changes/login.
|
||||
user = db.scalar(select(User).where(User.id == session.user_id).with_for_update())
|
||||
session = db.scalar(
|
||||
select(AuthSession)
|
||||
.where(AuthSession.token_hash == token_hash(raw))
|
||||
.execution_options(populate_existing=True)
|
||||
)
|
||||
if (
|
||||
user is None
|
||||
or session is None
|
||||
or session.expires_at <= utc_now()
|
||||
or session.credential_version != user.credential_version
|
||||
):
|
||||
raise HTTPException(401, "登录已失效,请重新登录")
|
||||
if request.method in UNSAFE_METHODS:
|
||||
require_origin(request)
|
||||
submitted = request.headers.get("x-csrf-token", "")
|
||||
if not hmac.compare_digest(submitted.encode(), csrf_token(raw).encode()):
|
||||
raise HTTPException(403, "CSRF 校验失败,请刷新登录状态后重试")
|
||||
return CurrentSession(user, session, raw)
|
||||
|
||||
|
||||
def require_admin(current: CurrentSession = Depends(require_session)):
|
||||
if current.user.role != "admin":
|
||||
raise HTTPException(403, "需要管理员权限")
|
||||
return current
|
||||
|
||||
|
||||
def issue_session(db: Session, user: User, request: Request, response: Response):
|
||||
now = utc_now()
|
||||
old = request.cookies.get(COOKIE_NAME, "")
|
||||
db.execute(delete(AuthSession).where(AuthSession.expires_at <= now))
|
||||
if old:
|
||||
db.execute(delete(AuthSession).where(AuthSession.token_hash == token_hash(old)))
|
||||
raw = secrets.token_urlsafe(32)
|
||||
session = AuthSession(
|
||||
token_hash=token_hash(raw),
|
||||
user_id=user.id,
|
||||
credential_version=user.credential_version,
|
||||
expires_at=now + timedelta(seconds=settings.auth_session_seconds),
|
||||
)
|
||||
db.add(session)
|
||||
db.commit()
|
||||
response.set_cookie(
|
||||
COOKIE_NAME,
|
||||
raw,
|
||||
max_age=settings.auth_session_seconds,
|
||||
httponly=True,
|
||||
secure=settings.auth_cookie_secure,
|
||||
samesite="lax",
|
||||
path="/",
|
||||
)
|
||||
return CurrentSession(user, session, raw).response()
|
||||
|
||||
|
||||
def clear_cookie(response: Response):
|
||||
response.delete_cookie(
|
||||
COOKIE_NAME, path="/", httponly=True, secure=settings.auth_cookie_secure, samesite="lax"
|
||||
)
|
||||
@@ -0,0 +1,120 @@
|
||||
from fastapi import HTTPException
|
||||
from fastapi.encoders import jsonable_encoder
|
||||
from sqlalchemy import func, inspect, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.models import AuditEvent, Document, KnowledgeItem, Source
|
||||
from app.models.catalog import utc_now
|
||||
|
||||
|
||||
def get_item(db: Session, model, item_id: int, *, include_deleted=False, lock=False):
|
||||
if lock:
|
||||
item = db.scalar(
|
||||
select(model)
|
||||
.where(model.id == item_id)
|
||||
.with_for_update()
|
||||
.execution_options(populate_existing=True)
|
||||
)
|
||||
else:
|
||||
item = db.get(model, item_id)
|
||||
if item is None or (item.is_deleted and not include_deleted):
|
||||
raise HTTPException(404, "记录不存在")
|
||||
return item
|
||||
|
||||
|
||||
def ensure_version(item, expected_version: int):
|
||||
if item.version != expected_version:
|
||||
raise HTTPException(409, "记录已更新,请刷新后重试")
|
||||
|
||||
|
||||
def values(item) -> dict:
|
||||
return jsonable_encoder(
|
||||
{column.key: getattr(item, column.key) for column in inspect(type(item)).columns}
|
||||
)
|
||||
|
||||
|
||||
def record_event(db: Session, item, action: str, *, actor=None, notes=""):
|
||||
db.flush()
|
||||
snapshot = {"item": values(item)}
|
||||
if isinstance(item, KnowledgeItem):
|
||||
source = db.get(Source, item.source_id) if item.source_id else None
|
||||
document = db.get(Document, item.document_id) if item.document_id else None
|
||||
snapshot["source"] = values(source) if source else None
|
||||
snapshot["document"] = values(document) if document else None
|
||||
db.add(
|
||||
AuditEvent(
|
||||
entity_type=item.__tablename__,
|
||||
entity_id=item.id,
|
||||
action=action,
|
||||
actor=actor,
|
||||
notes=notes,
|
||||
snapshot=snapshot,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def update_values(item, payload):
|
||||
data = payload.model_dump(mode="json", exclude_unset=True, exclude={"expected_version"})
|
||||
data.update(
|
||||
{
|
||||
key: value
|
||||
for key, value in payload.model_dump(exclude_unset=True).items()
|
||||
if key.endswith("_at")
|
||||
}
|
||||
)
|
||||
for name, value in data.items():
|
||||
setattr(item, name, value)
|
||||
item.updated_at = utc_now()
|
||||
|
||||
|
||||
def reset_review(item: KnowledgeItem):
|
||||
item.verification_status = "pending"
|
||||
item.reviewer = None
|
||||
item.reviewed_at = None
|
||||
item.verification_notes = None
|
||||
item.updated_at = utc_now()
|
||||
|
||||
|
||||
def invalidate_knowledge(db: Session, condition, *, source_url=None, update_url=False):
|
||||
for item in db.scalars(
|
||||
select(KnowledgeItem).where(condition, KnowledgeItem.is_deleted.is_(False))
|
||||
):
|
||||
reset_review(item)
|
||||
if update_url:
|
||||
item.source_url = source_url
|
||||
record_event(db, item, "invalidate", notes="来源或文档已变更,需要重新核验")
|
||||
|
||||
|
||||
def resolve_knowledge_links(db: Session, item: KnowledgeItem):
|
||||
# 先锁依据记录再保存条目,避免引用与父记录修改/删除同时发生。
|
||||
with db.no_autoflush:
|
||||
if item.document_id is not None:
|
||||
document = get_item(db, Document, item.document_id, lock=True)
|
||||
if item.source_id is not None and item.source_id != document.source_id:
|
||||
raise HTTPException(422, "文档与来源不一致")
|
||||
item.source_id = document.source_id
|
||||
if item.source_id is not None:
|
||||
source = get_item(db, Source, item.source_id, lock=True)
|
||||
if item.source_url and item.source_url != source.source_url:
|
||||
raise HTTPException(422, "来源网址与登记来源不一致")
|
||||
item.source_url = source.source_url
|
||||
|
||||
|
||||
def page_result(db: Session, statement, page: int, page_size: int):
|
||||
total = db.scalar(select(func.count()).select_from(statement.order_by(None).subquery()))
|
||||
return {
|
||||
"items": list(db.scalars(statement.offset((page - 1) * page_size).limit(page_size))),
|
||||
"total": total,
|
||||
"page": page,
|
||||
"page_size": page_size,
|
||||
}
|
||||
|
||||
|
||||
def history(db: Session, model, item_id: int, page: int, page_size: int):
|
||||
get_item(db, model, item_id, include_deleted=True)
|
||||
statement = (
|
||||
select(AuditEvent)
|
||||
.where(AuditEvent.entity_type == model.__tablename__, AuditEvent.entity_id == item_id)
|
||||
.order_by(AuditEvent.id.desc())
|
||||
)
|
||||
return page_result(db, statement, page, page_size)
|
||||
@@ -0,0 +1,165 @@
|
||||
"""Optional local embeddings; importing this module never loads model weights or PyTorch."""
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import re
|
||||
from dataclasses import asdict, dataclass
|
||||
from typing import Protocol
|
||||
|
||||
QUERY_INSTRUCTION = "为这个句子生成表示以用于检索相关文章:"
|
||||
|
||||
|
||||
class EmbeddingError(ValueError):
|
||||
pass
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class EmbeddingSpec:
|
||||
model: str
|
||||
revision: str
|
||||
dimension: int
|
||||
token_limit: int = 512
|
||||
query_instruction: str = QUERY_INSTRUCTION
|
||||
implementation: str = "hf-cls-l2-cpu-fp32-v1"
|
||||
checkpoint_format: str = "safetensors"
|
||||
|
||||
def __post_init__(self):
|
||||
if not isinstance(self.revision, str) or not re.fullmatch(r"[0-9a-f]{40}", self.revision):
|
||||
raise EmbeddingError("模型必须固定到完整 revision SHA")
|
||||
if (
|
||||
type(self.dimension) is not int
|
||||
or type(self.token_limit) is not int
|
||||
or self.dimension < 1
|
||||
or self.token_limit < 2
|
||||
):
|
||||
raise EmbeddingError("向量维度和 token 上限无效")
|
||||
if (
|
||||
not isinstance(self.model, str)
|
||||
or not self.model
|
||||
or not isinstance(self.query_instruction, str)
|
||||
):
|
||||
raise EmbeddingError("模型名称或查询指令无效")
|
||||
if self.checkpoint_format not in {"safetensors", "pytorch-weights-only"}:
|
||||
raise EmbeddingError("不支持的权重格式")
|
||||
|
||||
@property
|
||||
def space_id(self) -> str:
|
||||
return hashlib.sha256(
|
||||
json.dumps(asdict(self), sort_keys=True, ensure_ascii=False).encode()
|
||||
).hexdigest()
|
||||
|
||||
|
||||
SMALL_ZH = EmbeddingSpec("BAAI/bge-small-zh-v1.5", "7999e1d3359715c523056ef9478215996d62a620", 512)
|
||||
BASE_ZH = EmbeddingSpec(
|
||||
"BAAI/bge-base-zh-v1.5",
|
||||
"f03589ceff5aac7111bd60cfc7d497ca17ecac65",
|
||||
768,
|
||||
checkpoint_format="pytorch-weights-only",
|
||||
)
|
||||
LOCAL_MODELS = {"small": SMALL_ZH, "base": BASE_ZH}
|
||||
|
||||
|
||||
class EmbeddingProvider(Protocol):
|
||||
spec: EmbeddingSpec
|
||||
|
||||
def encode(self, texts: list[str], *, query: bool = False) -> list[list[float]]: ...
|
||||
|
||||
|
||||
def validate_vectors(vectors: list[list[float]], count: int, spec: EmbeddingSpec):
|
||||
if len(vectors) != count:
|
||||
raise EmbeddingError("模型返回数量与输入不一致")
|
||||
for vector in vectors:
|
||||
try:
|
||||
finite = all(type(x) in {float, int} and math.isfinite(x) for x in vector)
|
||||
except (TypeError, OverflowError):
|
||||
finite = False
|
||||
if len(vector) != spec.dimension or not finite:
|
||||
raise EmbeddingError("模型返回维度错误或非有限数值")
|
||||
if not math.isclose(sum(x * x for x in vector), 1, abs_tol=1e-4):
|
||||
raise EmbeddingError("模型输出没有正确 L2 归一化")
|
||||
|
||||
|
||||
class LocalEmbeddingProvider:
|
||||
def __init__(
|
||||
self,
|
||||
spec: EmbeddingSpec,
|
||||
*,
|
||||
batch_size: int = 8,
|
||||
threads: int = 2,
|
||||
cache_dir=None,
|
||||
offline: bool = True,
|
||||
):
|
||||
if (
|
||||
type(batch_size) is not int
|
||||
or type(threads) is not int
|
||||
or not 1 <= batch_size <= 32
|
||||
or not 1 <= threads <= 4
|
||||
):
|
||||
raise EmbeddingError("CPU batch_size 必须为 1–32,threads 必须为 1–4")
|
||||
import torch
|
||||
from huggingface_hub import snapshot_download
|
||||
from transformers import AutoModel, AutoTokenizer
|
||||
|
||||
torch.set_num_threads(threads)
|
||||
self.spec, self.batch_size, self.torch = spec, batch_size, torch
|
||||
path = snapshot_download(
|
||||
repo_id=spec.model,
|
||||
revision=spec.revision,
|
||||
cache_dir=cache_dir,
|
||||
local_files_only=offline,
|
||||
token=False,
|
||||
allow_patterns=[
|
||||
"config.json",
|
||||
"model.safetensors"
|
||||
if spec.checkpoint_format == "safetensors"
|
||||
else "pytorch_model.bin",
|
||||
"tokenizer.json",
|
||||
"tokenizer_config.json",
|
||||
"special_tokens_map.json",
|
||||
"vocab.txt",
|
||||
"added_tokens.json",
|
||||
],
|
||||
)
|
||||
self.tokenizer = AutoTokenizer.from_pretrained(
|
||||
path, local_files_only=True, trust_remote_code=False
|
||||
)
|
||||
self.model = (
|
||||
AutoModel.from_pretrained(
|
||||
path,
|
||||
local_files_only=True,
|
||||
trust_remote_code=False,
|
||||
use_safetensors=spec.checkpoint_format == "safetensors",
|
||||
weights_only=True,
|
||||
dtype=torch.float32,
|
||||
)
|
||||
.to("cpu")
|
||||
.eval()
|
||||
)
|
||||
if self.model.config.hidden_size != spec.dimension:
|
||||
raise EmbeddingError("模型实际维度与配置不一致")
|
||||
if self.tokenizer.model_max_length != spec.token_limit:
|
||||
raise EmbeddingError("模型 tokenizer 上限与配置不一致")
|
||||
|
||||
def encode(self, texts: list[str], *, query: bool = False) -> list[list[float]]:
|
||||
if not isinstance(texts, list) or any(
|
||||
not isinstance(text, str) or not text.strip() or len(text) > 10_000 for text in texts
|
||||
):
|
||||
raise EmbeddingError("每条输入须为非空文本且最多 10000 字符")
|
||||
torch = self.torch
|
||||
vectors = []
|
||||
for offset in range(0, len(texts), self.batch_size):
|
||||
batch = texts[offset : offset + self.batch_size]
|
||||
if query:
|
||||
batch = [self.spec.query_instruction + text for text in batch]
|
||||
encoded = self.tokenizer(
|
||||
batch, padding=True, truncation=False, return_tensors="pt", verbose=False
|
||||
)
|
||||
if encoded["input_ids"].shape[1] > self.spec.token_limit:
|
||||
raise EmbeddingError("输入超出模型 token 上限;请重新切分,禁止静默截断")
|
||||
with torch.inference_mode():
|
||||
features = self.model(**encoded).last_hidden_state[:, 0]
|
||||
normalized = torch.nn.functional.normalize(features, p=2, dim=1)
|
||||
vectors.extend(normalized.tolist())
|
||||
validate_vectors(vectors, len(texts), self.spec)
|
||||
return vectors
|
||||
@@ -0,0 +1,49 @@
|
||||
"""Offline, lazy CPU runtime; one encoder call per process, with a fast busy response."""
|
||||
|
||||
from functools import lru_cache
|
||||
from threading import Lock
|
||||
|
||||
from fastapi import HTTPException
|
||||
|
||||
from app.core.config import settings
|
||||
from app.services.embedding import LOCAL_MODELS, EmbeddingSpec, LocalEmbeddingProvider
|
||||
|
||||
|
||||
class EmbeddingRuntime:
|
||||
def __init__(self, spec: EmbeddingSpec, cache_dir, batch_size: int, threads: int):
|
||||
self.spec = spec
|
||||
self.cache_dir, self.batch_size, self.threads = cache_dir, batch_size, threads
|
||||
self._provider = None
|
||||
self._lock = Lock()
|
||||
|
||||
def encode(self, texts: list[str], *, query: bool = False) -> list[list[float]]:
|
||||
if not self._lock.acquire(blocking=False):
|
||||
raise HTTPException(503, "本地编码器忙,请稍后重试")
|
||||
try:
|
||||
if self._provider is None:
|
||||
self._provider = LocalEmbeddingProvider(
|
||||
self.spec,
|
||||
cache_dir=self.cache_dir,
|
||||
batch_size=self.batch_size,
|
||||
threads=self.threads,
|
||||
offline=True,
|
||||
)
|
||||
return self._provider.encode(texts, query=query)
|
||||
finally:
|
||||
self._lock.release()
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def runtime(spec, cache_dir, batch_size, threads):
|
||||
return EmbeddingRuntime(spec, cache_dir, batch_size, threads)
|
||||
|
||||
|
||||
def get_embedding_provider():
|
||||
if not settings.embedding_enabled:
|
||||
raise HTTPException(503, "本地 Embedding 未启用,请先完成模型缓存和运行环境配置")
|
||||
return runtime(
|
||||
LOCAL_MODELS[settings.embedding_model],
|
||||
settings.embedding_cache_dir,
|
||||
settings.embedding_batch_size,
|
||||
settings.embedding_threads,
|
||||
)
|
||||
@@ -0,0 +1,205 @@
|
||||
"""Bounded chunk BM25 and equal-weight reciprocal rank fusion on one publication."""
|
||||
|
||||
from collections import Counter
|
||||
from time import perf_counter
|
||||
from uuid import uuid4
|
||||
|
||||
from fastapi import HTTPException
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.models.catalog import utc_now
|
||||
from app.models.indexing import Chunk
|
||||
from app.models.retrieval import RetrievalRun
|
||||
from app.schemas.hybrid import (
|
||||
ChunkKeywordRequest,
|
||||
ChunkKeywordResponse,
|
||||
ChunkRetrievalConfig,
|
||||
HybridConfig,
|
||||
HybridHit,
|
||||
HybridRequest,
|
||||
HybridResponse,
|
||||
RankedCandidate,
|
||||
)
|
||||
from app.schemas.vectors import SpaceRead, VectorHit
|
||||
from app.services.indexing import locked_state
|
||||
from app.services.retrieval import CONFIG, bm25, tokenize
|
||||
from app.services.vectors import (
|
||||
checked_chunks,
|
||||
current_text,
|
||||
encode,
|
||||
rank_vectors,
|
||||
retrieval_scope,
|
||||
)
|
||||
|
||||
RRF_K = 60
|
||||
CHUNK_LEXICAL_CONFIG = CONFIG.model_copy(update={"scorer": "bm25-chunk-body-v1", "title_weight": 0})
|
||||
|
||||
|
||||
def chunk_key(chunk):
|
||||
return chunk.knowledge_id, chunk.ordinal, chunk.id
|
||||
|
||||
|
||||
def chunks_in_scope(db, text_version, category):
|
||||
statement = select(Chunk).where(Chunk.index_version_id == text_version.id)
|
||||
if category is not None:
|
||||
statement = statement.where(Chunk.provenance["category"].as_string() == category)
|
||||
return list(db.scalars(statement.order_by(Chunk.knowledge_id, Chunk.ordinal, Chunk.id)))
|
||||
|
||||
|
||||
def lexical_ranking(chunks, terms):
|
||||
counters = {chunk.id: Counter(tokenize(chunk.text)) for chunk in chunks}
|
||||
scores = bm25(list(counters.values()), terms)
|
||||
ranked = sorted(
|
||||
((chunk, score) for chunk, score in zip(chunks, scores) if score > 0),
|
||||
key=lambda pair: (-pair[1], chunk_key(pair[0])),
|
||||
)
|
||||
return ranked, counters
|
||||
|
||||
|
||||
def candidate_list(rows):
|
||||
return [
|
||||
RankedCandidate(chunk_id=chunk.id, rank=rank, score=score)
|
||||
for rank, (chunk, score) in enumerate(rows, 1)
|
||||
]
|
||||
|
||||
|
||||
def fuse(lexical, vector):
|
||||
"""Ranks start at one; an absent candidate contributes zero, never a padded rank."""
|
||||
scores = {}
|
||||
for leg in (lexical, vector):
|
||||
for candidate in leg:
|
||||
key = str(candidate.chunk_id)
|
||||
scores[key] = scores.get(key, 0.0) + 1.0 / (RRF_K + candidate.rank)
|
||||
return scores
|
||||
|
||||
|
||||
def hit_fields(chunk, terms, counters):
|
||||
return dict(
|
||||
**chunk.provenance,
|
||||
chunk_id=chunk.id,
|
||||
chunk_ordinal=chunk.ordinal,
|
||||
snippet=chunk.text,
|
||||
snippet_start=chunk.start_offset,
|
||||
snippet_end=chunk.end_offset,
|
||||
matched_terms=[term for term in terms if counters[chunk.id][term]],
|
||||
)
|
||||
|
||||
|
||||
def save_run(db, response):
|
||||
db.add(
|
||||
RetrievalRun(
|
||||
id=str(response.run_id),
|
||||
query=response.query,
|
||||
mode=response.mode,
|
||||
corpus_fingerprint=response.corpus_fingerprint,
|
||||
response=response.model_dump(mode="json"),
|
||||
created_at=response.created_at,
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
return response
|
||||
|
||||
|
||||
def response_fields(payload, state, text_version, chunks, terms, total, started):
|
||||
return dict(
|
||||
run_id=uuid4(),
|
||||
query=payload.query,
|
||||
status="ok" if total else "no_evidence",
|
||||
top_k=payload.top_k,
|
||||
category=payload.category,
|
||||
query_terms=terms,
|
||||
text_index_version_id=text_version.id,
|
||||
text_generation=state.generation,
|
||||
catalog_revision=state.catalog_revision,
|
||||
corpus_fingerprint=text_version.corpus_fingerprint,
|
||||
corpus_size=len(chunks),
|
||||
total_hits=total,
|
||||
elapsed_ms=round((perf_counter() - started) * 1000, 3),
|
||||
created_at=utc_now(),
|
||||
)
|
||||
|
||||
|
||||
def retrieve_chunks(db: Session, payload: ChunkKeywordRequest):
|
||||
started = perf_counter()
|
||||
terms = list(dict.fromkeys(tokenize(payload.query)))
|
||||
if not terms:
|
||||
raise HTTPException(422, "问题中没有可检索词,请补充具体关键词")
|
||||
state = locked_state(db)
|
||||
text_version, corpus = current_text(db, state)
|
||||
checked_chunks(db, text_version, corpus)
|
||||
chunks = chunks_in_scope(db, text_version, payload.category)
|
||||
ranked, counters = lexical_ranking(chunks, terms)
|
||||
hits = [
|
||||
VectorHit(**hit_fields(chunk, terms, counters), rank=rank, score=score)
|
||||
for rank, (chunk, score) in enumerate(ranked[: payload.top_k], 1)
|
||||
]
|
||||
return save_run(
|
||||
db,
|
||||
ChunkKeywordResponse(
|
||||
**response_fields(payload, state, text_version, chunks, terms, len(ranked), started),
|
||||
results=hits,
|
||||
config=ChunkRetrievalConfig(lexical=CHUNK_LEXICAL_CONFIG, chunking=text_version.config),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def retrieve_hybrid(db: Session, payload: HybridRequest, provider):
|
||||
started = perf_counter()
|
||||
terms = list(dict.fromkeys(tokenize(payload.query)))
|
||||
*_, token = retrieval_scope(db, provider)
|
||||
db.commit()
|
||||
query_vector = encode(provider, [payload.query], query=True)[0]
|
||||
state, vector_state, text_version, version, space, current = retrieval_scope(db, provider)
|
||||
if token != current:
|
||||
raise HTTPException(409, "查询编码期间发布版本或审核依据已变化,请重试")
|
||||
# Hold the shared catalog/publication locks until both rankings and the run are saved.
|
||||
chunks = chunks_in_scope(db, text_version, payload.category)
|
||||
ranked, counters = lexical_ranking(chunks, terms)
|
||||
lexical = candidate_list(ranked[: payload.candidate_k])
|
||||
vector_total, vector_rows = rank_vectors(
|
||||
db, version, space, payload.category, payload.candidate_k, query_vector
|
||||
)
|
||||
vector = candidate_list(
|
||||
[(chunk, max(-1.0, min(1.0, 1.0 - float(distance)))) for chunk, distance in vector_rows]
|
||||
)
|
||||
scores = fuse(lexical, vector)
|
||||
by_id = {chunk.id: chunk for chunk in chunks}
|
||||
ordered = sorted(scores, key=lambda key: (-scores[key], chunk_key(by_id[key])))
|
||||
lexical_by_id = {str(c.chunk_id): c for c in lexical}
|
||||
vector_by_id = {str(c.chunk_id): c for c in vector}
|
||||
hits = []
|
||||
for rank, key in enumerate(ordered[: payload.top_k], 1):
|
||||
left, right = lexical_by_id.get(key), vector_by_id.get(key)
|
||||
hits.append(
|
||||
HybridHit(
|
||||
**hit_fields(by_id[key], terms, counters),
|
||||
rank=rank,
|
||||
score=scores[key],
|
||||
lexical_rank=left.rank if left else None,
|
||||
lexical_score=left.score if left else None,
|
||||
vector_rank=right.rank if right else None,
|
||||
vector_score=right.score if right else None,
|
||||
lexical_contribution=1.0 / (RRF_K + left.rank) if left else 0.0,
|
||||
vector_contribution=1.0 / (RRF_K + right.rank) if right else 0.0,
|
||||
)
|
||||
)
|
||||
return save_run(
|
||||
db,
|
||||
HybridResponse(
|
||||
**response_fields(payload, state, text_version, chunks, terms, len(scores), started),
|
||||
vector_version_id=version.id,
|
||||
generation=vector_state.generation,
|
||||
lexical_total_hits=len(ranked),
|
||||
vector_total_hits=vector_total,
|
||||
lexical_candidates=lexical,
|
||||
vector_candidates=vector,
|
||||
results=hits,
|
||||
config=HybridConfig(
|
||||
lexical=CHUNK_LEXICAL_CONFIG,
|
||||
chunking=text_version.config,
|
||||
candidate_k=payload.candidate_k,
|
||||
space=SpaceRead.model_validate(space),
|
||||
),
|
||||
),
|
||||
)
|
||||
@@ -0,0 +1,147 @@
|
||||
"""Bounded chunk snapshots and serialized, optimistic publication."""
|
||||
|
||||
import hashlib
|
||||
from uuid import UUID, uuid4, uuid5
|
||||
|
||||
from fastapi import Depends, HTTPException, Request
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.db.session import get_db
|
||||
from app.models.catalog import utc_now
|
||||
from app.models.indexing import Chunk, IndexPublication, IndexState, IndexVersion
|
||||
from app.schemas.indexing import ActiveIndex, BuildIndex, IndexRead
|
||||
from app.services.retrieval import corpus_fingerprint, load_corpus
|
||||
|
||||
MAX_CHUNKS = 20_000
|
||||
|
||||
|
||||
def locked_state(db: Session) -> IndexState:
|
||||
state = db.scalar(
|
||||
select(IndexState)
|
||||
.where(IndexState.id == 1)
|
||||
.with_for_update()
|
||||
.execution_options(populate_existing=True)
|
||||
)
|
||||
if state is None:
|
||||
raise HTTPException(503, "索引状态尚未初始化,请检查数据库迁移")
|
||||
return state
|
||||
|
||||
|
||||
def catalog_write_guard(request: Request, db: Session = Depends(get_db)):
|
||||
if request.method in {"POST", "PATCH", "PUT", "DELETE"}:
|
||||
# Always before catalog/parent row locks; failures roll back this revision as well.
|
||||
state = locked_state(db)
|
||||
state.catalog_revision += 1
|
||||
|
||||
|
||||
def version_or_404(db: Session, version_id: str) -> IndexVersion:
|
||||
version = db.get(IndexVersion, version_id)
|
||||
if version is None:
|
||||
raise HTTPException(404, "索引版本不存在")
|
||||
return version
|
||||
|
||||
|
||||
def split_text(content: str, size: int, overlap: int):
|
||||
"""Offsets are original Python Unicode character offsets, end-exclusive."""
|
||||
for start in range(0, len(content), size - overlap):
|
||||
end = min(len(content), start + size)
|
||||
yield start, end, content[start:end]
|
||||
if end == len(content):
|
||||
break
|
||||
|
||||
|
||||
def build_index(db: Session, payload: BuildIndex, actor_id: int) -> IndexRead:
|
||||
state = locked_state(db)
|
||||
corpus = load_corpus(db, None)
|
||||
version_id = uuid4()
|
||||
chunks = []
|
||||
for item in corpus:
|
||||
provenance = {key: value for key, value in item.items() if key != "content"}
|
||||
for ordinal, (start, end, content) in enumerate(
|
||||
split_text(item["content"], payload.chunk_size, payload.overlap)
|
||||
):
|
||||
if len(chunks) >= MAX_CHUNKS:
|
||||
raise HTTPException(503, "切分数量超过本阶段容量,请增加切分长度或减少重叠")
|
||||
chunks.append(
|
||||
Chunk(
|
||||
id=str(uuid5(version_id, f"{item['knowledge_id']}:{ordinal}")),
|
||||
index_version_id=str(version_id),
|
||||
knowledge_id=item["knowledge_id"],
|
||||
ordinal=ordinal,
|
||||
start_offset=start,
|
||||
end_offset=end,
|
||||
text=content,
|
||||
text_sha256=hashlib.sha256(content.encode()).hexdigest(),
|
||||
provenance=provenance,
|
||||
)
|
||||
)
|
||||
version = IndexVersion(
|
||||
id=str(version_id),
|
||||
status="ready",
|
||||
config={"splitter": "unicode-window-v1", **payload.model_dump()},
|
||||
corpus_fingerprint=corpus_fingerprint(corpus),
|
||||
catalog_revision=state.catalog_revision,
|
||||
item_count=len(corpus),
|
||||
chunk_count=len(chunks),
|
||||
created_by=actor_id,
|
||||
)
|
||||
db.add(version)
|
||||
db.flush()
|
||||
db.add_all(chunks)
|
||||
result = IndexRead.model_validate(version)
|
||||
db.commit()
|
||||
return result
|
||||
|
||||
|
||||
def active_index(db: Session, state: IndexState | None = None) -> ActiveIndex:
|
||||
state = state or locked_state(db)
|
||||
version = db.get(IndexVersion, state.active_version_id) if state.active_version_id else None
|
||||
current = bool(version and version.catalog_revision == state.catalog_revision)
|
||||
if current:
|
||||
current = version.corpus_fingerprint == corpus_fingerprint(load_corpus(db, None))
|
||||
return ActiveIndex(
|
||||
generation=state.generation,
|
||||
catalog_revision=state.catalog_revision,
|
||||
is_current=current,
|
||||
active_version=IndexRead.model_validate(version) if version else None,
|
||||
)
|
||||
|
||||
|
||||
def publish_index(db: Session, version_id: UUID, expected_generation: int, actor_id: int):
|
||||
state = locked_state(db)
|
||||
if state.generation != expected_generation:
|
||||
raise HTTPException(409, "已发布版本发生变化,请刷新后重试")
|
||||
version = version_or_404(db, str(version_id))
|
||||
if (
|
||||
version.catalog_revision != state.catalog_revision
|
||||
or version.corpus_fingerprint != corpus_fingerprint(load_corpus(db, None))
|
||||
):
|
||||
raise HTTPException(409, "资料或审核依据已变化,请重新构建后发布")
|
||||
if state.active_version_id == version.id:
|
||||
return active_index(db, state)
|
||||
previous = state.active_version_id
|
||||
if previous:
|
||||
db.get(IndexVersion, previous).status = "superseded"
|
||||
state.active_version_id = version.id
|
||||
state.generation += 1
|
||||
version.status = "published"
|
||||
version.published_at = utc_now()
|
||||
db.add(
|
||||
IndexPublication(
|
||||
generation=state.generation,
|
||||
previous_version_id=previous,
|
||||
version_id=version.id,
|
||||
actor_id=actor_id,
|
||||
published_at=version.published_at,
|
||||
)
|
||||
)
|
||||
# Freeze the response before commit expires ORM objects and releases the lock.
|
||||
result = ActiveIndex(
|
||||
generation=expected_generation + 1,
|
||||
catalog_revision=version.catalog_revision,
|
||||
is_current=True,
|
||||
active_version=IndexRead.model_validate(version),
|
||||
)
|
||||
db.commit()
|
||||
return result
|
||||
@@ -0,0 +1,219 @@
|
||||
"""Deterministic lexical baseline over a bounded, reviewed knowledge corpus."""
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
import math
|
||||
import unicodedata
|
||||
from collections import Counter
|
||||
from functools import lru_cache
|
||||
from time import perf_counter
|
||||
from uuid import uuid4
|
||||
|
||||
import jieba
|
||||
from fastapi import HTTPException
|
||||
from sqlalchemy import func, or_, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.models import Document, KnowledgeItem, RetrievalRun, Source
|
||||
from app.models.catalog import utc_now
|
||||
from app.schemas.retrieval import RetrievalConfig, RetrievalHit, RetrievalRequest, RetrievalResponse
|
||||
|
||||
MAX_ITEMS = 5000
|
||||
MAX_CHARACTERS = 2_000_000
|
||||
SNIPPET_LENGTH = 240
|
||||
STOPWORDS = frozenset("的 了 和 与 是 在 有 哪些 什么 请问 如何 吗 呢 我 你 想 了解".split())
|
||||
CONFIG = RetrievalConfig(
|
||||
tokenizer="jieba-0.42.1-search-hmm-off-nalati-v1",
|
||||
scorer="bm25-weighted-text-v1",
|
||||
k1=1.2,
|
||||
b=0.75,
|
||||
title_weight=2,
|
||||
stopwords_version="zh-small-v1",
|
||||
)
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def tokenizer():
|
||||
jieba.setLogLevel(logging.WARNING)
|
||||
instance = jieba.Tokenizer()
|
||||
instance.initialize()
|
||||
instance.add_word("那拉提", freq=100000)
|
||||
return instance
|
||||
|
||||
|
||||
def normalized_with_offsets(text: str) -> tuple[str, list[int]]:
|
||||
pieces, offsets = [], []
|
||||
for position, character in enumerate(text):
|
||||
piece = unicodedata.normalize("NFKC", character).casefold()
|
||||
pieces.append(piece)
|
||||
offsets.extend([position] * len(piece))
|
||||
return "".join(pieces), offsets
|
||||
|
||||
|
||||
def tokenize(text: str) -> list[str]:
|
||||
normalized, _ = normalized_with_offsets(text)
|
||||
return [
|
||||
term
|
||||
for term in tokenizer().cut_for_search(normalized, HMM=False)
|
||||
if term not in STOPWORDS and any(character.isalnum() for character in term)
|
||||
]
|
||||
|
||||
|
||||
def bm25(counters: list[Counter], query_terms: list[str]) -> list[float]:
|
||||
"""Positive log-IDF BM25; repeated query terms count once."""
|
||||
count = len(counters)
|
||||
lengths = [sum(counter.values()) for counter in counters]
|
||||
average = sum(lengths) / count if count else 0
|
||||
if not average:
|
||||
return [0.0] * count
|
||||
scores = [0.0] * count
|
||||
for term in dict.fromkeys(query_terms):
|
||||
frequency = sum(counter[term] > 0 for counter in counters)
|
||||
if not frequency:
|
||||
continue
|
||||
idf = math.log(1 + (count - frequency + 0.5) / (frequency + 0.5))
|
||||
for index, counter in enumerate(counters):
|
||||
tf = counter[term]
|
||||
if tf:
|
||||
denominator = tf + CONFIG.k1 * (1 - CONFIG.b + CONFIG.b * lengths[index] / average)
|
||||
scores[index] += idf * tf * (CONFIG.k1 + 1) / denominator
|
||||
return scores
|
||||
|
||||
|
||||
def excerpt(content: str, matched_terms: list[str]) -> tuple[str, int, int]:
|
||||
normalized, offsets = normalized_with_offsets(content)
|
||||
positions = [normalized.find(term) for term in matched_terms if term in normalized]
|
||||
start = max(0, offsets[min(positions)] - 60) if positions else 0
|
||||
end = min(len(content), start + SNIPPET_LENGTH)
|
||||
return content[start:end], start, end
|
||||
|
||||
|
||||
def load_corpus(db: Session, category: str | None) -> list[dict]:
|
||||
statement = (
|
||||
select(
|
||||
KnowledgeItem,
|
||||
Source.name.label("source_name"),
|
||||
Source.version.label("source_version"),
|
||||
Document.version.label("document_version"),
|
||||
func.count().over().label("corpus_total"),
|
||||
func.sum(func.length(KnowledgeItem.title) + func.length(KnowledgeItem.content))
|
||||
.over()
|
||||
.label("corpus_characters"),
|
||||
)
|
||||
.outerjoin(Source, KnowledgeItem.source_id == Source.id)
|
||||
.outerjoin(Document, KnowledgeItem.document_id == Document.id)
|
||||
.where(
|
||||
KnowledgeItem.verification_status == "verified",
|
||||
KnowledgeItem.is_deleted.is_(False),
|
||||
KnowledgeItem.source_url.is_not(None),
|
||||
func.length(func.trim(KnowledgeItem.source_url)) > 0,
|
||||
KnowledgeItem.collected_at.is_not(None),
|
||||
KnowledgeItem.reviewed_at.is_not(None),
|
||||
KnowledgeItem.reviewer.is_not(None),
|
||||
KnowledgeItem.verification_notes.is_not(None),
|
||||
or_(KnowledgeItem.source_id.is_(None), Source.is_deleted.is_(False)),
|
||||
or_(
|
||||
KnowledgeItem.document_id.is_(None),
|
||||
(Document.is_deleted.is_(False)) & (Document.source_id == KnowledgeItem.source_id),
|
||||
),
|
||||
or_(KnowledgeItem.source_id.is_(None), KnowledgeItem.source_url == Source.source_url),
|
||||
)
|
||||
.order_by(KnowledgeItem.id)
|
||||
.limit(MAX_ITEMS + 1)
|
||||
)
|
||||
if category is not None:
|
||||
statement = statement.where(KnowledgeItem.category == category)
|
||||
corpus = []
|
||||
# 一条 SQL 的快照同时给出语料与容量;流式读取避免先载入超限正文。
|
||||
with db.execute(statement.execution_options(stream_results=True, max_row_buffer=1)) as rows:
|
||||
for row in rows:
|
||||
if row.corpus_total > MAX_ITEMS or row.corpus_characters > MAX_CHARACTERS:
|
||||
raise HTTPException(
|
||||
503, "已核验语料超过关键词基线容量,请缩小分类范围或建立检索索引"
|
||||
)
|
||||
item = row.KnowledgeItem
|
||||
corpus.append(
|
||||
{
|
||||
"knowledge_id": item.id,
|
||||
"knowledge_version": item.version,
|
||||
"document_id": item.document_id,
|
||||
"document_version": row.document_version,
|
||||
"source_id": item.source_id,
|
||||
"source_version": row.source_version,
|
||||
"source_name": row.source_name,
|
||||
"source_url": item.source_url,
|
||||
"title": item.title,
|
||||
"category": item.category,
|
||||
"content": item.content,
|
||||
}
|
||||
)
|
||||
return corpus
|
||||
|
||||
|
||||
def corpus_fingerprint(corpus: list[dict]) -> str:
|
||||
return hashlib.sha256(
|
||||
json.dumps(corpus, ensure_ascii=False, sort_keys=True, separators=(",", ":")).encode()
|
||||
).hexdigest()
|
||||
|
||||
|
||||
def retrieve(db: Session, payload: RetrievalRequest) -> RetrievalResponse:
|
||||
started = perf_counter()
|
||||
terms = list(dict.fromkeys(tokenize(payload.query)))
|
||||
if not terms:
|
||||
raise HTTPException(422, "问题中没有可检索词,请补充具体关键词")
|
||||
corpus = load_corpus(db, payload.category)
|
||||
fingerprint = corpus_fingerprint(corpus)
|
||||
counters = [
|
||||
Counter(tokenize(item["title"]) * CONFIG.title_weight + tokenize(item["content"]))
|
||||
for item in corpus
|
||||
]
|
||||
scores = bm25(counters, terms)
|
||||
ranked = sorted(
|
||||
(index for index, score in enumerate(scores) if score > 0),
|
||||
key=lambda index: (-scores[index], corpus[index]["knowledge_id"]),
|
||||
)
|
||||
results = []
|
||||
for rank, index in enumerate(ranked[: payload.top_k], start=1):
|
||||
item = corpus[index]
|
||||
matched = [term for term in terms if counters[index][term]]
|
||||
snippet, start, end = excerpt(item["content"], matched)
|
||||
results.append(
|
||||
RetrievalHit(
|
||||
**{key: value for key, value in item.items() if key != "content"},
|
||||
rank=rank,
|
||||
score=round(scores[index], 8),
|
||||
matched_terms=matched,
|
||||
snippet=snippet,
|
||||
snippet_start=start,
|
||||
snippet_end=end,
|
||||
)
|
||||
)
|
||||
response = RetrievalResponse(
|
||||
run_id=uuid4(),
|
||||
query=payload.query,
|
||||
mode="keyword",
|
||||
status="ok" if results else "no_evidence",
|
||||
top_k=payload.top_k,
|
||||
category=payload.category,
|
||||
query_terms=terms,
|
||||
corpus_size=len(corpus),
|
||||
corpus_fingerprint=fingerprint,
|
||||
total_hits=len(ranked),
|
||||
results=results,
|
||||
elapsed_ms=round((perf_counter() - started) * 1000, 3),
|
||||
created_at=utc_now(),
|
||||
config=CONFIG,
|
||||
)
|
||||
db.add(
|
||||
RetrievalRun(
|
||||
id=str(response.run_id),
|
||||
query=response.query,
|
||||
mode=response.mode,
|
||||
corpus_fingerprint=fingerprint,
|
||||
response=response.model_dump(mode="json"),
|
||||
created_at=response.created_at,
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
return response
|
||||
@@ -0,0 +1,356 @@
|
||||
"""Generate outside publication locks; recheck reviewed evidence before persistence/query."""
|
||||
|
||||
import hashlib
|
||||
from dataclasses import asdict
|
||||
from time import perf_counter
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
from fastapi import HTTPException
|
||||
from sqlalchemy import Float, func, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.models.catalog import utc_now
|
||||
from app.models.indexing import Chunk, IndexVersion
|
||||
from app.models.retrieval import RetrievalRun
|
||||
from app.models.vectors import (
|
||||
ChunkVector,
|
||||
EmbeddingSpace,
|
||||
VectorPublication,
|
||||
VectorState,
|
||||
VectorVersion,
|
||||
)
|
||||
from app.schemas.vectors import (
|
||||
ActiveVectors,
|
||||
SpaceRead,
|
||||
VectorHit,
|
||||
VectorRead,
|
||||
VectorRequest,
|
||||
VectorRetrievalConfig,
|
||||
VectorRetrievalResponse,
|
||||
)
|
||||
from app.services.embedding import EmbeddingError, EmbeddingProvider, validate_vectors
|
||||
from app.services.indexing import locked_state, split_text
|
||||
from app.services.retrieval import corpus_fingerprint, load_corpus
|
||||
|
||||
MAX_VECTOR_CHUNKS = 2000
|
||||
|
||||
|
||||
def vector_state(db: Session) -> VectorState:
|
||||
# Always acquire index_state first: the same ordering as catalog/text publication.
|
||||
state = db.scalar(
|
||||
select(VectorState)
|
||||
.where(VectorState.id == 1)
|
||||
.with_for_update()
|
||||
.execution_options(populate_existing=True)
|
||||
)
|
||||
if state is None:
|
||||
raise HTTPException(503, "向量状态未初始化,请检查迁移")
|
||||
return state
|
||||
|
||||
|
||||
def vector_or_404(db: Session, version_id: str) -> VectorVersion:
|
||||
version = db.get(VectorVersion, version_id)
|
||||
if version is None:
|
||||
raise HTTPException(404, "向量版本不存在")
|
||||
return version
|
||||
|
||||
|
||||
def current_text(db: Session, state):
|
||||
version = db.get(IndexVersion, state.active_version_id) if state.active_version_id else None
|
||||
if version is None or version.catalog_revision != state.catalog_revision:
|
||||
raise HTTPException(409, "尚无当前有效的文本发布版本,请构建并发布")
|
||||
corpus = load_corpus(db, None)
|
||||
if version.corpus_fingerprint != corpus_fingerprint(corpus):
|
||||
raise HTTPException(409, "审核依据已变化,请重新构建并发布文本版本")
|
||||
return version, corpus
|
||||
|
||||
|
||||
def encode(provider: EmbeddingProvider, texts: list[str], *, query=False):
|
||||
try:
|
||||
spec = provider.spec
|
||||
result = provider.encode(texts, query=query)
|
||||
if provider.spec != spec:
|
||||
raise EmbeddingError("编码期间模型身份改变")
|
||||
validate_vectors(result, len(texts), spec)
|
||||
return result
|
||||
except HTTPException:
|
||||
raise
|
||||
except EmbeddingError as exc:
|
||||
raise HTTPException(422, "编码校验失败,请核对文本 token 上限、向量维度和归一化") from exc
|
||||
except Exception as exc:
|
||||
# Optional dependency/cache failures must not expose server paths or library tracebacks.
|
||||
raise HTTPException(503, "本地模型不可用,请检查可选运行依赖和固定版本缓存") from exc
|
||||
|
||||
|
||||
def active_vectors(db: Session) -> ActiveVectors:
|
||||
state = locked_state(db)
|
||||
vector = vector_state(db)
|
||||
version = db.get(VectorVersion, vector.active_version_id) if vector.active_version_id else None
|
||||
current = False
|
||||
if version and version.text_index_version_id == state.active_version_id:
|
||||
try:
|
||||
text_version, corpus = current_text(db, state)
|
||||
checked_chunks(db, text_version, corpus)
|
||||
complete(db, version)
|
||||
current = version.chunk_count == text_version.chunk_count
|
||||
except HTTPException as exc:
|
||||
if exc.status_code != 409:
|
||||
raise
|
||||
return ActiveVectors(
|
||||
generation=vector.generation,
|
||||
text_generation=state.generation,
|
||||
catalog_revision=state.catalog_revision,
|
||||
is_current=current,
|
||||
active_version=VectorRead.model_validate(version) if version else None,
|
||||
)
|
||||
|
||||
|
||||
def checked_chunks(db, text_version, corpus):
|
||||
if text_version.chunk_count > MAX_VECTOR_CHUNKS:
|
||||
raise HTTPException(503, "超过同步向量构建容量(2000 chunks)")
|
||||
chunks = list(
|
||||
db.scalars(
|
||||
select(Chunk)
|
||||
.where(Chunk.index_version_id == text_version.id)
|
||||
.order_by(Chunk.knowledge_id, Chunk.ordinal)
|
||||
.limit(MAX_VECTOR_CHUNKS + 1)
|
||||
)
|
||||
)
|
||||
if len(chunks) != text_version.chunk_count:
|
||||
raise HTTPException(409, "文本切分快照不完整,请重新构建")
|
||||
expected = [
|
||||
(
|
||||
item["knowledge_id"],
|
||||
ordinal,
|
||||
start,
|
||||
end,
|
||||
body,
|
||||
hashlib.sha256(body.encode()).hexdigest(),
|
||||
{k: v for k, v in item.items() if k != "content"},
|
||||
)
|
||||
for item in corpus
|
||||
for ordinal, (start, end, body) in enumerate(
|
||||
split_text(
|
||||
item["content"], text_version.config["chunk_size"], text_version.config["overlap"]
|
||||
)
|
||||
)
|
||||
]
|
||||
actual = [
|
||||
(
|
||||
c.knowledge_id,
|
||||
c.ordinal,
|
||||
c.start_offset,
|
||||
c.end_offset,
|
||||
c.text,
|
||||
c.text_sha256,
|
||||
c.provenance,
|
||||
)
|
||||
for c in chunks
|
||||
]
|
||||
if actual != expected:
|
||||
raise HTTPException(409, "文本切分与审核语料不一致,请重新构建")
|
||||
return [(c.id, c.text) for c in chunks]
|
||||
|
||||
|
||||
def build_vectors(db: Session, text_version_id: UUID, actor_id: int, provider: EmbeddingProvider):
|
||||
state = locked_state(db)
|
||||
text_version, corpus = current_text(db, state)
|
||||
if str(text_version_id) != text_version.id:
|
||||
raise HTTPException(409, "只能对当前已发布的文本版本构建向量")
|
||||
inputs = checked_chunks(db, text_version, corpus)
|
||||
token = (state.generation, state.catalog_revision, text_version.id)
|
||||
spec = provider.spec
|
||||
db.commit() # No publication/catalog row locks or open read transaction during model work.
|
||||
vectors = encode(provider, [body for _, body in inputs]) if inputs else []
|
||||
if provider.spec != spec:
|
||||
raise HTTPException(409, "模型身份已变化,请重试")
|
||||
state = locked_state(db)
|
||||
new_text, new_corpus = current_text(db, state)
|
||||
if token != (state.generation, state.catalog_revision, state.active_version_id):
|
||||
raise HTTPException(409, "编码期间文本或审核版本已变化,请重新构建")
|
||||
if checked_chunks(db, new_text, new_corpus) != inputs:
|
||||
raise HTTPException(409, "编码期间文本切分快照已变化,请重新构建")
|
||||
space = db.get(EmbeddingSpace, spec.space_id)
|
||||
if space is None:
|
||||
db.add(EmbeddingSpace(id=spec.space_id, dimension=spec.dimension, spec=asdict(spec)))
|
||||
db.flush()
|
||||
elif space.spec != asdict(spec) or space.dimension != spec.dimension:
|
||||
raise HTTPException(409, "已保存向量空间配置不一致")
|
||||
version = VectorVersion(
|
||||
id=str(uuid4()),
|
||||
text_index_version_id=str(text_version_id),
|
||||
space_id=spec.space_id,
|
||||
dimension=spec.dimension,
|
||||
chunk_count=len(inputs),
|
||||
created_by=actor_id,
|
||||
status="ready",
|
||||
)
|
||||
db.add(version)
|
||||
db.flush()
|
||||
db.add_all(
|
||||
[
|
||||
ChunkVector(
|
||||
version_id=version.id,
|
||||
chunk_id=chunk_id,
|
||||
text_index_version_id=version.text_index_version_id,
|
||||
space_id=spec.space_id,
|
||||
dimension=spec.dimension,
|
||||
embedding=embedding,
|
||||
)
|
||||
for (chunk_id, _), embedding in zip(inputs, vectors, strict=True)
|
||||
]
|
||||
)
|
||||
result = VectorRead.model_validate(version)
|
||||
db.commit()
|
||||
return result
|
||||
|
||||
|
||||
def complete(db, version):
|
||||
count = db.scalar(
|
||||
select(func.count()).select_from(ChunkVector).where(ChunkVector.version_id == version.id)
|
||||
)
|
||||
if count != version.chunk_count:
|
||||
raise HTTPException(409, "向量快照不完整,请重新构建")
|
||||
|
||||
|
||||
def publish_vectors(db: Session, version_id: UUID, expected_generation: int, actor_id: int):
|
||||
state = locked_state(db)
|
||||
vector = vector_state(db)
|
||||
if vector.generation != expected_generation:
|
||||
raise HTTPException(409, "向量发布代数已变化,请刷新后重试")
|
||||
version = vector_or_404(db, str(version_id))
|
||||
text_version, corpus = current_text(db, state)
|
||||
if version.text_index_version_id != text_version.id:
|
||||
raise HTTPException(409, "该向量版本不属于当前文本发布版本")
|
||||
checked_chunks(db, text_version, corpus)
|
||||
complete(db, version)
|
||||
if version.chunk_count != text_version.chunk_count:
|
||||
raise HTTPException(409, "向量与文本切分数量不一致")
|
||||
if vector.active_version_id == version.id:
|
||||
return active_vectors(db)
|
||||
previous = vector.active_version_id
|
||||
if previous:
|
||||
db.get(VectorVersion, previous).status = "superseded"
|
||||
vector.active_version_id = version.id
|
||||
vector.generation += 1
|
||||
version.status, version.published_at = "published", utc_now()
|
||||
db.add(
|
||||
VectorPublication(
|
||||
generation=vector.generation,
|
||||
previous_version_id=previous,
|
||||
version_id=version.id,
|
||||
actor_id=actor_id,
|
||||
published_at=version.published_at,
|
||||
)
|
||||
)
|
||||
result = ActiveVectors(
|
||||
generation=vector.generation,
|
||||
text_generation=state.generation,
|
||||
catalog_revision=state.catalog_revision,
|
||||
is_current=True,
|
||||
active_version=VectorRead.model_validate(version),
|
||||
)
|
||||
db.commit()
|
||||
return result
|
||||
|
||||
|
||||
def retrieval_scope(db, provider):
|
||||
state = locked_state(db)
|
||||
vector = vector_state(db)
|
||||
text_version, corpus = current_text(db, state)
|
||||
version = db.get(VectorVersion, vector.active_version_id) if vector.active_version_id else None
|
||||
if version is None or version.text_index_version_id != text_version.id:
|
||||
raise HTTPException(409, "尚无匹配当前文本版本的向量发布,请先构建并发布")
|
||||
checked_chunks(db, text_version, corpus)
|
||||
space = db.get(EmbeddingSpace, version.space_id)
|
||||
if (
|
||||
space.id != provider.spec.space_id
|
||||
or space.spec != asdict(provider.spec)
|
||||
or space.dimension != provider.spec.dimension
|
||||
or version.dimension != provider.spec.dimension
|
||||
):
|
||||
raise HTTPException(
|
||||
409, "当前编码器与已发布向量空间不一致,请使用同一配置重建或切换运行配置"
|
||||
)
|
||||
complete(db, version)
|
||||
if version.chunk_count != text_version.chunk_count:
|
||||
raise HTTPException(409, "向量与文本切分数量不一致")
|
||||
token = (state.generation, state.catalog_revision, vector.generation, version.id)
|
||||
return state, vector, text_version, version, space, token
|
||||
|
||||
|
||||
def retrieve_vectors(db: Session, payload: VectorRequest, provider: EmbeddingProvider):
|
||||
started = perf_counter()
|
||||
*_, token = retrieval_scope(db, provider)
|
||||
db.commit()
|
||||
query_vector = encode(provider, [payload.query], query=True)[0]
|
||||
state, vector, text_version, version, space, current = retrieval_scope(db, provider)
|
||||
if token != current:
|
||||
raise HTTPException(409, "查询编码期间发布版本或审核依据已变化,请重试")
|
||||
total, rows = rank_vectors(db, version, space, payload.category, payload.top_k, query_vector)
|
||||
hits = [
|
||||
VectorHit(
|
||||
**chunk.provenance,
|
||||
rank=rank,
|
||||
chunk_id=chunk.id,
|
||||
chunk_ordinal=chunk.ordinal,
|
||||
snippet=chunk.text,
|
||||
snippet_start=chunk.start_offset,
|
||||
snippet_end=chunk.end_offset,
|
||||
score=round(max(-1.0, min(1.0, 1.0 - float(dist))), 8),
|
||||
matched_terms=[],
|
||||
)
|
||||
for rank, (chunk, dist) in enumerate(rows, 1)
|
||||
]
|
||||
result = VectorRetrievalResponse(
|
||||
run_id=uuid4(),
|
||||
query=payload.query,
|
||||
status="ok" if hits else "no_evidence",
|
||||
top_k=payload.top_k,
|
||||
category=payload.category,
|
||||
vector_version_id=version.id,
|
||||
text_index_version_id=text_version.id,
|
||||
generation=vector.generation,
|
||||
text_generation=state.generation,
|
||||
corpus_fingerprint=text_version.corpus_fingerprint,
|
||||
corpus_size=total,
|
||||
total_hits=total,
|
||||
results=hits,
|
||||
elapsed_ms=round((perf_counter() - started) * 1000, 3),
|
||||
created_at=utc_now(),
|
||||
config=VectorRetrievalConfig(space=SpaceRead.model_validate(space)),
|
||||
)
|
||||
db.add(
|
||||
RetrievalRun(
|
||||
id=str(result.run_id),
|
||||
query=result.query,
|
||||
mode="vector",
|
||||
corpus_fingerprint=result.corpus_fingerprint,
|
||||
response=result.model_dump(mode="json"),
|
||||
created_at=result.created_at,
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
return result
|
||||
|
||||
|
||||
def rank_vectors(db, version, space, category, limit, query_vector):
|
||||
# Extension-qualified operator works even with an isolated business schema search_path.
|
||||
distance = ChunkVector.embedding.op("OPERATOR(public.<=>)", return_type=Float)(query_vector)
|
||||
statement = (
|
||||
select(Chunk, distance.label("distance"))
|
||||
.join(ChunkVector, ChunkVector.chunk_id == Chunk.id)
|
||||
.where(
|
||||
ChunkVector.version_id == version.id,
|
||||
ChunkVector.space_id == space.id,
|
||||
ChunkVector.text_index_version_id == version.text_index_version_id,
|
||||
ChunkVector.dimension == space.dimension,
|
||||
)
|
||||
)
|
||||
if category is not None:
|
||||
statement = statement.where(Chunk.provenance["category"].as_string() == category)
|
||||
total = db.scalar(select(func.count()).select_from(statement.subquery()))
|
||||
rows = db.execute(
|
||||
statement.order_by(distance, Chunk.knowledge_id, Chunk.ordinal, Chunk.id).limit(limit)
|
||||
)
|
||||
return total, list(rows)
|
||||
@@ -0,0 +1 @@
|
||||
"""Offline, synthetic local-model evaluation; never writes the application database."""
|
||||
@@ -0,0 +1,221 @@
|
||||
"""Reproducible CPU smoke evaluation; no application database or remote inference service."""
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import importlib.metadata
|
||||
import json
|
||||
import math
|
||||
import platform
|
||||
import statistics
|
||||
import sys
|
||||
from dataclasses import asdict
|
||||
from pathlib import Path
|
||||
from time import perf_counter
|
||||
|
||||
from app.services.embedding import LOCAL_MODELS, EmbeddingError, LocalEmbeddingProvider
|
||||
|
||||
|
||||
def load_dataset(path: Path) -> tuple[dict, str]:
|
||||
with path.open("rb") as source:
|
||||
raw = source.read(2_000_001)
|
||||
if len(raw) > 2_000_000:
|
||||
raise ValueError("评估输入不得超过 2MB")
|
||||
data = json.loads(raw)
|
||||
if not isinstance(data, dict):
|
||||
raise ValueError("评估输入须为 JSON 对象")
|
||||
documents, queries = data["documents"], data["queries"]
|
||||
if not isinstance(documents, list) or not isinstance(queries, list):
|
||||
raise ValueError("documents 与 queries 须为列表")
|
||||
if not 1 <= len(documents) <= 1000 or not 1 <= len(queries) <= 1000:
|
||||
raise ValueError("文档和问题各须为 1–1000 条")
|
||||
for rows in [documents, queries]:
|
||||
if any(not isinstance(row, dict) for row in rows):
|
||||
raise ValueError("每条记录须为对象")
|
||||
ids = [row["id"] for row in rows]
|
||||
if any(not isinstance(x, str) or not x or len(x) > 100 for x in ids) or len(
|
||||
set(ids)
|
||||
) != len(ids):
|
||||
raise ValueError("每组 ID 必须非空、唯一且最多 100 字符")
|
||||
if any(
|
||||
not isinstance(row["text"], str) or not row["text"].strip() or len(row["text"]) > 10000
|
||||
for row in rows
|
||||
):
|
||||
raise ValueError("每条文本须为非空且最多 10000 字符")
|
||||
ids = {row["id"] for row in documents}
|
||||
for query in queries:
|
||||
relevant = query["relevant_ids"]
|
||||
if (
|
||||
not isinstance(relevant, list)
|
||||
or not relevant
|
||||
or any(not isinstance(x, str) for x in relevant)
|
||||
or len(set(relevant)) != len(relevant)
|
||||
or not set(relevant) <= ids
|
||||
):
|
||||
raise ValueError("相关文档标注必须非空、唯一并引用现有文档")
|
||||
if not isinstance(data["dataset_id"], str) or not isinstance(data["purpose"], str):
|
||||
raise ValueError("必须填写数据集 ID 与用途")
|
||||
return data, hashlib.sha256(raw).hexdigest()
|
||||
|
||||
|
||||
def rank_and_measure(documents, queries, document_vectors, query_vectors, *, top_k=3):
|
||||
if not documents or not queries or top_k < 1:
|
||||
raise ValueError("评估须有文档、问题且 top_k 为正数")
|
||||
if len(document_vectors) != len(documents) or len(query_vectors) != len(queries):
|
||||
raise ValueError("评估向量数量与输入不一致")
|
||||
rows = []
|
||||
for query, vector in zip(queries, query_vectors, strict=True):
|
||||
ranked = sorted(
|
||||
[
|
||||
(doc["id"], math.fsum(x * y for x, y in zip(vector, embedding, strict=True)))
|
||||
for doc, embedding in zip(documents, document_vectors, strict=True)
|
||||
],
|
||||
key=lambda pair: (-pair[1], pair[0]),
|
||||
)
|
||||
relevant = set(query["relevant_ids"])
|
||||
first = next(
|
||||
(rank for rank, (id_, _) in enumerate(ranked[:top_k], 1) if id_ in relevant), None
|
||||
)
|
||||
rows.append(
|
||||
{
|
||||
"query_id": query["id"],
|
||||
"top_results": [
|
||||
{"id": id_, "score": round(score, 8)} for id_, score in ranked[:top_k]
|
||||
],
|
||||
"hit_at_1": int(ranked[0][0] in relevant),
|
||||
"recall_at_k": len({id_ for id_, _ in ranked[:top_k]} & relevant) / len(relevant),
|
||||
"reciprocal_rank_at_k": 1 / first if first else 0,
|
||||
}
|
||||
)
|
||||
return {
|
||||
"top_k": top_k,
|
||||
"hit_at_1": statistics.mean(row["hit_at_1"] for row in rows),
|
||||
"recall_at_k": statistics.mean(row["recall_at_k"] for row in rows),
|
||||
"mrr_at_k": statistics.mean(row["reciprocal_rank_at_k"] for row in rows),
|
||||
"per_query": rows,
|
||||
}
|
||||
|
||||
|
||||
def measure(call, repeats):
|
||||
call() # Warm this exact input shape before taking the recorded samples.
|
||||
timings = []
|
||||
result = None
|
||||
for _ in range(repeats):
|
||||
start = perf_counter()
|
||||
result = call()
|
||||
timings.append((perf_counter() - start) * 1000)
|
||||
return result, {
|
||||
"samples_ms": [round(value, 3) for value in timings],
|
||||
"median_ms": round(statistics.median(timings), 3),
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="本地 CPU Embedding 实测;合成结果不能用于论文结论"
|
||||
)
|
||||
parser.add_argument("--model", choices=LOCAL_MODELS, required=True)
|
||||
parser.add_argument(
|
||||
"--dataset", type=Path, default=Path(__file__).with_name("synthetic_zh.json")
|
||||
)
|
||||
parser.add_argument("--batch-size", type=int, default=8)
|
||||
parser.add_argument("--threads", type=int, default=2)
|
||||
parser.add_argument("--repeats", type=int, default=5)
|
||||
parser.add_argument("--cache-dir", type=Path)
|
||||
parser.add_argument(
|
||||
"--download", action="store_true", help="允许下载固定 revision 的公开模型文件"
|
||||
)
|
||||
parser.add_argument("--output", type=Path)
|
||||
args = parser.parse_args()
|
||||
if sys.platform != "linux":
|
||||
parser.error("当前 RSS 测量仅支持 Linux;Provider 可在其他支持的平台单独使用")
|
||||
import resource
|
||||
|
||||
if not 1 <= args.repeats <= 20:
|
||||
parser.error("repeats 必须为 1–20")
|
||||
dataset, digest = load_dataset(args.dataset)
|
||||
print("初始化固定版本的 CPU 模型。", file=sys.stderr, flush=True)
|
||||
started = perf_counter()
|
||||
provider = LocalEmbeddingProvider(
|
||||
LOCAL_MODELS[args.model],
|
||||
batch_size=args.batch_size,
|
||||
threads=args.threads,
|
||||
cache_dir=args.cache_dir,
|
||||
offline=not args.download,
|
||||
)
|
||||
initialization_seconds = perf_counter() - started
|
||||
print("模型就绪,执行预热和编码测量。", file=sys.stderr, flush=True)
|
||||
provider.encode(["预热测试"], query=True)
|
||||
corpus = [row["text"] for row in dataset["documents"]]
|
||||
queries = [row["text"] for row in dataset["queries"]]
|
||||
documents, corpus_time = measure(lambda: provider.encode(corpus), args.repeats)
|
||||
vectors, query_time = measure(lambda: provider.encode(queries, query=True), args.repeats)
|
||||
_, single_time = measure(lambda: provider.encode([queries[0]], query=True), args.repeats)
|
||||
long_text = "合成测试资料仅用于测量本地编码资源,不表示任何真实景区信息。" * 12
|
||||
_, long_time = measure(lambda: provider.encode([long_text] * args.batch_size), args.repeats)
|
||||
try:
|
||||
provider.encode(["超长测试" * 500])
|
||||
except EmbeddingError:
|
||||
length_guard = True
|
||||
else:
|
||||
raise AssertionError("超长输入未被拒绝")
|
||||
import torch
|
||||
|
||||
report = {
|
||||
"schema_version": 1,
|
||||
"evaluation_kind": "synthetic_smoke"
|
||||
if dataset["purpose"] == "synthetic_smoke_not_domain_benchmark"
|
||||
else "provided_dataset_not_independently_verified",
|
||||
"dataset": {
|
||||
"id": dataset["dataset_id"],
|
||||
"purpose": dataset["purpose"],
|
||||
"sha256": digest,
|
||||
"documents": len(corpus),
|
||||
"queries": len(queries),
|
||||
},
|
||||
"embedding_spec": asdict(provider.spec),
|
||||
"space_id": provider.spec.space_id,
|
||||
"runtime": {
|
||||
"python": platform.python_version(),
|
||||
"torch": torch.__version__,
|
||||
"transformers": importlib.metadata.version("transformers"),
|
||||
"huggingface_hub": importlib.metadata.version("huggingface_hub"),
|
||||
"numpy": importlib.metadata.version("numpy"),
|
||||
"tokenizers": importlib.metadata.version("tokenizers"),
|
||||
"safetensors": importlib.metadata.version("safetensors"),
|
||||
"device": "cpu",
|
||||
"threads": torch.get_num_threads(),
|
||||
"interop_threads": torch.get_num_interop_threads(),
|
||||
"batch_size": args.batch_size,
|
||||
"repeats": args.repeats,
|
||||
"warmup_per_shape": 1,
|
||||
},
|
||||
"initialization_seconds": round(initialization_seconds, 3),
|
||||
"timings": {
|
||||
"corpus_encode": corpus_time,
|
||||
"query_batch_encode": query_time,
|
||||
"single_query_encode": single_time,
|
||||
"long_text_batch_encode": long_time,
|
||||
},
|
||||
"long_text_characters": len(long_text),
|
||||
"long_text_tokens": len(provider.tokenizer(long_text)["input_ids"]),
|
||||
"peak_process_rss_mib": round(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024, 1),
|
||||
"token_overflow_rejected": length_guard,
|
||||
"retrieval": rank_and_measure(dataset["documents"], dataset["queries"], documents, vectors),
|
||||
"limitations": [
|
||||
"合成小样本仅检查流程,不能代表那拉提领域质量或正式实验分数",
|
||||
"计时为本机预热后的编码,包含分词/补齐/归一化/结果转换,不含 API、数据库与网络请求",
|
||||
"峰值 RSS 是整个独立进程的高水位,包含解释器和模型;初始化包含缓存检查/下载与首次模型加载",
|
||||
],
|
||||
}
|
||||
encoded = json.dumps(report, ensure_ascii=False, indent=2) + "\n"
|
||||
if args.output:
|
||||
args.output.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.output.write_text(encoded, encoding="utf-8")
|
||||
args.output.chmod(0o600)
|
||||
print("评估完成,报告已写入指定本机位置。", file=sys.stderr)
|
||||
else:
|
||||
print(encoded, end="")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"dataset_id": "synthetic-zh-travel-smoke-v1",
|
||||
"purpose": "synthetic_smoke_not_domain_benchmark",
|
||||
"documents": [
|
||||
{"id":"d01","text":"【合成测试】示例园区的停车场设有接驳车,步行游客可换乘到游客中心。"},
|
||||
{"id":"d02","text":"【合成测试】示例停车区域按车辆停留时段管理,离场时到出口办理停车手续。"},
|
||||
{"id":"d03","text":"【合成测试】示例展馆提供轮椅通道和电梯,行动不便的访客可从无障碍入口进入。"},
|
||||
{"id":"d04","text":"【合成测试】示例演出因降雨取消,已购票观众可沿原购票渠道申请退款。"},
|
||||
{"id":"d05","text":"【合成测试】示例红色文化展厅设有预约讲解,访客可提前登记并选择导览时段。"},
|
||||
{"id":"d06","text":"【合成测试】示例资料管理员核查发布机构、来源链接和更新时间,只有审核通过的资料才能成为问答依据。"},
|
||||
{"id":"d07","text":"【合成测试】示例民宿在前台办理入住,旅客需要出示有效身份证件并确认预订信息。"},
|
||||
{"id":"d08","text":"【合成测试】示例住宿提供早餐,有素食需求的客人可以提前向接待人员说明。"},
|
||||
{"id":"d09","text":"【合成测试】示例山间步道遇到恶劣天气暂停通行,游客应查看关闭提示并选择室内活动。"},
|
||||
{"id":"d10","text":"【合成测试】示例游客服务台统一接收捡到的物品,并登记失物线索协助寻找主人。"},
|
||||
{"id":"d11","text":"【合成测试】示例停车区设有电动车充电点,司机可按现场设备说明补充电量。"},
|
||||
{"id":"d12","text":"【合成测试】示例场馆的开放安排以入口公告为准,参观者应提前确认当天是否接待。"}
|
||||
],
|
||||
"queries": [
|
||||
{"id":"q01","text":"没有开车的人怎样从停车区去游客中心?","relevant_ids":["d01"]},
|
||||
{"id":"q02","text":"同行者坐轮椅,可以从哪里进入展馆?","relevant_ids":["d03"]},
|
||||
{"id":"q03","text":"演出被雨取消,已经付的钱能退吗?","relevant_ids":["d04"]},
|
||||
{"id":"q04","text":"想听展厅里的故事讲解,如何安排导览?","relevant_ids":["d05"]},
|
||||
{"id":"q05","text":"怎么确保问答使用的旅游介绍有可靠出处?","relevant_ids":["d06"]},
|
||||
{"id":"q06","text":"到了住宿地点办理入住需要准备什么证件?","relevant_ids":["d07"]},
|
||||
{"id":"q07","text":"早饭不吃肉,应该提前找谁说明?","relevant_ids":["d08"]},
|
||||
{"id":"q08","text":"天气恶劣时还能在山路上游玩吗?","relevant_ids":["d09"]},
|
||||
{"id":"q09","text":"在路上捡到游客遗落的东西应交到哪里?","relevant_ids":["d10"]},
|
||||
{"id":"q10","text":"电动汽车快没电了,到哪儿补充电量?","relevant_ids":["d11"]}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
from alembic import context
|
||||
from sqlalchemy import create_engine, pool
|
||||
|
||||
from app import models # noqa: F401
|
||||
from app.core.config import settings
|
||||
from app.db.session import Base
|
||||
|
||||
config = context.config
|
||||
|
||||
|
||||
def run_migrations(connection):
|
||||
context.configure(connection=connection, target_metadata=Base.metadata, compare_type=True)
|
||||
with context.begin_transaction():
|
||||
context.run_migrations()
|
||||
|
||||
|
||||
if context.is_offline_mode():
|
||||
raise RuntimeError("迁移需要连接数据库,以检查并保留已有表;请使用 alembic upgrade head")
|
||||
elif config.attributes.get("connection") is not None:
|
||||
run_migrations(config.attributes["connection"])
|
||||
else:
|
||||
engine = create_engine(
|
||||
settings.database_url, poolclass=pool.NullPool, connect_args={"connect_timeout": 5}
|
||||
)
|
||||
with engine.connect() as connection:
|
||||
run_migrations(connection)
|
||||
@@ -0,0 +1,17 @@
|
||||
"""${message}"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
${imports if imports else ""}
|
||||
|
||||
revision = ${repr(up_revision)}
|
||||
down_revision = ${repr(down_revision)}
|
||||
branch_labels = ${repr(branch_labels)}
|
||||
depends_on = ${repr(depends_on)}
|
||||
|
||||
|
||||
def upgrade():
|
||||
${upgrades if upgrades else "pass"}
|
||||
|
||||
|
||||
def downgrade():
|
||||
${downgrades if downgrades else "pass"}
|
||||
@@ -0,0 +1,48 @@
|
||||
"""Register the original knowledge table without replacing existing data."""
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision = "0001_baseline"
|
||||
down_revision = None
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
op.execute("CREATE EXTENSION IF NOT EXISTS vector")
|
||||
inspector = sa.inspect(bind)
|
||||
if inspector.has_table("knowledge_items"):
|
||||
columns = {column["name"] for column in inspector.get_columns("knowledge_items")}
|
||||
expected = {
|
||||
"id",
|
||||
"title",
|
||||
"category",
|
||||
"content",
|
||||
"source_url",
|
||||
"verification_status",
|
||||
"created_at",
|
||||
}
|
||||
if columns != expected:
|
||||
raise RuntimeError("已有 knowledge_items 结构与基础版本不同,请先核对结构并备份")
|
||||
return
|
||||
op.create_table(
|
||||
"knowledge_items",
|
||||
sa.Column("id", sa.Integer(), primary_key=True),
|
||||
sa.Column("title", sa.String(250), nullable=False),
|
||||
sa.Column("category", sa.String(80), nullable=False),
|
||||
sa.Column("content", sa.Text(), nullable=False),
|
||||
sa.Column("source_url", sa.String(1000)),
|
||||
sa.Column("verification_status", sa.String(30), nullable=False),
|
||||
sa.Column(
|
||||
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
|
||||
),
|
||||
)
|
||||
op.create_index("ix_knowledge_items_id", "knowledge_items", ["id"])
|
||||
op.create_index("ix_knowledge_items_title", "knowledge_items", ["title"])
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.drop_table("knowledge_items")
|
||||
# 扩展可能由其他项目共享,不在回滚时删除。
|
||||
@@ -0,0 +1,110 @@
|
||||
"""Add source, document, review and audit management."""
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision = "0002_catalog"
|
||||
down_revision = "0001_baseline"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def version_columns():
|
||||
return [
|
||||
sa.Column("version", sa.Integer(), server_default="1", nullable=False),
|
||||
sa.Column(
|
||||
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
|
||||
),
|
||||
sa.Column(
|
||||
"updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
|
||||
),
|
||||
sa.Column("is_deleted", sa.Boolean(), server_default=sa.false(), nullable=False),
|
||||
]
|
||||
|
||||
|
||||
def upgrade():
|
||||
op.create_table(
|
||||
"sources",
|
||||
sa.Column("id", sa.Integer(), primary_key=True),
|
||||
sa.Column("name", sa.String(250), nullable=False),
|
||||
sa.Column("source_url", sa.String(1000)),
|
||||
sa.Column("notes", sa.Text(), nullable=False),
|
||||
*version_columns(),
|
||||
)
|
||||
op.create_table(
|
||||
"documents",
|
||||
sa.Column("id", sa.Integer(), primary_key=True),
|
||||
sa.Column("source_id", sa.Integer(), sa.ForeignKey("sources.id"), nullable=False),
|
||||
sa.Column("title", sa.String(250), nullable=False),
|
||||
sa.Column("category", sa.String(80), nullable=False),
|
||||
sa.Column("content", sa.Text(), nullable=False),
|
||||
sa.Column("collected_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("notes", sa.Text(), nullable=False),
|
||||
*version_columns(),
|
||||
)
|
||||
op.create_index("ix_documents_source_id", "documents", ["source_id"])
|
||||
op.create_table(
|
||||
"audit_events",
|
||||
sa.Column("id", sa.Integer(), primary_key=True),
|
||||
sa.Column("entity_type", sa.String(30), nullable=False),
|
||||
sa.Column("entity_id", sa.Integer(), nullable=False),
|
||||
sa.Column("action", sa.String(30), nullable=False),
|
||||
sa.Column("actor", sa.String(100)),
|
||||
sa.Column("notes", sa.Text(), nullable=False),
|
||||
sa.Column("snapshot", sa.JSON(), nullable=False),
|
||||
sa.Column(
|
||||
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
|
||||
),
|
||||
)
|
||||
op.create_index("ix_audit_events_entity_type", "audit_events", ["entity_type"])
|
||||
op.create_index("ix_audit_events_entity_id", "audit_events", ["entity_id"])
|
||||
for column in version_columns():
|
||||
if column.name != "created_at":
|
||||
op.add_column("knowledge_items", column)
|
||||
for name, kind in [
|
||||
("source_id", sa.Integer()),
|
||||
("document_id", sa.Integer()),
|
||||
("collected_at", sa.DateTime(timezone=True)),
|
||||
("reviewer", sa.String(100)),
|
||||
("reviewed_at", sa.DateTime(timezone=True)),
|
||||
("verification_notes", sa.Text()),
|
||||
]:
|
||||
op.add_column("knowledge_items", sa.Column(name, kind))
|
||||
for name, parent in [("source_id", "sources"), ("document_id", "documents")]:
|
||||
op.create_foreign_key(f"fk_knowledge_{name}", "knowledge_items", parent, [name], ["id"])
|
||||
op.create_index(f"ix_knowledge_items_{name}", "knowledge_items", [name])
|
||||
op.create_check_constraint(
|
||||
"ck_knowledge_verification_status",
|
||||
"knowledge_items",
|
||||
"verification_status IN ('pending', 'verified', 'rejected')",
|
||||
)
|
||||
op.execute("""INSERT INTO audit_events (entity_type, entity_id, action, notes, snapshot)
|
||||
SELECT 'knowledge_items', id, 'import', '保留基础版本数据及原状态',
|
||||
json_build_object('item', row_to_json(k)) FROM knowledge_items k""")
|
||||
op.execute("""UPDATE knowledge_items SET verification_status = 'pending', version = version + 1
|
||||
WHERE verification_status = 'verified'""")
|
||||
op.execute("""INSERT INTO audit_events (entity_type, entity_id, action, notes, snapshot)
|
||||
SELECT 'knowledge_items', id, 'invalidate', '旧版本没有完整审核记录,需要重新核验',
|
||||
json_build_object('item', row_to_json(k)) FROM knowledge_items k WHERE version = 2""")
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.drop_constraint("ck_knowledge_verification_status", "knowledge_items", type_="check")
|
||||
for name in ["source_id", "document_id"]:
|
||||
op.drop_index(f"ix_knowledge_items_{name}", table_name="knowledge_items")
|
||||
op.drop_constraint(f"fk_knowledge_{name}", "knowledge_items", type_="foreignkey")
|
||||
for name in [
|
||||
"source_id",
|
||||
"document_id",
|
||||
"collected_at",
|
||||
"reviewer",
|
||||
"reviewed_at",
|
||||
"verification_notes",
|
||||
"version",
|
||||
"updated_at",
|
||||
"is_deleted",
|
||||
]:
|
||||
op.drop_column("knowledge_items", name)
|
||||
op.drop_table("audit_events")
|
||||
op.drop_table("documents")
|
||||
op.drop_table("sources")
|
||||
@@ -0,0 +1,28 @@
|
||||
"""Persist keyword retrieval evidence and reproducible configuration."""
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision = "0003_retrieval_runs"
|
||||
down_revision = "0002_catalog"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
op.create_table(
|
||||
"retrieval_runs",
|
||||
sa.Column("id", sa.String(36), primary_key=True),
|
||||
sa.Column("query", sa.Text(), nullable=False),
|
||||
sa.Column("mode", sa.String(20), nullable=False),
|
||||
sa.Column("corpus_fingerprint", sa.String(64), nullable=False),
|
||||
sa.Column("response", sa.JSON(), nullable=False),
|
||||
sa.Column(
|
||||
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
|
||||
),
|
||||
)
|
||||
op.create_index("ix_retrieval_runs_created_at", "retrieval_runs", ["created_at"])
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.drop_table("retrieval_runs")
|
||||
@@ -0,0 +1,52 @@
|
||||
"""Add users, revocable opaque sessions and shared authentication budgets."""
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision = "0004_auth_sessions"
|
||||
down_revision = "0003_retrieval_runs"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
op.create_table(
|
||||
"users",
|
||||
sa.Column("id", sa.Integer(), primary_key=True),
|
||||
sa.Column("username", sa.String(32), nullable=False, unique=True),
|
||||
sa.Column("password_hash", sa.String(255), nullable=False),
|
||||
sa.Column("nickname", sa.String(80)),
|
||||
sa.Column("email", sa.String(254)),
|
||||
sa.Column("role", sa.String(10), nullable=False),
|
||||
sa.Column("credential_version", sa.Integer(), nullable=False),
|
||||
sa.Column(
|
||||
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
|
||||
),
|
||||
sa.CheckConstraint("role IN ('user', 'admin')", name="ck_users_role"),
|
||||
)
|
||||
op.create_table(
|
||||
"auth_sessions",
|
||||
sa.Column("token_hash", sa.String(64), primary_key=True),
|
||||
sa.Column(
|
||||
"user_id", sa.Integer(), sa.ForeignKey("users.id", ondelete="CASCADE"), nullable=False
|
||||
),
|
||||
sa.Column("credential_version", sa.Integer(), nullable=False),
|
||||
sa.Column(
|
||||
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
|
||||
),
|
||||
sa.Column("expires_at", sa.DateTime(timezone=True), nullable=False),
|
||||
)
|
||||
op.create_index("ix_auth_sessions_user_id", "auth_sessions", ["user_id"])
|
||||
op.create_index("ix_auth_sessions_expires_at", "auth_sessions", ["expires_at"])
|
||||
op.create_table(
|
||||
"auth_rate_limits",
|
||||
sa.Column("key", sa.String(64), primary_key=True),
|
||||
sa.Column("window_start", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("attempts", sa.Integer(), nullable=False),
|
||||
)
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.drop_table("auth_rate_limits")
|
||||
op.drop_table("auth_sessions")
|
||||
op.drop_table("users")
|
||||
@@ -0,0 +1,81 @@
|
||||
"""Persist reviewed chunks, version snapshots and atomic publication history."""
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision = "0005_chunk_indexes"
|
||||
down_revision = "0004_auth_sessions"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
op.create_table(
|
||||
"index_versions",
|
||||
sa.Column("id", sa.String(36), primary_key=True),
|
||||
sa.Column("status", sa.String(20), nullable=False),
|
||||
sa.Column("config", sa.JSON(), nullable=False),
|
||||
sa.Column("corpus_fingerprint", sa.String(64), nullable=False),
|
||||
sa.Column("catalog_revision", sa.Integer(), nullable=False),
|
||||
sa.Column("item_count", sa.Integer(), nullable=False),
|
||||
sa.Column("chunk_count", sa.Integer(), nullable=False),
|
||||
sa.Column("created_by", sa.Integer(), sa.ForeignKey("users.id"), nullable=False),
|
||||
sa.Column(
|
||||
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
|
||||
),
|
||||
sa.Column("published_at", sa.DateTime(timezone=True)),
|
||||
sa.CheckConstraint(
|
||||
"status IN ('ready', 'published', 'superseded')", name="ck_index_version_status"
|
||||
),
|
||||
)
|
||||
op.create_table(
|
||||
"index_state",
|
||||
sa.Column("id", sa.Integer(), primary_key=True),
|
||||
sa.Column("active_version_id", sa.String(36), sa.ForeignKey("index_versions.id")),
|
||||
sa.Column("generation", sa.Integer(), nullable=False),
|
||||
sa.Column("catalog_revision", sa.Integer(), nullable=False),
|
||||
sa.CheckConstraint("id = 1", name="ck_index_state_singleton"),
|
||||
)
|
||||
op.execute(
|
||||
sa.text("INSERT INTO index_state (id, generation, catalog_revision) VALUES (1, 0, 0)")
|
||||
)
|
||||
op.create_table(
|
||||
"chunks",
|
||||
sa.Column("id", sa.String(36), primary_key=True),
|
||||
sa.Column(
|
||||
"index_version_id", sa.String(36), sa.ForeignKey("index_versions.id"), nullable=False
|
||||
),
|
||||
sa.Column(
|
||||
"knowledge_id", sa.Integer(), sa.ForeignKey("knowledge_items.id"), nullable=False
|
||||
),
|
||||
sa.Column("ordinal", sa.Integer(), nullable=False),
|
||||
sa.Column("start_offset", sa.Integer(), nullable=False),
|
||||
sa.Column("end_offset", sa.Integer(), nullable=False),
|
||||
sa.Column("text", sa.Text(), nullable=False),
|
||||
sa.Column("text_sha256", sa.String(64), nullable=False),
|
||||
sa.Column("provenance", sa.JSON(), nullable=False),
|
||||
sa.UniqueConstraint(
|
||||
"index_version_id", "knowledge_id", "ordinal", name="uq_chunk_position"
|
||||
),
|
||||
sa.CheckConstraint(
|
||||
"start_offset >= 0 AND end_offset > start_offset", name="ck_chunk_offsets"
|
||||
),
|
||||
)
|
||||
op.create_index("ix_chunks_index_version_id", "chunks", ["index_version_id"])
|
||||
op.create_table(
|
||||
"index_publications",
|
||||
sa.Column("generation", sa.Integer(), primary_key=True),
|
||||
sa.Column("previous_version_id", sa.String(36), sa.ForeignKey("index_versions.id")),
|
||||
sa.Column("version_id", sa.String(36), sa.ForeignKey("index_versions.id"), nullable=False),
|
||||
sa.Column("actor_id", sa.Integer(), sa.ForeignKey("users.id"), nullable=False),
|
||||
sa.Column(
|
||||
"published_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.drop_table("index_publications")
|
||||
op.drop_table("chunks")
|
||||
op.drop_table("index_state")
|
||||
op.drop_table("index_versions")
|
||||
@@ -0,0 +1,122 @@
|
||||
"""Persist dimension-checked vector snapshots with composite scope constraints."""
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision = "0006_vectors"
|
||||
down_revision = "0005_chunk_indexes"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
class ExtensionVector(sa.types.UserDefinedType):
|
||||
"""Qualify extension type without adding public business tables to search_path."""
|
||||
|
||||
def get_col_spec(self, **kw):
|
||||
return "public.vector"
|
||||
|
||||
|
||||
def upgrade():
|
||||
schema = op.get_bind().scalar(
|
||||
sa.text("SELECT extnamespace::regnamespace::text FROM pg_extension WHERE extname='vector'")
|
||||
)
|
||||
if schema != "public":
|
||||
raise RuntimeError("vector 扩展须安装在 public;请由数据库管理员核对,迁移不会移动共享扩展")
|
||||
op.create_unique_constraint("uq_chunk_text_scope", "chunks", ["id", "index_version_id"])
|
||||
op.create_table(
|
||||
"embedding_spaces",
|
||||
sa.Column("id", sa.String(64), primary_key=True),
|
||||
sa.Column("dimension", sa.Integer(), nullable=False),
|
||||
sa.Column("spec", sa.JSON(), nullable=False),
|
||||
sa.UniqueConstraint("id", "dimension", name="uq_embedding_space_dimension"),
|
||||
sa.CheckConstraint("dimension BETWEEN 1 AND 2000", name="ck_embedding_space_dimension"),
|
||||
)
|
||||
op.create_table(
|
||||
"vector_versions",
|
||||
sa.Column("id", sa.String(36), primary_key=True),
|
||||
sa.Column(
|
||||
"text_index_version_id",
|
||||
sa.String(36),
|
||||
sa.ForeignKey("index_versions.id"),
|
||||
nullable=False,
|
||||
),
|
||||
sa.Column("space_id", sa.String(64), nullable=False),
|
||||
sa.Column("dimension", sa.Integer(), nullable=False),
|
||||
sa.Column("chunk_count", sa.Integer(), nullable=False),
|
||||
sa.Column("status", sa.String(20), nullable=False),
|
||||
sa.Column("created_by", sa.Integer(), sa.ForeignKey("users.id"), nullable=False),
|
||||
sa.Column(
|
||||
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
|
||||
),
|
||||
sa.Column("published_at", sa.DateTime(timezone=True)),
|
||||
sa.ForeignKeyConstraint(
|
||||
["space_id", "dimension"],
|
||||
["embedding_spaces.id", "embedding_spaces.dimension"],
|
||||
name="fk_vector_version_space",
|
||||
),
|
||||
sa.UniqueConstraint(
|
||||
"id", "text_index_version_id", "space_id", "dimension", name="uq_vector_version_scope"
|
||||
),
|
||||
sa.CheckConstraint(
|
||||
"status IN ('ready', 'published', 'superseded')", name="ck_vector_version_status"
|
||||
),
|
||||
sa.CheckConstraint("chunk_count BETWEEN 0 AND 2000", name="ck_vector_version_count"),
|
||||
)
|
||||
op.create_table(
|
||||
"chunk_vectors",
|
||||
sa.Column("version_id", sa.String(36), primary_key=True),
|
||||
sa.Column("chunk_id", sa.String(36), primary_key=True),
|
||||
sa.Column("text_index_version_id", sa.String(36), nullable=False),
|
||||
sa.Column("space_id", sa.String(64), nullable=False),
|
||||
sa.Column("dimension", sa.Integer(), nullable=False),
|
||||
sa.Column("embedding", ExtensionVector(), nullable=False),
|
||||
sa.ForeignKeyConstraint(
|
||||
["version_id", "text_index_version_id", "space_id", "dimension"],
|
||||
[
|
||||
"vector_versions.id",
|
||||
"vector_versions.text_index_version_id",
|
||||
"vector_versions.space_id",
|
||||
"vector_versions.dimension",
|
||||
],
|
||||
name="fk_chunk_vector_scope",
|
||||
),
|
||||
sa.ForeignKeyConstraint(
|
||||
["chunk_id", "text_index_version_id"],
|
||||
["chunks.id", "chunks.index_version_id"],
|
||||
name="fk_chunk_vector_text",
|
||||
),
|
||||
sa.CheckConstraint("public.vector_dims(embedding) = dimension", name="ck_vector_dimension"),
|
||||
sa.CheckConstraint(
|
||||
"public.vector_norm(embedding) BETWEEN 0.9999 AND 1.0001", name="ck_vector_unit_norm"
|
||||
),
|
||||
)
|
||||
op.create_table(
|
||||
"vector_state",
|
||||
sa.Column("id", sa.Integer(), primary_key=True),
|
||||
sa.Column("active_version_id", sa.String(36), sa.ForeignKey("vector_versions.id")),
|
||||
sa.Column("generation", sa.Integer(), nullable=False),
|
||||
sa.CheckConstraint("id = 1", name="ck_vector_state_singleton"),
|
||||
)
|
||||
op.execute(sa.text("INSERT INTO vector_state (id, generation) VALUES (1, 0)"))
|
||||
op.create_table(
|
||||
"vector_publications",
|
||||
sa.Column("generation", sa.Integer(), primary_key=True),
|
||||
sa.Column("previous_version_id", sa.String(36), sa.ForeignKey("vector_versions.id")),
|
||||
sa.Column("version_id", sa.String(36), sa.ForeignKey("vector_versions.id"), nullable=False),
|
||||
sa.Column("actor_id", sa.Integer(), sa.ForeignKey("users.id"), nullable=False),
|
||||
sa.Column(
|
||||
"published_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def downgrade():
|
||||
for table in [
|
||||
"vector_publications",
|
||||
"vector_state",
|
||||
"chunk_vectors",
|
||||
"vector_versions",
|
||||
"embedding_spaces",
|
||||
]:
|
||||
op.drop_table(table)
|
||||
op.drop_constraint("uq_chunk_text_scope", "chunks", type_="unique")
|
||||
@@ -0,0 +1,10 @@
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
pythonpath = ["."]
|
||||
|
||||
[tool.ruff]
|
||||
target-version = "py311"
|
||||
line-length = 100
|
||||
|
||||
[tool.ruff.lint]
|
||||
select = ["E4", "E7", "E9", "F", "I"]
|
||||
@@ -0,0 +1,6 @@
|
||||
-r requirements.txt
|
||||
pytest>=8,<10
|
||||
httpx>=0.27,<1
|
||||
# FastAPI/Starlette TestClient uses the httpx2 package in this runtime.
|
||||
httpx2>=2.13,<3
|
||||
ruff>=0.9,<1
|
||||
@@ -0,0 +1,8 @@
|
||||
# Optional Linux CPU evaluation environment; not required by FastAPI startup.
|
||||
# Install the CPU torch wheel from its official index first; see the evaluation guide.
|
||||
torch==2.14.1+cpu
|
||||
transformers==5.20.0
|
||||
huggingface-hub==2.2.0
|
||||
numpy==2.4.6
|
||||
tokenizers==0.23.3
|
||||
safetensors==0.8.0
|
||||
@@ -5,3 +5,7 @@ pydantic-settings>=2.4,<3.0
|
||||
SQLAlchemy>=2.0,<3.0
|
||||
psycopg[binary]>=3.2,<4.0
|
||||
pgvector>=0.3,<1.0
|
||||
alembic>=1.13,<2.0
|
||||
jieba==0.42.1
|
||||
argon2-cffi>=25.1,<26
|
||||
email-validator>=2.2,<3
|
||||
|
||||
@@ -0,0 +1,109 @@
|
||||
"""Real PostgreSQL tests, isolated in a fresh schema for every test."""
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
from uuid import uuid4
|
||||
|
||||
import pytest
|
||||
from alembic import command
|
||||
from alembic.config import Config
|
||||
from fastapi.testclient import TestClient
|
||||
from sqlalchemy import create_engine, text
|
||||
from sqlalchemy.engine import make_url
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from app.db.session import get_db
|
||||
from app.main import create_app
|
||||
from app.models.auth import User
|
||||
from app.services.auth import PASSWORD_HASHER
|
||||
|
||||
BACKEND = Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
def migrate(engine, revision="head", *, downgrade=False):
|
||||
config = Config(str(BACKEND / "alembic.ini"))
|
||||
with engine.begin() as connection:
|
||||
config.attributes["connection"] = connection
|
||||
(command.downgrade if downgrade else command.upgrade)(config, revision)
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def postgres_url(tmp_path_factory):
|
||||
url = os.environ.get("TEST_DATABASE_URL")
|
||||
server = None
|
||||
if not url:
|
||||
try:
|
||||
import pgserver
|
||||
except ImportError:
|
||||
pytest.fail("请设置 TEST_DATABASE_URL,或安装可选的本机测试依赖 pgserver")
|
||||
server = pgserver.get_server(tmp_path_factory.mktemp("postgres"), cleanup_mode="stop")
|
||||
url = server.get_uri().replace("postgresql://", "postgresql+psycopg://", 1)
|
||||
if not url.startswith("postgresql+psycopg://"):
|
||||
pytest.fail("TEST_DATABASE_URL 必须使用 postgresql+psycopg://")
|
||||
bootstrap = create_engine(url)
|
||||
with bootstrap.begin() as connection:
|
||||
connection.execute(text("CREATE EXTENSION IF NOT EXISTS vector WITH SCHEMA public"))
|
||||
bootstrap.dispose()
|
||||
yield url
|
||||
if server:
|
||||
server.cleanup()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def database(postgres_url):
|
||||
schema = "test_" + uuid4().hex
|
||||
admin = create_engine(postgres_url)
|
||||
with admin.begin() as connection:
|
||||
connection.execute(text(f'CREATE SCHEMA "{schema}"'))
|
||||
# 不包含 public,防止误操作已有业务表。
|
||||
url = make_url(postgres_url).update_query_dict({"options": f"-csearch_path={schema}"})
|
||||
engine = create_engine(url)
|
||||
yield engine
|
||||
engine.dispose()
|
||||
with admin.begin() as connection:
|
||||
connection.execute(text(f'DROP SCHEMA "{schema}" CASCADE'))
|
||||
admin.dispose()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def migrated_database(database):
|
||||
migrate(database)
|
||||
return database
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def anonymous_client(migrated_database):
|
||||
app = create_app(migrated_database)
|
||||
factory = sessionmaker(bind=migrated_database)
|
||||
|
||||
def test_session():
|
||||
with factory() as session:
|
||||
try:
|
||||
yield session
|
||||
except Exception:
|
||||
session.rollback()
|
||||
raise
|
||||
|
||||
app.dependency_overrides[get_db] = test_session
|
||||
with TestClient(app, base_url="https://testserver") as client:
|
||||
client.headers["Origin"] = "http://localhost:5173"
|
||||
yield client
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client(anonymous_client, migrated_database):
|
||||
with sessionmaker(bind=migrated_database)() as db:
|
||||
db.add(
|
||||
User(
|
||||
username="test_admin",
|
||||
password_hash=PASSWORD_HASHER.hash("test-password-123"),
|
||||
role="admin",
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
response = anonymous_client.post(
|
||||
"/api/auth/login", json={"username": "test_admin", "password": "test-password-123"}
|
||||
)
|
||||
assert response.status_code == 200, response.text
|
||||
anonymous_client.headers["X-CSRF-Token"] = response.json()["csrf_token"]
|
||||
return anonymous_client
|
||||
@@ -0,0 +1,342 @@
|
||||
"""Authentication, authorization and CSRF boundaries over real PostgreSQL."""
|
||||
|
||||
import json
|
||||
from datetime import timedelta
|
||||
|
||||
import pytest
|
||||
from pydantic import ValidationError
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.core.config import Settings
|
||||
from app.models.auth import AuthSession, User
|
||||
from app.models.catalog import utc_now
|
||||
from app.services.auth import COOKIE_NAME, check_password, token_hash
|
||||
|
||||
PASSWORD = "test-password-123"
|
||||
ORIGIN = "http://localhost:5173"
|
||||
|
||||
|
||||
def register(client, username="test_user", **extra):
|
||||
return client.post(
|
||||
"/api/auth/register", json={"username": username, "password": PASSWORD, **extra}
|
||||
)
|
||||
|
||||
|
||||
def login(client, username="test_user", password=PASSWORD):
|
||||
response = client.post("/api/auth/login", json={"username": username, "password": password})
|
||||
if response.status_code == 200:
|
||||
client.headers["X-CSRF-Token"] = response.json()["csrf_token"]
|
||||
return response
|
||||
|
||||
|
||||
def test_registration_hashes_password_and_cannot_select_privilege(
|
||||
anonymous_client, migrated_database
|
||||
):
|
||||
c = anonymous_client
|
||||
for extra in [{"role": "admin"}, {"id": 1}, {"password_hash": "injected"}]:
|
||||
assert register(c, **extra).status_code == 422
|
||||
result = register(c, username=" Test_User ", email="user@example.com", nickname="占位用户")
|
||||
assert result.status_code == 201
|
||||
assert result.json()["username"] == "test_user" and result.json()["role"] == "user"
|
||||
assert c.get("/api/auth/me").status_code == 401
|
||||
assert register(c).status_code == 409
|
||||
with Session(migrated_database) as db:
|
||||
user = db.scalar(select(User))
|
||||
assert user.password_hash.startswith("$argon2id$")
|
||||
assert PASSWORD not in user.password_hash and check_password(user.password_hash, PASSWORD)
|
||||
assert db.scalar(select(func.count()).select_from(AuthSession)) == 0
|
||||
assert "password" not in result.text and "credential_version" not in result.text
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"data",
|
||||
[
|
||||
{"password": "short"},
|
||||
{"password": "x" * 129},
|
||||
{"username": "a"},
|
||||
{"username": "a b"},
|
||||
{"email": "invalid"},
|
||||
{"nickname": "x" * 81},
|
||||
{"nickname": "bad\u0000name"},
|
||||
],
|
||||
)
|
||||
def test_validation_does_not_echo_credentials(anonymous_client, data):
|
||||
submitted = {"username": "test_user", "password": PASSWORD, **data}
|
||||
response = anonymous_client.post("/api/auth/register", json=submitted)
|
||||
assert response.status_code == 422
|
||||
assert '"input"' not in response.text
|
||||
assert json.dumps(submitted["password"]) not in response.text
|
||||
assert response.headers["cache-control"] == "no-store"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("origin", [None, "null", "https://evil.example", ORIGIN + ".evil.example"])
|
||||
def test_login_and_registration_reject_untrusted_origin(
|
||||
anonymous_client, migrated_database, origin
|
||||
):
|
||||
c = anonymous_client
|
||||
c.headers.pop("Origin", None)
|
||||
headers = {"Origin": origin} if origin else {}
|
||||
for path in ["register", "login"]:
|
||||
response = c.post(
|
||||
f"/api/auth/{path}",
|
||||
json={"username": "test_user", "password": PASSWORD},
|
||||
headers=headers,
|
||||
)
|
||||
assert response.status_code == 403
|
||||
with Session(migrated_database) as db:
|
||||
assert db.scalar(select(func.count()).select_from(User)) == 0
|
||||
|
||||
|
||||
def test_login_cookie_rotation_logout_and_opaque_storage(anonymous_client, migrated_database):
|
||||
c = anonymous_client
|
||||
register(c)
|
||||
response = login(c)
|
||||
assert response.status_code == 200
|
||||
assert all(
|
||||
value in response.headers["set-cookie"]
|
||||
for value in ["HttpOnly", "Secure", "SameSite=lax", "Path=/"]
|
||||
)
|
||||
first = c.cookies.get(COOKIE_NAME)
|
||||
csrf = response.json()["csrf_token"]
|
||||
assert c.get("/api/auth/me").json()["csrf_token"] == csrf
|
||||
with Session(migrated_database) as db:
|
||||
stored = db.scalar(select(AuthSession))
|
||||
assert stored.token_hash == token_hash(first) and first not in stored.token_hash
|
||||
assert login(c).status_code == 200
|
||||
second = c.cookies.get(COOKIE_NAME)
|
||||
assert first != second
|
||||
c.cookies.clear()
|
||||
c.cookies.set(COOKIE_NAME, first)
|
||||
assert c.get("/api/auth/me").status_code == 401
|
||||
c.cookies.clear()
|
||||
c.cookies.set(COOKIE_NAME, second)
|
||||
response = c.post("/api/auth/logout")
|
||||
assert response.status_code == 204 and "Max-Age=0" in response.headers["set-cookie"]
|
||||
c.cookies.clear()
|
||||
c.cookies.set(COOKIE_NAME, second)
|
||||
assert c.get("/api/auth/me").status_code == 401
|
||||
|
||||
|
||||
def test_login_failure_is_generic_and_throttled_in_database(anonymous_client):
|
||||
c = anonymous_client
|
||||
register(c)
|
||||
absent = login(c, "absent_user", "wrong").json()
|
||||
wrong = login(c, password="wrong")
|
||||
assert wrong.status_code == 401 and wrong.json() == absent
|
||||
for _ in range(9):
|
||||
assert login(c, password="wrong").status_code == 401
|
||||
blocked = login(c)
|
||||
assert blocked.status_code == 429 and blocked.headers["retry-after"] == "300"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("username", ["bad\u0000name", "invalid name", "../user"])
|
||||
def test_login_rejects_invalid_username_before_database_lookup(anonymous_client, username):
|
||||
assert login(anonymous_client, username=username).status_code == 422
|
||||
|
||||
|
||||
def test_expired_forged_and_credential_version_sessions_are_rejected(
|
||||
anonymous_client, migrated_database
|
||||
):
|
||||
c = anonymous_client
|
||||
register(c)
|
||||
login(c)
|
||||
with Session(migrated_database) as db:
|
||||
session = db.scalar(select(AuthSession))
|
||||
session.expires_at = utc_now() - timedelta(seconds=1)
|
||||
db.commit()
|
||||
assert c.get("/api/auth/me").status_code == 401
|
||||
login(c)
|
||||
with Session(migrated_database) as db:
|
||||
db.scalar(select(User)).credential_version += 1
|
||||
db.commit()
|
||||
assert c.get("/api/auth/me").status_code == 401
|
||||
c.cookies.clear()
|
||||
c.cookies.set(COOKIE_NAME, "x" * 43)
|
||||
assert c.get("/api/auth/me").status_code == 401
|
||||
c.cookies.clear()
|
||||
c.cookies.set(COOKIE_NAME, "invalid")
|
||||
assert c.get("/api/auth/me").status_code == 401
|
||||
|
||||
|
||||
def test_csrf_is_bound_to_session_and_required_for_all_authenticated_writes(anonymous_client):
|
||||
c = anonymous_client
|
||||
register(c)
|
||||
login(c)
|
||||
first = c.headers["X-CSRF-Token"]
|
||||
login(c)
|
||||
for token in ["", "invalid", first]:
|
||||
assert (
|
||||
c.patch(
|
||||
"/api/users/me", json={"nickname": "占位"}, headers={"X-CSRF-Token": token}
|
||||
).status_code
|
||||
== 403
|
||||
)
|
||||
assert c.post("/api/auth/logout", headers={"X-CSRF-Token": token}).status_code == 403
|
||||
assert (
|
||||
c.patch(
|
||||
"/api/users/me", json={"nickname": "占位"}, headers={"Origin": "https://evil.example"}
|
||||
).status_code
|
||||
== 403
|
||||
)
|
||||
c.headers.pop("Origin")
|
||||
assert c.patch("/api/users/me", json={"nickname": "占位"}).status_code == 403
|
||||
c.headers["Origin"] = ORIGIN
|
||||
assert c.patch("/api/users/me", json={"nickname": " 占位 "}).json()["nickname"] == "占位"
|
||||
|
||||
|
||||
def test_profile_updates_only_own_allowed_fields(anonymous_client, migrated_database):
|
||||
c = anonymous_client
|
||||
first = register(c).json()
|
||||
register(c, "other_user")
|
||||
login(c)
|
||||
for payload in [
|
||||
{"id": 2},
|
||||
{"role": "admin"},
|
||||
{"username": "other_user"},
|
||||
{"password": "changed"},
|
||||
{},
|
||||
]:
|
||||
assert c.patch("/api/users/me", json=payload).status_code == 422
|
||||
assert (
|
||||
c.patch("/api/users/me", json={"email": "changed@example.com", "nickname": "示例"}).json()[
|
||||
"id"
|
||||
]
|
||||
== first["id"]
|
||||
)
|
||||
assert c.patch("/api/users/me", json={"email": "", "nickname": None}).json()["email"] is None
|
||||
with Session(migrated_database) as db:
|
||||
assert db.scalar(select(User).where(User.username == "other_user")).email is None
|
||||
assert db.scalar(select(User).where(User.id == first["id"])).role == "user"
|
||||
|
||||
|
||||
def test_password_change_revokes_all_sessions(anonymous_client):
|
||||
c = anonymous_client
|
||||
register(c)
|
||||
login(c)
|
||||
first = c.cookies.get(COOKIE_NAME)
|
||||
# Independent second browser: existing first session must also be invalidated.
|
||||
c.cookies.clear()
|
||||
login(c)
|
||||
assert (
|
||||
c.post(
|
||||
"/api/auth/change-password",
|
||||
json={"current_password": "wrong", "new_password": "new-password-123"},
|
||||
).status_code
|
||||
== 400
|
||||
)
|
||||
response = c.post(
|
||||
"/api/auth/change-password",
|
||||
json={"current_password": PASSWORD, "new_password": "new-password-123"},
|
||||
)
|
||||
assert response.status_code == 204
|
||||
assert c.get("/api/auth/me").status_code == 401
|
||||
c.cookies.set(COOKIE_NAME, first)
|
||||
assert c.get("/api/auth/me").status_code == 401
|
||||
assert login(c).status_code == 401
|
||||
assert login(c, password="new-password-123").status_code == 200
|
||||
|
||||
|
||||
def test_all_management_routes_require_admin_and_history_is_private(anonymous_client):
|
||||
c = anonymous_client
|
||||
paths = [
|
||||
("GET", "/api/admin/users"),
|
||||
("GET", "/api/knowledge"),
|
||||
("GET", "/api/knowledge/page"),
|
||||
("GET", "/api/knowledge/1"),
|
||||
("POST", "/api/knowledge"),
|
||||
("PATCH", "/api/knowledge/1"),
|
||||
("DELETE", "/api/knowledge/1?expected_version=1"),
|
||||
("POST", "/api/knowledge/1/review"),
|
||||
("GET", "/api/knowledge/1/history"),
|
||||
("GET", "/api/sources"),
|
||||
("POST", "/api/sources"),
|
||||
("GET", "/api/sources/1"),
|
||||
("PATCH", "/api/sources/1"),
|
||||
("DELETE", "/api/sources/1?expected_version=1"),
|
||||
("GET", "/api/sources/1/history"),
|
||||
("GET", "/api/documents"),
|
||||
("POST", "/api/documents"),
|
||||
("GET", "/api/documents/1"),
|
||||
("PATCH", "/api/documents/1"),
|
||||
("DELETE", "/api/documents/1?expected_version=1"),
|
||||
("GET", "/api/documents/1/history"),
|
||||
("GET", "/api/retrieval/runs"),
|
||||
("GET", "/api/retrieval/runs/00000000-0000-0000-0000-000000000000"),
|
||||
]
|
||||
for method, path in paths:
|
||||
assert c.request(method, path, json={}).status_code == 401, path
|
||||
register(c)
|
||||
login(c)
|
||||
for method, path in paths:
|
||||
assert c.request(method, path, json={}).status_code == 403, path
|
||||
assert c.post("/api/retrieval", json={"query": "占位"}).json()["status"] == "no_evidence"
|
||||
assert c.post("/api/chat", json={"question": "占位"}).json()["status"] == "stub"
|
||||
|
||||
|
||||
def test_admin_access_live_role_and_no_csrf_bypass(client, migrated_database):
|
||||
c = client
|
||||
users = c.get("/api/admin/users").json()
|
||||
assert users[0]["role"] == "admin" and "password_hash" not in str(users)
|
||||
assert c.get("/api/admin/users?limit=101").status_code == 422
|
||||
assert (
|
||||
c.post(
|
||||
"/api/knowledge",
|
||||
json={"title": "示例", "content": "示例"},
|
||||
headers={"X-CSRF-Token": ""},
|
||||
).status_code
|
||||
== 403
|
||||
)
|
||||
assert c.post("/api/knowledge", json={"title": "示例", "content": "示例"}).status_code == 201
|
||||
with Session(migrated_database) as db:
|
||||
db.scalar(select(User)).role = "user"
|
||||
db.commit()
|
||||
assert c.get("/api/admin/users").status_code == 403
|
||||
assert c.get("/api/knowledge").status_code == 403
|
||||
|
||||
|
||||
def test_auth_settings_require_explicit_origins_and_secure_production():
|
||||
with pytest.raises(ValidationError):
|
||||
Settings(_env_file=None, app_env="production", auth_cookie_secure=False)
|
||||
for origin in [
|
||||
"*",
|
||||
"null",
|
||||
"https://example.com/path",
|
||||
"https://user@example.com",
|
||||
"https://*.example.com",
|
||||
]:
|
||||
with pytest.raises(ValidationError):
|
||||
Settings(_env_file=None, auth_allowed_origins=origin)
|
||||
|
||||
|
||||
def test_local_admin_provisioning_has_no_default_account(migrated_database, monkeypatch, capsys):
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from app import manage
|
||||
|
||||
monkeypatch.setattr(manage, "SessionLocal", sessionmaker(bind=migrated_database))
|
||||
monkeypatch.setattr("sys.argv", ["manage", "create-admin", "--username", "local_admin"])
|
||||
monkeypatch.setattr(manage.getpass, "getpass", lambda _: PASSWORD)
|
||||
manage.main()
|
||||
with Session(migrated_database) as db:
|
||||
user = db.scalar(select(User))
|
||||
assert user.role == "admin" and check_password(user.password_hash, PASSWORD)
|
||||
with pytest.raises(SystemExit) as error:
|
||||
manage.main()
|
||||
assert error.value.code == 1
|
||||
assert PASSWORD not in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_client_budget_is_shared_across_names_and_registration(anonymous_client, migrated_database):
|
||||
c = anonymous_client
|
||||
for index in range(30):
|
||||
assert login(c, username=f"absent_{index}").status_code == 401
|
||||
assert register(c).status_code == 429
|
||||
# Expire the persisted budget instead of waiting in a test.
|
||||
from app.models.auth import AuthRateLimit
|
||||
|
||||
with Session(migrated_database) as db:
|
||||
for bucket in db.scalars(select(AuthRateLimit)):
|
||||
bucket.window_start = utc_now() - timedelta(minutes=6)
|
||||
db.commit()
|
||||
assert register(c).status_code == 201
|
||||
@@ -0,0 +1,269 @@
|
||||
from datetime import datetime
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session
|
||||
from sqlalchemy.orm.exc import StaleDataError
|
||||
|
||||
from app.models import KnowledgeItem
|
||||
|
||||
COLLECTED = "2026-10-15T09:00:00+08:00"
|
||||
|
||||
|
||||
def create(client, path, data):
|
||||
response = client.post("/api/" + path, json=data)
|
||||
assert response.status_code == 201, response.text
|
||||
return response.json()
|
||||
|
||||
|
||||
def source(client):
|
||||
return create(client, "sources", {"name": "示例来源", "source_url": "https://example.com/"})
|
||||
|
||||
|
||||
def document(client, source_id):
|
||||
return create(
|
||||
client,
|
||||
"documents",
|
||||
{
|
||||
"source_id": source_id,
|
||||
"title": "示例文档",
|
||||
"content": "仅用于自动化测试的占位内容",
|
||||
"collected_at": COLLECTED,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def knowledge(client, **overrides):
|
||||
return create(
|
||||
client,
|
||||
"knowledge",
|
||||
{
|
||||
"title": "示例知识",
|
||||
"content": "测试资料,不代表文旅事实",
|
||||
"source_url": "https://example.com/",
|
||||
"collected_at": COLLECTED,
|
||||
**overrides,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def review(client, item, status="verified"):
|
||||
return client.post(
|
||||
f"/api/knowledge/{item['id']}/review",
|
||||
json={
|
||||
"expected_version": item["version"],
|
||||
"verification_status": status,
|
||||
"reviewer": "test-reviewer",
|
||||
"verification_notes": "仅验证审核流程,非真实核验结论",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def test_health_docs_and_stub(client):
|
||||
assert client.get("/api/health").json()["status"] == "ok"
|
||||
assert client.get("/api/ready").json()["database"] == "connected"
|
||||
assert client.get("/docs").status_code == 200
|
||||
assert "/api/knowledge/{item_id}/review" in client.get("/openapi.json").json()["paths"]
|
||||
assert client.post("/api/chat", json={"question": "测试"}).json()["status"] == "stub"
|
||||
|
||||
|
||||
def test_legacy_create_and_missing_evidence(client):
|
||||
item = create(client, "knowledge", {"title": "旧版调用", "content": "示例"})
|
||||
assert item["verification_status"] == "pending"
|
||||
assert item["collected_at"] is None
|
||||
assert isinstance(client.get("/api/knowledge").json(), list)
|
||||
assert review(client, item).status_code == 422
|
||||
assert client.get(f"/api/knowledge/{item['id']}/history").json()["total"] == 1
|
||||
|
||||
|
||||
def test_optional_empty_url_matches_existing_frontend(client):
|
||||
item = create(client, "knowledge", {"title": "示例", "content": "示例", "source_url": ""})
|
||||
assert item["source_url"] is None
|
||||
|
||||
|
||||
def test_reassign_document_resolves_new_source(client):
|
||||
first = source(client)
|
||||
first_doc = document(client, first["id"])
|
||||
second = source(client)
|
||||
second_doc = document(client, second["id"])
|
||||
item = knowledge(client, document_id=first_doc["id"])
|
||||
response = client.patch(
|
||||
f"/api/knowledge/{item['id']}",
|
||||
json={
|
||||
"expected_version": item["version"],
|
||||
"document_id": second_doc["id"],
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
assert response.json()["source_id"] == second["id"]
|
||||
|
||||
|
||||
def test_browser_preflight_and_document_source_lists(client):
|
||||
response = client.options(
|
||||
"/api/knowledge",
|
||||
headers={
|
||||
"Origin": "http://localhost:5173",
|
||||
"Access-Control-Request-Method": "POST",
|
||||
"Access-Control-Request-Headers": "content-type",
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
assert response.headers["access-control-allow-origin"] == "http://localhost:5173"
|
||||
src = source(client)
|
||||
doc = document(client, src["id"])
|
||||
assert client.get("/api/sources?q=示例").json()["items"][0]["id"] == src["id"]
|
||||
result = client.get(f"/api/documents?source_id={src['id']}&q=占位").json()
|
||||
assert result["total"] == 1 and result["items"][0]["id"] == doc["id"]
|
||||
|
||||
|
||||
def test_review_edit_and_history(client):
|
||||
item = knowledge(client)
|
||||
verified = review(client, item)
|
||||
assert verified.status_code == 200
|
||||
verified = verified.json()
|
||||
assert verified["version"] == 2
|
||||
assert datetime.fromisoformat(verified["reviewed_at"]).tzinfo is not None
|
||||
assert review(client, item).status_code == 409
|
||||
response = client.patch(
|
||||
f"/api/knowledge/{item['id']}",
|
||||
json={
|
||||
"expected_version": verified["version"],
|
||||
"content": "修订后的示例正文",
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
updated = response.json()
|
||||
assert updated["verification_status"] == "pending"
|
||||
assert updated["reviewer"] is None and updated["reviewed_at"] is None
|
||||
events = client.get(f"/api/knowledge/{item['id']}/history").json()
|
||||
assert [event["action"] for event in events["items"]] == ["update", "review", "create"]
|
||||
assert events["items"][1]["snapshot"]["item"]["verification_status"] == "verified"
|
||||
|
||||
|
||||
def test_source_document_changes_invalidate_evidence(client):
|
||||
src = source(client)
|
||||
doc = document(client, src["id"])
|
||||
item = knowledge(client, source_id=src["id"], document_id=doc["id"])
|
||||
assert review(client, item).status_code == 200
|
||||
response = client.patch(
|
||||
f"/api/documents/{doc['id']}",
|
||||
json={
|
||||
"expected_version": doc["version"],
|
||||
"content": "修订后的文档",
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
current = client.get(f"/api/knowledge/{item['id']}").json()
|
||||
assert current["verification_status"] == "pending"
|
||||
assert review(client, current).status_code == 200
|
||||
response = client.patch(
|
||||
f"/api/sources/{src['id']}",
|
||||
json={
|
||||
"expected_version": src["version"],
|
||||
"source_url": "https://example.com/new",
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
current = client.get(f"/api/knowledge/{item['id']}").json()
|
||||
assert current["verification_status"] == "pending"
|
||||
assert current["source_url"] == "https://example.com/new"
|
||||
events = client.get(f"/api/knowledge/{item['id']}/history").json()["items"]
|
||||
reviewed = next(e for e in events if e["action"] == "review")
|
||||
assert reviewed["snapshot"]["source"]["source_url"] == "https://example.com/"
|
||||
assert reviewed["snapshot"]["document"]["content"] == "修订后的文档"
|
||||
|
||||
|
||||
def test_delete_preserves_history_and_guards_references(client):
|
||||
src = source(client)
|
||||
doc = document(client, src["id"])
|
||||
item = knowledge(client, document_id=doc["id"])
|
||||
assert item["source_id"] == src["id"]
|
||||
assert client.delete(f"/api/sources/{src['id']}?expected_version=1").status_code == 409
|
||||
assert client.delete(f"/api/documents/{doc['id']}?expected_version=1").status_code == 409
|
||||
assert client.delete(f"/api/knowledge/{item['id']}?expected_version=2").status_code == 409
|
||||
assert client.delete(f"/api/knowledge/{item['id']}?expected_version=1").status_code == 204
|
||||
assert client.get(f"/api/knowledge/{item['id']}").status_code == 404
|
||||
assert client.get("/api/knowledge").json() == []
|
||||
assert (
|
||||
client.get(f"/api/knowledge/{item['id']}/history").json()["items"][0]["action"] == "delete"
|
||||
)
|
||||
assert client.delete(f"/api/documents/{doc['id']}?expected_version=1").status_code == 204
|
||||
assert client.delete(f"/api/sources/{src['id']}?expected_version=1").status_code == 204
|
||||
assert client.get("/api/sources").json()["total"] == 0
|
||||
|
||||
|
||||
def test_filter_pagination_and_literal_search(client):
|
||||
knowledge(client, title="示例%资料", category="交通出行")
|
||||
other = knowledge(client, title="其他资料", category="民宿服务")
|
||||
assert review(client, other, "rejected").status_code == 200
|
||||
page = client.get("/api/knowledge/page?page_size=1").json()
|
||||
assert page["total"] == 2 and len(page["items"]) == 1
|
||||
assert page["items"][0]["id"] == other["id"]
|
||||
assert client.get("/api/knowledge/page?page=3&page_size=1").json()["items"] == []
|
||||
assert client.get("/api/knowledge/page?q=%25").json()["total"] == 1
|
||||
assert len(client.get("/api/knowledge?category=交通出行").json()) == 1
|
||||
assert client.get("/api/knowledge/page?verification_status=verified").json()["total"] == 0
|
||||
assert client.get("/api/knowledge/page?verification_status=rejected").json()["total"] == 1
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"patch",
|
||||
[
|
||||
{"title": " "},
|
||||
{"title": None},
|
||||
{"content": ""},
|
||||
{"category": ""},
|
||||
{"collected_at": "2026-10-15T09:00:00"},
|
||||
{"source_url": "javascript:alert(1)"},
|
||||
{"verification_status": "verified"},
|
||||
{},
|
||||
],
|
||||
)
|
||||
def test_invalid_update_cannot_mutate_or_audit(client, patch):
|
||||
item = knowledge(client)
|
||||
response = client.patch(f"/api/knowledge/{item['id']}", json={"expected_version": 1, **patch})
|
||||
assert response.status_code == 422
|
||||
assert client.get(f"/api/knowledge/{item['id']}").json()["version"] == 1
|
||||
assert client.get(f"/api/knowledge/{item['id']}/history").json()["total"] == 1
|
||||
|
||||
|
||||
def test_invalid_relationship_and_query(client):
|
||||
src = source(client)
|
||||
doc = document(client, src["id"])
|
||||
second = source(client)
|
||||
response = client.post(
|
||||
"/api/knowledge",
|
||||
json={
|
||||
"title": "示例",
|
||||
"content": "示例",
|
||||
"source_id": second["id"],
|
||||
"document_id": doc["id"],
|
||||
},
|
||||
)
|
||||
assert response.status_code == 422
|
||||
assert (
|
||||
client.post(
|
||||
"/api/knowledge", json={"title": "示例", "content": "示例", "document_id": 9999}
|
||||
).status_code
|
||||
== 404
|
||||
)
|
||||
for path in ["knowledge", "sources", "documents"]:
|
||||
assert client.get(f"/api/{path}/9999").status_code == 404
|
||||
assert client.get(f"/api/{path}/9999/history").status_code == 404
|
||||
assert client.get("/api/knowledge/page?page_size=101").status_code == 422
|
||||
assert client.get("/api/knowledge/page?verification_status=unknown").status_code == 422
|
||||
assert client.get("/api/knowledge/page?page=0").status_code == 422
|
||||
|
||||
|
||||
def test_optimistic_lock_prevents_concurrent_overwrite(client, migrated_database):
|
||||
item = knowledge(client)
|
||||
with Session(migrated_database) as first, Session(migrated_database) as second:
|
||||
first_item = first.scalar(select(KnowledgeItem).where(KnowledgeItem.id == item["id"]))
|
||||
second_item = second.get(KnowledgeItem, item["id"])
|
||||
first_item.content = "先提交的内容"
|
||||
first.commit()
|
||||
second_item.content = "过期编辑"
|
||||
with pytest.raises(StaleDataError):
|
||||
second.commit()
|
||||
second.rollback()
|
||||
assert client.get(f"/api/knowledge/{item['id']}").json()["content"] == "先提交的内容"
|
||||
@@ -0,0 +1,127 @@
|
||||
"""Lightweight contract/metric tests do not require model packages or network access."""
|
||||
|
||||
import json
|
||||
from dataclasses import FrozenInstanceError, replace
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from app.services.embedding import (
|
||||
SMALL_ZH,
|
||||
EmbeddingError,
|
||||
EmbeddingSpec,
|
||||
LocalEmbeddingProvider,
|
||||
validate_vectors,
|
||||
)
|
||||
from evaluation.local_embeddings import load_dataset, rank_and_measure
|
||||
|
||||
|
||||
def test_model_space_identity_is_immutable_and_includes_semantic_configuration():
|
||||
assert SMALL_ZH.space_id == replace(SMALL_ZH).space_id
|
||||
for change in [
|
||||
{"revision": "a" * 40},
|
||||
{"dimension": 768},
|
||||
{"query_instruction": ""},
|
||||
{"token_limit": 256},
|
||||
{"implementation": "another-pooling"},
|
||||
]:
|
||||
assert replace(SMALL_ZH, **change).space_id != SMALL_ZH.space_id
|
||||
with pytest.raises(FrozenInstanceError):
|
||||
SMALL_ZH.dimension = 768
|
||||
|
||||
|
||||
@pytest.mark.parametrize("revision", ["main", "v1.5", "a" * 39, None])
|
||||
def test_model_requires_full_revision(revision):
|
||||
with pytest.raises(EmbeddingError):
|
||||
EmbeddingSpec("example", revision, 2)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"vectors,count",
|
||||
[
|
||||
([[1.0, 0.0]], 2),
|
||||
([[1.0]], 1),
|
||||
([[0.0, 0.0]], 1),
|
||||
([[float("nan"), 0.0]], 1),
|
||||
([[float("inf"), 0.0]], 1),
|
||||
([["invalid", 0.0]], 1),
|
||||
([[True, 0.0]], 1),
|
||||
],
|
||||
)
|
||||
def test_invalid_vectors_cannot_pass_contract(vectors, count):
|
||||
spec = EmbeddingSpec("example", "a" * 40, 2)
|
||||
with pytest.raises(EmbeddingError):
|
||||
validate_vectors(vectors, count, spec)
|
||||
validate_vectors([[1.0, 0.0]], 1, spec)
|
||||
|
||||
|
||||
def test_token_overflow_is_rejected_before_any_model_execution():
|
||||
provider = object.__new__(LocalEmbeddingProvider)
|
||||
provider.spec = SMALL_ZH
|
||||
provider.batch_size = 8
|
||||
provider.torch = None
|
||||
|
||||
class Shape:
|
||||
shape = (1, 513)
|
||||
|
||||
provider.tokenizer = lambda *args, **kwargs: {"input_ids": Shape()}
|
||||
# There is intentionally no model: exceeding the budget must never execute one.
|
||||
with pytest.raises(EmbeddingError, match="token"):
|
||||
provider.encode(["占位文本"])
|
||||
|
||||
|
||||
def test_ranking_metrics_match_hand_calculation_and_ties_are_stable():
|
||||
docs = [{"id": id_} for id_ in ["a", "b", "c"]]
|
||||
queries = [{"id": "q1", "relevant_ids": ["b", "c"]}, {"id": "q2", "relevant_ids": ["c"]}]
|
||||
result = rank_and_measure(docs, queries, [[1, 0], [0, 1], [-1, 0]], [[1, 1], [-1, 0]], top_k=2)
|
||||
assert result["hit_at_1"] == 0.5
|
||||
assert result["recall_at_k"] == 0.75
|
||||
assert result["mrr_at_k"] == 0.75
|
||||
assert [x["id"] for x in result["per_query"][0]["top_results"]] == ["a", "b"]
|
||||
|
||||
|
||||
def test_dataset_validation_rejects_unknown_labels_and_duplicate_ids(tmp_path):
|
||||
data = {
|
||||
"dataset_id": "placeholder",
|
||||
"purpose": "synthetic_smoke_not_domain_benchmark",
|
||||
"documents": [{"id": "d", "text": "占位"}],
|
||||
"queries": [{"id": "q", "text": "测试", "relevant_ids": ["missing"]}],
|
||||
}
|
||||
path = tmp_path / "dataset.json"
|
||||
path.write_text(json.dumps(data))
|
||||
with pytest.raises(ValueError):
|
||||
load_dataset(path)
|
||||
data["queries"][0]["relevant_ids"] = ["d"]
|
||||
data["documents"].append({"id": "d", "text": "重复"})
|
||||
path.write_text(json.dumps(data))
|
||||
with pytest.raises(ValueError):
|
||||
load_dataset(path)
|
||||
|
||||
|
||||
def test_bundled_dataset_is_explicitly_synthetic_and_reproducible():
|
||||
path = Path(__file__).resolve().parents[1] / "evaluation" / "synthetic_zh.json"
|
||||
dataset, digest = load_dataset(path)
|
||||
assert dataset["purpose"] == "synthetic_smoke_not_domain_benchmark"
|
||||
assert all(row["text"].startswith("【合成测试】") for row in dataset["documents"])
|
||||
assert load_dataset(path)[1] == digest
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"options", [{"batch_size": 0}, {"batch_size": 33}, {"threads": 5}, {"batch_size": 1.5}]
|
||||
)
|
||||
def test_invalid_resource_options_fail_without_loading_optional_dependencies(options):
|
||||
with pytest.raises(EmbeddingError):
|
||||
LocalEmbeddingProvider(SMALL_ZH, **options)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("checkpoint", ["pickle", "automatic-fallback"])
|
||||
def test_unsupported_checkpoint_format_is_rejected(checkpoint):
|
||||
with pytest.raises(EmbeddingError):
|
||||
replace(SMALL_ZH, checkpoint_format=checkpoint)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("texts", [None, "不能逐字编码的字符串", ("占位",), [None], [" "]])
|
||||
def test_input_shape_and_text_are_validated_before_encoding(texts):
|
||||
provider = object.__new__(LocalEmbeddingProvider)
|
||||
with pytest.raises(EmbeddingError):
|
||||
provider.encode(texts)
|
||||
@@ -0,0 +1,306 @@
|
||||
"""Synthetic coordinates/evidence test fusion semantics, not domain/model quality."""
|
||||
|
||||
import math
|
||||
from dataclasses import replace
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import func, select, text
|
||||
from sqlalchemy.orm import Session
|
||||
from test_indexing import build
|
||||
from test_retrieval import add_item
|
||||
from test_vectors import ToyProvider, text_version, vector_build, vector_publish
|
||||
|
||||
from app.models.retrieval import RetrievalRun
|
||||
from app.schemas.hybrid import RankedCandidate
|
||||
from app.services import vectors
|
||||
from app.services.embedding import EmbeddingError
|
||||
from app.services.embedding_runtime import get_embedding_provider
|
||||
from app.services.hybrid import fuse
|
||||
from app.services.indexing import publish_index
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def provider(client):
|
||||
value = ToyProvider()
|
||||
client.app.dependency_overrides[get_embedding_provider] = lambda: value
|
||||
return value
|
||||
|
||||
|
||||
def search(client, path="hybrid", **kwargs):
|
||||
return client.post(f"/api/retrieval/{path}", json={"query": "query", **kwargs})
|
||||
|
||||
|
||||
def publish_vectors(client):
|
||||
text_ = text_version(client)
|
||||
version = vector_build(client, text_)
|
||||
assert vector_publish(client, version).status_code == 200
|
||||
return text_, version
|
||||
|
||||
|
||||
def test_rrf_hand_calculation_absent_leg_and_raw_score_independence():
|
||||
a, b, c = uuid4(), uuid4(), uuid4()
|
||||
lexical = [
|
||||
RankedCandidate(chunk_id=a, rank=1, score=1000),
|
||||
RankedCandidate(chunk_id=b, rank=2, score=1),
|
||||
]
|
||||
vector = [
|
||||
RankedCandidate(chunk_id=b, rank=1, score=-0.9),
|
||||
RankedCandidate(chunk_id=c, rank=2, score=-1),
|
||||
]
|
||||
scores = fuse(lexical, vector)
|
||||
assert scores == {str(a): 1 / 61, str(b): 1 / 62 + 1 / 61, str(c): 1 / 62}
|
||||
for candidate in lexical + vector:
|
||||
candidate.score *= 10000
|
||||
assert fuse(lexical, vector) == scores
|
||||
assert fuse([], []) == {}
|
||||
|
||||
|
||||
def test_same_chunk_scope_standalone_rankings_formula_and_history(client, provider):
|
||||
items = [
|
||||
add_item(client, content="alpha query extra words words"),
|
||||
add_item(client, content="query"),
|
||||
add_item(client, content="beta"),
|
||||
]
|
||||
add_item(client, status="pending", content="query alpha")
|
||||
text_, version = publish_vectors(client)
|
||||
response = search(client, top_k=3, candidate_k=3)
|
||||
assert response.status_code == 200, response.text
|
||||
result = response.json()
|
||||
lexical = search(client, "chunks", top_k=3).json()
|
||||
vector = search(client, "vector", top_k=3).json()
|
||||
assert result["corpus_size"] == lexical["corpus_size"] == vector["corpus_size"] == 3
|
||||
assert (
|
||||
result["text_index_version_id"]
|
||||
== lexical["text_index_version_id"]
|
||||
== vector["text_index_version_id"]
|
||||
== text_["id"]
|
||||
)
|
||||
assert result["vector_version_id"] == version["id"]
|
||||
assert result["config"]["space"]["id"] == provider.spec.space_id
|
||||
assert result["config"]["lexical"]["title_weight"] == 0
|
||||
assert result["config"]["lexical"]["k1"] == 1.2
|
||||
assert result["config"]["lexical"]["b"] == 0.75
|
||||
assert result["config"]["chunking"] == text_["config"]
|
||||
assert result["lexical_total_hits"] == 2 and result["vector_total_hits"] == 3
|
||||
assert [c["chunk_id"] for c in result["lexical_candidates"]] == [
|
||||
c["chunk_id"] for c in lexical["results"]
|
||||
]
|
||||
assert [c["chunk_id"] for c in result["vector_candidates"]] == [
|
||||
c["chunk_id"] for c in vector["results"]
|
||||
]
|
||||
# Independently calculate positive-log-IDF BM25 on bodies of lengths 5, 1, 1.
|
||||
expected = math.log(1 + 1.5 / 2.5) * 2.2 / (1 + 1.2 * (0.25 + 0.75 / (7 / 3)))
|
||||
assert lexical["results"][0]["knowledge_id"] == items[1]["id"]
|
||||
assert lexical["results"][0]["score"] == pytest.approx(expected)
|
||||
for hit in result["results"]:
|
||||
left = next(
|
||||
(c for c in result["lexical_candidates"] if c["chunk_id"] == hit["chunk_id"]), None
|
||||
)
|
||||
right = next(
|
||||
(c for c in result["vector_candidates"] if c["chunk_id"] == hit["chunk_id"]), None
|
||||
)
|
||||
assert hit["score"] == pytest.approx(
|
||||
(1 / (60 + left["rank"]) if left else 0) + (1 / (60 + right["rank"]) if right else 0)
|
||||
)
|
||||
assert hit["score"] == hit["lexical_contribution"] + hit["vector_contribution"]
|
||||
assert hit["source_url"] == "https://example.com/"
|
||||
assert search(client, candidate_k=3, top_k=3).json()["results"] == result["results"]
|
||||
for value in (result, lexical, vector):
|
||||
assert client.get(f"/api/retrieval/runs/{value['run_id']}").json() == value
|
||||
old = client.post("/api/retrieval", json={"query": "query"}).json()
|
||||
assert old["mode"] == "keyword" and "chunk_id" not in old["results"][0]
|
||||
history = client.get("/api/retrieval/runs?page_size=100").json()
|
||||
assert {r["mode"] for r in history["items"]} == {
|
||||
"keyword",
|
||||
"vector",
|
||||
"keyword_chunks",
|
||||
"hybrid",
|
||||
}
|
||||
client.cookies.clear()
|
||||
assert search(client).status_code == search(client, "chunks").status_code == 200
|
||||
assert client.get("/api/retrieval/runs").status_code == 401
|
||||
|
||||
|
||||
def test_candidate_cutoffs_disjoint_legs_and_deterministic_tie(client, provider):
|
||||
first = add_item(client, content="alpha")
|
||||
second = add_item(client, content="query")
|
||||
publish_vectors(client)
|
||||
small = search(client, candidate_k=1, top_k=1).json()
|
||||
assert small["total_hits"] == 2 and len(small["results"]) == 1
|
||||
assert small["results"][0]["knowledge_id"] == first["id"]
|
||||
assert small["results"][0]["lexical_rank"] is None
|
||||
assert small["results"][0]["lexical_score"] is None
|
||||
assert small["results"][0]["lexical_contribution"] == 0
|
||||
assert small["results"][0]["score"] == 1 / 61
|
||||
full = search(client, candidate_k=2, top_k=2).json()
|
||||
assert full["results"][0]["knowledge_id"] == second["id"]
|
||||
assert full["results"][0]["lexical_rank"] == 1 and full["results"][0]["vector_rank"] == 2
|
||||
|
||||
|
||||
def test_chunk_bodies_offsets_overlap_and_category_apply_before_both_rankings(client, provider):
|
||||
item = add_item(client, title="uniquetitle", content="alpha query 文本。" * 30, category="交通")
|
||||
add_item(client, content="query", category="其他")
|
||||
publish_vectors(client)
|
||||
result = search(client, category="交通", top_k=20, candidate_k=20).json()
|
||||
assert result["corpus_size"] > 1 and result["total_hits"] == result["corpus_size"]
|
||||
assert len({h["chunk_id"] for h in result["results"]}) == len(result["results"])
|
||||
for hit in result["results"]:
|
||||
assert hit["knowledge_id"] == item["id"]
|
||||
assert hit["snippet"] == item["content"][hit["snippet_start"] : hit["snippet_end"]]
|
||||
assert search(client, "chunks", query="uniquetitle").json()["status"] == "no_evidence"
|
||||
empty = search(client, category="不存在分类").json()
|
||||
assert empty["status"] == "no_evidence" and empty["results"] == []
|
||||
assert empty["lexical_candidates"] == empty["vector_candidates"] == []
|
||||
|
||||
|
||||
def test_empty_lexical_leg_and_empty_corpus_are_explicit(client, provider):
|
||||
add_item(client, content="alpha")
|
||||
publish_vectors(client)
|
||||
for query in ["unmatchedword", "!!"]:
|
||||
result = search(client, query=query).json()
|
||||
assert result["lexical_candidates"] == [] and result["status"] == "ok"
|
||||
assert result["results"][0]["score"] == 1 / 61
|
||||
assert search(client, "chunks", query="!!").status_code == 422
|
||||
|
||||
|
||||
def test_empty_published_corpus(client, provider):
|
||||
publish_vectors(client)
|
||||
for path in ["chunks", "hybrid"]:
|
||||
result = search(client, path).json()
|
||||
assert result["status"] == "no_evidence" and result["total_hits"] == 0
|
||||
|
||||
|
||||
def test_keyword_chunks_need_no_runtime_or_vector_publication(client):
|
||||
add_item(client, content="query")
|
||||
assert search(client, "chunks").status_code == 409
|
||||
text_version(client)
|
||||
assert search(client, "chunks").status_code == 200
|
||||
assert search(client).status_code == 503
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"payload",
|
||||
[
|
||||
{"candidate_k": 1, "top_k": 2},
|
||||
{"candidate_k": 201},
|
||||
{"candidate_k": 0},
|
||||
{"top_k": 21},
|
||||
{"top_k": 0},
|
||||
{"mode": "vector"},
|
||||
{"rrf_k": 10},
|
||||
{"query": " "},
|
||||
{"query": "x" * 501},
|
||||
],
|
||||
)
|
||||
def test_request_budget_and_fixed_config_validation(client, provider, payload):
|
||||
assert search(client, **payload).status_code == 422
|
||||
|
||||
|
||||
@pytest.mark.parametrize("change", ["catalog", "text", "vector"])
|
||||
def test_query_encoding_releases_locks_and_rechecks_publication(
|
||||
client, provider, migrated_database, change
|
||||
):
|
||||
item = add_item(client, content="alpha query")
|
||||
text_, first = publish_vectors(client)
|
||||
replacement_text = build(client)
|
||||
replacement_vector = vector_build(client, text_)
|
||||
|
||||
def mutate(query):
|
||||
assert query
|
||||
with Session(migrated_database) as other:
|
||||
other.execute(text("SET LOCAL lock_timeout='1s'"))
|
||||
if change == "catalog":
|
||||
state = vectors.locked_state(other)
|
||||
state.catalog_revision += 1
|
||||
other.execute(
|
||||
text(
|
||||
"UPDATE knowledge_items SET content='changed synthetic evidence' WHERE id=:id"
|
||||
),
|
||||
{"id": item["id"]},
|
||||
)
|
||||
other.commit()
|
||||
elif change == "text":
|
||||
publish_index(other, UUID(replacement_text["id"]), 1, first["created_by"])
|
||||
else:
|
||||
vectors.publish_vectors(
|
||||
other, UUID(replacement_vector["id"]), 1, first["created_by"]
|
||||
)
|
||||
|
||||
provider.callback = mutate
|
||||
assert search(client).status_code == 409
|
||||
with Session(migrated_database) as db:
|
||||
assert db.scalar(select(func.count()).select_from(RetrievalRun)) == 0
|
||||
|
||||
|
||||
@pytest.mark.parametrize("change", ["body", "chunks", "vectors", "space"])
|
||||
def test_corrupt_or_incompatible_snapshot_fails_closed(client, provider, migrated_database, change):
|
||||
add_item(client, content="alpha query")
|
||||
publish_vectors(client)
|
||||
if change == "space":
|
||||
provider.spec = replace(provider.spec, revision="b" * 40)
|
||||
else:
|
||||
with migrated_database.begin() as connection:
|
||||
connection.execute(
|
||||
text(
|
||||
{
|
||||
"body": "UPDATE knowledge_items SET content='out of band change'",
|
||||
"chunks": "UPDATE chunks SET text='tampered'",
|
||||
"vectors": "DELETE FROM chunk_vectors",
|
||||
}[change]
|
||||
)
|
||||
)
|
||||
assert search(client).status_code == 409
|
||||
with Session(migrated_database) as db:
|
||||
assert db.scalar(select(func.count()).select_from(RetrievalRun)) == 0
|
||||
|
||||
|
||||
@pytest.mark.parametrize("failure,status", [("token", 422), ("runtime", 503)])
|
||||
def test_model_failure_has_no_fallback_or_partial_history(
|
||||
client, provider, migrated_database, failure, status
|
||||
):
|
||||
add_item(client, content="alpha query")
|
||||
publish_vectors(client)
|
||||
|
||||
def fail(*args, **kwargs):
|
||||
if failure == "token":
|
||||
raise EmbeddingError("token budget")
|
||||
raise OSError("private-runtime-path")
|
||||
|
||||
provider.encode = fail
|
||||
response = search(client)
|
||||
assert response.status_code == status and "private-runtime-path" not in response.text
|
||||
with Session(migrated_database) as db:
|
||||
assert db.scalar(select(func.count()).select_from(RetrievalRun)) == 0
|
||||
|
||||
|
||||
def test_failed_history_insert_rolls_back_both_leg_result(client, provider, migrated_database):
|
||||
add_item(client, content="alpha query")
|
||||
publish_vectors(client)
|
||||
with migrated_database.begin() as connection:
|
||||
connection.execute(
|
||||
text("ALTER TABLE retrieval_runs ADD CONSTRAINT test_no_runs CHECK (false) NOT VALID")
|
||||
)
|
||||
assert search(client).status_code == 409
|
||||
with Session(migrated_database) as db:
|
||||
assert db.scalar(select(func.count()).select_from(RetrievalRun)) == 0
|
||||
|
||||
|
||||
def test_stale_history_survives_evidence_change_and_capacity_is_explicit(
|
||||
client, provider, migrated_database, monkeypatch
|
||||
):
|
||||
item = add_item(client, content="alpha query")
|
||||
publish_vectors(client)
|
||||
result = search(client).json()
|
||||
monkeypatch.setattr(vectors, "MAX_VECTOR_CHUNKS", 0)
|
||||
assert search(client).status_code == search(client, "chunks").status_code == 503
|
||||
monkeypatch.undo()
|
||||
assert (
|
||||
client.patch(
|
||||
f"/api/knowledge/{item['id']}",
|
||||
json={"expected_version": item["version"], "content": "updated synthetic body"},
|
||||
).status_code
|
||||
== 200
|
||||
)
|
||||
assert search(client).status_code == search(client, "chunks").status_code == 409
|
||||
assert client.get(f"/api/retrieval/runs/{result['run_id']}").json() == result
|
||||
@@ -0,0 +1,320 @@
|
||||
"""Versioned chunks remain evidence snapshots; publication uses real PostgreSQL locks."""
|
||||
|
||||
import hashlib
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
import pytest
|
||||
from fastapi import HTTPException
|
||||
from sqlalchemy import func, select, text
|
||||
from sqlalchemy.exc import OperationalError
|
||||
from sqlalchemy.orm import Session
|
||||
from test_retrieval import add_item
|
||||
|
||||
from app.models.indexing import Chunk, IndexPublication, IndexState, IndexVersion
|
||||
from app.services import indexing
|
||||
from app.services.indexing import locked_state
|
||||
|
||||
|
||||
def build(client, **payload):
|
||||
response = client.post("/api/indexes", json=payload)
|
||||
assert response.status_code == 201, response.text
|
||||
return response.json()
|
||||
|
||||
|
||||
def publish(client, version, generation=0):
|
||||
return client.post(
|
||||
f"/api/indexes/{version['id']}/publish", json={"expected_generation": generation}
|
||||
)
|
||||
|
||||
|
||||
def chunks(client, version):
|
||||
return client.get(f"/api/indexes/{version['id']}/chunks?page_size=100").json()["items"]
|
||||
|
||||
|
||||
def test_build_preserves_original_offsets_content_hash_and_provenance(client):
|
||||
content = ("A字符\n那拉提🙂占位资料。" * 37).strip()
|
||||
item = add_item(client, content=content)
|
||||
first = build(client, chunk_size=100, overlap=20)
|
||||
second = build(client, chunk_size=100, overlap=20)
|
||||
assert first["status"] == "ready" and first["item_count"] == 1
|
||||
assert first["corpus_fingerprint"] == second["corpus_fingerprint"]
|
||||
assert first["config"] == second["config"] and first["id"] != second["id"]
|
||||
actual = chunks(client, first)
|
||||
assert len(actual) == first["chunk_count"]
|
||||
assert [c["ordinal"] for c in actual] == list(range(len(actual)))
|
||||
assert actual[0]["start_offset"] == 0 and actual[-1]["end_offset"] == len(content)
|
||||
restored = [""] * len(content)
|
||||
for chunk in actual:
|
||||
start, end = chunk["start_offset"], chunk["end_offset"]
|
||||
assert chunk["text"] == content[start:end]
|
||||
assert chunk["text_sha256"] == hashlib.sha256(chunk["text"].encode()).hexdigest()
|
||||
assert chunk["provenance"]["knowledge_version"] == item["version"]
|
||||
assert chunk["provenance"]["source_url"] == "https://example.com/"
|
||||
restored[start:end] = list(chunk["text"])
|
||||
assert "".join(restored) == content
|
||||
assert [(c["text"], c["start_offset"], c["end_offset"]) for c in actual] == [
|
||||
(c["text"], c["start_offset"], c["end_offset"]) for c in chunks(client, second)
|
||||
]
|
||||
state = client.get("/api/indexes/active").json()
|
||||
assert state["generation"] == 0 and state["active_version"] is None and not state["is_current"]
|
||||
|
||||
|
||||
def test_only_valid_reviewed_evidence_is_chunked(client):
|
||||
good = add_item(client)
|
||||
add_item(client, status="pending")
|
||||
add_item(client, status="rejected")
|
||||
removed = add_item(client)
|
||||
assert (
|
||||
client.delete(
|
||||
f"/api/knowledge/{removed['id']}?expected_version={removed['version']}"
|
||||
).status_code
|
||||
== 204
|
||||
)
|
||||
version = build(client)
|
||||
assert version["item_count"] == 1
|
||||
assert {c["knowledge_id"] for c in chunks(client, version)} == {good["id"]}
|
||||
|
||||
|
||||
def test_publication_is_atomic_optimistic_and_audited(client):
|
||||
add_item(client)
|
||||
first, second = build(client), build(client)
|
||||
result = publish(client, first)
|
||||
assert result.status_code == 200 and result.json()["generation"] == 1
|
||||
assert result.json()["is_current"]
|
||||
assert publish(client, second).status_code == 409
|
||||
assert publish(client, second, 1).json()["generation"] == 2
|
||||
assert client.get(f"/api/indexes/{first['id']}").json()["status"] == "superseded"
|
||||
assert client.get(f"/api/indexes/{second['id']}").json()["status"] == "published"
|
||||
# Same active version with the current generation is an idempotent no-op.
|
||||
assert publish(client, second, 2).json()["generation"] == 2
|
||||
history = client.get("/api/indexes/publications").json()
|
||||
assert history["total"] == 2
|
||||
assert history["items"][0]["previous_version_id"] == first["id"]
|
||||
assert history["items"][0]["version_id"] == second["id"]
|
||||
assert client.get("/api/indexes/active").json()["active_version"]["id"] == second["id"]
|
||||
|
||||
|
||||
def test_catalog_mutation_marks_active_stale_and_rejects_old_build(client):
|
||||
item = add_item(client)
|
||||
version = build(client)
|
||||
assert publish(client, version).status_code == 200
|
||||
old = chunks(client, version)
|
||||
updated = client.patch(
|
||||
f"/api/knowledge/{item['id']}",
|
||||
json={"expected_version": item["version"], "content": "修订的占位正文"},
|
||||
)
|
||||
assert updated.status_code == 200
|
||||
active = client.get("/api/indexes/active").json()
|
||||
assert not active["is_current"] and active["generation"] == 1
|
||||
assert active["active_version"]["id"] == version["id"]
|
||||
assert chunks(client, version) == old
|
||||
assert publish(client, version, 1).status_code == 409
|
||||
replacement = build(client)
|
||||
assert replacement["item_count"] == 0 # revised body is pending, not approved evidence
|
||||
assert publish(client, replacement, 1).json()["is_current"]
|
||||
assert chunks(client, replacement) == []
|
||||
|
||||
|
||||
def test_failed_catalog_write_does_not_invalidate_current_version(client):
|
||||
item = add_item(client)
|
||||
version = build(client)
|
||||
publish(client, version)
|
||||
before = client.get("/api/indexes/active").json()
|
||||
assert (
|
||||
client.patch(
|
||||
f"/api/knowledge/{item['id']}", json={"expected_version": 999, "content": "占位"}
|
||||
).status_code
|
||||
== 409
|
||||
)
|
||||
assert client.post("/api/knowledge", json={"title": "", "content": "占位"}).status_code == 422
|
||||
assert client.get("/api/indexes/active").json() == before
|
||||
|
||||
|
||||
def test_source_and_document_mutations_preserve_old_provenance(client):
|
||||
src = client.post(
|
||||
"/api/sources", json={"name": "原始占位来源", "source_url": "https://example.com/"}
|
||||
).json()
|
||||
doc = client.post(
|
||||
"/api/documents",
|
||||
json={
|
||||
"source_id": src["id"],
|
||||
"title": "占位文档",
|
||||
"content": "占位正文",
|
||||
"collected_at": "2026-10-16T00:00:00Z",
|
||||
},
|
||||
).json()
|
||||
add_item(client, document_id=doc["id"])
|
||||
version = build(client)
|
||||
publish(client, version)
|
||||
snapshot = chunks(client, version)
|
||||
assert snapshot[0]["provenance"]["document_version"] == doc["version"]
|
||||
assert snapshot[0]["provenance"]["source_name"] == src["name"]
|
||||
assert (
|
||||
client.patch(
|
||||
f"/api/documents/{doc['id']}", json={"expected_version": 1, "content": "文档新正文"}
|
||||
).status_code
|
||||
== 200
|
||||
)
|
||||
assert not client.get("/api/indexes/active").json()["is_current"]
|
||||
empty = build(client)
|
||||
publish(client, empty, 1)
|
||||
assert (
|
||||
client.patch(
|
||||
f"/api/sources/{src['id']}", json={"expected_version": 1, "name": "新来源名"}
|
||||
).status_code
|
||||
== 200
|
||||
)
|
||||
assert not client.get("/api/indexes/active").json()["is_current"]
|
||||
assert chunks(client, version) == snapshot
|
||||
assert publish(client, version, 2).status_code == 409
|
||||
|
||||
|
||||
def test_fingerprint_recheck_rejects_out_of_band_catalog_changes(client, migrated_database):
|
||||
item = add_item(client)
|
||||
version = build(client)
|
||||
publish(client, version)
|
||||
with migrated_database.begin() as c:
|
||||
c.execute(
|
||||
text("UPDATE knowledge_items SET content=:body WHERE id=:id"),
|
||||
{"body": "直接修改的占位正文", "id": item["id"]},
|
||||
)
|
||||
assert not client.get("/api/indexes/active").json()["is_current"]
|
||||
assert publish(client, version, 1).status_code == 409
|
||||
|
||||
|
||||
@pytest.mark.parametrize("failure", ["chunk_budget", "corpus_budget", "database_write"])
|
||||
def test_failed_build_keeps_previous_publication_and_leaves_no_partial_rows(
|
||||
client, migrated_database, monkeypatch, failure
|
||||
):
|
||||
add_item(client, content="占位正文" * 100)
|
||||
previous = build(client)
|
||||
publish(client, previous)
|
||||
if failure == "chunk_budget":
|
||||
monkeypatch.setattr(indexing, "MAX_CHUNKS", 1)
|
||||
response = client.post("/api/indexes", json={"chunk_size": 100, "overlap": 0})
|
||||
assert response.status_code == 503
|
||||
elif failure == "corpus_budget":
|
||||
from app.services import retrieval
|
||||
|
||||
monkeypatch.setattr(retrieval, "MAX_ITEMS", 0)
|
||||
assert client.post("/api/indexes", json={}).status_code == 503
|
||||
monkeypatch.undo()
|
||||
else:
|
||||
# Real PostgreSQL constraint failure during chunk insert; the new version rolls back too.
|
||||
with migrated_database.begin() as c:
|
||||
c.execute(
|
||||
text("ALTER TABLE chunks ADD CONSTRAINT test_reject_chunk CHECK (false) NOT VALID")
|
||||
)
|
||||
assert client.post("/api/indexes", json={}).status_code == 409
|
||||
with Session(migrated_database) as db:
|
||||
assert db.scalar(select(func.count()).select_from(IndexVersion)) == 1
|
||||
assert db.scalar(select(func.count()).select_from(Chunk)) == previous["chunk_count"]
|
||||
assert db.get(IndexState, 1).active_version_id == previous["id"]
|
||||
assert db.scalar(select(func.count()).select_from(IndexPublication)) == 1
|
||||
assert client.get("/api/indexes/active").json()["active_version"]["id"] == previous["id"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"payload",
|
||||
[
|
||||
{"chunk_size": 99},
|
||||
{"chunk_size": 2001},
|
||||
{"overlap": -1},
|
||||
{"chunk_size": 100, "overlap": 51},
|
||||
{"unexpected": 1},
|
||||
],
|
||||
)
|
||||
def test_build_validation_has_no_side_effects(client, payload):
|
||||
assert client.post("/api/indexes", json=payload).status_code == 422
|
||||
assert client.get("/api/indexes").json()["total"] == 0
|
||||
|
||||
|
||||
def test_index_management_permissions_csrf_and_pagination(client, anonymous_client):
|
||||
# Same TestClient fixture is authenticated as admin here; first verify authorized validation.
|
||||
version = build(client)
|
||||
assert client.post("/api/indexes", json={}, headers={"X-CSRF-Token": ""}).status_code == 403
|
||||
assert client.get("/api/indexes?page_size=101").status_code == 422
|
||||
assert client.get(f"/api/indexes/{uuid4()}").status_code == 404
|
||||
assert client.get(f"/api/indexes/{uuid4()}/chunks").status_code == 404
|
||||
assert client.get("/api/indexes/invalid").status_code == 422
|
||||
assert client.get(f"/api/indexes/{version['id']}/chunks?page=2").json()["items"] == []
|
||||
client.cookies.clear()
|
||||
for method, path in [
|
||||
("GET", "/api/indexes"),
|
||||
("GET", "/api/indexes/active"),
|
||||
("GET", "/api/indexes/publications"),
|
||||
("POST", "/api/indexes"),
|
||||
("GET", f"/api/indexes/{version['id']}/chunks"),
|
||||
("POST", f"/api/indexes/{version['id']}/publish"),
|
||||
]:
|
||||
assert client.request(method, path, json={}).status_code == 401
|
||||
assert (
|
||||
client.post(
|
||||
"/api/auth/register", json={"username": "normal_user", "password": "test-password-123"}
|
||||
).status_code
|
||||
== 201
|
||||
)
|
||||
auth = client.post(
|
||||
"/api/auth/login", json={"username": "normal_user", "password": "test-password-123"}
|
||||
).json()
|
||||
client.headers["X-CSRF-Token"] = auth["csrf_token"]
|
||||
assert client.post("/api/indexes", json={}).status_code == 403
|
||||
assert client.get("/api/indexes/active").status_code == 403
|
||||
|
||||
|
||||
def test_publication_lock_serializes_competing_publishers(client, migrated_database):
|
||||
add_item(client)
|
||||
first, second = build(client), build(client)
|
||||
actor = first["created_by"]
|
||||
with Session(migrated_database) as owner, Session(migrated_database) as competitor:
|
||||
locked_state(owner)
|
||||
competitor.execute(text("SET LOCAL lock_timeout = '100ms'"))
|
||||
with pytest.raises(OperationalError) as blocked:
|
||||
indexing.publish_index(competitor, UUID(second["id"]), 0, actor)
|
||||
assert blocked.value.orig.sqlstate == "55P03"
|
||||
competitor.rollback()
|
||||
assert indexing.publish_index(owner, UUID(first["id"]), 0, actor).generation == 1
|
||||
with pytest.raises(HTTPException) as stale:
|
||||
indexing.publish_index(competitor, UUID(second["id"]), 0, actor)
|
||||
assert stale.value.status_code == 409
|
||||
competitor.rollback()
|
||||
assert client.get("/api/indexes/publications").json()["total"] == 1
|
||||
assert client.get("/api/indexes/active").json()["active_version"]["id"] == first["id"]
|
||||
|
||||
|
||||
def test_catalog_writes_use_the_publication_lock(client, migrated_database):
|
||||
# The guard must run before a catalog mutation, even for creation with no parent row.
|
||||
with Session(migrated_database) as owner:
|
||||
locked_state(owner)
|
||||
from app.db.session import get_db
|
||||
|
||||
factory = client.app.dependency_overrides[get_db]
|
||||
|
||||
def short_timeout():
|
||||
for db in factory():
|
||||
db.execute(text("SET LOCAL lock_timeout = '100ms'"))
|
||||
yield db
|
||||
|
||||
client.app.dependency_overrides[get_db] = short_timeout
|
||||
response = client.post("/api/knowledge", json={"title": "占位", "content": "占位"})
|
||||
assert response.status_code == 503
|
||||
client.app.dependency_overrides[get_db] = factory
|
||||
owner.rollback()
|
||||
assert client.get("/api/knowledge").json() == []
|
||||
|
||||
|
||||
def test_publication_failure_rolls_back_old_status_pointer_and_audit(client, migrated_database):
|
||||
add_item(client)
|
||||
first, second = build(client), build(client)
|
||||
publish(client, first)
|
||||
with migrated_database.begin() as c:
|
||||
c.execute(
|
||||
text(
|
||||
"ALTER TABLE index_publications ADD CONSTRAINT test_reject_publication CHECK (false) NOT VALID"
|
||||
)
|
||||
)
|
||||
assert publish(client, second, 1).status_code == 409
|
||||
assert client.get("/api/indexes/active").json()["active_version"]["id"] == first["id"]
|
||||
assert client.get(f"/api/indexes/{first['id']}").json()["status"] == "published"
|
||||
assert client.get(f"/api/indexes/{second['id']}").json()["status"] == "ready"
|
||||
assert client.get("/api/indexes/publications").json()["total"] == 1
|
||||
@@ -0,0 +1,123 @@
|
||||
import pytest
|
||||
from alembic.autogenerate import compare_metadata
|
||||
from alembic.migration import MigrationContext
|
||||
from conftest import migrate
|
||||
from fastapi.testclient import TestClient
|
||||
from sqlalchemy import text
|
||||
|
||||
from app.db.session import Base
|
||||
from app.main import create_app
|
||||
|
||||
|
||||
def test_startup_requires_migration(database):
|
||||
with pytest.raises(RuntimeError, match="alembic upgrade head"):
|
||||
with TestClient(create_app(database)):
|
||||
pass
|
||||
|
||||
|
||||
def test_startup_rejects_previous_schema(database):
|
||||
migrate(database, "0005_chunk_indexes")
|
||||
with pytest.raises(RuntimeError, match="alembic upgrade head"):
|
||||
with TestClient(create_app(database)):
|
||||
pass
|
||||
|
||||
|
||||
def test_upgrade_preserves_original_rows_and_repeated_upgrade(database):
|
||||
migrate(database, "0001_baseline")
|
||||
with database.begin() as connection:
|
||||
connection.execute(
|
||||
text("""INSERT INTO knowledge_items
|
||||
(title, category, content, source_url, verification_status)
|
||||
VALUES ('示例旧数据', '未分类', '原始正文', 'https://example.com/', 'verified')""")
|
||||
)
|
||||
# 模拟 create_all 建表后没有 Alembic 版本记录的旧工程。
|
||||
connection.execute(text("DELETE FROM alembic_version"))
|
||||
migrate(database)
|
||||
migrate(database)
|
||||
with database.connect() as connection:
|
||||
row = connection.execute(text("SELECT * FROM knowledge_items")).mappings().one()
|
||||
assert row["title"] == "示例旧数据" and row["content"] == "原始正文"
|
||||
assert row["collected_at"] is None
|
||||
assert row["verification_status"] == "pending"
|
||||
assert connection.scalar(text("SELECT count(*) FROM audit_events")) == 2
|
||||
assert connection.scalar(text("SELECT version_num FROM alembic_version")) == "0006_vectors"
|
||||
assert compare_metadata(MigrationContext.configure(connection), Base.metadata) == []
|
||||
migrate(database, "0001_baseline", downgrade=True)
|
||||
with database.connect() as connection:
|
||||
assert connection.scalar(text("SELECT content FROM knowledge_items")) == "原始正文"
|
||||
migrate(database)
|
||||
|
||||
|
||||
def test_auth_upgrade_and_downgrade_preserve_knowledge_and_retrieval(database):
|
||||
migrate(database, "0003_retrieval_runs")
|
||||
with database.begin() as c:
|
||||
c.execute(
|
||||
text("""INSERT INTO knowledge_items
|
||||
(title, category, content, verification_status, version, updated_at, is_deleted)
|
||||
VALUES ('占位', '其他', '保留的占位正文', 'pending', 1, now(), false)""")
|
||||
)
|
||||
c.execute(
|
||||
text("""INSERT INTO retrieval_runs
|
||||
(id, query, mode, corpus_fingerprint, response)
|
||||
VALUES ('00000000-0000-0000-0000-000000000001', '占位', 'keyword', 'placeholder', '{}')""")
|
||||
)
|
||||
migrate(database)
|
||||
with database.connect() as c:
|
||||
assert c.scalar(text("SELECT count(*) FROM users")) == 0
|
||||
assert c.scalar(text("SELECT count(*) FROM auth_sessions")) == 0
|
||||
migrate(database, "0003_retrieval_runs", downgrade=True)
|
||||
with database.connect() as c:
|
||||
assert c.scalar(text("SELECT content FROM knowledge_items")) == "保留的占位正文"
|
||||
assert c.scalar(text("SELECT count(*) FROM retrieval_runs")) == 1
|
||||
migrate(database)
|
||||
|
||||
|
||||
def test_chunk_index_downgrade_preserves_users_sessions_knowledge_and_retrieval(
|
||||
client, migrated_database
|
||||
):
|
||||
from test_indexing import build, publish
|
||||
from test_retrieval import add_item, search
|
||||
|
||||
item = add_item(client)
|
||||
run = search(client)
|
||||
version = build(client)
|
||||
assert publish(client, version).status_code == 200
|
||||
migrate(migrated_database, "0004_auth_sessions", downgrade=True)
|
||||
with migrated_database.connect() as c:
|
||||
assert c.scalar(text("SELECT count(*) FROM users")) == 1
|
||||
assert c.scalar(text("SELECT count(*) FROM auth_sessions")) == 1
|
||||
assert (
|
||||
c.scalar(text("SELECT content FROM knowledge_items WHERE id=:id"), {"id": item["id"]})
|
||||
== item["content"]
|
||||
)
|
||||
assert c.scalar(text("SELECT id FROM retrieval_runs")) == run["run_id"]
|
||||
migrate(migrated_database)
|
||||
assert client.get("/api/auth/me").status_code == 200
|
||||
assert client.get("/api/indexes").json()["total"] == 0
|
||||
assert client.get("/api/indexes/active").json()["generation"] == 0
|
||||
|
||||
|
||||
def test_vector_downgrade_retains_text_publications_and_prior_retrieval_runs(
|
||||
client, migrated_database
|
||||
):
|
||||
from test_retrieval import add_item
|
||||
from test_vectors import ToyProvider, text_version, vector_build, vector_publish, vector_search
|
||||
|
||||
from app.services.embedding_runtime import get_embedding_provider
|
||||
|
||||
client.app.dependency_overrides[get_embedding_provider] = ToyProvider
|
||||
add_item(client)
|
||||
text_ = text_version(client)
|
||||
version = vector_build(client, text_)
|
||||
vector_publish(client, version)
|
||||
history = vector_search(client).json()
|
||||
migrate(migrated_database, "0005_chunk_indexes", downgrade=True)
|
||||
with migrated_database.connect() as c:
|
||||
assert c.scalar(text("SELECT active_version_id FROM index_state")) == text_["id"]
|
||||
assert c.scalar(text("SELECT count(*) FROM chunks")) == 1
|
||||
assert c.scalar(text("SELECT count(*) FROM users")) == 1
|
||||
assert c.scalar(text("SELECT count(*) FROM index_publications")) == 1
|
||||
assert c.scalar(text("SELECT id FROM retrieval_runs")) == history["run_id"]
|
||||
migrate(migrated_database)
|
||||
assert client.get("/api/vector-indexes/active").json()["generation"] == 0
|
||||
assert client.get(f"/api/retrieval/runs/{history['run_id']}").json() == history
|
||||
@@ -0,0 +1,212 @@
|
||||
import math
|
||||
from collections import Counter
|
||||
from uuid import uuid4
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import text
|
||||
|
||||
from app.services import retrieval
|
||||
|
||||
|
||||
def add_item(client, *, status="verified", **fields):
|
||||
response = client.post(
|
||||
"/api/knowledge",
|
||||
json={
|
||||
"title": "检索占位资料",
|
||||
"content": "那拉提旅游资源检索测试,不代表真实文旅事实。",
|
||||
"source_url": "https://example.com/",
|
||||
"collected_at": "2026-10-15T09:00:00+08:00",
|
||||
**fields,
|
||||
},
|
||||
)
|
||||
assert response.status_code == 201, response.text
|
||||
item = response.json()
|
||||
if status != "pending":
|
||||
response = client.post(
|
||||
f"/api/knowledge/{item['id']}/review",
|
||||
json={
|
||||
"expected_version": item["version"],
|
||||
"verification_status": status,
|
||||
"reviewer": "test-reviewer",
|
||||
"verification_notes": "仅为自动化审核流程测试",
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200, response.text
|
||||
item = response.json()
|
||||
return item
|
||||
|
||||
|
||||
def search(client, query="那拉提", **fields):
|
||||
response = client.post("/api/retrieval", json={"query": query, **fields})
|
||||
assert response.status_code == 200, response.text
|
||||
return response.json()
|
||||
|
||||
|
||||
def test_bm25_matches_hand_calculated_scores_and_query_deduplication():
|
||||
counters = [Counter({"x": 1}), Counter({"x": 1, "y": 1})]
|
||||
scores = retrieval.bm25(counters, ["x"])
|
||||
idf = math.log(1 + 0.5 / 2.5)
|
||||
assert scores[0] == pytest.approx(idf * 2.2 / (1 + 1.2 * (0.25 + 0.75 / 1.5)))
|
||||
assert scores[1] == pytest.approx(idf * 2.2 / (1 + 1.2 * (0.25 + 0.75 * 2 / 1.5)))
|
||||
assert scores[0] > scores[1]
|
||||
assert retrieval.bm25(counters, ["x", "x"]) == scores
|
||||
assert retrieval.bm25([], ["x"]) == []
|
||||
assert retrieval.bm25([Counter()], ["x"]) == [0.0]
|
||||
|
||||
|
||||
def test_tokenization_and_exact_original_snippet_offsets():
|
||||
terms = retrieval.tokenize("请问那拉提 API api!")
|
||||
assert "那拉提" in terms and terms.count("api") == 2
|
||||
assert "请问" not in terms
|
||||
original = "占位前文。" * 80 + "这里有API检索说明。" + "占位后文。" * 80
|
||||
snippet, start, end = retrieval.excerpt(original, ["api"])
|
||||
assert start > 0 and "API" in snippet
|
||||
assert snippet == original[start:end] and len(snippet) <= 240
|
||||
|
||||
|
||||
def test_only_reviewed_active_knowledge_is_retrieved(client):
|
||||
good = add_item(client)
|
||||
add_item(client, status="pending")
|
||||
add_item(client, status="rejected")
|
||||
deleted = add_item(client)
|
||||
assert (
|
||||
client.delete(
|
||||
f"/api/knowledge/{deleted['id']}?expected_version={deleted['version']}"
|
||||
).status_code
|
||||
== 204
|
||||
)
|
||||
result = search(client)
|
||||
assert result["corpus_size"] == 1 and result["total_hits"] == 1
|
||||
assert result["results"][0]["knowledge_id"] == good["id"]
|
||||
assert result["results"][0]["knowledge_version"] == good["version"]
|
||||
assert result["results"][0]["source_url"] == "https://example.com/"
|
||||
assert result["elapsed_ms"] >= 0
|
||||
|
||||
|
||||
def test_tied_results_category_top_k_and_reproducibility(client):
|
||||
first = add_item(client, category="交通出行")
|
||||
second = add_item(client, category="交通出行")
|
||||
third = add_item(client, category="民宿服务")
|
||||
result = search(client, top_k=2)
|
||||
again = search(client, top_k=2)
|
||||
assert result["results"] == again["results"]
|
||||
assert result["corpus_fingerprint"] == again["corpus_fingerprint"]
|
||||
assert result["config"] == again["config"]
|
||||
assert result["run_id"] != again["run_id"]
|
||||
assert result["total_hits"] == 3
|
||||
assert [hit["knowledge_id"] for hit in result["results"]] == [first["id"], second["id"]]
|
||||
filtered = search(client, category="民宿服务")
|
||||
assert filtered["corpus_size"] == 1
|
||||
assert filtered["results"][0]["knowledge_id"] == third["id"]
|
||||
|
||||
|
||||
def test_title_terms_have_more_weight_than_body_terms(client):
|
||||
title_match = add_item(client, title="旅游", content="交通")
|
||||
body_match = add_item(client, title="交通", content="旅游")
|
||||
result = search(client, query="旅游")
|
||||
assert [hit["knowledge_id"] for hit in result["results"]] == [
|
||||
title_match["id"],
|
||||
body_match["id"],
|
||||
]
|
||||
assert result["results"][0]["score"] > result["results"][1]["score"]
|
||||
|
||||
|
||||
def test_no_evidence_and_history_are_persisted(client):
|
||||
empty = search(client)
|
||||
assert empty["status"] == "no_evidence" and empty["results"] == []
|
||||
add_item(client)
|
||||
unmatched = search(client, query="unmatcheduniquelexeme")
|
||||
assert unmatched["status"] == "no_evidence" and unmatched["corpus_size"] == 1
|
||||
assert client.get(f"/api/retrieval/runs/{empty['run_id']}").json() == empty
|
||||
history = client.get("/api/retrieval/runs?page_size=1").json()
|
||||
assert history["total"] == 2 and len(history["items"]) == 1
|
||||
assert history["items"][0] == unmatched
|
||||
assert client.get(f"/api/retrieval/runs/{uuid4()}").status_code == 404
|
||||
assert client.get("/api/retrieval/runs/invalid").status_code == 422
|
||||
assert client.get("/api/retrieval/runs?page_size=101").status_code == 422
|
||||
|
||||
|
||||
def test_source_snapshot_and_original_run_survive_invalidation(client):
|
||||
src = client.post(
|
||||
"/api/sources",
|
||||
json={
|
||||
"name": "测试来源",
|
||||
"source_url": "https://example.com/",
|
||||
},
|
||||
).json()
|
||||
content = "占位前言。" * 80 + "那拉提检索关键词。" + "占位后文。" * 80
|
||||
doc = client.post(
|
||||
"/api/documents",
|
||||
json={
|
||||
"source_id": src["id"],
|
||||
"title": "测试文档",
|
||||
"content": content,
|
||||
"collected_at": "2026-10-15T09:00:00+08:00",
|
||||
},
|
||||
).json()
|
||||
item = add_item(client, document_id=doc["id"], content=content)
|
||||
original = search(client)
|
||||
hit = original["results"][0]
|
||||
assert hit["document_id"] == doc["id"] and hit["document_version"] == 1
|
||||
assert hit["source_id"] == src["id"] and hit["source_version"] == 1
|
||||
assert hit["source_name"] == "测试来源"
|
||||
assert "那拉提" in hit["snippet"] and hit["snippet_start"] > 0
|
||||
assert hit["snippet"] == content[hit["snippet_start"] : hit["snippet_end"]]
|
||||
assert (
|
||||
client.patch(
|
||||
f"/api/sources/{src['id']}",
|
||||
json={
|
||||
"expected_version": src["version"],
|
||||
"name": "修订来源名称",
|
||||
},
|
||||
).status_code
|
||||
== 200
|
||||
)
|
||||
revised = search(client)
|
||||
assert revised["corpus_size"] == 0 and revised["results"] == []
|
||||
assert revised["corpus_fingerprint"] != original["corpus_fingerprint"]
|
||||
stored = client.get(f"/api/retrieval/runs/{original['run_id']}").json()
|
||||
assert stored == original and stored["results"][0]["knowledge_id"] == item["id"]
|
||||
|
||||
|
||||
def test_inactive_parent_is_excluded_even_if_status_was_not_reset(client, migrated_database):
|
||||
src = client.post(
|
||||
"/api/sources", json={"name": "测试", "source_url": "https://example.com/"}
|
||||
).json()
|
||||
add_item(client, source_id=src["id"])
|
||||
assert search(client)["corpus_size"] == 1
|
||||
with migrated_database.begin() as connection:
|
||||
connection.execute(
|
||||
text("UPDATE sources SET is_deleted = true WHERE id = :id"), {"id": src["id"]}
|
||||
)
|
||||
assert search(client)["corpus_size"] == 0
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"payload",
|
||||
[
|
||||
{"query": " "},
|
||||
{"query": "!!!"},
|
||||
{"query": "的了"},
|
||||
{"query": "那拉提", "top_k": 0},
|
||||
{"query": "那拉提", "top_k": 21},
|
||||
{"query": "那拉提", "mode": "vector"},
|
||||
{"query": "那拉提", "unexpected": True},
|
||||
{"query": "a" * 501},
|
||||
],
|
||||
)
|
||||
def test_invalid_query_does_not_create_run(client, payload):
|
||||
assert client.post("/api/retrieval", json=payload).status_code == 422
|
||||
assert client.get("/api/retrieval/runs").json()["total"] == 0
|
||||
|
||||
|
||||
@pytest.mark.parametrize("setting,value", [("MAX_ITEMS", 1), ("MAX_CHARACTERS", 1)])
|
||||
def test_corpus_budget_fails_explicitly_without_silent_truncation(
|
||||
client, monkeypatch, setting, value
|
||||
):
|
||||
add_item(client)
|
||||
add_item(client)
|
||||
monkeypatch.setattr(retrieval, setting, value)
|
||||
response = client.post("/api/retrieval", json={"query": "那拉提"})
|
||||
assert response.status_code == 503 and "容量" in response.json()["detail"]
|
||||
assert client.get("/api/retrieval/runs").json()["total"] == 0
|
||||
@@ -0,0 +1,193 @@
|
||||
"""Run the default application lifecycle over HTTP and verify persistence."""
|
||||
|
||||
import os
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from contextlib import contextmanager
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.models.auth import User
|
||||
from app.services.auth import PASSWORD_HASHER
|
||||
|
||||
|
||||
@contextmanager
|
||||
def server(engine, *, authenticate=True, environment_overrides=None):
|
||||
with socket.socket() as socket_:
|
||||
socket_.bind(("127.0.0.1", 0))
|
||||
port = socket_.getsockname()[1]
|
||||
environment = {
|
||||
**os.environ,
|
||||
"DATABASE_URL": engine.url.render_as_string(hide_password=False),
|
||||
"APP_ENV": "test",
|
||||
"AUTH_COOKIE_SECURE": "false",
|
||||
**(environment_overrides or {}),
|
||||
}
|
||||
if authenticate:
|
||||
with Session(engine) as db:
|
||||
if db.scalar(select(User.id).where(User.username == "http_admin")) is None:
|
||||
db.add(
|
||||
User(
|
||||
username="http_admin",
|
||||
password_hash=PASSWORD_HASHER.hash("test-password-123"),
|
||||
role="admin",
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
backend = Path(__file__).resolve().parents[1]
|
||||
process = subprocess.Popen(
|
||||
[
|
||||
sys.executable,
|
||||
"-m",
|
||||
"uvicorn",
|
||||
"app.main:app",
|
||||
"--host",
|
||||
"127.0.0.1",
|
||||
"--port",
|
||||
str(port),
|
||||
"--log-level",
|
||||
"error",
|
||||
],
|
||||
cwd=backend,
|
||||
env=environment,
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL,
|
||||
)
|
||||
try:
|
||||
with httpx.Client(base_url=f"http://127.0.0.1:{port}", timeout=2) as client:
|
||||
client.headers["Origin"] = "http://localhost:5173"
|
||||
deadline = time.monotonic() + 15
|
||||
while time.monotonic() < deadline:
|
||||
if process.poll() is not None:
|
||||
raise AssertionError("默认后端进程启动失败")
|
||||
try:
|
||||
if client.get("/api/ready").status_code == 200:
|
||||
break
|
||||
except httpx.TransportError:
|
||||
pass
|
||||
time.sleep(0.05)
|
||||
else:
|
||||
raise AssertionError("默认后端未在时限内就绪")
|
||||
if authenticate:
|
||||
login = client.post(
|
||||
"/api/auth/login",
|
||||
json={"username": "http_admin", "password": "test-password-123"},
|
||||
)
|
||||
assert login.status_code == 200, login.text
|
||||
client.headers["X-CSRF-Token"] = login.json()["csrf_token"]
|
||||
yield client
|
||||
finally:
|
||||
process.terminate()
|
||||
try:
|
||||
process.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
process.kill()
|
||||
process.wait(timeout=5)
|
||||
|
||||
|
||||
def test_default_startup_and_persistence_across_restart(migrated_database):
|
||||
with server(migrated_database) as client:
|
||||
assert client.get("/docs").status_code == 200
|
||||
response = client.post(
|
||||
"/api/knowledge",
|
||||
json={
|
||||
"title": "持久化验收占位",
|
||||
"content": "仅为自动化测试数据",
|
||||
"source_url": "",
|
||||
},
|
||||
)
|
||||
assert response.status_code == 201
|
||||
item = response.json()
|
||||
with server(migrated_database) as client:
|
||||
assert client.get(f"/api/knowledge/{item['id']}").json()["content"] == "仅为自动化测试数据"
|
||||
assert client.get("/api/knowledge").json()[0]["id"] == item["id"]
|
||||
|
||||
|
||||
def test_retrieval_run_persists_across_restart(migrated_database):
|
||||
with server(migrated_database) as client:
|
||||
item = client.post(
|
||||
"/api/knowledge",
|
||||
json={
|
||||
"title": "HTTP检索测试",
|
||||
"content": "仅用于自动化测试的那拉提关键词。",
|
||||
"source_url": "https://example.com/",
|
||||
"collected_at": "2026-10-15T09:00:00+08:00",
|
||||
},
|
||||
).json()
|
||||
assert (
|
||||
client.post(
|
||||
f"/api/knowledge/{item['id']}/review",
|
||||
json={
|
||||
"expected_version": item["version"],
|
||||
"verification_status": "verified",
|
||||
"reviewer": "test-reviewer",
|
||||
"verification_notes": "仅验证检索流程",
|
||||
},
|
||||
).status_code
|
||||
== 200
|
||||
)
|
||||
response = client.post("/api/retrieval", json={"query": "那拉提"})
|
||||
assert response.status_code == 200
|
||||
run = response.json()
|
||||
assert run["results"][0]["knowledge_id"] == item["id"]
|
||||
with server(migrated_database) as client:
|
||||
assert client.get(f"/api/retrieval/runs/{run['run_id']}").json() == run
|
||||
|
||||
|
||||
def test_session_and_logout_survive_http_restart(migrated_database):
|
||||
from app.services.auth import COOKIE_NAME
|
||||
|
||||
with server(migrated_database, authenticate=False) as client:
|
||||
payload = {"username": "http_user", "password": "test-password-123"}
|
||||
assert client.post("/api/auth/register", json=payload).status_code == 201
|
||||
response = client.post("/api/auth/login", json=payload)
|
||||
cookie = client.cookies.get(COOKIE_NAME)
|
||||
token = response.json()["csrf_token"]
|
||||
with server(migrated_database, authenticate=False) as client:
|
||||
client.cookies.set(COOKIE_NAME, cookie)
|
||||
assert client.get("/api/auth/me").json()["username"] == "http_user"
|
||||
assert client.post("/api/auth/logout", headers={"X-CSRF-Token": token}).status_code == 204
|
||||
with server(migrated_database, authenticate=False) as client:
|
||||
client.cookies.set(COOKIE_NAME, cookie)
|
||||
assert client.get("/api/auth/me").status_code == 401
|
||||
|
||||
|
||||
def test_published_index_and_chunks_survive_http_restart(migrated_database):
|
||||
with server(migrated_database) as client:
|
||||
item = client.post(
|
||||
"/api/knowledge",
|
||||
json={
|
||||
"title": "切分持久化占位",
|
||||
"content": "那拉提测试正文。" * 50,
|
||||
"source_url": "https://example.com/",
|
||||
"collected_at": "2026-10-16T00:00:00Z",
|
||||
},
|
||||
).json()
|
||||
assert (
|
||||
client.post(
|
||||
f"/api/knowledge/{item['id']}/review",
|
||||
json={
|
||||
"expected_version": item["version"],
|
||||
"verification_status": "verified",
|
||||
"reviewer": "test-reviewer",
|
||||
"verification_notes": "仅为切分持久化测试",
|
||||
},
|
||||
).status_code
|
||||
== 200
|
||||
)
|
||||
built = client.post("/api/indexes", json={"chunk_size": 100, "overlap": 20}).json()
|
||||
response = client.post(
|
||||
f"/api/indexes/{built['id']}/publish", json={"expected_generation": 0}
|
||||
)
|
||||
assert response.status_code == 200
|
||||
active = response.json()
|
||||
chunks = client.get(f"/api/indexes/{built['id']}/chunks").json()
|
||||
with server(migrated_database) as client:
|
||||
assert client.get("/api/indexes/active").json() == active
|
||||
assert client.get(f"/api/indexes/{built['id']}/chunks").json() == chunks
|
||||
assert client.get("/api/indexes/publications").json()["total"] == 1
|
||||
@@ -0,0 +1,115 @@
|
||||
"""Explicit opt-in: actual pinned model -> HTTP -> PostgreSQL -> HTTP process restart."""
|
||||
|
||||
import os
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import text
|
||||
from test_runtime import server
|
||||
|
||||
from app.services.embedding import SMALL_ZH
|
||||
|
||||
|
||||
@pytest.mark.skipif(
|
||||
os.environ.get("RUN_LOCAL_EMBEDDING_TEST") != "1",
|
||||
reason="requires optional CPU runtime and offline pinned model cache",
|
||||
)
|
||||
def test_real_local_model_vectors_retrieval_and_http_restart(migrated_database):
|
||||
options = {
|
||||
"EMBEDDING_ENABLED": "true",
|
||||
"EMBEDDING_MODEL": "small",
|
||||
"EMBEDDING_CACHE_DIR": os.environ["EMBEDDING_CACHE_DIR"],
|
||||
"HF_HUB_OFFLINE": "1",
|
||||
"HF_HUB_DISABLE_TELEMETRY": "1",
|
||||
"HF_HUB_DISABLE_IMPLICIT_TOKEN": "1",
|
||||
"TOKENIZERS_PARALLELISM": "false",
|
||||
}
|
||||
query = {"query": "合成测试中交通安排应查阅哪份资料?", "top_k": 3}
|
||||
with server(migrated_database, environment_overrides=options) as client:
|
||||
ids = []
|
||||
for topic, body in [
|
||||
("交通", "【合成测试】占位交通资料介绍测试公交线路与班次的查询入口。"),
|
||||
("住宿", "【合成测试】占位住宿资料介绍测试房型与入住须知。"),
|
||||
("审核", "【合成测试】占位审核资料说明测试出处核验流程。"),
|
||||
]:
|
||||
item = client.post(
|
||||
"/api/knowledge",
|
||||
json={
|
||||
"title": f"{topic}合成测试",
|
||||
"content": body,
|
||||
"source_url": "https://example.com/",
|
||||
"collected_at": "2026-10-16T00:00:00Z",
|
||||
},
|
||||
).json()
|
||||
response = client.post(
|
||||
f"/api/knowledge/{item['id']}/review",
|
||||
json={
|
||||
"expected_version": item["version"],
|
||||
"verification_status": "verified",
|
||||
"reviewer": "test-reviewer",
|
||||
"verification_notes": "仅为模型/HTTP/数据库持久化测试,不是事实资料",
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
ids.append(item["id"])
|
||||
text_ = client.post("/api/indexes", json={"chunk_size": 200, "overlap": 20}).json()
|
||||
assert (
|
||||
client.post(
|
||||
f"/api/indexes/{text_['id']}/publish", json={"expected_generation": 0}
|
||||
).status_code
|
||||
== 200
|
||||
)
|
||||
built = client.post(
|
||||
"/api/vector-indexes", json={"text_index_version_id": text_["id"]}, timeout=60
|
||||
)
|
||||
assert built.status_code == 201, built.text
|
||||
version = built.json()
|
||||
assert version["space_id"] == SMALL_ZH.space_id and version["dimension"] == 512
|
||||
published = client.post(
|
||||
f"/api/vector-indexes/{version['id']}/publish", json={"expected_generation": 0}
|
||||
)
|
||||
assert published.status_code == 200
|
||||
active = published.json()
|
||||
response = client.post("/api/retrieval/vector", json=query, timeout=60)
|
||||
assert response.status_code == 200, response.text
|
||||
first = response.json()
|
||||
assert {h["knowledge_id"] for h in first["results"]} == set(ids)
|
||||
assert all(-1 <= h["score"] <= 1 for h in first["results"])
|
||||
assert first["config"]["space"]["spec"]["revision"] == SMALL_ZH.revision
|
||||
response = client.post(
|
||||
"/api/retrieval/hybrid", json={**query, "candidate_k": 3}, timeout=60
|
||||
)
|
||||
assert response.status_code == 200, response.text
|
||||
hybrid = response.json()
|
||||
assert hybrid["vector_version_id"] == version["id"]
|
||||
assert hybrid["text_index_version_id"] == text_["id"]
|
||||
assert hybrid["config"]["space"]["spec"]["revision"] == SMALL_ZH.revision
|
||||
assert {h["knowledge_id"] for h in hybrid["results"]} == set(ids)
|
||||
assert hybrid["vector_candidates"]
|
||||
keyword = client.post("/api/retrieval/chunks", json=query).json()
|
||||
assert keyword["mode"] == "keyword_chunks" and keyword["results"]
|
||||
with migrated_database.connect() as c:
|
||||
assert (
|
||||
c.scalar(
|
||||
text("SELECT count(*) FROM chunk_vectors WHERE public.vector_dims(embedding)=512")
|
||||
)
|
||||
== 3
|
||||
)
|
||||
assert c.scalar(text("SELECT count(*) FROM vector_publications")) == 1
|
||||
with server(migrated_database, environment_overrides=options) as client:
|
||||
assert client.get("/api/vector-indexes/active").json() == active
|
||||
assert client.get(f"/api/retrieval/runs/{first['run_id']}").json() == first
|
||||
response = client.post("/api/retrieval/vector", json=query, timeout=60)
|
||||
assert response.status_code == 200, response.text
|
||||
assert response.json()["results"] == first["results"]
|
||||
for result, endpoint in [(hybrid, "hybrid"), (keyword, "chunks")]:
|
||||
assert client.get(f"/api/retrieval/runs/{result['run_id']}").json() == result
|
||||
response = client.post(
|
||||
f"/api/retrieval/{endpoint}",
|
||||
json={**query, **({"candidate_k": 3} if endpoint == "hybrid" else {})},
|
||||
timeout=60,
|
||||
)
|
||||
assert response.status_code == 200, response.text
|
||||
assert response.json()["results"] == result["results"]
|
||||
if endpoint == "hybrid":
|
||||
assert response.json()["lexical_candidates"] == result["lexical_candidates"]
|
||||
assert response.json()["vector_candidates"] == result["vector_candidates"]
|
||||
@@ -0,0 +1,552 @@
|
||||
"""Toy vectors test database mathematics and isolation, never model/domain quality."""
|
||||
|
||||
from dataclasses import replace
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
import pytest
|
||||
from fastapi import HTTPException
|
||||
from sqlalchemy import func, select, text, update
|
||||
from sqlalchemy.exc import DBAPIError, OperationalError
|
||||
from sqlalchemy.orm import Session
|
||||
from test_indexing import build, publish
|
||||
from test_retrieval import add_item
|
||||
|
||||
from app.models.indexing import Chunk
|
||||
from app.models.retrieval import RetrievalRun
|
||||
from app.models.vectors import ChunkVector, EmbeddingSpace, VectorPublication, VectorVersion
|
||||
from app.services import vectors
|
||||
from app.services.embedding import EmbeddingError, EmbeddingSpec
|
||||
from app.services.embedding_runtime import EmbeddingRuntime, get_embedding_provider
|
||||
from app.services.indexing import locked_state, publish_index
|
||||
|
||||
|
||||
class ToyProvider:
|
||||
spec = EmbeddingSpec("test-only-coordinate-fixture", "a" * 40, 3)
|
||||
callback = None
|
||||
|
||||
def encode(self, texts, *, query=False):
|
||||
if self.callback:
|
||||
self.callback(query)
|
||||
return [[1.0, 0.0, 0.0] if query or "alpha" in t else [0.0, 1.0, 0.0] for t in texts]
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def provider(client):
|
||||
value = ToyProvider()
|
||||
client.app.dependency_overrides[get_embedding_provider] = lambda: value
|
||||
return value
|
||||
|
||||
|
||||
def text_version(client, **kwargs):
|
||||
version = build(client, chunk_size=100, overlap=0, **kwargs)
|
||||
generation = client.get("/api/indexes/active").json()["generation"]
|
||||
assert publish(client, version, generation).status_code == 200
|
||||
return version
|
||||
|
||||
|
||||
def vector_build(client, version):
|
||||
response = client.post("/api/vector-indexes", json={"text_index_version_id": version["id"]})
|
||||
assert response.status_code == 201, response.text
|
||||
return response.json()
|
||||
|
||||
|
||||
def vector_publish(client, version, generation=0):
|
||||
return client.post(
|
||||
f"/api/vector-indexes/{version['id']}/publish", json={"expected_generation": generation}
|
||||
)
|
||||
|
||||
|
||||
def vector_search(client, **kwargs):
|
||||
return client.post("/api/retrieval/vector", json={"query": "alpha query", **kwargs})
|
||||
|
||||
|
||||
def test_exact_cosine_original_chunks_category_and_mixed_history(client, provider):
|
||||
first = add_item(client, content="alpha 占位正文", category="交通")
|
||||
second = add_item(client, content="beta 占位正文", category="其他")
|
||||
third = add_item(client, content="gamma 占位正文", category="其他")
|
||||
add_item(client, status="pending", content="alpha 待审核不应进入")
|
||||
text_ = text_version(client)
|
||||
version = vector_build(client, text_)
|
||||
assert (
|
||||
version["status"] == "ready" and version["dimension"] == 3 and version["chunk_count"] == 3
|
||||
)
|
||||
assert vector_search(client).status_code == 409
|
||||
assert vector_publish(client, version).json()["is_current"]
|
||||
result = vector_search(client).json()
|
||||
assert result["mode"] == "vector" and result["total_hits"] == 3
|
||||
assert [h["knowledge_id"] for h in result["results"]] == [
|
||||
first["id"],
|
||||
second["id"],
|
||||
third["id"],
|
||||
]
|
||||
assert [h["score"] for h in result["results"]] == [1.0, 0.0, 0.0]
|
||||
assert result["config"]["space"]["id"] == provider.spec.space_id
|
||||
assert result["text_index_version_id"] == text_["id"]
|
||||
assert result["vector_version_id"] == version["id"]
|
||||
assert result["corpus_fingerprint"] == text_["corpus_fingerprint"]
|
||||
assert result["elapsed_ms"] > 0
|
||||
assert result["results"][0]["snippet"] == first["content"]
|
||||
assert result["results"][0]["snippet_start"] == 0
|
||||
assert result["results"][0]["snippet_end"] == len(first["content"])
|
||||
assert result["results"][0]["source_url"] == "https://example.com/"
|
||||
filtered = vector_search(client, category="其他", top_k=1).json()
|
||||
assert filtered["corpus_size"] == 2 and filtered["results"][0]["knowledge_id"] == second["id"]
|
||||
empty = vector_search(client, category="不存在分类").json()
|
||||
assert empty["status"] == "no_evidence" and empty["results"] == []
|
||||
keyword = client.post("/api/retrieval", json={"query": "alpha"}).json()
|
||||
history = client.get("/api/retrieval/runs?page_size=100").json()
|
||||
assert {r["mode"] for r in history["items"]} == {"keyword", "vector"}
|
||||
assert client.get(f"/api/retrieval/runs/{result['run_id']}").json() == result
|
||||
assert client.get(f"/api/retrieval/runs/{keyword['run_id']}").json() == keyword
|
||||
client.cookies.clear()
|
||||
assert vector_search(client).status_code == 200
|
||||
assert client.get("/api/retrieval/runs").status_code == 401
|
||||
|
||||
|
||||
def test_multiple_chunks_keep_original_offsets_and_order(client, provider):
|
||||
item = add_item(client, content="alpha 占位正文。" * 45)
|
||||
version = vector_build(client, text_version(client))
|
||||
vector_publish(client, version)
|
||||
hits = vector_search(client, top_k=20).json()["results"]
|
||||
assert len(hits) > 1
|
||||
assert [h["chunk_ordinal"] for h in hits] == list(range(len(hits)))
|
||||
assert len({h["chunk_id"] for h in hits}) == len(hits)
|
||||
for h in hits:
|
||||
assert h["snippet"] == item["content"][h["snippet_start"] : h["snippet_end"]]
|
||||
|
||||
|
||||
def test_same_dimension_different_revision_is_a_separate_space(client, provider, migrated_database):
|
||||
add_item(client)
|
||||
text_ = text_version(client)
|
||||
first = vector_build(client, text_)
|
||||
vector_publish(client, first)
|
||||
old_space = provider.spec.space_id
|
||||
provider.spec = replace(provider.spec, revision="b" * 40)
|
||||
assert vector_search(client).status_code == 409
|
||||
second = vector_build(client, text_)
|
||||
assert second["space_id"] != old_space and second["dimension"] == first["dimension"]
|
||||
assert vector_publish(client, second, 1).status_code == 200
|
||||
assert vector_search(client).json()["config"]["space"]["id"] == provider.spec.space_id
|
||||
assert client.get("/api/vector-indexes/spaces").json()["total"] == 2
|
||||
with Session(migrated_database) as db:
|
||||
assert db.scalar(select(func.count()).select_from(EmbeddingSpace)) == 2
|
||||
assert db.scalar(select(func.count()).select_from(ChunkVector)) == 2
|
||||
|
||||
|
||||
@pytest.mark.parametrize("bad", ["dimension", "zero", "nan", "count", "overflow", "unavailable"])
|
||||
def test_invalid_or_failed_encode_never_persists_partial_vectors(
|
||||
client, provider, migrated_database, bad
|
||||
):
|
||||
add_item(client, content="alpha 占位")
|
||||
text_ = text_version(client)
|
||||
original = vector_build(client, text_)
|
||||
vector_publish(client, original)
|
||||
before = client.get("/api/vector-indexes/active").json()
|
||||
|
||||
def fail(texts, **kwargs):
|
||||
if bad == "overflow":
|
||||
raise EmbeddingError("token limit")
|
||||
if bad == "unavailable":
|
||||
raise OSError("private-path-must-not-appear")
|
||||
return {
|
||||
"dimension": [[1, 0]],
|
||||
"zero": [[0, 0, 0]],
|
||||
"nan": [[float("nan"), 0, 0]],
|
||||
"count": [],
|
||||
}[bad]
|
||||
|
||||
provider.encode = fail
|
||||
response = client.post("/api/vector-indexes", json={"text_index_version_id": text_["id"]})
|
||||
assert response.status_code == (503 if bad == "unavailable" else 422)
|
||||
assert "private-path" not in response.text
|
||||
assert client.get("/api/vector-indexes/active").json() == before
|
||||
with Session(migrated_database) as db:
|
||||
assert db.scalar(select(func.count()).select_from(VectorVersion)) == 1
|
||||
assert db.scalar(select(func.count()).select_from(ChunkVector)) == 1
|
||||
|
||||
|
||||
def test_build_and_query_encode_release_locks_but_recheck_catalog(
|
||||
client, provider, migrated_database
|
||||
):
|
||||
item = add_item(client, content="alpha 占位")
|
||||
text_ = text_version(client)
|
||||
original = vector_build(client, text_)
|
||||
vector_publish(client, original)
|
||||
|
||||
def mutate(query):
|
||||
with Session(migrated_database) as other:
|
||||
other.execute(text("SET LOCAL lock_timeout = '100ms'"))
|
||||
state = locked_state(other) # Must succeed during both document and query inference.
|
||||
state.catalog_revision += 1
|
||||
other.execute(
|
||||
text("UPDATE knowledge_items SET content=:value WHERE id=:id"),
|
||||
{"value": "修订占位", "id": item["id"]},
|
||||
)
|
||||
other.commit()
|
||||
|
||||
provider.callback = mutate
|
||||
response = client.post("/api/vector-indexes", json={"text_index_version_id": text_["id"]})
|
||||
assert response.status_code == 409
|
||||
assert not client.get("/api/vector-indexes/active").json()["is_current"]
|
||||
assert vector_search(client).status_code == 409
|
||||
# Restore an active text/vector snapshot, then race during query encoding too.
|
||||
provider.callback = None
|
||||
text_ = text_version(client)
|
||||
replacement = vector_build(client, text_)
|
||||
vector_publish(client, replacement, 1)
|
||||
provider.callback = mutate
|
||||
assert vector_search(client).status_code == 409
|
||||
with Session(migrated_database) as db:
|
||||
assert db.scalar(select(func.count()).select_from(VectorVersion)) == 2
|
||||
assert db.scalar(select(func.count()).select_from(RetrievalRun)) == 0
|
||||
|
||||
|
||||
def test_query_rejects_publication_changed_during_encode(client, provider, migrated_database):
|
||||
add_item(client)
|
||||
text_ = text_version(client)
|
||||
first, second = vector_build(client, text_), vector_build(client, text_)
|
||||
vector_publish(client, first)
|
||||
|
||||
def switch(query):
|
||||
if query:
|
||||
with Session(migrated_database) as other:
|
||||
vectors.publish_vectors(other, UUID(second["id"]), 1, second["created_by"])
|
||||
|
||||
provider.callback = switch
|
||||
assert vector_search(client).status_code == 409
|
||||
assert client.get("/api/vector-indexes/active").json()["active_version"]["id"] == second["id"]
|
||||
|
||||
|
||||
def test_text_switch_during_build_discards_new_vectors(client, provider, migrated_database):
|
||||
add_item(client)
|
||||
first = text_version(client)
|
||||
replacement = build(client)
|
||||
|
||||
def switch(query):
|
||||
with Session(migrated_database) as other:
|
||||
publish_index(other, UUID(replacement["id"]), 1, replacement["created_by"])
|
||||
|
||||
provider.callback = switch
|
||||
assert (
|
||||
client.post("/api/vector-indexes", json={"text_index_version_id": first["id"]}).status_code
|
||||
== 409
|
||||
)
|
||||
assert client.get("/api/vector-indexes").json()["total"] == 0
|
||||
|
||||
|
||||
def test_out_of_band_evidence_change_fails_closed(client, provider, migrated_database):
|
||||
item = add_item(client)
|
||||
text_ = text_version(client)
|
||||
version = vector_build(client, text_)
|
||||
vector_publish(client, version)
|
||||
with migrated_database.begin() as c:
|
||||
c.execute(
|
||||
text("UPDATE knowledge_items SET content='SQL 更新占位' WHERE id=:id"),
|
||||
{"id": item["id"]},
|
||||
)
|
||||
assert not client.get("/api/vector-indexes/active").json()["is_current"]
|
||||
assert vector_search(client).status_code == 409
|
||||
assert vector_publish(client, version, 1).status_code == 409
|
||||
|
||||
|
||||
def test_optimistic_publication_audit_and_rollback(client, provider, migrated_database):
|
||||
add_item(client)
|
||||
text_ = text_version(client)
|
||||
first, second = vector_build(client, text_), vector_build(client, text_)
|
||||
assert vector_publish(client, first).json()["generation"] == 1
|
||||
assert vector_publish(client, second).status_code == 409
|
||||
assert vector_publish(client, first, 1).json()["generation"] == 1
|
||||
with migrated_database.begin() as c:
|
||||
c.execute(
|
||||
text(
|
||||
"ALTER TABLE vector_publications ADD CONSTRAINT test_reject_publication CHECK (false) NOT VALID"
|
||||
)
|
||||
)
|
||||
assert vector_publish(client, second, 1).status_code == 409
|
||||
assert client.get("/api/vector-indexes/active").json()["active_version"]["id"] == first["id"]
|
||||
assert client.get(f"/api/vector-indexes/{first['id']}").json()["status"] == "published"
|
||||
assert client.get(f"/api/vector-indexes/{second['id']}").json()["status"] == "ready"
|
||||
assert client.get("/api/vector-indexes/publications").json()["total"] == 1
|
||||
with migrated_database.begin() as c:
|
||||
c.execute(text("ALTER TABLE vector_publications DROP CONSTRAINT test_reject_publication"))
|
||||
assert vector_publish(client, second, 1).json()["generation"] == 2
|
||||
assert client.get(f"/api/vector-indexes/{first['id']}").json()["status"] == "superseded"
|
||||
|
||||
|
||||
def test_vector_insert_failure_rolls_back_entire_build(client, provider, migrated_database):
|
||||
add_item(client)
|
||||
text_ = text_version(client)
|
||||
with migrated_database.begin() as c:
|
||||
c.execute(
|
||||
text(
|
||||
"ALTER TABLE chunk_vectors ADD CONSTRAINT test_reject_vector CHECK (false) NOT VALID"
|
||||
)
|
||||
)
|
||||
assert (
|
||||
client.post("/api/vector-indexes", json={"text_index_version_id": text_["id"]}).status_code
|
||||
== 409
|
||||
)
|
||||
with Session(migrated_database) as db:
|
||||
assert db.scalar(select(func.count()).select_from(VectorVersion)) == 0
|
||||
assert db.scalar(select(func.count()).select_from(ChunkVector)) == 0
|
||||
assert db.scalar(select(func.count()).select_from(EmbeddingSpace)) == 0
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"column,value",
|
||||
[
|
||||
("dimension", 2),
|
||||
("space_id", "0" * 64),
|
||||
("text_index_version_id", str(uuid4())),
|
||||
("embedding", [1, 0]),
|
||||
("embedding", [2, 0, 0]),
|
||||
("embedding", [float("nan"), 0, 0]),
|
||||
],
|
||||
)
|
||||
def test_postgresql_rejects_cross_scope_dimensions_and_nonunit_values(
|
||||
client, provider, migrated_database, column, value
|
||||
):
|
||||
add_item(client)
|
||||
version = vector_build(client, text_version(client))
|
||||
with Session(migrated_database) as db:
|
||||
with pytest.raises(DBAPIError):
|
||||
db.execute(
|
||||
update(ChunkVector)
|
||||
.where(ChunkVector.version_id == version["id"])
|
||||
.values(**{column: value})
|
||||
)
|
||||
db.commit()
|
||||
db.rollback()
|
||||
row = db.scalar(select(ChunkVector))
|
||||
assert row.space_id == provider.spec.space_id and row.dimension == 3
|
||||
|
||||
|
||||
def test_same_width_cross_space_and_cross_text_chunks_are_rejected_by_database(
|
||||
client, provider, migrated_database
|
||||
):
|
||||
add_item(client)
|
||||
original_text = text_version(client)
|
||||
first = vector_build(client, original_text)
|
||||
provider.spec = replace(provider.spec, revision="c" * 40)
|
||||
second = vector_build(client, original_text)
|
||||
newer_text = text_version(client)
|
||||
with Session(migrated_database) as db:
|
||||
other_chunk = db.scalar(select(Chunk.id).where(Chunk.index_version_id == newer_text["id"]))
|
||||
for fields in [{"space_id": second["space_id"]}, {"chunk_id": other_chunk}]:
|
||||
with pytest.raises(DBAPIError):
|
||||
db.execute(
|
||||
update(ChunkVector)
|
||||
.where(ChunkVector.version_id == first["id"])
|
||||
.values(**fields)
|
||||
)
|
||||
db.commit()
|
||||
db.rollback()
|
||||
|
||||
|
||||
def test_incomplete_vector_set_cannot_be_published_or_queried(client, provider, migrated_database):
|
||||
add_item(client)
|
||||
version = vector_build(client, text_version(client))
|
||||
vector_publish(client, version)
|
||||
with migrated_database.begin() as c:
|
||||
c.execute(text("DELETE FROM chunk_vectors"))
|
||||
assert vector_publish(client, version, 1).status_code == 409
|
||||
assert vector_search(client).status_code == 409
|
||||
|
||||
|
||||
def test_only_current_text_and_bounded_complete_snapshots_can_build(
|
||||
client, provider, migrated_database, monkeypatch
|
||||
):
|
||||
assert vector_search(client).status_code == 409
|
||||
add_item(client)
|
||||
unpublished = build(client)
|
||||
assert (
|
||||
client.post(
|
||||
"/api/vector-indexes", json={"text_index_version_id": unpublished["id"]}
|
||||
).status_code
|
||||
== 409
|
||||
)
|
||||
text_ = text_version(client)
|
||||
assert (
|
||||
client.post(
|
||||
"/api/vector-indexes", json={"text_index_version_id": unpublished["id"]}
|
||||
).status_code
|
||||
== 409
|
||||
)
|
||||
monkeypatch.setattr(vectors, "MAX_VECTOR_CHUNKS", 0)
|
||||
assert (
|
||||
client.post("/api/vector-indexes", json={"text_index_version_id": text_["id"]}).status_code
|
||||
== 503
|
||||
)
|
||||
monkeypatch.undo()
|
||||
with migrated_database.begin() as c:
|
||||
c.execute(text("DELETE FROM chunks"))
|
||||
assert (
|
||||
client.post("/api/vector-indexes", json={"text_index_version_id": text_["id"]}).status_code
|
||||
== 409
|
||||
)
|
||||
|
||||
|
||||
def test_empty_reviewed_corpus_can_be_published_without_fabricated_results(client, provider):
|
||||
version = vector_build(client, text_version(client))
|
||||
assert version["chunk_count"] == 0
|
||||
assert vector_publish(client, version).json()["is_current"]
|
||||
result = vector_search(client).json()
|
||||
assert (
|
||||
result["status"] == "no_evidence" and result["results"] == [] and result["corpus_size"] == 0
|
||||
)
|
||||
|
||||
|
||||
def test_metadata_and_build_require_admin_and_csrf(client, provider):
|
||||
add_item(client)
|
||||
version = vector_build(client, text_version(client))
|
||||
assert (
|
||||
client.post(
|
||||
"/api/vector-indexes",
|
||||
json={"text_index_version_id": version["text_index_version_id"]},
|
||||
headers={"X-CSRF-Token": ""},
|
||||
).status_code
|
||||
== 403
|
||||
)
|
||||
client.cookies.clear()
|
||||
for method, path in [
|
||||
("GET", "/api/vector-indexes"),
|
||||
("GET", "/api/vector-indexes/active"),
|
||||
("GET", "/api/vector-indexes/spaces"),
|
||||
("GET", "/api/vector-indexes/publications"),
|
||||
("GET", f"/api/vector-indexes/{version['id']}"),
|
||||
("POST", "/api/vector-indexes"),
|
||||
("POST", f"/api/vector-indexes/{version['id']}/publish"),
|
||||
]:
|
||||
assert client.request(method, path, json={}).status_code == 401
|
||||
assert (
|
||||
client.post(
|
||||
"/api/auth/register", json={"username": "vector_user", "password": "test-password-123"}
|
||||
).status_code
|
||||
== 201
|
||||
)
|
||||
auth = client.post(
|
||||
"/api/auth/login", json={"username": "vector_user", "password": "test-password-123"}
|
||||
).json()
|
||||
client.headers["X-CSRF-Token"] = auth["csrf_token"]
|
||||
assert client.get("/api/vector-indexes").status_code == 403
|
||||
assert client.post("/api/vector-indexes", json={}).status_code == 403
|
||||
|
||||
|
||||
def test_vector_publication_serializes_with_catalog_and_other_publishers(
|
||||
client, provider, migrated_database
|
||||
):
|
||||
add_item(client)
|
||||
text_ = text_version(client)
|
||||
first, second = vector_build(client, text_), vector_build(client, text_)
|
||||
with Session(migrated_database) as owner, Session(migrated_database) as other:
|
||||
locked_state(owner)
|
||||
other.execute(text("SET LOCAL lock_timeout = '100ms'"))
|
||||
with pytest.raises(OperationalError) as blocked:
|
||||
vectors.publish_vectors(other, UUID(first["id"]), 0, first["created_by"])
|
||||
assert blocked.value.orig.sqlstate == "55P03"
|
||||
other.rollback()
|
||||
vectors.publish_vectors(owner, UUID(first["id"]), 0, first["created_by"])
|
||||
with pytest.raises(HTTPException) as stale:
|
||||
vectors.publish_vectors(other, UUID(second["id"]), 0, second["created_by"])
|
||||
assert stale.value.status_code == 409
|
||||
other.rollback()
|
||||
with Session(migrated_database) as db:
|
||||
assert db.scalar(select(func.count()).select_from(VectorPublication)) == 1
|
||||
|
||||
|
||||
def test_lazy_runtime_is_offline_and_does_not_wait_when_busy(monkeypatch):
|
||||
from app.services import embedding_runtime
|
||||
|
||||
calls = []
|
||||
|
||||
def load(spec, **kwargs):
|
||||
calls.append(kwargs)
|
||||
return ToyProvider()
|
||||
|
||||
monkeypatch.setattr(embedding_runtime, "LocalEmbeddingProvider", load)
|
||||
runtime = EmbeddingRuntime(ToyProvider.spec, None, 8, 2)
|
||||
assert calls == []
|
||||
assert runtime.encode(["alpha"]) == [[1, 0, 0]]
|
||||
assert calls[0]["offline"] is True
|
||||
runtime.encode(["alpha"], query=True)
|
||||
assert len(calls) == 1
|
||||
with runtime._lock:
|
||||
with pytest.raises(HTTPException) as busy:
|
||||
runtime.encode(["alpha"])
|
||||
assert busy.value.status_code == 503
|
||||
|
||||
|
||||
def test_disabled_runtime_is_explicit_and_keyword_baseline_remains_available(client):
|
||||
assert vector_search(client).status_code == 503
|
||||
assert client.post("/api/retrieval", json={"query": "那拉提"}).status_code == 200
|
||||
|
||||
|
||||
def test_different_width_spaces_coexist_and_queries_never_compare_across_widths(client, provider):
|
||||
add_item(client)
|
||||
text_ = text_version(client)
|
||||
first = vector_build(client, text_)
|
||||
vector_publish(client, first)
|
||||
provider.spec = replace(provider.spec, dimension=2)
|
||||
provider.encode = lambda texts, **kwargs: [[1, 0] for _ in texts]
|
||||
second = vector_build(client, text_)
|
||||
assert vector_publish(client, second, 1).status_code == 200
|
||||
result = vector_search(client)
|
||||
assert result.status_code == 200, result.text
|
||||
assert result.json()["total_hits"] == 1 and result.json()["results"][0]["score"] == 1
|
||||
assert client.get("/api/vector-indexes/spaces").json()["total"] == 2
|
||||
|
||||
|
||||
def test_new_text_publication_requires_corresponding_vectors_and_keeps_history(client, provider):
|
||||
add_item(client)
|
||||
old_text = text_version(client)
|
||||
old = vector_build(client, old_text)
|
||||
vector_publish(client, old)
|
||||
history = vector_search(client).json()
|
||||
new_text = text_version(client)
|
||||
active = client.get("/api/vector-indexes/active").json()
|
||||
assert not active["is_current"] and active["active_version"]["id"] == old["id"]
|
||||
assert vector_search(client).status_code == 409
|
||||
assert vector_publish(client, old, 1).status_code == 409
|
||||
replacement = vector_build(client, new_text)
|
||||
assert vector_publish(client, replacement, 1).json()["is_current"]
|
||||
assert client.get(f"/api/retrieval/runs/{history['run_id']}").json() == history
|
||||
assert client.get("/api/vector-indexes/publications").json()["total"] == 2
|
||||
|
||||
|
||||
def test_snapshot_text_corruption_cannot_be_encoded_published_or_served(
|
||||
client, provider, migrated_database
|
||||
):
|
||||
add_item(client)
|
||||
text_ = text_version(client)
|
||||
version = vector_build(client, text_)
|
||||
vector_publish(client, version)
|
||||
with migrated_database.begin() as c:
|
||||
c.execute(text("UPDATE chunks SET text='篡改占位'"))
|
||||
assert not client.get("/api/vector-indexes/active").json()["is_current"]
|
||||
assert (
|
||||
client.post("/api/vector-indexes", json={"text_index_version_id": text_["id"]}).status_code
|
||||
== 409
|
||||
)
|
||||
assert vector_publish(client, version, 1).status_code == 409
|
||||
assert vector_search(client).status_code == 409
|
||||
|
||||
|
||||
def test_request_shape_pagination_and_missing_vector_version(client, provider):
|
||||
assert (
|
||||
client.post(
|
||||
"/api/vector-indexes", json={"text_index_version_id": str(uuid4()), "embedding": [1, 0]}
|
||||
).status_code
|
||||
== 422
|
||||
)
|
||||
assert vector_search(client, top_k=21).status_code == 422
|
||||
assert vector_search(client, mode="keyword").status_code == 422
|
||||
assert vector_search(client, query=" ").status_code == 422
|
||||
assert client.get("/api/vector-indexes?page=0").status_code == 422
|
||||
assert client.get(f"/api/vector-indexes/{uuid4()}").status_code == 404
|
||||
assert client.get("/api/vector-indexes/spaces").json()["total"] == 0
|
||||
|
||||
|
||||
def test_blank_optional_cache_path_is_not_the_working_directory():
|
||||
from app.core.config import Settings
|
||||
|
||||
assert Settings(embedding_cache_dir="", _env_file=None).embedding_cache_dir is None
|
||||
+38
-2
@@ -14,7 +14,43 @@ FastAPI + Pydantic
|
||||
|-- 来源获取 -> 人工核验 -> 文本切分 -> 生成向量 -> 发布版本
|
||||
```
|
||||
|
||||
当前工程仅实现健康检查、知识条目新增/列表及问答占位接口;pgvector 扩展已在启动时启用,但未实现向量字段、索引和 RAG。前端来源管理与实验页为静态原型。
|
||||
当前后端实现健康/数据库就绪检查、来源与文档管理、知识增删改查、分页筛选、人工审核、历史快照、认证会话/角色权限、文本/向量快照及独立关键词/向量/RRF 检索;问答仍为占位接口。数据库由 Alembic 显式迁移并启用 pgvector,DeepSeek RAG 待明确恢复后实现。前端已接入登录/注册、个人中心、账户设置、会话恢复、路由守卫和角色菜单;实验页不伪造数据,文档、审核、完整分页、浏览器 Cookie 与 PostgreSQL 联调仍需验收。
|
||||
|
||||
## 阶段 2 数据边界
|
||||
|
||||
- `sources` 登记来源名称、HTTP(S) 网址和备注;`documents` 登记所属来源、正文、分类与采集时间。
|
||||
- `knowledge_items` 保留基础版本全部字段,新增可选来源/文档引用、采集时间、人工审核字段、版本和软删除状态。历史条目不强行补造来源或时间。
|
||||
- `audit_events` 在同一事务中保存创建、修改、审核、失效、删除和迁移导入事件;知识快照包含当时关联的来源与文档。
|
||||
- `version` 用于内容并发控制,PATCH/审核/DELETE 必须匹配读取版本;ORM 版本检查处理并发写入冲突,引用操作对来源/文档加行锁,避免与修改和删除交错。
|
||||
- 修改知识或关联依据会撤销当前审核结论。未来正式检索必须同时限定 `verification_status = 'verified'` 和 `is_deleted = false`;当前管理列表不承担检索或索引发布功能。
|
||||
- 文档版本通过递增版本号及完整历史快照留存;`chunks`、Embedding 模型版本与索引发布版本在检索阶段单独实现,不能把记录版本当作已实现索引切换。
|
||||
- 旧库首次升级自动接管基础表,保留条目与导入快照。没有完整审核记录的旧 `verified` 状态回到待审核。启动不自动迁移,缺失或不匹配的版本会明确报错。
|
||||
|
||||
## 安全边界
|
||||
当前无认证与权限控制,仅供本机开发;正式部署前须补充登录鉴权、接口限流、数据库迁移、密钥管理、日志脱敏、CORS 收敛和 HTTPS。
|
||||
新增 `users`、`auth_sessions` 与 `auth_rate_limits`。Argon2id 保存密码哈希,随机 Cookie 会话只保存摘要,角色逐次由数据库校验。来源白名单与会话 CSRF 保护写入;改密撤销所有会话。管理资源和检索历史均限管理员,公开检索只返回已核验资料。具体字段、来源、Cookie、预算和迁移见 [认证接口契约](AUTH_API.md)。
|
||||
|
||||
当前仍供本机开发与联调;完整生产部署、网关限流、访问审计、邮箱验证、账号恢复以及浏览器端权限流程验收仍待后续工程验收。
|
||||
|
||||
## 关键词检索基线(T06)
|
||||
|
||||
`POST /api/retrieval` 在同一 SQL 语料快照中筛选有效的已审核知识,读取出处和记录版本,再进行固定中文分词、标题加权 BM25 排序和原文片段截取。语料指纹覆盖本次实际参与排序的内容与出处版本;算法标识和参数独立记录。管理页的字面子串筛选继续用于管理,正式关键词基线使用专用检索接口。
|
||||
|
||||
新增 `retrieval_runs` 保存查询、模式、语料指纹和完整结果快照,支持分页读取及按 UUID 复核。零命中如实记录;当前依据修改后失效不会改写历史,但后续检索必须重新检查审核与父记录状态。响应耗时是本次真实读取、分词、排序及片段生成耗时,不含写日志或模型调用。
|
||||
|
||||
关键词基线仍按知识条目检索,显示片段最长 240 字符;独立的持久化 chunks 和文本索引版本见下方。该接口不读取 chunks;chunk 与向量/RRF 的独立接口见后文。全语料计算明确限制条目数及总字符数,超限报错,适用于首批小规模资料与后续对照实验准备,不能当作生产检索性能结论。
|
||||
|
||||
|
||||
## 切分与索引版本(T05)
|
||||
|
||||
`index_versions/chunks` 保存已审核语料的原文切分、出处与版本、参数、内容哈希及语料指纹。构建先保存 ready 快照,不切换当前发布;`index_state` 管理发布指针、代数与目录修订号,发布同一事务更新前后状态并追加 `index_publications`。
|
||||
|
||||
目录写入、构建与发布共用状态行锁,目录写入先于父记录锁获取该锁并递增修订号。发布核对代数、修订号和当前语料指纹;失败回滚,保留旧版本。目录变化后旧指针仍可追溯,但 is_current=false,不可据此绕过正式审核。历史快照不会被改写。接口均限管理员并沿用 Cookie/CSRF,完整边界见 [INDEX_API.md](INDEX_API.md)。
|
||||
|
||||
这只是后续索引的文本版本基础,关键词基线行为不变;没有生成向量或宣称已实现 ANN/混合检索。同步构建和全局写锁面向当前有限语料,后续异步/增量发布须重新设计并验证并发协议。
|
||||
|
||||
|
||||
## 向量与 RRF 检索(T07)
|
||||
|
||||
文本发布之后,固定 Embedding 空间的完整向量快照通过独立代数发布,精确余弦检索只使用当前审核与发布版本。编码在发布锁外进行,完成后重新核对语料、空间及发布状态,失败保留旧版本。运行与空间身份见 [VECTOR_API.md](VECTOR_API.md)。
|
||||
|
||||
新增 chunk 正文 BM25 与等权 RRF;两路限定同一文本/向量快照和分类,固定 k=60,先各取 candidate_k 再融合。保存完整候选排名、原始分数、贡献、切分/分词/BM25/模型配置及出处。旧条目关键词模式保留,评测不能混同两种证据单元。两路排序至检索记录写入共用当前性锁,面向最多 2000 chunks 的同步精确计算。字段、失败及回滚边界见 [HYBRID_API.md](HYBRID_API.md)。RAG、ANN 与真实领域质量验收仍待后续阶段。
|
||||
|
||||
@@ -0,0 +1,67 @@
|
||||
# 认证与角色权限接口(Issue #5)
|
||||
|
||||
本分支实现 [Issue #5](https://gitea.kronecker.cc/saint/border-red-post-nalati/issues/5) 的七个接口。认证分支 `feat/backend-auth-sessions` 基于 PR #6 的已验证后端提交,迁移依次为 `0002_catalog → 0003_retrieval_runs → 0004_auth_sessions`;PR #6 仍须独立审阅,本分支没有自行合并它。
|
||||
|
||||
## 启动与管理员初始化
|
||||
|
||||
在 `backend/` 安装更新依赖并执行 `alembic upgrade head`,再运行:
|
||||
|
||||
```bash
|
||||
python -m app.manage create-admin --username local_admin
|
||||
```
|
||||
|
||||
命令通过终端隐藏输入和确认密码,不接受密码参数,不创建默认账号;已有用户名不会被升级为管理员或覆盖密码。生产密码请使用独立长口令,不把账号、密码、Cookie 或响应保存到仓库。迁移只新增表,不预置任何用户。
|
||||
|
||||
默认 `AUTH_COOKIE_SECURE=true`,必须通过 HTTPS 使用会话。本机 HTTP 联调时可在未入库的 `backend/.env` 显式设置 `APP_ENV=development` 和 `AUTH_COOKIE_SECURE=false`,保持服务只绑定回环地址。其他环境拒绝关闭 Secure;非开发/测试环境的认证与 CORS Origin 必须均为 HTTPS。不要把本机例外带到部署环境。
|
||||
|
||||
`AUTH_ALLOWED_ORIGINS` 独立控制写请求的来源,默认允许 `localhost/127.0.0.1` 的 5173、8000 端口;`CORS_ORIGINS` 控制前端跨域读取。两者均须填写完整 `scheme://host:port`,不含路径/尾斜杠/通配符。部署时按实际地址同时收敛这两项。代理层仅信任受控代理,不能任意接受外部转发 IP 头。
|
||||
|
||||
## 接口契约
|
||||
|
||||
| 接口 | 请求 | 成功响应与权限 |
|
||||
|---|---|---|
|
||||
| POST `/api/auth/register` | `username/password`,可选 `nickname/email` | 201 用户资料;仅创建 `user`,不自动登录 |
|
||||
| POST `/api/auth/login` | `username/password` | 200 用户资料及 `csrf_token`;设置会话 Cookie |
|
||||
| POST `/api/auth/logout` | 无正文 | 204;要求当前会话和 CSRF,作废该会话并清除 Cookie |
|
||||
| GET `/api/auth/me` | 无 | 200 用户资料及当前 `csrf_token`;要求有效会话 |
|
||||
| PATCH `/api/users/me` | `nickname` 和/或 `email` | 200 当前用户资料;会话与 CSRF 必需 |
|
||||
| POST `/api/auth/change-password` | `current_password/new_password` | 204;验证当前密码后撤销该用户全部会话并清除 Cookie,须重新登录 |
|
||||
| GET `/api/admin/users` | `limit` 默认/最大 100,`offset` 默认 0 | 200 用户数组,按 ID 升序;仅 `admin` |
|
||||
|
||||
所有输入拒绝未知字段。注册和资料修改均不接受 `role/id/password_hash`,不能修改他人资料或用户名。用户名去首尾空白并转小写,须为字母开头的 3–32 位 ASCII 字母、数字或下划线。新密码为 15–128 字符,不修剪、不截断;旧密码只作校验。昵称最长 80 字符且不能含控制字符;邮箱须为合法地址,最长 254 字符。昵称/邮箱的空字符串或 `null` 用于清空;邮箱尚未验证,不能用于登录、找回密码或唯一身份判定。
|
||||
|
||||
用户资料为 `{id, username, nickname, email, role, created_at}`;登录和 `/me` 额外返回 `csrf_token`。响应不包含密码哈希、会话 Cookie 值或内部凭据版本。示例中的 ID、时间和名字均为占位:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": 1,
|
||||
"username": "example_user",
|
||||
"nickname": null,
|
||||
"email": null,
|
||||
"role": "user",
|
||||
"created_at": "2026-10-16T00:00:00Z",
|
||||
"csrf_token": "<由服务端返回的当前会话 CSRF 值>"
|
||||
}
|
||||
```
|
||||
|
||||
错误使用 `detail`:参数 422、用户名冲突 409、登录错误/会话失效 401、权限/来源/CSRF 错误 403、当前密码错误 400、认证请求过多 429(`Retry-After: 300`)。参数错误不回显输入值,避免原样返回密码。认证及其他 `/api/` 响应均使用 `Cache-Control: no-store`。
|
||||
|
||||
## Cookie、CSRF 与授权
|
||||
|
||||
- Cookie 名 `nalati_session`,HttpOnly、Secure(默认)、SameSite=Lax、Path=/、不设置 Domain。服务端生成 256 位随机会话标识;数据库仅保存其 SHA-256 摘要、用户、凭据版本及到期时间。固定有效期 `AUTH_SESSION_SECONDS` 默认 28800 秒,可配置 300–604800 秒,不自动续期。
|
||||
- 登录轮换本浏览器旧会话;退出仅撤销当前会话;改密通过用户行锁、凭据版本递增与删除全部会话使旧凭据立即失效。授权每次从数据库读取当前角色,不信任前端角色或 Cookie 内的角色声明。
|
||||
- 登录/注册即检查精确匹配的 `Origin`,缺失或不可信均拒绝,避免登录 CSRF。已认证的 POST/PATCH/DELETE 等写请求还须带 `X-CSRF-Token`,值来自登录或 `/me`。该值由当前随机 Cookie 经独立 HMAC 派生,不能跨会话复用。后端不从请求正文、URL 或客户端角色获取权限。
|
||||
- `knowledge`、`sources`、`documents` 的全部读写及历史、管理员用户列表、检索历史列表/详情均要求 `admin`。因此旧匿名知识管理调用现在返回 401,普通用户返回 403。健康检查、公开的已核验关键词检索和当前占位问答保持公开;正式问答、实验接口接入时须重新明确各自授权。
|
||||
- 登录及注册共用每客户端地址 30 次/5 分钟预算;登录另有规范用户名 10 次/5 分钟预算,成功/失败都计数。计数通过 PostgreSQL 原子更新,跨进程/重启有效;仅保存带类型前缀的摘要,旧预算在下一次认证请求时清理。此处为基础防暴力尝试,不代替部署网关限流与监控。
|
||||
|
||||
前端 `src/services/auth.ts` 和 `src/services/session.ts` 提供真实 Cookie 请求封装:CSRF 只存内存,刷新后通过 `/me` 恢复;知识写入也使用同一封装。`dev` 已提供登录、注册、个人中心、账户设置、管理员用户列表与路由守卫;导航菜单按当前角色隐藏管理员入口。账户资料更新与改密均调用真实接口,前端的新密码规则与后端一致为至少 15 个字符。浏览器端 Cookie、CORS 和 PostgreSQL 联调仍须在目标部署环境验收,不能由静态构建替代。
|
||||
|
||||
## 验证与回滚
|
||||
|
||||
在 `backend/` 执行 `ruff check .`、`ruff format --check .`、`pytest -q`;在 `frontend/` 执行 `npm run build`。测试使用隔离 PostgreSQL 和占位账号,覆盖权限拒绝、CSRF、输入越权、哈希、轮换/撤销/过期、改密、重启后会话恢复与退出持久化、预算、迁移及既有检索回归。测试库不代表 Docker 或浏览器验收。
|
||||
|
||||
备份后可配合旧后端执行 `alembic downgrade 0003_retrieval_runs`,删除用户、会话与认证预算表,保留知识和检索历史;账号与哈希恢复依赖备份。降级会撤除授权边界,旧服务只能按其原开发范围使用,不应用于公开部署。继续降至 `0002_catalog` 还会删除检索历史。
|
||||
|
||||
尚未实现邮箱验证、找回密码、MFA、角色管理 API、会话管理页面、访问审计和完整生产部署;没有把这些功能标记为完成。
|
||||
|
||||
实现依据:[argon2-cffi API](https://argon2-cffi.readthedocs.io/en/stable/api.html)、[OWASP 密码存储](https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html)、[OWASP 认证](https://cheatsheetseries.owasp.org/cheatsheets/Authentication_Cheat_Sheet.html)、[OWASP CSRF 防护](https://cheatsheetseries.owasp.org/cheatsheets/Cross-Site_Request_Forgery_Prevention_Cheat_Sheet.html)。本项目使用 Argon2id(19 MiB、2 次迭代、并行度 1);会话与来源策略以本接口契约及代码为准。
|
||||
@@ -14,3 +14,38 @@
|
||||
|
||||
## ADR-005 更新策略
|
||||
状态:设计确定、尚未实现。新文档版本后台生成向量,完成后原子发布;失败保留旧索引。
|
||||
|
||||
## ADR-006 阶段 2 知识管理
|
||||
|
||||
状态:已通过 PR #4 合入 `dev`。采用来源、文档、知识条目和追加式事件日志;记录版本用于并发控制,快照用于追溯。所有删除均为软删除,被有效记录引用的父记录禁止删除。依据或正文修改即回待审核;核验通过必须有来源链接、采集时间、核验人和说明。保留旧版知识创建请求及列表数组结构,新页面使用独立分页接口。当前为本机人工审核流程,认证身份、向量切分和索引发布仍属后续阶段。
|
||||
|
||||
## ADR-007 显式迁移与旧数据保留
|
||||
|
||||
状态:已通过 PR #4 合入 `dev`。以 Alembic 取代应用启动时的 `create_all`。基础迁移接管旧知识表,新增字段和表通过增量迁移完成;不推测旧数据采集时间。旧审核结论缺少完整记录时保留原快照并转待审核。迁移回滚需先备份,新增历史与审核元数据无法通过降级完整保留。
|
||||
|
||||
## ADR-008 可复核的中文关键词基线
|
||||
|
||||
状态:分支实现,待审阅。采用固定 Jieba 搜索分词及关闭 HMM 的词典模式,加入项目名称词,并使用固定停用词表。选择标题重复权重 2、正文权重 1 的 BM25 正值平滑 IDF 变体,不称为 BM25F;查询词去重,同分按知识 ID 排序。参数、分词标识、实际语料指纹及出处版本随每次结果保存,方便冻结后续实验配置。分词接口参考 [Jieba 官方说明](https://github.com/fxsjy/jieba),BM25 基础参考 [Stanford IR 教材](https://nlp.stanford.edu/IR-book/html/htmledition/okapi-bm25-a-non-binary-model-1.html);具体平滑公式与权重以本仓库实现和手算测试为准。
|
||||
|
||||
当前检索单元为已审核知识条目,在设置的容量内按请求计算,不提前实现向量切分/索引更新;超过容量明确失败。`retrieval_runs` 仅记录当时结果,不代表该条目之后仍然获得审核批准。网络搜索、模型选择和真实论文实验保持原阶段边界。
|
||||
|
||||
|
||||
## ADR-009 服务端会话与最小角色边界
|
||||
|
||||
状态:Issue #5 分支实现,待审阅。使用 Argon2id 密码哈希、数据库保存摘要的随机 Cookie 会话,以及独立 CSRF 请求头与显式 Origin 白名单;不把角色或会话凭据放入浏览器持久存储。普通注册只能创建 user,管理员仅由本机隐藏密码输入命令创建。知识/来源/文档管理及检索历史限 admin,公开检索继续只使用已核验依据。完整契约与实现依据见 [AUTH_API.md](AUTH_API.md)。
|
||||
|
||||
为改密和会话撤销使用用户行锁与凭据版本,认证预算使用数据库原子计数;不依赖单进程内存限流。现有前端写请求同步 Cookie/CSRF 封装,页面和路由联调仍待前端完成。本分支叠加 PR #6,后续审阅需关注迁移顺序及匿名管理调用变为 401 的行为变化。
|
||||
|
||||
|
||||
## ADR-010 切分快照与显式发布
|
||||
|
||||
状态:T05 分支实现,待审阅。先对当前有效审核语料构建带原文偏移/哈希/出处的 chunks,再以独立的发布请求切换指针。固定 Unicode 字符窗口配置留存在版本中,关键词接口继续按条目运行,防止未经评估就改变实验口径。
|
||||
|
||||
为当前小规模语料选择统一数据库行锁:应用目录写入、构建、发布串行,发布同时校验代数、目录修订和语料指纹;失败回滚全部更新,旧版本保留为历史。任何成功目录写入保守地使旧索引过期,历史不能视作当前有效依据。实现、容量与后续并发扩展边界见 [INDEX_API.md](INDEX_API.md)。
|
||||
|
||||
|
||||
## ADR-011 同粒度 chunk 排序与等权 RRF
|
||||
|
||||
状态:后端分支实现,待审阅。基于已合入的向量快照,新增正文 BM25 的 chunk 基线,与向量模式使用同一文本发布及分类范围;保留条目关键词接口作为独立口径。两路候选各限 candidate_k,使用从 1 起的排名、固定 k=60、等权倒数排名融合,缺席一路贡献为零。保存候选全排名、原始分数、两路贡献和配置,确保实验准备可复核;不把 BM25 与余弦原始分数直接相加。
|
||||
|
||||
公式与常数参考 [RRF 原论文](https://cormack.uwaterloo.ca/cormacksigir09-rrf.pdf),本项目尚未依据核验领域问题集调优。同步精确计算限制 2000 chunks,固定配置、版本一致性、历史兼容与回滚见 [HYBRID_API.md](HYBRID_API.md)。
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
# 前端页面与功能设计(阶段一)
|
||||
|
||||
## 设计目标
|
||||
|
||||
前端不是展示虚构文旅结论的门户,而是让团队能看清资料从来源登记、文档采集到知识审核的状态。所有列表均以接口返回为准;接口不可用时展示明确的连接状态,不使用模拟数据兜底。
|
||||
|
||||
## 页面结构
|
||||
|
||||
| 页面 | 已实现 | 下一步功能 | 验收标准 |
|
||||
|---|---|---|---|
|
||||
| 智能问答 | 输入、会话、来源链接、请求中与不可用提示 | 检索命中片段、追问、会话清空 | 每条回答显示后端状态和可打开来源 |
|
||||
| 知识库管理 | 新建条目、搜索、状态统计、来源跳转 | 审核抽屉、编辑、版本冲突提示、历史记录 | 通过审核前必须有来源网址与采集时间 |
|
||||
| 来源与更新 | 登记来源、检索来源、来源状态说明 | 文档列表、关联创建、编辑/软删除、历史记录 | 不自动抓取;来源登记不等同于事实核验 |
|
||||
| 实验评估 | 指标边界与对照方案说明 | 问题集、人工评分、结果导出 | 不展示未经真实实验产生的数值 |
|
||||
|
||||
## 核心操作链路
|
||||
|
||||
1. 资料负责人登记来源:填写名称、网址和可靠性备注。
|
||||
2. 采集文档:必须选定一个来源,记录正文、分类和真实采集时间。
|
||||
3. 前端创建知识条目:关联文档或填写可访问的来源网址,默认状态为 `pending`。
|
||||
4. 审核人核验:填写审核说明;仅在来源网址和采集时间齐全时可标为 `verified`。
|
||||
5. 检索阶段只使用已核验内容;问答页展示来源链接与“原型阶段”提示。
|
||||
|
||||
## 交互与状态规范
|
||||
|
||||
- 提交按钮在请求期间禁用,完成后刷新对应列表。
|
||||
- 接口错误显示后端返回的业务信息;网络错误提示检查 FastAPI 和 PostgreSQL。
|
||||
- 删除采用后端软删除;前端须在删除前显示关联文档/知识条目可能阻止删除的说明。
|
||||
- 并发修改以接口 `version` 为准;返回 409 时提示刷新列表后重试,禁止静默覆盖。
|
||||
- 移动端优先展示表格的标题、状态和主要动作,其他字段通过横向滚动查看。
|
||||
|
||||
## 前端与后端边界
|
||||
|
||||
- 前端不决定资料真实性、不生成审核结论,也不伪造统计结果。
|
||||
- 前端仅调用 `/api/health`、`/api/chat`、`/api/knowledge`、`/api/sources` 和后续 `/api/documents`;鉴权接入后统一由 API 层处理。
|
||||
- 网页抓取、Embedding、向量检索与定时更新属于后端后续阶段,当前页面只说明其规划状态。
|
||||
@@ -0,0 +1,67 @@
|
||||
# Chunk 关键词与 RRF 混合检索
|
||||
|
||||
本阶段基于已合入 `dev` 的文本/向量发布,增加可独立调用的 chunk BM25 和 RRF。既有条目关键词接口继续保留。三个 chunk 模式使用相同的当前文本版本、原文 chunk 与分类范围;比较时应固定发布版本、分类和 `top_k`,同时记录混合检索的 `candidate_k`。
|
||||
|
||||
| 接口 | 请求 mode | 排序/证据单元 | 前置条件 |
|
||||
|---|---|---|---|
|
||||
| `POST /api/retrieval` | `keyword` | 知识条目,标题权重 2、正文权重 1 | 当前有效审核语料 |
|
||||
| `POST /api/retrieval/chunks` | `keyword_chunks` | chunk 正文 BM25,标题不参与打分 | 当前有效文本发布;无需模型或向量发布 |
|
||||
| `POST /api/retrieval/vector` | `vector` | chunk 精确余弦 | 当前有效文本/向量发布,运行模型空间匹配 |
|
||||
| `POST /api/retrieval/hybrid` | `hybrid` | 同一批 chunk 的两路 RRF | 与向量模式相同 |
|
||||
|
||||
`mode` 可省略,由接口固定;传入其他模式或未知字段返回 422。公开检索无需登录;历史分页及详情继续限管理员。问答接口仍为占位,当前响应不生成 LLM 答案。
|
||||
|
||||
## 请求与排名
|
||||
|
||||
```json
|
||||
{"query":"合成资料中的交通查询入口","mode":"hybrid","top_k":5,"candidate_k":50,"category":"交通"}
|
||||
```
|
||||
|
||||
- `query` 去除首尾空白后 1–500 字符,`top_k` 默认 5、范围 1–20,`category` 可省略。chunk 关键词请求不接收 `candidate_k`。
|
||||
- 混合检索每路最多取 `candidate_k` 条候选,默认 50、范围 1–200,并须不小于 `top_k`。这是固定排名后的候选截断,不会改变文本或输入 token。最终从候选并集返回最多 `top_k` 条。
|
||||
- chunk BM25 只对正文分词。沿用固定 Jieba 0.42.1 搜索分词、NFKC/casefold、HMM 关闭、项目词与 `zh-small-v1` 停用词表;查询词去重,`k1=1.2`、`b=0.75`,正值平滑 IDF。语料长度和词频在分类过滤后的全部 chunks 上计算,只将正分候选纳入关键词一路。标题仅作为出处字段。
|
||||
- 向量一路复用精确余弦排序,同样先按分类筛选,不设置相似度阈值。负分和零分仍可能进入候选,分数不是事实置信度,`status=ok` 只表示存在排序结果。
|
||||
- 两路先按各自原始分数排序;同分按 `knowledge_id, chunk_ordinal, chunk_id` 升序,排名从 1 开始。RRF 使用等权、固定 `k=60`:`score(c) = sum(1 / (60 + rank_leg(c)))`。不在某路候选列表中的 chunk,该路贡献为 0。原始 BM25 和余弦分数不直接相加;融合分数同分沿用相同 ID 次序。排序与日志使用未预先四舍五入的分数。
|
||||
- 融合以 chunk ID 去重。同一知识的不同片段(含重叠片段)保留各自排名和原文偏移,不做条目折叠或多样性重排。条目基线与 chunk 模式的分数、命中数、语料数及评测单元不同,不能直接混算指标。
|
||||
|
||||
RRF 方法与常数参考 [Cormack、Clarke、Büttcher,SIGIR 2009 原论文](https://cormack.uwaterloo.ca/cormacksigir09-rrf.pdf)。这里把 `k=60` 作为首版可复现配置,并未通过本项目领域问题集调优。
|
||||
|
||||
## 响应与复核
|
||||
|
||||
全部模式保留原文、字符偏移(左闭右开)、知识/文档/来源 ID 及版本、URL、排名与耗时。chunk 关键词和混合响应另含:
|
||||
|
||||
| 字段 | 含义 |
|
||||
|---|---|
|
||||
| `mode` | `keyword_chunks` 或 `hybrid`,可用于历史类型判别 |
|
||||
| `text_index_version_id`, `text_generation`, `catalog_revision` | 本次文本发布和目录修订 |
|
||||
| `corpus_fingerprint` | 整份已审核语料的指纹,分类筛选不改变它 |
|
||||
| `corpus_size` | 分类范围内参与两路打分的 chunk 数 |
|
||||
| `total_hits` | 关键词模式为全部正分 chunk 数;混合模式为两路截断后的候选并集大小,可能大于返回条数 |
|
||||
| `query_terms`, `config.lexical`, `config.chunking`, `config.tie_break` | 实际分词、BM25 参数、切分配置、稳定排序规则 |
|
||||
| `elapsed_ms` | 语料检查、模型编码(混合模式)、排序及响应组装耗时;不含记录写入及网络时间 |
|
||||
|
||||
混合模式还保存:
|
||||
|
||||
- `vector_version_id`、`generation`:向量发布版本与代数;`config.space` 包含完整模型 revision、维度、token 预算、指令、实现/池化/归一化与权重格式身份;与向量契约一致。
|
||||
- `config.rrf_k/weights/candidate_k/vector_scorer`:融合与候选预算,不允许请求覆盖固定算法参数。
|
||||
- `lexical_candidates/vector_candidates`:两路完整的截断候选列表,每条为 `chunk_id/rank/score`,即使未进最终前 `top_k` 也保留。
|
||||
- `lexical_total_hits/vector_total_hits`:截断前正分关键词候选数、分类范围内向量候选总数。
|
||||
- 命中的 `lexical_rank/lexical_score/vector_rank/vector_score` 和两路 `*_contribution`:缺席的排名/原始分数为 `null`、贡献为 0。`score` 等于两路贡献之和。
|
||||
|
||||
结果完整写入现有 `retrieval_runs`,历史接口联合类型支持四种 mode,不需要新增迁移,当前 Schema 仍为 `0006_vectors`。修改依据后历史保持原样,只代表当时快照;重查须通过当前性验证。原始查询和运行记录属于数据库数据,不应提交到仓库。
|
||||
|
||||
## 一致性、失败与容量
|
||||
|
||||
混合检索先检查发布版本与模型空间,然后提交只读事务释放发布锁,再真实编码查询;编码后重新获取目录/向量状态锁,复核目录修订、文本/向量代数、审核指纹、实际 chunks、完整向量集及空间身份。变化时 409,且不保存部分结果。两路排名及写入检索历史在同一个受锁保护的事务内完成。
|
||||
|
||||
关键词一路没有命中或没有分词时,混合模式明确保留空候选列表,可仅有向量一路;纯关键词模式没有可检索词则 422。分类无资料或空的有效发布返回 `no_evidence`。模型 token 超限/输出校验失败返回 422,运行环境不可用或编码忙返回 503;模型失败不降级成一份声称成功的混合结果。
|
||||
|
||||
两种新接口均沿用向量阶段最多 2000 chunks、5000 条知识/200 万字符的整体核验预算,超限明确 503。只在分类范围内打分,但当前性及容量检查针对整份发布。文本快照允许的 20000 chunks 不等同于本接口容量。同步精确排序及 BM25 在锁内运行,会短暂串行化目录写入;当前没有 ANN、后台队列或生产吞吐量结论。
|
||||
|
||||
运行依赖、固定缓存、默认关闭的本地模型、构建/发布步骤见 [VECTOR_API.md](VECTOR_API.md)。前端新增 `services/hybrid.ts` 两个调用及历史类型;模式切换页面、Docker/浏览器端和真实领域验收仍待交接。
|
||||
|
||||
## 验证与回滚
|
||||
|
||||
轻量测试使用明确的合成正文及坐标 fixture,覆盖手算 BM25/RRF、原始分数与融合的独立性、候选边界、两路缺席、稳定排序、chunk 偏移、分类、权限、历史、失效/破坏快照、编码期间版本竞争、无部分日志及容量拒绝。真实模型专项沿用 `test_vector_runtime.py` 的显式 opt-in,增加固定 BGE small 的 HTTP 混合检索和进程重启后的两路排名/历史复核。运行命令见 [VECTOR_API.md](VECTOR_API.md)。这些用例验证实际模型/数据库/API 贯通,合成问题不代表领域质量或论文实验结论。
|
||||
|
||||
撤回本功能代码即可撤回两个新增接口;数据库无需降级。旧后端历史响应模型不认识 `keyword_chunks/hybrid`,回滚时应限制旧历史端点或先以备份保留新模式记录,避免把新记录交给旧模型解析;不要为回滚而无备份删除检索历史。
|
||||
@@ -0,0 +1,81 @@
|
||||
# 文本切分与索引版本接口(T05)
|
||||
|
||||
本阶段保存可复核的已审核语料切分及发布指针,为后续检索索引提供版本边界。这里的“索引版本”是文本语料快照,不包含 Embedding、pgvector 向量字段/ANN 索引或 RRF。现有 `POST /api/retrieval` 仍运行原关键词基线,不读取这些 chunks;不能把本阶段称为向量检索已完成。
|
||||
|
||||
## 依赖、迁移与权限
|
||||
|
||||
分支 `feat/backend-chunk-index-versions` 叠加待审 PR #7(认证),并继承 PR #6(关键词)。三项须分别跟踪审阅;本分支不自行合并其他请求。升级时停止旧后端实例,安装依赖、备份并执行 `alembic upgrade head`,再启动新实例。所有实例必须使用当前版本,才能让目录写入遵守同一发布锁。
|
||||
|
||||
新增迁移 `0005_chunk_indexes`,依赖 `0004_auth_sessions`,创建:
|
||||
|
||||
- `index_versions`:切分配置、语料指纹、目录修订号、条目/chunk 数、构建人、构建时间及发布状态。
|
||||
- `chunks`:所属版本、知识 ID、从 0 开始的片序、原文起止偏移、正文、UTF-8 内容 SHA-256 与出处快照。
|
||||
- `index_state`:单例发布指针、发布代数 `generation`、目录修订号 `catalog_revision`。
|
||||
- `index_publications`:每次发布的前后版本、代数、操作人和时间;重复发布当前版本不会新增记录。
|
||||
|
||||
迁移不改写既有知识、审核、用户、会话或检索记录,不创建初始发布版本。所有 `/api/indexes` 接口均要求 `admin`;POST 还必须携带有效 Cookie、明确 Origin 与 `X-CSRF-Token`,与 [认证契约](AUTH_API.md) 相同。
|
||||
|
||||
## 构建与原文切分
|
||||
|
||||
`POST /api/indexes` 创建一个 `ready` 版本,返回 201,不切换当前发布指针。请求可用默认值 `{}` 或:
|
||||
|
||||
```json
|
||||
{"chunk_size": 500, "overlap": 80}
|
||||
```
|
||||
|
||||
`chunk_size` 为 100–2000 个 Python Unicode 字符,默认 500;`overlap` 默认 80,非负且不得超过切分长度一半。自定义较短长度时也须指定兼容的重叠,例如 `{"chunk_size":100,"overlap":20}`。未知字段拒绝,不支持按类别单独发布。
|
||||
|
||||
固定算法 `unicode-window-v1` 从知识条目原正文位置 0 开始(不是关联文档或原网页的偏移),以 `chunk_size-overlap` 前进,末片到正文结束即停止。保留空白、全半角、组合字符和 Emoji,不标准化或截断原文。`start_offset` 包含起点,`end_offset` 不包含终点;每片 `text == 原文[start_offset:end_offset]`。标题放在出处信息中,不拼进正文偏移。客户端若使用 JavaScript 字符串偏移,须注意其 UTF-16 索引与 Python Unicode 字符偏移的区别。
|
||||
|
||||
- 语料条件与关键词基线共用:只读取未删除、审核及采集信息完整的 `verified` 知识,关联来源/文档须有效、归属及来源网址一致;pending/rejected 不进入构建。
|
||||
- 语料指纹与关键词基线使用同一规范 JSON/算法,涵盖实际正文、标题、类别、出处及知识/父记录版本;切分参数独立保存在 `config`。
|
||||
- 出处快照包含 `knowledge_id/knowledge_version`、可空 `document_id/document_version`、可空 `source_id/source_version/source_name`、`source_url/title/category`。以后原资料修改或失效,不改写旧 chunk 内容及出处。
|
||||
- chunk ID 在一个版本内由版本 UUID、知识 ID 与顺序确定;新构建使用新版本 ID。相同资料及参数可重现偏移/正文/内容哈希,但新旧版本的 chunk ID 不相同。
|
||||
- 本阶段同步构建,沿用最多 5000 条及标题/正文总计 200 万字符的语料限制,每版本另限 20000 个 chunks。超限返回 503;事务写入失败也完整回滚,不保留半个版本,不改变旧发布指针。
|
||||
- 可以构建空语料版本,以便所有依据撤回后发布空快照;这不表示存在可用知识。
|
||||
|
||||
## 发布、过期与并发
|
||||
|
||||
先通过 `GET /api/indexes/active` 读取当前代数,再提交:
|
||||
|
||||
```text
|
||||
POST /api/indexes/{version_id}/publish
|
||||
```
|
||||
|
||||
```json
|
||||
{"expected_generation": 0}
|
||||
```
|
||||
|
||||
发布时锁定 `index_state`,检查代数、目录修订号及当前有效语料指纹;资料或审核已变化、代数过期均返回 409,要求重新构建或刷新。成功后同一事务把旧版本设为 `superseded`、新版本设为 `published`、切换指针、增加代数并追加发布记录。校验失败或写入失败保留此前全部发布状态。持有当前代数重复发布当前版本是无变化的 200;仍有效的历史版本可用最新代数重新发布,发布记录保留每次转换。
|
||||
|
||||
知识、来源、文档的应用写入在修改父记录前持有同一状态锁,事务内将 `catalog_revision` 增加一次;失败写入一起回滚修订号。构建、发布和这些写入串行,避免校验到提交之间发生目录修改。为首批小规模资料选择此保守策略,长构建会暂时阻塞管理写入;尚无后台任务或大规模增量索引。
|
||||
|
||||
任一成功目录写入都会使旧快照过期,包括尚未核验的新条目;此规则比只比较有效语料更保守。`GET /active` 同时核对修订号与实际语料指纹,返回:
|
||||
|
||||
```json
|
||||
{"generation": 1, "catalog_revision": 2, "is_current": true, "active_version": {"id": "<版本 UUID>", "status": "published", "config": {"splitter":"unicode-window-v1","chunk_size":500,"overlap":80}, "...":"其余版本字段见 OpenAPI"}}
|
||||
```
|
||||
|
||||
初始无发布时 `active_version=null/is_current=false`。资料变化后指针及历史保留,但 `is_current=false`;`published` 只表示仍是发布指针,不表示资料仍有效。历史 chunks 是当时的依据快照,不能绕过 `is_current` 直接用于后续正式检索。任何结果只描述该事务快照,之后的目录写入仍可让它过期。直接改库不属于支持的维护流程;额外指纹检查可发现绕过修订号的正文/出处变化,不能替代应用写锁。
|
||||
|
||||
## 查询与交接
|
||||
|
||||
| 接口 | 响应 |
|
||||
|---|---|
|
||||
| GET `/api/indexes` | 版本分页,构建时间降序,同时间按 ID |
|
||||
| GET `/api/indexes/active` | 发布代数、目录修订号、当前性与当前版本 |
|
||||
| GET `/api/indexes/publications` | 发布记录分页,代数降序 |
|
||||
| GET `/api/indexes/{version_id}` | 版本元数据 |
|
||||
| GET `/api/indexes/{version_id}/chunks` | chunk 分页;可按 `knowledge_id` 筛选,按知识 ID/片序排列 |
|
||||
|
||||
分页沿用 `{items,total,page,page_size}`,默认 20、最大 100。版本元数据完整字段为 `id/status/config/corpus_fingerprint/catalog_revision/item_count/chunk_count/created_by/created_at/published_at`;`published_at` 是最近一次发布的时间,逐次时间保存在发布记录。版本及发布记录时间统一输出 UTC ISO 8601,避免数据库时区让重读结果出现不同表示。chunk 字段为 `id/index_version_id/knowledge_id/ordinal/start_offset/end_offset/text/text_sha256/provenance`。
|
||||
|
||||
前端 `src/services/indexes.ts` 提供类型和调用封装,写入共用 Cookie/CSRF;未增加页面或假数据。接口新增需由前端按该契约接入、审阅并验收。独立 T05 Issue 因当前缺少 Gitea 写入认证仍待补建,本 PR 和任务板先保留实现与验证记录。
|
||||
|
||||
## 验证及回滚
|
||||
|
||||
验证包含真实 PostgreSQL 的原文重建与哈希、审核筛选、重复构建、过期拒绝、来源/文档更新、失败写入、发布历史、数据库锁竞争、失败构建/发布的真实约束回滚、迁移兼容及真实 HTTP 重启后指针/chunks 持久化。这里只使用占位资料,不是论文实验结果。Docker 和浏览器端验收保持待办。
|
||||
|
||||
备份后配合旧后端执行 `alembic downgrade 0004_auth_sessions`,只撤回这四张表,保留用户、会话、知识及检索记录;索引历史与 chunks 恢复依赖备份。旧应用不可使用新版本数据库混跑。
|
||||
|
||||
并发实现参考 [PostgreSQL 16 行锁说明](https://www.postgresql.org/docs/16/explicit-locking.html) 与 [SQLAlchemy FOR UPDATE](https://docs.sqlalchemy.org/en/20/core/selectable.html#sqlalchemy.sql.expression.Select.with_for_update)。
|
||||
@@ -0,0 +1,79 @@
|
||||
# T07:本地 Embedding 与 CPU 资源评估
|
||||
|
||||
本阶段从 `dev` 独立开发,交付可替换 Provider 和可复现的本地 CPU 评估工具。它不依赖待审 PR #6/#7/#8 的接口或数据库迁移;后续向量检索集成再组合这些基础。普通后端启动及普通自动化测试均不导入 PyTorch、不加载模型,也不要求模型下载。
|
||||
|
||||
## 固定配置与边界
|
||||
|
||||
| CLI 名称 | 官方模型 | 固定 revision | 维度 | Token 上限 | 权重格式 |
|
||||
|---|---|---|---:|---:|---|
|
||||
| small | BAAI/bge-small-zh-v1.5 | `7999e1d3359715c523056ef9478215996d62a620` | 512 | 512 | Safetensors |
|
||||
| base | BAAI/bge-base-zh-v1.5 | `f03589ceff5aac7111bd60cfc7d497ca17ecac65` | 768 | 512 | PyTorch,仅权重读取 |
|
||||
|
||||
两者使用 CLS 输出、L2 归一化、CPU FP32;查询添加官方检索指令,正文不添加该指令。`EmbeddingSpec` 冻结模型/revision/维度/token 上限/查询指令/实现版本/权重格式,由规范 JSON 生成 `space_id`。不同空间必须分开保存和查询;目前只提供身份契约,数据库隔离约束尚待向量阶段实现。
|
||||
|
||||
模型仓库均标注 MIT。基础版该修订只提供 `pytorch_model.bin`,显式使用 Transformers 的 `weights_only=True` 读取;不会自动回退未知格式或开启远程模型代码。公开模型下载使用 `token=False`,不要求或读取用户保存的 Hugging Face 凭据。默认只用本机缓存,CLI `--download` 才允许下载固定修订。
|
||||
|
||||
`LocalEmbeddingProvider` 按批处理,默认 batch=8、CPU 线程=2;显式限制 batch 1–32、线程 1–4。输入先按真实 tokenizer 计数,包含查询指令和特殊 token;超过模型上限明确报错,不静默截断。输出检查数量、维度、有限数值和 L2 范数。字符切分长度不等于 token 数,后续接入 chunks 时须据此重新检查。
|
||||
|
||||
## 输入与指标
|
||||
|
||||
项目当前资料台账只有未核验的占位行,没有可用于正式选型的领域数据。本阶段自带 `backend/evaluation/synthetic_zh.json`,含 12 条明确标注“合成测试”的示例文本及 10 个问题,ID/相关文档标注在运行前固定。这些句子不描述那拉提真实事实,不写入应用数据库。
|
||||
|
||||
评估保存输入文件 SHA-256、模型完整配置与空间 ID、运行库版本、实际命中 ID/分数、命中率、Recall@3 和 MRR@3。排序采用归一化向量内积,同分按文档 ID;数值不是正确率或置信度。命中率检查首位相关文档;Recall@k 为前 k 命中相关文档数/标注相关文档总数;MRR@k 为首个相关命中的倒数,未命中为 0。
|
||||
|
||||
每项计时先按同一输入形状预热一次,然后记录 5 次及中位数:12 条语料编码、10 个查询批量编码、单个查询、8 条长文本批量编码。长文本均为相同的合成句子重复形成,360 字符/362 token,仅测试资源。包含分词、补齐、前向、归一化和结果转换;不包含网络、数据库、HTTP 或整个 RAG 延迟。初始化时间单独记录,可能包含缓存检查/下载与首次加载,不用于模型推理速度比较。Linux 峰值 RSS 是整个独立进程高水位,包含解释器、运行库和模型,不能当作仅权重大小。
|
||||
|
||||
## 本机实测
|
||||
|
||||
运行环境:Linux、Python 3.11.17、4 个逻辑 CPU、内存总计约 9.7 GiB,检查时可用约 5.0 GiB;GPU 驱动不可用,本轮仅用 CPU。PyTorch `2.14.1+cpu`、Transformers `5.20.0`、huggingface-hub `2.2.0`、NumPy `2.4.6`;每个模型用独立进程测量,线程 2、batch 8、每项 5 次。未记录用户名、主机名或本机个人路径。
|
||||
|
||||
输入文件 SHA-256:`9f4762a10dc1502b871c84ccedf9fbbdbd93067b8318476a77c78aa14789b243`。
|
||||
|
||||
| 模型 | 12 条正文批编码中位数 | 10 个查询批编码中位数 | 单查询中位数 | 8 条长文批编码中位数 | 峰值进程 RSS | 合成 Hit@1 / Recall@3 / MRR@3 |
|
||||
|---|---:|---:|---:|---:|---:|---|
|
||||
| small | 98.708 ms | 72.877 ms | 8.927 ms | 457.944 ms | 531.2 MiB | 0.90 / 0.90 / 0.90 |
|
||||
| base | 582.315 ms | 416.287 ms | 52.912 ms | 3016.157 ms | 865.5 MiB | 1.00 / 1.00 / 1.00 |
|
||||
|
||||
两个模型在该负载下都能使用本机 CPU。small 在合成问题 q05(出处可靠性)把住宿条目 d07 排在首位,目标资料审核条目 d06 未进入前三;base 将 d06 排首位。这里保留失败结果,不修改数据以提高分数。两个模型的相似度分布不同,不能共用固定分数阈值。
|
||||
|
||||
建议:资源优先的后端开发验证可先使用 small,保留 base 作对照。正式默认模型暂不冻结,先在已核验语料与独立问题集上比较相关性、无依据情形和实际端到端延迟。当前未测 GPU、并发服务、最大 token 负载或完整索引构建。
|
||||
|
||||
这些是合成小样本的功能与资源结果,不是项目正式实验分数。不能据此断言真实文旅问答质量或生产尾延迟。应先用资料负责人核验后的语料及独立问题集复测,再固定默认模型;不能通过修改测试题迎合已观察的排名。
|
||||
|
||||
## 复现(Linux,仓库根目录)
|
||||
|
||||
可使用 `uv` 创建独立 Python 3.11 环境;或以等价 venv/pip 命令安装。依赖独立于 `requirements.txt`,不把 GPU 依赖或模型加入默认 API 启动流程。
|
||||
|
||||
```bash
|
||||
task_eval_root=$(git rev-parse --absolute-git-dir)
|
||||
test -x "$task_eval_root/embedding-eval-venv/bin/python" || \
|
||||
uv venv --python 3.11 "$task_eval_root/embedding-eval-venv"
|
||||
uv pip install --python "$task_eval_root/embedding-eval-venv/bin/python" \
|
||||
--index-url https://download.pytorch.org/whl/cpu torch==2.14.1+cpu
|
||||
uv pip install --python "$task_eval_root/embedding-eval-venv/bin/python" \
|
||||
-r backend/requirements-embedding-cpu.txt
|
||||
export HF_HUB_DISABLE_TELEMETRY=1 HF_HUB_DISABLE_IMPLICIT_TOKEN=1
|
||||
export HF_HUB_DISABLE_PROGRESS_BARS=1 TOKENIZERS_PARALLELISM=false
|
||||
export HF_HUB_DISABLE_XET=1
|
||||
(
|
||||
cd backend
|
||||
"$task_eval_root/embedding-eval-venv/bin/python" -m evaluation.local_embeddings \
|
||||
--model small --download --cache-dir "$task_eval_root/embedding-model-cache" \
|
||||
--output "$task_eval_root/embedding-evaluation/small.json"
|
||||
"$task_eval_root/embedding-eval-venv/bin/python" -m evaluation.local_embeddings \
|
||||
--model base --download --cache-dir "$task_eval_root/embedding-model-cache" \
|
||||
--output "$task_eval_root/embedding-evaluation/base.json"
|
||||
)
|
||||
```
|
||||
|
||||
后续离线复跑去掉 `--download`。可用 `--dataset` 指定同结构数据,文档/问题各最多 1000 条、文件最多 2MB;拒绝重复 ID、未知或重复的相关文档标注。报告不含输入文本,但实际问题/标签可能仍敏感,原始报告、运行输出、环境及模型缓存都应只留在本机 Git 元数据目录,不提交。参数错误或模型失败不会生成伪造的成功结果。
|
||||
|
||||
普通后端验证仍是在 `backend/` 执行 `ruff check .`、`ruff format --check .`、`pytest -q`。轻量测试验证模型身份、非法输出、资源参数、token 拒绝、输入标注及手算排序指标,不需要模型包或联网;真实模型执行由上述独立 CLI 验证。
|
||||
|
||||
## 下一步
|
||||
|
||||
将 Provider 接入已发布的文本版本,完成 pgvector 存储与严格空间隔离、向量检索及 RRF,再使用真实审核语料评测。模型编码耗时应在发布锁外运行,发布前重新核对目录/语料版本;过期或失败时保留旧版本。当前没有新增向量数据库字段、检索 API 或论文结论。
|
||||
|
||||
以上描述 PR #9 独立评估阶段的范围;后续向量存储与精确检索集成另见 [VECTOR_API.md](VECTOR_API.md),不改变本页已记录的合成评估结果。
|
||||
|
||||
依据:[BGE small 官方模型说明](https://huggingface.co/BAAI/bge-small-zh-v1.5)、[BGE base 官方文件](https://huggingface.co/BAAI/bge-base-zh-v1.5/tree/f03589ceff5aac7111bd60cfc7d497ca17ecac65)、[Transformers 模型加载参数](https://huggingface.co/docs/transformers/main_classes/model#transformers.PreTrainedModel.from_pretrained)、[PyTorch CPU 安装](https://pytorch.org/get-started/locally/)、[Hugging Face 固定修订下载](https://huggingface.co/docs/huggingface_hub/guides/download)。
|
||||
@@ -0,0 +1,181 @@
|
||||
# 阶段 1 详细开发计划:环境联调与需求冻结
|
||||
|
||||
> 计划周期:2026 年 10 月 16 日—10 月 18 日
|
||||
> 阶段目标:让团队三人都能按 README 启动项目,并完成“前端 → API → PostgreSQL”的一次真实知识条目持久化闭环。
|
||||
> 本阶段不实现 Embedding、向量检索、RAG、联网搜索或真实大模型调用。
|
||||
|
||||
## 1. 阶段完成标准
|
||||
|
||||
全部满足才可将阶段 1 标记为完成:
|
||||
|
||||
1. Docker Desktop 引擎正常,`docker version` 同时显示 Client 和 Server;`nalati-postgres` 容器健康运行。
|
||||
2. 后端可通过默认生命周期启动,`http://127.0.0.1:8000/docs` 可访问,`GET /api/health` 返回 200。
|
||||
3. `POST /api/knowledge` 新增一条测试知识,`GET /api/knowledge` 能读取该条目;重启后端后数据仍存在。
|
||||
4. 前端 `npm run build` 通过,`http://localhost:5173` 可访问,页面显示 API 已连接。
|
||||
5. 在浏览器完成“新增知识 → 保存 → 列表刷新 → 重新打开页面仍存在”的完整流程。
|
||||
6. 三人分别在 Gitea Issue 中留下命令、结果、截图或日志位置;未提交真实密钥、真实密码和未授权资料。
|
||||
|
||||
## 2. 当前基线与阻塞
|
||||
|
||||
| 项目 | 当前状态 | 负责人 | 下一步 |
|
||||
|---|---|---|---|
|
||||
| Git 仓库、`main`、`dev` | 已准备 | 全员 | 每次工作从最新 `dev` 拉分支 |
|
||||
| 前端依赖与生产构建 | 已通过 | 吉海燕 | 启动 Vite,做浏览器端验收 |
|
||||
| 后端虚拟环境与依赖 | 已准备 | 范涵宇 | 数据库就绪后按默认生命周期启动 |
|
||||
| `/api/health` 路由 | 已验证 | 范涵宇 | 数据库就绪后复验默认启动 |
|
||||
| Docker Desktop/WSL | 受阻 | 范涵宇 | 在管理员 PowerShell 完成 WSL 前置、重启后启动 Docker Desktop |
|
||||
| 资料台账、审核规则 | 待创建 | 杨星萱 | 建立模板并明确状态含义 |
|
||||
|
||||
> Docker/WSL 未恢复前,可以完成前端构建、资料模板和文档工作;不得把数据库、新增知识持久化或“API 已连接”写成已通过。
|
||||
|
||||
## 3. 每日安排
|
||||
|
||||
### 10 月 16 日:环境恢复与单服务启动
|
||||
|
||||
| 成员 | 必做工作 | 交付物 | 验收证据 |
|
||||
|---|---|---|---|
|
||||
| 范涵宇 | 处理 Docker Desktop/WSL;启动 PostgreSQL;检查后端 `.env` 和 Python 环境 | 运行中的数据库容器、后端启动记录 | `docker compose ps`、`docker logs nalati-postgres`、后端启动日志 |
|
||||
| 吉海燕 | 从 `dev` 拉前端分支;执行依赖安装、构建和 Vite 启动;检查四个页面切换 | 前端启动记录、四页初始截图 | `npm run build` 输出、浏览器截图 |
|
||||
| 杨星萱 | 建立资料台账字段和审核规则;建立联调验收记录模板 | 台账模板、审核规则、验收记录模板 | 文档提交与 Issue 链接 |
|
||||
|
||||
### 10 月 17 日:API 与浏览器端持久化闭环
|
||||
|
||||
| 成员 | 必做工作 | 交付物 | 验收证据 |
|
||||
|---|---|---|---|
|
||||
| 范涵宇 | 复验 `/docs`、健康检查、知识新增/读取;记录 API 请求和响应 | T02/T03 验收记录 | 201 创建响应、列表读取响应、数据库容器日志 |
|
||||
| 吉海燕 | 按后端真实 Schema 调用新增与列表 API;处理加载、保存失败、API 未连接状态 | 前端联调截图或缺陷清单 | 新增后列表可见、刷新后仍存在的截图 |
|
||||
| 杨星萱 | 用非敏感测试资料完成一条待审核知识的登记;核对前端字段是否满足资料追溯需要 | 测试条目内容、字段反馈 | 测试条目 ID、字段核对记录 |
|
||||
|
||||
### 10 月 18 日:复现验收、问题冻结与交接
|
||||
|
||||
| 成员 | 必做工作 | 交付物 | 验收证据 |
|
||||
|---|---|---|---|
|
||||
| 范涵宇 | 重启后端或容器后复验数据;汇总未解决环境/接口问题 | 后端验收结论、遗留问题列表 | 重启后 `GET /api/knowledge` 结果 |
|
||||
| 吉海燕 | 在干净浏览器会话复验问答页、知识管理页和错误状态;整理可用于软著的阶段截图 | 前端验收结论、截图清单 | 浏览器操作步骤和截图 |
|
||||
| 杨星萱 | 汇总资料台账、审核规则与测试记录;确认阶段 2 所需字段和资料数量 | 阶段 2 知识收集清单 | Issue 中的验收记录与待办 |
|
||||
|
||||
## 4. 范涵宇执行清单:后端、数据库
|
||||
|
||||
### 4.1 Docker 与 PostgreSQL
|
||||
|
||||
在管理员完成 WSL/重启后,先执行:
|
||||
|
||||
```powershell
|
||||
cd E:\边疆红驿\border-red-post-nalati-ready
|
||||
docker version
|
||||
docker compose up -d db
|
||||
docker compose ps
|
||||
docker logs nalati-postgres
|
||||
```
|
||||
|
||||
通过条件:`docker version` 存在 Server 信息;`docker compose ps` 显示 `nalati-postgres` 运行且健康检查通过。若失败,只在 T03 Issue 记录完整错误,不删除 Docker 卷或重置数据。
|
||||
|
||||
### 4.2 后端启动与接口验证
|
||||
|
||||
```powershell
|
||||
cd E:\边疆红驿\border-red-post-nalati-ready\backend
|
||||
if (-not (Test-Path .env)) { Copy-Item .env.example .env }
|
||||
.\.venv\Scripts\Activate.ps1
|
||||
uvicorn app.main:app --reload
|
||||
```
|
||||
|
||||
另开 PowerShell 验证:
|
||||
|
||||
```powershell
|
||||
Invoke-RestMethod http://127.0.0.1:8000/api/health
|
||||
Invoke-RestMethod http://127.0.0.1:8000/api/knowledge
|
||||
$body = @{ title = '阶段1联调测试条目'; category = '其他'; content = '仅用于验证知识新增与持久化,不是正式事实资料。' } | ConvertTo-Json
|
||||
Invoke-RestMethod http://127.0.0.1:8000/api/knowledge -Method Post -ContentType 'application/json' -Body $body
|
||||
```
|
||||
|
||||
通过条件:健康检查返回 `status: ok`;新增请求返回 201;列表能看到返回的 ID 和标题。将测试条目保留并标注为测试,不作为正式检索资料。
|
||||
|
||||
### 4.3 给前端的交接内容
|
||||
|
||||
在 T02/T03 Issue 粘贴以下内容后再通知吉海燕联调:
|
||||
|
||||
- 后端地址、启动命令和 `/docs` 地址;
|
||||
- 最终确认的知识条目请求/响应 JSON;
|
||||
- 可用的 `verification_status` 状态值;
|
||||
- 当前已知限制与错误响应格式。
|
||||
|
||||
## 5. 吉海燕执行清单:前端与浏览器联调
|
||||
|
||||
### 5.1 拉取、构建与启动
|
||||
|
||||
```powershell
|
||||
cd E:\边疆红驿\border-red-post-nalati-ready
|
||||
git switch dev
|
||||
git pull --ff-only origin dev
|
||||
git switch -c feat/frontend-phase1-validation
|
||||
cd frontend
|
||||
npm ci
|
||||
npm run build
|
||||
npm run dev
|
||||
```
|
||||
|
||||
通过条件:构建命令退出码为 0;Vite 显示本地访问地址;浏览器可打开四个导航页。
|
||||
|
||||
### 5.2 浏览器检查顺序
|
||||
|
||||
1. 打开 `http://localhost:5173`,检查顶部/侧边导航能切换“智能问答、知识库管理、来源与更新、实验评估”。
|
||||
2. 后端未启动时,确认界面显示“API 未连接”或明确错误提示;不要修改前端来伪造连接成功。
|
||||
3. 后端启动后刷新页面,确认状态变为“API 已连接”。
|
||||
4. 在“知识库管理”点击“新增知识”,填写后端给出的测试条目,保存并确认列表出现该条目。
|
||||
5. 刷新浏览器页面,确认条目仍然存在;记录浏览器、时间和截图。
|
||||
6. 在“智能问答”发送一个测试问题,确认基础占位回答或后端错误能在页面被清楚展示,不白屏、不无限加载。
|
||||
|
||||
### 5.3 前端缺陷处理边界
|
||||
|
||||
- 请求、响应类型和错误信息统一维护在 `frontend/src/api.ts`;页面中不复制 `fetch` 逻辑。
|
||||
- 只按范涵宇确认的 API 字段改动;发现字段不一致时开 Issue,不在前端猜测字段或拼接数据库连接。
|
||||
- 每个修复独立提交,提交信息如 `fix: handle knowledge api error`;合并请求附构建结果与前后截图。
|
||||
- 当前阶段不实现检索模式选择、LLM 配置、真实实验图表或自动抓取来源,它们属于后续阶段。
|
||||
|
||||
## 6. 杨星萱执行清单:资料、审核与验收记录
|
||||
|
||||
### 6.1 资料台账字段
|
||||
|
||||
在 `docs/` 下的个人工作分支建立台账模板(Markdown、CSV 或学校统一表格均可),每条候选资料至少填写:
|
||||
|
||||
| 字段 | 要求 |
|
||||
|---|---|
|
||||
| 标题、类别 | 可识别且分类明确,如红色文化、旅游资源、民宿服务、交通出行 |
|
||||
| 来源名称、URL | 可回访来源;无 URL 时注明线下材料出处 |
|
||||
| 获取日期、更新时间 | 使用 `YYYY-MM-DD` |
|
||||
| 正文摘要 | 保留原意,不编造无法核实的具体事实 |
|
||||
| 核验状态 | `pending`、`verified` 或 `rejected` |
|
||||
| 核验依据、核验人、核验日期 | 说明为什么可以使用或拒绝使用 |
|
||||
|
||||
### 6.2 审核与测试资料规则
|
||||
|
||||
1. `pending`:已收集但未完成核验;可用于界面演示,不得进入正式检索。
|
||||
2. `verified`:来源可定位、信息完整且已人工核验;可交给范涵宇入库。
|
||||
3. `rejected`:来源不明、过期、重复或不可靠;保留记录但不入库。
|
||||
4. 阶段 1 的测试条目必须写明“测试”,不得伪装为那拉提真实资料。
|
||||
|
||||
### 6.3 联调验收记录
|
||||
|
||||
每次测试至少记录:测试日期、测试人、Git 提交、运行环境、操作步骤、预期结果、实际结果、截图/日志、结论和待处理问题。阶段 1 最少记录一次“知识新增—读取—页面刷新”测试。
|
||||
|
||||
## 7. Issue、提交与审阅顺序
|
||||
|
||||
| 任务 | 建议分支 | 提交/合并目标 | 审阅与批准 |
|
||||
|---|---|---|---|
|
||||
| T01 前端启动与浏览器验收 | `feat/frontend-phase1-validation` | 合并请求 → `dev` | 吉海燕提交;Codex 审阅并明确回复“批准合并”后由吉海燕合并 |
|
||||
| T02 健康检查与后端启动 | `feat/backend-phase1-api-validation` | 合并请求 → `dev` | 范涵宇提交;吉海燕审阅批准后合并 |
|
||||
| T03 Docker 与知识持久化 | `feat/backend-phase1-database` | 合并请求 → `dev` | 范涵宇提交;吉海燕审阅批准后合并 |
|
||||
| T04 审核规则与验收模板 | `docs/phase1-knowledge-governance` | 合并请求 → `dev` | 杨星萱提交;吉海燕审阅批准后合并 |
|
||||
|
||||
阶段 1 的 `dev → main` 合并请求只能在本文件第 1 节全部满足后创建,由吉海燕发起并确认。若 Docker/WSL 仍受阻,保留阶段状态为“进行中”,先记录阻塞和恢复动作,不跳过持久化验收。
|
||||
|
||||
## 8. 阶段结束交接包
|
||||
|
||||
阶段 1 结束时,三人共同整理以下内容到 Issue 或 `docs/`:
|
||||
|
||||
- 经过验证的启动命令与版本信息;
|
||||
- 前端四页和知识新增闭环截图;
|
||||
- API 请求/响应样例与已知限制;
|
||||
- 测试资料与正式资料的区分说明;
|
||||
- 未解决问题清单、负责人和预计恢复日期;
|
||||
- 阶段 2 可直接开始的接口字段与资料台账模板。
|
||||
+65
-8
@@ -4,16 +4,16 @@
|
||||
|
||||
| 编号 | 模块 | 任务 | 负责人 | 状态 |
|
||||
|---|---|---|---|---|
|
||||
| T01 | 前端 | 验证 Vue 页面启动、补充截图 | 吉海燕 | 进行中(生产构建已通过,待浏览器启动验证) |
|
||||
| T02 | 后端 | 验证 FastAPI `/docs`、`/api/health` | 范涵宇 | 进行中(`/api/health` 已通过,待数据库就绪后验证完整启动和 `/docs`) |
|
||||
| T03 | 数据库 | 启动 PostgreSQL 并验证知识入库 | 范涵宇 | 受阻(Docker Desktop/WSL 引擎尚未就绪) |
|
||||
| T01 | 前端 | 验证 Vue 页面启动、补充截图 | 吉海燕 | 进行中(生产构建已通过;登录、账户设置与角色导航已合入,待浏览器 Cookie 联调) |
|
||||
| T02 | 后端 | 验证 FastAPI `/docs`、`/api/health` | 范涵宇 | 已完成自动化/真实 HTTP 验证;待目标 Docker 环境复核 |
|
||||
| T03 | 数据库 | 启动 PostgreSQL 并验证知识入库 | 范涵宇 | 已用隔离 PostgreSQL 完成迁移与回归;本机 Docker Compose 验收受阻 |
|
||||
| T04 | 文档 | 明确接口字段、知识来源审核规则 | 杨星萱 | 待办 |
|
||||
| T05 | 知识库 | 设计 sources/documents/chunks/version 表 | 范涵宇 | 待办 |
|
||||
| T06 | 检索 | 实现关键词检索基线 | 范涵宇 | 待办 |
|
||||
| T07 | 向量 | 评估 Embedding 并实现 pgvector | 范涵宇 | 待办 |
|
||||
| T08 | RAG | 集成 LLM 与来源展示 | 范涵宇 | 待办 |
|
||||
| T05 | 知识库 | 设计 sources/documents/chunks/version 表 | 范涵宇 | 已完成(迁移、发布快照和接口已合入;待 Docker 联调) |
|
||||
| T06 | 检索 | 实现关键词检索基线 | 范涵宇 | 已完成(已核验知识 BM25 与检索历史已合入) |
|
||||
| T07 | 向量 | 评估 Embedding 并实现 pgvector | 范涵宇 | 已完成(本地模型、精确向量和 RRF 已合入;待真实领域评测) |
|
||||
| T08 | RAG | 集成 LLM 与来源展示 | 范涵宇 | 暂缓(已确定 DeepSeek,待明确恢复外部模型调用授权) |
|
||||
| T09 | 实验 | 构建可核验问题集及评分规范 | 杨星萱 | 待办 |
|
||||
| T10 | 工程 | 补充测试、Alembic、鉴权与部署 | 范涵宇(主)、全员 | 待办 |
|
||||
| T10 | 工程 | 补充测试、Alembic、鉴权与部署 | 范涵宇(主)、全员 | 进行中(迁移、认证、CSRF 与自动化回归已完成;本机 pytest/Docker/部署验收待补) |
|
||||
|
||||
## 2026-10-15 首次联调记录
|
||||
|
||||
@@ -22,3 +22,60 @@
|
||||
- 已使用临时 Uvicorn 服务(关闭数据库生命周期初始化)验证 `GET /api/health` 返回 `200` 和预期状态数据,验证后服务已关闭。
|
||||
- 本机 Docker CLI 已安装,但 Docker Desktop 引擎无法启动,且 WSL 尚未处于可用状态。因此尚未执行 PostgreSQL 容器启动、知识条目新增/读取、默认生命周期启动和浏览器端联调。
|
||||
- 后续操作:重启 Windows 后启动 Docker Desktop;确认 `docker version` 能显示 Server 信息,再在项目根目录运行 `docker compose up -d db`,随后继续 T02、T03 和 T01 的剩余验收。
|
||||
|
||||
## 2026-10-15 后端阶段 1–2 分支验收
|
||||
|
||||
分支:`feat/backend-knowledge-management`。以下为分支验收结果,须经既定审阅流程合入后再关闭任务;不覆盖前述其他环境的历史记录。
|
||||
|
||||
| 范围 | 已完成的验证 | 剩余事项 |
|
||||
|---|---|---|
|
||||
| T02 | 默认生命周期真实启动 Uvicorn;HTTP `/docs`、健康检查、数据库就绪检查通过 | 合并请求审阅、团队环境复核 |
|
||||
| T03 | 真实 PostgreSQL 16 测试实例完成迁移、知识录入和读取;后端进程重启后数据仍可读取 | 当前主机 Docker 守护进程未运行,启动需管理员权限;Compose 容器与浏览器端联调待复核 |
|
||||
| T05(阶段 2 范围) | 来源、文档、知识关联、版本、人工审核、软删除和历史快照;验证旧基础表迁移保留数据 | `chunks` 和索引版本留到检索阶段;字段与前端、资料负责人共同确认 |
|
||||
| T10(部分) | Alembic 迁移、并发版本检查、PostgreSQL 自动化测试、启动说明 | 登录鉴权、限流、生产部署留到后续工程阶段 |
|
||||
|
||||
验证环境:Python 3.11;使用可选 `pgserver` 启动独立 PostgreSQL 16.2 测试实例并启用 pgvector。该方式用于验证 SQL 和接口,不等同于 Docker 容器验收。
|
||||
|
||||
实际命令与结果(在 `backend/` 执行):
|
||||
|
||||
- `ruff check .`:通过。
|
||||
- `ruff format --check .`:通过。
|
||||
- `pytest -q`:22 passed;依赖库产生一条 TestClient 弃用提示,不影响本轮用例。
|
||||
- 覆盖:新库迁移、已有基础表接管、重复升级、阶段 2 降级后基础正文保留、ORM 与迁移结构一致、审核前置条件、正文/来源/文档变更后重新审核、历史快照、软删除引用保护、筛选与分页、并发写入冲突、跨域预检、真实 HTTP 与服务重启持久化。
|
||||
|
||||
所有测试正文、来源和审核人均为占位数据,临时数据库不入库。当前问答仍是占位接口,未开展论文对照实验。Issue、合并请求和最终批准结论须补入 Gitea 后按团队规则归档。
|
||||
|
||||
## 2026-10-15 合并后续开发:关键词检索基线
|
||||
|
||||
- 阶段 2 后端功能已通过普通功能分支的 [PR #4](https://gitea.kronecker.cc/saint/border-red-post-nalati/pulls/4) 合入 `dev`(合并提交 `4d437ca`),包含测试客户端依赖修正。旧 AGit PR #2 仍打开,不作为该批功能是否已合入的唯一判据。
|
||||
- 从最新 `dev` 创建 `feat/backend-keyword-retrieval`,推进 T06 的独立关键词接口与检索记录。T05 的持久化切分/索引版本、T07 的模型评估和向量检索仍待后续实现。
|
||||
- 当前主机 Docker 服务仍未运行;本轮继续使用隔离 PostgreSQL 做接口与迁移测试,不把容器或浏览器端联调标记为完成。独立任务 Issue 仍须补齐。
|
||||
- 新增检索接口、中文分词和 BM25 基线、有效审核筛选、原文片段、来源与版本、语料指纹及检索历史;前端调用层同步类型与接口,页面接入由前端继续联调。
|
||||
- 分支验证:`ruff check .`、`ruff format --check .`、`pytest -q`;本轮 42 项 PostgreSQL/单元/真实 HTTP 测试通过,覆盖手算分数、标题权重、排序复现、审核边界、来源失效、容量拒绝、迁移兼容和服务重启后的检索历史。前端 `npm run build` 通过,包含新增检索调用类型的检查。全部使用占位资料,不是论文实验结果。
|
||||
|
||||
## 2026-10-16 Issue #5:认证与角色权限
|
||||
|
||||
- 按继续开发的授权,在 `feat/backend-auth-sessions` 推进 T10 的认证部分;叠加 PR #6(`8ac9dca`),不等待其合并,不自行批准或合并。PR #6 与本分支分别保持可追踪记录。
|
||||
- 实现注册、登录、退出、当前会话、本人资料修改、改密及管理员用户列表;注册拒绝角色字段,知识/来源/文档管理及检索历史均由服务端要求 admin。
|
||||
- 新增 `0004_auth_sessions`,保存 Argon2id 密码哈希、会话摘要与到期/凭据版本、共享认证请求预算;不预置用户,提供本机隐藏密码输入的管理员初始化命令。迁移和回滚保留知识及检索记录。
|
||||
- 默认 Secure/HttpOnly/SameSite Cookie;写操作校验明确 Origin 及当前会话 CSRF。改密使所有旧会话失效,退出持久化撤销;参数错误不回显密码。字段、来源、CSRF 和前端交接见 [认证接口契约](AUTH_API.md)。
|
||||
- 实际验证:`ruff check .`、`ruff format --check .` 通过;`pytest -q` **70 passed**,包含认证、授权、CSRF、输入越权、过期/轮换/撤销、限额、迁移及既有知识/检索回归;真实 HTTP 服务重启验证会话恢复和退出后不再有效。前端 `npm run build` 通过。
|
||||
- 当前仍使用隔离 PostgreSQL 做自动化验证;Docker 容器、前端登录/角色页面及真实浏览器端流程仍待验收。远程 main 的注册原型最低密码提示需要按本契约同步到 15 字符,并接入带 Cookie/CSRF 的调用层。没有宣称已完成前端集成或生产部署。
|
||||
|
||||
## 2026-10-16 T05:持久化 chunks 与索引版本
|
||||
|
||||
- 分支 `feat/backend-chunk-index-versions` 叠加待审 PR #7(`f9827fe`),继承 #6;不等待合并继续后端阶段,同时保留各 PR 的审阅记录。
|
||||
- 新增 `0005_chunk_indexes`:版本、原文切分及出处、发布状态单例、逐次发布记录。构建 ready 快照与发布分开,保留字符偏移、内容哈希、知识/来源/文档版本和语料指纹。
|
||||
- 知识、来源、文档写入与构建/发布共用数据库行锁;目录修订与实际语料指纹一起决定当前性,旧构建/旧代数拒绝发布。构建或发布失败完整回滚,旧发布指针及历史不丢失。
|
||||
- 管理接口统一要求 admin 与写入 CSRF;前端只同步类型和调用层,页面仍待联调。现有关键词基线行为不变,向量、ANN、RRF 与 RAG 未在本阶段实现。
|
||||
- 实际验证:`ruff check .`、`ruff format --check .` 通过;`pytest -q` **91 passed**(隔离 PostgreSQL 16 + pgvector),包含真实锁竞争、失败构建/发布的数据库约束回滚、迁移保留账号/知识/检索记录、原文重建与来源快照、真实 HTTP 重启后的发布指针和 chunks 持久化。修正了发布时间在 UTC 与数据库时区之间表示不一致的问题,索引接口统一 UTC 输出。前端 `npm run build` 通过。
|
||||
- 完整请求/响应、切分/容量、失效策略及回滚说明见 [INDEX_API.md](INDEX_API.md)。本阶段仍是有限语料的同步文本快照;Docker/浏览器验收、独立 T05 Issue 补建及后续 Embedding 方案选择仍有待完成。
|
||||
|
||||
|
||||
## 2026-10-16 T07:合并后的 RRF 混合检索
|
||||
|
||||
- 本地模型评估和向量持久化/精确检索分别经 PR #11/#13 合入 `dev`;#9/#10 是被替代的旧 AGit 入口。读取已确认合并结果后,从最新 `dev` 创建 `feat/backend-rrf-retrieval`,遵循每阶段 PR 合并后才开发下一阶段。
|
||||
- 新增公开 chunk 正文 BM25 与 RRF 接口,三种 chunk 模式限定同一文本快照及分类;原条目关键词接口保留独立口径。固定 k=60、等权、每路候选预算,记录两路完整候选排名/原始分数、贡献、模型空间、切分/BM25/分词参数与出处。前端仅同步调用与历史类型。
|
||||
- 模型编码在发布锁外,完成后重新检查审核语料、文本/向量代数与空间;两路排序和记录保存受同一事务保护。竞争、模型或写入失败不保存部分混合结果;2000 chunks 容量与原向量阶段一致。没有新增数据库迁移。完整字段、比较口径及回滚见 [HYBRID_API.md](HYBRID_API.md)。
|
||||
- 常规测试 `pytest -q` **180 passed, 1 skipped**(真实模型专项须显式启用);新增 27 项覆盖手算排序/融合、候选截断/缺席/同分、相同粒度及分类、原文、历史、权限、编码期间锁释放和版本竞争、损坏/失效快照、失败回滚与容量。`ruff check`、`ruff format --check` 和前端 `npm run build` 通过。真实模型专项在独立 CPU 环境另执行 **1 passed**:固定 BGE small 经真实 HTTP 完成向量/混合/关键词 chunk 查询,进程重启后两路候选排名、结果和历史保持一致。首次专项请求因 candidate_k 小于默认 top_k 被按契约拒绝,修正测试请求后通过;未改变该输入约束。
|
||||
- Issue #5 本轮复查无新评论,后端实现已合入,前端登录/CSRF/角色页面及浏览器流程仍待交接;Docker 容器、核验领域语料/问题集、正式模型冻结与领域评测同样未验收。所有新增自动化资料均为合成测试,不作为文旅事实或论文分数。下一阶段须等待本轮 PR 真正合入 `dev`。
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
# T07 向量快照、发布与精确检索
|
||||
|
||||
本分支 `feat/backend-vector-retrieval` 组合待审 #8 的文本版本与 #9 的固定版本 CPU Provider,继承 #7/#6。合并到 `dev` 前须按依赖审阅;不自行批准或合并。双方阶段文档均保留,独立阶段差异从组合基线 `bd0b738` 起查看。
|
||||
|
||||
交付 pgvector 存储、空间约束、独立发布、精确余弦检索与历史。关键词 `POST /api/retrieval` 保持原请求/分数/条目粒度;向量使用新的 `POST /api/retrieval/vector`,粒度为 chunk。本阶段尚未实现 RRF、ANN、RAG、持久化后台任务或正式领域评测。
|
||||
|
||||
## 配置与迁移
|
||||
|
||||
执行 `alembic upgrade head` 到 `0006_vectors`。迁移增加 `embedding_spaces`、`vector_versions`、`chunk_vectors`、`vector_state` 和 `vector_publications`,以及 chunks 的复合唯一约束。保留账号、目录、原文版本及既有检索日志,不预置模型或向量。
|
||||
|
||||
扩展类型/函数/距离算子显式引用 `public.vector`,业务表仍放在连接的当前 Schema,不把 public 加入测试搜索路径。已有 vector 扩展须位于 public;其他安装位置会明确报错,迁移不会移动共享扩展。新数据库可先由有权限的数据库管理员执行 `CREATE EXTENSION IF NOT EXISTS vector WITH SCHEMA public`。
|
||||
|
||||
默认 `EMBEDDING_ENABLED=false`,普通 API 启动不导入 PyTorch 或加载权重。模型相关请求返回 503,不回退占位向量或伪造空结果;关键词和管理元数据接口可继续使用。启用前按 [本地评估说明](LOCAL_EMBEDDING_EVALUATION.md) 安装独立 CPU 环境、缓存固定修订;该运行环境还需安装 `backend/requirements.txt` 的 API 依赖。
|
||||
|
||||
| 变量 | 默认值 | 作用 |
|
||||
|---|---|---|
|
||||
| `EMBEDDING_ENABLED` | false | 显式启用本地编码 |
|
||||
| `EMBEDDING_MODEL` | small | 仅 small/base;完整修订、维度等由固定 Spec 决定 |
|
||||
| `EMBEDDING_CACHE_DIR` | 未设置 | 固定模型缓存目录;空值使用库默认缓存 |
|
||||
| `EMBEDDING_BATCH_SIZE` | 8 | 1–32 |
|
||||
| `EMBEDDING_THREADS` | 2 | 1–4 个 CPU 线程 |
|
||||
|
||||
开发资源优先暂用 small,正式模型仍待已核验领域资料/问题集评测。API 始终只从本机固定修订缓存读取,禁止在请求中下载权重,不接收模型路径、任意仓库名或用户向量。CPU Provider 首次使用才加载,单进程同时只允许一个编码调用,繁忙立即返回 503;当前建议单 worker 本机运行,不能据此宣称已实现跨进程队列或生产并发容量。
|
||||
|
||||
Linux 仓库根目录示例(先完成模型准备及数据库迁移;本机 Cookie/Origin 配置仍按认证说明设置):
|
||||
|
||||
```bash
|
||||
task_vector_root=$(git rev-parse --absolute-git-dir)
|
||||
uv pip install --python "$task_vector_root/embedding-eval-venv/bin/python" \
|
||||
-r backend/requirements.txt
|
||||
export EMBEDDING_ENABLED=true EMBEDDING_MODEL=small
|
||||
export EMBEDDING_CACHE_DIR="$task_vector_root/embedding-model-cache"
|
||||
export HF_HUB_OFFLINE=1 HF_HUB_DISABLE_TELEMETRY=1 HF_HUB_DISABLE_IMPLICIT_TOKEN=1
|
||||
export TOKENIZERS_PARALLELISM=false
|
||||
cd backend
|
||||
"$task_vector_root/embedding-eval-venv/bin/python" -m uvicorn app.main:app --host 127.0.0.1
|
||||
```
|
||||
|
||||
环境、模型权重和实际日志只留本机,不提交。普通 API 环境不自动增加 PyTorch 依赖。
|
||||
|
||||
## 构建与发布协议
|
||||
|
||||
1. 通过 `/api/indexes` 构建并发布当前有效审核语料的文本版本。建议开发先用 `chunk_size=200, overlap=20`;实际 tokenizer 仍须检查,字符数不保证 token 数。
|
||||
2. 管理员向 `POST /api/vector-indexes` 提交 `{"text_index_version_id":"<已发布文本版本 UUID>"}`。服务仅接受当前有效的文本发布版本;整份快照最多 2000 chunks,超限返回 503,不截断语料。
|
||||
3. 服务短暂锁住目录/文本状态,复核实际审核语料指纹、原文窗口、偏移、哈希、出处和完整数量;复制输入后提交并释放事务。模型加载与正文编码全部在锁外运行。
|
||||
4. 编码成功且向量数量/维度/有限数/L2 范数通过后,再取得同一状态锁,重新校验文本代数、目录修订、当前指针、语料指纹及实际 chunks。变化则 409,整批不入库。成功只保存 ready 向量版本,不自动发布。
|
||||
5. 管理员向 `POST /api/vector-indexes/{id}/publish` 提交 `{"expected_generation":0}`。向量代数独立于文本代数;原子切换当前指针、前后版本状态和发布历史。旧代数、过期文本或不完整向量集均拒绝。相同当前版本和当前代数为幂等读取。
|
||||
|
||||
上述构建同步执行,没有后台任务 ID 或自动重试。超过模型 token 上限返回 422,应重新切分并发布文本,再重建;不会静默截断。编码不可用/繁忙为 503;数据库约束或版本竞争为 409。失败不会改写旧指针或留下部分向量版本。空的有效审核语料允许生成/发布 0 chunks 的版本;随后检索如实返回 `no_evidence`。
|
||||
|
||||
目录变化会使既有文本和向量版本过期;文本切换即使内容相同也要求对应新文本版本的向量。旧指针和日志保留为历史,公开检索对过期状态返回 409,不继续使用旧审核依据。切换模型配置后,若当前空间不一致,也须重新构建/发布相应空间或恢复匹配的运行配置。
|
||||
|
||||
## 数据库空间隔离
|
||||
|
||||
`space_id` 为固定 Spec 的 SHA-256,包含模型名、完整 revision、维度、token 上限、查询指令、编码实现(CLS/L2/CPU FP32)及权重格式。同维度的不同修订/指令/配置仍是不同空间。
|
||||
|
||||
数据库同时约束:
|
||||
|
||||
- 版本的 `(space_id, dimension)` 必须引用同一 EmbeddingSpace。
|
||||
- 每条向量的 `(version_id, text_index_version_id, space_id, dimension)` 必须引用同一向量版本。
|
||||
- `(chunk_id, text_index_version_id)` 必须引用对应文本版本中的 chunk。
|
||||
- `public.vector_dims` 必须等于声明维度;`public.vector_norm` 须在 `0.9999–1.0001`,pgvector 拒绝非有限元素。
|
||||
|
||||
使用无固定维度的 vector 列容纳独立空间,查询显式限定版本、空间、文本版本和维度,然后使用余弦距离。当前为有限语料精确扫描,没有 HNSW/IVFFlat;模型空间和向量发布均不由客户端参数任意指定。
|
||||
|
||||
## 接口与字段交接
|
||||
|
||||
`/api/vector-indexes` 下所有读写要求 admin;写入沿用当前 Cookie、CSRF 和明确 Origin。元数据不要求加载模型。分页使用 `{items,total,page,page_size}`,`page_size` 最大 100。
|
||||
|
||||
| 方法与路径 | 请求/用途 | 响应 |
|
||||
|---|---|---|
|
||||
| POST `/api/vector-indexes` | `text_index_version_id` | 201,VectorVersion |
|
||||
| GET `/api/vector-indexes` | page/page_size | 版本分页 |
|
||||
| GET `/api/vector-indexes/{id}` | UUID | VectorVersion,缺失 404 |
|
||||
| GET `/api/vector-indexes/active` | 当前元数据 | generation/text_generation/catalog_revision/is_current/active_version |
|
||||
| GET `/api/vector-indexes/spaces` | page/page_size | 空间 id/dimension/spec |
|
||||
| POST `/api/vector-indexes/{id}/publish` | expected_generation | 当前向量元数据 |
|
||||
| GET `/api/vector-indexes/publications` | page/page_size | 代数、前后向量版本、actor_id、UTC 时间 |
|
||||
| POST `/api/retrieval/vector` | 下方请求 | VectorRetrievalResponse |
|
||||
|
||||
VectorVersion 字段:`id/text_index_version_id/space_id/dimension/chunk_count/status/created_by/created_at/published_at`,状态为 ready/published/superseded;时间统一 UTC ISO 8601。发布历史的 `version_id` 是向量版本,不能与文本发布历史混用。
|
||||
|
||||
公开检索请求示例:
|
||||
|
||||
```json
|
||||
{"query":"待检索问题","mode":"vector","top_k":5,"category":"交通"}
|
||||
```
|
||||
|
||||
`mode` 可省略,固定为 vector;query 1–500 字符,top_k 1–20,category 可省略。未知字段拒绝。流程在查询编码前后均复核当前发布/审核状态和完整向量集;编码时不持有目录发布锁,期间任何发布代数变化都会返回 409,重试使用新快照。
|
||||
|
||||
响应记录:`run_id/query/mode/status/top_k/category/vector_version_id/text_index_version_id/generation/text_generation/corpus_fingerprint/corpus_size/total_hits/results/elapsed_ms/created_at/config`。`config` 固定 scorer=`pgvector-cosine-exact-v1`、ranking_unit=`chunk`,并保存完整空间 Spec。
|
||||
|
||||
每个 result 沿用关键词出处字段,再加 `chunk_id/chunk_ordinal`;snippet 为完整原文 chunk,snippet_start/end 是原正文的 Unicode 字符偏移(左闭右开),matched_terms 为空数组。score 为 `1 - cosine_distance`,不是正确率或置信度,也未设领域无依据阈值。按实际距离、knowledge_id、chunk_ordinal、chunk_id 排序;同一知识可出现多个 chunks,不能把结果数量当作命中的知识数。
|
||||
|
||||
`corpus_size/total_hits` 为分类筛选后的候选 chunk 数,不是超过置信阈值的数量;corpus_fingerprint 是整个已发布文本语料指纹,分类另行保存。耗时包括两次审核/快照检查、查询编码(首次可含模型加载)、SQL 距离排序与片段生成,不含最终日志提交及网络。不同于关键词基线的条目粒度,正式对照前必须统一评测单位。
|
||||
|
||||
成功和零候选均保存 `retrieval_runs` 不可变快照。既有历史接口现在返回 keyword/vector 判别联合,仍限 admin;不改写过去结果,也不因历史可读取就代表资料仍有效。前端调用层已同步 `services/vectors.ts` 与历史联合类型,页面和浏览器仍待接入验收。
|
||||
|
||||
## 回滚与验证
|
||||
|
||||
备份后 `alembic downgrade 0005_chunk_indexes` 删除本阶段空间、向量、向量状态和发布历史,保留文本 chunks/发布、账号、知识及所有检索结果快照。切回对应旧版后端前须处理其历史接口只认识 keyword 响应的兼容性;重升后向量指针为空,需要重新构建,不会复原被删权重结果。当前 API 启动会拒绝旧 Schema。
|
||||
|
||||
默认测试用明确的三维坐标 fixture 验证 SQL 数学、空间隔离、并发与回滚,绝不代表实际模型质量。真实模型贯通用例显式启用,在独立模型环境运行:
|
||||
|
||||
```bash
|
||||
task_vector_root=$(git rev-parse --absolute-git-dir)
|
||||
uv pip install --python "$task_vector_root/embedding-eval-venv/bin/python" \
|
||||
-r backend/requirements-dev.txt pgserver
|
||||
RUN_LOCAL_EMBEDDING_TEST=1 EMBEDDING_CACHE_DIR="$task_vector_root/embedding-model-cache" \
|
||||
"$task_vector_root/embedding-eval-venv/bin/python" -m pytest -q \
|
||||
backend/tests/test_vector_runtime.py
|
||||
```
|
||||
|
||||
该用例用合成测试文字,验证真实 small 的 512 维编码、HTTP 构建/发布/查询、pgvector 持久化及新进程重启后的空间、发布、命中和历史一致;不是领域分数或生产性能实验。Docker 容器、前端浏览器、RRF 和领域验收保持待办。
|
||||
|
||||
实现依据:[pgvector 精确查询与余弦算子](https://github.com/pgvector/pgvector#querying)、[混合维度存储约束](https://github.com/pgvector/pgvector#can-i-store-vectors-with-different-dimensions-in-the-same-column)、[SQLAlchemy 集成](https://github.com/pgvector/pgvector-python#sqlalchemy)。
|
||||
Generated
+23
-1
@@ -9,7 +9,8 @@
|
||||
"version": "0.1.0",
|
||||
"dependencies": {
|
||||
"lucide-vue-next": "^0.468.0",
|
||||
"vue": "^3.5.13"
|
||||
"vue": "^3.5.13",
|
||||
"vue-router": "^4.6.4"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "^22.10.2",
|
||||
@@ -1043,6 +1044,12 @@
|
||||
"he": "^1.2.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@vue/devtools-api": {
|
||||
"version": "6.6.4",
|
||||
"resolved": "https://registry.npmmirror.com/@vue/devtools-api/-/devtools-api-6.6.4.tgz",
|
||||
"integrity": "sha512-sGhTPMuXqZ1rVOk32RylztWkfXTRhuS7vgAKv0zjqk8gbsHkJ7xfFf+jbySxt7tWObEJwyKaHMikV/WGDiQm8g==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/@vue/language-core": {
|
||||
"version": "2.2.12",
|
||||
"resolved": "https://registry.npmmirror.com/@vue/language-core/-/language-core-2.2.12.tgz",
|
||||
@@ -1565,6 +1572,21 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"node_modules/vue-router": {
|
||||
"version": "4.6.4",
|
||||
"resolved": "https://registry.npmmirror.com/vue-router/-/vue-router-4.6.4.tgz",
|
||||
"integrity": "sha512-Hz9q5sa33Yhduglwz6g9skT8OBPii+4bFn88w6J+J4MfEo4KRRpmiNG/hHHkdbRFlLBOqxN8y8gf2Fb0MTUgVg==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@vue/devtools-api": "^6.6.4"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://github.com/sponsors/posva"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"vue": "^3.5.0"
|
||||
}
|
||||
},
|
||||
"node_modules/vue-tsc": {
|
||||
"version": "2.2.12",
|
||||
"resolved": "https://registry.npmmirror.com/vue-tsc/-/vue-tsc-2.2.12.tgz",
|
||||
|
||||
+21
-4
@@ -1,6 +1,23 @@
|
||||
{
|
||||
"name": "nalati-rag-frontend", "private": true, "version": "0.1.0", "type": "module",
|
||||
"scripts": {"dev": "vite", "build": "vue-tsc -b && vite build", "preview": "vite preview"},
|
||||
"dependencies": {"lucide-vue-next": "^0.468.0", "vue": "^3.5.13"},
|
||||
"devDependencies": {"@types/node": "^22.10.2", "@vitejs/plugin-vue": "^5.2.1", "typescript": "~5.7.2", "vite": "^6.0.5", "vue-tsc": "^2.2.0"}
|
||||
"name": "nalati-rag-frontend",
|
||||
"private": true,
|
||||
"version": "0.1.0",
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"dev": "vite",
|
||||
"build": "vue-tsc -b && vite build",
|
||||
"preview": "vite preview"
|
||||
},
|
||||
"dependencies": {
|
||||
"lucide-vue-next": "^0.468.0",
|
||||
"vue": "^3.5.13",
|
||||
"vue-router": "^4.6.4"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "^22.10.2",
|
||||
"@vitejs/plugin-vue": "^5.2.1",
|
||||
"typescript": "~5.7.2",
|
||||
"vite": "^6.0.5",
|
||||
"vue-tsc": "^2.2.0"
|
||||
}
|
||||
}
|
||||
|
||||
+20
-9
@@ -1,20 +1,31 @@
|
||||
<script setup lang="ts">
|
||||
import { computed, onMounted, ref } from 'vue'
|
||||
import { computed, onMounted, ref, watch } from 'vue'
|
||||
import { Activity, BookOpen, Bot, CheckCircle2, ChevronRight, Database, FileText, Globe2, Link2, MessageSquareText, Plus, Search, Send, ShieldCheck, Sparkles, AlertCircle } from 'lucide-vue-next'
|
||||
import { askQuestion, createKnowledge, getHealth, getKnowledge } from './api'
|
||||
import { createKnowledge, createSource, getHealth, getKnowledge, getSources, type SourceItem } from './api'
|
||||
import { retrieveHybrid } from './services/hybrid'
|
||||
import { getCurrentUser } from './services/auth'
|
||||
import { useRouter } from 'vue-router'
|
||||
import type { SessionUser } from './types/auth'
|
||||
type Page='chat'|'knowledge'|'sources'|'evaluation'
|
||||
const page=ref<Page>('chat'), apiStatus=ref('检测中'), question=ref(''), busy=ref(false), search=ref(''), showForm=ref(false), error=ref(''), notice=ref('')
|
||||
const router=useRouter(),currentUser=ref<SessionUser>(),page=ref<Page>('chat'), apiStatus=ref('检测中'), question=ref(''), busy=ref(false), search=ref(''), showForm=ref(false), error=ref(''), notice=ref('')
|
||||
const messages=ref([{role:'assistant',content:'你好!我是边疆红驿文旅问答原型。当前后端仅为基础接口,尚未接入知识检索和大语言模型。'}] as {role:string;content:string;sources?:{title:string;source_url?:string|null}[]}[])
|
||||
const items=ref<any[]>([]), form=ref({title:'',category:'旅游资源',content:'',source_url:''})
|
||||
const nav=[{id:'chat',label:'智能问答',icon:MessageSquareText},{id:'knowledge',label:'知识库管理',icon:Database},{id:'sources',label:'来源与更新',icon:Globe2},{id:'evaluation',label:'实验评估',icon:Activity}] as const
|
||||
const sources=ref<SourceItem[]>([]), sourceSearch=ref(''), showSourceForm=ref(false), sourceForm=ref({name:'',source_url:'',notes:''})
|
||||
const nav=[{id:'chat',label:'智能问答',icon:MessageSquareText},{id:'knowledge',label:'知识库管理',icon:Database,adminOnly:true},{id:'sources',label:'来源与更新',icon:Globe2,adminOnly:true},{id:'evaluation',label:'实验评估',icon:Activity,adminOnly:true}] as const
|
||||
const visibleNav=computed(()=>nav.filter(item=>!('adminOnly' in item)||!item.adminOnly||currentUser.value?.role==='admin'))
|
||||
const filtered=computed(()=>items.value.filter(x=>`${x.title} ${x.category} ${x.content}`.toLowerCase().includes(search.value.toLowerCase())))
|
||||
async function refresh(){try{items.value=await getKnowledge();error.value=''}catch{error.value='无法读取后端知识库,请确认 API 和 PostgreSQL 已启动。'}}
|
||||
onMounted(async()=>{try{await getHealth();apiStatus.value='已连接'}catch{apiStatus.value='未连接'}await refresh()})
|
||||
async function send(){const q=question.value.trim();if(!q||busy.value)return;messages.value.push({role:'user',content:q});question.value='';busy.value=true;try{const r=await askQuestion(q);messages.value.push({role:'assistant',content:r.answer,sources:r.sources})}catch{messages.value.push({role:'assistant',content:'暂时无法连接后端 API。请确认 FastAPI 已启动。'})}finally{busy.value=false}}
|
||||
async function save(){error.value='';notice.value='';try{await createKnowledge({...form.value,...(form.value.source_url?{source_url:form.value.source_url}:{})});form.value={title:'',category:'旅游资源',content:'',source_url:''};showForm.value=false;notice.value='已保存为待审核条目,尚未进入检索索引。';await refresh()}catch(e){error.value=e instanceof Error?e.message:'保存失败'}}
|
||||
async function refreshSources(){try{sources.value=(await getSources(sourceSearch.value)).items;error.value=''}catch{error.value='无法读取来源列表,请确认 API 和 PostgreSQL 已启动。'}}
|
||||
async function saveSource(){error.value='';notice.value='';if(!sourceForm.value.name.trim()){error.value='请填写来源名称。';return}if(!validUrl(sourceForm.value.source_url)){error.value='来源网址必须是有效的 http(s) 地址。';return}try{await createSource({...sourceForm.value,name:sourceForm.value.name.trim(),...(sourceForm.value.source_url?{source_url:sourceForm.value.source_url}:{})});sourceForm.value={name:'',source_url:'',notes:''};showSourceForm.value=false;notice.value='来源已登记,可继续创建关联文档和知识条目。';await refreshSources()}catch(e){error.value=e instanceof Error?e.message:'保存失败'}}
|
||||
let sourceSearchTimer:number|undefined
|
||||
watch(sourceSearch,()=>{window.clearTimeout(sourceSearchTimer);sourceSearchTimer=window.setTimeout(refreshSources,260)})
|
||||
onMounted(async()=>{try{currentUser.value=await getCurrentUser()}catch{router.replace('/login');return}try{await getHealth();apiStatus.value='已连接'}catch{apiStatus.value='未连接'}if(currentUser.value.role==='admin')await Promise.all([refresh(),refreshSources()])})
|
||||
async function send(){const q=question.value.trim();if(!q||busy.value)return;messages.value.push({role:'user',content:q});question.value='';busy.value=true;try{const r=await retrieveHybrid({query:q,top_k:5,candidate_k:50});const content=r.status==='no_evidence'?'未检索到可用的已核验知识依据。请调整问题或补充审核后的资料。':`已通过 RRF 混合检索命中 ${r.total_hits} 条依据(关键词 ${r.lexical_total_hits},向量 ${r.vector_total_hits},${r.elapsed_ms.toFixed(1)} ms)。当前仅展示检索证据,尚未接入大语言模型生成回答。`;messages.value.push({role:'assistant',content,sources:r.results.map(hit=>({title:`${hit.title}:${hit.snippet}`,source_url:hit.source_url}))})}catch(e){messages.value.push({role:'assistant',content:e instanceof Error?e.message:'暂时无法连接混合检索服务。'})}finally{busy.value=false}}
|
||||
function validUrl(value:string){try{if(value)new URL(value);return true}catch{return false}}
|
||||
async function save(){error.value='';notice.value='';if(!form.value.title.trim()||!form.value.content.trim()){error.value='请填写知识标题和正文。';return}if(!validUrl(form.value.source_url)){error.value='来源网址必须是有效的 http(s) 地址。';return}try{await createKnowledge({...form.value,title:form.value.title.trim(),content:form.value.content.trim(),...(form.value.source_url?{source_url:form.value.source_url}:{})});form.value={title:'',category:'旅游资源',content:'',source_url:''};showForm.value=false;notice.value='已保存为待审核条目,尚未进入检索索引。';await refresh()}catch(e){error.value=e instanceof Error?e.message:'保存失败'}}
|
||||
</script>
|
||||
<template><div class="shell"><aside class="sidebar"><div class="brand"><div class="mark"><Bot :size="23"/></div><div><b>边疆红驿</b><small>NALATI KNOWLEDGE HUB</small></div></div><p class="muted label">工作空间</p><nav><button v-for="n in nav" :key="n.id" :class="{active:page===n.id}" @click="page=n.id"><component :is="n.icon" :size="18"/>{{n.label}}<ChevronRight v-if="page===n.id" class="chev" :size="15"/></button></nav><div class="side-bottom"><div class="tip"><ShieldCheck :size="17"/><div><b>可信知识服务</b><p>优先展示可核验来源,信息不足时明确提示。</p></div></div><p class="api">● API {{apiStatus}} <span>v0.1.0</span></p></div></aside><main><header><span>边疆红驿 / {{nav.find(n=>n.id===page)?.label}}</span><div><span class="pill">● 原型阶段</span><i>研</i></div></header><section class="content">
|
||||
<template><div class="shell"><aside class="sidebar"><div class="brand"><div class="mark"><Bot :size="23"/></div><div><b>边疆红驿</b><small>NALATI KNOWLEDGE HUB</small></div></div><p class="muted label">功能列表</p><nav><button v-for="n in visibleNav" :key="n.id" :class="{active:page===n.id}" @click="page=n.id"><component :is="n.icon" :size="18"/>{{n.label}}<ChevronRight v-if="page===n.id" class="chev" :size="15"/></button><button @click="router.push('/profile')"><ShieldCheck :size="18"/>个人中心</button><button v-if="currentUser?.role==='admin'" @click="router.push('/admin')"><Activity :size="18"/>用户管理</button></nav><div class="quick-actions"><p class="muted label">快捷操作</p><button v-if="page==='knowledge'" class="side-action" @click="showForm=true"><Plus :size="16"/>新增知识</button><button v-if="page==='sources'" class="side-action" @click="showSourceForm=true"><Plus :size="16"/>新增来源</button><p v-if="page==='chat'" class="side-hint">选择示例问题,或直接开始提问。</p><p v-if="page==='evaluation'" class="side-hint">实验结果只展示真实运行数据。</p></div><div class="side-bottom"><div class="tip"><ShieldCheck :size="17"/><div><b>{{currentUser?.role==='admin'?'管理员':'已登录用户'}}</b><p>{{currentUser?.nickname||currentUser?.username||'正在载入账户'}}</p></div></div><p class="api">● API {{apiStatus}} <span>v0.1.0</span></p></div></aside><main><header><span>边疆红驿 / {{nav.find(n=>n.id===page)?.label}}</span><div><span class="pill">● {{currentUser?.role==='admin'?'管理员':'用户'}}</span><i>研</i></div></header><section class="content">
|
||||
<div v-if="page==='chat'"><div class="heading"><div><small class="eyebrow"><Sparkles :size="14"/> LOCAL KNOWLEDGE ASSISTANT</small><h1>文旅知识,问得明白</h1><p>面向那拉提文旅与民宿服务的知识问答原型</p></div><span class="chip"><ShieldCheck :size="15"/> 来源可追溯 · 设计目标</span></div><div class="chat-layout"><div class="card chat"><div class="chat-head"><div class="mark"><Bot :size="20"/></div><div><b>边疆红驿助手</b><small>问答服务原型</small></div></div><div class="messages"><div v-for="(m,i) in messages" :key="i" class="message" :class="m.role"><small>{{m.role==='assistant'?'智能助手':'你'}}</small><p>{{m.content}}</p><div v-if="m.sources?.length" class="sources"><b><Link2 :size="14"/> 参考来源</b><a v-for="s in m.sources" :key="s.title" :href="s.source_url ?? undefined" target="_blank">{{s.title}}</a></div></div><p v-if="busy">正在请求后端…</p></div><form class="composer" @submit.prevent="send"><textarea v-model="question" placeholder="输入你想了解的文旅问题…" rows="2" @keydown.enter.exact.prevent="send"/><div><small>当前为基础接口,尚未接入真实检索</small><button class="primary" :disabled="busy||!question.trim()"><Send :size="15"/> 发送</button></div></form></div><aside class="suggest"><b><Sparkles :size="16"/> 你可以这样问</b><button @click="question='那拉提有哪些值得了解的旅游资源?'"><Globe2/>旅游资源 <ChevronRight/></button><button @click="question='有哪些经过核验的红色文化资料?'"><BookOpen/>红色文化 <ChevronRight/></button><button @click="question='民宿服务信息应如何核验?'"><ShieldCheck/>民宿服务 <ChevronRight/></button><div class="notice"><b><AlertCircle :size="15"/> 信息说明</b><p>本项目仍处于原型阶段。演示内容不代表已核验的地方文旅事实。</p></div></aside></div></div>
|
||||
<div v-else-if="page==='knowledge'"><div class="heading"><div><small class="eyebrow"><Database :size="14"/> KNOWLEDGE MANAGEMENT</small><h1>知识库管理</h1><p>集中管理领域知识、来源和审核状态。</p></div><button class="primary" @click="showForm=!showForm"><Plus :size="16"/> 新增知识</button></div><div v-if="showForm" class="card form-card"><b>新增知识条目</b><form class="knowledge-form" @submit.prevent="save"><label>标题<input v-model="form.title" required maxlength="250"/></label><label>分类<select v-model="form.category"><option>红色文化</option><option>旅游资源</option><option>民宿服务</option><option>交通出行</option><option>其他</option></select></label><label class="wide">正文<textarea v-model="form.content" required rows="3"/></label><label class="wide">来源网址(可选)<input v-model="form.source_url" type="url" placeholder="https://..."/></label><div class="wide actions"><button type="button" class="secondary" @click="showForm=false">取消</button><button class="primary">保存条目</button></div></form></div><div class="stats"><div class="card stat"><small>知识条目</small><strong>{{items.length}}</strong><span>来自当前数据库</span></div><div class="card stat"><small>已核验</small><strong>{{items.filter(x=>x.verification_status==='verified').length}}</strong><span>需经过真实审核</span></div><div class="card stat"><small>待审核</small><strong>{{items.filter(x=>x.verification_status==='pending').length}}</strong><span>尚未进入检索索引</span></div></div><p v-if="notice" class="success">{{notice}}</p><p v-if="error" class="error">{{error}}</p><div class="card table-card"><div class="table-head"><div><b>知识条目</b><p>列表来自后端接口,不含预设真实资料。</p></div><label class="search"><Search :size="15"/><input v-model="search" placeholder="搜索知识"/></label></div><div class="scroll"><table><thead><tr><th>标题</th><th>分类</th><th>来源</th><th>状态</th><th>创建时间</th></tr></thead><tbody><tr v-for="x in filtered" :key="x.id"><td><b>{{x.title}}</b><small>{{x.content.slice(0,45)}}</small></td><td>{{x.category}}</td><td><a v-if="x.source_url" :href="x.source_url" target="_blank">打开来源</a><span v-else>未填写</span></td><td><em>{{x.verification_status==='pending'?'待审核':x.verification_status}}</em></td><td>{{new Date(x.created_at).toLocaleDateString()}}</td></tr><tr v-if="!filtered.length"><td colspan="5" class="empty">暂无条目,点击“新增知识”添加。</td></tr></tbody></table></div></div></div>
|
||||
<div v-else-if="page==='sources'"><div class="heading"><div><small class="eyebrow"><Globe2 :size="14"/> SOURCE OPERATIONS</small><h1>来源与更新</h1><p>为后续网页采集、审核、向量生成和版本切换预留界面。</p></div><span class="pill">● 功能规划中</span></div><div class="notice wide"><b><AlertCircle :size="16"/> 当前状态</b><p>此页面为静态原型。尚未连接网页抓取、定时任务或 Embedding 流程,不会自动更新知识。</p></div><div class="stats"><div class="card stat"><small>来源总数</small><strong>—</strong><span>待接入</span></div><div class="card stat"><small>待处理任务</small><strong>—</strong><span>后台队列尚未实现</span></div><div class="card stat"><small>最近更新</small><strong>—</strong><span>待接入更新记录</span></div></div><div class="card roadmap"><b>后续更新流程</b><div><span>01 获取来源</span><ChevronRight/><span>02 核验内容</span><ChevronRight/><span>03 生成向量</span><ChevronRight/><span>04 发布版本</span></div></div></div>
|
||||
<div v-else-if="page==='sources'"><div class="heading"><div><small class="eyebrow"><Globe2 :size="14"/> SOURCE OPERATIONS</small><h1>来源与更新</h1><p>登记可追溯来源,为文档采集和知识审核建立基础。</p></div><button class="primary" @click="showSourceForm=!showSourceForm"><Plus :size="16"/> 新增来源</button></div><div v-if="showSourceForm" class="card form-card"><b>登记来源</b><form class="knowledge-form" @submit.prevent="saveSource"><label>来源名称<input v-model="sourceForm.name" required maxlength="250" placeholder="例如:那拉提景区官网"/></label><label>来源网址(可选)<input v-model="sourceForm.source_url" type="url" placeholder="https://..."/></label><label class="wide">备注<textarea v-model="sourceForm.notes" rows="3" placeholder="记录来源用途、可靠性判断或待核验事项"/></label><div class="wide actions"><button type="button" class="secondary" @click="showSourceForm=false">取消</button><button class="primary">保存来源</button></div></form></div><p v-if="notice" class="success">{{notice}}</p><p v-if="error" class="error">{{error}}</p><div class="stats"><div class="card stat"><small>来源总数</small><strong>{{sources.length}}</strong><span>来自当前数据库</span></div><div class="card stat"><small>关联文档</small><strong>—</strong><span>文档工作台待接入</span></div><div class="card stat"><small>自动更新</small><strong class="pending">未启用</strong><span>当前不会自动抓取网页</span></div></div><div class="card table-card"><div class="table-head"><div><b>已登记来源</b><p>来源不是审核通过的证明,关键事实仍需交叉核验。</p></div><label class="search"><Search :size="15"/><input v-model="sourceSearch" placeholder="搜索名称或备注" @keyup.enter="refreshSources"/></label></div><div class="scroll"><table><thead><tr><th>名称</th><th>网址</th><th>备注</th><th>登记时间</th></tr></thead><tbody><tr v-for="source in sources" :key="source.id"><td><b>{{source.name}}</b></td><td><a v-if="source.source_url" :href="source.source_url" target="_blank" rel="noreferrer">打开来源</a><span v-else>未填写</span></td><td>{{source.notes||'—'}}</td><td>{{new Date(source.created_at).toLocaleDateString()}}</td></tr><tr v-if="!sources.length"><td colspan="4" class="empty">暂无来源,点击“新增来源”登记。</td></tr></tbody></table></div></div><div class="card roadmap"><b>来源处理边界</b><div><span>01 登记来源</span><ChevronRight/><span>02 人工核验</span><ChevronRight/><span>03 关联文档</span><ChevronRight/><span>04 后续检索</span></div></div></div>
|
||||
<div v-else><div class="heading"><div><small class="eyebrow"><Activity :size="14"/> EVALUATION WORKSPACE</small><h1>实验评估</h1><p>展示实验方案与指标规划;当前不显示虚构实验结果。</p></div><span class="pill">● 尚未开展实验</span></div><div class="notice wide"><b><ShieldCheck :size="16"/> 实验原则</b><p>所有指标应来自真实测试问题、运行日志和明确的人工评分规则。下方仅列评价维度,不包含实验数值。</p></div><div class="stats"><div class="card stat"><small>回答正确性</small><strong class="pending">待测试</strong><span>依据人工评分规范</span></div><div class="card stat"><small>来源可追溯性</small><strong class="pending">待测试</strong><span>核对来源是否支持答案</span></div><div class="card stat"><small>响应时间与成本</small><strong class="pending">待测试</strong><span>记录耗时与实际费用</span></div></div><div class="card roadmap"><b>计划中的对照方案</b><div class="methods"><p><b>A 基础大语言模型</b><span>不提供项目知识库检索结果</span></p><p><b>B 关键词检索 RAG</b><span>关键词检索后生成答案</span></p><p><b>C 混合检索 RAG</b><span>融合关键词与向量语义检索</span></p></div></div></div><footer>边疆红驿 · 那拉提文旅智能问答 <span>工程原型 v0.1.0 · 尚非完整系统</span></footer></section></main></div></template>
|
||||
|
||||
@@ -0,0 +1,2 @@
|
||||
<script setup lang="ts">import { RouterView } from 'vue-router';</script>
|
||||
<template><RouterView/></template>
|
||||
+21
-4
@@ -1,4 +1,21 @@
|
||||
export async function getHealth(){const r=await fetch('/api/health');if(!r.ok)throw new Error('health failed');return r.json()}
|
||||
export async function askQuestion(question:string){const r=await fetch('/api/chat',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({question})});if(!r.ok)throw new Error('chat failed');return r.json() as Promise<{answer:string;status:string;sources:{title:string;source_url?:string|null}[]}>}
|
||||
export async function getKnowledge(){const r=await fetch('/api/knowledge');if(!r.ok)throw new Error('knowledge failed');return r.json() as Promise<any[]>}
|
||||
export async function createKnowledge(payload:{title:string;category:string;content:string;source_url?:string}){const r=await fetch('/api/knowledge',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify(payload)});if(!r.ok)throw new Error('保存失败,请检查网址格式及后端状态');return r.json()}
|
||||
import { apiRequest } from './services/session'
|
||||
import type { VectorRetrievalResult } from './services/vectors'
|
||||
import type { ChunkKeywordResult, HybridResult } from './services/hybrid'
|
||||
|
||||
export const getHealth = () => apiRequest<{status:string}>('/api/health')
|
||||
export const askQuestion = (question:string) => apiRequest<{answer:string;status:string;sources:{title:string;source_url?:string|null}[]}>('/api/chat',{method:'POST',body:JSON.stringify({question})})
|
||||
export const getKnowledge = () => apiRequest<{id:number;title:string;category:string;content:string;source_url:string|null;verification_status:string;version:number}[]>('/api/knowledge')
|
||||
export const createKnowledge = (payload:{title:string;category:string;content:string;source_url?:string}) => apiRequest('/api/knowledge',{method:'POST',body:JSON.stringify(payload)},true)
|
||||
|
||||
export type VerificationStatus='pending'|'verified'|'rejected'
|
||||
export type SourceItem={id:number;version:number;name:string;source_url:string|null;notes:string;created_at:string}
|
||||
export function getSources(q=''){return apiRequest<{items:SourceItem[];total:number}>(`/api/sources?q=${encodeURIComponent(q)}`)}
|
||||
export function createSource(payload:{name:string;source_url?:string;notes:string}){return apiRequest<SourceItem>('/api/sources',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify(payload)},true)}
|
||||
export function reviewKnowledge(id:number,payload:{expected_version:number;verification_status:VerificationStatus;reviewer:string;verification_notes:string}){return apiRequest(`/api/knowledge/${id}/review`,{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify(payload)},true)}
|
||||
|
||||
export type RetrievalHit={rank:number;knowledge_id:number;knowledge_version:number;document_id:number|null;document_version:number|null;source_id:number|null;source_version:number|null;source_name:string|null;source_url:string;title:string;category:string;snippet:string;snippet_start:number;snippet_end:number;score:number;matched_terms:string[]}
|
||||
export type RetrievalResult={run_id:string;query:string;mode:'keyword';status:'ok'|'no_evidence';top_k:number;category:string|null;query_terms:string[];corpus_size:number;corpus_fingerprint:string;total_hits:number;results:RetrievalHit[];elapsed_ms:number;created_at:string;config:{tokenizer:string;scorer:string;k1:number;b:number;title_weight:number;stopwords_version:string}}
|
||||
export function retrieveKnowledge(payload:{query:string;mode?:'keyword';top_k?:number;category?:string}){return apiRequest<RetrievalResult>('/api/retrieval',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify(payload)})}
|
||||
export type RetrievalRunResult = RetrievalResult | VectorRetrievalResult | ChunkKeywordResult | HybridResult
|
||||
export function getRetrievalRun(runId:string){return apiRequest<RetrievalRunResult>(`/api/retrieval/runs/${encodeURIComponent(runId)}`)}
|
||||
export function getRetrievalRuns(page=1,pageSize=20){return apiRequest<{items:RetrievalRunResult[];total:number;page:number;page_size:number}>(`/api/retrieval/runs?page=${page}&page_size=${pageSize}`)}
|
||||
|
||||
@@ -0,0 +1,2 @@
|
||||
.search:focus-within{border-color:#8eaa91;box-shadow:0 0 0 3px #dcebdc}.knowledge-form input:focus,.knowledge-form select:focus,.knowledge-form textarea:focus{border-color:#7e9f83;box-shadow:0 0 0 3px #e4f0e4}.table-card tbody tr{transition:background .15s ease}.table-card tbody tr:hover{background:#fafbf7}.success,.error{animation:feedback-in .2s ease-out}@keyframes feedback-in{from{opacity:0;transform:translateY(-4px)}to{opacity:1;transform:translateY(0)}}
|
||||
.quick-actions{margin-top:18px}.quick-actions .label{margin:0 0 8px}.side-action{width:100%;display:flex;align-items:center;gap:8px;border:1px solid #dbe6da;border-radius:9px;padding:10px 12px;background:#f1f7ef;color:#355c48;font-size:11px;font-weight:600}.side-action:hover{background:#e2efdf}.side-hint{margin:0 10px;color:#8a9086;font-size:10px;line-height:1.7}.heading>.primary{display:none}
|
||||
@@ -1,4 +1,6 @@
|
||||
import { createApp } from 'vue'
|
||||
import './style.css'
|
||||
import App from './App.vue'
|
||||
createApp(App).mount('#app')
|
||||
import './enhancements.css'
|
||||
import Root from './Root.vue'
|
||||
import router from './router'
|
||||
createApp(Root).use(router).mount('#app')
|
||||
|
||||
@@ -0,0 +1,16 @@
|
||||
import { createRouter, createWebHistory } from 'vue-router'
|
||||
import App from './App.vue'
|
||||
import { getCurrentUser } from './services/auth'
|
||||
import type { UserRole } from './types/auth'
|
||||
|
||||
let role:UserRole|undefined
|
||||
const router=createRouter({history:createWebHistory(),routes:[
|
||||
{path:'/',component:App},
|
||||
{path:'/login',component:()=>import('./views/LoginView.vue'),meta:{public:true}},
|
||||
{path:'/register',component:()=>import('./views/RegisterView.vue'),meta:{public:true}},
|
||||
{path:'/profile',component:()=>import('./views/ProfileView.vue')},
|
||||
{path:'/settings',component:()=>import('./views/AccountSettingsView.vue')},
|
||||
{path:'/admin',component:()=>import('./views/AdminView.vue'),meta:{role:'admin'}},
|
||||
]})
|
||||
router.beforeEach(async to=>{if(to.meta.public)return true;try{role=(await getCurrentUser()).role}catch{return {path:'/login',query:{redirect:to.fullPath}}}if(to.meta.role&&role!==to.meta.role)return {path:'/profile',query:{forbidden:'1'}};return true})
|
||||
export default router
|
||||
@@ -0,0 +1,20 @@
|
||||
import type { ChangePasswordPayload, CurrentUser, LoginPayload, RegisterPayload, SessionUser, UpdateProfilePayload } from '../types/auth'
|
||||
import { apiRequest, ApiError, clearSessionToken } from './session'
|
||||
|
||||
export { ApiError as AuthApiUnavailableError }
|
||||
export const register = (payload: RegisterPayload) => apiRequest<CurrentUser>('/api/auth/register', { method: 'POST', body: JSON.stringify(payload) })
|
||||
export const login = async (payload: LoginPayload) => {
|
||||
clearSessionToken()
|
||||
return apiRequest<SessionUser>('/api/auth/login', { method: 'POST', body: JSON.stringify(payload) })
|
||||
}
|
||||
export const getCurrentUser = () => apiRequest<SessionUser>('/api/auth/me')
|
||||
export const updateMyProfile = (payload: UpdateProfilePayload) => apiRequest<CurrentUser>('/api/users/me', { method: 'PATCH', body: JSON.stringify(payload) }, true)
|
||||
export const logout = async () => {
|
||||
await apiRequest<void>('/api/auth/logout', { method: 'POST' }, true)
|
||||
clearSessionToken()
|
||||
}
|
||||
export const changePassword = async (payload: ChangePasswordPayload) => {
|
||||
await apiRequest<void>('/api/auth/change-password', { method: 'POST', body: JSON.stringify(payload) }, true)
|
||||
clearSessionToken()
|
||||
}
|
||||
export const getUsers = (limit = 100, offset = 0) => apiRequest<CurrentUser[]>(`/api/admin/users?limit=${limit}&offset=${offset}`)
|
||||
@@ -0,0 +1,19 @@
|
||||
import { apiRequest } from './session'
|
||||
import type { RetrievalHit, RetrievalResult } from '../api'
|
||||
import type { IndexVersion } from './indexes'
|
||||
import type { EmbeddingSpace } from './vectors'
|
||||
|
||||
type ChunkHit = RetrievalHit & { chunk_id: string; chunk_ordinal: number }
|
||||
type ChunkConfig = { ranking_unit: 'chunk'; lexical: RetrievalResult['config']; chunking: IndexVersion['config']; tie_break: 'knowledge_id,chunk_ordinal,chunk_id' }
|
||||
type ChunkSnapshot = { run_id: string; query: string; status: 'ok' | 'no_evidence'; top_k: number; category: string | null; query_terms: string[]; text_index_version_id: string; text_generation: number; catalog_revision: number; corpus_fingerprint: string; corpus_size: number; total_hits: number; elapsed_ms: number; created_at: string }
|
||||
export type ChunkKeywordResult = ChunkSnapshot & { mode: 'keyword_chunks'; results: ChunkHit[]; config: ChunkConfig }
|
||||
export type RankedCandidate = { chunk_id: string; rank: number; score: number }
|
||||
export type HybridResult = ChunkSnapshot & {
|
||||
mode: 'hybrid'; vector_version_id: string; generation: number; lexical_total_hits: number; vector_total_hits: number;
|
||||
lexical_candidates: RankedCandidate[]; vector_candidates: RankedCandidate[];
|
||||
results: (ChunkHit & { lexical_rank: number | null; lexical_score: number | null; vector_rank: number | null; vector_score: number | null; lexical_contribution: number; vector_contribution: number })[];
|
||||
config: ChunkConfig & { scorer: 'rrf-v1'; rrf_k: 60; weights: [1, 1]; candidate_k: number; vector_scorer: 'pgvector-cosine-exact-v1'; space: EmbeddingSpace }
|
||||
}
|
||||
type Request = { query: string; top_k?: number; category?: string }
|
||||
export const retrieveKeywordChunks = (payload: Request & { mode?: 'keyword_chunks' }) => apiRequest<ChunkKeywordResult>('/api/retrieval/chunks', { method: 'POST', body: JSON.stringify(payload) })
|
||||
export const retrieveHybrid = (payload: Request & { mode?: 'hybrid'; candidate_k?: number }) => apiRequest<HybridResult>('/api/retrieval/hybrid', { method: 'POST', body: JSON.stringify(payload) })
|
||||
@@ -0,0 +1,16 @@
|
||||
import { apiRequest } from './session'
|
||||
import type { RetrievalHit } from '../api'
|
||||
|
||||
type Page<T> = { items: T[]; total: number; page: number; page_size: number }
|
||||
export type IndexConfig = { splitter: 'unicode-window-v1'; chunk_size: number; overlap: number }
|
||||
export type IndexVersion = { id: string; status: 'ready' | 'published' | 'superseded'; config: IndexConfig; corpus_fingerprint: string; catalog_revision: number; item_count: number; chunk_count: number; created_by: number; created_at: string; published_at: string | null }
|
||||
export type ActiveIndex = { generation: number; catalog_revision: number; is_current: boolean; active_version: IndexVersion | null }
|
||||
export type Chunk = { id: string; index_version_id: string; knowledge_id: number; ordinal: number; start_offset: number; end_offset: number; text: string; text_sha256: string; provenance: Pick<RetrievalHit, 'knowledge_id' | 'knowledge_version' | 'document_id' | 'document_version' | 'source_id' | 'source_version' | 'source_name' | 'source_url' | 'title' | 'category'> }
|
||||
export type IndexPublication = { generation: number; previous_version_id: string | null; version_id: string; actor_id: number; published_at: string }
|
||||
export const buildIndex = (config: Partial<Pick<IndexConfig, 'chunk_size' | 'overlap'>> = {}) => apiRequest<IndexVersion>('/api/indexes', { method: 'POST', body: JSON.stringify(config) }, true)
|
||||
export const getActiveIndex = () => apiRequest<ActiveIndex>('/api/indexes/active')
|
||||
export const getIndexVersions = (page = 1, pageSize = 20) => apiRequest<Page<IndexVersion>>(`/api/indexes?page=${page}&page_size=${pageSize}`)
|
||||
export const getIndexVersion = (id: string) => apiRequest<IndexVersion>(`/api/indexes/${encodeURIComponent(id)}`)
|
||||
export const publishIndex = (id: string, expectedGeneration: number) => apiRequest<ActiveIndex>(`/api/indexes/${encodeURIComponent(id)}/publish`, { method: 'POST', body: JSON.stringify({ expected_generation: expectedGeneration }) }, true)
|
||||
export const getChunks = (id: string, page = 1, pageSize = 20, knowledgeId?: number) => apiRequest<Page<Chunk>>(`/api/indexes/${encodeURIComponent(id)}/chunks?page=${page}&page_size=${pageSize}${knowledgeId ? `&knowledge_id=${knowledgeId}` : ''}`)
|
||||
export const getIndexPublications = (page = 1, pageSize = 20) => apiRequest<Page<IndexPublication>>(`/api/indexes/publications?page=${page}&page_size=${pageSize}`)
|
||||
@@ -0,0 +1,28 @@
|
||||
let csrfToken: string | null = null
|
||||
|
||||
export class ApiError extends Error {
|
||||
constructor(public readonly status: number, message: string) { super(message) }
|
||||
}
|
||||
|
||||
export function clearSessionToken() { csrfToken = null }
|
||||
|
||||
export async function apiRequest<T>(path: string, init: RequestInit = {}, protect = false): Promise<T> {
|
||||
if (protect && !csrfToken) await apiRequest('/api/auth/me')
|
||||
const headers = new Headers(init.headers)
|
||||
headers.set('Accept', 'application/json')
|
||||
if (init.body) headers.set('Content-Type', 'application/json')
|
||||
if (protect && csrfToken) headers.set('X-CSRF-Token', csrfToken)
|
||||
const response = await fetch(path, { ...init, credentials: 'include', headers })
|
||||
if (!response.ok) {
|
||||
if (response.status === 401) clearSessionToken()
|
||||
const body: unknown = await response.json().catch(() => null)
|
||||
const detail = body && typeof body === 'object' && 'detail' in body ? body.detail : null
|
||||
throw new ApiError(response.status, typeof detail === 'string' ? detail : `请求失败(${response.status}),请检查登录状态或填写内容。`)
|
||||
}
|
||||
if (response.status === 204) return undefined as T
|
||||
const data = await response.json()
|
||||
if (path === '/api/auth/login' || path === '/api/auth/me') {
|
||||
csrfToken = typeof data.csrf_token === 'string' ? data.csrf_token : null
|
||||
}
|
||||
return data as T
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
import { apiRequest } from './session'
|
||||
import type { RetrievalHit } from '../api'
|
||||
import type { IndexPublication } from './indexes'
|
||||
|
||||
type Page<T> = { items: T[]; total: number; page: number; page_size: number }
|
||||
export type EmbeddingSpec = { model: string; revision: string; dimension: number; token_limit: number; query_instruction: string; implementation: string; checkpoint_format: string }
|
||||
export type EmbeddingSpace = { id: string; dimension: number; spec: EmbeddingSpec }
|
||||
export type VectorVersion = { id: string; text_index_version_id: string; space_id: string; dimension: number; chunk_count: number; status: 'ready' | 'published' | 'superseded'; created_by: number; created_at: string; published_at: string | null }
|
||||
export type ActiveVectors = { generation: number; text_generation: number; catalog_revision: number; is_current: boolean; active_version: VectorVersion | null }
|
||||
export type VectorRetrievalResult = { run_id: string; query: string; mode: 'vector'; status: 'ok' | 'no_evidence'; top_k: number; category: string | null; vector_version_id: string; text_index_version_id: string; generation: number; text_generation: number; corpus_fingerprint: string; corpus_size: number; total_hits: number; results: (RetrievalHit & { chunk_id: string; chunk_ordinal: number })[]; elapsed_ms: number; created_at: string; config: { scorer: 'pgvector-cosine-exact-v1'; ranking_unit: 'chunk'; space: EmbeddingSpace } }
|
||||
export const buildVectorIndex = (textIndexVersionId: string) => apiRequest<VectorVersion>('/api/vector-indexes', { method: 'POST', body: JSON.stringify({ text_index_version_id: textIndexVersionId }) }, true)
|
||||
export const getVectorVersions = (page = 1, pageSize = 20) => apiRequest<Page<VectorVersion>>(`/api/vector-indexes?page=${page}&page_size=${pageSize}`)
|
||||
export const getVectorVersion = (id: string) => apiRequest<VectorVersion>(`/api/vector-indexes/${encodeURIComponent(id)}`)
|
||||
export const getActiveVectors = () => apiRequest<ActiveVectors>('/api/vector-indexes/active')
|
||||
export const getEmbeddingSpaces = (page = 1, pageSize = 20) => apiRequest<Page<EmbeddingSpace>>(`/api/vector-indexes/spaces?page=${page}&page_size=${pageSize}`)
|
||||
export const publishVectorIndex = (id: string, expectedGeneration: number) => apiRequest<ActiveVectors>(`/api/vector-indexes/${encodeURIComponent(id)}/publish`, { method: 'POST', body: JSON.stringify({ expected_generation: expectedGeneration }) }, true)
|
||||
export const getVectorPublications = (page = 1, pageSize = 20) => apiRequest<Page<IndexPublication>>(`/api/vector-indexes/publications?page=${page}&page_size=${pageSize}`)
|
||||
export const retrieveVectors = (payload: { query: string; mode?: 'vector'; top_k?: number; category?: string }) => apiRequest<VectorRetrievalResult>('/api/retrieval/vector', { method: 'POST', body: JSON.stringify(payload) })
|
||||
@@ -0,0 +1,14 @@
|
||||
export type UserRole = 'user' | 'admin'
|
||||
export type CurrentUser = {
|
||||
id: number
|
||||
username: string
|
||||
nickname: string | null
|
||||
email: string | null
|
||||
role: UserRole
|
||||
created_at: string
|
||||
}
|
||||
export type SessionUser = CurrentUser & { csrf_token: string }
|
||||
export type RegisterPayload = { username: string; password: string; email?: string; nickname?: string }
|
||||
export type LoginPayload = { username: string; password: string }
|
||||
export type UpdateProfilePayload = { nickname?: string | null; email?: string | null }
|
||||
export type ChangePasswordPayload = { current_password: string; new_password: string }
|
||||
@@ -0,0 +1,96 @@
|
||||
<script setup lang="ts">
|
||||
import { onMounted, ref } from 'vue'
|
||||
import { useRouter } from 'vue-router'
|
||||
import { changePassword, getCurrentUser, updateMyProfile } from '../services/auth'
|
||||
import type { SessionUser } from '../types/auth'
|
||||
|
||||
const router = useRouter()
|
||||
const user = ref<SessionUser>()
|
||||
const profile = ref({ nickname: '', email: '' })
|
||||
const password = ref({ current_password: '', new_password: '', confirm_password: '' })
|
||||
const loading = ref(true)
|
||||
const savingProfile = ref(false)
|
||||
const savingPassword = ref(false)
|
||||
const error = ref('')
|
||||
const success = ref('')
|
||||
|
||||
onMounted(async () => {
|
||||
try {
|
||||
user.value = await getCurrentUser()
|
||||
profile.value = { nickname: user.value.nickname ?? '', email: user.value.email ?? '' }
|
||||
} catch (cause) {
|
||||
error.value = cause instanceof Error ? cause.message : '无法读取账户资料'
|
||||
} finally {
|
||||
loading.value = false
|
||||
}
|
||||
})
|
||||
|
||||
async function saveProfile() {
|
||||
error.value = ''
|
||||
success.value = ''
|
||||
if (profile.value.email && !/^\S+@\S+\.\S+$/.test(profile.value.email)) {
|
||||
error.value = '请输入有效邮箱地址。'
|
||||
return
|
||||
}
|
||||
savingProfile.value = true
|
||||
try {
|
||||
const updated = await updateMyProfile({ nickname: profile.value.nickname || null, email: profile.value.email || null })
|
||||
if (user.value) user.value = { ...user.value, ...updated }
|
||||
success.value = '基本资料已保存。'
|
||||
} catch (cause) {
|
||||
error.value = cause instanceof Error ? cause.message : '保存资料失败'
|
||||
} finally {
|
||||
savingProfile.value = false
|
||||
}
|
||||
}
|
||||
|
||||
async function savePassword() {
|
||||
error.value = ''
|
||||
success.value = ''
|
||||
if (password.value.new_password.length < 15) {
|
||||
error.value = '新密码至少需要 15 个字符。'
|
||||
return
|
||||
}
|
||||
if (password.value.new_password !== password.value.confirm_password) {
|
||||
error.value = '两次输入的新密码不一致。'
|
||||
return
|
||||
}
|
||||
savingPassword.value = true
|
||||
try {
|
||||
await changePassword({ current_password: password.value.current_password, new_password: password.value.new_password })
|
||||
password.value = { current_password: '', new_password: '', confirm_password: '' }
|
||||
router.replace('/login?changed=1')
|
||||
} catch (cause) {
|
||||
error.value = cause instanceof Error ? cause.message : '修改密码失败'
|
||||
} finally {
|
||||
savingPassword.value = false
|
||||
}
|
||||
}
|
||||
</script>
|
||||
|
||||
<template>
|
||||
<main>
|
||||
<RouterLink to="/profile">返回个人中心</RouterLink>
|
||||
<h1>账户设置</h1>
|
||||
<p v-if="loading">正在读取账户资料…</p>
|
||||
<template v-else>
|
||||
<p v-if="error" role="alert">{{ error }}</p>
|
||||
<p v-if="success" role="status">{{ success }}</p>
|
||||
<form @submit.prevent="saveProfile">
|
||||
<h2>基本资料</h2>
|
||||
<label>用户名<input :value="user?.username" disabled /></label>
|
||||
<label>昵称<input v-model.trim="profile.nickname" maxlength="80" autocomplete="nickname" /></label>
|
||||
<label>邮箱<input v-model.trim="profile.email" type="email" maxlength="320" autocomplete="email" /></label>
|
||||
<button :disabled="savingProfile">{{ savingProfile ? '保存中…' : '保存资料' }}</button>
|
||||
</form>
|
||||
<form @submit.prevent="savePassword">
|
||||
<h2>修改密码</h2>
|
||||
<label>当前密码<input v-model="password.current_password" type="password" required autocomplete="current-password" /></label>
|
||||
<label>新密码<input v-model="password.new_password" type="password" required minlength="15" autocomplete="new-password" /></label>
|
||||
<label>确认新密码<input v-model="password.confirm_password" type="password" required minlength="15" autocomplete="new-password" /></label>
|
||||
<p>修改密码后会退出当前会话,请使用新密码重新登录。</p>
|
||||
<button :disabled="savingPassword">{{ savingPassword ? '提交中…' : '修改密码' }}</button>
|
||||
</form>
|
||||
</template>
|
||||
</main>
|
||||
</template>
|
||||
@@ -0,0 +1 @@
|
||||
<script setup lang="ts">import { onMounted,ref } from 'vue';import { getUsers } from '../services/auth';const users=ref<any[]>([]),error=ref('');onMounted(async()=>{try{users.value=await getUsers()}catch(e){error.value=e instanceof Error?e.message:'无管理员权限'}});</script><template><main><h1>管理员用户管理</h1><p v-if="error">{{error}}</p><table v-else><tr><th>用户名</th><th>邮箱</th><th>角色</th></tr><tr v-for="u in users" :key="u.id"><td>{{u.username}}</td><td>{{u.email}}</td><td>{{u.role}}</td></tr></table></main></template>
|
||||
@@ -0,0 +1 @@
|
||||
<script setup lang="ts">import { ref } from 'vue';import { useRoute,useRouter } from 'vue-router';import { login } from '../services/auth';const username=ref(''),password=ref(''),error=ref(''),router=useRouter(),route=useRoute();async function submit(){try{await login({username:username.value,password:password.value});await router.replace(typeof route.query.redirect==='string'?route.query.redirect:'/')}catch(e){error.value=e instanceof Error?e.message:'登录失败'}}</script><template><main><h1>登录</h1><form @submit.prevent="submit"><input v-model="username" autocomplete="username" required placeholder="用户名"><input v-model="password" type="password" autocomplete="current-password" required placeholder="密码"><p v-if="error">{{error}}</p><button>登录</button></form><RouterLink to="/register">注册账户</RouterLink></main></template>
|
||||
@@ -0,0 +1 @@
|
||||
<script setup lang="ts">import { onMounted,ref } from 'vue';import { getCurrentUser,logout } from '../services/auth';import { useRouter } from 'vue-router';import type { SessionUser } from '../types/auth';const user=ref<SessionUser>(),error=ref(''),router=useRouter();onMounted(async()=>{try{user.value=await getCurrentUser()}catch(e){error.value=e instanceof Error?e.message:'无法读取资料'}});async function signOut(){try{await logout()}finally{router.replace('/login')}}</script><template><main><h1>个人中心</h1><p v-if="error">{{error}}</p><dl v-else-if="user"><dt>用户名</dt><dd>{{user.username}}</dd><dt>昵称</dt><dd>{{user.nickname||'未设置'}}</dd><dt>邮箱</dt><dd>{{user.email||'未设置'}}</dd><dt>角色</dt><dd>{{user.role}}</dd><dt>注册时间</dt><dd>{{new Date(user.created_at).toLocaleString()}}</dd></dl><div><RouterLink to="/settings">账户设置</RouterLink><button @click="signOut">退出登录</button></div></main></template>
|
||||
@@ -0,0 +1 @@
|
||||
<script setup lang="ts">import { ref } from 'vue';import { useRouter } from 'vue-router';import { register } from '../services/auth';const username=ref(''),password=ref(''),error=ref(''),router=useRouter();async function submit(){try{await register({username:username.value,password:password.value});await router.push('/login')}catch(e){error.value=e instanceof Error?e.message:'注册失败'}}</script><template><main><h1>注册</h1><form @submit.prevent="submit"><input v-model="username" required minlength="3" placeholder="用户名"><input v-model="password" required minlength="15" type="password" placeholder="至少 15 位密码"><p v-if="error">{{error}}</p><button>注册</button></form></main></template>
|
||||
Reference in New Issue
Block a user