Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
05a129bf51 | ||
|
|
b28b3a18d6 | ||
|
|
783c5d7cf9 |
@@ -8,6 +8,3 @@ dist/
|
||||
.vscode/
|
||||
.idea/
|
||||
*.log
|
||||
.ruff_cache/
|
||||
.coverage
|
||||
htmlcov/
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
# 那拉提文旅智能问答
|
||||
# 那拉提文旅智能问答:前后端基础壳子
|
||||
|
||||
Vue 3 + TypeScript + Vite 前端;FastAPI + Pydantic + SQLAlchemy 后端;PostgreSQL + pgvector。当前后端提供知识管理、来源、文档、人工审核、关键词检索、历史记录及认证/角色权限,问答仍为占位接口。
|
||||
基础工程骨架:Vue 3 + TypeScript + Vite 前端;FastAPI + Pydantic + SQLAlchemy 后端;PostgreSQL + pgvector 数据库配置。
|
||||
|
||||
## 启动数据库
|
||||
需要可用的 Docker Engine(Windows 可使用 Docker Desktop),在项目根目录运行:
|
||||
需要 Docker Desktop,在项目根目录运行:
|
||||
```bash
|
||||
docker compose up -d db
|
||||
```
|
||||
@@ -15,27 +15,11 @@ cd backend
|
||||
py -3.12 -m venv .venv
|
||||
.\.venv\Scripts\Activate.ps1
|
||||
pip install -r requirements.txt
|
||||
if (!(Test-Path .env)) { Copy-Item .env.example .env }
|
||||
alembic upgrade head
|
||||
Copy-Item .env.example .env
|
||||
uvicorn app.main:app --reload
|
||||
```
|
||||
API 文档:http://127.0.0.1:8000/docs
|
||||
|
||||
Linux / macOS 可使用以下等价命令(需要 Python 3.11+;`.env` 已存在时不要覆盖):
|
||||
```bash
|
||||
cd backend
|
||||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
pip install -r requirements.txt
|
||||
test -f .env || cp .env.example .env
|
||||
alembic upgrade head
|
||||
uvicorn app.main:app --reload --host 127.0.0.1
|
||||
```
|
||||
|
||||
配置始终从 `backend/.env` 读取,环境变量优先。必须先迁移再启动;应用启动只检查数据库版本,不再执行 `create_all`。旧版 `knowledge_items` 会被非破坏性接管;首次迁移前仍须备份。迁移会保留原条目及导入快照,不推测缺失的采集时间。旧版 `verified` 条目因缺少完整审核记录,会转为 `pending`,原状态保存在导入日志中。
|
||||
|
||||
认证迁移为 `0004_auth_sessions`。知识管理、来源、文档及检索历史现要求管理员登录;迁移不创建默认账号。运行 `python -m app.manage create-admin --username local_admin`,通过隐藏输入设置独立密码。默认 Cookie 要求 HTTPS;本机 HTTP 联调须在未入库的 `.env` 显式设置 `AUTH_COOKIE_SECURE=false`(仅 development/test 允许)。完整来源、CSRF、字段和权限说明见 [认证接口契约](docs/AUTH_API.md)。当前数据库最新迁移为 `0005_chunk_indexes`,索引构建/发布说明见 [切分与索引版本接口](docs/INDEX_API.md)。
|
||||
|
||||
## 启动前端(另开终端)
|
||||
```powershell
|
||||
cd frontend
|
||||
@@ -45,87 +29,19 @@ npm run dev
|
||||
Vite 通常使用 http://localhost:5173。
|
||||
|
||||
## 已有接口
|
||||
- 注册/登录/退出、当前会话、资料修改、改密与管理员用户列表,详见 [认证接口契约](docs/AUTH_API.md)
|
||||
- GET `/api/health`
|
||||
- GET `/api/ready`:检查数据库连接
|
||||
- POST `/api/chat`:占位接口,尚未接入检索和 LLM
|
||||
- GET `/api/knowledge`
|
||||
- POST `/api/knowledge`
|
||||
- GET `/api/knowledge/page`、GET/PATCH/DELETE `/api/knowledge/{id}`
|
||||
- POST `/api/knowledge/{id}/review`、GET `/api/knowledge/{id}/history`
|
||||
- GET/POST `/api/sources`、GET/PATCH/DELETE `/api/sources/{id}`、GET `/api/sources/{id}/history`
|
||||
- GET/POST `/api/documents`、GET/PATCH/DELETE `/api/documents/{id}`、GET `/api/documents/{id}/history`
|
||||
- POST `/api/retrieval`:已核验知识的关键词检索
|
||||
- GET `/api/retrieval/runs`、GET `/api/retrieval/runs/{run_id}`:检索历史和原始结果
|
||||
- `/api/indexes`:管理员构建/查询切分快照、发布/查询当前版本、查询 chunks 和发布历史,详见 [接口契约](docs/INDEX_API.md)
|
||||
|
||||
### 阶段 2 接口约定
|
||||
|
||||
- 所有日期均为带时区的 ISO 8601。输入拒绝未知字段,标题/正文不能全为空白,网址只接受 HTTP(S);可选网址空字符串按未填写处理。
|
||||
- `GET /api/knowledge` 保留数组响应,支持 `q`、`category`、`verification_status`、`limit`(默认 100,最大 200)和 `offset`。新页面应使用 `/api/knowledge/page`,避免把首批条目数量当作总数。
|
||||
- 分页响应为 `{items, total, page, page_size}`;`page` 从 1 开始,`page_size` 默认 20、最大 100。来源、文档、知识分页和各类历史接口均支持分页。
|
||||
- 来源分页支持 `q`;文档分页支持 `q`、`source_id`、`category`;知识分页支持 `q`、`category`、`verification_status`、`source_id`、`document_id`。`q` 为字面文本子串搜索,尚非论文中的关键词检索基线。
|
||||
- 创建知识沿用 `title/category/content/source_url`,新增可选 `source_id/document_id/collected_at`;状态固定为 `pending`。关联文档时自动解析其来源,填写的来源或网址若冲突会报错。直接来源链接的旧调用仍受支持。
|
||||
- 来源创建字段:`name`、可选 `source_url`、`notes`。文档创建字段:`source_id/title/content/collected_at`,可选 `category/notes`。文档来源归属固定,换来源需新建文档。
|
||||
- PATCH 请求必须携带读取时获得的 `expected_version`,并至少提交一个修改字段;DELETE 使用查询参数 `expected_version`。版本过期返回 409,防止覆盖他人修改。
|
||||
- 审核请求字段见下方示例。`verification_status` 只允许 `pending/verified/rejected`;核验人和说明必填。通过审核前必须有来源网址及真实采集时间。这里只记录人工结论,不会自动核实网页。
|
||||
- 修改条目、关联来源或关联文档都会将相关知识转为 `pending`,清除当前审核字段并记录失效原因。历史快照保留当时的条目、来源和文档。
|
||||
- 删除采用软删除;被有效文档或知识引用的来源、被有效知识引用的文档不能删除(409)。已删除记录不出现在列表,但历史仍可查看。
|
||||
- 错误响应使用 `detail`:参数错误 422、记录不存在 404、版本/引用冲突 409、数据库暂不可用 503。
|
||||
|
||||
审核请求示例(仅为占位测试数据):
|
||||
```json
|
||||
{
|
||||
"expected_version": 1,
|
||||
"verification_status": "verified",
|
||||
"reviewer": "test-reviewer",
|
||||
"verification_notes": "示例审核说明,实际使用须填写真实核验依据"
|
||||
}
|
||||
```
|
||||
|
||||
响应新增 `version/updated_at/is_deleted` 及关联、采集、审核字段;完整类型以 `/openapi.json` 和 `/docs` 为准。前端已接入知识录入和来源登记/查询,文档、审核及完整分页页面仍需按接口联调。
|
||||
|
||||
### 关键词检索基线
|
||||
|
||||
先安装更新后的依赖并执行 `alembic upgrade head`,再启动后端。检索请求示例:
|
||||
```json
|
||||
{"query": "那拉提旅游资源", "mode": "keyword", "top_k": 5, "category": "旅游资源"}
|
||||
```
|
||||
|
||||
`query` 为 1–500 字符;`top_k` 默认 5、最大 20;`category` 可省略。当前只接受 `keyword` 模式。纯标点、空白或仅含停用词的查询返回 422。
|
||||
|
||||
- 仅使用 `verified` 且未删除、采集/审核字段齐全的知识。关联来源/文档须仍有效,来源网址须与登记来源一致;`pending/rejected` 不参与分词、IDF 统计或结果排名。
|
||||
- 使用固定 `jieba==0.42.1` 搜索分词,关闭 HMM,加入项目词“那拉提”,采用字符级 NFKC 和大小写折叠、小型固定停用词表。标题词频权重 2,正文权重 1;BM25 参数 `k1=1.2/b=0.75`,IDF 为 `ln(1+(N-df+0.5)/(df+0.5))`。重复查询词计一次,同分按知识 ID 升序。
|
||||
- 返回 `run_id/status/query_terms/corpus_size/corpus_fingerprint/total_hits/results/elapsed_ms/config` 等字段。命中包含知识及关联文档/来源的 ID 与版本、标题、来源网址、分数、命中词和最长 240 字符的原文片段;`snippet_start/end` 是原文字符偏移,结束位置不包含在片段内。分数仅用于排序,不是事实正确率或置信度。独立网址的旧条目可没有文档 ID 或来源名称。
|
||||
- 关键词基线的检索单元仍是已审核知识条目;该接口的显示片段独立于新增的持久化 chunks,尚无向量索引。没有匹配证据时返回 `status=no_evidence` 和空结果,当前不会调用模型生成答案。
|
||||
- 每次成功检索(含零命中)保存不可变结果快照、实际检索耗时、语料指纹和算法参数。`elapsed_ms` 包含读取、分词、排序和片段生成,不包含记录写入和网络时间。历史记录用于复核,不能当作仍然有效的当前知识;依据变更后的新检索会重新应用审核条件。
|
||||
- `GET /api/retrieval/runs` 使用标准分页参数;`GET /api/retrieval/runs/{run_id}` 读取当时结果。当前知识修改、失效或删除不会改写历史结果。检索记录保存在数据库,运行日志和实际查询不要提交到 Git。
|
||||
- 本基线按请求计算语料词频,面向小规模资料集:筛选后的语料最多 5000 条、标题与正文合计最多 200 万字符。超过容量明确返回 503,不截断语料或伪造空结果;可缩小类别范围,大规模索引留到后续实现。
|
||||
|
||||
前端 `src/api.ts` 已提供带类型的检索及历史调用;检索页面交互仍由前端阶段接入。本阶段没有接入 Embedding、向量/混合检索或 LLM,不产生论文实验分数。
|
||||
|
||||
前端包括问答、知识库管理、来源与更新、实验评估四个页面。实验评估页面没有虚构数据;所有页面明确标记当前功能状态。
|
||||
|
||||
注意:当前没有实现 Embedding、RAG、网络搜索或后台更新任务。管理接口已有认证与角色校验;审核表单的核验人仍是人工输入字段。服务保持仅本机访问,完整生产部署及前端登录联调尚未验收。`.env` 不要提交到 Git。
|
||||
|
||||
### 后端测试和迁移回滚
|
||||
|
||||
在 `backend` 目录安装 `requirements-dev.txt` 后执行:
|
||||
```bash
|
||||
ruff check .
|
||||
ruff format --check .
|
||||
pytest -q
|
||||
```
|
||||
|
||||
测试需要真实 PostgreSQL + pgvector:设置 `TEST_DATABASE_URL`(`postgresql+psycopg://...`),或另行 `pip install pgserver` 使用可选的本机临时实例。`pgserver` 仅用于测试,不是应用运行依赖;外部测试数据库账号需有建 Schema 权限且已启用 vector 扩展。每个测试使用独立 Schema,不会清空业务表。测试使用的占位正文和审核人不代表真实资料。
|
||||
|
||||
迁移命令为 `alembic upgrade head`,可重复执行。仅撤回切分/索引版本时,备份后执行 `alembic downgrade 0004_auth_sessions`,保留账号、会话、知识和检索记录。继续撤回认证时,备份后执行 `alembic downgrade 0003_retrieval_runs`,删除用户/会话/预算表,保留知识和检索历史。继续执行 `alembic downgrade 0002_catalog` 还会删除检索历史。需要撤回阶段 2 时,先备份,再执行 `alembic downgrade 0001_baseline`;它保留基础知识字段,但会删除新增来源、文档、审核字段与所有历史记录,软删除的基础条目也会重新可见。降级须配合对应旧版后端,当前应用会拒绝旧 Schema。恢复完整数据应使用备份。不要在有业务数据的数据库执行 `downgrade base`。
|
||||
注意:当前没有实现 Embedding、RAG、网络搜索、后台更新任务、用户认证或生产级数据库迁移。下一步建议先验证前后端联通,再按模块实现。`.env` 不要提交到 Git。
|
||||
|
||||
## 团队协作文档
|
||||
|
||||
- [项目章程](docs/PROJECT_CHARTER.md)
|
||||
- [开发计划](docs/DEVELOPMENT_PLAN.md)
|
||||
- [阶段 1 详细开发计划](docs/PHASE_1_DEVELOPMENT_PLAN.md)
|
||||
- [共同开发规范](docs/CONTRIBUTING.md)
|
||||
- [系统架构](docs/ARCHITECTURE.md)
|
||||
- [任务分工](docs/TASKS.md)
|
||||
|
||||
@@ -3,7 +3,3 @@ APP_ENV=development
|
||||
API_PREFIX=/api
|
||||
DATABASE_URL=postgresql+psycopg://nalati:change-this-password@127.0.0.1:5432/nalati_rag
|
||||
CORS_ORIGINS=http://localhost:5173,http://127.0.0.1:5173
|
||||
AUTH_ALLOWED_ORIGINS=http://localhost:5173,http://127.0.0.1:5173,http://localhost:8000,http://127.0.0.1:8000
|
||||
# 默认要求 HTTPS;仅本机 HTTP 联调可显式改为 false。
|
||||
AUTH_COOKIE_SECURE=true
|
||||
AUTH_SESSION_SECONDS=28800
|
||||
|
||||
@@ -1,4 +0,0 @@
|
||||
[alembic]
|
||||
script_location = %(here)s/migrations
|
||||
prepend_sys_path = %(here)s
|
||||
path_separator = os
|
||||
@@ -1,122 +0,0 @@
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Request, Response
|
||||
from sqlalchemy import delete, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.db.session import get_db
|
||||
from app.models.auth import AuthSession, User
|
||||
from app.schemas.auth import (
|
||||
LoginInput,
|
||||
PasswordChangeInput,
|
||||
ProfileInput,
|
||||
RegisterInput,
|
||||
SessionRead,
|
||||
UserRead,
|
||||
)
|
||||
from app.services.auth import (
|
||||
PASSWORD_HASHER,
|
||||
CurrentSession,
|
||||
check_password,
|
||||
clear_cookie,
|
||||
dummy_hash,
|
||||
issue_session,
|
||||
limit_auth_requests,
|
||||
require_admin,
|
||||
require_origin,
|
||||
require_session,
|
||||
)
|
||||
|
||||
router = APIRouter(tags=["auth"])
|
||||
|
||||
|
||||
@router.post("/auth/register", response_model=UserRead, status_code=201)
|
||||
def register(
|
||||
payload: RegisterInput,
|
||||
request: Request,
|
||||
db: Session = Depends(get_db),
|
||||
origin=Depends(require_origin),
|
||||
):
|
||||
"""Registration never accepts role, ID, password hash or credential version fields."""
|
||||
limit_auth_requests(db, request)
|
||||
user = User(
|
||||
username=payload.username,
|
||||
password_hash=PASSWORD_HASHER.hash(payload.password.get_secret_value()),
|
||||
nickname=payload.nickname,
|
||||
email=str(payload.email) if payload.email else None,
|
||||
role="user",
|
||||
)
|
||||
db.add(user)
|
||||
db.commit()
|
||||
return user
|
||||
|
||||
|
||||
@router.post("/auth/login", response_model=SessionRead)
|
||||
def login(
|
||||
payload: LoginInput,
|
||||
request: Request,
|
||||
response: Response,
|
||||
db: Session = Depends(get_db),
|
||||
origin=Depends(require_origin),
|
||||
):
|
||||
limit_auth_requests(db, request, payload.username)
|
||||
user = db.scalar(select(User).where(User.username == payload.username).with_for_update())
|
||||
encoded = user.password_hash if user else dummy_hash()
|
||||
if not check_password(encoded, payload.password.get_secret_value()) or user is None:
|
||||
raise HTTPException(401, "用户名或密码错误")
|
||||
if PASSWORD_HASHER.check_needs_rehash(user.password_hash):
|
||||
user.password_hash = PASSWORD_HASHER.hash(payload.password.get_secret_value())
|
||||
return issue_session(db, user, request, response)
|
||||
|
||||
|
||||
@router.get("/auth/me", response_model=SessionRead)
|
||||
def me(current: CurrentSession = Depends(require_session)):
|
||||
return current.response()
|
||||
|
||||
|
||||
@router.post("/auth/logout", status_code=204)
|
||||
def logout(
|
||||
response: Response,
|
||||
current: CurrentSession = Depends(require_session),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
db.delete(current.session)
|
||||
db.commit()
|
||||
clear_cookie(response)
|
||||
|
||||
|
||||
@router.patch("/users/me", response_model=UserRead)
|
||||
def update_profile(
|
||||
payload: ProfileInput,
|
||||
current: CurrentSession = Depends(require_session),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
if not payload.model_fields_set:
|
||||
raise HTTPException(422, "请至少填写一个可修改字段")
|
||||
for key, value in payload.model_dump(exclude_unset=True, mode="json").items():
|
||||
setattr(current.user, key, value)
|
||||
db.commit()
|
||||
return current.user
|
||||
|
||||
|
||||
@router.post("/auth/change-password", status_code=204)
|
||||
def change_password(
|
||||
payload: PasswordChangeInput,
|
||||
response: Response,
|
||||
current: CurrentSession = Depends(require_session),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
if not check_password(current.user.password_hash, payload.current_password.get_secret_value()):
|
||||
raise HTTPException(400, "当前密码不正确")
|
||||
current.user.password_hash = PASSWORD_HASHER.hash(payload.new_password.get_secret_value())
|
||||
current.user.credential_version += 1
|
||||
db.execute(delete(AuthSession).where(AuthSession.user_id == current.user.id))
|
||||
db.commit()
|
||||
clear_cookie(response)
|
||||
|
||||
|
||||
@router.get("/admin/users", response_model=list[UserRead], dependencies=[Depends(require_admin)])
|
||||
def list_users(
|
||||
limit: int = Query(100, ge=1, le=100),
|
||||
offset: int = Query(0, ge=0),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
return list(db.scalars(select(User).order_by(User.id).offset(offset).limit(limit)))
|
||||
@@ -1,15 +1,7 @@
|
||||
from fastapi import APIRouter
|
||||
|
||||
from app.schemas.chat import ChatRequest, ChatResponse
|
||||
|
||||
router = APIRouter(tags=["chat"])
|
||||
|
||||
|
||||
@router.post("/chat", response_model=ChatResponse)
|
||||
def chat(payload: ChatRequest) -> ChatResponse:
|
||||
# TODO: retrieval -> context assembly -> LLM -> evidence/source checks
|
||||
return ChatResponse(
|
||||
answer=f"已收到问题:{payload.question}。当前为基础壳子,尚未接入知识检索或大语言模型。",
|
||||
status="stub",
|
||||
sources=[],
|
||||
)
|
||||
return ChatResponse(answer=f"已收到问题:{payload.question}。当前为基础壳子,尚未接入知识检索或大语言模型。", status="stub", sources=[])
|
||||
|
||||
@@ -1,105 +0,0 @@
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Response
|
||||
from sqlalchemy import or_, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.db.session import get_db
|
||||
from app.models import Document, KnowledgeItem, Source
|
||||
from app.schemas.catalog import AuditRead, DocumentCreate, DocumentRead, DocumentUpdate, Page
|
||||
from app.services.auth import require_admin
|
||||
from app.services.catalog import (
|
||||
ensure_version,
|
||||
get_item,
|
||||
history,
|
||||
invalidate_knowledge,
|
||||
page_result,
|
||||
record_event,
|
||||
update_values,
|
||||
)
|
||||
from app.services.indexing import catalog_write_guard
|
||||
|
||||
router = APIRouter(
|
||||
prefix="/documents",
|
||||
tags=["documents"],
|
||||
dependencies=[Depends(require_admin), Depends(catalog_write_guard)],
|
||||
)
|
||||
|
||||
|
||||
@router.get("", response_model=Page[DocumentRead])
|
||||
def list_documents(
|
||||
q: str = Query("", max_length=200),
|
||||
source_id: int | None = Query(None, gt=0),
|
||||
category: str | None = Query(None, max_length=80),
|
||||
page: int = Query(1, ge=1),
|
||||
page_size: int = Query(20, ge=1, le=100),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
statement = select(Document).where(Document.is_deleted.is_(False))
|
||||
if source_id is not None:
|
||||
statement = statement.where(Document.source_id == source_id)
|
||||
if category is not None:
|
||||
statement = statement.where(Document.category == category)
|
||||
if q:
|
||||
statement = statement.where(
|
||||
or_(
|
||||
Document.title.icontains(q, autoescape=True),
|
||||
Document.content.icontains(q, autoescape=True),
|
||||
)
|
||||
)
|
||||
return page_result(db, statement.order_by(Document.id.desc()), page, page_size)
|
||||
|
||||
|
||||
@router.post("", response_model=DocumentRead, status_code=201)
|
||||
def create_document(payload: DocumentCreate, db: Session = Depends(get_db)):
|
||||
get_item(db, Source, payload.source_id, lock=True)
|
||||
item = Document(**payload.model_dump())
|
||||
db.add(item)
|
||||
record_event(db, item, "create")
|
||||
db.commit()
|
||||
return item
|
||||
|
||||
|
||||
@router.get("/{item_id}", response_model=DocumentRead)
|
||||
def read_document(item_id: int, db: Session = Depends(get_db)):
|
||||
return get_item(db, Document, item_id)
|
||||
|
||||
|
||||
@router.patch("/{item_id}", response_model=DocumentRead)
|
||||
def update_document(item_id: int, payload: DocumentUpdate, db: Session = Depends(get_db)):
|
||||
item = get_item(db, Document, item_id, lock=True)
|
||||
ensure_version(item, payload.expected_version)
|
||||
update_values(item, payload)
|
||||
record_event(db, item, "update")
|
||||
invalidate_knowledge(db, KnowledgeItem.document_id == item.id)
|
||||
db.commit()
|
||||
return item
|
||||
|
||||
|
||||
@router.delete("/{item_id}", status_code=204)
|
||||
def delete_document(
|
||||
item_id: int, expected_version: int = Query(..., ge=1), db: Session = Depends(get_db)
|
||||
):
|
||||
item = get_item(db, Document, item_id, lock=True)
|
||||
ensure_version(item, expected_version)
|
||||
if (
|
||||
db.scalar(
|
||||
select(KnowledgeItem.id)
|
||||
.where(KnowledgeItem.document_id == item_id, KnowledgeItem.is_deleted.is_(False))
|
||||
.limit(1)
|
||||
)
|
||||
is not None
|
||||
):
|
||||
raise HTTPException(409, "文档仍被知识条目引用,不能删除")
|
||||
item.is_deleted = True
|
||||
record_event(db, item, "delete")
|
||||
db.commit()
|
||||
return Response(status_code=204)
|
||||
|
||||
|
||||
@router.get("/{item_id}/history", response_model=Page[AuditRead])
|
||||
def document_history(
|
||||
item_id: int,
|
||||
page: int = Query(1, ge=1),
|
||||
page_size: int = Query(20, ge=1, le=100),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
return history(db, Document, item_id, page, page_size)
|
||||
@@ -1,18 +1,5 @@
|
||||
from fastapi import APIRouter, Depends
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.db.session import get_db
|
||||
|
||||
from fastapi import APIRouter
|
||||
router = APIRouter(tags=["health"])
|
||||
|
||||
|
||||
@router.get("/health")
|
||||
def health():
|
||||
return {"status": "ok", "service": "nalati-rag-api"}
|
||||
|
||||
|
||||
@router.get("/ready")
|
||||
def ready(db: Session = Depends(get_db)):
|
||||
db.execute(text("SELECT 1"))
|
||||
return {"status": "ok", "database": "connected"}
|
||||
|
||||
@@ -1,91 +0,0 @@
|
||||
from uuid import UUID
|
||||
|
||||
from fastapi import APIRouter, Depends, Query
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.db.session import get_db
|
||||
from app.models.indexing import Chunk, IndexPublication, IndexVersion
|
||||
from app.schemas.catalog import Page
|
||||
from app.schemas.indexing import (
|
||||
ActiveIndex,
|
||||
BuildIndex,
|
||||
ChunkRead,
|
||||
IndexRead,
|
||||
PublicationRead,
|
||||
PublishIndex,
|
||||
)
|
||||
from app.services.auth import CurrentSession, require_admin
|
||||
from app.services.catalog import page_result
|
||||
from app.services.indexing import active_index, build_index, publish_index, version_or_404
|
||||
|
||||
router = APIRouter(prefix="/indexes", tags=["indexes"], dependencies=[Depends(require_admin)])
|
||||
|
||||
|
||||
@router.post("", response_model=IndexRead, status_code=201)
|
||||
def build(
|
||||
payload: BuildIndex,
|
||||
current: CurrentSession = Depends(require_admin),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
return build_index(db, payload, current.user.id)
|
||||
|
||||
|
||||
@router.get("", response_model=Page[IndexRead])
|
||||
def list_versions(
|
||||
page: int = Query(1, ge=1),
|
||||
page_size: int = Query(20, ge=1, le=100),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
return page_result(
|
||||
db,
|
||||
select(IndexVersion).order_by(IndexVersion.created_at.desc(), IndexVersion.id),
|
||||
page,
|
||||
page_size,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/active", response_model=ActiveIndex)
|
||||
def active(db: Session = Depends(get_db)):
|
||||
return active_index(db)
|
||||
|
||||
|
||||
@router.get("/publications", response_model=Page[PublicationRead])
|
||||
def publications(
|
||||
page: int = Query(1, ge=1),
|
||||
page_size: int = Query(20, ge=1, le=100),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
return page_result(
|
||||
db, select(IndexPublication).order_by(IndexPublication.generation.desc()), page, page_size
|
||||
)
|
||||
|
||||
|
||||
@router.get("/{version_id}", response_model=IndexRead)
|
||||
def read_version(version_id: UUID, db: Session = Depends(get_db)):
|
||||
return version_or_404(db, str(version_id))
|
||||
|
||||
|
||||
@router.post("/{version_id}/publish", response_model=ActiveIndex)
|
||||
def publish(
|
||||
version_id: UUID,
|
||||
payload: PublishIndex,
|
||||
current: CurrentSession = Depends(require_admin),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
return publish_index(db, version_id, payload.expected_generation, current.user.id)
|
||||
|
||||
|
||||
@router.get("/{version_id}/chunks", response_model=Page[ChunkRead])
|
||||
def chunks(
|
||||
version_id: UUID,
|
||||
knowledge_id: int | None = Query(None, gt=0),
|
||||
page: int = Query(1, ge=1),
|
||||
page_size: int = Query(20, ge=1, le=100),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
version_or_404(db, str(version_id))
|
||||
statement = select(Chunk).where(Chunk.index_version_id == str(version_id))
|
||||
if knowledge_id is not None:
|
||||
statement = statement.where(Chunk.knowledge_id == knowledge_id)
|
||||
return page_result(db, statement.order_by(Chunk.knowledge_id, Chunk.ordinal), page, page_size)
|
||||
@@ -1,157 +1,15 @@
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Response
|
||||
from sqlalchemy import or_, select
|
||||
from fastapi import APIRouter, Depends
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.db.session import get_db
|
||||
from app.models import KnowledgeItem
|
||||
from app.models.catalog import utc_now
|
||||
from app.schemas.catalog import AuditRead, Page, Status
|
||||
from app.schemas.knowledge import KnowledgeCreate, KnowledgeRead, KnowledgeReview, KnowledgeUpdate
|
||||
from app.services.auth import require_admin
|
||||
from app.services.catalog import (
|
||||
ensure_version,
|
||||
get_item,
|
||||
history,
|
||||
page_result,
|
||||
record_event,
|
||||
reset_review,
|
||||
resolve_knowledge_links,
|
||||
update_values,
|
||||
)
|
||||
from app.services.indexing import catalog_write_guard
|
||||
|
||||
router = APIRouter(
|
||||
prefix="/knowledge",
|
||||
tags=["knowledge"],
|
||||
dependencies=[Depends(require_admin), Depends(catalog_write_guard)],
|
||||
)
|
||||
|
||||
|
||||
def filtered_query(q="", category=None, verification_status=None, source_id=None, document_id=None):
|
||||
statement = select(KnowledgeItem).where(KnowledgeItem.is_deleted.is_(False))
|
||||
for column, value in (
|
||||
(KnowledgeItem.category, category),
|
||||
(KnowledgeItem.verification_status, verification_status),
|
||||
(KnowledgeItem.source_id, source_id),
|
||||
(KnowledgeItem.document_id, document_id),
|
||||
):
|
||||
if value is not None:
|
||||
statement = statement.where(column == value)
|
||||
if q:
|
||||
statement = statement.where(
|
||||
or_(
|
||||
KnowledgeItem.title.icontains(q, autoescape=True),
|
||||
KnowledgeItem.content.icontains(q, autoescape=True),
|
||||
)
|
||||
)
|
||||
return statement.order_by(KnowledgeItem.id.desc())
|
||||
|
||||
|
||||
@router.get("", response_model=list[KnowledgeRead])
|
||||
def list_knowledge(
|
||||
q: str = Query("", max_length=200),
|
||||
category: str | None = Query(None, max_length=80),
|
||||
verification_status: Status | None = None,
|
||||
limit: int = Query(100, ge=1, le=200),
|
||||
offset: int = Query(0, ge=0),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""保留数组响应;完整分页与关联筛选使用 /knowledge/page。"""
|
||||
return list(
|
||||
db.scalars(filtered_query(q, category, verification_status).offset(offset).limit(limit))
|
||||
)
|
||||
|
||||
|
||||
@router.get("/page", response_model=Page[KnowledgeRead])
|
||||
def paged_knowledge(
|
||||
q: str = Query("", max_length=200),
|
||||
category: str | None = Query(None, max_length=80),
|
||||
verification_status: Status | None = None,
|
||||
source_id: int | None = Query(None, gt=0),
|
||||
document_id: int | None = Query(None, gt=0),
|
||||
page: int = Query(1, ge=1),
|
||||
page_size: int = Query(20, ge=1, le=100),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
return page_result(
|
||||
db,
|
||||
filtered_query(q, category, verification_status, source_id, document_id),
|
||||
page,
|
||||
page_size,
|
||||
)
|
||||
|
||||
|
||||
@router.post("", response_model=KnowledgeRead, status_code=201)
|
||||
from app.models.knowledge import KnowledgeItem
|
||||
from app.schemas.knowledge import KnowledgeCreate, KnowledgeRead
|
||||
router = APIRouter(tags=["knowledge"])
|
||||
@router.get("/knowledge", response_model=list[KnowledgeRead])
|
||||
def list_knowledge(db: Session = Depends(get_db)):
|
||||
return list(db.scalars(select(KnowledgeItem).order_by(KnowledgeItem.id.desc())).all())
|
||||
@router.post("/knowledge", response_model=KnowledgeRead, status_code=201)
|
||||
def create_knowledge(payload: KnowledgeCreate, db: Session = Depends(get_db)):
|
||||
item = KnowledgeItem(
|
||||
**payload.model_dump(exclude={"source_url"}),
|
||||
source_url=str(payload.source_url) if payload.source_url else None,
|
||||
verification_status="pending",
|
||||
)
|
||||
resolve_knowledge_links(db, item)
|
||||
db.add(item)
|
||||
record_event(db, item, "create")
|
||||
db.commit()
|
||||
item = KnowledgeItem(title=payload.title, category=payload.category, content=payload.content, source_url=str(payload.source_url) if payload.source_url else None, verification_status="pending")
|
||||
db.add(item); db.commit(); db.refresh(item)
|
||||
return item
|
||||
|
||||
|
||||
@router.get("/{item_id}", response_model=KnowledgeRead)
|
||||
def read_knowledge(item_id: int, db: Session = Depends(get_db)):
|
||||
return get_item(db, KnowledgeItem, item_id)
|
||||
|
||||
|
||||
@router.patch("/{item_id}", response_model=KnowledgeRead)
|
||||
def update_knowledge(item_id: int, payload: KnowledgeUpdate, db: Session = Depends(get_db)):
|
||||
item = get_item(db, KnowledgeItem, item_id)
|
||||
ensure_version(item, payload.expected_version)
|
||||
update_values(item, payload)
|
||||
if payload.document_id is not None and "source_id" not in payload.model_fields_set:
|
||||
item.source_id = None
|
||||
if {
|
||||
"source_id",
|
||||
"document_id",
|
||||
} & payload.model_fields_set and "source_url" not in payload.model_fields_set:
|
||||
item.source_url = None
|
||||
resolve_knowledge_links(db, item)
|
||||
reset_review(item)
|
||||
record_event(db, item, "update")
|
||||
db.commit()
|
||||
return item
|
||||
|
||||
|
||||
@router.post("/{item_id}/review", response_model=KnowledgeRead)
|
||||
def review_knowledge(item_id: int, payload: KnowledgeReview, db: Session = Depends(get_db)):
|
||||
item = get_item(db, KnowledgeItem, item_id)
|
||||
ensure_version(item, payload.expected_version)
|
||||
resolve_knowledge_links(db, item)
|
||||
if payload.verification_status == "verified" and (not item.source_url or not item.collected_at):
|
||||
raise HTTPException(422, "核验通过前必须补全来源网址和真实采集时间")
|
||||
item.verification_status = payload.verification_status
|
||||
item.reviewer = payload.reviewer
|
||||
item.reviewed_at = utc_now()
|
||||
item.verification_notes = payload.verification_notes
|
||||
record_event(db, item, "review", actor=payload.reviewer, notes=payload.verification_notes)
|
||||
db.commit()
|
||||
return item
|
||||
|
||||
|
||||
@router.delete("/{item_id}", status_code=204)
|
||||
def delete_knowledge(
|
||||
item_id: int, expected_version: int = Query(..., ge=1), db: Session = Depends(get_db)
|
||||
):
|
||||
item = get_item(db, KnowledgeItem, item_id)
|
||||
ensure_version(item, expected_version)
|
||||
item.is_deleted = True
|
||||
record_event(db, item, "delete")
|
||||
db.commit()
|
||||
return Response(status_code=204)
|
||||
|
||||
|
||||
@router.get("/{item_id}/history", response_model=Page[AuditRead])
|
||||
def knowledge_history(
|
||||
item_id: int,
|
||||
page: int = Query(1, ge=1),
|
||||
page_size: int = Query(20, ge=1, le=100),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
return history(db, KnowledgeItem, item_id, page, page_size)
|
||||
|
||||
@@ -1,50 +0,0 @@
|
||||
from uuid import UUID
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.db.session import get_db
|
||||
from app.models import RetrievalRun
|
||||
from app.schemas.catalog import Page
|
||||
from app.schemas.retrieval import RetrievalRequest, RetrievalResponse
|
||||
from app.services.auth import require_admin
|
||||
from app.services.retrieval import retrieve
|
||||
|
||||
router = APIRouter(prefix="/retrieval", tags=["retrieval"])
|
||||
|
||||
|
||||
@router.post("", response_model=RetrievalResponse)
|
||||
def keyword_retrieval(payload: RetrievalRequest, db: Session = Depends(get_db)):
|
||||
"""只检索已核验知识;保存原始检索结果供复核,不调用大模型。"""
|
||||
return retrieve(db, payload)
|
||||
|
||||
|
||||
@router.get("/runs", response_model=Page[RetrievalResponse], dependencies=[Depends(require_admin)])
|
||||
def list_runs(
|
||||
page: int = Query(1, ge=1),
|
||||
page_size: int = Query(20, ge=1, le=100),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
rows = db.scalars(
|
||||
select(RetrievalRun)
|
||||
.order_by(RetrievalRun.created_at.desc(), RetrievalRun.id)
|
||||
.offset((page - 1) * page_size)
|
||||
.limit(page_size)
|
||||
)
|
||||
return {
|
||||
"items": [row.response for row in rows],
|
||||
"total": db.scalar(select(func.count()).select_from(RetrievalRun)),
|
||||
"page": page,
|
||||
"page_size": page_size,
|
||||
}
|
||||
|
||||
|
||||
@router.get(
|
||||
"/runs/{run_id}", response_model=RetrievalResponse, dependencies=[Depends(require_admin)]
|
||||
)
|
||||
def read_run(run_id: UUID, db: Session = Depends(get_db)):
|
||||
row = db.get(RetrievalRun, str(run_id))
|
||||
if row is None:
|
||||
raise HTTPException(404, "检索记录不存在")
|
||||
return row.response
|
||||
@@ -1,101 +0,0 @@
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Response
|
||||
from sqlalchemy import or_, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.db.session import get_db
|
||||
from app.models import Document, KnowledgeItem, Source
|
||||
from app.schemas.catalog import AuditRead, Page, SourceCreate, SourceRead, SourceUpdate
|
||||
from app.services.auth import require_admin
|
||||
from app.services.catalog import (
|
||||
ensure_version,
|
||||
get_item,
|
||||
history,
|
||||
invalidate_knowledge,
|
||||
page_result,
|
||||
record_event,
|
||||
update_values,
|
||||
)
|
||||
from app.services.indexing import catalog_write_guard
|
||||
|
||||
router = APIRouter(
|
||||
prefix="/sources",
|
||||
tags=["sources"],
|
||||
dependencies=[Depends(require_admin), Depends(catalog_write_guard)],
|
||||
)
|
||||
|
||||
|
||||
@router.get("", response_model=Page[SourceRead])
|
||||
def list_sources(
|
||||
q: str = Query("", max_length=200),
|
||||
page: int = Query(1, ge=1),
|
||||
page_size: int = Query(20, ge=1, le=100),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
statement = select(Source).where(Source.is_deleted.is_(False))
|
||||
if q:
|
||||
statement = statement.where(
|
||||
or_(
|
||||
Source.name.icontains(q, autoescape=True),
|
||||
Source.notes.icontains(q, autoescape=True),
|
||||
)
|
||||
)
|
||||
return page_result(db, statement.order_by(Source.id.desc()), page, page_size)
|
||||
|
||||
|
||||
@router.post("", response_model=SourceRead, status_code=201)
|
||||
def create_source(payload: SourceCreate, db: Session = Depends(get_db)):
|
||||
item = Source(**payload.model_dump(mode="json"))
|
||||
db.add(item)
|
||||
record_event(db, item, "create")
|
||||
db.commit()
|
||||
return item
|
||||
|
||||
|
||||
@router.get("/{item_id}", response_model=SourceRead)
|
||||
def read_source(item_id: int, db: Session = Depends(get_db)):
|
||||
return get_item(db, Source, item_id)
|
||||
|
||||
|
||||
@router.patch("/{item_id}", response_model=SourceRead)
|
||||
def update_source(item_id: int, payload: SourceUpdate, db: Session = Depends(get_db)):
|
||||
item = get_item(db, Source, item_id, lock=True)
|
||||
ensure_version(item, payload.expected_version)
|
||||
update_values(item, payload)
|
||||
record_event(db, item, "update")
|
||||
invalidate_knowledge(
|
||||
db, KnowledgeItem.source_id == item.id, source_url=item.source_url, update_url=True
|
||||
)
|
||||
db.commit()
|
||||
return item
|
||||
|
||||
|
||||
@router.delete("/{item_id}", status_code=204)
|
||||
def delete_source(
|
||||
item_id: int, expected_version: int = Query(..., ge=1), db: Session = Depends(get_db)
|
||||
):
|
||||
item = get_item(db, Source, item_id, lock=True)
|
||||
ensure_version(item, expected_version)
|
||||
for model in (Document, KnowledgeItem):
|
||||
if (
|
||||
db.scalar(
|
||||
select(model.id)
|
||||
.where(model.source_id == item_id, model.is_deleted.is_(False))
|
||||
.limit(1)
|
||||
)
|
||||
is not None
|
||||
):
|
||||
raise HTTPException(409, "来源仍被文档或知识条目引用,不能删除")
|
||||
item.is_deleted = True
|
||||
record_event(db, item, "delete")
|
||||
db.commit()
|
||||
return Response(status_code=204)
|
||||
|
||||
|
||||
@router.get("/{item_id}/history", response_model=Page[AuditRead])
|
||||
def source_history(
|
||||
item_id: int,
|
||||
page: int = Query(1, ge=1),
|
||||
page_size: int = Query(20, ge=1, le=100),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
return history(db, Source, item_id, page, page_size)
|
||||
@@ -1,53 +1,14 @@
|
||||
from pathlib import Path
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from pydantic import Field, model_validator
|
||||
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||
|
||||
|
||||
class Settings(BaseSettings):
|
||||
app_name: str = "Nalati Tourism QA API"
|
||||
app_env: str = "development"
|
||||
api_prefix: str = "/api"
|
||||
database_url: str = "postgresql+psycopg://nalati:change-this-password@127.0.0.1:5432/nalati_rag"
|
||||
cors_origins: str = "http://localhost:5173,http://127.0.0.1:5173"
|
||||
auth_allowed_origins: str = (
|
||||
"http://localhost:5173,http://127.0.0.1:5173,http://localhost:8000,http://127.0.0.1:8000"
|
||||
)
|
||||
auth_cookie_secure: bool = True
|
||||
auth_session_seconds: int = Field(28800, ge=300, le=604800)
|
||||
model_config = SettingsConfigDict(
|
||||
env_file=Path(__file__).resolve().parents[2] / ".env", extra="ignore"
|
||||
)
|
||||
|
||||
model_config = SettingsConfigDict(env_file=".env", extra="ignore")
|
||||
@property
|
||||
def cors_origin_list(self) -> list[str]:
|
||||
return [x.strip() for x in self.cors_origins.split(",") if x.strip()]
|
||||
|
||||
@property
|
||||
def auth_origin_list(self) -> list[str]:
|
||||
return [x.strip() for x in self.auth_allowed_origins.split(",") if x.strip()]
|
||||
|
||||
@model_validator(mode="after")
|
||||
def validate_auth_settings(self):
|
||||
if not self.auth_cookie_secure and self.app_env not in {"development", "test"}:
|
||||
raise ValueError("非开发/测试环境必须使用 Secure Cookie 与 HTTPS")
|
||||
if not self.auth_origin_list:
|
||||
raise ValueError("必须显式设置认证允许的来源")
|
||||
for origin in [*self.auth_origin_list, *self.cors_origin_list]:
|
||||
parsed = urlsplit(origin)
|
||||
if (
|
||||
parsed.scheme not in {"http", "https"}
|
||||
or not parsed.hostname
|
||||
or parsed.netloc != parsed.hostname + (f":{parsed.port}" if parsed.port else "")
|
||||
or parsed.path
|
||||
or parsed.query
|
||||
or parsed.fragment
|
||||
or "*" in origin
|
||||
or (self.app_env not in {"development", "test"} and parsed.scheme != "https")
|
||||
):
|
||||
raise ValueError("认证和跨域来源须为明确的 HTTP(S) Origin,不允许通配符或路径")
|
||||
return self
|
||||
|
||||
|
||||
settings = Settings()
|
||||
|
||||
@@ -1,25 +1,16 @@
|
||||
from sqlalchemy import create_engine
|
||||
from sqlalchemy.orm import DeclarativeBase, sessionmaker
|
||||
|
||||
from app.core.config import settings
|
||||
|
||||
|
||||
class Base(DeclarativeBase):
|
||||
pass
|
||||
|
||||
|
||||
engine = create_engine(
|
||||
settings.database_url, pool_pre_ping=True, connect_args={"connect_timeout": 5}
|
||||
)
|
||||
engine = create_engine(settings.database_url, pool_pre_ping=True)
|
||||
SessionLocal = sessionmaker(bind=engine, autoflush=False, autocommit=False)
|
||||
|
||||
|
||||
def get_db():
|
||||
db = SessionLocal()
|
||||
try:
|
||||
yield db
|
||||
except Exception:
|
||||
db.rollback()
|
||||
raise
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
+18
-91
@@ -1,98 +1,25 @@
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
from fastapi import FastAPI
|
||||
from fastapi.exceptions import RequestValidationError
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
from fastapi.responses import JSONResponse
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.exc import IntegrityError, OperationalError, ProgrammingError
|
||||
from sqlalchemy.orm.exc import StaleDataError
|
||||
|
||||
from app.api.routes import auth, chat, documents, health, indexes, knowledge, retrieval, sources
|
||||
from app.core.config import settings
|
||||
from app.db.session import engine
|
||||
from app.services.auth import dummy_hash
|
||||
from app.services.retrieval import tokenizer
|
||||
from app.db.session import Base, engine
|
||||
from app.models import KnowledgeItem # noqa: F401
|
||||
from app.api.routes import health, chat, knowledge
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
# Convenience for local bootstrap. Use Alembic migrations before production.
|
||||
with engine.begin() as connection:
|
||||
connection.execute(text("CREATE EXTENSION IF NOT EXISTS vector"))
|
||||
Base.metadata.create_all(bind=engine)
|
||||
yield
|
||||
|
||||
def create_app(db_engine=engine) -> FastAPI:
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
try:
|
||||
with db_engine.connect() as connection:
|
||||
revision = connection.scalar(text("SELECT version_num FROM alembic_version"))
|
||||
if revision != "0005_chunk_indexes":
|
||||
raise RuntimeError(
|
||||
"数据库版本不匹配,请先在 backend 目录执行 alembic upgrade head"
|
||||
)
|
||||
except ProgrammingError as exc:
|
||||
raise RuntimeError(
|
||||
"数据库尚未初始化,请先在 backend 目录执行 alembic upgrade head"
|
||||
) from exc
|
||||
tokenizer()
|
||||
dummy_hash()
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
db_engine.dispose()
|
||||
|
||||
app = FastAPI(title=settings.app_name, version="0.5.0", lifespan=lifespan)
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=settings.cors_origin_list,
|
||||
allow_credentials=True,
|
||||
allow_methods=["*"],
|
||||
allow_headers=["*"],
|
||||
)
|
||||
for router in (
|
||||
health.router,
|
||||
auth.router,
|
||||
indexes.router,
|
||||
chat.router,
|
||||
knowledge.router,
|
||||
sources.router,
|
||||
documents.router,
|
||||
retrieval.router,
|
||||
):
|
||||
app.include_router(router, prefix=settings.api_prefix)
|
||||
|
||||
@app.middleware("http")
|
||||
async def no_cache_api(request, call_next):
|
||||
response = await call_next(request)
|
||||
if request.url.path.startswith(settings.api_prefix + "/"):
|
||||
response.headers["Cache-Control"] = "no-store"
|
||||
return response
|
||||
|
||||
@app.exception_handler(RequestValidationError)
|
||||
async def validation_handler(request, exc):
|
||||
# FastAPI's default includes raw input (including passwords) in error responses.
|
||||
return JSONResponse(
|
||||
status_code=422,
|
||||
content={
|
||||
"detail": [
|
||||
{"loc": list(error["loc"]), "msg": error["msg"], "type": error["type"]}
|
||||
for error in exc.errors()
|
||||
]
|
||||
},
|
||||
)
|
||||
|
||||
@app.exception_handler(StaleDataError)
|
||||
async def stale_data_handler(request, exc):
|
||||
return JSONResponse(status_code=409, content={"detail": "记录已更新,请刷新后重试"})
|
||||
|
||||
@app.exception_handler(IntegrityError)
|
||||
async def integrity_handler(request, exc):
|
||||
return JSONResponse(status_code=409, content={"detail": "数据关联或约束冲突"})
|
||||
|
||||
@app.exception_handler(OperationalError)
|
||||
async def database_unavailable_handler(request, exc):
|
||||
return JSONResponse(status_code=503, content={"detail": "数据库暂不可用"})
|
||||
|
||||
@app.get("/")
|
||||
def root():
|
||||
return {"message": "Nalati Tourism QA API", "docs": "/docs"}
|
||||
|
||||
return app
|
||||
|
||||
|
||||
app = create_app()
|
||||
app = FastAPI(title=settings.app_name, version="0.1.0", lifespan=lifespan)
|
||||
app.add_middleware(CORSMiddleware, allow_origins=settings.cors_origin_list, allow_credentials=True, allow_methods=["*"], allow_headers=["*"])
|
||||
app.include_router(health.router, prefix=settings.api_prefix)
|
||||
app.include_router(chat.router, prefix=settings.api_prefix)
|
||||
app.include_router(knowledge.router, prefix=settings.api_prefix)
|
||||
@app.get("/")
|
||||
def root():
|
||||
return {"message": "Nalati Tourism QA API", "docs": "/docs"}
|
||||
|
||||
@@ -1,47 +0,0 @@
|
||||
"""Local-only administrator provisioning. Passwords are read from the terminal, never argv."""
|
||||
|
||||
import argparse
|
||||
import getpass
|
||||
|
||||
from pydantic import ValidationError
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
|
||||
from app.db.session import SessionLocal
|
||||
from app.models.auth import User
|
||||
from app.schemas.auth import RegisterInput
|
||||
from app.services.auth import PASSWORD_HASHER
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="本机管理员初始化(须先完成数据库迁移)")
|
||||
parser.add_argument("command", choices=["create-admin"])
|
||||
parser.add_argument("--username", required=True)
|
||||
args = parser.parse_args()
|
||||
password = getpass.getpass("管理员密码(至少 15 字符):")
|
||||
if password != getpass.getpass("再次输入密码:"):
|
||||
parser.exit(1, "两次密码不一致\n")
|
||||
try:
|
||||
payload = RegisterInput(username=args.username, password=password)
|
||||
except ValidationError:
|
||||
parser.exit(1, "用户名须为 3–32 位字母开头的小写字母/数字/下划线;密码须为 15–128 字符\n")
|
||||
with SessionLocal() as db:
|
||||
if db.scalar(select(User.id).where(User.username == payload.username)) is not None:
|
||||
parser.exit(1, "用户已存在,未修改原有用户或权限\n")
|
||||
db.add(
|
||||
User(
|
||||
username=payload.username,
|
||||
password_hash=PASSWORD_HASHER.hash(password),
|
||||
role="admin",
|
||||
)
|
||||
)
|
||||
try:
|
||||
db.commit()
|
||||
except IntegrityError:
|
||||
db.rollback()
|
||||
parser.exit(1, "用户已存在,未修改原有用户或权限\n")
|
||||
print("管理员已创建;请通过登录接口建立会话。")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,20 +1 @@
|
||||
from app.models.auth import AuthRateLimit, AuthSession, User
|
||||
from app.models.catalog import AuditEvent, Document, Source
|
||||
from app.models.indexing import Chunk, IndexPublication, IndexState, IndexVersion
|
||||
from app.models.knowledge import KnowledgeItem
|
||||
from app.models.retrieval import RetrievalRun
|
||||
|
||||
__all__ = [
|
||||
"AuditEvent",
|
||||
"AuthRateLimit",
|
||||
"AuthSession",
|
||||
"Chunk",
|
||||
"Document",
|
||||
"IndexPublication",
|
||||
"IndexState",
|
||||
"IndexVersion",
|
||||
"KnowledgeItem",
|
||||
"RetrievalRun",
|
||||
"Source",
|
||||
"User",
|
||||
]
|
||||
|
||||
@@ -1,42 +0,0 @@
|
||||
"""Users, opaque sessions and shared authentication request budgets."""
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import CheckConstraint, DateTime, ForeignKey, Integer, String, func
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from app.db.session import Base
|
||||
|
||||
|
||||
class User(Base):
|
||||
__tablename__ = "users"
|
||||
__table_args__ = (CheckConstraint("role IN ('user', 'admin')", name="ck_users_role"),)
|
||||
|
||||
id: Mapped[int] = mapped_column(primary_key=True)
|
||||
username: Mapped[str] = mapped_column(String(32), unique=True, nullable=False)
|
||||
password_hash: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||
nickname: Mapped[str | None] = mapped_column(String(80))
|
||||
email: Mapped[str | None] = mapped_column(String(254))
|
||||
role: Mapped[str] = mapped_column(String(10), default="user", nullable=False)
|
||||
credential_version: Mapped[int] = mapped_column(Integer, default=1, nullable=False)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
|
||||
|
||||
|
||||
class AuthSession(Base):
|
||||
__tablename__ = "auth_sessions"
|
||||
|
||||
token_hash: Mapped[str] = mapped_column(String(64), primary_key=True)
|
||||
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), index=True)
|
||||
credential_version: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
|
||||
expires_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, index=True
|
||||
)
|
||||
|
||||
|
||||
class AuthRateLimit(Base):
|
||||
__tablename__ = "auth_rate_limits"
|
||||
|
||||
key: Mapped[str] = mapped_column(String(64), primary_key=True)
|
||||
window_start: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||
attempts: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
@@ -1,54 +0,0 @@
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy import JSON, Boolean, DateTime, ForeignKey, Integer, String, Text, func
|
||||
from sqlalchemy.orm import Mapped, declared_attr, mapped_column
|
||||
|
||||
from app.db.session import Base
|
||||
|
||||
|
||||
def utc_now() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
class Versioned:
|
||||
version: Mapped[int] = mapped_column(Integer, nullable=False, default=1)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
|
||||
updated_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), default=utc_now, onupdate=utc_now
|
||||
)
|
||||
is_deleted: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
|
||||
|
||||
@declared_attr.directive
|
||||
def __mapper_args__(cls):
|
||||
return {"version_id_col": cls.version}
|
||||
|
||||
|
||||
class Source(Versioned, Base):
|
||||
__tablename__ = "sources"
|
||||
id: Mapped[int] = mapped_column(primary_key=True)
|
||||
name: Mapped[str] = mapped_column(String(250), nullable=False)
|
||||
source_url: Mapped[str | None] = mapped_column(String(1000))
|
||||
notes: Mapped[str] = mapped_column(Text, nullable=False, default="")
|
||||
|
||||
|
||||
class Document(Versioned, Base):
|
||||
__tablename__ = "documents"
|
||||
id: Mapped[int] = mapped_column(primary_key=True)
|
||||
source_id: Mapped[int] = mapped_column(ForeignKey("sources.id"), index=True)
|
||||
title: Mapped[str] = mapped_column(String(250), nullable=False)
|
||||
category: Mapped[str] = mapped_column(String(80), nullable=False)
|
||||
content: Mapped[str] = mapped_column(Text, nullable=False)
|
||||
collected_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||
notes: Mapped[str] = mapped_column(Text, nullable=False, default="")
|
||||
|
||||
|
||||
class AuditEvent(Base):
|
||||
__tablename__ = "audit_events"
|
||||
id: Mapped[int] = mapped_column(primary_key=True)
|
||||
entity_type: Mapped[str] = mapped_column(String(30), nullable=False, index=True)
|
||||
entity_id: Mapped[int] = mapped_column(Integer, nullable=False, index=True)
|
||||
action: Mapped[str] = mapped_column(String(30), nullable=False)
|
||||
actor: Mapped[str | None] = mapped_column(String(100))
|
||||
notes: Mapped[str] = mapped_column(Text, nullable=False, default="")
|
||||
snapshot: Mapped[dict] = mapped_column(JSON, nullable=False)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
|
||||
@@ -1,74 +0,0 @@
|
||||
"""Immutable chunk snapshots and atomic publication metadata (no vector index yet)."""
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import (
|
||||
JSON,
|
||||
CheckConstraint,
|
||||
DateTime,
|
||||
ForeignKey,
|
||||
Integer,
|
||||
String,
|
||||
Text,
|
||||
UniqueConstraint,
|
||||
func,
|
||||
)
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from app.db.session import Base
|
||||
|
||||
|
||||
class IndexVersion(Base):
|
||||
__tablename__ = "index_versions"
|
||||
__table_args__ = (
|
||||
CheckConstraint(
|
||||
"status IN ('ready', 'published', 'superseded')", name="ck_index_version_status"
|
||||
),
|
||||
)
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True)
|
||||
status: Mapped[str] = mapped_column(String(20), nullable=False, default="ready")
|
||||
config: Mapped[dict] = mapped_column(JSON, nullable=False)
|
||||
corpus_fingerprint: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
catalog_revision: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
item_count: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
chunk_count: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
created_by: Mapped[int] = mapped_column(ForeignKey("users.id"), nullable=False)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
|
||||
published_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
|
||||
|
||||
class IndexState(Base):
|
||||
__tablename__ = "index_state"
|
||||
__table_args__ = (CheckConstraint("id = 1", name="ck_index_state_singleton"),)
|
||||
id: Mapped[int] = mapped_column(primary_key=True)
|
||||
active_version_id: Mapped[str | None] = mapped_column(ForeignKey("index_versions.id"))
|
||||
generation: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
catalog_revision: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
|
||||
|
||||
class Chunk(Base):
|
||||
__tablename__ = "chunks"
|
||||
__table_args__ = (
|
||||
UniqueConstraint("index_version_id", "knowledge_id", "ordinal", name="uq_chunk_position"),
|
||||
CheckConstraint("start_offset >= 0 AND end_offset > start_offset", name="ck_chunk_offsets"),
|
||||
)
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True)
|
||||
index_version_id: Mapped[str] = mapped_column(ForeignKey("index_versions.id"), index=True)
|
||||
knowledge_id: Mapped[int] = mapped_column(ForeignKey("knowledge_items.id"), nullable=False)
|
||||
ordinal: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
start_offset: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
end_offset: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
text: Mapped[str] = mapped_column(Text, nullable=False)
|
||||
text_sha256: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
provenance: Mapped[dict] = mapped_column(JSON, nullable=False)
|
||||
|
||||
|
||||
class IndexPublication(Base):
|
||||
__tablename__ = "index_publications"
|
||||
generation: Mapped[int] = mapped_column(primary_key=True)
|
||||
previous_version_id: Mapped[str | None] = mapped_column(ForeignKey("index_versions.id"))
|
||||
version_id: Mapped[str] = mapped_column(ForeignKey("index_versions.id"), nullable=False)
|
||||
actor_id: Mapped[int] = mapped_column(ForeignKey("users.id"), nullable=False)
|
||||
published_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), server_default=func.now()
|
||||
)
|
||||
@@ -1,29 +1,14 @@
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import CheckConstraint, DateTime, ForeignKey, Integer, String, Text
|
||||
from sqlalchemy import DateTime, Integer, String, Text, func
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from app.db.session import Base
|
||||
from app.models.catalog import Versioned
|
||||
|
||||
|
||||
class KnowledgeItem(Versioned, Base):
|
||||
class KnowledgeItem(Base):
|
||||
__tablename__ = "knowledge_items"
|
||||
__table_args__ = (
|
||||
CheckConstraint(
|
||||
"verification_status IN ('pending', 'verified', 'rejected')",
|
||||
name="ck_knowledge_verification_status",
|
||||
),
|
||||
)
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True, index=True)
|
||||
title: Mapped[str] = mapped_column(String(250), nullable=False, index=True)
|
||||
category: Mapped[str] = mapped_column(String(80), nullable=False, default="未分类")
|
||||
content: Mapped[str] = mapped_column(Text, nullable=False)
|
||||
source_url: Mapped[str | None] = mapped_column(String(1000), nullable=True)
|
||||
verification_status: Mapped[str] = mapped_column(String(30), nullable=False, default="pending")
|
||||
source_id: Mapped[int | None] = mapped_column(ForeignKey("sources.id"), index=True)
|
||||
document_id: Mapped[int | None] = mapped_column(ForeignKey("documents.id"), index=True)
|
||||
collected_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
reviewer: Mapped[str | None] = mapped_column(String(100))
|
||||
reviewed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
verification_notes: Mapped[str | None] = mapped_column(Text)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
|
||||
|
||||
@@ -1,18 +0,0 @@
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import JSON, DateTime, String, Text, func
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from app.db.session import Base
|
||||
|
||||
|
||||
class RetrievalRun(Base):
|
||||
__tablename__ = "retrieval_runs"
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True)
|
||||
query: Mapped[str] = mapped_column(Text, nullable=False)
|
||||
mode: Mapped[str] = mapped_column(String(20), nullable=False)
|
||||
corpus_fingerprint: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
response: Mapped[dict] = mapped_column(JSON, nullable=False)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), server_default=func.now(), index=True
|
||||
)
|
||||
@@ -1,67 +0,0 @@
|
||||
from datetime import datetime
|
||||
from typing import Annotated, Literal
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, EmailStr, Field, SecretStr, field_validator
|
||||
|
||||
Password = Annotated[SecretStr, Field(min_length=15, max_length=128)]
|
||||
|
||||
|
||||
class AuthInput(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
|
||||
class ProfileInput(AuthInput):
|
||||
nickname: str | None = Field(None, min_length=1, max_length=80)
|
||||
email: EmailStr | None = Field(None, max_length=254)
|
||||
|
||||
@field_validator("nickname", "email", mode="before")
|
||||
@classmethod
|
||||
def optional_text(cls, value):
|
||||
return (value.strip() or None) if isinstance(value, str) else value
|
||||
|
||||
@field_validator("nickname")
|
||||
@classmethod
|
||||
def no_control_characters(cls, value):
|
||||
if value and any(ord(char) < 32 or ord(char) == 127 for char in value):
|
||||
raise ValueError("昵称不能包含控制字符")
|
||||
return value
|
||||
|
||||
|
||||
class RegisterInput(ProfileInput):
|
||||
username: str = Field(min_length=3, max_length=32, pattern=r"^[a-z][a-z0-9_]{2,31}$")
|
||||
password: Password
|
||||
|
||||
@field_validator("username", mode="before")
|
||||
@classmethod
|
||||
def canonical_username(cls, value):
|
||||
return value.strip().lower() if isinstance(value, str) else value
|
||||
|
||||
|
||||
class LoginInput(AuthInput):
|
||||
username: str = Field(min_length=3, max_length=32, pattern=r"^[a-z][a-z0-9_]{2,31}$")
|
||||
password: SecretStr = Field(min_length=1, max_length=128)
|
||||
|
||||
@field_validator("username", mode="before")
|
||||
@classmethod
|
||||
def canonical_username(cls, value):
|
||||
return value.strip().lower() if isinstance(value, str) else value
|
||||
|
||||
|
||||
class PasswordChangeInput(AuthInput):
|
||||
current_password: SecretStr = Field(min_length=1, max_length=128)
|
||||
new_password: Password
|
||||
|
||||
|
||||
class UserRead(BaseModel):
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
id: int
|
||||
username: str
|
||||
nickname: str | None
|
||||
email: str | None
|
||||
role: Literal["user", "admin"]
|
||||
created_at: datetime
|
||||
|
||||
|
||||
class SessionRead(UserRead):
|
||||
csrf_token: str
|
||||
@@ -1,124 +0,0 @@
|
||||
from datetime import datetime
|
||||
from typing import Annotated, Generic, Literal, TypeVar
|
||||
|
||||
from pydantic import (
|
||||
AwareDatetime,
|
||||
BaseModel,
|
||||
ConfigDict,
|
||||
Field,
|
||||
HttpUrl,
|
||||
field_validator,
|
||||
model_validator,
|
||||
)
|
||||
|
||||
Title = Annotated[str, Field(min_length=1, max_length=250)]
|
||||
Category = Annotated[str, Field(min_length=1, max_length=80)]
|
||||
Body = Annotated[str, Field(min_length=1, max_length=100_000)]
|
||||
Notes = Annotated[str, Field(max_length=10_000)]
|
||||
Status = Literal["pending", "verified", "rejected"]
|
||||
PositiveId = Annotated[int, Field(gt=0)]
|
||||
|
||||
|
||||
class Input(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid", str_strip_whitespace=True)
|
||||
|
||||
@field_validator("source_url", mode="before", check_fields=False)
|
||||
@classmethod
|
||||
def empty_url_is_absent(cls, value):
|
||||
return None if isinstance(value, str) and not value.strip() else value
|
||||
|
||||
|
||||
class Patch(Input):
|
||||
expected_version: PositiveId
|
||||
|
||||
@model_validator(mode="after")
|
||||
def require_changes(self):
|
||||
if not self.model_fields_set - {"expected_version"}:
|
||||
raise ValueError("至少提供一个需要更新的字段")
|
||||
return self
|
||||
|
||||
@field_validator("*", mode="before")
|
||||
@classmethod
|
||||
def disallow_required_null(cls, value, info):
|
||||
if (
|
||||
info.field_name
|
||||
in {"name", "title", "category", "content", "notes", "collected_at", "expected_version"}
|
||||
and value is None
|
||||
):
|
||||
raise ValueError("该字段不能为 null")
|
||||
return value
|
||||
|
||||
|
||||
class Read(BaseModel):
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
id: int
|
||||
version: int
|
||||
created_at: datetime
|
||||
updated_at: datetime
|
||||
is_deleted: bool
|
||||
|
||||
|
||||
class SourceCreate(Input):
|
||||
name: Title
|
||||
source_url: HttpUrl | None = Field(default=None, max_length=1000)
|
||||
notes: Notes = ""
|
||||
|
||||
|
||||
class SourceUpdate(Patch):
|
||||
name: Title | None = None
|
||||
source_url: HttpUrl | None = Field(default=None, max_length=1000)
|
||||
notes: Notes | None = None
|
||||
|
||||
|
||||
class SourceRead(Read):
|
||||
name: str
|
||||
source_url: str | None
|
||||
notes: str
|
||||
|
||||
|
||||
class DocumentCreate(Input):
|
||||
source_id: PositiveId
|
||||
title: Title
|
||||
category: Category = "未分类"
|
||||
content: Body
|
||||
collected_at: AwareDatetime
|
||||
notes: Notes = ""
|
||||
|
||||
|
||||
class DocumentUpdate(Patch):
|
||||
title: Title | None = None
|
||||
category: Category | None = None
|
||||
content: Body | None = None
|
||||
collected_at: AwareDatetime | None = None
|
||||
notes: Notes | None = None
|
||||
|
||||
|
||||
class DocumentRead(Read):
|
||||
source_id: int
|
||||
title: str
|
||||
category: str
|
||||
content: str
|
||||
collected_at: datetime
|
||||
notes: str
|
||||
|
||||
|
||||
T = TypeVar("T")
|
||||
|
||||
|
||||
class Page(BaseModel, Generic[T]):
|
||||
items: list[T]
|
||||
total: int
|
||||
page: int
|
||||
page_size: int
|
||||
|
||||
|
||||
class AuditRead(BaseModel):
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
id: int
|
||||
entity_type: str
|
||||
entity_id: int
|
||||
action: str
|
||||
actor: str | None
|
||||
notes: str
|
||||
snapshot: dict
|
||||
created_at: datetime
|
||||
@@ -1,15 +1,12 @@
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class ChatRequest(BaseModel):
|
||||
question: str = Field(min_length=1, max_length=2000)
|
||||
|
||||
|
||||
class SourcePreview(BaseModel):
|
||||
title: str
|
||||
source_url: str | None = None
|
||||
|
||||
|
||||
class ChatResponse(BaseModel):
|
||||
answer: str
|
||||
status: str
|
||||
|
||||
@@ -1,89 +0,0 @@
|
||||
from datetime import datetime, timezone
|
||||
from typing import Literal
|
||||
from uuid import UUID
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
|
||||
|
||||
from app.schemas.catalog import Input
|
||||
|
||||
|
||||
class BuildIndex(Input):
|
||||
chunk_size: int = Field(500, ge=100, le=2000)
|
||||
overlap: int = Field(80, ge=0, le=1000)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def bounded_overlap(self):
|
||||
if self.overlap > self.chunk_size // 2:
|
||||
raise ValueError("重叠不得超过切分长度的一半")
|
||||
return self
|
||||
|
||||
|
||||
class PublishIndex(Input):
|
||||
expected_generation: int = Field(ge=0)
|
||||
|
||||
|
||||
class IndexConfig(BuildIndex):
|
||||
splitter: Literal["unicode-window-v1"]
|
||||
|
||||
|
||||
class ChunkProvenance(BaseModel):
|
||||
knowledge_id: int
|
||||
knowledge_version: int
|
||||
document_id: int | None
|
||||
document_version: int | None
|
||||
source_id: int | None
|
||||
source_version: int | None
|
||||
source_name: str | None
|
||||
source_url: str
|
||||
title: str
|
||||
category: str
|
||||
|
||||
|
||||
class TimestampRead(BaseModel):
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
@field_validator("created_at", "published_at", check_fields=False)
|
||||
@classmethod
|
||||
def utc_timestamp(cls, value):
|
||||
return value.astimezone(timezone.utc) if value is not None else None
|
||||
|
||||
|
||||
class IndexRead(TimestampRead):
|
||||
id: UUID
|
||||
status: Literal["ready", "published", "superseded"]
|
||||
config: IndexConfig
|
||||
corpus_fingerprint: str
|
||||
catalog_revision: int
|
||||
item_count: int
|
||||
chunk_count: int
|
||||
created_by: int
|
||||
created_at: datetime
|
||||
published_at: datetime | None
|
||||
|
||||
|
||||
class ActiveIndex(BaseModel):
|
||||
generation: int
|
||||
catalog_revision: int
|
||||
is_current: bool
|
||||
active_version: IndexRead | None
|
||||
|
||||
|
||||
class ChunkRead(BaseModel):
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
id: UUID
|
||||
index_version_id: UUID
|
||||
knowledge_id: int
|
||||
ordinal: int
|
||||
start_offset: int
|
||||
end_offset: int
|
||||
text: str
|
||||
text_sha256: str
|
||||
provenance: ChunkProvenance
|
||||
|
||||
|
||||
class PublicationRead(TimestampRead):
|
||||
generation: int
|
||||
previous_version_id: UUID | None
|
||||
version_id: UUID
|
||||
actor_id: int
|
||||
published_at: datetime
|
||||
@@ -1,46 +1,18 @@
|
||||
from datetime import datetime
|
||||
from pydantic import BaseModel, ConfigDict, Field, HttpUrl
|
||||
|
||||
from pydantic import AwareDatetime, Field, HttpUrl
|
||||
class KnowledgeCreate(BaseModel):
|
||||
title: str = Field(min_length=1, max_length=250)
|
||||
category: str = Field(default="未分类", max_length=80)
|
||||
content: str = Field(min_length=1)
|
||||
source_url: HttpUrl | None = None
|
||||
|
||||
from app.schemas.catalog import Body, Category, Input, Patch, PositiveId, Read, Status, Title
|
||||
|
||||
|
||||
class KnowledgeCreate(Input):
|
||||
title: Title
|
||||
category: Category = "未分类"
|
||||
content: Body
|
||||
source_url: HttpUrl | None = Field(default=None, max_length=1000)
|
||||
source_id: PositiveId | None = None
|
||||
document_id: PositiveId | None = None
|
||||
collected_at: AwareDatetime | None = None
|
||||
|
||||
|
||||
class KnowledgeUpdate(Patch):
|
||||
title: Title | None = None
|
||||
category: Category | None = None
|
||||
content: Body | None = None
|
||||
source_url: HttpUrl | None = Field(default=None, max_length=1000)
|
||||
source_id: PositiveId | None = None
|
||||
document_id: PositiveId | None = None
|
||||
collected_at: AwareDatetime | None = None
|
||||
|
||||
|
||||
class KnowledgeReview(Input):
|
||||
expected_version: PositiveId
|
||||
verification_status: Status
|
||||
reviewer: str = Field(min_length=1, max_length=100)
|
||||
verification_notes: str = Field(min_length=1, max_length=10_000)
|
||||
|
||||
|
||||
class KnowledgeRead(Read):
|
||||
class KnowledgeRead(BaseModel):
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
id: int
|
||||
title: str
|
||||
category: str
|
||||
content: str
|
||||
source_url: str | None
|
||||
verification_status: Status
|
||||
source_id: int | None
|
||||
document_id: int | None
|
||||
collected_at: datetime | None
|
||||
reviewer: str | None
|
||||
reviewed_at: datetime | None
|
||||
verification_notes: str | None
|
||||
verification_status: str
|
||||
created_at: datetime
|
||||
|
||||
@@ -1,59 +0,0 @@
|
||||
from datetime import datetime
|
||||
from typing import Literal
|
||||
from uuid import UUID
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from app.schemas.catalog import Category, Input
|
||||
|
||||
|
||||
class RetrievalRequest(Input):
|
||||
query: str = Field(min_length=1, max_length=500)
|
||||
mode: Literal["keyword"] = "keyword"
|
||||
top_k: int = Field(default=5, ge=1, le=20)
|
||||
category: Category | None = None
|
||||
|
||||
|
||||
class RetrievalHit(BaseModel):
|
||||
rank: int
|
||||
knowledge_id: int
|
||||
knowledge_version: int
|
||||
document_id: int | None
|
||||
document_version: int | None
|
||||
source_id: int | None
|
||||
source_version: int | None
|
||||
source_name: str | None
|
||||
source_url: str
|
||||
title: str
|
||||
category: str
|
||||
snippet: str
|
||||
snippet_start: int
|
||||
snippet_end: int
|
||||
score: float
|
||||
matched_terms: list[str]
|
||||
|
||||
|
||||
class RetrievalConfig(BaseModel):
|
||||
tokenizer: str
|
||||
scorer: str
|
||||
k1: float
|
||||
b: float
|
||||
title_weight: int
|
||||
stopwords_version: str
|
||||
|
||||
|
||||
class RetrievalResponse(BaseModel):
|
||||
run_id: UUID
|
||||
query: str
|
||||
mode: Literal["keyword"]
|
||||
status: Literal["ok", "no_evidence"]
|
||||
top_k: int
|
||||
category: str | None
|
||||
query_terms: list[str]
|
||||
corpus_size: int
|
||||
corpus_fingerprint: str
|
||||
total_hits: int
|
||||
results: list[RetrievalHit]
|
||||
elapsed_ms: float
|
||||
created_at: datetime
|
||||
config: RetrievalConfig
|
||||
@@ -1,161 +0,0 @@
|
||||
"""Database-backed sessions; no bearer secrets are stored in the database."""
|
||||
|
||||
import hashlib
|
||||
import hmac
|
||||
import re
|
||||
import secrets
|
||||
from dataclasses import dataclass
|
||||
from datetime import timedelta
|
||||
from functools import lru_cache
|
||||
|
||||
from argon2 import PasswordHasher
|
||||
from argon2.exceptions import InvalidHashError, VerificationError
|
||||
from fastapi import Depends, HTTPException, Request, Response
|
||||
from sqlalchemy import case, delete, select
|
||||
from sqlalchemy.dialects.postgresql import insert
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.core.config import settings
|
||||
from app.db.session import get_db
|
||||
from app.models.auth import AuthRateLimit, AuthSession, User
|
||||
from app.models.catalog import utc_now
|
||||
from app.schemas.auth import SessionRead, UserRead
|
||||
|
||||
COOKIE_NAME = "nalati_session"
|
||||
UNSAFE_METHODS = {"POST", "PUT", "PATCH", "DELETE"}
|
||||
PASSWORD_HASHER = PasswordHasher(time_cost=2, memory_cost=19456, parallelism=1)
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def dummy_hash():
|
||||
return PASSWORD_HASHER.hash(secrets.token_urlsafe(32))
|
||||
|
||||
|
||||
def check_password(encoded: str, password: str) -> bool:
|
||||
try:
|
||||
return PASSWORD_HASHER.verify(encoded, password)
|
||||
except (VerificationError, InvalidHashError):
|
||||
return False
|
||||
|
||||
|
||||
def token_hash(raw: str) -> str:
|
||||
return hashlib.sha256(raw.encode()).hexdigest()
|
||||
|
||||
|
||||
def csrf_token(raw: str) -> str:
|
||||
# Domain-separated derivative; knowing the CSRF token does not reveal the cookie.
|
||||
return hmac.new(raw.encode(), b"nalati-csrf-v1", hashlib.sha256).hexdigest()
|
||||
|
||||
|
||||
def require_origin(request: Request):
|
||||
if request.headers.get("origin") not in settings.auth_origin_list:
|
||||
raise HTTPException(403, "请求来源不受信任")
|
||||
|
||||
|
||||
def limit_auth_requests(db: Session, request: Request, username: str | None = None):
|
||||
"""Atomic, shared across workers and restarts; never trust forwarded client headers here."""
|
||||
now = utc_now()
|
||||
window = timedelta(minutes=5)
|
||||
buckets = [("ip:" + (request.client.host if request.client else "unknown"), 30)]
|
||||
if username is not None:
|
||||
buckets.append(("username:" + username, 10))
|
||||
blocked = False
|
||||
for identity, maximum in buckets:
|
||||
statement = insert(AuthRateLimit).values(
|
||||
key=token_hash(identity), window_start=now, attempts=1
|
||||
)
|
||||
expired = AuthRateLimit.window_start <= now - window
|
||||
statement = statement.on_conflict_do_update(
|
||||
index_elements=[AuthRateLimit.key],
|
||||
set_={
|
||||
"window_start": case((expired, now), else_=AuthRateLimit.window_start),
|
||||
"attempts": case((expired, 1), else_=AuthRateLimit.attempts + 1),
|
||||
},
|
||||
).returning(AuthRateLimit.attempts)
|
||||
blocked |= db.scalar(statement) > maximum
|
||||
db.execute(delete(AuthRateLimit).where(AuthRateLimit.window_start < now - timedelta(days=1)))
|
||||
# Persist even failed authentication; no password or raw address is logged.
|
||||
db.commit()
|
||||
if blocked:
|
||||
raise HTTPException(429, "认证请求过于频繁,请稍后重试", headers={"Retry-After": "300"})
|
||||
|
||||
|
||||
@dataclass
|
||||
class CurrentSession:
|
||||
user: User
|
||||
session: AuthSession
|
||||
raw_token: str
|
||||
|
||||
def response(self):
|
||||
return SessionRead(
|
||||
**UserRead.model_validate(self.user).model_dump(),
|
||||
csrf_token=csrf_token(self.raw_token),
|
||||
)
|
||||
|
||||
|
||||
def require_session(request: Request, db: Session = Depends(get_db)) -> CurrentSession:
|
||||
raw = request.cookies.get(COOKIE_NAME, "")
|
||||
if not re.fullmatch(r"[A-Za-z0-9_-]{43}", raw):
|
||||
raise HTTPException(401, "请先登录")
|
||||
session = db.get(AuthSession, token_hash(raw))
|
||||
if session is None:
|
||||
raise HTTPException(401, "登录已失效,请重新登录")
|
||||
# Lock the user before checking the current session again, serializing password changes/login.
|
||||
user = db.scalar(select(User).where(User.id == session.user_id).with_for_update())
|
||||
session = db.scalar(
|
||||
select(AuthSession)
|
||||
.where(AuthSession.token_hash == token_hash(raw))
|
||||
.execution_options(populate_existing=True)
|
||||
)
|
||||
if (
|
||||
user is None
|
||||
or session is None
|
||||
or session.expires_at <= utc_now()
|
||||
or session.credential_version != user.credential_version
|
||||
):
|
||||
raise HTTPException(401, "登录已失效,请重新登录")
|
||||
if request.method in UNSAFE_METHODS:
|
||||
require_origin(request)
|
||||
submitted = request.headers.get("x-csrf-token", "")
|
||||
if not hmac.compare_digest(submitted.encode(), csrf_token(raw).encode()):
|
||||
raise HTTPException(403, "CSRF 校验失败,请刷新登录状态后重试")
|
||||
return CurrentSession(user, session, raw)
|
||||
|
||||
|
||||
def require_admin(current: CurrentSession = Depends(require_session)):
|
||||
if current.user.role != "admin":
|
||||
raise HTTPException(403, "需要管理员权限")
|
||||
return current
|
||||
|
||||
|
||||
def issue_session(db: Session, user: User, request: Request, response: Response):
|
||||
now = utc_now()
|
||||
old = request.cookies.get(COOKIE_NAME, "")
|
||||
db.execute(delete(AuthSession).where(AuthSession.expires_at <= now))
|
||||
if old:
|
||||
db.execute(delete(AuthSession).where(AuthSession.token_hash == token_hash(old)))
|
||||
raw = secrets.token_urlsafe(32)
|
||||
session = AuthSession(
|
||||
token_hash=token_hash(raw),
|
||||
user_id=user.id,
|
||||
credential_version=user.credential_version,
|
||||
expires_at=now + timedelta(seconds=settings.auth_session_seconds),
|
||||
)
|
||||
db.add(session)
|
||||
db.commit()
|
||||
response.set_cookie(
|
||||
COOKIE_NAME,
|
||||
raw,
|
||||
max_age=settings.auth_session_seconds,
|
||||
httponly=True,
|
||||
secure=settings.auth_cookie_secure,
|
||||
samesite="lax",
|
||||
path="/",
|
||||
)
|
||||
return CurrentSession(user, session, raw).response()
|
||||
|
||||
|
||||
def clear_cookie(response: Response):
|
||||
response.delete_cookie(
|
||||
COOKIE_NAME, path="/", httponly=True, secure=settings.auth_cookie_secure, samesite="lax"
|
||||
)
|
||||
@@ -1,120 +0,0 @@
|
||||
from fastapi import HTTPException
|
||||
from fastapi.encoders import jsonable_encoder
|
||||
from sqlalchemy import func, inspect, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.models import AuditEvent, Document, KnowledgeItem, Source
|
||||
from app.models.catalog import utc_now
|
||||
|
||||
|
||||
def get_item(db: Session, model, item_id: int, *, include_deleted=False, lock=False):
|
||||
if lock:
|
||||
item = db.scalar(
|
||||
select(model)
|
||||
.where(model.id == item_id)
|
||||
.with_for_update()
|
||||
.execution_options(populate_existing=True)
|
||||
)
|
||||
else:
|
||||
item = db.get(model, item_id)
|
||||
if item is None or (item.is_deleted and not include_deleted):
|
||||
raise HTTPException(404, "记录不存在")
|
||||
return item
|
||||
|
||||
|
||||
def ensure_version(item, expected_version: int):
|
||||
if item.version != expected_version:
|
||||
raise HTTPException(409, "记录已更新,请刷新后重试")
|
||||
|
||||
|
||||
def values(item) -> dict:
|
||||
return jsonable_encoder(
|
||||
{column.key: getattr(item, column.key) for column in inspect(type(item)).columns}
|
||||
)
|
||||
|
||||
|
||||
def record_event(db: Session, item, action: str, *, actor=None, notes=""):
|
||||
db.flush()
|
||||
snapshot = {"item": values(item)}
|
||||
if isinstance(item, KnowledgeItem):
|
||||
source = db.get(Source, item.source_id) if item.source_id else None
|
||||
document = db.get(Document, item.document_id) if item.document_id else None
|
||||
snapshot["source"] = values(source) if source else None
|
||||
snapshot["document"] = values(document) if document else None
|
||||
db.add(
|
||||
AuditEvent(
|
||||
entity_type=item.__tablename__,
|
||||
entity_id=item.id,
|
||||
action=action,
|
||||
actor=actor,
|
||||
notes=notes,
|
||||
snapshot=snapshot,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def update_values(item, payload):
|
||||
data = payload.model_dump(mode="json", exclude_unset=True, exclude={"expected_version"})
|
||||
data.update(
|
||||
{
|
||||
key: value
|
||||
for key, value in payload.model_dump(exclude_unset=True).items()
|
||||
if key.endswith("_at")
|
||||
}
|
||||
)
|
||||
for name, value in data.items():
|
||||
setattr(item, name, value)
|
||||
item.updated_at = utc_now()
|
||||
|
||||
|
||||
def reset_review(item: KnowledgeItem):
|
||||
item.verification_status = "pending"
|
||||
item.reviewer = None
|
||||
item.reviewed_at = None
|
||||
item.verification_notes = None
|
||||
item.updated_at = utc_now()
|
||||
|
||||
|
||||
def invalidate_knowledge(db: Session, condition, *, source_url=None, update_url=False):
|
||||
for item in db.scalars(
|
||||
select(KnowledgeItem).where(condition, KnowledgeItem.is_deleted.is_(False))
|
||||
):
|
||||
reset_review(item)
|
||||
if update_url:
|
||||
item.source_url = source_url
|
||||
record_event(db, item, "invalidate", notes="来源或文档已变更,需要重新核验")
|
||||
|
||||
|
||||
def resolve_knowledge_links(db: Session, item: KnowledgeItem):
|
||||
# 先锁依据记录再保存条目,避免引用与父记录修改/删除同时发生。
|
||||
with db.no_autoflush:
|
||||
if item.document_id is not None:
|
||||
document = get_item(db, Document, item.document_id, lock=True)
|
||||
if item.source_id is not None and item.source_id != document.source_id:
|
||||
raise HTTPException(422, "文档与来源不一致")
|
||||
item.source_id = document.source_id
|
||||
if item.source_id is not None:
|
||||
source = get_item(db, Source, item.source_id, lock=True)
|
||||
if item.source_url and item.source_url != source.source_url:
|
||||
raise HTTPException(422, "来源网址与登记来源不一致")
|
||||
item.source_url = source.source_url
|
||||
|
||||
|
||||
def page_result(db: Session, statement, page: int, page_size: int):
|
||||
total = db.scalar(select(func.count()).select_from(statement.order_by(None).subquery()))
|
||||
return {
|
||||
"items": list(db.scalars(statement.offset((page - 1) * page_size).limit(page_size))),
|
||||
"total": total,
|
||||
"page": page,
|
||||
"page_size": page_size,
|
||||
}
|
||||
|
||||
|
||||
def history(db: Session, model, item_id: int, page: int, page_size: int):
|
||||
get_item(db, model, item_id, include_deleted=True)
|
||||
statement = (
|
||||
select(AuditEvent)
|
||||
.where(AuditEvent.entity_type == model.__tablename__, AuditEvent.entity_id == item_id)
|
||||
.order_by(AuditEvent.id.desc())
|
||||
)
|
||||
return page_result(db, statement, page, page_size)
|
||||
@@ -1,165 +0,0 @@
|
||||
"""Optional local embeddings; importing this module never loads model weights or PyTorch."""
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import re
|
||||
from dataclasses import asdict, dataclass
|
||||
from typing import Protocol
|
||||
|
||||
QUERY_INSTRUCTION = "为这个句子生成表示以用于检索相关文章:"
|
||||
|
||||
|
||||
class EmbeddingError(ValueError):
|
||||
pass
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class EmbeddingSpec:
|
||||
model: str
|
||||
revision: str
|
||||
dimension: int
|
||||
token_limit: int = 512
|
||||
query_instruction: str = QUERY_INSTRUCTION
|
||||
implementation: str = "hf-cls-l2-cpu-fp32-v1"
|
||||
checkpoint_format: str = "safetensors"
|
||||
|
||||
def __post_init__(self):
|
||||
if not isinstance(self.revision, str) or not re.fullmatch(r"[0-9a-f]{40}", self.revision):
|
||||
raise EmbeddingError("模型必须固定到完整 revision SHA")
|
||||
if (
|
||||
type(self.dimension) is not int
|
||||
or type(self.token_limit) is not int
|
||||
or self.dimension < 1
|
||||
or self.token_limit < 2
|
||||
):
|
||||
raise EmbeddingError("向量维度和 token 上限无效")
|
||||
if (
|
||||
not isinstance(self.model, str)
|
||||
or not self.model
|
||||
or not isinstance(self.query_instruction, str)
|
||||
):
|
||||
raise EmbeddingError("模型名称或查询指令无效")
|
||||
if self.checkpoint_format not in {"safetensors", "pytorch-weights-only"}:
|
||||
raise EmbeddingError("不支持的权重格式")
|
||||
|
||||
@property
|
||||
def space_id(self) -> str:
|
||||
return hashlib.sha256(
|
||||
json.dumps(asdict(self), sort_keys=True, ensure_ascii=False).encode()
|
||||
).hexdigest()
|
||||
|
||||
|
||||
SMALL_ZH = EmbeddingSpec("BAAI/bge-small-zh-v1.5", "7999e1d3359715c523056ef9478215996d62a620", 512)
|
||||
BASE_ZH = EmbeddingSpec(
|
||||
"BAAI/bge-base-zh-v1.5",
|
||||
"f03589ceff5aac7111bd60cfc7d497ca17ecac65",
|
||||
768,
|
||||
checkpoint_format="pytorch-weights-only",
|
||||
)
|
||||
LOCAL_MODELS = {"small": SMALL_ZH, "base": BASE_ZH}
|
||||
|
||||
|
||||
class EmbeddingProvider(Protocol):
|
||||
spec: EmbeddingSpec
|
||||
|
||||
def encode(self, texts: list[str], *, query: bool = False) -> list[list[float]]: ...
|
||||
|
||||
|
||||
def validate_vectors(vectors: list[list[float]], count: int, spec: EmbeddingSpec):
|
||||
if len(vectors) != count:
|
||||
raise EmbeddingError("模型返回数量与输入不一致")
|
||||
for vector in vectors:
|
||||
try:
|
||||
finite = all(type(x) in {float, int} and math.isfinite(x) for x in vector)
|
||||
except (TypeError, OverflowError):
|
||||
finite = False
|
||||
if len(vector) != spec.dimension or not finite:
|
||||
raise EmbeddingError("模型返回维度错误或非有限数值")
|
||||
if not math.isclose(sum(x * x for x in vector), 1, abs_tol=1e-4):
|
||||
raise EmbeddingError("模型输出没有正确 L2 归一化")
|
||||
|
||||
|
||||
class LocalEmbeddingProvider:
|
||||
def __init__(
|
||||
self,
|
||||
spec: EmbeddingSpec,
|
||||
*,
|
||||
batch_size: int = 8,
|
||||
threads: int = 2,
|
||||
cache_dir=None,
|
||||
offline: bool = True,
|
||||
):
|
||||
if (
|
||||
type(batch_size) is not int
|
||||
or type(threads) is not int
|
||||
or not 1 <= batch_size <= 32
|
||||
or not 1 <= threads <= 4
|
||||
):
|
||||
raise EmbeddingError("CPU batch_size 必须为 1–32,threads 必须为 1–4")
|
||||
import torch
|
||||
from huggingface_hub import snapshot_download
|
||||
from transformers import AutoModel, AutoTokenizer
|
||||
|
||||
torch.set_num_threads(threads)
|
||||
self.spec, self.batch_size, self.torch = spec, batch_size, torch
|
||||
path = snapshot_download(
|
||||
repo_id=spec.model,
|
||||
revision=spec.revision,
|
||||
cache_dir=cache_dir,
|
||||
local_files_only=offline,
|
||||
token=False,
|
||||
allow_patterns=[
|
||||
"config.json",
|
||||
"model.safetensors"
|
||||
if spec.checkpoint_format == "safetensors"
|
||||
else "pytorch_model.bin",
|
||||
"tokenizer.json",
|
||||
"tokenizer_config.json",
|
||||
"special_tokens_map.json",
|
||||
"vocab.txt",
|
||||
"added_tokens.json",
|
||||
],
|
||||
)
|
||||
self.tokenizer = AutoTokenizer.from_pretrained(
|
||||
path, local_files_only=True, trust_remote_code=False
|
||||
)
|
||||
self.model = (
|
||||
AutoModel.from_pretrained(
|
||||
path,
|
||||
local_files_only=True,
|
||||
trust_remote_code=False,
|
||||
use_safetensors=spec.checkpoint_format == "safetensors",
|
||||
weights_only=True,
|
||||
dtype=torch.float32,
|
||||
)
|
||||
.to("cpu")
|
||||
.eval()
|
||||
)
|
||||
if self.model.config.hidden_size != spec.dimension:
|
||||
raise EmbeddingError("模型实际维度与配置不一致")
|
||||
if self.tokenizer.model_max_length != spec.token_limit:
|
||||
raise EmbeddingError("模型 tokenizer 上限与配置不一致")
|
||||
|
||||
def encode(self, texts: list[str], *, query: bool = False) -> list[list[float]]:
|
||||
if not isinstance(texts, list) or any(
|
||||
not isinstance(text, str) or not text.strip() or len(text) > 10_000 for text in texts
|
||||
):
|
||||
raise EmbeddingError("每条输入须为非空文本且最多 10000 字符")
|
||||
torch = self.torch
|
||||
vectors = []
|
||||
for offset in range(0, len(texts), self.batch_size):
|
||||
batch = texts[offset : offset + self.batch_size]
|
||||
if query:
|
||||
batch = [self.spec.query_instruction + text for text in batch]
|
||||
encoded = self.tokenizer(
|
||||
batch, padding=True, truncation=False, return_tensors="pt", verbose=False
|
||||
)
|
||||
if encoded["input_ids"].shape[1] > self.spec.token_limit:
|
||||
raise EmbeddingError("输入超出模型 token 上限;请重新切分,禁止静默截断")
|
||||
with torch.inference_mode():
|
||||
features = self.model(**encoded).last_hidden_state[:, 0]
|
||||
normalized = torch.nn.functional.normalize(features, p=2, dim=1)
|
||||
vectors.extend(normalized.tolist())
|
||||
validate_vectors(vectors, len(texts), self.spec)
|
||||
return vectors
|
||||
@@ -1,147 +0,0 @@
|
||||
"""Bounded chunk snapshots and serialized, optimistic publication."""
|
||||
|
||||
import hashlib
|
||||
from uuid import UUID, uuid4, uuid5
|
||||
|
||||
from fastapi import Depends, HTTPException, Request
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.db.session import get_db
|
||||
from app.models.catalog import utc_now
|
||||
from app.models.indexing import Chunk, IndexPublication, IndexState, IndexVersion
|
||||
from app.schemas.indexing import ActiveIndex, BuildIndex, IndexRead
|
||||
from app.services.retrieval import corpus_fingerprint, load_corpus
|
||||
|
||||
MAX_CHUNKS = 20_000
|
||||
|
||||
|
||||
def locked_state(db: Session) -> IndexState:
|
||||
state = db.scalar(
|
||||
select(IndexState)
|
||||
.where(IndexState.id == 1)
|
||||
.with_for_update()
|
||||
.execution_options(populate_existing=True)
|
||||
)
|
||||
if state is None:
|
||||
raise HTTPException(503, "索引状态尚未初始化,请检查数据库迁移")
|
||||
return state
|
||||
|
||||
|
||||
def catalog_write_guard(request: Request, db: Session = Depends(get_db)):
|
||||
if request.method in {"POST", "PATCH", "PUT", "DELETE"}:
|
||||
# Always before catalog/parent row locks; failures roll back this revision as well.
|
||||
state = locked_state(db)
|
||||
state.catalog_revision += 1
|
||||
|
||||
|
||||
def version_or_404(db: Session, version_id: str) -> IndexVersion:
|
||||
version = db.get(IndexVersion, version_id)
|
||||
if version is None:
|
||||
raise HTTPException(404, "索引版本不存在")
|
||||
return version
|
||||
|
||||
|
||||
def split_text(content: str, size: int, overlap: int):
|
||||
"""Offsets are original Python Unicode character offsets, end-exclusive."""
|
||||
for start in range(0, len(content), size - overlap):
|
||||
end = min(len(content), start + size)
|
||||
yield start, end, content[start:end]
|
||||
if end == len(content):
|
||||
break
|
||||
|
||||
|
||||
def build_index(db: Session, payload: BuildIndex, actor_id: int) -> IndexRead:
|
||||
state = locked_state(db)
|
||||
corpus = load_corpus(db, None)
|
||||
version_id = uuid4()
|
||||
chunks = []
|
||||
for item in corpus:
|
||||
provenance = {key: value for key, value in item.items() if key != "content"}
|
||||
for ordinal, (start, end, content) in enumerate(
|
||||
split_text(item["content"], payload.chunk_size, payload.overlap)
|
||||
):
|
||||
if len(chunks) >= MAX_CHUNKS:
|
||||
raise HTTPException(503, "切分数量超过本阶段容量,请增加切分长度或减少重叠")
|
||||
chunks.append(
|
||||
Chunk(
|
||||
id=str(uuid5(version_id, f"{item['knowledge_id']}:{ordinal}")),
|
||||
index_version_id=str(version_id),
|
||||
knowledge_id=item["knowledge_id"],
|
||||
ordinal=ordinal,
|
||||
start_offset=start,
|
||||
end_offset=end,
|
||||
text=content,
|
||||
text_sha256=hashlib.sha256(content.encode()).hexdigest(),
|
||||
provenance=provenance,
|
||||
)
|
||||
)
|
||||
version = IndexVersion(
|
||||
id=str(version_id),
|
||||
status="ready",
|
||||
config={"splitter": "unicode-window-v1", **payload.model_dump()},
|
||||
corpus_fingerprint=corpus_fingerprint(corpus),
|
||||
catalog_revision=state.catalog_revision,
|
||||
item_count=len(corpus),
|
||||
chunk_count=len(chunks),
|
||||
created_by=actor_id,
|
||||
)
|
||||
db.add(version)
|
||||
db.flush()
|
||||
db.add_all(chunks)
|
||||
result = IndexRead.model_validate(version)
|
||||
db.commit()
|
||||
return result
|
||||
|
||||
|
||||
def active_index(db: Session, state: IndexState | None = None) -> ActiveIndex:
|
||||
state = state or locked_state(db)
|
||||
version = db.get(IndexVersion, state.active_version_id) if state.active_version_id else None
|
||||
current = bool(version and version.catalog_revision == state.catalog_revision)
|
||||
if current:
|
||||
current = version.corpus_fingerprint == corpus_fingerprint(load_corpus(db, None))
|
||||
return ActiveIndex(
|
||||
generation=state.generation,
|
||||
catalog_revision=state.catalog_revision,
|
||||
is_current=current,
|
||||
active_version=IndexRead.model_validate(version) if version else None,
|
||||
)
|
||||
|
||||
|
||||
def publish_index(db: Session, version_id: UUID, expected_generation: int, actor_id: int):
|
||||
state = locked_state(db)
|
||||
if state.generation != expected_generation:
|
||||
raise HTTPException(409, "已发布版本发生变化,请刷新后重试")
|
||||
version = version_or_404(db, str(version_id))
|
||||
if (
|
||||
version.catalog_revision != state.catalog_revision
|
||||
or version.corpus_fingerprint != corpus_fingerprint(load_corpus(db, None))
|
||||
):
|
||||
raise HTTPException(409, "资料或审核依据已变化,请重新构建后发布")
|
||||
if state.active_version_id == version.id:
|
||||
return active_index(db, state)
|
||||
previous = state.active_version_id
|
||||
if previous:
|
||||
db.get(IndexVersion, previous).status = "superseded"
|
||||
state.active_version_id = version.id
|
||||
state.generation += 1
|
||||
version.status = "published"
|
||||
version.published_at = utc_now()
|
||||
db.add(
|
||||
IndexPublication(
|
||||
generation=state.generation,
|
||||
previous_version_id=previous,
|
||||
version_id=version.id,
|
||||
actor_id=actor_id,
|
||||
published_at=version.published_at,
|
||||
)
|
||||
)
|
||||
# Freeze the response before commit expires ORM objects and releases the lock.
|
||||
result = ActiveIndex(
|
||||
generation=expected_generation + 1,
|
||||
catalog_revision=version.catalog_revision,
|
||||
is_current=True,
|
||||
active_version=IndexRead.model_validate(version),
|
||||
)
|
||||
db.commit()
|
||||
return result
|
||||
@@ -1,219 +0,0 @@
|
||||
"""Deterministic lexical baseline over a bounded, reviewed knowledge corpus."""
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
import math
|
||||
import unicodedata
|
||||
from collections import Counter
|
||||
from functools import lru_cache
|
||||
from time import perf_counter
|
||||
from uuid import uuid4
|
||||
|
||||
import jieba
|
||||
from fastapi import HTTPException
|
||||
from sqlalchemy import func, or_, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.models import Document, KnowledgeItem, RetrievalRun, Source
|
||||
from app.models.catalog import utc_now
|
||||
from app.schemas.retrieval import RetrievalConfig, RetrievalHit, RetrievalRequest, RetrievalResponse
|
||||
|
||||
MAX_ITEMS = 5000
|
||||
MAX_CHARACTERS = 2_000_000
|
||||
SNIPPET_LENGTH = 240
|
||||
STOPWORDS = frozenset("的 了 和 与 是 在 有 哪些 什么 请问 如何 吗 呢 我 你 想 了解".split())
|
||||
CONFIG = RetrievalConfig(
|
||||
tokenizer="jieba-0.42.1-search-hmm-off-nalati-v1",
|
||||
scorer="bm25-weighted-text-v1",
|
||||
k1=1.2,
|
||||
b=0.75,
|
||||
title_weight=2,
|
||||
stopwords_version="zh-small-v1",
|
||||
)
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def tokenizer():
|
||||
jieba.setLogLevel(logging.WARNING)
|
||||
instance = jieba.Tokenizer()
|
||||
instance.initialize()
|
||||
instance.add_word("那拉提", freq=100000)
|
||||
return instance
|
||||
|
||||
|
||||
def normalized_with_offsets(text: str) -> tuple[str, list[int]]:
|
||||
pieces, offsets = [], []
|
||||
for position, character in enumerate(text):
|
||||
piece = unicodedata.normalize("NFKC", character).casefold()
|
||||
pieces.append(piece)
|
||||
offsets.extend([position] * len(piece))
|
||||
return "".join(pieces), offsets
|
||||
|
||||
|
||||
def tokenize(text: str) -> list[str]:
|
||||
normalized, _ = normalized_with_offsets(text)
|
||||
return [
|
||||
term
|
||||
for term in tokenizer().cut_for_search(normalized, HMM=False)
|
||||
if term not in STOPWORDS and any(character.isalnum() for character in term)
|
||||
]
|
||||
|
||||
|
||||
def bm25(counters: list[Counter], query_terms: list[str]) -> list[float]:
|
||||
"""Positive log-IDF BM25; repeated query terms count once."""
|
||||
count = len(counters)
|
||||
lengths = [sum(counter.values()) for counter in counters]
|
||||
average = sum(lengths) / count if count else 0
|
||||
if not average:
|
||||
return [0.0] * count
|
||||
scores = [0.0] * count
|
||||
for term in dict.fromkeys(query_terms):
|
||||
frequency = sum(counter[term] > 0 for counter in counters)
|
||||
if not frequency:
|
||||
continue
|
||||
idf = math.log(1 + (count - frequency + 0.5) / (frequency + 0.5))
|
||||
for index, counter in enumerate(counters):
|
||||
tf = counter[term]
|
||||
if tf:
|
||||
denominator = tf + CONFIG.k1 * (1 - CONFIG.b + CONFIG.b * lengths[index] / average)
|
||||
scores[index] += idf * tf * (CONFIG.k1 + 1) / denominator
|
||||
return scores
|
||||
|
||||
|
||||
def excerpt(content: str, matched_terms: list[str]) -> tuple[str, int, int]:
|
||||
normalized, offsets = normalized_with_offsets(content)
|
||||
positions = [normalized.find(term) for term in matched_terms if term in normalized]
|
||||
start = max(0, offsets[min(positions)] - 60) if positions else 0
|
||||
end = min(len(content), start + SNIPPET_LENGTH)
|
||||
return content[start:end], start, end
|
||||
|
||||
|
||||
def load_corpus(db: Session, category: str | None) -> list[dict]:
|
||||
statement = (
|
||||
select(
|
||||
KnowledgeItem,
|
||||
Source.name.label("source_name"),
|
||||
Source.version.label("source_version"),
|
||||
Document.version.label("document_version"),
|
||||
func.count().over().label("corpus_total"),
|
||||
func.sum(func.length(KnowledgeItem.title) + func.length(KnowledgeItem.content))
|
||||
.over()
|
||||
.label("corpus_characters"),
|
||||
)
|
||||
.outerjoin(Source, KnowledgeItem.source_id == Source.id)
|
||||
.outerjoin(Document, KnowledgeItem.document_id == Document.id)
|
||||
.where(
|
||||
KnowledgeItem.verification_status == "verified",
|
||||
KnowledgeItem.is_deleted.is_(False),
|
||||
KnowledgeItem.source_url.is_not(None),
|
||||
func.length(func.trim(KnowledgeItem.source_url)) > 0,
|
||||
KnowledgeItem.collected_at.is_not(None),
|
||||
KnowledgeItem.reviewed_at.is_not(None),
|
||||
KnowledgeItem.reviewer.is_not(None),
|
||||
KnowledgeItem.verification_notes.is_not(None),
|
||||
or_(KnowledgeItem.source_id.is_(None), Source.is_deleted.is_(False)),
|
||||
or_(
|
||||
KnowledgeItem.document_id.is_(None),
|
||||
(Document.is_deleted.is_(False)) & (Document.source_id == KnowledgeItem.source_id),
|
||||
),
|
||||
or_(KnowledgeItem.source_id.is_(None), KnowledgeItem.source_url == Source.source_url),
|
||||
)
|
||||
.order_by(KnowledgeItem.id)
|
||||
.limit(MAX_ITEMS + 1)
|
||||
)
|
||||
if category is not None:
|
||||
statement = statement.where(KnowledgeItem.category == category)
|
||||
corpus = []
|
||||
# 一条 SQL 的快照同时给出语料与容量;流式读取避免先载入超限正文。
|
||||
with db.execute(statement.execution_options(stream_results=True, max_row_buffer=1)) as rows:
|
||||
for row in rows:
|
||||
if row.corpus_total > MAX_ITEMS or row.corpus_characters > MAX_CHARACTERS:
|
||||
raise HTTPException(
|
||||
503, "已核验语料超过关键词基线容量,请缩小分类范围或建立检索索引"
|
||||
)
|
||||
item = row.KnowledgeItem
|
||||
corpus.append(
|
||||
{
|
||||
"knowledge_id": item.id,
|
||||
"knowledge_version": item.version,
|
||||
"document_id": item.document_id,
|
||||
"document_version": row.document_version,
|
||||
"source_id": item.source_id,
|
||||
"source_version": row.source_version,
|
||||
"source_name": row.source_name,
|
||||
"source_url": item.source_url,
|
||||
"title": item.title,
|
||||
"category": item.category,
|
||||
"content": item.content,
|
||||
}
|
||||
)
|
||||
return corpus
|
||||
|
||||
|
||||
def corpus_fingerprint(corpus: list[dict]) -> str:
|
||||
return hashlib.sha256(
|
||||
json.dumps(corpus, ensure_ascii=False, sort_keys=True, separators=(",", ":")).encode()
|
||||
).hexdigest()
|
||||
|
||||
|
||||
def retrieve(db: Session, payload: RetrievalRequest) -> RetrievalResponse:
|
||||
started = perf_counter()
|
||||
terms = list(dict.fromkeys(tokenize(payload.query)))
|
||||
if not terms:
|
||||
raise HTTPException(422, "问题中没有可检索词,请补充具体关键词")
|
||||
corpus = load_corpus(db, payload.category)
|
||||
fingerprint = corpus_fingerprint(corpus)
|
||||
counters = [
|
||||
Counter(tokenize(item["title"]) * CONFIG.title_weight + tokenize(item["content"]))
|
||||
for item in corpus
|
||||
]
|
||||
scores = bm25(counters, terms)
|
||||
ranked = sorted(
|
||||
(index for index, score in enumerate(scores) if score > 0),
|
||||
key=lambda index: (-scores[index], corpus[index]["knowledge_id"]),
|
||||
)
|
||||
results = []
|
||||
for rank, index in enumerate(ranked[: payload.top_k], start=1):
|
||||
item = corpus[index]
|
||||
matched = [term for term in terms if counters[index][term]]
|
||||
snippet, start, end = excerpt(item["content"], matched)
|
||||
results.append(
|
||||
RetrievalHit(
|
||||
**{key: value for key, value in item.items() if key != "content"},
|
||||
rank=rank,
|
||||
score=round(scores[index], 8),
|
||||
matched_terms=matched,
|
||||
snippet=snippet,
|
||||
snippet_start=start,
|
||||
snippet_end=end,
|
||||
)
|
||||
)
|
||||
response = RetrievalResponse(
|
||||
run_id=uuid4(),
|
||||
query=payload.query,
|
||||
mode="keyword",
|
||||
status="ok" if results else "no_evidence",
|
||||
top_k=payload.top_k,
|
||||
category=payload.category,
|
||||
query_terms=terms,
|
||||
corpus_size=len(corpus),
|
||||
corpus_fingerprint=fingerprint,
|
||||
total_hits=len(ranked),
|
||||
results=results,
|
||||
elapsed_ms=round((perf_counter() - started) * 1000, 3),
|
||||
created_at=utc_now(),
|
||||
config=CONFIG,
|
||||
)
|
||||
db.add(
|
||||
RetrievalRun(
|
||||
id=str(response.run_id),
|
||||
query=response.query,
|
||||
mode=response.mode,
|
||||
corpus_fingerprint=fingerprint,
|
||||
response=response.model_dump(mode="json"),
|
||||
created_at=response.created_at,
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
return response
|
||||
@@ -1 +0,0 @@
|
||||
"""Offline, synthetic local-model evaluation; never writes the application database."""
|
||||
@@ -1,221 +0,0 @@
|
||||
"""Reproducible CPU smoke evaluation; no application database or remote inference service."""
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import importlib.metadata
|
||||
import json
|
||||
import math
|
||||
import platform
|
||||
import statistics
|
||||
import sys
|
||||
from dataclasses import asdict
|
||||
from pathlib import Path
|
||||
from time import perf_counter
|
||||
|
||||
from app.services.embedding import LOCAL_MODELS, EmbeddingError, LocalEmbeddingProvider
|
||||
|
||||
|
||||
def load_dataset(path: Path) -> tuple[dict, str]:
|
||||
with path.open("rb") as source:
|
||||
raw = source.read(2_000_001)
|
||||
if len(raw) > 2_000_000:
|
||||
raise ValueError("评估输入不得超过 2MB")
|
||||
data = json.loads(raw)
|
||||
if not isinstance(data, dict):
|
||||
raise ValueError("评估输入须为 JSON 对象")
|
||||
documents, queries = data["documents"], data["queries"]
|
||||
if not isinstance(documents, list) or not isinstance(queries, list):
|
||||
raise ValueError("documents 与 queries 须为列表")
|
||||
if not 1 <= len(documents) <= 1000 or not 1 <= len(queries) <= 1000:
|
||||
raise ValueError("文档和问题各须为 1–1000 条")
|
||||
for rows in [documents, queries]:
|
||||
if any(not isinstance(row, dict) for row in rows):
|
||||
raise ValueError("每条记录须为对象")
|
||||
ids = [row["id"] for row in rows]
|
||||
if any(not isinstance(x, str) or not x or len(x) > 100 for x in ids) or len(
|
||||
set(ids)
|
||||
) != len(ids):
|
||||
raise ValueError("每组 ID 必须非空、唯一且最多 100 字符")
|
||||
if any(
|
||||
not isinstance(row["text"], str) or not row["text"].strip() or len(row["text"]) > 10000
|
||||
for row in rows
|
||||
):
|
||||
raise ValueError("每条文本须为非空且最多 10000 字符")
|
||||
ids = {row["id"] for row in documents}
|
||||
for query in queries:
|
||||
relevant = query["relevant_ids"]
|
||||
if (
|
||||
not isinstance(relevant, list)
|
||||
or not relevant
|
||||
or any(not isinstance(x, str) for x in relevant)
|
||||
or len(set(relevant)) != len(relevant)
|
||||
or not set(relevant) <= ids
|
||||
):
|
||||
raise ValueError("相关文档标注必须非空、唯一并引用现有文档")
|
||||
if not isinstance(data["dataset_id"], str) or not isinstance(data["purpose"], str):
|
||||
raise ValueError("必须填写数据集 ID 与用途")
|
||||
return data, hashlib.sha256(raw).hexdigest()
|
||||
|
||||
|
||||
def rank_and_measure(documents, queries, document_vectors, query_vectors, *, top_k=3):
|
||||
if not documents or not queries or top_k < 1:
|
||||
raise ValueError("评估须有文档、问题且 top_k 为正数")
|
||||
if len(document_vectors) != len(documents) or len(query_vectors) != len(queries):
|
||||
raise ValueError("评估向量数量与输入不一致")
|
||||
rows = []
|
||||
for query, vector in zip(queries, query_vectors, strict=True):
|
||||
ranked = sorted(
|
||||
[
|
||||
(doc["id"], math.fsum(x * y for x, y in zip(vector, embedding, strict=True)))
|
||||
for doc, embedding in zip(documents, document_vectors, strict=True)
|
||||
],
|
||||
key=lambda pair: (-pair[1], pair[0]),
|
||||
)
|
||||
relevant = set(query["relevant_ids"])
|
||||
first = next(
|
||||
(rank for rank, (id_, _) in enumerate(ranked[:top_k], 1) if id_ in relevant), None
|
||||
)
|
||||
rows.append(
|
||||
{
|
||||
"query_id": query["id"],
|
||||
"top_results": [
|
||||
{"id": id_, "score": round(score, 8)} for id_, score in ranked[:top_k]
|
||||
],
|
||||
"hit_at_1": int(ranked[0][0] in relevant),
|
||||
"recall_at_k": len({id_ for id_, _ in ranked[:top_k]} & relevant) / len(relevant),
|
||||
"reciprocal_rank_at_k": 1 / first if first else 0,
|
||||
}
|
||||
)
|
||||
return {
|
||||
"top_k": top_k,
|
||||
"hit_at_1": statistics.mean(row["hit_at_1"] for row in rows),
|
||||
"recall_at_k": statistics.mean(row["recall_at_k"] for row in rows),
|
||||
"mrr_at_k": statistics.mean(row["reciprocal_rank_at_k"] for row in rows),
|
||||
"per_query": rows,
|
||||
}
|
||||
|
||||
|
||||
def measure(call, repeats):
|
||||
call() # Warm this exact input shape before taking the recorded samples.
|
||||
timings = []
|
||||
result = None
|
||||
for _ in range(repeats):
|
||||
start = perf_counter()
|
||||
result = call()
|
||||
timings.append((perf_counter() - start) * 1000)
|
||||
return result, {
|
||||
"samples_ms": [round(value, 3) for value in timings],
|
||||
"median_ms": round(statistics.median(timings), 3),
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="本地 CPU Embedding 实测;合成结果不能用于论文结论"
|
||||
)
|
||||
parser.add_argument("--model", choices=LOCAL_MODELS, required=True)
|
||||
parser.add_argument(
|
||||
"--dataset", type=Path, default=Path(__file__).with_name("synthetic_zh.json")
|
||||
)
|
||||
parser.add_argument("--batch-size", type=int, default=8)
|
||||
parser.add_argument("--threads", type=int, default=2)
|
||||
parser.add_argument("--repeats", type=int, default=5)
|
||||
parser.add_argument("--cache-dir", type=Path)
|
||||
parser.add_argument(
|
||||
"--download", action="store_true", help="允许下载固定 revision 的公开模型文件"
|
||||
)
|
||||
parser.add_argument("--output", type=Path)
|
||||
args = parser.parse_args()
|
||||
if sys.platform != "linux":
|
||||
parser.error("当前 RSS 测量仅支持 Linux;Provider 可在其他支持的平台单独使用")
|
||||
import resource
|
||||
|
||||
if not 1 <= args.repeats <= 20:
|
||||
parser.error("repeats 必须为 1–20")
|
||||
dataset, digest = load_dataset(args.dataset)
|
||||
print("初始化固定版本的 CPU 模型。", file=sys.stderr, flush=True)
|
||||
started = perf_counter()
|
||||
provider = LocalEmbeddingProvider(
|
||||
LOCAL_MODELS[args.model],
|
||||
batch_size=args.batch_size,
|
||||
threads=args.threads,
|
||||
cache_dir=args.cache_dir,
|
||||
offline=not args.download,
|
||||
)
|
||||
initialization_seconds = perf_counter() - started
|
||||
print("模型就绪,执行预热和编码测量。", file=sys.stderr, flush=True)
|
||||
provider.encode(["预热测试"], query=True)
|
||||
corpus = [row["text"] for row in dataset["documents"]]
|
||||
queries = [row["text"] for row in dataset["queries"]]
|
||||
documents, corpus_time = measure(lambda: provider.encode(corpus), args.repeats)
|
||||
vectors, query_time = measure(lambda: provider.encode(queries, query=True), args.repeats)
|
||||
_, single_time = measure(lambda: provider.encode([queries[0]], query=True), args.repeats)
|
||||
long_text = "合成测试资料仅用于测量本地编码资源,不表示任何真实景区信息。" * 12
|
||||
_, long_time = measure(lambda: provider.encode([long_text] * args.batch_size), args.repeats)
|
||||
try:
|
||||
provider.encode(["超长测试" * 500])
|
||||
except EmbeddingError:
|
||||
length_guard = True
|
||||
else:
|
||||
raise AssertionError("超长输入未被拒绝")
|
||||
import torch
|
||||
|
||||
report = {
|
||||
"schema_version": 1,
|
||||
"evaluation_kind": "synthetic_smoke"
|
||||
if dataset["purpose"] == "synthetic_smoke_not_domain_benchmark"
|
||||
else "provided_dataset_not_independently_verified",
|
||||
"dataset": {
|
||||
"id": dataset["dataset_id"],
|
||||
"purpose": dataset["purpose"],
|
||||
"sha256": digest,
|
||||
"documents": len(corpus),
|
||||
"queries": len(queries),
|
||||
},
|
||||
"embedding_spec": asdict(provider.spec),
|
||||
"space_id": provider.spec.space_id,
|
||||
"runtime": {
|
||||
"python": platform.python_version(),
|
||||
"torch": torch.__version__,
|
||||
"transformers": importlib.metadata.version("transformers"),
|
||||
"huggingface_hub": importlib.metadata.version("huggingface_hub"),
|
||||
"numpy": importlib.metadata.version("numpy"),
|
||||
"tokenizers": importlib.metadata.version("tokenizers"),
|
||||
"safetensors": importlib.metadata.version("safetensors"),
|
||||
"device": "cpu",
|
||||
"threads": torch.get_num_threads(),
|
||||
"interop_threads": torch.get_num_interop_threads(),
|
||||
"batch_size": args.batch_size,
|
||||
"repeats": args.repeats,
|
||||
"warmup_per_shape": 1,
|
||||
},
|
||||
"initialization_seconds": round(initialization_seconds, 3),
|
||||
"timings": {
|
||||
"corpus_encode": corpus_time,
|
||||
"query_batch_encode": query_time,
|
||||
"single_query_encode": single_time,
|
||||
"long_text_batch_encode": long_time,
|
||||
},
|
||||
"long_text_characters": len(long_text),
|
||||
"long_text_tokens": len(provider.tokenizer(long_text)["input_ids"]),
|
||||
"peak_process_rss_mib": round(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024, 1),
|
||||
"token_overflow_rejected": length_guard,
|
||||
"retrieval": rank_and_measure(dataset["documents"], dataset["queries"], documents, vectors),
|
||||
"limitations": [
|
||||
"合成小样本仅检查流程,不能代表那拉提领域质量或正式实验分数",
|
||||
"计时为本机预热后的编码,包含分词/补齐/归一化/结果转换,不含 API、数据库与网络请求",
|
||||
"峰值 RSS 是整个独立进程的高水位,包含解释器和模型;初始化包含缓存检查/下载与首次模型加载",
|
||||
],
|
||||
}
|
||||
encoded = json.dumps(report, ensure_ascii=False, indent=2) + "\n"
|
||||
if args.output:
|
||||
args.output.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.output.write_text(encoded, encoding="utf-8")
|
||||
args.output.chmod(0o600)
|
||||
print("评估完成,报告已写入指定本机位置。", file=sys.stderr)
|
||||
else:
|
||||
print(encoded, end="")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,30 +0,0 @@
|
||||
{
|
||||
"dataset_id": "synthetic-zh-travel-smoke-v1",
|
||||
"purpose": "synthetic_smoke_not_domain_benchmark",
|
||||
"documents": [
|
||||
{"id":"d01","text":"【合成测试】示例园区的停车场设有接驳车,步行游客可换乘到游客中心。"},
|
||||
{"id":"d02","text":"【合成测试】示例停车区域按车辆停留时段管理,离场时到出口办理停车手续。"},
|
||||
{"id":"d03","text":"【合成测试】示例展馆提供轮椅通道和电梯,行动不便的访客可从无障碍入口进入。"},
|
||||
{"id":"d04","text":"【合成测试】示例演出因降雨取消,已购票观众可沿原购票渠道申请退款。"},
|
||||
{"id":"d05","text":"【合成测试】示例红色文化展厅设有预约讲解,访客可提前登记并选择导览时段。"},
|
||||
{"id":"d06","text":"【合成测试】示例资料管理员核查发布机构、来源链接和更新时间,只有审核通过的资料才能成为问答依据。"},
|
||||
{"id":"d07","text":"【合成测试】示例民宿在前台办理入住,旅客需要出示有效身份证件并确认预订信息。"},
|
||||
{"id":"d08","text":"【合成测试】示例住宿提供早餐,有素食需求的客人可以提前向接待人员说明。"},
|
||||
{"id":"d09","text":"【合成测试】示例山间步道遇到恶劣天气暂停通行,游客应查看关闭提示并选择室内活动。"},
|
||||
{"id":"d10","text":"【合成测试】示例游客服务台统一接收捡到的物品,并登记失物线索协助寻找主人。"},
|
||||
{"id":"d11","text":"【合成测试】示例停车区设有电动车充电点,司机可按现场设备说明补充电量。"},
|
||||
{"id":"d12","text":"【合成测试】示例场馆的开放安排以入口公告为准,参观者应提前确认当天是否接待。"}
|
||||
],
|
||||
"queries": [
|
||||
{"id":"q01","text":"没有开车的人怎样从停车区去游客中心?","relevant_ids":["d01"]},
|
||||
{"id":"q02","text":"同行者坐轮椅,可以从哪里进入展馆?","relevant_ids":["d03"]},
|
||||
{"id":"q03","text":"演出被雨取消,已经付的钱能退吗?","relevant_ids":["d04"]},
|
||||
{"id":"q04","text":"想听展厅里的故事讲解,如何安排导览?","relevant_ids":["d05"]},
|
||||
{"id":"q05","text":"怎么确保问答使用的旅游介绍有可靠出处?","relevant_ids":["d06"]},
|
||||
{"id":"q06","text":"到了住宿地点办理入住需要准备什么证件?","relevant_ids":["d07"]},
|
||||
{"id":"q07","text":"早饭不吃肉,应该提前找谁说明?","relevant_ids":["d08"]},
|
||||
{"id":"q08","text":"天气恶劣时还能在山路上游玩吗?","relevant_ids":["d09"]},
|
||||
{"id":"q09","text":"在路上捡到游客遗落的东西应交到哪里?","relevant_ids":["d10"]},
|
||||
{"id":"q10","text":"电动汽车快没电了,到哪儿补充电量?","relevant_ids":["d11"]}
|
||||
]
|
||||
}
|
||||
@@ -1,26 +0,0 @@
|
||||
from alembic import context
|
||||
from sqlalchemy import create_engine, pool
|
||||
|
||||
from app import models # noqa: F401
|
||||
from app.core.config import settings
|
||||
from app.db.session import Base
|
||||
|
||||
config = context.config
|
||||
|
||||
|
||||
def run_migrations(connection):
|
||||
context.configure(connection=connection, target_metadata=Base.metadata, compare_type=True)
|
||||
with context.begin_transaction():
|
||||
context.run_migrations()
|
||||
|
||||
|
||||
if context.is_offline_mode():
|
||||
raise RuntimeError("迁移需要连接数据库,以检查并保留已有表;请使用 alembic upgrade head")
|
||||
elif config.attributes.get("connection") is not None:
|
||||
run_migrations(config.attributes["connection"])
|
||||
else:
|
||||
engine = create_engine(
|
||||
settings.database_url, poolclass=pool.NullPool, connect_args={"connect_timeout": 5}
|
||||
)
|
||||
with engine.connect() as connection:
|
||||
run_migrations(connection)
|
||||
@@ -1,17 +0,0 @@
|
||||
"""${message}"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
${imports if imports else ""}
|
||||
|
||||
revision = ${repr(up_revision)}
|
||||
down_revision = ${repr(down_revision)}
|
||||
branch_labels = ${repr(branch_labels)}
|
||||
depends_on = ${repr(depends_on)}
|
||||
|
||||
|
||||
def upgrade():
|
||||
${upgrades if upgrades else "pass"}
|
||||
|
||||
|
||||
def downgrade():
|
||||
${downgrades if downgrades else "pass"}
|
||||
@@ -1,48 +0,0 @@
|
||||
"""Register the original knowledge table without replacing existing data."""
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision = "0001_baseline"
|
||||
down_revision = None
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
op.execute("CREATE EXTENSION IF NOT EXISTS vector")
|
||||
inspector = sa.inspect(bind)
|
||||
if inspector.has_table("knowledge_items"):
|
||||
columns = {column["name"] for column in inspector.get_columns("knowledge_items")}
|
||||
expected = {
|
||||
"id",
|
||||
"title",
|
||||
"category",
|
||||
"content",
|
||||
"source_url",
|
||||
"verification_status",
|
||||
"created_at",
|
||||
}
|
||||
if columns != expected:
|
||||
raise RuntimeError("已有 knowledge_items 结构与基础版本不同,请先核对结构并备份")
|
||||
return
|
||||
op.create_table(
|
||||
"knowledge_items",
|
||||
sa.Column("id", sa.Integer(), primary_key=True),
|
||||
sa.Column("title", sa.String(250), nullable=False),
|
||||
sa.Column("category", sa.String(80), nullable=False),
|
||||
sa.Column("content", sa.Text(), nullable=False),
|
||||
sa.Column("source_url", sa.String(1000)),
|
||||
sa.Column("verification_status", sa.String(30), nullable=False),
|
||||
sa.Column(
|
||||
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
|
||||
),
|
||||
)
|
||||
op.create_index("ix_knowledge_items_id", "knowledge_items", ["id"])
|
||||
op.create_index("ix_knowledge_items_title", "knowledge_items", ["title"])
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.drop_table("knowledge_items")
|
||||
# 扩展可能由其他项目共享,不在回滚时删除。
|
||||
@@ -1,110 +0,0 @@
|
||||
"""Add source, document, review and audit management."""
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision = "0002_catalog"
|
||||
down_revision = "0001_baseline"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def version_columns():
|
||||
return [
|
||||
sa.Column("version", sa.Integer(), server_default="1", nullable=False),
|
||||
sa.Column(
|
||||
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
|
||||
),
|
||||
sa.Column(
|
||||
"updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
|
||||
),
|
||||
sa.Column("is_deleted", sa.Boolean(), server_default=sa.false(), nullable=False),
|
||||
]
|
||||
|
||||
|
||||
def upgrade():
|
||||
op.create_table(
|
||||
"sources",
|
||||
sa.Column("id", sa.Integer(), primary_key=True),
|
||||
sa.Column("name", sa.String(250), nullable=False),
|
||||
sa.Column("source_url", sa.String(1000)),
|
||||
sa.Column("notes", sa.Text(), nullable=False),
|
||||
*version_columns(),
|
||||
)
|
||||
op.create_table(
|
||||
"documents",
|
||||
sa.Column("id", sa.Integer(), primary_key=True),
|
||||
sa.Column("source_id", sa.Integer(), sa.ForeignKey("sources.id"), nullable=False),
|
||||
sa.Column("title", sa.String(250), nullable=False),
|
||||
sa.Column("category", sa.String(80), nullable=False),
|
||||
sa.Column("content", sa.Text(), nullable=False),
|
||||
sa.Column("collected_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("notes", sa.Text(), nullable=False),
|
||||
*version_columns(),
|
||||
)
|
||||
op.create_index("ix_documents_source_id", "documents", ["source_id"])
|
||||
op.create_table(
|
||||
"audit_events",
|
||||
sa.Column("id", sa.Integer(), primary_key=True),
|
||||
sa.Column("entity_type", sa.String(30), nullable=False),
|
||||
sa.Column("entity_id", sa.Integer(), nullable=False),
|
||||
sa.Column("action", sa.String(30), nullable=False),
|
||||
sa.Column("actor", sa.String(100)),
|
||||
sa.Column("notes", sa.Text(), nullable=False),
|
||||
sa.Column("snapshot", sa.JSON(), nullable=False),
|
||||
sa.Column(
|
||||
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
|
||||
),
|
||||
)
|
||||
op.create_index("ix_audit_events_entity_type", "audit_events", ["entity_type"])
|
||||
op.create_index("ix_audit_events_entity_id", "audit_events", ["entity_id"])
|
||||
for column in version_columns():
|
||||
if column.name != "created_at":
|
||||
op.add_column("knowledge_items", column)
|
||||
for name, kind in [
|
||||
("source_id", sa.Integer()),
|
||||
("document_id", sa.Integer()),
|
||||
("collected_at", sa.DateTime(timezone=True)),
|
||||
("reviewer", sa.String(100)),
|
||||
("reviewed_at", sa.DateTime(timezone=True)),
|
||||
("verification_notes", sa.Text()),
|
||||
]:
|
||||
op.add_column("knowledge_items", sa.Column(name, kind))
|
||||
for name, parent in [("source_id", "sources"), ("document_id", "documents")]:
|
||||
op.create_foreign_key(f"fk_knowledge_{name}", "knowledge_items", parent, [name], ["id"])
|
||||
op.create_index(f"ix_knowledge_items_{name}", "knowledge_items", [name])
|
||||
op.create_check_constraint(
|
||||
"ck_knowledge_verification_status",
|
||||
"knowledge_items",
|
||||
"verification_status IN ('pending', 'verified', 'rejected')",
|
||||
)
|
||||
op.execute("""INSERT INTO audit_events (entity_type, entity_id, action, notes, snapshot)
|
||||
SELECT 'knowledge_items', id, 'import', '保留基础版本数据及原状态',
|
||||
json_build_object('item', row_to_json(k)) FROM knowledge_items k""")
|
||||
op.execute("""UPDATE knowledge_items SET verification_status = 'pending', version = version + 1
|
||||
WHERE verification_status = 'verified'""")
|
||||
op.execute("""INSERT INTO audit_events (entity_type, entity_id, action, notes, snapshot)
|
||||
SELECT 'knowledge_items', id, 'invalidate', '旧版本没有完整审核记录,需要重新核验',
|
||||
json_build_object('item', row_to_json(k)) FROM knowledge_items k WHERE version = 2""")
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.drop_constraint("ck_knowledge_verification_status", "knowledge_items", type_="check")
|
||||
for name in ["source_id", "document_id"]:
|
||||
op.drop_index(f"ix_knowledge_items_{name}", table_name="knowledge_items")
|
||||
op.drop_constraint(f"fk_knowledge_{name}", "knowledge_items", type_="foreignkey")
|
||||
for name in [
|
||||
"source_id",
|
||||
"document_id",
|
||||
"collected_at",
|
||||
"reviewer",
|
||||
"reviewed_at",
|
||||
"verification_notes",
|
||||
"version",
|
||||
"updated_at",
|
||||
"is_deleted",
|
||||
]:
|
||||
op.drop_column("knowledge_items", name)
|
||||
op.drop_table("audit_events")
|
||||
op.drop_table("documents")
|
||||
op.drop_table("sources")
|
||||
@@ -1,28 +0,0 @@
|
||||
"""Persist keyword retrieval evidence and reproducible configuration."""
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision = "0003_retrieval_runs"
|
||||
down_revision = "0002_catalog"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
op.create_table(
|
||||
"retrieval_runs",
|
||||
sa.Column("id", sa.String(36), primary_key=True),
|
||||
sa.Column("query", sa.Text(), nullable=False),
|
||||
sa.Column("mode", sa.String(20), nullable=False),
|
||||
sa.Column("corpus_fingerprint", sa.String(64), nullable=False),
|
||||
sa.Column("response", sa.JSON(), nullable=False),
|
||||
sa.Column(
|
||||
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
|
||||
),
|
||||
)
|
||||
op.create_index("ix_retrieval_runs_created_at", "retrieval_runs", ["created_at"])
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.drop_table("retrieval_runs")
|
||||
@@ -1,52 +0,0 @@
|
||||
"""Add users, revocable opaque sessions and shared authentication budgets."""
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision = "0004_auth_sessions"
|
||||
down_revision = "0003_retrieval_runs"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
op.create_table(
|
||||
"users",
|
||||
sa.Column("id", sa.Integer(), primary_key=True),
|
||||
sa.Column("username", sa.String(32), nullable=False, unique=True),
|
||||
sa.Column("password_hash", sa.String(255), nullable=False),
|
||||
sa.Column("nickname", sa.String(80)),
|
||||
sa.Column("email", sa.String(254)),
|
||||
sa.Column("role", sa.String(10), nullable=False),
|
||||
sa.Column("credential_version", sa.Integer(), nullable=False),
|
||||
sa.Column(
|
||||
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
|
||||
),
|
||||
sa.CheckConstraint("role IN ('user', 'admin')", name="ck_users_role"),
|
||||
)
|
||||
op.create_table(
|
||||
"auth_sessions",
|
||||
sa.Column("token_hash", sa.String(64), primary_key=True),
|
||||
sa.Column(
|
||||
"user_id", sa.Integer(), sa.ForeignKey("users.id", ondelete="CASCADE"), nullable=False
|
||||
),
|
||||
sa.Column("credential_version", sa.Integer(), nullable=False),
|
||||
sa.Column(
|
||||
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
|
||||
),
|
||||
sa.Column("expires_at", sa.DateTime(timezone=True), nullable=False),
|
||||
)
|
||||
op.create_index("ix_auth_sessions_user_id", "auth_sessions", ["user_id"])
|
||||
op.create_index("ix_auth_sessions_expires_at", "auth_sessions", ["expires_at"])
|
||||
op.create_table(
|
||||
"auth_rate_limits",
|
||||
sa.Column("key", sa.String(64), primary_key=True),
|
||||
sa.Column("window_start", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("attempts", sa.Integer(), nullable=False),
|
||||
)
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.drop_table("auth_rate_limits")
|
||||
op.drop_table("auth_sessions")
|
||||
op.drop_table("users")
|
||||
@@ -1,81 +0,0 @@
|
||||
"""Persist reviewed chunks, version snapshots and atomic publication history."""
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision = "0005_chunk_indexes"
|
||||
down_revision = "0004_auth_sessions"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
op.create_table(
|
||||
"index_versions",
|
||||
sa.Column("id", sa.String(36), primary_key=True),
|
||||
sa.Column("status", sa.String(20), nullable=False),
|
||||
sa.Column("config", sa.JSON(), nullable=False),
|
||||
sa.Column("corpus_fingerprint", sa.String(64), nullable=False),
|
||||
sa.Column("catalog_revision", sa.Integer(), nullable=False),
|
||||
sa.Column("item_count", sa.Integer(), nullable=False),
|
||||
sa.Column("chunk_count", sa.Integer(), nullable=False),
|
||||
sa.Column("created_by", sa.Integer(), sa.ForeignKey("users.id"), nullable=False),
|
||||
sa.Column(
|
||||
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
|
||||
),
|
||||
sa.Column("published_at", sa.DateTime(timezone=True)),
|
||||
sa.CheckConstraint(
|
||||
"status IN ('ready', 'published', 'superseded')", name="ck_index_version_status"
|
||||
),
|
||||
)
|
||||
op.create_table(
|
||||
"index_state",
|
||||
sa.Column("id", sa.Integer(), primary_key=True),
|
||||
sa.Column("active_version_id", sa.String(36), sa.ForeignKey("index_versions.id")),
|
||||
sa.Column("generation", sa.Integer(), nullable=False),
|
||||
sa.Column("catalog_revision", sa.Integer(), nullable=False),
|
||||
sa.CheckConstraint("id = 1", name="ck_index_state_singleton"),
|
||||
)
|
||||
op.execute(
|
||||
sa.text("INSERT INTO index_state (id, generation, catalog_revision) VALUES (1, 0, 0)")
|
||||
)
|
||||
op.create_table(
|
||||
"chunks",
|
||||
sa.Column("id", sa.String(36), primary_key=True),
|
||||
sa.Column(
|
||||
"index_version_id", sa.String(36), sa.ForeignKey("index_versions.id"), nullable=False
|
||||
),
|
||||
sa.Column(
|
||||
"knowledge_id", sa.Integer(), sa.ForeignKey("knowledge_items.id"), nullable=False
|
||||
),
|
||||
sa.Column("ordinal", sa.Integer(), nullable=False),
|
||||
sa.Column("start_offset", sa.Integer(), nullable=False),
|
||||
sa.Column("end_offset", sa.Integer(), nullable=False),
|
||||
sa.Column("text", sa.Text(), nullable=False),
|
||||
sa.Column("text_sha256", sa.String(64), nullable=False),
|
||||
sa.Column("provenance", sa.JSON(), nullable=False),
|
||||
sa.UniqueConstraint(
|
||||
"index_version_id", "knowledge_id", "ordinal", name="uq_chunk_position"
|
||||
),
|
||||
sa.CheckConstraint(
|
||||
"start_offset >= 0 AND end_offset > start_offset", name="ck_chunk_offsets"
|
||||
),
|
||||
)
|
||||
op.create_index("ix_chunks_index_version_id", "chunks", ["index_version_id"])
|
||||
op.create_table(
|
||||
"index_publications",
|
||||
sa.Column("generation", sa.Integer(), primary_key=True),
|
||||
sa.Column("previous_version_id", sa.String(36), sa.ForeignKey("index_versions.id")),
|
||||
sa.Column("version_id", sa.String(36), sa.ForeignKey("index_versions.id"), nullable=False),
|
||||
sa.Column("actor_id", sa.Integer(), sa.ForeignKey("users.id"), nullable=False),
|
||||
sa.Column(
|
||||
"published_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.drop_table("index_publications")
|
||||
op.drop_table("chunks")
|
||||
op.drop_table("index_state")
|
||||
op.drop_table("index_versions")
|
||||
@@ -1,10 +0,0 @@
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
pythonpath = ["."]
|
||||
|
||||
[tool.ruff]
|
||||
target-version = "py311"
|
||||
line-length = 100
|
||||
|
||||
[tool.ruff.lint]
|
||||
select = ["E4", "E7", "E9", "F", "I"]
|
||||
@@ -1,6 +0,0 @@
|
||||
-r requirements.txt
|
||||
pytest>=8,<10
|
||||
httpx>=0.27,<1
|
||||
# FastAPI/Starlette TestClient uses the httpx2 package in this runtime.
|
||||
httpx2>=2.13,<3
|
||||
ruff>=0.9,<1
|
||||
@@ -1,8 +0,0 @@
|
||||
# Optional Linux CPU evaluation environment; not required by FastAPI startup.
|
||||
# Install the CPU torch wheel from its official index first; see the evaluation guide.
|
||||
torch==2.14.1+cpu
|
||||
transformers==5.20.0
|
||||
huggingface-hub==2.2.0
|
||||
numpy==2.4.6
|
||||
tokenizers==0.23.3
|
||||
safetensors==0.8.0
|
||||
@@ -5,7 +5,3 @@ pydantic-settings>=2.4,<3.0
|
||||
SQLAlchemy>=2.0,<3.0
|
||||
psycopg[binary]>=3.2,<4.0
|
||||
pgvector>=0.3,<1.0
|
||||
alembic>=1.13,<2.0
|
||||
jieba==0.42.1
|
||||
argon2-cffi>=25.1,<26
|
||||
email-validator>=2.2,<3
|
||||
|
||||
@@ -1,109 +0,0 @@
|
||||
"""Real PostgreSQL tests, isolated in a fresh schema for every test."""
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
from uuid import uuid4
|
||||
|
||||
import pytest
|
||||
from alembic import command
|
||||
from alembic.config import Config
|
||||
from fastapi.testclient import TestClient
|
||||
from sqlalchemy import create_engine, text
|
||||
from sqlalchemy.engine import make_url
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from app.db.session import get_db
|
||||
from app.main import create_app
|
||||
from app.models.auth import User
|
||||
from app.services.auth import PASSWORD_HASHER
|
||||
|
||||
BACKEND = Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
def migrate(engine, revision="head", *, downgrade=False):
|
||||
config = Config(str(BACKEND / "alembic.ini"))
|
||||
with engine.begin() as connection:
|
||||
config.attributes["connection"] = connection
|
||||
(command.downgrade if downgrade else command.upgrade)(config, revision)
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def postgres_url(tmp_path_factory):
|
||||
url = os.environ.get("TEST_DATABASE_URL")
|
||||
server = None
|
||||
if not url:
|
||||
try:
|
||||
import pgserver
|
||||
except ImportError:
|
||||
pytest.fail("请设置 TEST_DATABASE_URL,或安装可选的本机测试依赖 pgserver")
|
||||
server = pgserver.get_server(tmp_path_factory.mktemp("postgres"), cleanup_mode="stop")
|
||||
url = server.get_uri().replace("postgresql://", "postgresql+psycopg://", 1)
|
||||
if not url.startswith("postgresql+psycopg://"):
|
||||
pytest.fail("TEST_DATABASE_URL 必须使用 postgresql+psycopg://")
|
||||
bootstrap = create_engine(url)
|
||||
with bootstrap.begin() as connection:
|
||||
connection.execute(text("CREATE EXTENSION IF NOT EXISTS vector WITH SCHEMA public"))
|
||||
bootstrap.dispose()
|
||||
yield url
|
||||
if server:
|
||||
server.cleanup()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def database(postgres_url):
|
||||
schema = "test_" + uuid4().hex
|
||||
admin = create_engine(postgres_url)
|
||||
with admin.begin() as connection:
|
||||
connection.execute(text(f'CREATE SCHEMA "{schema}"'))
|
||||
# 不包含 public,防止误操作已有业务表。
|
||||
url = make_url(postgres_url).update_query_dict({"options": f"-csearch_path={schema}"})
|
||||
engine = create_engine(url)
|
||||
yield engine
|
||||
engine.dispose()
|
||||
with admin.begin() as connection:
|
||||
connection.execute(text(f'DROP SCHEMA "{schema}" CASCADE'))
|
||||
admin.dispose()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def migrated_database(database):
|
||||
migrate(database)
|
||||
return database
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def anonymous_client(migrated_database):
|
||||
app = create_app(migrated_database)
|
||||
factory = sessionmaker(bind=migrated_database)
|
||||
|
||||
def test_session():
|
||||
with factory() as session:
|
||||
try:
|
||||
yield session
|
||||
except Exception:
|
||||
session.rollback()
|
||||
raise
|
||||
|
||||
app.dependency_overrides[get_db] = test_session
|
||||
with TestClient(app, base_url="https://testserver") as client:
|
||||
client.headers["Origin"] = "http://localhost:5173"
|
||||
yield client
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client(anonymous_client, migrated_database):
|
||||
with sessionmaker(bind=migrated_database)() as db:
|
||||
db.add(
|
||||
User(
|
||||
username="test_admin",
|
||||
password_hash=PASSWORD_HASHER.hash("test-password-123"),
|
||||
role="admin",
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
response = anonymous_client.post(
|
||||
"/api/auth/login", json={"username": "test_admin", "password": "test-password-123"}
|
||||
)
|
||||
assert response.status_code == 200, response.text
|
||||
anonymous_client.headers["X-CSRF-Token"] = response.json()["csrf_token"]
|
||||
return anonymous_client
|
||||
@@ -1,342 +0,0 @@
|
||||
"""Authentication, authorization and CSRF boundaries over real PostgreSQL."""
|
||||
|
||||
import json
|
||||
from datetime import timedelta
|
||||
|
||||
import pytest
|
||||
from pydantic import ValidationError
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.core.config import Settings
|
||||
from app.models.auth import AuthSession, User
|
||||
from app.models.catalog import utc_now
|
||||
from app.services.auth import COOKIE_NAME, check_password, token_hash
|
||||
|
||||
PASSWORD = "test-password-123"
|
||||
ORIGIN = "http://localhost:5173"
|
||||
|
||||
|
||||
def register(client, username="test_user", **extra):
|
||||
return client.post(
|
||||
"/api/auth/register", json={"username": username, "password": PASSWORD, **extra}
|
||||
)
|
||||
|
||||
|
||||
def login(client, username="test_user", password=PASSWORD):
|
||||
response = client.post("/api/auth/login", json={"username": username, "password": password})
|
||||
if response.status_code == 200:
|
||||
client.headers["X-CSRF-Token"] = response.json()["csrf_token"]
|
||||
return response
|
||||
|
||||
|
||||
def test_registration_hashes_password_and_cannot_select_privilege(
|
||||
anonymous_client, migrated_database
|
||||
):
|
||||
c = anonymous_client
|
||||
for extra in [{"role": "admin"}, {"id": 1}, {"password_hash": "injected"}]:
|
||||
assert register(c, **extra).status_code == 422
|
||||
result = register(c, username=" Test_User ", email="user@example.com", nickname="占位用户")
|
||||
assert result.status_code == 201
|
||||
assert result.json()["username"] == "test_user" and result.json()["role"] == "user"
|
||||
assert c.get("/api/auth/me").status_code == 401
|
||||
assert register(c).status_code == 409
|
||||
with Session(migrated_database) as db:
|
||||
user = db.scalar(select(User))
|
||||
assert user.password_hash.startswith("$argon2id$")
|
||||
assert PASSWORD not in user.password_hash and check_password(user.password_hash, PASSWORD)
|
||||
assert db.scalar(select(func.count()).select_from(AuthSession)) == 0
|
||||
assert "password" not in result.text and "credential_version" not in result.text
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"data",
|
||||
[
|
||||
{"password": "short"},
|
||||
{"password": "x" * 129},
|
||||
{"username": "a"},
|
||||
{"username": "a b"},
|
||||
{"email": "invalid"},
|
||||
{"nickname": "x" * 81},
|
||||
{"nickname": "bad\u0000name"},
|
||||
],
|
||||
)
|
||||
def test_validation_does_not_echo_credentials(anonymous_client, data):
|
||||
submitted = {"username": "test_user", "password": PASSWORD, **data}
|
||||
response = anonymous_client.post("/api/auth/register", json=submitted)
|
||||
assert response.status_code == 422
|
||||
assert '"input"' not in response.text
|
||||
assert json.dumps(submitted["password"]) not in response.text
|
||||
assert response.headers["cache-control"] == "no-store"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("origin", [None, "null", "https://evil.example", ORIGIN + ".evil.example"])
|
||||
def test_login_and_registration_reject_untrusted_origin(
|
||||
anonymous_client, migrated_database, origin
|
||||
):
|
||||
c = anonymous_client
|
||||
c.headers.pop("Origin", None)
|
||||
headers = {"Origin": origin} if origin else {}
|
||||
for path in ["register", "login"]:
|
||||
response = c.post(
|
||||
f"/api/auth/{path}",
|
||||
json={"username": "test_user", "password": PASSWORD},
|
||||
headers=headers,
|
||||
)
|
||||
assert response.status_code == 403
|
||||
with Session(migrated_database) as db:
|
||||
assert db.scalar(select(func.count()).select_from(User)) == 0
|
||||
|
||||
|
||||
def test_login_cookie_rotation_logout_and_opaque_storage(anonymous_client, migrated_database):
|
||||
c = anonymous_client
|
||||
register(c)
|
||||
response = login(c)
|
||||
assert response.status_code == 200
|
||||
assert all(
|
||||
value in response.headers["set-cookie"]
|
||||
for value in ["HttpOnly", "Secure", "SameSite=lax", "Path=/"]
|
||||
)
|
||||
first = c.cookies.get(COOKIE_NAME)
|
||||
csrf = response.json()["csrf_token"]
|
||||
assert c.get("/api/auth/me").json()["csrf_token"] == csrf
|
||||
with Session(migrated_database) as db:
|
||||
stored = db.scalar(select(AuthSession))
|
||||
assert stored.token_hash == token_hash(first) and first not in stored.token_hash
|
||||
assert login(c).status_code == 200
|
||||
second = c.cookies.get(COOKIE_NAME)
|
||||
assert first != second
|
||||
c.cookies.clear()
|
||||
c.cookies.set(COOKIE_NAME, first)
|
||||
assert c.get("/api/auth/me").status_code == 401
|
||||
c.cookies.clear()
|
||||
c.cookies.set(COOKIE_NAME, second)
|
||||
response = c.post("/api/auth/logout")
|
||||
assert response.status_code == 204 and "Max-Age=0" in response.headers["set-cookie"]
|
||||
c.cookies.clear()
|
||||
c.cookies.set(COOKIE_NAME, second)
|
||||
assert c.get("/api/auth/me").status_code == 401
|
||||
|
||||
|
||||
def test_login_failure_is_generic_and_throttled_in_database(anonymous_client):
|
||||
c = anonymous_client
|
||||
register(c)
|
||||
absent = login(c, "absent_user", "wrong").json()
|
||||
wrong = login(c, password="wrong")
|
||||
assert wrong.status_code == 401 and wrong.json() == absent
|
||||
for _ in range(9):
|
||||
assert login(c, password="wrong").status_code == 401
|
||||
blocked = login(c)
|
||||
assert blocked.status_code == 429 and blocked.headers["retry-after"] == "300"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("username", ["bad\u0000name", "invalid name", "../user"])
|
||||
def test_login_rejects_invalid_username_before_database_lookup(anonymous_client, username):
|
||||
assert login(anonymous_client, username=username).status_code == 422
|
||||
|
||||
|
||||
def test_expired_forged_and_credential_version_sessions_are_rejected(
|
||||
anonymous_client, migrated_database
|
||||
):
|
||||
c = anonymous_client
|
||||
register(c)
|
||||
login(c)
|
||||
with Session(migrated_database) as db:
|
||||
session = db.scalar(select(AuthSession))
|
||||
session.expires_at = utc_now() - timedelta(seconds=1)
|
||||
db.commit()
|
||||
assert c.get("/api/auth/me").status_code == 401
|
||||
login(c)
|
||||
with Session(migrated_database) as db:
|
||||
db.scalar(select(User)).credential_version += 1
|
||||
db.commit()
|
||||
assert c.get("/api/auth/me").status_code == 401
|
||||
c.cookies.clear()
|
||||
c.cookies.set(COOKIE_NAME, "x" * 43)
|
||||
assert c.get("/api/auth/me").status_code == 401
|
||||
c.cookies.clear()
|
||||
c.cookies.set(COOKIE_NAME, "invalid")
|
||||
assert c.get("/api/auth/me").status_code == 401
|
||||
|
||||
|
||||
def test_csrf_is_bound_to_session_and_required_for_all_authenticated_writes(anonymous_client):
|
||||
c = anonymous_client
|
||||
register(c)
|
||||
login(c)
|
||||
first = c.headers["X-CSRF-Token"]
|
||||
login(c)
|
||||
for token in ["", "invalid", first]:
|
||||
assert (
|
||||
c.patch(
|
||||
"/api/users/me", json={"nickname": "占位"}, headers={"X-CSRF-Token": token}
|
||||
).status_code
|
||||
== 403
|
||||
)
|
||||
assert c.post("/api/auth/logout", headers={"X-CSRF-Token": token}).status_code == 403
|
||||
assert (
|
||||
c.patch(
|
||||
"/api/users/me", json={"nickname": "占位"}, headers={"Origin": "https://evil.example"}
|
||||
).status_code
|
||||
== 403
|
||||
)
|
||||
c.headers.pop("Origin")
|
||||
assert c.patch("/api/users/me", json={"nickname": "占位"}).status_code == 403
|
||||
c.headers["Origin"] = ORIGIN
|
||||
assert c.patch("/api/users/me", json={"nickname": " 占位 "}).json()["nickname"] == "占位"
|
||||
|
||||
|
||||
def test_profile_updates_only_own_allowed_fields(anonymous_client, migrated_database):
|
||||
c = anonymous_client
|
||||
first = register(c).json()
|
||||
register(c, "other_user")
|
||||
login(c)
|
||||
for payload in [
|
||||
{"id": 2},
|
||||
{"role": "admin"},
|
||||
{"username": "other_user"},
|
||||
{"password": "changed"},
|
||||
{},
|
||||
]:
|
||||
assert c.patch("/api/users/me", json=payload).status_code == 422
|
||||
assert (
|
||||
c.patch("/api/users/me", json={"email": "changed@example.com", "nickname": "示例"}).json()[
|
||||
"id"
|
||||
]
|
||||
== first["id"]
|
||||
)
|
||||
assert c.patch("/api/users/me", json={"email": "", "nickname": None}).json()["email"] is None
|
||||
with Session(migrated_database) as db:
|
||||
assert db.scalar(select(User).where(User.username == "other_user")).email is None
|
||||
assert db.scalar(select(User).where(User.id == first["id"])).role == "user"
|
||||
|
||||
|
||||
def test_password_change_revokes_all_sessions(anonymous_client):
|
||||
c = anonymous_client
|
||||
register(c)
|
||||
login(c)
|
||||
first = c.cookies.get(COOKIE_NAME)
|
||||
# Independent second browser: existing first session must also be invalidated.
|
||||
c.cookies.clear()
|
||||
login(c)
|
||||
assert (
|
||||
c.post(
|
||||
"/api/auth/change-password",
|
||||
json={"current_password": "wrong", "new_password": "new-password-123"},
|
||||
).status_code
|
||||
== 400
|
||||
)
|
||||
response = c.post(
|
||||
"/api/auth/change-password",
|
||||
json={"current_password": PASSWORD, "new_password": "new-password-123"},
|
||||
)
|
||||
assert response.status_code == 204
|
||||
assert c.get("/api/auth/me").status_code == 401
|
||||
c.cookies.set(COOKIE_NAME, first)
|
||||
assert c.get("/api/auth/me").status_code == 401
|
||||
assert login(c).status_code == 401
|
||||
assert login(c, password="new-password-123").status_code == 200
|
||||
|
||||
|
||||
def test_all_management_routes_require_admin_and_history_is_private(anonymous_client):
|
||||
c = anonymous_client
|
||||
paths = [
|
||||
("GET", "/api/admin/users"),
|
||||
("GET", "/api/knowledge"),
|
||||
("GET", "/api/knowledge/page"),
|
||||
("GET", "/api/knowledge/1"),
|
||||
("POST", "/api/knowledge"),
|
||||
("PATCH", "/api/knowledge/1"),
|
||||
("DELETE", "/api/knowledge/1?expected_version=1"),
|
||||
("POST", "/api/knowledge/1/review"),
|
||||
("GET", "/api/knowledge/1/history"),
|
||||
("GET", "/api/sources"),
|
||||
("POST", "/api/sources"),
|
||||
("GET", "/api/sources/1"),
|
||||
("PATCH", "/api/sources/1"),
|
||||
("DELETE", "/api/sources/1?expected_version=1"),
|
||||
("GET", "/api/sources/1/history"),
|
||||
("GET", "/api/documents"),
|
||||
("POST", "/api/documents"),
|
||||
("GET", "/api/documents/1"),
|
||||
("PATCH", "/api/documents/1"),
|
||||
("DELETE", "/api/documents/1?expected_version=1"),
|
||||
("GET", "/api/documents/1/history"),
|
||||
("GET", "/api/retrieval/runs"),
|
||||
("GET", "/api/retrieval/runs/00000000-0000-0000-0000-000000000000"),
|
||||
]
|
||||
for method, path in paths:
|
||||
assert c.request(method, path, json={}).status_code == 401, path
|
||||
register(c)
|
||||
login(c)
|
||||
for method, path in paths:
|
||||
assert c.request(method, path, json={}).status_code == 403, path
|
||||
assert c.post("/api/retrieval", json={"query": "占位"}).json()["status"] == "no_evidence"
|
||||
assert c.post("/api/chat", json={"question": "占位"}).json()["status"] == "stub"
|
||||
|
||||
|
||||
def test_admin_access_live_role_and_no_csrf_bypass(client, migrated_database):
|
||||
c = client
|
||||
users = c.get("/api/admin/users").json()
|
||||
assert users[0]["role"] == "admin" and "password_hash" not in str(users)
|
||||
assert c.get("/api/admin/users?limit=101").status_code == 422
|
||||
assert (
|
||||
c.post(
|
||||
"/api/knowledge",
|
||||
json={"title": "示例", "content": "示例"},
|
||||
headers={"X-CSRF-Token": ""},
|
||||
).status_code
|
||||
== 403
|
||||
)
|
||||
assert c.post("/api/knowledge", json={"title": "示例", "content": "示例"}).status_code == 201
|
||||
with Session(migrated_database) as db:
|
||||
db.scalar(select(User)).role = "user"
|
||||
db.commit()
|
||||
assert c.get("/api/admin/users").status_code == 403
|
||||
assert c.get("/api/knowledge").status_code == 403
|
||||
|
||||
|
||||
def test_auth_settings_require_explicit_origins_and_secure_production():
|
||||
with pytest.raises(ValidationError):
|
||||
Settings(_env_file=None, app_env="production", auth_cookie_secure=False)
|
||||
for origin in [
|
||||
"*",
|
||||
"null",
|
||||
"https://example.com/path",
|
||||
"https://user@example.com",
|
||||
"https://*.example.com",
|
||||
]:
|
||||
with pytest.raises(ValidationError):
|
||||
Settings(_env_file=None, auth_allowed_origins=origin)
|
||||
|
||||
|
||||
def test_local_admin_provisioning_has_no_default_account(migrated_database, monkeypatch, capsys):
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from app import manage
|
||||
|
||||
monkeypatch.setattr(manage, "SessionLocal", sessionmaker(bind=migrated_database))
|
||||
monkeypatch.setattr("sys.argv", ["manage", "create-admin", "--username", "local_admin"])
|
||||
monkeypatch.setattr(manage.getpass, "getpass", lambda _: PASSWORD)
|
||||
manage.main()
|
||||
with Session(migrated_database) as db:
|
||||
user = db.scalar(select(User))
|
||||
assert user.role == "admin" and check_password(user.password_hash, PASSWORD)
|
||||
with pytest.raises(SystemExit) as error:
|
||||
manage.main()
|
||||
assert error.value.code == 1
|
||||
assert PASSWORD not in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_client_budget_is_shared_across_names_and_registration(anonymous_client, migrated_database):
|
||||
c = anonymous_client
|
||||
for index in range(30):
|
||||
assert login(c, username=f"absent_{index}").status_code == 401
|
||||
assert register(c).status_code == 429
|
||||
# Expire the persisted budget instead of waiting in a test.
|
||||
from app.models.auth import AuthRateLimit
|
||||
|
||||
with Session(migrated_database) as db:
|
||||
for bucket in db.scalars(select(AuthRateLimit)):
|
||||
bucket.window_start = utc_now() - timedelta(minutes=6)
|
||||
db.commit()
|
||||
assert register(c).status_code == 201
|
||||
@@ -1,269 +0,0 @@
|
||||
from datetime import datetime
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session
|
||||
from sqlalchemy.orm.exc import StaleDataError
|
||||
|
||||
from app.models import KnowledgeItem
|
||||
|
||||
COLLECTED = "2026-10-15T09:00:00+08:00"
|
||||
|
||||
|
||||
def create(client, path, data):
|
||||
response = client.post("/api/" + path, json=data)
|
||||
assert response.status_code == 201, response.text
|
||||
return response.json()
|
||||
|
||||
|
||||
def source(client):
|
||||
return create(client, "sources", {"name": "示例来源", "source_url": "https://example.com/"})
|
||||
|
||||
|
||||
def document(client, source_id):
|
||||
return create(
|
||||
client,
|
||||
"documents",
|
||||
{
|
||||
"source_id": source_id,
|
||||
"title": "示例文档",
|
||||
"content": "仅用于自动化测试的占位内容",
|
||||
"collected_at": COLLECTED,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def knowledge(client, **overrides):
|
||||
return create(
|
||||
client,
|
||||
"knowledge",
|
||||
{
|
||||
"title": "示例知识",
|
||||
"content": "测试资料,不代表文旅事实",
|
||||
"source_url": "https://example.com/",
|
||||
"collected_at": COLLECTED,
|
||||
**overrides,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def review(client, item, status="verified"):
|
||||
return client.post(
|
||||
f"/api/knowledge/{item['id']}/review",
|
||||
json={
|
||||
"expected_version": item["version"],
|
||||
"verification_status": status,
|
||||
"reviewer": "test-reviewer",
|
||||
"verification_notes": "仅验证审核流程,非真实核验结论",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def test_health_docs_and_stub(client):
|
||||
assert client.get("/api/health").json()["status"] == "ok"
|
||||
assert client.get("/api/ready").json()["database"] == "connected"
|
||||
assert client.get("/docs").status_code == 200
|
||||
assert "/api/knowledge/{item_id}/review" in client.get("/openapi.json").json()["paths"]
|
||||
assert client.post("/api/chat", json={"question": "测试"}).json()["status"] == "stub"
|
||||
|
||||
|
||||
def test_legacy_create_and_missing_evidence(client):
|
||||
item = create(client, "knowledge", {"title": "旧版调用", "content": "示例"})
|
||||
assert item["verification_status"] == "pending"
|
||||
assert item["collected_at"] is None
|
||||
assert isinstance(client.get("/api/knowledge").json(), list)
|
||||
assert review(client, item).status_code == 422
|
||||
assert client.get(f"/api/knowledge/{item['id']}/history").json()["total"] == 1
|
||||
|
||||
|
||||
def test_optional_empty_url_matches_existing_frontend(client):
|
||||
item = create(client, "knowledge", {"title": "示例", "content": "示例", "source_url": ""})
|
||||
assert item["source_url"] is None
|
||||
|
||||
|
||||
def test_reassign_document_resolves_new_source(client):
|
||||
first = source(client)
|
||||
first_doc = document(client, first["id"])
|
||||
second = source(client)
|
||||
second_doc = document(client, second["id"])
|
||||
item = knowledge(client, document_id=first_doc["id"])
|
||||
response = client.patch(
|
||||
f"/api/knowledge/{item['id']}",
|
||||
json={
|
||||
"expected_version": item["version"],
|
||||
"document_id": second_doc["id"],
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
assert response.json()["source_id"] == second["id"]
|
||||
|
||||
|
||||
def test_browser_preflight_and_document_source_lists(client):
|
||||
response = client.options(
|
||||
"/api/knowledge",
|
||||
headers={
|
||||
"Origin": "http://localhost:5173",
|
||||
"Access-Control-Request-Method": "POST",
|
||||
"Access-Control-Request-Headers": "content-type",
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
assert response.headers["access-control-allow-origin"] == "http://localhost:5173"
|
||||
src = source(client)
|
||||
doc = document(client, src["id"])
|
||||
assert client.get("/api/sources?q=示例").json()["items"][0]["id"] == src["id"]
|
||||
result = client.get(f"/api/documents?source_id={src['id']}&q=占位").json()
|
||||
assert result["total"] == 1 and result["items"][0]["id"] == doc["id"]
|
||||
|
||||
|
||||
def test_review_edit_and_history(client):
|
||||
item = knowledge(client)
|
||||
verified = review(client, item)
|
||||
assert verified.status_code == 200
|
||||
verified = verified.json()
|
||||
assert verified["version"] == 2
|
||||
assert datetime.fromisoformat(verified["reviewed_at"]).tzinfo is not None
|
||||
assert review(client, item).status_code == 409
|
||||
response = client.patch(
|
||||
f"/api/knowledge/{item['id']}",
|
||||
json={
|
||||
"expected_version": verified["version"],
|
||||
"content": "修订后的示例正文",
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
updated = response.json()
|
||||
assert updated["verification_status"] == "pending"
|
||||
assert updated["reviewer"] is None and updated["reviewed_at"] is None
|
||||
events = client.get(f"/api/knowledge/{item['id']}/history").json()
|
||||
assert [event["action"] for event in events["items"]] == ["update", "review", "create"]
|
||||
assert events["items"][1]["snapshot"]["item"]["verification_status"] == "verified"
|
||||
|
||||
|
||||
def test_source_document_changes_invalidate_evidence(client):
|
||||
src = source(client)
|
||||
doc = document(client, src["id"])
|
||||
item = knowledge(client, source_id=src["id"], document_id=doc["id"])
|
||||
assert review(client, item).status_code == 200
|
||||
response = client.patch(
|
||||
f"/api/documents/{doc['id']}",
|
||||
json={
|
||||
"expected_version": doc["version"],
|
||||
"content": "修订后的文档",
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
current = client.get(f"/api/knowledge/{item['id']}").json()
|
||||
assert current["verification_status"] == "pending"
|
||||
assert review(client, current).status_code == 200
|
||||
response = client.patch(
|
||||
f"/api/sources/{src['id']}",
|
||||
json={
|
||||
"expected_version": src["version"],
|
||||
"source_url": "https://example.com/new",
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
current = client.get(f"/api/knowledge/{item['id']}").json()
|
||||
assert current["verification_status"] == "pending"
|
||||
assert current["source_url"] == "https://example.com/new"
|
||||
events = client.get(f"/api/knowledge/{item['id']}/history").json()["items"]
|
||||
reviewed = next(e for e in events if e["action"] == "review")
|
||||
assert reviewed["snapshot"]["source"]["source_url"] == "https://example.com/"
|
||||
assert reviewed["snapshot"]["document"]["content"] == "修订后的文档"
|
||||
|
||||
|
||||
def test_delete_preserves_history_and_guards_references(client):
|
||||
src = source(client)
|
||||
doc = document(client, src["id"])
|
||||
item = knowledge(client, document_id=doc["id"])
|
||||
assert item["source_id"] == src["id"]
|
||||
assert client.delete(f"/api/sources/{src['id']}?expected_version=1").status_code == 409
|
||||
assert client.delete(f"/api/documents/{doc['id']}?expected_version=1").status_code == 409
|
||||
assert client.delete(f"/api/knowledge/{item['id']}?expected_version=2").status_code == 409
|
||||
assert client.delete(f"/api/knowledge/{item['id']}?expected_version=1").status_code == 204
|
||||
assert client.get(f"/api/knowledge/{item['id']}").status_code == 404
|
||||
assert client.get("/api/knowledge").json() == []
|
||||
assert (
|
||||
client.get(f"/api/knowledge/{item['id']}/history").json()["items"][0]["action"] == "delete"
|
||||
)
|
||||
assert client.delete(f"/api/documents/{doc['id']}?expected_version=1").status_code == 204
|
||||
assert client.delete(f"/api/sources/{src['id']}?expected_version=1").status_code == 204
|
||||
assert client.get("/api/sources").json()["total"] == 0
|
||||
|
||||
|
||||
def test_filter_pagination_and_literal_search(client):
|
||||
knowledge(client, title="示例%资料", category="交通出行")
|
||||
other = knowledge(client, title="其他资料", category="民宿服务")
|
||||
assert review(client, other, "rejected").status_code == 200
|
||||
page = client.get("/api/knowledge/page?page_size=1").json()
|
||||
assert page["total"] == 2 and len(page["items"]) == 1
|
||||
assert page["items"][0]["id"] == other["id"]
|
||||
assert client.get("/api/knowledge/page?page=3&page_size=1").json()["items"] == []
|
||||
assert client.get("/api/knowledge/page?q=%25").json()["total"] == 1
|
||||
assert len(client.get("/api/knowledge?category=交通出行").json()) == 1
|
||||
assert client.get("/api/knowledge/page?verification_status=verified").json()["total"] == 0
|
||||
assert client.get("/api/knowledge/page?verification_status=rejected").json()["total"] == 1
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"patch",
|
||||
[
|
||||
{"title": " "},
|
||||
{"title": None},
|
||||
{"content": ""},
|
||||
{"category": ""},
|
||||
{"collected_at": "2026-10-15T09:00:00"},
|
||||
{"source_url": "javascript:alert(1)"},
|
||||
{"verification_status": "verified"},
|
||||
{},
|
||||
],
|
||||
)
|
||||
def test_invalid_update_cannot_mutate_or_audit(client, patch):
|
||||
item = knowledge(client)
|
||||
response = client.patch(f"/api/knowledge/{item['id']}", json={"expected_version": 1, **patch})
|
||||
assert response.status_code == 422
|
||||
assert client.get(f"/api/knowledge/{item['id']}").json()["version"] == 1
|
||||
assert client.get(f"/api/knowledge/{item['id']}/history").json()["total"] == 1
|
||||
|
||||
|
||||
def test_invalid_relationship_and_query(client):
|
||||
src = source(client)
|
||||
doc = document(client, src["id"])
|
||||
second = source(client)
|
||||
response = client.post(
|
||||
"/api/knowledge",
|
||||
json={
|
||||
"title": "示例",
|
||||
"content": "示例",
|
||||
"source_id": second["id"],
|
||||
"document_id": doc["id"],
|
||||
},
|
||||
)
|
||||
assert response.status_code == 422
|
||||
assert (
|
||||
client.post(
|
||||
"/api/knowledge", json={"title": "示例", "content": "示例", "document_id": 9999}
|
||||
).status_code
|
||||
== 404
|
||||
)
|
||||
for path in ["knowledge", "sources", "documents"]:
|
||||
assert client.get(f"/api/{path}/9999").status_code == 404
|
||||
assert client.get(f"/api/{path}/9999/history").status_code == 404
|
||||
assert client.get("/api/knowledge/page?page_size=101").status_code == 422
|
||||
assert client.get("/api/knowledge/page?verification_status=unknown").status_code == 422
|
||||
assert client.get("/api/knowledge/page?page=0").status_code == 422
|
||||
|
||||
|
||||
def test_optimistic_lock_prevents_concurrent_overwrite(client, migrated_database):
|
||||
item = knowledge(client)
|
||||
with Session(migrated_database) as first, Session(migrated_database) as second:
|
||||
first_item = first.scalar(select(KnowledgeItem).where(KnowledgeItem.id == item["id"]))
|
||||
second_item = second.get(KnowledgeItem, item["id"])
|
||||
first_item.content = "先提交的内容"
|
||||
first.commit()
|
||||
second_item.content = "过期编辑"
|
||||
with pytest.raises(StaleDataError):
|
||||
second.commit()
|
||||
second.rollback()
|
||||
assert client.get(f"/api/knowledge/{item['id']}").json()["content"] == "先提交的内容"
|
||||
@@ -1,127 +0,0 @@
|
||||
"""Lightweight contract/metric tests do not require model packages or network access."""
|
||||
|
||||
import json
|
||||
from dataclasses import FrozenInstanceError, replace
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from app.services.embedding import (
|
||||
SMALL_ZH,
|
||||
EmbeddingError,
|
||||
EmbeddingSpec,
|
||||
LocalEmbeddingProvider,
|
||||
validate_vectors,
|
||||
)
|
||||
from evaluation.local_embeddings import load_dataset, rank_and_measure
|
||||
|
||||
|
||||
def test_model_space_identity_is_immutable_and_includes_semantic_configuration():
|
||||
assert SMALL_ZH.space_id == replace(SMALL_ZH).space_id
|
||||
for change in [
|
||||
{"revision": "a" * 40},
|
||||
{"dimension": 768},
|
||||
{"query_instruction": ""},
|
||||
{"token_limit": 256},
|
||||
{"implementation": "another-pooling"},
|
||||
]:
|
||||
assert replace(SMALL_ZH, **change).space_id != SMALL_ZH.space_id
|
||||
with pytest.raises(FrozenInstanceError):
|
||||
SMALL_ZH.dimension = 768
|
||||
|
||||
|
||||
@pytest.mark.parametrize("revision", ["main", "v1.5", "a" * 39, None])
|
||||
def test_model_requires_full_revision(revision):
|
||||
with pytest.raises(EmbeddingError):
|
||||
EmbeddingSpec("example", revision, 2)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"vectors,count",
|
||||
[
|
||||
([[1.0, 0.0]], 2),
|
||||
([[1.0]], 1),
|
||||
([[0.0, 0.0]], 1),
|
||||
([[float("nan"), 0.0]], 1),
|
||||
([[float("inf"), 0.0]], 1),
|
||||
([["invalid", 0.0]], 1),
|
||||
([[True, 0.0]], 1),
|
||||
],
|
||||
)
|
||||
def test_invalid_vectors_cannot_pass_contract(vectors, count):
|
||||
spec = EmbeddingSpec("example", "a" * 40, 2)
|
||||
with pytest.raises(EmbeddingError):
|
||||
validate_vectors(vectors, count, spec)
|
||||
validate_vectors([[1.0, 0.0]], 1, spec)
|
||||
|
||||
|
||||
def test_token_overflow_is_rejected_before_any_model_execution():
|
||||
provider = object.__new__(LocalEmbeddingProvider)
|
||||
provider.spec = SMALL_ZH
|
||||
provider.batch_size = 8
|
||||
provider.torch = None
|
||||
|
||||
class Shape:
|
||||
shape = (1, 513)
|
||||
|
||||
provider.tokenizer = lambda *args, **kwargs: {"input_ids": Shape()}
|
||||
# There is intentionally no model: exceeding the budget must never execute one.
|
||||
with pytest.raises(EmbeddingError, match="token"):
|
||||
provider.encode(["占位文本"])
|
||||
|
||||
|
||||
def test_ranking_metrics_match_hand_calculation_and_ties_are_stable():
|
||||
docs = [{"id": id_} for id_ in ["a", "b", "c"]]
|
||||
queries = [{"id": "q1", "relevant_ids": ["b", "c"]}, {"id": "q2", "relevant_ids": ["c"]}]
|
||||
result = rank_and_measure(docs, queries, [[1, 0], [0, 1], [-1, 0]], [[1, 1], [-1, 0]], top_k=2)
|
||||
assert result["hit_at_1"] == 0.5
|
||||
assert result["recall_at_k"] == 0.75
|
||||
assert result["mrr_at_k"] == 0.75
|
||||
assert [x["id"] for x in result["per_query"][0]["top_results"]] == ["a", "b"]
|
||||
|
||||
|
||||
def test_dataset_validation_rejects_unknown_labels_and_duplicate_ids(tmp_path):
|
||||
data = {
|
||||
"dataset_id": "placeholder",
|
||||
"purpose": "synthetic_smoke_not_domain_benchmark",
|
||||
"documents": [{"id": "d", "text": "占位"}],
|
||||
"queries": [{"id": "q", "text": "测试", "relevant_ids": ["missing"]}],
|
||||
}
|
||||
path = tmp_path / "dataset.json"
|
||||
path.write_text(json.dumps(data))
|
||||
with pytest.raises(ValueError):
|
||||
load_dataset(path)
|
||||
data["queries"][0]["relevant_ids"] = ["d"]
|
||||
data["documents"].append({"id": "d", "text": "重复"})
|
||||
path.write_text(json.dumps(data))
|
||||
with pytest.raises(ValueError):
|
||||
load_dataset(path)
|
||||
|
||||
|
||||
def test_bundled_dataset_is_explicitly_synthetic_and_reproducible():
|
||||
path = Path(__file__).resolve().parents[1] / "evaluation" / "synthetic_zh.json"
|
||||
dataset, digest = load_dataset(path)
|
||||
assert dataset["purpose"] == "synthetic_smoke_not_domain_benchmark"
|
||||
assert all(row["text"].startswith("【合成测试】") for row in dataset["documents"])
|
||||
assert load_dataset(path)[1] == digest
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"options", [{"batch_size": 0}, {"batch_size": 33}, {"threads": 5}, {"batch_size": 1.5}]
|
||||
)
|
||||
def test_invalid_resource_options_fail_without_loading_optional_dependencies(options):
|
||||
with pytest.raises(EmbeddingError):
|
||||
LocalEmbeddingProvider(SMALL_ZH, **options)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("checkpoint", ["pickle", "automatic-fallback"])
|
||||
def test_unsupported_checkpoint_format_is_rejected(checkpoint):
|
||||
with pytest.raises(EmbeddingError):
|
||||
replace(SMALL_ZH, checkpoint_format=checkpoint)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("texts", [None, "不能逐字编码的字符串", ("占位",), [None], [" "]])
|
||||
def test_input_shape_and_text_are_validated_before_encoding(texts):
|
||||
provider = object.__new__(LocalEmbeddingProvider)
|
||||
with pytest.raises(EmbeddingError):
|
||||
provider.encode(texts)
|
||||
@@ -1,320 +0,0 @@
|
||||
"""Versioned chunks remain evidence snapshots; publication uses real PostgreSQL locks."""
|
||||
|
||||
import hashlib
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
import pytest
|
||||
from fastapi import HTTPException
|
||||
from sqlalchemy import func, select, text
|
||||
from sqlalchemy.exc import OperationalError
|
||||
from sqlalchemy.orm import Session
|
||||
from test_retrieval import add_item
|
||||
|
||||
from app.models.indexing import Chunk, IndexPublication, IndexState, IndexVersion
|
||||
from app.services import indexing
|
||||
from app.services.indexing import locked_state
|
||||
|
||||
|
||||
def build(client, **payload):
|
||||
response = client.post("/api/indexes", json=payload)
|
||||
assert response.status_code == 201, response.text
|
||||
return response.json()
|
||||
|
||||
|
||||
def publish(client, version, generation=0):
|
||||
return client.post(
|
||||
f"/api/indexes/{version['id']}/publish", json={"expected_generation": generation}
|
||||
)
|
||||
|
||||
|
||||
def chunks(client, version):
|
||||
return client.get(f"/api/indexes/{version['id']}/chunks?page_size=100").json()["items"]
|
||||
|
||||
|
||||
def test_build_preserves_original_offsets_content_hash_and_provenance(client):
|
||||
content = ("A字符\n那拉提🙂占位资料。" * 37).strip()
|
||||
item = add_item(client, content=content)
|
||||
first = build(client, chunk_size=100, overlap=20)
|
||||
second = build(client, chunk_size=100, overlap=20)
|
||||
assert first["status"] == "ready" and first["item_count"] == 1
|
||||
assert first["corpus_fingerprint"] == second["corpus_fingerprint"]
|
||||
assert first["config"] == second["config"] and first["id"] != second["id"]
|
||||
actual = chunks(client, first)
|
||||
assert len(actual) == first["chunk_count"]
|
||||
assert [c["ordinal"] for c in actual] == list(range(len(actual)))
|
||||
assert actual[0]["start_offset"] == 0 and actual[-1]["end_offset"] == len(content)
|
||||
restored = [""] * len(content)
|
||||
for chunk in actual:
|
||||
start, end = chunk["start_offset"], chunk["end_offset"]
|
||||
assert chunk["text"] == content[start:end]
|
||||
assert chunk["text_sha256"] == hashlib.sha256(chunk["text"].encode()).hexdigest()
|
||||
assert chunk["provenance"]["knowledge_version"] == item["version"]
|
||||
assert chunk["provenance"]["source_url"] == "https://example.com/"
|
||||
restored[start:end] = list(chunk["text"])
|
||||
assert "".join(restored) == content
|
||||
assert [(c["text"], c["start_offset"], c["end_offset"]) for c in actual] == [
|
||||
(c["text"], c["start_offset"], c["end_offset"]) for c in chunks(client, second)
|
||||
]
|
||||
state = client.get("/api/indexes/active").json()
|
||||
assert state["generation"] == 0 and state["active_version"] is None and not state["is_current"]
|
||||
|
||||
|
||||
def test_only_valid_reviewed_evidence_is_chunked(client):
|
||||
good = add_item(client)
|
||||
add_item(client, status="pending")
|
||||
add_item(client, status="rejected")
|
||||
removed = add_item(client)
|
||||
assert (
|
||||
client.delete(
|
||||
f"/api/knowledge/{removed['id']}?expected_version={removed['version']}"
|
||||
).status_code
|
||||
== 204
|
||||
)
|
||||
version = build(client)
|
||||
assert version["item_count"] == 1
|
||||
assert {c["knowledge_id"] for c in chunks(client, version)} == {good["id"]}
|
||||
|
||||
|
||||
def test_publication_is_atomic_optimistic_and_audited(client):
|
||||
add_item(client)
|
||||
first, second = build(client), build(client)
|
||||
result = publish(client, first)
|
||||
assert result.status_code == 200 and result.json()["generation"] == 1
|
||||
assert result.json()["is_current"]
|
||||
assert publish(client, second).status_code == 409
|
||||
assert publish(client, second, 1).json()["generation"] == 2
|
||||
assert client.get(f"/api/indexes/{first['id']}").json()["status"] == "superseded"
|
||||
assert client.get(f"/api/indexes/{second['id']}").json()["status"] == "published"
|
||||
# Same active version with the current generation is an idempotent no-op.
|
||||
assert publish(client, second, 2).json()["generation"] == 2
|
||||
history = client.get("/api/indexes/publications").json()
|
||||
assert history["total"] == 2
|
||||
assert history["items"][0]["previous_version_id"] == first["id"]
|
||||
assert history["items"][0]["version_id"] == second["id"]
|
||||
assert client.get("/api/indexes/active").json()["active_version"]["id"] == second["id"]
|
||||
|
||||
|
||||
def test_catalog_mutation_marks_active_stale_and_rejects_old_build(client):
|
||||
item = add_item(client)
|
||||
version = build(client)
|
||||
assert publish(client, version).status_code == 200
|
||||
old = chunks(client, version)
|
||||
updated = client.patch(
|
||||
f"/api/knowledge/{item['id']}",
|
||||
json={"expected_version": item["version"], "content": "修订的占位正文"},
|
||||
)
|
||||
assert updated.status_code == 200
|
||||
active = client.get("/api/indexes/active").json()
|
||||
assert not active["is_current"] and active["generation"] == 1
|
||||
assert active["active_version"]["id"] == version["id"]
|
||||
assert chunks(client, version) == old
|
||||
assert publish(client, version, 1).status_code == 409
|
||||
replacement = build(client)
|
||||
assert replacement["item_count"] == 0 # revised body is pending, not approved evidence
|
||||
assert publish(client, replacement, 1).json()["is_current"]
|
||||
assert chunks(client, replacement) == []
|
||||
|
||||
|
||||
def test_failed_catalog_write_does_not_invalidate_current_version(client):
|
||||
item = add_item(client)
|
||||
version = build(client)
|
||||
publish(client, version)
|
||||
before = client.get("/api/indexes/active").json()
|
||||
assert (
|
||||
client.patch(
|
||||
f"/api/knowledge/{item['id']}", json={"expected_version": 999, "content": "占位"}
|
||||
).status_code
|
||||
== 409
|
||||
)
|
||||
assert client.post("/api/knowledge", json={"title": "", "content": "占位"}).status_code == 422
|
||||
assert client.get("/api/indexes/active").json() == before
|
||||
|
||||
|
||||
def test_source_and_document_mutations_preserve_old_provenance(client):
|
||||
src = client.post(
|
||||
"/api/sources", json={"name": "原始占位来源", "source_url": "https://example.com/"}
|
||||
).json()
|
||||
doc = client.post(
|
||||
"/api/documents",
|
||||
json={
|
||||
"source_id": src["id"],
|
||||
"title": "占位文档",
|
||||
"content": "占位正文",
|
||||
"collected_at": "2026-10-16T00:00:00Z",
|
||||
},
|
||||
).json()
|
||||
add_item(client, document_id=doc["id"])
|
||||
version = build(client)
|
||||
publish(client, version)
|
||||
snapshot = chunks(client, version)
|
||||
assert snapshot[0]["provenance"]["document_version"] == doc["version"]
|
||||
assert snapshot[0]["provenance"]["source_name"] == src["name"]
|
||||
assert (
|
||||
client.patch(
|
||||
f"/api/documents/{doc['id']}", json={"expected_version": 1, "content": "文档新正文"}
|
||||
).status_code
|
||||
== 200
|
||||
)
|
||||
assert not client.get("/api/indexes/active").json()["is_current"]
|
||||
empty = build(client)
|
||||
publish(client, empty, 1)
|
||||
assert (
|
||||
client.patch(
|
||||
f"/api/sources/{src['id']}", json={"expected_version": 1, "name": "新来源名"}
|
||||
).status_code
|
||||
== 200
|
||||
)
|
||||
assert not client.get("/api/indexes/active").json()["is_current"]
|
||||
assert chunks(client, version) == snapshot
|
||||
assert publish(client, version, 2).status_code == 409
|
||||
|
||||
|
||||
def test_fingerprint_recheck_rejects_out_of_band_catalog_changes(client, migrated_database):
|
||||
item = add_item(client)
|
||||
version = build(client)
|
||||
publish(client, version)
|
||||
with migrated_database.begin() as c:
|
||||
c.execute(
|
||||
text("UPDATE knowledge_items SET content=:body WHERE id=:id"),
|
||||
{"body": "直接修改的占位正文", "id": item["id"]},
|
||||
)
|
||||
assert not client.get("/api/indexes/active").json()["is_current"]
|
||||
assert publish(client, version, 1).status_code == 409
|
||||
|
||||
|
||||
@pytest.mark.parametrize("failure", ["chunk_budget", "corpus_budget", "database_write"])
|
||||
def test_failed_build_keeps_previous_publication_and_leaves_no_partial_rows(
|
||||
client, migrated_database, monkeypatch, failure
|
||||
):
|
||||
add_item(client, content="占位正文" * 100)
|
||||
previous = build(client)
|
||||
publish(client, previous)
|
||||
if failure == "chunk_budget":
|
||||
monkeypatch.setattr(indexing, "MAX_CHUNKS", 1)
|
||||
response = client.post("/api/indexes", json={"chunk_size": 100, "overlap": 0})
|
||||
assert response.status_code == 503
|
||||
elif failure == "corpus_budget":
|
||||
from app.services import retrieval
|
||||
|
||||
monkeypatch.setattr(retrieval, "MAX_ITEMS", 0)
|
||||
assert client.post("/api/indexes", json={}).status_code == 503
|
||||
monkeypatch.undo()
|
||||
else:
|
||||
# Real PostgreSQL constraint failure during chunk insert; the new version rolls back too.
|
||||
with migrated_database.begin() as c:
|
||||
c.execute(
|
||||
text("ALTER TABLE chunks ADD CONSTRAINT test_reject_chunk CHECK (false) NOT VALID")
|
||||
)
|
||||
assert client.post("/api/indexes", json={}).status_code == 409
|
||||
with Session(migrated_database) as db:
|
||||
assert db.scalar(select(func.count()).select_from(IndexVersion)) == 1
|
||||
assert db.scalar(select(func.count()).select_from(Chunk)) == previous["chunk_count"]
|
||||
assert db.get(IndexState, 1).active_version_id == previous["id"]
|
||||
assert db.scalar(select(func.count()).select_from(IndexPublication)) == 1
|
||||
assert client.get("/api/indexes/active").json()["active_version"]["id"] == previous["id"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"payload",
|
||||
[
|
||||
{"chunk_size": 99},
|
||||
{"chunk_size": 2001},
|
||||
{"overlap": -1},
|
||||
{"chunk_size": 100, "overlap": 51},
|
||||
{"unexpected": 1},
|
||||
],
|
||||
)
|
||||
def test_build_validation_has_no_side_effects(client, payload):
|
||||
assert client.post("/api/indexes", json=payload).status_code == 422
|
||||
assert client.get("/api/indexes").json()["total"] == 0
|
||||
|
||||
|
||||
def test_index_management_permissions_csrf_and_pagination(client, anonymous_client):
|
||||
# Same TestClient fixture is authenticated as admin here; first verify authorized validation.
|
||||
version = build(client)
|
||||
assert client.post("/api/indexes", json={}, headers={"X-CSRF-Token": ""}).status_code == 403
|
||||
assert client.get("/api/indexes?page_size=101").status_code == 422
|
||||
assert client.get(f"/api/indexes/{uuid4()}").status_code == 404
|
||||
assert client.get(f"/api/indexes/{uuid4()}/chunks").status_code == 404
|
||||
assert client.get("/api/indexes/invalid").status_code == 422
|
||||
assert client.get(f"/api/indexes/{version['id']}/chunks?page=2").json()["items"] == []
|
||||
client.cookies.clear()
|
||||
for method, path in [
|
||||
("GET", "/api/indexes"),
|
||||
("GET", "/api/indexes/active"),
|
||||
("GET", "/api/indexes/publications"),
|
||||
("POST", "/api/indexes"),
|
||||
("GET", f"/api/indexes/{version['id']}/chunks"),
|
||||
("POST", f"/api/indexes/{version['id']}/publish"),
|
||||
]:
|
||||
assert client.request(method, path, json={}).status_code == 401
|
||||
assert (
|
||||
client.post(
|
||||
"/api/auth/register", json={"username": "normal_user", "password": "test-password-123"}
|
||||
).status_code
|
||||
== 201
|
||||
)
|
||||
auth = client.post(
|
||||
"/api/auth/login", json={"username": "normal_user", "password": "test-password-123"}
|
||||
).json()
|
||||
client.headers["X-CSRF-Token"] = auth["csrf_token"]
|
||||
assert client.post("/api/indexes", json={}).status_code == 403
|
||||
assert client.get("/api/indexes/active").status_code == 403
|
||||
|
||||
|
||||
def test_publication_lock_serializes_competing_publishers(client, migrated_database):
|
||||
add_item(client)
|
||||
first, second = build(client), build(client)
|
||||
actor = first["created_by"]
|
||||
with Session(migrated_database) as owner, Session(migrated_database) as competitor:
|
||||
locked_state(owner)
|
||||
competitor.execute(text("SET LOCAL lock_timeout = '100ms'"))
|
||||
with pytest.raises(OperationalError) as blocked:
|
||||
indexing.publish_index(competitor, UUID(second["id"]), 0, actor)
|
||||
assert blocked.value.orig.sqlstate == "55P03"
|
||||
competitor.rollback()
|
||||
assert indexing.publish_index(owner, UUID(first["id"]), 0, actor).generation == 1
|
||||
with pytest.raises(HTTPException) as stale:
|
||||
indexing.publish_index(competitor, UUID(second["id"]), 0, actor)
|
||||
assert stale.value.status_code == 409
|
||||
competitor.rollback()
|
||||
assert client.get("/api/indexes/publications").json()["total"] == 1
|
||||
assert client.get("/api/indexes/active").json()["active_version"]["id"] == first["id"]
|
||||
|
||||
|
||||
def test_catalog_writes_use_the_publication_lock(client, migrated_database):
|
||||
# The guard must run before a catalog mutation, even for creation with no parent row.
|
||||
with Session(migrated_database) as owner:
|
||||
locked_state(owner)
|
||||
from app.db.session import get_db
|
||||
|
||||
factory = client.app.dependency_overrides[get_db]
|
||||
|
||||
def short_timeout():
|
||||
for db in factory():
|
||||
db.execute(text("SET LOCAL lock_timeout = '100ms'"))
|
||||
yield db
|
||||
|
||||
client.app.dependency_overrides[get_db] = short_timeout
|
||||
response = client.post("/api/knowledge", json={"title": "占位", "content": "占位"})
|
||||
assert response.status_code == 503
|
||||
client.app.dependency_overrides[get_db] = factory
|
||||
owner.rollback()
|
||||
assert client.get("/api/knowledge").json() == []
|
||||
|
||||
|
||||
def test_publication_failure_rolls_back_old_status_pointer_and_audit(client, migrated_database):
|
||||
add_item(client)
|
||||
first, second = build(client), build(client)
|
||||
publish(client, first)
|
||||
with migrated_database.begin() as c:
|
||||
c.execute(
|
||||
text(
|
||||
"ALTER TABLE index_publications ADD CONSTRAINT test_reject_publication CHECK (false) NOT VALID"
|
||||
)
|
||||
)
|
||||
assert publish(client, second, 1).status_code == 409
|
||||
assert client.get("/api/indexes/active").json()["active_version"]["id"] == first["id"]
|
||||
assert client.get(f"/api/indexes/{first['id']}").json()["status"] == "published"
|
||||
assert client.get(f"/api/indexes/{second['id']}").json()["status"] == "ready"
|
||||
assert client.get("/api/indexes/publications").json()["total"] == 1
|
||||
@@ -1,100 +0,0 @@
|
||||
import pytest
|
||||
from alembic.autogenerate import compare_metadata
|
||||
from alembic.migration import MigrationContext
|
||||
from conftest import migrate
|
||||
from fastapi.testclient import TestClient
|
||||
from sqlalchemy import text
|
||||
|
||||
from app.db.session import Base
|
||||
from app.main import create_app
|
||||
|
||||
|
||||
def test_startup_requires_migration(database):
|
||||
with pytest.raises(RuntimeError, match="alembic upgrade head"):
|
||||
with TestClient(create_app(database)):
|
||||
pass
|
||||
|
||||
|
||||
def test_startup_rejects_previous_schema(database):
|
||||
migrate(database, "0004_auth_sessions")
|
||||
with pytest.raises(RuntimeError, match="alembic upgrade head"):
|
||||
with TestClient(create_app(database)):
|
||||
pass
|
||||
|
||||
|
||||
def test_upgrade_preserves_original_rows_and_repeated_upgrade(database):
|
||||
migrate(database, "0001_baseline")
|
||||
with database.begin() as connection:
|
||||
connection.execute(
|
||||
text("""INSERT INTO knowledge_items
|
||||
(title, category, content, source_url, verification_status)
|
||||
VALUES ('示例旧数据', '未分类', '原始正文', 'https://example.com/', 'verified')""")
|
||||
)
|
||||
# 模拟 create_all 建表后没有 Alembic 版本记录的旧工程。
|
||||
connection.execute(text("DELETE FROM alembic_version"))
|
||||
migrate(database)
|
||||
migrate(database)
|
||||
with database.connect() as connection:
|
||||
row = connection.execute(text("SELECT * FROM knowledge_items")).mappings().one()
|
||||
assert row["title"] == "示例旧数据" and row["content"] == "原始正文"
|
||||
assert row["collected_at"] is None
|
||||
assert row["verification_status"] == "pending"
|
||||
assert connection.scalar(text("SELECT count(*) FROM audit_events")) == 2
|
||||
assert (
|
||||
connection.scalar(text("SELECT version_num FROM alembic_version"))
|
||||
== "0005_chunk_indexes"
|
||||
)
|
||||
assert compare_metadata(MigrationContext.configure(connection), Base.metadata) == []
|
||||
migrate(database, "0001_baseline", downgrade=True)
|
||||
with database.connect() as connection:
|
||||
assert connection.scalar(text("SELECT content FROM knowledge_items")) == "原始正文"
|
||||
migrate(database)
|
||||
|
||||
|
||||
def test_auth_upgrade_and_downgrade_preserve_knowledge_and_retrieval(database):
|
||||
migrate(database, "0003_retrieval_runs")
|
||||
with database.begin() as c:
|
||||
c.execute(
|
||||
text("""INSERT INTO knowledge_items
|
||||
(title, category, content, verification_status, version, updated_at, is_deleted)
|
||||
VALUES ('占位', '其他', '保留的占位正文', 'pending', 1, now(), false)""")
|
||||
)
|
||||
c.execute(
|
||||
text("""INSERT INTO retrieval_runs
|
||||
(id, query, mode, corpus_fingerprint, response)
|
||||
VALUES ('00000000-0000-0000-0000-000000000001', '占位', 'keyword', 'placeholder', '{}')""")
|
||||
)
|
||||
migrate(database)
|
||||
with database.connect() as c:
|
||||
assert c.scalar(text("SELECT count(*) FROM users")) == 0
|
||||
assert c.scalar(text("SELECT count(*) FROM auth_sessions")) == 0
|
||||
migrate(database, "0003_retrieval_runs", downgrade=True)
|
||||
with database.connect() as c:
|
||||
assert c.scalar(text("SELECT content FROM knowledge_items")) == "保留的占位正文"
|
||||
assert c.scalar(text("SELECT count(*) FROM retrieval_runs")) == 1
|
||||
migrate(database)
|
||||
|
||||
|
||||
def test_chunk_index_downgrade_preserves_users_sessions_knowledge_and_retrieval(
|
||||
client, migrated_database
|
||||
):
|
||||
from test_indexing import build, publish
|
||||
from test_retrieval import add_item, search
|
||||
|
||||
item = add_item(client)
|
||||
run = search(client)
|
||||
version = build(client)
|
||||
assert publish(client, version).status_code == 200
|
||||
migrate(migrated_database, "0004_auth_sessions", downgrade=True)
|
||||
with migrated_database.connect() as c:
|
||||
assert c.scalar(text("SELECT count(*) FROM users")) == 1
|
||||
assert c.scalar(text("SELECT count(*) FROM auth_sessions")) == 1
|
||||
assert (
|
||||
c.scalar(text("SELECT content FROM knowledge_items WHERE id=:id"), {"id": item["id"]})
|
||||
== item["content"]
|
||||
)
|
||||
assert c.scalar(text("SELECT id FROM retrieval_runs")) == run["run_id"]
|
||||
migrate(migrated_database)
|
||||
assert client.get("/api/auth/me").status_code == 200
|
||||
assert client.get("/api/indexes").json()["total"] == 0
|
||||
assert client.get("/api/indexes/active").json()["generation"] == 0
|
||||
@@ -1,212 +0,0 @@
|
||||
import math
|
||||
from collections import Counter
|
||||
from uuid import uuid4
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import text
|
||||
|
||||
from app.services import retrieval
|
||||
|
||||
|
||||
def add_item(client, *, status="verified", **fields):
|
||||
response = client.post(
|
||||
"/api/knowledge",
|
||||
json={
|
||||
"title": "检索占位资料",
|
||||
"content": "那拉提旅游资源检索测试,不代表真实文旅事实。",
|
||||
"source_url": "https://example.com/",
|
||||
"collected_at": "2026-10-15T09:00:00+08:00",
|
||||
**fields,
|
||||
},
|
||||
)
|
||||
assert response.status_code == 201, response.text
|
||||
item = response.json()
|
||||
if status != "pending":
|
||||
response = client.post(
|
||||
f"/api/knowledge/{item['id']}/review",
|
||||
json={
|
||||
"expected_version": item["version"],
|
||||
"verification_status": status,
|
||||
"reviewer": "test-reviewer",
|
||||
"verification_notes": "仅为自动化审核流程测试",
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200, response.text
|
||||
item = response.json()
|
||||
return item
|
||||
|
||||
|
||||
def search(client, query="那拉提", **fields):
|
||||
response = client.post("/api/retrieval", json={"query": query, **fields})
|
||||
assert response.status_code == 200, response.text
|
||||
return response.json()
|
||||
|
||||
|
||||
def test_bm25_matches_hand_calculated_scores_and_query_deduplication():
|
||||
counters = [Counter({"x": 1}), Counter({"x": 1, "y": 1})]
|
||||
scores = retrieval.bm25(counters, ["x"])
|
||||
idf = math.log(1 + 0.5 / 2.5)
|
||||
assert scores[0] == pytest.approx(idf * 2.2 / (1 + 1.2 * (0.25 + 0.75 / 1.5)))
|
||||
assert scores[1] == pytest.approx(idf * 2.2 / (1 + 1.2 * (0.25 + 0.75 * 2 / 1.5)))
|
||||
assert scores[0] > scores[1]
|
||||
assert retrieval.bm25(counters, ["x", "x"]) == scores
|
||||
assert retrieval.bm25([], ["x"]) == []
|
||||
assert retrieval.bm25([Counter()], ["x"]) == [0.0]
|
||||
|
||||
|
||||
def test_tokenization_and_exact_original_snippet_offsets():
|
||||
terms = retrieval.tokenize("请问那拉提 API api!")
|
||||
assert "那拉提" in terms and terms.count("api") == 2
|
||||
assert "请问" not in terms
|
||||
original = "占位前文。" * 80 + "这里有API检索说明。" + "占位后文。" * 80
|
||||
snippet, start, end = retrieval.excerpt(original, ["api"])
|
||||
assert start > 0 and "API" in snippet
|
||||
assert snippet == original[start:end] and len(snippet) <= 240
|
||||
|
||||
|
||||
def test_only_reviewed_active_knowledge_is_retrieved(client):
|
||||
good = add_item(client)
|
||||
add_item(client, status="pending")
|
||||
add_item(client, status="rejected")
|
||||
deleted = add_item(client)
|
||||
assert (
|
||||
client.delete(
|
||||
f"/api/knowledge/{deleted['id']}?expected_version={deleted['version']}"
|
||||
).status_code
|
||||
== 204
|
||||
)
|
||||
result = search(client)
|
||||
assert result["corpus_size"] == 1 and result["total_hits"] == 1
|
||||
assert result["results"][0]["knowledge_id"] == good["id"]
|
||||
assert result["results"][0]["knowledge_version"] == good["version"]
|
||||
assert result["results"][0]["source_url"] == "https://example.com/"
|
||||
assert result["elapsed_ms"] >= 0
|
||||
|
||||
|
||||
def test_tied_results_category_top_k_and_reproducibility(client):
|
||||
first = add_item(client, category="交通出行")
|
||||
second = add_item(client, category="交通出行")
|
||||
third = add_item(client, category="民宿服务")
|
||||
result = search(client, top_k=2)
|
||||
again = search(client, top_k=2)
|
||||
assert result["results"] == again["results"]
|
||||
assert result["corpus_fingerprint"] == again["corpus_fingerprint"]
|
||||
assert result["config"] == again["config"]
|
||||
assert result["run_id"] != again["run_id"]
|
||||
assert result["total_hits"] == 3
|
||||
assert [hit["knowledge_id"] for hit in result["results"]] == [first["id"], second["id"]]
|
||||
filtered = search(client, category="民宿服务")
|
||||
assert filtered["corpus_size"] == 1
|
||||
assert filtered["results"][0]["knowledge_id"] == third["id"]
|
||||
|
||||
|
||||
def test_title_terms_have_more_weight_than_body_terms(client):
|
||||
title_match = add_item(client, title="旅游", content="交通")
|
||||
body_match = add_item(client, title="交通", content="旅游")
|
||||
result = search(client, query="旅游")
|
||||
assert [hit["knowledge_id"] for hit in result["results"]] == [
|
||||
title_match["id"],
|
||||
body_match["id"],
|
||||
]
|
||||
assert result["results"][0]["score"] > result["results"][1]["score"]
|
||||
|
||||
|
||||
def test_no_evidence_and_history_are_persisted(client):
|
||||
empty = search(client)
|
||||
assert empty["status"] == "no_evidence" and empty["results"] == []
|
||||
add_item(client)
|
||||
unmatched = search(client, query="unmatcheduniquelexeme")
|
||||
assert unmatched["status"] == "no_evidence" and unmatched["corpus_size"] == 1
|
||||
assert client.get(f"/api/retrieval/runs/{empty['run_id']}").json() == empty
|
||||
history = client.get("/api/retrieval/runs?page_size=1").json()
|
||||
assert history["total"] == 2 and len(history["items"]) == 1
|
||||
assert history["items"][0] == unmatched
|
||||
assert client.get(f"/api/retrieval/runs/{uuid4()}").status_code == 404
|
||||
assert client.get("/api/retrieval/runs/invalid").status_code == 422
|
||||
assert client.get("/api/retrieval/runs?page_size=101").status_code == 422
|
||||
|
||||
|
||||
def test_source_snapshot_and_original_run_survive_invalidation(client):
|
||||
src = client.post(
|
||||
"/api/sources",
|
||||
json={
|
||||
"name": "测试来源",
|
||||
"source_url": "https://example.com/",
|
||||
},
|
||||
).json()
|
||||
content = "占位前言。" * 80 + "那拉提检索关键词。" + "占位后文。" * 80
|
||||
doc = client.post(
|
||||
"/api/documents",
|
||||
json={
|
||||
"source_id": src["id"],
|
||||
"title": "测试文档",
|
||||
"content": content,
|
||||
"collected_at": "2026-10-15T09:00:00+08:00",
|
||||
},
|
||||
).json()
|
||||
item = add_item(client, document_id=doc["id"], content=content)
|
||||
original = search(client)
|
||||
hit = original["results"][0]
|
||||
assert hit["document_id"] == doc["id"] and hit["document_version"] == 1
|
||||
assert hit["source_id"] == src["id"] and hit["source_version"] == 1
|
||||
assert hit["source_name"] == "测试来源"
|
||||
assert "那拉提" in hit["snippet"] and hit["snippet_start"] > 0
|
||||
assert hit["snippet"] == content[hit["snippet_start"] : hit["snippet_end"]]
|
||||
assert (
|
||||
client.patch(
|
||||
f"/api/sources/{src['id']}",
|
||||
json={
|
||||
"expected_version": src["version"],
|
||||
"name": "修订来源名称",
|
||||
},
|
||||
).status_code
|
||||
== 200
|
||||
)
|
||||
revised = search(client)
|
||||
assert revised["corpus_size"] == 0 and revised["results"] == []
|
||||
assert revised["corpus_fingerprint"] != original["corpus_fingerprint"]
|
||||
stored = client.get(f"/api/retrieval/runs/{original['run_id']}").json()
|
||||
assert stored == original and stored["results"][0]["knowledge_id"] == item["id"]
|
||||
|
||||
|
||||
def test_inactive_parent_is_excluded_even_if_status_was_not_reset(client, migrated_database):
|
||||
src = client.post(
|
||||
"/api/sources", json={"name": "测试", "source_url": "https://example.com/"}
|
||||
).json()
|
||||
add_item(client, source_id=src["id"])
|
||||
assert search(client)["corpus_size"] == 1
|
||||
with migrated_database.begin() as connection:
|
||||
connection.execute(
|
||||
text("UPDATE sources SET is_deleted = true WHERE id = :id"), {"id": src["id"]}
|
||||
)
|
||||
assert search(client)["corpus_size"] == 0
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"payload",
|
||||
[
|
||||
{"query": " "},
|
||||
{"query": "!!!"},
|
||||
{"query": "的了"},
|
||||
{"query": "那拉提", "top_k": 0},
|
||||
{"query": "那拉提", "top_k": 21},
|
||||
{"query": "那拉提", "mode": "vector"},
|
||||
{"query": "那拉提", "unexpected": True},
|
||||
{"query": "a" * 501},
|
||||
],
|
||||
)
|
||||
def test_invalid_query_does_not_create_run(client, payload):
|
||||
assert client.post("/api/retrieval", json=payload).status_code == 422
|
||||
assert client.get("/api/retrieval/runs").json()["total"] == 0
|
||||
|
||||
|
||||
@pytest.mark.parametrize("setting,value", [("MAX_ITEMS", 1), ("MAX_CHARACTERS", 1)])
|
||||
def test_corpus_budget_fails_explicitly_without_silent_truncation(
|
||||
client, monkeypatch, setting, value
|
||||
):
|
||||
add_item(client)
|
||||
add_item(client)
|
||||
monkeypatch.setattr(retrieval, setting, value)
|
||||
response = client.post("/api/retrieval", json={"query": "那拉提"})
|
||||
assert response.status_code == 503 and "容量" in response.json()["detail"]
|
||||
assert client.get("/api/retrieval/runs").json()["total"] == 0
|
||||
@@ -1,192 +0,0 @@
|
||||
"""Run the default application lifecycle over HTTP and verify persistence."""
|
||||
|
||||
import os
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from contextlib import contextmanager
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.models.auth import User
|
||||
from app.services.auth import PASSWORD_HASHER
|
||||
|
||||
|
||||
@contextmanager
|
||||
def server(engine, *, authenticate=True):
|
||||
with socket.socket() as socket_:
|
||||
socket_.bind(("127.0.0.1", 0))
|
||||
port = socket_.getsockname()[1]
|
||||
environment = {
|
||||
**os.environ,
|
||||
"DATABASE_URL": engine.url.render_as_string(hide_password=False),
|
||||
"APP_ENV": "test",
|
||||
"AUTH_COOKIE_SECURE": "false",
|
||||
}
|
||||
if authenticate:
|
||||
with Session(engine) as db:
|
||||
if db.scalar(select(User.id).where(User.username == "http_admin")) is None:
|
||||
db.add(
|
||||
User(
|
||||
username="http_admin",
|
||||
password_hash=PASSWORD_HASHER.hash("test-password-123"),
|
||||
role="admin",
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
backend = Path(__file__).resolve().parents[1]
|
||||
process = subprocess.Popen(
|
||||
[
|
||||
sys.executable,
|
||||
"-m",
|
||||
"uvicorn",
|
||||
"app.main:app",
|
||||
"--host",
|
||||
"127.0.0.1",
|
||||
"--port",
|
||||
str(port),
|
||||
"--log-level",
|
||||
"error",
|
||||
],
|
||||
cwd=backend,
|
||||
env=environment,
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL,
|
||||
)
|
||||
try:
|
||||
with httpx.Client(base_url=f"http://127.0.0.1:{port}", timeout=2) as client:
|
||||
client.headers["Origin"] = "http://localhost:5173"
|
||||
deadline = time.monotonic() + 15
|
||||
while time.monotonic() < deadline:
|
||||
if process.poll() is not None:
|
||||
raise AssertionError("默认后端进程启动失败")
|
||||
try:
|
||||
if client.get("/api/ready").status_code == 200:
|
||||
break
|
||||
except httpx.TransportError:
|
||||
pass
|
||||
time.sleep(0.05)
|
||||
else:
|
||||
raise AssertionError("默认后端未在时限内就绪")
|
||||
if authenticate:
|
||||
login = client.post(
|
||||
"/api/auth/login",
|
||||
json={"username": "http_admin", "password": "test-password-123"},
|
||||
)
|
||||
assert login.status_code == 200, login.text
|
||||
client.headers["X-CSRF-Token"] = login.json()["csrf_token"]
|
||||
yield client
|
||||
finally:
|
||||
process.terminate()
|
||||
try:
|
||||
process.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
process.kill()
|
||||
process.wait(timeout=5)
|
||||
|
||||
|
||||
def test_default_startup_and_persistence_across_restart(migrated_database):
|
||||
with server(migrated_database) as client:
|
||||
assert client.get("/docs").status_code == 200
|
||||
response = client.post(
|
||||
"/api/knowledge",
|
||||
json={
|
||||
"title": "持久化验收占位",
|
||||
"content": "仅为自动化测试数据",
|
||||
"source_url": "",
|
||||
},
|
||||
)
|
||||
assert response.status_code == 201
|
||||
item = response.json()
|
||||
with server(migrated_database) as client:
|
||||
assert client.get(f"/api/knowledge/{item['id']}").json()["content"] == "仅为自动化测试数据"
|
||||
assert client.get("/api/knowledge").json()[0]["id"] == item["id"]
|
||||
|
||||
|
||||
def test_retrieval_run_persists_across_restart(migrated_database):
|
||||
with server(migrated_database) as client:
|
||||
item = client.post(
|
||||
"/api/knowledge",
|
||||
json={
|
||||
"title": "HTTP检索测试",
|
||||
"content": "仅用于自动化测试的那拉提关键词。",
|
||||
"source_url": "https://example.com/",
|
||||
"collected_at": "2026-10-15T09:00:00+08:00",
|
||||
},
|
||||
).json()
|
||||
assert (
|
||||
client.post(
|
||||
f"/api/knowledge/{item['id']}/review",
|
||||
json={
|
||||
"expected_version": item["version"],
|
||||
"verification_status": "verified",
|
||||
"reviewer": "test-reviewer",
|
||||
"verification_notes": "仅验证检索流程",
|
||||
},
|
||||
).status_code
|
||||
== 200
|
||||
)
|
||||
response = client.post("/api/retrieval", json={"query": "那拉提"})
|
||||
assert response.status_code == 200
|
||||
run = response.json()
|
||||
assert run["results"][0]["knowledge_id"] == item["id"]
|
||||
with server(migrated_database) as client:
|
||||
assert client.get(f"/api/retrieval/runs/{run['run_id']}").json() == run
|
||||
|
||||
|
||||
def test_session_and_logout_survive_http_restart(migrated_database):
|
||||
from app.services.auth import COOKIE_NAME
|
||||
|
||||
with server(migrated_database, authenticate=False) as client:
|
||||
payload = {"username": "http_user", "password": "test-password-123"}
|
||||
assert client.post("/api/auth/register", json=payload).status_code == 201
|
||||
response = client.post("/api/auth/login", json=payload)
|
||||
cookie = client.cookies.get(COOKIE_NAME)
|
||||
token = response.json()["csrf_token"]
|
||||
with server(migrated_database, authenticate=False) as client:
|
||||
client.cookies.set(COOKIE_NAME, cookie)
|
||||
assert client.get("/api/auth/me").json()["username"] == "http_user"
|
||||
assert client.post("/api/auth/logout", headers={"X-CSRF-Token": token}).status_code == 204
|
||||
with server(migrated_database, authenticate=False) as client:
|
||||
client.cookies.set(COOKIE_NAME, cookie)
|
||||
assert client.get("/api/auth/me").status_code == 401
|
||||
|
||||
|
||||
def test_published_index_and_chunks_survive_http_restart(migrated_database):
|
||||
with server(migrated_database) as client:
|
||||
item = client.post(
|
||||
"/api/knowledge",
|
||||
json={
|
||||
"title": "切分持久化占位",
|
||||
"content": "那拉提测试正文。" * 50,
|
||||
"source_url": "https://example.com/",
|
||||
"collected_at": "2026-10-16T00:00:00Z",
|
||||
},
|
||||
).json()
|
||||
assert (
|
||||
client.post(
|
||||
f"/api/knowledge/{item['id']}/review",
|
||||
json={
|
||||
"expected_version": item["version"],
|
||||
"verification_status": "verified",
|
||||
"reviewer": "test-reviewer",
|
||||
"verification_notes": "仅为切分持久化测试",
|
||||
},
|
||||
).status_code
|
||||
== 200
|
||||
)
|
||||
built = client.post("/api/indexes", json={"chunk_size": 100, "overlap": 20}).json()
|
||||
response = client.post(
|
||||
f"/api/indexes/{built['id']}/publish", json={"expected_generation": 0}
|
||||
)
|
||||
assert response.status_code == 200
|
||||
active = response.json()
|
||||
chunks = client.get(f"/api/indexes/{built['id']}/chunks").json()
|
||||
with server(migrated_database) as client:
|
||||
assert client.get("/api/indexes/active").json() == active
|
||||
assert client.get(f"/api/indexes/{built['id']}/chunks").json() == chunks
|
||||
assert client.get("/api/indexes/publications").json()["total"] == 1
|
||||
+2
-31
@@ -14,36 +14,7 @@ FastAPI + Pydantic
|
||||
|-- 来源获取 -> 人工核验 -> 文本切分 -> 生成向量 -> 发布版本
|
||||
```
|
||||
|
||||
当前后端实现健康/数据库就绪检查、来源与文档管理、知识增删改查、分页筛选、人工审核、历史快照、认证会话/角色权限及独立关键词检索;问答仍为占位接口。数据库由 Alembic 显式迁移并启用 pgvector,尚无向量字段、向量索引和 RAG。前端来源登记/查询已接入,实验页仍为静态原型,文档、审核、完整分页与检索页面需继续联调。
|
||||
|
||||
## 阶段 2 数据边界
|
||||
|
||||
- `sources` 登记来源名称、HTTP(S) 网址和备注;`documents` 登记所属来源、正文、分类与采集时间。
|
||||
- `knowledge_items` 保留基础版本全部字段,新增可选来源/文档引用、采集时间、人工审核字段、版本和软删除状态。历史条目不强行补造来源或时间。
|
||||
- `audit_events` 在同一事务中保存创建、修改、审核、失效、删除和迁移导入事件;知识快照包含当时关联的来源与文档。
|
||||
- `version` 用于内容并发控制,PATCH/审核/DELETE 必须匹配读取版本;ORM 版本检查处理并发写入冲突,引用操作对来源/文档加行锁,避免与修改和删除交错。
|
||||
- 修改知识或关联依据会撤销当前审核结论。未来正式检索必须同时限定 `verification_status = 'verified'` 和 `is_deleted = false`;当前管理列表不承担检索或索引发布功能。
|
||||
- 文档版本通过递增版本号及完整历史快照留存;`chunks`、Embedding 模型版本与索引发布版本在检索阶段单独实现,不能把记录版本当作已实现索引切换。
|
||||
- 旧库首次升级自动接管基础表,保留条目与导入快照。没有完整审核记录的旧 `verified` 状态回到待审核。启动不自动迁移,缺失或不匹配的版本会明确报错。
|
||||
当前工程仅实现健康检查、知识条目新增/列表及问答占位接口;pgvector 扩展已在启动时启用,但未实现向量字段、索引和 RAG。前端来源管理与实验页为静态原型。
|
||||
|
||||
## 安全边界
|
||||
新增 `users`、`auth_sessions` 与 `auth_rate_limits`。Argon2id 保存密码哈希,随机 Cookie 会话只保存摘要,角色逐次由数据库校验。来源白名单与会话 CSRF 保护写入;改密撤销所有会话。管理资源和检索历史均限管理员,公开检索只返回已核验资料。具体字段、来源、Cookie、预算和迁移见 [认证接口契约](AUTH_API.md)。
|
||||
|
||||
当前仍供本机开发与联调;完整生产部署、网关限流、访问审计、邮箱验证、账号恢复和前端权限页面仍待后续工程验收。
|
||||
|
||||
## 关键词检索基线(T06)
|
||||
|
||||
`POST /api/retrieval` 在同一 SQL 语料快照中筛选有效的已审核知识,读取出处和记录版本,再进行固定中文分词、标题加权 BM25 排序和原文片段截取。语料指纹覆盖本次实际参与排序的内容与出处版本;算法标识和参数独立记录。管理页的字面子串筛选继续用于管理,正式关键词基线使用专用检索接口。
|
||||
|
||||
新增 `retrieval_runs` 保存查询、模式、语料指纹和完整结果快照,支持分页读取及按 UUID 复核。零命中如实记录;当前依据修改后失效不会改写历史,但后续检索必须重新检查审核与父记录状态。响应耗时是本次真实读取、分词、排序及片段生成耗时,不含写日志或模型调用。
|
||||
|
||||
关键词基线仍按知识条目检索,显示片段最长 240 字符;独立的持久化 chunks 和文本索引版本见下方。当前尚无倒排索引、Embedding、向量检索、RRF 或 RAG。全语料计算明确限制条目数及总字符数,超限报错,适用于首批小规模资料与后续对照实验准备,不能当作生产检索性能结论。
|
||||
|
||||
|
||||
## 切分与索引版本(T05)
|
||||
|
||||
`index_versions/chunks` 保存已审核语料的原文切分、出处与版本、参数、内容哈希及语料指纹。构建先保存 ready 快照,不切换当前发布;`index_state` 管理发布指针、代数与目录修订号,发布同一事务更新前后状态并追加 `index_publications`。
|
||||
|
||||
目录写入、构建与发布共用状态行锁,目录写入先于父记录锁获取该锁并递增修订号。发布核对代数、修订号和当前语料指纹;失败回滚,保留旧版本。目录变化后旧指针仍可追溯,但 is_current=false,不可据此绕过正式审核。历史快照不会被改写。接口均限管理员并沿用 Cookie/CSRF,完整边界见 [INDEX_API.md](INDEX_API.md)。
|
||||
|
||||
这只是后续索引的文本版本基础,关键词基线行为不变;没有生成向量或宣称已实现 ANN/混合检索。同步构建和全局写锁面向当前有限语料,后续异步/增量发布须重新设计并验证并发协议。
|
||||
当前无认证与权限控制,仅供本机开发;正式部署前须补充登录鉴权、接口限流、数据库迁移、密钥管理、日志脱敏、CORS 收敛和 HTTPS。
|
||||
|
||||
@@ -1,67 +0,0 @@
|
||||
# 认证与角色权限接口(Issue #5)
|
||||
|
||||
本分支实现 [Issue #5](https://gitea.kronecker.cc/saint/border-red-post-nalati/issues/5) 的七个接口。认证分支 `feat/backend-auth-sessions` 基于 PR #6 的已验证后端提交,迁移依次为 `0002_catalog → 0003_retrieval_runs → 0004_auth_sessions`;PR #6 仍须独立审阅,本分支没有自行合并它。
|
||||
|
||||
## 启动与管理员初始化
|
||||
|
||||
在 `backend/` 安装更新依赖并执行 `alembic upgrade head`,再运行:
|
||||
|
||||
```bash
|
||||
python -m app.manage create-admin --username local_admin
|
||||
```
|
||||
|
||||
命令通过终端隐藏输入和确认密码,不接受密码参数,不创建默认账号;已有用户名不会被升级为管理员或覆盖密码。生产密码请使用独立长口令,不把账号、密码、Cookie 或响应保存到仓库。迁移只新增表,不预置任何用户。
|
||||
|
||||
默认 `AUTH_COOKIE_SECURE=true`,必须通过 HTTPS 使用会话。本机 HTTP 联调时可在未入库的 `backend/.env` 显式设置 `APP_ENV=development` 和 `AUTH_COOKIE_SECURE=false`,保持服务只绑定回环地址。其他环境拒绝关闭 Secure;非开发/测试环境的认证与 CORS Origin 必须均为 HTTPS。不要把本机例外带到部署环境。
|
||||
|
||||
`AUTH_ALLOWED_ORIGINS` 独立控制写请求的来源,默认允许 `localhost/127.0.0.1` 的 5173、8000 端口;`CORS_ORIGINS` 控制前端跨域读取。两者均须填写完整 `scheme://host:port`,不含路径/尾斜杠/通配符。部署时按实际地址同时收敛这两项。代理层仅信任受控代理,不能任意接受外部转发 IP 头。
|
||||
|
||||
## 接口契约
|
||||
|
||||
| 接口 | 请求 | 成功响应与权限 |
|
||||
|---|---|---|
|
||||
| POST `/api/auth/register` | `username/password`,可选 `nickname/email` | 201 用户资料;仅创建 `user`,不自动登录 |
|
||||
| POST `/api/auth/login` | `username/password` | 200 用户资料及 `csrf_token`;设置会话 Cookie |
|
||||
| POST `/api/auth/logout` | 无正文 | 204;要求当前会话和 CSRF,作废该会话并清除 Cookie |
|
||||
| GET `/api/auth/me` | 无 | 200 用户资料及当前 `csrf_token`;要求有效会话 |
|
||||
| PATCH `/api/users/me` | `nickname` 和/或 `email` | 200 当前用户资料;会话与 CSRF 必需 |
|
||||
| POST `/api/auth/change-password` | `current_password/new_password` | 204;验证当前密码后撤销该用户全部会话并清除 Cookie,须重新登录 |
|
||||
| GET `/api/admin/users` | `limit` 默认/最大 100,`offset` 默认 0 | 200 用户数组,按 ID 升序;仅 `admin` |
|
||||
|
||||
所有输入拒绝未知字段。注册和资料修改均不接受 `role/id/password_hash`,不能修改他人资料或用户名。用户名去首尾空白并转小写,须为字母开头的 3–32 位 ASCII 字母、数字或下划线。新密码为 15–128 字符,不修剪、不截断;旧密码只作校验。昵称最长 80 字符且不能含控制字符;邮箱须为合法地址,最长 254 字符。昵称/邮箱的空字符串或 `null` 用于清空;邮箱尚未验证,不能用于登录、找回密码或唯一身份判定。
|
||||
|
||||
用户资料为 `{id, username, nickname, email, role, created_at}`;登录和 `/me` 额外返回 `csrf_token`。响应不包含密码哈希、会话 Cookie 值或内部凭据版本。示例中的 ID、时间和名字均为占位:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": 1,
|
||||
"username": "example_user",
|
||||
"nickname": null,
|
||||
"email": null,
|
||||
"role": "user",
|
||||
"created_at": "2026-10-16T00:00:00Z",
|
||||
"csrf_token": "<由服务端返回的当前会话 CSRF 值>"
|
||||
}
|
||||
```
|
||||
|
||||
错误使用 `detail`:参数 422、用户名冲突 409、登录错误/会话失效 401、权限/来源/CSRF 错误 403、当前密码错误 400、认证请求过多 429(`Retry-After: 300`)。参数错误不回显输入值,避免原样返回密码。认证及其他 `/api/` 响应均使用 `Cache-Control: no-store`。
|
||||
|
||||
## Cookie、CSRF 与授权
|
||||
|
||||
- Cookie 名 `nalati_session`,HttpOnly、Secure(默认)、SameSite=Lax、Path=/、不设置 Domain。服务端生成 256 位随机会话标识;数据库仅保存其 SHA-256 摘要、用户、凭据版本及到期时间。固定有效期 `AUTH_SESSION_SECONDS` 默认 28800 秒,可配置 300–604800 秒,不自动续期。
|
||||
- 登录轮换本浏览器旧会话;退出仅撤销当前会话;改密通过用户行锁、凭据版本递增与删除全部会话使旧凭据立即失效。授权每次从数据库读取当前角色,不信任前端角色或 Cookie 内的角色声明。
|
||||
- 登录/注册即检查精确匹配的 `Origin`,缺失或不可信均拒绝,避免登录 CSRF。已认证的 POST/PATCH/DELETE 等写请求还须带 `X-CSRF-Token`,值来自登录或 `/me`。该值由当前随机 Cookie 经独立 HMAC 派生,不能跨会话复用。后端不从请求正文、URL 或客户端角色获取权限。
|
||||
- `knowledge`、`sources`、`documents` 的全部读写及历史、管理员用户列表、检索历史列表/详情均要求 `admin`。因此旧匿名知识管理调用现在返回 401,普通用户返回 403。健康检查、公开的已核验关键词检索和当前占位问答保持公开;正式问答、实验接口接入时须重新明确各自授权。
|
||||
- 登录及注册共用每客户端地址 30 次/5 分钟预算;登录另有规范用户名 10 次/5 分钟预算,成功/失败都计数。计数通过 PostgreSQL 原子更新,跨进程/重启有效;仅保存带类型前缀的摘要,旧预算在下一次认证请求时清理。此处为基础防暴力尝试,不代替部署网关限流与监控。
|
||||
|
||||
前端 `src/services/auth.ts` 和 `src/services/session.ts` 提供真实 Cookie 请求封装:CSRF 只存内存,刷新后通过 `/me` 恢复;知识写入也使用同一封装。登录入口、路由守卫和权限页面交互仍由前端接入;远程 main 注册原型的最低密码提示须同步为 15 字符。当前 `dev` 页面没有完整登录流程,不能把 API 完成视作浏览器验收完成。
|
||||
|
||||
## 验证与回滚
|
||||
|
||||
在 `backend/` 执行 `ruff check .`、`ruff format --check .`、`pytest -q`;在 `frontend/` 执行 `npm run build`。测试使用隔离 PostgreSQL 和占位账号,覆盖权限拒绝、CSRF、输入越权、哈希、轮换/撤销/过期、改密、重启后会话恢复与退出持久化、预算、迁移及既有检索回归。测试库不代表 Docker 或浏览器验收。
|
||||
|
||||
备份后可配合旧后端执行 `alembic downgrade 0003_retrieval_runs`,删除用户、会话与认证预算表,保留知识和检索历史;账号与哈希恢复依赖备份。降级会撤除授权边界,旧服务只能按其原开发范围使用,不应用于公开部署。继续降至 `0002_catalog` 还会删除检索历史。
|
||||
|
||||
尚未实现邮箱验证、找回密码、MFA、角色管理 API、会话管理页面、访问审计和完整生产部署;没有把这些功能标记为完成。
|
||||
|
||||
实现依据:[argon2-cffi API](https://argon2-cffi.readthedocs.io/en/stable/api.html)、[OWASP 密码存储](https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html)、[OWASP 认证](https://cheatsheetseries.owasp.org/cheatsheets/Authentication_Cheat_Sheet.html)、[OWASP CSRF 防护](https://cheatsheetseries.owasp.org/cheatsheets/Cross-Site_Request_Forgery_Prevention_Cheat_Sheet.html)。本项目使用 Argon2id(19 MiB、2 次迭代、并行度 1);会话与来源策略以本接口契约及代码为准。
|
||||
@@ -14,31 +14,3 @@
|
||||
|
||||
## ADR-005 更新策略
|
||||
状态:设计确定、尚未实现。新文档版本后台生成向量,完成后原子发布;失败保留旧索引。
|
||||
|
||||
## ADR-006 阶段 2 知识管理
|
||||
|
||||
状态:已通过 PR #4 合入 `dev`。采用来源、文档、知识条目和追加式事件日志;记录版本用于并发控制,快照用于追溯。所有删除均为软删除,被有效记录引用的父记录禁止删除。依据或正文修改即回待审核;核验通过必须有来源链接、采集时间、核验人和说明。保留旧版知识创建请求及列表数组结构,新页面使用独立分页接口。当前为本机人工审核流程,认证身份、向量切分和索引发布仍属后续阶段。
|
||||
|
||||
## ADR-007 显式迁移与旧数据保留
|
||||
|
||||
状态:已通过 PR #4 合入 `dev`。以 Alembic 取代应用启动时的 `create_all`。基础迁移接管旧知识表,新增字段和表通过增量迁移完成;不推测旧数据采集时间。旧审核结论缺少完整记录时保留原快照并转待审核。迁移回滚需先备份,新增历史与审核元数据无法通过降级完整保留。
|
||||
|
||||
## ADR-008 可复核的中文关键词基线
|
||||
|
||||
状态:分支实现,待审阅。采用固定 Jieba 搜索分词及关闭 HMM 的词典模式,加入项目名称词,并使用固定停用词表。选择标题重复权重 2、正文权重 1 的 BM25 正值平滑 IDF 变体,不称为 BM25F;查询词去重,同分按知识 ID 排序。参数、分词标识、实际语料指纹及出处版本随每次结果保存,方便冻结后续实验配置。分词接口参考 [Jieba 官方说明](https://github.com/fxsjy/jieba),BM25 基础参考 [Stanford IR 教材](https://nlp.stanford.edu/IR-book/html/htmledition/okapi-bm25-a-non-binary-model-1.html);具体平滑公式与权重以本仓库实现和手算测试为准。
|
||||
|
||||
当前检索单元为已审核知识条目,在设置的容量内按请求计算,不提前实现向量切分/索引更新;超过容量明确失败。`retrieval_runs` 仅记录当时结果,不代表该条目之后仍然获得审核批准。网络搜索、模型选择和真实论文实验保持原阶段边界。
|
||||
|
||||
|
||||
## ADR-009 服务端会话与最小角色边界
|
||||
|
||||
状态:Issue #5 分支实现,待审阅。使用 Argon2id 密码哈希、数据库保存摘要的随机 Cookie 会话,以及独立 CSRF 请求头与显式 Origin 白名单;不把角色或会话凭据放入浏览器持久存储。普通注册只能创建 user,管理员仅由本机隐藏密码输入命令创建。知识/来源/文档管理及检索历史限 admin,公开检索继续只使用已核验依据。完整契约与实现依据见 [AUTH_API.md](AUTH_API.md)。
|
||||
|
||||
为改密和会话撤销使用用户行锁与凭据版本,认证预算使用数据库原子计数;不依赖单进程内存限流。现有前端写请求同步 Cookie/CSRF 封装,页面和路由联调仍待前端完成。本分支叠加 PR #6,后续审阅需关注迁移顺序及匿名管理调用变为 401 的行为变化。
|
||||
|
||||
|
||||
## ADR-010 切分快照与显式发布
|
||||
|
||||
状态:T05 分支实现,待审阅。先对当前有效审核语料构建带原文偏移/哈希/出处的 chunks,再以独立的发布请求切换指针。固定 Unicode 字符窗口配置留存在版本中,关键词接口继续按条目运行,防止未经评估就改变实验口径。
|
||||
|
||||
为当前小规模语料选择统一数据库行锁:应用目录写入、构建、发布串行,发布同时校验代数、目录修订和语料指纹;失败回滚全部更新,旧版本保留为历史。任何成功目录写入保守地使旧索引过期,历史不能视作当前有效依据。实现、容量与后续并发扩展边界见 [INDEX_API.md](INDEX_API.md)。
|
||||
|
||||
@@ -1,36 +0,0 @@
|
||||
# 前端页面与功能设计(阶段一)
|
||||
|
||||
## 设计目标
|
||||
|
||||
前端不是展示虚构文旅结论的门户,而是让团队能看清资料从来源登记、文档采集到知识审核的状态。所有列表均以接口返回为准;接口不可用时展示明确的连接状态,不使用模拟数据兜底。
|
||||
|
||||
## 页面结构
|
||||
|
||||
| 页面 | 已实现 | 下一步功能 | 验收标准 |
|
||||
|---|---|---|---|
|
||||
| 智能问答 | 输入、会话、来源链接、请求中与不可用提示 | 检索命中片段、追问、会话清空 | 每条回答显示后端状态和可打开来源 |
|
||||
| 知识库管理 | 新建条目、搜索、状态统计、来源跳转 | 审核抽屉、编辑、版本冲突提示、历史记录 | 通过审核前必须有来源网址与采集时间 |
|
||||
| 来源与更新 | 登记来源、检索来源、来源状态说明 | 文档列表、关联创建、编辑/软删除、历史记录 | 不自动抓取;来源登记不等同于事实核验 |
|
||||
| 实验评估 | 指标边界与对照方案说明 | 问题集、人工评分、结果导出 | 不展示未经真实实验产生的数值 |
|
||||
|
||||
## 核心操作链路
|
||||
|
||||
1. 资料负责人登记来源:填写名称、网址和可靠性备注。
|
||||
2. 采集文档:必须选定一个来源,记录正文、分类和真实采集时间。
|
||||
3. 前端创建知识条目:关联文档或填写可访问的来源网址,默认状态为 `pending`。
|
||||
4. 审核人核验:填写审核说明;仅在来源网址和采集时间齐全时可标为 `verified`。
|
||||
5. 检索阶段只使用已核验内容;问答页展示来源链接与“原型阶段”提示。
|
||||
|
||||
## 交互与状态规范
|
||||
|
||||
- 提交按钮在请求期间禁用,完成后刷新对应列表。
|
||||
- 接口错误显示后端返回的业务信息;网络错误提示检查 FastAPI 和 PostgreSQL。
|
||||
- 删除采用后端软删除;前端须在删除前显示关联文档/知识条目可能阻止删除的说明。
|
||||
- 并发修改以接口 `version` 为准;返回 409 时提示刷新列表后重试,禁止静默覆盖。
|
||||
- 移动端优先展示表格的标题、状态和主要动作,其他字段通过横向滚动查看。
|
||||
|
||||
## 前端与后端边界
|
||||
|
||||
- 前端不决定资料真实性、不生成审核结论,也不伪造统计结果。
|
||||
- 前端仅调用 `/api/health`、`/api/chat`、`/api/knowledge`、`/api/sources` 和后续 `/api/documents`;鉴权接入后统一由 API 层处理。
|
||||
- 网页抓取、Embedding、向量检索与定时更新属于后端后续阶段,当前页面只说明其规划状态。
|
||||
@@ -1,81 +0,0 @@
|
||||
# 文本切分与索引版本接口(T05)
|
||||
|
||||
本阶段保存可复核的已审核语料切分及发布指针,为后续检索索引提供版本边界。这里的“索引版本”是文本语料快照,不包含 Embedding、pgvector 向量字段/ANN 索引或 RRF。现有 `POST /api/retrieval` 仍运行原关键词基线,不读取这些 chunks;不能把本阶段称为向量检索已完成。
|
||||
|
||||
## 依赖、迁移与权限
|
||||
|
||||
分支 `feat/backend-chunk-index-versions` 叠加待审 PR #7(认证),并继承 PR #6(关键词)。三项须分别跟踪审阅;本分支不自行合并其他请求。升级时停止旧后端实例,安装依赖、备份并执行 `alembic upgrade head`,再启动新实例。所有实例必须使用当前版本,才能让目录写入遵守同一发布锁。
|
||||
|
||||
新增迁移 `0005_chunk_indexes`,依赖 `0004_auth_sessions`,创建:
|
||||
|
||||
- `index_versions`:切分配置、语料指纹、目录修订号、条目/chunk 数、构建人、构建时间及发布状态。
|
||||
- `chunks`:所属版本、知识 ID、从 0 开始的片序、原文起止偏移、正文、UTF-8 内容 SHA-256 与出处快照。
|
||||
- `index_state`:单例发布指针、发布代数 `generation`、目录修订号 `catalog_revision`。
|
||||
- `index_publications`:每次发布的前后版本、代数、操作人和时间;重复发布当前版本不会新增记录。
|
||||
|
||||
迁移不改写既有知识、审核、用户、会话或检索记录,不创建初始发布版本。所有 `/api/indexes` 接口均要求 `admin`;POST 还必须携带有效 Cookie、明确 Origin 与 `X-CSRF-Token`,与 [认证契约](AUTH_API.md) 相同。
|
||||
|
||||
## 构建与原文切分
|
||||
|
||||
`POST /api/indexes` 创建一个 `ready` 版本,返回 201,不切换当前发布指针。请求可用默认值 `{}` 或:
|
||||
|
||||
```json
|
||||
{"chunk_size": 500, "overlap": 80}
|
||||
```
|
||||
|
||||
`chunk_size` 为 100–2000 个 Python Unicode 字符,默认 500;`overlap` 默认 80,非负且不得超过切分长度一半。自定义较短长度时也须指定兼容的重叠,例如 `{"chunk_size":100,"overlap":20}`。未知字段拒绝,不支持按类别单独发布。
|
||||
|
||||
固定算法 `unicode-window-v1` 从知识条目原正文位置 0 开始(不是关联文档或原网页的偏移),以 `chunk_size-overlap` 前进,末片到正文结束即停止。保留空白、全半角、组合字符和 Emoji,不标准化或截断原文。`start_offset` 包含起点,`end_offset` 不包含终点;每片 `text == 原文[start_offset:end_offset]`。标题放在出处信息中,不拼进正文偏移。客户端若使用 JavaScript 字符串偏移,须注意其 UTF-16 索引与 Python Unicode 字符偏移的区别。
|
||||
|
||||
- 语料条件与关键词基线共用:只读取未删除、审核及采集信息完整的 `verified` 知识,关联来源/文档须有效、归属及来源网址一致;pending/rejected 不进入构建。
|
||||
- 语料指纹与关键词基线使用同一规范 JSON/算法,涵盖实际正文、标题、类别、出处及知识/父记录版本;切分参数独立保存在 `config`。
|
||||
- 出处快照包含 `knowledge_id/knowledge_version`、可空 `document_id/document_version`、可空 `source_id/source_version/source_name`、`source_url/title/category`。以后原资料修改或失效,不改写旧 chunk 内容及出处。
|
||||
- chunk ID 在一个版本内由版本 UUID、知识 ID 与顺序确定;新构建使用新版本 ID。相同资料及参数可重现偏移/正文/内容哈希,但新旧版本的 chunk ID 不相同。
|
||||
- 本阶段同步构建,沿用最多 5000 条及标题/正文总计 200 万字符的语料限制,每版本另限 20000 个 chunks。超限返回 503;事务写入失败也完整回滚,不保留半个版本,不改变旧发布指针。
|
||||
- 可以构建空语料版本,以便所有依据撤回后发布空快照;这不表示存在可用知识。
|
||||
|
||||
## 发布、过期与并发
|
||||
|
||||
先通过 `GET /api/indexes/active` 读取当前代数,再提交:
|
||||
|
||||
```text
|
||||
POST /api/indexes/{version_id}/publish
|
||||
```
|
||||
|
||||
```json
|
||||
{"expected_generation": 0}
|
||||
```
|
||||
|
||||
发布时锁定 `index_state`,检查代数、目录修订号及当前有效语料指纹;资料或审核已变化、代数过期均返回 409,要求重新构建或刷新。成功后同一事务把旧版本设为 `superseded`、新版本设为 `published`、切换指针、增加代数并追加发布记录。校验失败或写入失败保留此前全部发布状态。持有当前代数重复发布当前版本是无变化的 200;仍有效的历史版本可用最新代数重新发布,发布记录保留每次转换。
|
||||
|
||||
知识、来源、文档的应用写入在修改父记录前持有同一状态锁,事务内将 `catalog_revision` 增加一次;失败写入一起回滚修订号。构建、发布和这些写入串行,避免校验到提交之间发生目录修改。为首批小规模资料选择此保守策略,长构建会暂时阻塞管理写入;尚无后台任务或大规模增量索引。
|
||||
|
||||
任一成功目录写入都会使旧快照过期,包括尚未核验的新条目;此规则比只比较有效语料更保守。`GET /active` 同时核对修订号与实际语料指纹,返回:
|
||||
|
||||
```json
|
||||
{"generation": 1, "catalog_revision": 2, "is_current": true, "active_version": {"id": "<版本 UUID>", "status": "published", "config": {"splitter":"unicode-window-v1","chunk_size":500,"overlap":80}, "...":"其余版本字段见 OpenAPI"}}
|
||||
```
|
||||
|
||||
初始无发布时 `active_version=null/is_current=false`。资料变化后指针及历史保留,但 `is_current=false`;`published` 只表示仍是发布指针,不表示资料仍有效。历史 chunks 是当时的依据快照,不能绕过 `is_current` 直接用于后续正式检索。任何结果只描述该事务快照,之后的目录写入仍可让它过期。直接改库不属于支持的维护流程;额外指纹检查可发现绕过修订号的正文/出处变化,不能替代应用写锁。
|
||||
|
||||
## 查询与交接
|
||||
|
||||
| 接口 | 响应 |
|
||||
|---|---|
|
||||
| GET `/api/indexes` | 版本分页,构建时间降序,同时间按 ID |
|
||||
| GET `/api/indexes/active` | 发布代数、目录修订号、当前性与当前版本 |
|
||||
| GET `/api/indexes/publications` | 发布记录分页,代数降序 |
|
||||
| GET `/api/indexes/{version_id}` | 版本元数据 |
|
||||
| GET `/api/indexes/{version_id}/chunks` | chunk 分页;可按 `knowledge_id` 筛选,按知识 ID/片序排列 |
|
||||
|
||||
分页沿用 `{items,total,page,page_size}`,默认 20、最大 100。版本元数据完整字段为 `id/status/config/corpus_fingerprint/catalog_revision/item_count/chunk_count/created_by/created_at/published_at`;`published_at` 是最近一次发布的时间,逐次时间保存在发布记录。版本及发布记录时间统一输出 UTC ISO 8601,避免数据库时区让重读结果出现不同表示。chunk 字段为 `id/index_version_id/knowledge_id/ordinal/start_offset/end_offset/text/text_sha256/provenance`。
|
||||
|
||||
前端 `src/services/indexes.ts` 提供类型和调用封装,写入共用 Cookie/CSRF;未增加页面或假数据。接口新增需由前端按该契约接入、审阅并验收。独立 T05 Issue 因当前缺少 Gitea 写入认证仍待补建,本 PR 和任务板先保留实现与验证记录。
|
||||
|
||||
## 验证及回滚
|
||||
|
||||
验证包含真实 PostgreSQL 的原文重建与哈希、审核筛选、重复构建、过期拒绝、来源/文档更新、失败写入、发布历史、数据库锁竞争、失败构建/发布的真实约束回滚、迁移兼容及真实 HTTP 重启后指针/chunks 持久化。这里只使用占位资料,不是论文实验结果。Docker 和浏览器端验收保持待办。
|
||||
|
||||
备份后配合旧后端执行 `alembic downgrade 0004_auth_sessions`,只撤回这四张表,保留用户、会话、知识及检索记录;索引历史与 chunks 恢复依赖备份。旧应用不可使用新版本数据库混跑。
|
||||
|
||||
并发实现参考 [PostgreSQL 16 行锁说明](https://www.postgresql.org/docs/16/explicit-locking.html) 与 [SQLAlchemy FOR UPDATE](https://docs.sqlalchemy.org/en/20/core/selectable.html#sqlalchemy.sql.expression.Select.with_for_update)。
|
||||
@@ -1,77 +0,0 @@
|
||||
# T07:本地 Embedding 与 CPU 资源评估
|
||||
|
||||
本阶段从 `dev` 独立开发,交付可替换 Provider 和可复现的本地 CPU 评估工具。它不依赖待审 PR #6/#7/#8 的接口或数据库迁移;后续向量检索集成再组合这些基础。普通后端启动及普通自动化测试均不导入 PyTorch、不加载模型,也不要求模型下载。
|
||||
|
||||
## 固定配置与边界
|
||||
|
||||
| CLI 名称 | 官方模型 | 固定 revision | 维度 | Token 上限 | 权重格式 |
|
||||
|---|---|---|---:|---:|---|
|
||||
| small | BAAI/bge-small-zh-v1.5 | `7999e1d3359715c523056ef9478215996d62a620` | 512 | 512 | Safetensors |
|
||||
| base | BAAI/bge-base-zh-v1.5 | `f03589ceff5aac7111bd60cfc7d497ca17ecac65` | 768 | 512 | PyTorch,仅权重读取 |
|
||||
|
||||
两者使用 CLS 输出、L2 归一化、CPU FP32;查询添加官方检索指令,正文不添加该指令。`EmbeddingSpec` 冻结模型/revision/维度/token 上限/查询指令/实现版本/权重格式,由规范 JSON 生成 `space_id`。不同空间必须分开保存和查询;目前只提供身份契约,数据库隔离约束尚待向量阶段实现。
|
||||
|
||||
模型仓库均标注 MIT。基础版该修订只提供 `pytorch_model.bin`,显式使用 Transformers 的 `weights_only=True` 读取;不会自动回退未知格式或开启远程模型代码。公开模型下载使用 `token=False`,不要求或读取用户保存的 Hugging Face 凭据。默认只用本机缓存,CLI `--download` 才允许下载固定修订。
|
||||
|
||||
`LocalEmbeddingProvider` 按批处理,默认 batch=8、CPU 线程=2;显式限制 batch 1–32、线程 1–4。输入先按真实 tokenizer 计数,包含查询指令和特殊 token;超过模型上限明确报错,不静默截断。输出检查数量、维度、有限数值和 L2 范数。字符切分长度不等于 token 数,后续接入 chunks 时须据此重新检查。
|
||||
|
||||
## 输入与指标
|
||||
|
||||
项目当前资料台账只有未核验的占位行,没有可用于正式选型的领域数据。本阶段自带 `backend/evaluation/synthetic_zh.json`,含 12 条明确标注“合成测试”的示例文本及 10 个问题,ID/相关文档标注在运行前固定。这些句子不描述那拉提真实事实,不写入应用数据库。
|
||||
|
||||
评估保存输入文件 SHA-256、模型完整配置与空间 ID、运行库版本、实际命中 ID/分数、命中率、Recall@3 和 MRR@3。排序采用归一化向量内积,同分按文档 ID;数值不是正确率或置信度。命中率检查首位相关文档;Recall@k 为前 k 命中相关文档数/标注相关文档总数;MRR@k 为首个相关命中的倒数,未命中为 0。
|
||||
|
||||
每项计时先按同一输入形状预热一次,然后记录 5 次及中位数:12 条语料编码、10 个查询批量编码、单个查询、8 条长文本批量编码。长文本均为相同的合成句子重复形成,360 字符/362 token,仅测试资源。包含分词、补齐、前向、归一化和结果转换;不包含网络、数据库、HTTP 或整个 RAG 延迟。初始化时间单独记录,可能包含缓存检查/下载与首次加载,不用于模型推理速度比较。Linux 峰值 RSS 是整个独立进程高水位,包含解释器、运行库和模型,不能当作仅权重大小。
|
||||
|
||||
## 本机实测
|
||||
|
||||
运行环境:Linux、Python 3.11.17、4 个逻辑 CPU、内存总计约 9.7 GiB,检查时可用约 5.0 GiB;GPU 驱动不可用,本轮仅用 CPU。PyTorch `2.14.1+cpu`、Transformers `5.20.0`、huggingface-hub `2.2.0`、NumPy `2.4.6`;每个模型用独立进程测量,线程 2、batch 8、每项 5 次。未记录用户名、主机名或本机个人路径。
|
||||
|
||||
输入文件 SHA-256:`9f4762a10dc1502b871c84ccedf9fbbdbd93067b8318476a77c78aa14789b243`。
|
||||
|
||||
| 模型 | 12 条正文批编码中位数 | 10 个查询批编码中位数 | 单查询中位数 | 8 条长文批编码中位数 | 峰值进程 RSS | 合成 Hit@1 / Recall@3 / MRR@3 |
|
||||
|---|---:|---:|---:|---:|---:|---|
|
||||
| small | 98.708 ms | 72.877 ms | 8.927 ms | 457.944 ms | 531.2 MiB | 0.90 / 0.90 / 0.90 |
|
||||
| base | 582.315 ms | 416.287 ms | 52.912 ms | 3016.157 ms | 865.5 MiB | 1.00 / 1.00 / 1.00 |
|
||||
|
||||
两个模型在该负载下都能使用本机 CPU。small 在合成问题 q05(出处可靠性)把住宿条目 d07 排在首位,目标资料审核条目 d06 未进入前三;base 将 d06 排首位。这里保留失败结果,不修改数据以提高分数。两个模型的相似度分布不同,不能共用固定分数阈值。
|
||||
|
||||
建议:资源优先的后端开发验证可先使用 small,保留 base 作对照。正式默认模型暂不冻结,先在已核验语料与独立问题集上比较相关性、无依据情形和实际端到端延迟。当前未测 GPU、并发服务、最大 token 负载或完整索引构建。
|
||||
|
||||
这些是合成小样本的功能与资源结果,不是项目正式实验分数。不能据此断言真实文旅问答质量或生产尾延迟。应先用资料负责人核验后的语料及独立问题集复测,再固定默认模型;不能通过修改测试题迎合已观察的排名。
|
||||
|
||||
## 复现(Linux,仓库根目录)
|
||||
|
||||
可使用 `uv` 创建独立 Python 3.11 环境;或以等价 venv/pip 命令安装。依赖独立于 `requirements.txt`,不把 GPU 依赖或模型加入默认 API 启动流程。
|
||||
|
||||
```bash
|
||||
task_eval_root=$(git rev-parse --absolute-git-dir)
|
||||
test -x "$task_eval_root/embedding-eval-venv/bin/python" || \
|
||||
uv venv --python 3.11 "$task_eval_root/embedding-eval-venv"
|
||||
uv pip install --python "$task_eval_root/embedding-eval-venv/bin/python" \
|
||||
--index-url https://download.pytorch.org/whl/cpu torch==2.14.1+cpu
|
||||
uv pip install --python "$task_eval_root/embedding-eval-venv/bin/python" \
|
||||
-r backend/requirements-embedding-cpu.txt
|
||||
export HF_HUB_DISABLE_TELEMETRY=1 HF_HUB_DISABLE_IMPLICIT_TOKEN=1
|
||||
export HF_HUB_DISABLE_PROGRESS_BARS=1 TOKENIZERS_PARALLELISM=false
|
||||
export HF_HUB_DISABLE_XET=1
|
||||
(
|
||||
cd backend
|
||||
"$task_eval_root/embedding-eval-venv/bin/python" -m evaluation.local_embeddings \
|
||||
--model small --download --cache-dir "$task_eval_root/embedding-model-cache" \
|
||||
--output "$task_eval_root/embedding-evaluation/small.json"
|
||||
"$task_eval_root/embedding-eval-venv/bin/python" -m evaluation.local_embeddings \
|
||||
--model base --download --cache-dir "$task_eval_root/embedding-model-cache" \
|
||||
--output "$task_eval_root/embedding-evaluation/base.json"
|
||||
)
|
||||
```
|
||||
|
||||
后续离线复跑去掉 `--download`。可用 `--dataset` 指定同结构数据,文档/问题各最多 1000 条、文件最多 2MB;拒绝重复 ID、未知或重复的相关文档标注。报告不含输入文本,但实际问题/标签可能仍敏感,原始报告、运行输出、环境及模型缓存都应只留在本机 Git 元数据目录,不提交。参数错误或模型失败不会生成伪造的成功结果。
|
||||
|
||||
普通后端验证仍是在 `backend/` 执行 `ruff check .`、`ruff format --check .`、`pytest -q`。轻量测试验证模型身份、非法输出、资源参数、token 拒绝、输入标注及手算排序指标,不需要模型包或联网;真实模型执行由上述独立 CLI 验证。
|
||||
|
||||
## 下一步
|
||||
|
||||
将 Provider 接入已发布的文本版本,完成 pgvector 存储与严格空间隔离、向量检索及 RRF,再使用真实审核语料评测。模型编码耗时应在发布锁外运行,发布前重新核对目录/语料版本;过期或失败时保留旧版本。当前没有新增向量数据库字段、检索 API 或论文结论。
|
||||
|
||||
依据:[BGE small 官方模型说明](https://huggingface.co/BAAI/bge-small-zh-v1.5)、[BGE base 官方文件](https://huggingface.co/BAAI/bge-base-zh-v1.5/tree/f03589ceff5aac7111bd60cfc7d497ca17ecac65)、[Transformers 模型加载参数](https://huggingface.co/docs/transformers/main_classes/model#transformers.PreTrainedModel.from_pretrained)、[PyTorch CPU 安装](https://pytorch.org/get-started/locally/)、[Hugging Face 固定修订下载](https://huggingface.co/docs/huggingface_hub/guides/download)。
|
||||
@@ -1,181 +0,0 @@
|
||||
# 阶段 1 详细开发计划:环境联调与需求冻结
|
||||
|
||||
> 计划周期:2026 年 10 月 16 日—10 月 18 日
|
||||
> 阶段目标:让团队三人都能按 README 启动项目,并完成“前端 → API → PostgreSQL”的一次真实知识条目持久化闭环。
|
||||
> 本阶段不实现 Embedding、向量检索、RAG、联网搜索或真实大模型调用。
|
||||
|
||||
## 1. 阶段完成标准
|
||||
|
||||
全部满足才可将阶段 1 标记为完成:
|
||||
|
||||
1. Docker Desktop 引擎正常,`docker version` 同时显示 Client 和 Server;`nalati-postgres` 容器健康运行。
|
||||
2. 后端可通过默认生命周期启动,`http://127.0.0.1:8000/docs` 可访问,`GET /api/health` 返回 200。
|
||||
3. `POST /api/knowledge` 新增一条测试知识,`GET /api/knowledge` 能读取该条目;重启后端后数据仍存在。
|
||||
4. 前端 `npm run build` 通过,`http://localhost:5173` 可访问,页面显示 API 已连接。
|
||||
5. 在浏览器完成“新增知识 → 保存 → 列表刷新 → 重新打开页面仍存在”的完整流程。
|
||||
6. 三人分别在 Gitea Issue 中留下命令、结果、截图或日志位置;未提交真实密钥、真实密码和未授权资料。
|
||||
|
||||
## 2. 当前基线与阻塞
|
||||
|
||||
| 项目 | 当前状态 | 负责人 | 下一步 |
|
||||
|---|---|---|---|
|
||||
| Git 仓库、`main`、`dev` | 已准备 | 全员 | 每次工作从最新 `dev` 拉分支 |
|
||||
| 前端依赖与生产构建 | 已通过 | 吉海燕 | 启动 Vite,做浏览器端验收 |
|
||||
| 后端虚拟环境与依赖 | 已准备 | 范涵宇 | 数据库就绪后按默认生命周期启动 |
|
||||
| `/api/health` 路由 | 已验证 | 范涵宇 | 数据库就绪后复验默认启动 |
|
||||
| Docker Desktop/WSL | 受阻 | 范涵宇 | 在管理员 PowerShell 完成 WSL 前置、重启后启动 Docker Desktop |
|
||||
| 资料台账、审核规则 | 待创建 | 杨星萱 | 建立模板并明确状态含义 |
|
||||
|
||||
> Docker/WSL 未恢复前,可以完成前端构建、资料模板和文档工作;不得把数据库、新增知识持久化或“API 已连接”写成已通过。
|
||||
|
||||
## 3. 每日安排
|
||||
|
||||
### 10 月 16 日:环境恢复与单服务启动
|
||||
|
||||
| 成员 | 必做工作 | 交付物 | 验收证据 |
|
||||
|---|---|---|---|
|
||||
| 范涵宇 | 处理 Docker Desktop/WSL;启动 PostgreSQL;检查后端 `.env` 和 Python 环境 | 运行中的数据库容器、后端启动记录 | `docker compose ps`、`docker logs nalati-postgres`、后端启动日志 |
|
||||
| 吉海燕 | 从 `dev` 拉前端分支;执行依赖安装、构建和 Vite 启动;检查四个页面切换 | 前端启动记录、四页初始截图 | `npm run build` 输出、浏览器截图 |
|
||||
| 杨星萱 | 建立资料台账字段和审核规则;建立联调验收记录模板 | 台账模板、审核规则、验收记录模板 | 文档提交与 Issue 链接 |
|
||||
|
||||
### 10 月 17 日:API 与浏览器端持久化闭环
|
||||
|
||||
| 成员 | 必做工作 | 交付物 | 验收证据 |
|
||||
|---|---|---|---|
|
||||
| 范涵宇 | 复验 `/docs`、健康检查、知识新增/读取;记录 API 请求和响应 | T02/T03 验收记录 | 201 创建响应、列表读取响应、数据库容器日志 |
|
||||
| 吉海燕 | 按后端真实 Schema 调用新增与列表 API;处理加载、保存失败、API 未连接状态 | 前端联调截图或缺陷清单 | 新增后列表可见、刷新后仍存在的截图 |
|
||||
| 杨星萱 | 用非敏感测试资料完成一条待审核知识的登记;核对前端字段是否满足资料追溯需要 | 测试条目内容、字段反馈 | 测试条目 ID、字段核对记录 |
|
||||
|
||||
### 10 月 18 日:复现验收、问题冻结与交接
|
||||
|
||||
| 成员 | 必做工作 | 交付物 | 验收证据 |
|
||||
|---|---|---|---|
|
||||
| 范涵宇 | 重启后端或容器后复验数据;汇总未解决环境/接口问题 | 后端验收结论、遗留问题列表 | 重启后 `GET /api/knowledge` 结果 |
|
||||
| 吉海燕 | 在干净浏览器会话复验问答页、知识管理页和错误状态;整理可用于软著的阶段截图 | 前端验收结论、截图清单 | 浏览器操作步骤和截图 |
|
||||
| 杨星萱 | 汇总资料台账、审核规则与测试记录;确认阶段 2 所需字段和资料数量 | 阶段 2 知识收集清单 | Issue 中的验收记录与待办 |
|
||||
|
||||
## 4. 范涵宇执行清单:后端、数据库
|
||||
|
||||
### 4.1 Docker 与 PostgreSQL
|
||||
|
||||
在管理员完成 WSL/重启后,先执行:
|
||||
|
||||
```powershell
|
||||
cd E:\边疆红驿\border-red-post-nalati-ready
|
||||
docker version
|
||||
docker compose up -d db
|
||||
docker compose ps
|
||||
docker logs nalati-postgres
|
||||
```
|
||||
|
||||
通过条件:`docker version` 存在 Server 信息;`docker compose ps` 显示 `nalati-postgres` 运行且健康检查通过。若失败,只在 T03 Issue 记录完整错误,不删除 Docker 卷或重置数据。
|
||||
|
||||
### 4.2 后端启动与接口验证
|
||||
|
||||
```powershell
|
||||
cd E:\边疆红驿\border-red-post-nalati-ready\backend
|
||||
if (-not (Test-Path .env)) { Copy-Item .env.example .env }
|
||||
.\.venv\Scripts\Activate.ps1
|
||||
uvicorn app.main:app --reload
|
||||
```
|
||||
|
||||
另开 PowerShell 验证:
|
||||
|
||||
```powershell
|
||||
Invoke-RestMethod http://127.0.0.1:8000/api/health
|
||||
Invoke-RestMethod http://127.0.0.1:8000/api/knowledge
|
||||
$body = @{ title = '阶段1联调测试条目'; category = '其他'; content = '仅用于验证知识新增与持久化,不是正式事实资料。' } | ConvertTo-Json
|
||||
Invoke-RestMethod http://127.0.0.1:8000/api/knowledge -Method Post -ContentType 'application/json' -Body $body
|
||||
```
|
||||
|
||||
通过条件:健康检查返回 `status: ok`;新增请求返回 201;列表能看到返回的 ID 和标题。将测试条目保留并标注为测试,不作为正式检索资料。
|
||||
|
||||
### 4.3 给前端的交接内容
|
||||
|
||||
在 T02/T03 Issue 粘贴以下内容后再通知吉海燕联调:
|
||||
|
||||
- 后端地址、启动命令和 `/docs` 地址;
|
||||
- 最终确认的知识条目请求/响应 JSON;
|
||||
- 可用的 `verification_status` 状态值;
|
||||
- 当前已知限制与错误响应格式。
|
||||
|
||||
## 5. 吉海燕执行清单:前端与浏览器联调
|
||||
|
||||
### 5.1 拉取、构建与启动
|
||||
|
||||
```powershell
|
||||
cd E:\边疆红驿\border-red-post-nalati-ready
|
||||
git switch dev
|
||||
git pull --ff-only origin dev
|
||||
git switch -c feat/frontend-phase1-validation
|
||||
cd frontend
|
||||
npm ci
|
||||
npm run build
|
||||
npm run dev
|
||||
```
|
||||
|
||||
通过条件:构建命令退出码为 0;Vite 显示本地访问地址;浏览器可打开四个导航页。
|
||||
|
||||
### 5.2 浏览器检查顺序
|
||||
|
||||
1. 打开 `http://localhost:5173`,检查顶部/侧边导航能切换“智能问答、知识库管理、来源与更新、实验评估”。
|
||||
2. 后端未启动时,确认界面显示“API 未连接”或明确错误提示;不要修改前端来伪造连接成功。
|
||||
3. 后端启动后刷新页面,确认状态变为“API 已连接”。
|
||||
4. 在“知识库管理”点击“新增知识”,填写后端给出的测试条目,保存并确认列表出现该条目。
|
||||
5. 刷新浏览器页面,确认条目仍然存在;记录浏览器、时间和截图。
|
||||
6. 在“智能问答”发送一个测试问题,确认基础占位回答或后端错误能在页面被清楚展示,不白屏、不无限加载。
|
||||
|
||||
### 5.3 前端缺陷处理边界
|
||||
|
||||
- 请求、响应类型和错误信息统一维护在 `frontend/src/api.ts`;页面中不复制 `fetch` 逻辑。
|
||||
- 只按范涵宇确认的 API 字段改动;发现字段不一致时开 Issue,不在前端猜测字段或拼接数据库连接。
|
||||
- 每个修复独立提交,提交信息如 `fix: handle knowledge api error`;合并请求附构建结果与前后截图。
|
||||
- 当前阶段不实现检索模式选择、LLM 配置、真实实验图表或自动抓取来源,它们属于后续阶段。
|
||||
|
||||
## 6. 杨星萱执行清单:资料、审核与验收记录
|
||||
|
||||
### 6.1 资料台账字段
|
||||
|
||||
在 `docs/` 下的个人工作分支建立台账模板(Markdown、CSV 或学校统一表格均可),每条候选资料至少填写:
|
||||
|
||||
| 字段 | 要求 |
|
||||
|---|---|
|
||||
| 标题、类别 | 可识别且分类明确,如红色文化、旅游资源、民宿服务、交通出行 |
|
||||
| 来源名称、URL | 可回访来源;无 URL 时注明线下材料出处 |
|
||||
| 获取日期、更新时间 | 使用 `YYYY-MM-DD` |
|
||||
| 正文摘要 | 保留原意,不编造无法核实的具体事实 |
|
||||
| 核验状态 | `pending`、`verified` 或 `rejected` |
|
||||
| 核验依据、核验人、核验日期 | 说明为什么可以使用或拒绝使用 |
|
||||
|
||||
### 6.2 审核与测试资料规则
|
||||
|
||||
1. `pending`:已收集但未完成核验;可用于界面演示,不得进入正式检索。
|
||||
2. `verified`:来源可定位、信息完整且已人工核验;可交给范涵宇入库。
|
||||
3. `rejected`:来源不明、过期、重复或不可靠;保留记录但不入库。
|
||||
4. 阶段 1 的测试条目必须写明“测试”,不得伪装为那拉提真实资料。
|
||||
|
||||
### 6.3 联调验收记录
|
||||
|
||||
每次测试至少记录:测试日期、测试人、Git 提交、运行环境、操作步骤、预期结果、实际结果、截图/日志、结论和待处理问题。阶段 1 最少记录一次“知识新增—读取—页面刷新”测试。
|
||||
|
||||
## 7. Issue、提交与审阅顺序
|
||||
|
||||
| 任务 | 建议分支 | 提交/合并目标 | 审阅与批准 |
|
||||
|---|---|---|---|
|
||||
| T01 前端启动与浏览器验收 | `feat/frontend-phase1-validation` | 合并请求 → `dev` | 吉海燕提交;Codex 审阅并明确回复“批准合并”后由吉海燕合并 |
|
||||
| T02 健康检查与后端启动 | `feat/backend-phase1-api-validation` | 合并请求 → `dev` | 范涵宇提交;吉海燕审阅批准后合并 |
|
||||
| T03 Docker 与知识持久化 | `feat/backend-phase1-database` | 合并请求 → `dev` | 范涵宇提交;吉海燕审阅批准后合并 |
|
||||
| T04 审核规则与验收模板 | `docs/phase1-knowledge-governance` | 合并请求 → `dev` | 杨星萱提交;吉海燕审阅批准后合并 |
|
||||
|
||||
阶段 1 的 `dev → main` 合并请求只能在本文件第 1 节全部满足后创建,由吉海燕发起并确认。若 Docker/WSL 仍受阻,保留阶段状态为“进行中”,先记录阻塞和恢复动作,不跳过持久化验收。
|
||||
|
||||
## 8. 阶段结束交接包
|
||||
|
||||
阶段 1 结束时,三人共同整理以下内容到 Issue 或 `docs/`:
|
||||
|
||||
- 经过验证的启动命令与版本信息;
|
||||
- 前端四页和知识新增闭环截图;
|
||||
- API 请求/响应样例与已知限制;
|
||||
- 测试资料与正式资料的区分说明;
|
||||
- 未解决问题清单、负责人和预计恢复日期;
|
||||
- 阶段 2 可直接开始的接口字段与资料台账模板。
|
||||
@@ -22,51 +22,3 @@
|
||||
- 已使用临时 Uvicorn 服务(关闭数据库生命周期初始化)验证 `GET /api/health` 返回 `200` 和预期状态数据,验证后服务已关闭。
|
||||
- 本机 Docker CLI 已安装,但 Docker Desktop 引擎无法启动,且 WSL 尚未处于可用状态。因此尚未执行 PostgreSQL 容器启动、知识条目新增/读取、默认生命周期启动和浏览器端联调。
|
||||
- 后续操作:重启 Windows 后启动 Docker Desktop;确认 `docker version` 能显示 Server 信息,再在项目根目录运行 `docker compose up -d db`,随后继续 T02、T03 和 T01 的剩余验收。
|
||||
|
||||
## 2026-10-15 后端阶段 1–2 分支验收
|
||||
|
||||
分支:`feat/backend-knowledge-management`。以下为分支验收结果,须经既定审阅流程合入后再关闭任务;不覆盖前述其他环境的历史记录。
|
||||
|
||||
| 范围 | 已完成的验证 | 剩余事项 |
|
||||
|---|---|---|
|
||||
| T02 | 默认生命周期真实启动 Uvicorn;HTTP `/docs`、健康检查、数据库就绪检查通过 | 合并请求审阅、团队环境复核 |
|
||||
| T03 | 真实 PostgreSQL 16 测试实例完成迁移、知识录入和读取;后端进程重启后数据仍可读取 | 当前主机 Docker 守护进程未运行,启动需管理员权限;Compose 容器与浏览器端联调待复核 |
|
||||
| T05(阶段 2 范围) | 来源、文档、知识关联、版本、人工审核、软删除和历史快照;验证旧基础表迁移保留数据 | `chunks` 和索引版本留到检索阶段;字段与前端、资料负责人共同确认 |
|
||||
| T10(部分) | Alembic 迁移、并发版本检查、PostgreSQL 自动化测试、启动说明 | 登录鉴权、限流、生产部署留到后续工程阶段 |
|
||||
|
||||
验证环境:Python 3.11;使用可选 `pgserver` 启动独立 PostgreSQL 16.2 测试实例并启用 pgvector。该方式用于验证 SQL 和接口,不等同于 Docker 容器验收。
|
||||
|
||||
实际命令与结果(在 `backend/` 执行):
|
||||
|
||||
- `ruff check .`:通过。
|
||||
- `ruff format --check .`:通过。
|
||||
- `pytest -q`:22 passed;依赖库产生一条 TestClient 弃用提示,不影响本轮用例。
|
||||
- 覆盖:新库迁移、已有基础表接管、重复升级、阶段 2 降级后基础正文保留、ORM 与迁移结构一致、审核前置条件、正文/来源/文档变更后重新审核、历史快照、软删除引用保护、筛选与分页、并发写入冲突、跨域预检、真实 HTTP 与服务重启持久化。
|
||||
|
||||
所有测试正文、来源和审核人均为占位数据,临时数据库不入库。当前问答仍是占位接口,未开展论文对照实验。Issue、合并请求和最终批准结论须补入 Gitea 后按团队规则归档。
|
||||
|
||||
## 2026-10-15 合并后续开发:关键词检索基线
|
||||
|
||||
- 阶段 2 后端功能已通过普通功能分支的 [PR #4](https://gitea.kronecker.cc/saint/border-red-post-nalati/pulls/4) 合入 `dev`(合并提交 `4d437ca`),包含测试客户端依赖修正。旧 AGit PR #2 仍打开,不作为该批功能是否已合入的唯一判据。
|
||||
- 从最新 `dev` 创建 `feat/backend-keyword-retrieval`,推进 T06 的独立关键词接口与检索记录。T05 的持久化切分/索引版本、T07 的模型评估和向量检索仍待后续实现。
|
||||
- 当前主机 Docker 服务仍未运行;本轮继续使用隔离 PostgreSQL 做接口与迁移测试,不把容器或浏览器端联调标记为完成。独立任务 Issue 仍须补齐。
|
||||
- 新增检索接口、中文分词和 BM25 基线、有效审核筛选、原文片段、来源与版本、语料指纹及检索历史;前端调用层同步类型与接口,页面接入由前端继续联调。
|
||||
- 分支验证:`ruff check .`、`ruff format --check .`、`pytest -q`;本轮 42 项 PostgreSQL/单元/真实 HTTP 测试通过,覆盖手算分数、标题权重、排序复现、审核边界、来源失效、容量拒绝、迁移兼容和服务重启后的检索历史。前端 `npm run build` 通过,包含新增检索调用类型的检查。全部使用占位资料,不是论文实验结果。
|
||||
|
||||
## 2026-10-16 Issue #5:认证与角色权限
|
||||
|
||||
- 按继续开发的授权,在 `feat/backend-auth-sessions` 推进 T10 的认证部分;叠加 PR #6(`8ac9dca`),不等待其合并,不自行批准或合并。PR #6 与本分支分别保持可追踪记录。
|
||||
- 实现注册、登录、退出、当前会话、本人资料修改、改密及管理员用户列表;注册拒绝角色字段,知识/来源/文档管理及检索历史均由服务端要求 admin。
|
||||
- 新增 `0004_auth_sessions`,保存 Argon2id 密码哈希、会话摘要与到期/凭据版本、共享认证请求预算;不预置用户,提供本机隐藏密码输入的管理员初始化命令。迁移和回滚保留知识及检索记录。
|
||||
- 默认 Secure/HttpOnly/SameSite Cookie;写操作校验明确 Origin 及当前会话 CSRF。改密使所有旧会话失效,退出持久化撤销;参数错误不回显密码。字段、来源、CSRF 和前端交接见 [认证接口契约](AUTH_API.md)。
|
||||
- 实际验证:`ruff check .`、`ruff format --check .` 通过;`pytest -q` **70 passed**,包含认证、授权、CSRF、输入越权、过期/轮换/撤销、限额、迁移及既有知识/检索回归;真实 HTTP 服务重启验证会话恢复和退出后不再有效。前端 `npm run build` 通过。
|
||||
- 当前仍使用隔离 PostgreSQL 做自动化验证;Docker 容器、前端登录/角色页面及真实浏览器端流程仍待验收。远程 main 的注册原型最低密码提示需要按本契约同步到 15 字符,并接入带 Cookie/CSRF 的调用层。没有宣称已完成前端集成或生产部署。
|
||||
|
||||
## 2026-10-16 T05:持久化 chunks 与索引版本
|
||||
|
||||
- 分支 `feat/backend-chunk-index-versions` 叠加待审 PR #7(`f9827fe`),继承 #6;不等待合并继续后端阶段,同时保留各 PR 的审阅记录。
|
||||
- 新增 `0005_chunk_indexes`:版本、原文切分及出处、发布状态单例、逐次发布记录。构建 ready 快照与发布分开,保留字符偏移、内容哈希、知识/来源/文档版本和语料指纹。
|
||||
- 知识、来源、文档写入与构建/发布共用数据库行锁;目录修订与实际语料指纹一起决定当前性,旧构建/旧代数拒绝发布。构建或发布失败完整回滚,旧发布指针及历史不丢失。
|
||||
- 管理接口统一要求 admin 与写入 CSRF;前端只同步类型和调用层,页面仍待联调。现有关键词基线行为不变,向量、ANN、RRF 与 RAG 未在本阶段实现。
|
||||
- 实际验证:`ruff check .`、`ruff format --check .` 通过;`pytest -q` **91 passed**(隔离 PostgreSQL 16 + pgvector),包含真实锁竞争、失败构建/发布的数据库约束回滚、迁移保留账号/知识/检索记录、原文重建与来源快照、真实 HTTP 重启后的发布指针和 chunks 持久化。修正了发布时间在 UTC 与数据库时区之间表示不一致的问题,索引接口统一 UTC 输出。前端 `npm run build` 通过。
|
||||
- 完整请求/响应、切分/容量、失效策略及回滚说明见 [INDEX_API.md](INDEX_API.md)。本阶段仍是有限语料的同步文本快照;Docker/浏览器验收、独立 T05 Issue 补建及后续 Embedding 方案选择仍有待完成。
|
||||
|
||||
+8
-14
@@ -1,26 +1,20 @@
|
||||
<script setup lang="ts">
|
||||
import { computed, onMounted, ref, watch } from 'vue'
|
||||
import { computed, onMounted, ref } from 'vue'
|
||||
import { Activity, BookOpen, Bot, CheckCircle2, ChevronRight, Database, FileText, Globe2, Link2, MessageSquareText, Plus, Search, Send, ShieldCheck, Sparkles, AlertCircle } from 'lucide-vue-next'
|
||||
import { createKnowledge, createSource, getHealth, getKnowledge, getSources, retrieveKnowledge, retrieveVectorKnowledge, type SourceItem } from './api'
|
||||
import { askQuestion, createKnowledge, getHealth, getKnowledge } from './api'
|
||||
type Page='chat'|'knowledge'|'sources'|'evaluation'
|
||||
const page=ref<Page>('chat'), apiStatus=ref('检测中'), question=ref(''), busy=ref(false), retrievalMode=ref<'keyword'|'vector'>('keyword'), search=ref(''), showForm=ref(false), error=ref(''), notice=ref('')
|
||||
const page=ref<Page>('chat'), apiStatus=ref('检测中'), question=ref(''), busy=ref(false), search=ref(''), showForm=ref(false), error=ref(''), notice=ref('')
|
||||
const messages=ref([{role:'assistant',content:'你好!我是边疆红驿文旅问答原型。当前后端仅为基础接口,尚未接入知识检索和大语言模型。'}] as {role:string;content:string;sources?:{title:string;source_url?:string|null}[]}[])
|
||||
const items=ref<any[]>([]), form=ref({title:'',category:'旅游资源',content:'',source_url:''})
|
||||
const sources=ref<SourceItem[]>([]), sourceSearch=ref(''), showSourceForm=ref(false), sourceForm=ref({name:'',source_url:'',notes:''})
|
||||
const nav=[{id:'chat',label:'智能问答',icon:MessageSquareText},{id:'knowledge',label:'知识库管理',icon:Database},{id:'sources',label:'来源与更新',icon:Globe2},{id:'evaluation',label:'实验评估',icon:Activity}] as const
|
||||
const filtered=computed(()=>items.value.filter(x=>`${x.title} ${x.category} ${x.content}`.toLowerCase().includes(search.value.toLowerCase())))
|
||||
async function refresh(){try{items.value=await getKnowledge();error.value=''}catch{error.value='无法读取后端知识库,请确认 API 和 PostgreSQL 已启动。'}}
|
||||
async function refreshSources(){try{sources.value=(await getSources(sourceSearch.value)).items;error.value=''}catch{error.value='无法读取来源列表,请确认 API 和 PostgreSQL 已启动。'}}
|
||||
async function saveSource(){error.value='';notice.value='';if(!sourceForm.value.name.trim()){error.value='请填写来源名称。';return}if(!validUrl(sourceForm.value.source_url)){error.value='来源网址必须是有效的 http(s) 地址。';return}try{await createSource({...sourceForm.value,name:sourceForm.value.name.trim(),...(sourceForm.value.source_url?{source_url:sourceForm.value.source_url}:{})});sourceForm.value={name:'',source_url:'',notes:''};showSourceForm.value=false;notice.value='来源已登记,可继续创建关联文档和知识条目。';await refreshSources()}catch(e){error.value=e instanceof Error?e.message:'保存失败'}}
|
||||
let sourceSearchTimer:number|undefined
|
||||
watch(sourceSearch,()=>{window.clearTimeout(sourceSearchTimer);sourceSearchTimer=window.setTimeout(refreshSources,260)})
|
||||
onMounted(async()=>{try{await getHealth();apiStatus.value='已连接'}catch{apiStatus.value='未连接'}await Promise.all([refresh(),refreshSources()])})
|
||||
async function send(){const q=question.value.trim();if(!q||busy.value)return;messages.value.push({role:'user',content:q});question.value='';busy.value=true;try{const r=retrievalMode.value==='vector'?await retrieveVectorKnowledge({query:q,top_k:5}):await retrieveKnowledge({query:q,top_k:5});const content=r.status==='no_evidence'?'未检索到可用的已核验知识依据。请调整问题或补充审核后的资料。':`已通过${r.mode==='vector'?'向量':'关键词'}检索命中 ${r.total_hits} 条已核验依据(${r.elapsed_ms.toFixed(1)} ms)。当前仅展示检索证据,尚未接入大语言模型生成回答。`;messages.value.push({role:'assistant',content,sources:r.results.map(hit=>({title:`${hit.title}:${hit.snippet}`,source_url:hit.source_url}))})}catch(e){messages.value.push({role:'assistant',content:e instanceof Error?e.message:'暂时无法连接检索服务。'})}finally{busy.value=false}}
|
||||
function validUrl(value:string){try{if(value)new URL(value);return true}catch{return false}}
|
||||
async function save(){error.value='';notice.value='';if(!form.value.title.trim()||!form.value.content.trim()){error.value='请填写知识标题和正文。';return}if(!validUrl(form.value.source_url)){error.value='来源网址必须是有效的 http(s) 地址。';return}try{await createKnowledge({...form.value,title:form.value.title.trim(),content:form.value.content.trim(),...(form.value.source_url?{source_url:form.value.source_url}:{})});form.value={title:'',category:'旅游资源',content:'',source_url:''};showForm.value=false;notice.value='已保存为待审核条目,尚未进入检索索引。';await refresh()}catch(e){error.value=e instanceof Error?e.message:'保存失败'}}
|
||||
onMounted(async()=>{try{await getHealth();apiStatus.value='已连接'}catch{apiStatus.value='未连接'}await refresh()})
|
||||
async function send(){const q=question.value.trim();if(!q||busy.value)return;messages.value.push({role:'user',content:q});question.value='';busy.value=true;try{const r=await askQuestion(q);messages.value.push({role:'assistant',content:r.answer,sources:r.sources})}catch{messages.value.push({role:'assistant',content:'暂时无法连接后端 API。请确认 FastAPI 已启动。'})}finally{busy.value=false}}
|
||||
async function save(){error.value='';notice.value='';try{await createKnowledge({...form.value,...(form.value.source_url?{source_url:form.value.source_url}:{})});form.value={title:'',category:'旅游资源',content:'',source_url:''};showForm.value=false;notice.value='已保存为待审核条目,尚未进入检索索引。';await refresh()}catch(e){error.value=e instanceof Error?e.message:'保存失败'}}
|
||||
</script>
|
||||
<template><div class="shell"><aside class="sidebar"><div class="brand"><div class="mark"><Bot :size="23"/></div><div><b>边疆红驿</b><small>NALATI KNOWLEDGE HUB</small></div></div><p class="muted label">功能列表</p><nav><button v-for="n in nav" :key="n.id" :class="{active:page===n.id}" @click="page=n.id"><component :is="n.icon" :size="18"/>{{n.label}}<ChevronRight v-if="page===n.id" class="chev" :size="15"/></button></nav><div class="quick-actions"><p class="muted label">快捷操作</p><button v-if="page==='knowledge'" class="side-action" @click="showForm=true"><Plus :size="16"/>新增知识</button><button v-if="page==='sources'" class="side-action" @click="showSourceForm=true"><Plus :size="16"/>新增来源</button><p v-if="page==='chat'" class="side-hint">选择示例问题,或直接开始提问。</p><p v-if="page==='evaluation'" class="side-hint">实验结果只展示真实运行数据。</p></div><div class="side-bottom"><div class="tip"><ShieldCheck :size="17"/><div><b>可信知识服务</b><p>优先展示可核验来源,信息不足时明确提示。</p></div></div><p class="api">● API {{apiStatus}} <span>v0.1.0</span></p></div></aside><main><header><span>边疆红驿 / {{nav.find(n=>n.id===page)?.label}}</span><div><span class="pill">● 原型阶段</span><i>研</i></div></header><section class="content">
|
||||
<template><div class="shell"><aside class="sidebar"><div class="brand"><div class="mark"><Bot :size="23"/></div><div><b>边疆红驿</b><small>NALATI KNOWLEDGE HUB</small></div></div><p class="muted label">工作空间</p><nav><button v-for="n in nav" :key="n.id" :class="{active:page===n.id}" @click="page=n.id"><component :is="n.icon" :size="18"/>{{n.label}}<ChevronRight v-if="page===n.id" class="chev" :size="15"/></button></nav><div class="side-bottom"><div class="tip"><ShieldCheck :size="17"/><div><b>可信知识服务</b><p>优先展示可核验来源,信息不足时明确提示。</p></div></div><p class="api">● API {{apiStatus}} <span>v0.1.0</span></p></div></aside><main><header><span>边疆红驿 / {{nav.find(n=>n.id===page)?.label}}</span><div><span class="pill">● 原型阶段</span><i>研</i></div></header><section class="content">
|
||||
<div v-if="page==='chat'"><div class="heading"><div><small class="eyebrow"><Sparkles :size="14"/> LOCAL KNOWLEDGE ASSISTANT</small><h1>文旅知识,问得明白</h1><p>面向那拉提文旅与民宿服务的知识问答原型</p></div><span class="chip"><ShieldCheck :size="15"/> 来源可追溯 · 设计目标</span></div><div class="chat-layout"><div class="card chat"><div class="chat-head"><div class="mark"><Bot :size="20"/></div><div><b>边疆红驿助手</b><small>问答服务原型</small></div></div><div class="messages"><div v-for="(m,i) in messages" :key="i" class="message" :class="m.role"><small>{{m.role==='assistant'?'智能助手':'你'}}</small><p>{{m.content}}</p><div v-if="m.sources?.length" class="sources"><b><Link2 :size="14"/> 参考来源</b><a v-for="s in m.sources" :key="s.title" :href="s.source_url ?? undefined" target="_blank">{{s.title}}</a></div></div><p v-if="busy">正在请求后端…</p></div><form class="composer" @submit.prevent="send"><textarea v-model="question" placeholder="输入你想了解的文旅问题…" rows="2" @keydown.enter.exact.prevent="send"/><div><small>当前为基础接口,尚未接入真实检索</small><button class="primary" :disabled="busy||!question.trim()"><Send :size="15"/> 发送</button></div></form></div><aside class="suggest"><b><Sparkles :size="16"/> 你可以这样问</b><button @click="question='那拉提有哪些值得了解的旅游资源?'"><Globe2/>旅游资源 <ChevronRight/></button><button @click="question='有哪些经过核验的红色文化资料?'"><BookOpen/>红色文化 <ChevronRight/></button><button @click="question='民宿服务信息应如何核验?'"><ShieldCheck/>民宿服务 <ChevronRight/></button><div class="notice"><b><AlertCircle :size="15"/> 信息说明</b><p>本项目仍处于原型阶段。演示内容不代表已核验的地方文旅事实。</p></div></aside></div></div>
|
||||
<div v-else-if="page==='knowledge'"><div class="heading"><div><small class="eyebrow"><Database :size="14"/> KNOWLEDGE MANAGEMENT</small><h1>知识库管理</h1><p>集中管理领域知识、来源和审核状态。</p></div><button class="primary" @click="showForm=!showForm"><Plus :size="16"/> 新增知识</button></div><div v-if="showForm" class="card form-card"><b>新增知识条目</b><form class="knowledge-form" @submit.prevent="save"><label>标题<input v-model="form.title" required maxlength="250"/></label><label>分类<select v-model="form.category"><option>红色文化</option><option>旅游资源</option><option>民宿服务</option><option>交通出行</option><option>其他</option></select></label><label class="wide">正文<textarea v-model="form.content" required rows="3"/></label><label class="wide">来源网址(可选)<input v-model="form.source_url" type="url" placeholder="https://..."/></label><div class="wide actions"><button type="button" class="secondary" @click="showForm=false">取消</button><button class="primary">保存条目</button></div></form></div><div class="stats"><div class="card stat"><small>知识条目</small><strong>{{items.length}}</strong><span>来自当前数据库</span></div><div class="card stat"><small>已核验</small><strong>{{items.filter(x=>x.verification_status==='verified').length}}</strong><span>需经过真实审核</span></div><div class="card stat"><small>待审核</small><strong>{{items.filter(x=>x.verification_status==='pending').length}}</strong><span>尚未进入检索索引</span></div></div><p v-if="notice" class="success">{{notice}}</p><p v-if="error" class="error">{{error}}</p><div class="card table-card"><div class="table-head"><div><b>知识条目</b><p>列表来自后端接口,不含预设真实资料。</p></div><label class="search"><Search :size="15"/><input v-model="search" placeholder="搜索知识"/></label></div><div class="scroll"><table><thead><tr><th>标题</th><th>分类</th><th>来源</th><th>状态</th><th>创建时间</th></tr></thead><tbody><tr v-for="x in filtered" :key="x.id"><td><b>{{x.title}}</b><small>{{x.content.slice(0,45)}}</small></td><td>{{x.category}}</td><td><a v-if="x.source_url" :href="x.source_url" target="_blank">打开来源</a><span v-else>未填写</span></td><td><em>{{x.verification_status==='pending'?'待审核':x.verification_status}}</em></td><td>{{new Date(x.created_at).toLocaleDateString()}}</td></tr><tr v-if="!filtered.length"><td colspan="5" class="empty">暂无条目,点击“新增知识”添加。</td></tr></tbody></table></div></div></div>
|
||||
<div v-else-if="page==='sources'"><div class="heading"><div><small class="eyebrow"><Globe2 :size="14"/> SOURCE OPERATIONS</small><h1>来源与更新</h1><p>登记可追溯来源,为文档采集和知识审核建立基础。</p></div><button class="primary" @click="showSourceForm=!showSourceForm"><Plus :size="16"/> 新增来源</button></div><div v-if="showSourceForm" class="card form-card"><b>登记来源</b><form class="knowledge-form" @submit.prevent="saveSource"><label>来源名称<input v-model="sourceForm.name" required maxlength="250" placeholder="例如:那拉提景区官网"/></label><label>来源网址(可选)<input v-model="sourceForm.source_url" type="url" placeholder="https://..."/></label><label class="wide">备注<textarea v-model="sourceForm.notes" rows="3" placeholder="记录来源用途、可靠性判断或待核验事项"/></label><div class="wide actions"><button type="button" class="secondary" @click="showSourceForm=false">取消</button><button class="primary">保存来源</button></div></form></div><p v-if="notice" class="success">{{notice}}</p><p v-if="error" class="error">{{error}}</p><div class="stats"><div class="card stat"><small>来源总数</small><strong>{{sources.length}}</strong><span>来自当前数据库</span></div><div class="card stat"><small>关联文档</small><strong>—</strong><span>文档工作台待接入</span></div><div class="card stat"><small>自动更新</small><strong class="pending">未启用</strong><span>当前不会自动抓取网页</span></div></div><div class="card table-card"><div class="table-head"><div><b>已登记来源</b><p>来源不是审核通过的证明,关键事实仍需交叉核验。</p></div><label class="search"><Search :size="15"/><input v-model="sourceSearch" placeholder="搜索名称或备注" @keyup.enter="refreshSources"/></label></div><div class="scroll"><table><thead><tr><th>名称</th><th>网址</th><th>备注</th><th>登记时间</th></tr></thead><tbody><tr v-for="source in sources" :key="source.id"><td><b>{{source.name}}</b></td><td><a v-if="source.source_url" :href="source.source_url" target="_blank" rel="noreferrer">打开来源</a><span v-else>未填写</span></td><td>{{source.notes||'—'}}</td><td>{{new Date(source.created_at).toLocaleDateString()}}</td></tr><tr v-if="!sources.length"><td colspan="4" class="empty">暂无来源,点击“新增来源”登记。</td></tr></tbody></table></div></div><div class="card roadmap"><b>来源处理边界</b><div><span>01 登记来源</span><ChevronRight/><span>02 人工核验</span><ChevronRight/><span>03 关联文档</span><ChevronRight/><span>04 后续检索</span></div></div></div>
|
||||
<div v-else-if="page==='sources'"><div class="heading"><div><small class="eyebrow"><Globe2 :size="14"/> SOURCE OPERATIONS</small><h1>来源与更新</h1><p>为后续网页采集、审核、向量生成和版本切换预留界面。</p></div><span class="pill">● 功能规划中</span></div><div class="notice wide"><b><AlertCircle :size="16"/> 当前状态</b><p>此页面为静态原型。尚未连接网页抓取、定时任务或 Embedding 流程,不会自动更新知识。</p></div><div class="stats"><div class="card stat"><small>来源总数</small><strong>—</strong><span>待接入</span></div><div class="card stat"><small>待处理任务</small><strong>—</strong><span>后台队列尚未实现</span></div><div class="card stat"><small>最近更新</small><strong>—</strong><span>待接入更新记录</span></div></div><div class="card roadmap"><b>后续更新流程</b><div><span>01 获取来源</span><ChevronRight/><span>02 核验内容</span><ChevronRight/><span>03 生成向量</span><ChevronRight/><span>04 发布版本</span></div></div></div>
|
||||
<div v-else><div class="heading"><div><small class="eyebrow"><Activity :size="14"/> EVALUATION WORKSPACE</small><h1>实验评估</h1><p>展示实验方案与指标规划;当前不显示虚构实验结果。</p></div><span class="pill">● 尚未开展实验</span></div><div class="notice wide"><b><ShieldCheck :size="16"/> 实验原则</b><p>所有指标应来自真实测试问题、运行日志和明确的人工评分规则。下方仅列评价维度,不包含实验数值。</p></div><div class="stats"><div class="card stat"><small>回答正确性</small><strong class="pending">待测试</strong><span>依据人工评分规范</span></div><div class="card stat"><small>来源可追溯性</small><strong class="pending">待测试</strong><span>核对来源是否支持答案</span></div><div class="card stat"><small>响应时间与成本</small><strong class="pending">待测试</strong><span>记录耗时与实际费用</span></div></div><div class="card roadmap"><b>计划中的对照方案</b><div class="methods"><p><b>A 基础大语言模型</b><span>不提供项目知识库检索结果</span></p><p><b>B 关键词检索 RAG</b><span>关键词检索后生成答案</span></p><p><b>C 混合检索 RAG</b><span>融合关键词与向量语义检索</span></p></div></div></div><footer>边疆红驿 · 那拉提文旅智能问答 <span>工程原型 v0.1.0 · 尚非完整系统</span></footer></section></main></div></template>
|
||||
|
||||
+4
-21
@@ -1,21 +1,4 @@
|
||||
import { apiRequest } from './services/session'
|
||||
|
||||
export const getHealth = () => apiRequest<{status:string}>('/api/health')
|
||||
export const askQuestion = (question:string) => apiRequest<{answer:string;status:string;sources:{title:string;source_url?:string|null}[]}>('/api/chat',{method:'POST',body:JSON.stringify({question})})
|
||||
export const getKnowledge = () => apiRequest<{id:number;title:string;category:string;content:string;source_url:string|null;verification_status:string;version:number}[]>('/api/knowledge')
|
||||
export const createKnowledge = (payload:{title:string;category:string;content:string;source_url?:string}) => apiRequest('/api/knowledge',{method:'POST',body:JSON.stringify(payload)},true)
|
||||
|
||||
export type VerificationStatus='pending'|'verified'|'rejected'
|
||||
export type SourceItem={id:number;version:number;name:string;source_url:string|null;notes:string;created_at:string}
|
||||
export function getSources(q=''){return apiRequest<{items:SourceItem[];total:number}>(`/api/sources?q=${encodeURIComponent(q)}`)}
|
||||
export function createSource(payload:{name:string;source_url?:string;notes:string}){return apiRequest<SourceItem>('/api/sources',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify(payload)},true)}
|
||||
export function reviewKnowledge(id:number,payload:{expected_version:number;verification_status:VerificationStatus;reviewer:string;verification_notes:string}){return apiRequest(`/api/knowledge/${id}/review`,{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify(payload)},true)}
|
||||
|
||||
export type RetrievalHit={rank:number;knowledge_id:number;knowledge_version:number;document_id:number|null;document_version:number|null;source_id:number|null;source_version:number|null;source_name:string|null;source_url:string;title:string;category:string;snippet:string;snippet_start:number;snippet_end:number;score:number;matched_terms:string[]}
|
||||
export type RetrievalResult={run_id:string;query:string;mode:'keyword';status:'ok'|'no_evidence';top_k:number;category:string|null;query_terms:string[];corpus_size:number;corpus_fingerprint:string;total_hits:number;results:RetrievalHit[];elapsed_ms:number;created_at:string;config:{tokenizer:string;scorer:string;k1:number;b:number;title_weight:number;stopwords_version:string}}
|
||||
export function retrieveKnowledge(payload:{query:string;mode?:'keyword';top_k?:number;category?:string}){return apiRequest<RetrievalResult>('/api/retrieval',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify(payload)})}
|
||||
export function getRetrievalRun(runId:string){return apiRequest<RetrievalResult>(`/api/retrieval/runs/${encodeURIComponent(runId)}`)}
|
||||
export function getRetrievalRuns(page=1,pageSize=20){return apiRequest<{items:RetrievalResult[];total:number;page:number;page_size:number}>(`/api/retrieval/runs?page=${page}&page_size=${pageSize}`)}
|
||||
|
||||
export type VectorRetrievalResult=Omit<RetrievalResult,'mode'|'results'|'config'> & {mode:'vector';vector_version_id:string;text_index_version_id:string;generation:number;text_generation:number;results:(RetrievalHit & {chunk_id:string;chunk_ordinal:number})[];config:{scorer:'pgvector-cosine-exact-v1';ranking_unit:'chunk';space:{id:string;dimension:number}}}
|
||||
export function retrieveVectorKnowledge(payload:{query:string;top_k?:number;category?:string}){return apiRequest<VectorRetrievalResult>('/api/retrieval/vector',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({mode:'vector',top_k:5,...payload})})}
|
||||
export async function getHealth(){const r=await fetch('/api/health');if(!r.ok)throw new Error('health failed');return r.json()}
|
||||
export async function askQuestion(question:string){const r=await fetch('/api/chat',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({question})});if(!r.ok)throw new Error('chat failed');return r.json() as Promise<{answer:string;status:string;sources:{title:string;source_url?:string|null}[]}>}
|
||||
export async function getKnowledge(){const r=await fetch('/api/knowledge');if(!r.ok)throw new Error('knowledge failed');return r.json() as Promise<any[]>}
|
||||
export async function createKnowledge(payload:{title:string;category:string;content:string;source_url?:string}){const r=await fetch('/api/knowledge',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify(payload)});if(!r.ok)throw new Error('保存失败,请检查网址格式及后端状态');return r.json()}
|
||||
|
||||
@@ -1,2 +0,0 @@
|
||||
.search:focus-within{border-color:#8eaa91;box-shadow:0 0 0 3px #dcebdc}.knowledge-form input:focus,.knowledge-form select:focus,.knowledge-form textarea:focus{border-color:#7e9f83;box-shadow:0 0 0 3px #e4f0e4}.table-card tbody tr{transition:background .15s ease}.table-card tbody tr:hover{background:#fafbf7}.success,.error{animation:feedback-in .2s ease-out}@keyframes feedback-in{from{opacity:0;transform:translateY(-4px)}to{opacity:1;transform:translateY(0)}}
|
||||
.quick-actions{margin-top:18px}.quick-actions .label{margin:0 0 8px}.side-action{width:100%;display:flex;align-items:center;gap:8px;border:1px solid #dbe6da;border-radius:9px;padding:10px 12px;background:#f1f7ef;color:#355c48;font-size:11px;font-weight:600}.side-action:hover{background:#e2efdf}.side-hint{margin:0 10px;color:#8a9086;font-size:10px;line-height:1.7}.heading>.primary{display:none}
|
||||
@@ -1,5 +1,4 @@
|
||||
import { createApp } from 'vue'
|
||||
import './style.css'
|
||||
import './enhancements.css'
|
||||
import App from './App.vue'
|
||||
createApp(App).mount('#app')
|
||||
|
||||
@@ -1,20 +0,0 @@
|
||||
import type { ChangePasswordPayload, CurrentUser, LoginPayload, RegisterPayload, SessionUser, UpdateProfilePayload } from '../types/auth'
|
||||
import { apiRequest, ApiError, clearSessionToken } from './session'
|
||||
|
||||
export { ApiError as AuthApiUnavailableError }
|
||||
export const register = (payload: RegisterPayload) => apiRequest<CurrentUser>('/api/auth/register', { method: 'POST', body: JSON.stringify(payload) })
|
||||
export const login = async (payload: LoginPayload) => {
|
||||
clearSessionToken()
|
||||
return apiRequest<SessionUser>('/api/auth/login', { method: 'POST', body: JSON.stringify(payload) })
|
||||
}
|
||||
export const getCurrentUser = () => apiRequest<SessionUser>('/api/auth/me')
|
||||
export const updateMyProfile = (payload: UpdateProfilePayload) => apiRequest<CurrentUser>('/api/users/me', { method: 'PATCH', body: JSON.stringify(payload) }, true)
|
||||
export const logout = async () => {
|
||||
await apiRequest<void>('/api/auth/logout', { method: 'POST' }, true)
|
||||
clearSessionToken()
|
||||
}
|
||||
export const changePassword = async (payload: ChangePasswordPayload) => {
|
||||
await apiRequest<void>('/api/auth/change-password', { method: 'POST', body: JSON.stringify(payload) }, true)
|
||||
clearSessionToken()
|
||||
}
|
||||
export const getUsers = (limit = 100, offset = 0) => apiRequest<CurrentUser[]>(`/api/admin/users?limit=${limit}&offset=${offset}`)
|
||||
@@ -1,16 +0,0 @@
|
||||
import { apiRequest } from './session'
|
||||
import type { RetrievalHit } from '../api'
|
||||
|
||||
type Page<T> = { items: T[]; total: number; page: number; page_size: number }
|
||||
export type IndexConfig = { splitter: 'unicode-window-v1'; chunk_size: number; overlap: number }
|
||||
export type IndexVersion = { id: string; status: 'ready' | 'published' | 'superseded'; config: IndexConfig; corpus_fingerprint: string; catalog_revision: number; item_count: number; chunk_count: number; created_by: number; created_at: string; published_at: string | null }
|
||||
export type ActiveIndex = { generation: number; catalog_revision: number; is_current: boolean; active_version: IndexVersion | null }
|
||||
export type Chunk = { id: string; index_version_id: string; knowledge_id: number; ordinal: number; start_offset: number; end_offset: number; text: string; text_sha256: string; provenance: Pick<RetrievalHit, 'knowledge_id' | 'knowledge_version' | 'document_id' | 'document_version' | 'source_id' | 'source_version' | 'source_name' | 'source_url' | 'title' | 'category'> }
|
||||
export type IndexPublication = { generation: number; previous_version_id: string | null; version_id: string; actor_id: number; published_at: string }
|
||||
export const buildIndex = (config: Partial<Pick<IndexConfig, 'chunk_size' | 'overlap'>> = {}) => apiRequest<IndexVersion>('/api/indexes', { method: 'POST', body: JSON.stringify(config) }, true)
|
||||
export const getActiveIndex = () => apiRequest<ActiveIndex>('/api/indexes/active')
|
||||
export const getIndexVersions = (page = 1, pageSize = 20) => apiRequest<Page<IndexVersion>>(`/api/indexes?page=${page}&page_size=${pageSize}`)
|
||||
export const getIndexVersion = (id: string) => apiRequest<IndexVersion>(`/api/indexes/${encodeURIComponent(id)}`)
|
||||
export const publishIndex = (id: string, expectedGeneration: number) => apiRequest<ActiveIndex>(`/api/indexes/${encodeURIComponent(id)}/publish`, { method: 'POST', body: JSON.stringify({ expected_generation: expectedGeneration }) }, true)
|
||||
export const getChunks = (id: string, page = 1, pageSize = 20, knowledgeId?: number) => apiRequest<Page<Chunk>>(`/api/indexes/${encodeURIComponent(id)}/chunks?page=${page}&page_size=${pageSize}${knowledgeId ? `&knowledge_id=${knowledgeId}` : ''}`)
|
||||
export const getIndexPublications = (page = 1, pageSize = 20) => apiRequest<Page<IndexPublication>>(`/api/indexes/publications?page=${page}&page_size=${pageSize}`)
|
||||
@@ -1,28 +0,0 @@
|
||||
let csrfToken: string | null = null
|
||||
|
||||
export class ApiError extends Error {
|
||||
constructor(public readonly status: number, message: string) { super(message) }
|
||||
}
|
||||
|
||||
export function clearSessionToken() { csrfToken = null }
|
||||
|
||||
export async function apiRequest<T>(path: string, init: RequestInit = {}, protect = false): Promise<T> {
|
||||
if (protect && !csrfToken) await apiRequest('/api/auth/me')
|
||||
const headers = new Headers(init.headers)
|
||||
headers.set('Accept', 'application/json')
|
||||
if (init.body) headers.set('Content-Type', 'application/json')
|
||||
if (protect && csrfToken) headers.set('X-CSRF-Token', csrfToken)
|
||||
const response = await fetch(path, { ...init, credentials: 'include', headers })
|
||||
if (!response.ok) {
|
||||
if (response.status === 401) clearSessionToken()
|
||||
const body: unknown = await response.json().catch(() => null)
|
||||
const detail = body && typeof body === 'object' && 'detail' in body ? body.detail : null
|
||||
throw new ApiError(response.status, typeof detail === 'string' ? detail : `请求失败(${response.status}),请检查登录状态或填写内容。`)
|
||||
}
|
||||
if (response.status === 204) return undefined as T
|
||||
const data = await response.json()
|
||||
if (path === '/api/auth/login' || path === '/api/auth/me') {
|
||||
csrfToken = typeof data.csrf_token === 'string' ? data.csrf_token : null
|
||||
}
|
||||
return data as T
|
||||
}
|
||||
@@ -1,14 +0,0 @@
|
||||
export type UserRole = 'user' | 'admin'
|
||||
export type CurrentUser = {
|
||||
id: number
|
||||
username: string
|
||||
nickname: string | null
|
||||
email: string | null
|
||||
role: UserRole
|
||||
created_at: string
|
||||
}
|
||||
export type SessionUser = CurrentUser & { csrf_token: string }
|
||||
export type RegisterPayload = { username: string; password: string; email?: string; nickname?: string }
|
||||
export type LoginPayload = { username: string; password: string }
|
||||
export type UpdateProfilePayload = { nickname?: string | null; email?: string | null }
|
||||
export type ChangePasswordPayload = { current_password: string; new_password: string }
|
||||
Reference in New Issue
Block a user