Compare commits

..

11 Commits

Author SHA1 Message Date
codex-bot 9ee2bd219a Add chapter editing API with internal summaries and revision control 2026-10-30 19:58:40 +08:00
Zayden-Jung 6b4fa711c1 Merge pull request '补核六项既有业务并保留 Sponstar 年份冲突' (#28) from codex/ai-research-year-proofs into main
Verify and deploy Tabularium / verify-and-deploy (push) Successful in 8m34s
Reviewed-on: #28
2026-10-30 11:32:34 +00:00
codex-bot 76e32490e9 Verify six company histories and retain conflicting founding dates 2026-10-30 19:27:55 +08:00
Zayden-Jung fa0bcb15fb Merge pull request '补齐32项企业竞赛登记并核查19项既有业务' (#26) from codex/ai-research-2025-coverage into main
Verify and deploy Tabularium / verify-and-deploy (push) Has been cancelled
Reviewed-on: #26
2026-10-30 11:20:47 +00:00
codex-bot 12123c278f Resolve fourteen older Battlefield businesses with source evidence 2026-10-30 19:17:43 +08:00
codex-bot 4273dd77e0 Register all 32 enterprise Battlefield discovery entries 2026-10-30 19:09:13 +08:00
Zayden-Jung 3ab2f8e122 Merge pull request '汇总 AI 公司核查更新与文档上传 API' (#25) from codex/research-api-integration into main
Verify and deploy Tabularium / verify-and-deploy (push) Successful in 7m33s
Reviewed-on: #25
2026-10-30 10:28:45 +00:00
codex-bot 15024c3908 merge: include latest company research updates from PR #23 2026-10-30 18:20:34 +08:00
codex-bot 083f532609 merge: integrate document upload API from PR #24 2026-10-30 18:17:53 +08:00
codex-bot 2b3706c90d merge: integrate company categorization and deployment fixes from PR #23 2026-10-30 18:17:53 +08:00
codex-bot 6bfeea4e7e feat: add scoped document API and resumable batch uploader 2026-10-30 18:14:33 +08:00
23 changed files with 2441 additions and 46 deletions
+2 -1
View File
@@ -3,7 +3,8 @@
- 本仓库专门维护 Endor 文档与制图脚本。支持本地开发;CI/CD 仅在 main 使用现有 ces-control Runner,验证通过后自动部署;工作分支与 PR 不运行 CI,确认 Runner 不构成扩大触发范围的授权。不得修改其他项目、共享 Runner 配置或清理共享 Docker 缓存。
- Web 技术栈为 TypeScript、React、Tailwind CSS、Next.js Pages Router;不要切换到 App Router。
- 使用 pnpm 管理 JavaScript workspace,使用 uv 管理 `tools/figures` 的 Python 项目,保留两套锁文件。
- 文档正文、目录元数据与修订记录以站点现有 SQLite 数据库为唯一权威源,使用 docs 管理工具直接更新;不得把新正文、目录 JSON、数据库文件或日常内容快照提交到仓库。历史 Git 文档只用于显式的一次性迁移,不作为运行时种子。Python 图形源码放在 `tools/figures/src/endor_figures`,可复现的 SVG 输出放在 `apps/docs/public/figures`。
- 文档正文、目录元数据与修订记录以站点现有 SQLite 数据库为唯一权威源,使用文档 API 或 docs 管理工具直接更新;不得把新正文、目录 JSON、数据库文件或日常内容快照提交到仓库。历史 Git 文档只用于显式的一次性迁移,不作为运行时种子。Python 图形源码放在 `tools/figures/src/endor_figures`,可复现的 SVG 输出放在 `apps/docs/public/figures`。
- 编辑已有文章时优先使用章节 API:先读取章节目录与内部摘要,据此选择需要读取正文的章节,再提交单章或多章操作。摘要缺失时先读对应正文并补充;修改观点前必须核对该章原文,检查读取版本与提交 revision 一致。摘要只写入内部章节数据,不放入网页正文、目录摘要或 Git;改写正文时同步更新摘要。完整操作与 Python 客户端用法见 README 的「先读摘要,再按章节编辑」。
- 未确认的架构提议明确标注为草案,不能表述为现有实现。
- 本地开发与预览只监听 `127.0.0.1`。部署容器监听 `0.0.0.0:3000`,服务器入口默认 18135,可通过独立 production.env 配置。
- 对 gitea.entroia.com 仅使用 codex-bot,提交身份为 Codex Bot <codex-bot@gitea.entroia.local>。SSH 使用隔离配置:`ssh -F /dev/null -o IdentitiesOnly=yes -i /Users/zzyang/.codex/credentials/gitea-entroia/codex_entroia_ed25519 -p 22222`。
+118 -2
View File
@@ -8,7 +8,7 @@ Endor 的文档站与 AI 公司研究工作台,包含文档阅读、全文搜
| 内容 | 权威来源与维护方式 |
| --- | --- |
| 文档正文、分类、合集、分组、摘要、草案和推荐状态 | 数据库;通过服务器上的 `docs` 管理工具读写 |
| 文档正文、分类、合集、分组、摘要、草案和推荐状态 | 数据库;通过文档 API 或服务器上的 `docs` 管理工具读写 |
| 文档旧地址与修订历史 | 数据库;改名保留别名,修改追加修订 |
| AI 公司、观测、证据、候选与核查记录 | `content/research` 中的版本化快照;服务端显式按版本导入同库后查询 |
| 页面、组件、管理工具和部署逻辑 | 本仓库,通过工作分支与 PR 评审 |
@@ -32,7 +32,7 @@ Endor 的文档站与 AI 公司研究工作台,包含文档阅读、全文搜
## 管理文档
部署后,管理员可在服务器上使用 `/opt/endorphin-tabularium-docs/docs`。该工具在当前容器中操作数据库,需要服务器运维权限;站点没有公开的文档写入 API。
部署后,管理员可在服务器上使用 `/opt/endorphin-tabularium-docs/docs`。该工具在当前容器中操作数据库,需要服务器运维权限。自动化上传使用下文的文档 API,日常读写无需登录服务器。
```sh
sudo /opt/endorphin-tabularium-docs/docs list
@@ -63,6 +63,122 @@ sudo /opt/endorphin-tabularium-docs/docs save < "$doc_work_dir/document.json"
保存成功后,正文、目录和搜索通过后续请求读取新内容。发生版本冲突时重新读取并合并修改,不能仅改大 `revision` 覆盖并发编辑。调整分类或分组也走同一保存流程;`history` 使用稳定的 `sourceSlug` 查询修订。
## 文档上传 API
文档 API 直接读写同一个数据库,供自动化整理和大量文档上传使用。它只接受独立的文档 API Key,不接受网页 Cookie 或通用网页会话令牌;Key 不能用于研究、登录或其他站点 API。
| 方法与地址 | 用途 |
| --- | --- |
| `GET /api/documents?limit=50&offset=0` | 分页读取目录元数据;不返回全部正文 |
| `GET /api/documents/item/<slug>` | 读取单篇完整正文、目录字段和当前 revision;支持旧地址 |
| `GET /api/documents/item/<slug>?history=1&limit=50&offset=0` | 分页读取修订元数据与时间,按新到旧排列 |
| `GET /api/documents/item/<slug>?revision=1` | 读取指定修订的完整正文 |
| `POST /api/documents` | 保存一个完整文档 JSON;字段与 `docs save` 相同 |
| `POST /api/documents/batch` | 保存 `{ "documents": [...] }`,每个请求 1—100 篇 |
| `GET /api/documents/sections/<slug>` | 读取章节 ID、标题和内部摘要,不返回正文 |
| `GET /api/documents/sections/<slug>?section=<id>` | 读取选中章节正文;重复 `section` 可选多章 |
| `GET /api/documents/sections/<slug>?revision=2` | 读取对应修订的章节目录;也可搭配 `section` |
| `POST /api/documents/sections/<slug>` | 原子执行章节新增、更新、删除和移动 |
写入使用 `Authorization: Bearer <文档 API Key>`。新建传 `revision: 0`;更新先读取当前全文,再使用读到的 revision。每篇正文最多 2,000,000 个 JavaScript 字符单位;每次请求最多 8 MiB。目录和修订分页的 `limit` 范围为 1—100,响应包含 `total` 和 `nextOffset`。
单个批量请求在同一事务提交,任意一篇格式错误、版本冲突或地址冲突都会整体回滚。成功返回文档地址、稳定标识和新修订号;版本或幂等键冲突返回 `409`,格式错误返回 `400`,超出请求大小返回 `413`。缺少、过期或撤销的 Key 返回 `401`,只读 Key 写入返回 `403`。
建议写入时提供 `Idempotency-Key`,由 1—128 个字母、数字或 `._:-` 组成。同一个 Key、相同请求 ID 和相同输入重试时,返回原回执并标记 `replayed: true`,不追加修订;相同 ID 改换输入会冲突。回执描述原请求,不代表之后没有其他修改;需要当前内容时重新读取文档。改过正文或 revision 后,应使用新的请求 ID。
### 先读摘要,再按章节编辑
章节接口使用相同的文档 Key 和读写权限。编辑流程为:先读章节目录及摘要,选择需要深入的章节,再读取这些章节的正文,最后提交一组操作。不要只凭摘要修改未读正文;目录与正文响应的 `revision` 应一致,否则重新读取。接口不调用模型生成摘要:由编辑 Agent 阅读正文、撰写摘要,并在编辑请求中保存。
章节按行首 Markdown ATX 二级标题(`##`)划分,三级至六级标题随所属章节一起编辑;代码围栏内的标题不会切章。二级标题之前的文档标题及导言属于固定 ID `preamble`,可以修改但不能删除或移动。Setext 标题、带缩进或引用、列表标记的标题不作为章节边界。正文无行首二级标题时,全文作为导言处理。拆分及组合保持换行和未编辑内容原样,不自动格式化 Markdown。
目录中的每章包含 `id`、`title`、`position`、`summary`、`summaryStatus` 和 `contentHash`。摘要为编辑内部数据,单独保存在 SQLite 的 `document_sections` 和 `document_section_revisions` 中,不加入普通文档响应、网页 props、搜索索引或页面正文。完整数据库备份包含这些表;`docs export` 仍只导出当前文档,不包含内部摘要和修订。
旧文章无需重写,首次读取即可按原文返回目录,初始摘要为空,`summaryStatus` 为 `missing`。Agent 应按需读取这些章节,生成摘要并通过 `update` 保存;保存后状态为 `current`。这里的 `current` 仅表示摘要与该版正文一起保存,不保证模型的总结正确。全文接口仍兼容;未变化章节保留摘要与 ID,正文变化的章节清空摘要,等待更新。全文改名且改写章节时可能产生新 ID,因此全文保存后应重新读取目录。
章节编辑请求示例(示例 ID 需要替换为目录实际返回的值):
```json
{
"revision": 3,
"operations": [
{
"op": "update",
"id": "s_existing",
"content": "## 原有章节\n\n修改后的正文。\n\n",
"summary": "本章说明修改后的核心观点与边界。"
},
{
"op": "insert",
"after": "s_existing",
"content": "## 新章节\n\n新增论证。\n\n",
"summary": "补充上一章尚未覆盖的论证。"
}
]
}
```
`update` 可只改 `summary`,也可改正文;正文改变时若不提供新摘要,旧摘要会清空。`insert` 必须提供正文和非空摘要;摘要最多 8000 个 JavaScript 字符单位。`delete` 使用 `{ "op": "delete", "id": "s_existing" }`;`move` 使用 `{ "op": "move", "id": "s_existing", "after": "preamble" }`。按数组顺序执行,最多 100 个操作;新章 ID 由服务器生成,在回执目录中返回。章正文应从自己的 `##` 标题开始,只包含一个二级标题,并保留与下一章之间的换行;跨章代码围栏或章节边界改变会被拒绝,拆章请使用多个操作。
章节变更、内部摘要、组合后的全文和历史快照在同一事务内提交。每批操作只增加一次文章 `revision`;摘要单独更新也增加版本号。后续操作失败或全文版本过期时全部回滚。写入沿用幂等规则,回执包含新版本、章节目录和 `replayed`;章节接口与全文接口的幂等记录分开。修改章节不会改变其 ID,移动亦保留 ID;删除后的正文和摘要仍可从对应的旧修订读取。启用前的历史修订只有原文,摘要显示缺失。
Python 客户端对应命令:
```sh
python3 scripts/documents-client.py outline endor/endor-purpose-draft
python3 scripts/documents-client.py sections endor/endor-purpose-draft preamble s_existing
python3 scripts/documents-client.py edit-sections endor/endor-purpose-draft /private/path/edit.json --request-id chapter-edit-1
```
`outline` 和 `sections` 支持 `--revision`;请求文件只包含 `revision` 和 `operations`,不要把内部摘要文件提交到 Git。
### 创建、查看与撤销 Key
管理员只需通过已验证的服务器连接进行 Key 管理:
```sh
sudo /opt/endorphin-tabularium-docs/docs key-create codex-upload /auth/codex-docs-key.json read-write 90
sudo /opt/endorphin-tabularium-docs/docs key-list
sudo /opt/endorphin-tabularium-docs/docs key-revoke <key-id>
```
创建命令将完整 Key 写入认证私有目录中的指定新文件,权限 `0600`,终端仅输出 ID、权限和到期信息。文件已存在时拒绝覆盖。权限可选 `read` 或 `read-write`;有效期默认 90 天,可设 1—365 天。数据库仅保存 Key 摘要,列表不返回原文;撤销后不能继续认证。独立 Key 文件和认证 config.json 都不包含在 SQLite 整库备份中。
### 使用上传客户端
`scripts/documents-client.py` 仅使用 Python 标准库。通过 `TABULARIUM_DOCS_URL` 指定站点 origin,通过 `TABULARIUM_DOCS_KEY_FILE` 指定本机私有 Key 文件(支持创建命令输出的 JSON),不把 Key 放入命令行参数。
客户端要求 HTTPS;尚未配置 HTTPS 时,可通过已验证的 SSH 连接建立本机端口转发,连接 `http://127.0.0.1:18136`。以下 `$DOCS_SSH_TARGET` 应事先设置为受信任的 SSH 别名或 `user@host`:
```sh
ssh -N -L 18136:127.0.0.1:18135 "$DOCS_SSH_TARGET"
```
将服务器私有目录中的 Key 文件复制到受保护的本机位置后,在另一个终端配置客户端。`$TABULARIUM_DOCS_KEY_FILE` 文件须仅本人可读写,不能提交到仓库:
```sh
export TABULARIUM_DOCS_URL=http://127.0.0.1:18136
export TABULARIUM_DOCS_KEY_FILE="$HOME/.config/tabularium/codex-docs-key.json"
python3 scripts/documents-client.py list --limit 50
python3 scripts/documents-client.py get endor/endor-purpose-draft
python3 scripts/documents-client.py get endor/endor-purpose-draft --history
python3 scripts/documents-client.py get endor/endor-purpose-draft --revision 1
```
修改流程与 CLI 相同:
```sh
umask 077
api_work_dir=$(mktemp -d)
python3 scripts/documents-client.py get endor/endor-purpose-draft > "$api_work_dir/document.json"
# 编辑正文或目录字段,保留 sourceSlug 和读取到的 revision。
python3 scripts/documents-client.py save "$api_work_dir/document.json"
```
大量文档可使用仅含 `documents` 数组的私有 JSON 文件。客户端按 100 篇和 8 MiB 请求上限自动分批,并为各批派生固定的幂等请求 ID。每批独立提交;后续批次失败时,前面的成功批次保留。客户端会打印根请求 ID 和各批进度,使用相同 ID 与未修改的输入重试可跳过已完成的批次;可通过 `--request-id` 显式指定根 ID(客户端最多 100 个字符)。若后续批次因业务冲突失败,按进度中已提交的文档数量提取尚未成功的部分,修正后使用新的根 ID 上传。
客户端只自动重试一次临时网络或 `502/503/504` 错误,保持正文和请求 ID 不变;`409` 等业务错误不会自动覆盖。它不跟随 HTTP 重定向,避免将 Bearer 凭据转发到其他地址。正文和 Key 继续保存在数据库或私有文件中,不因使用 API 而进入 Git。
## 研究数据与检查
研究工作台提供公司筛选、经营指标、原始证据、收费与交付模式、技术材料及核查记录。它区分公司披露、支付平台指标、第三方估算、融资和客户收益;未知收入不填零,材料可访问也不代表性能或商业效果已经验证。研究范围、样本限制和个案结论以工作台记录为准。
+42 -1
View File
@@ -18,6 +18,7 @@ import {
totp,
isLoginRedirect,
} from "../src/lib/auth";
import { createDocumentKey, revokeDocumentKey } from "../src/lib/document-keys";
import { readDocument, saveDocument } from "../src/lib/docs";
import { consumeCode, initializeStore } from "../src/lib/auth-store";
@@ -183,7 +184,7 @@ async function main() {
const research = await researchResponse.json();
assert.equal(research.companies.length, 1023);
assert.equal(research.categories.length, 11);
assert.equal(research.evidence.length, 1259);
assert.equal(research.evidence.length, 1318);
assert.equal(research.observations.length, 189);
assert.equal((await request("/api/research/ai-companies", { method: "POST", headers })).status, 405);
const researchPage = await request("/docs/other-research/ai-company-research/overview", { headers });
@@ -210,6 +211,7 @@ async function main() {
assert.equal(raw.status, 200);
assert.ok((await raw.text()).startsWith("# Endor"));
if (process.env.DOCUMENT_SMOKE_TEST) {
if (command === "exercise") await exerciseDocumentApi(request, headers);
const sourceSlug = "ci-live-document";
const previous = readDocument(sourceSlug);
if (command === "exercise") {
@@ -356,3 +358,42 @@ main().catch((error) => {
);
process.exitCode = 1;
});
async function exerciseDocumentApi(request: (path: string, init?: RequestInit) => Promise<Response>, session: Record<string, string>) {
const writer = createDocumentKey("isolated API smoke", "read-write", 1);
const reader = createDocumentKey("isolated read smoke", "read", 1);
const headers = { Authorization: "Bearer " + writer.token, "Content-Type": "application/json", "Idempotency-Key": "smoke-batch" };
const readHeaders = { Authorization: "Bearer " + reader.token, "Content-Type": "application/json" };
const make = (name: string) => ({ slug: `endor/${name}`, sourceSlug: name, title: "API smoke",
category: "Endor", collection: "Endor 文档", group: "CI", description: "Isolated API smoke",
draft: true, featured: false, content: "# API smoke", revision: 0 });
const payload = { documents: [make("ci-api-first"), make("ci-api-second")] };
try {
assert.equal((await request("/api/documents")).status, 401);
assert.equal((await request("/api/documents", { method: "POST", headers: { ...session, "Content-Type": "application/json" }, body: JSON.stringify(payload.documents[0]) })).status, 401);
assert.equal((await request("/api/research/ai-companies", { headers })).status, 401, "Document keys cannot read research");
assert.equal((await request("/api/search", { headers })).status, 401, "Document keys cannot authenticate other APIs");
assert.equal((await request("/api/documents/batch", { method: "POST", headers: readHeaders, body: JSON.stringify(payload) })).status, 403);
const saved = await request("/api/documents/batch", { method: "POST", headers, body: JSON.stringify(payload) });
assert.equal(saved.status, 200);
assert.equal((await saved.json()).documents.length, 2);
const replay = await request("/api/documents/batch", { method: "POST", headers, body: JSON.stringify(payload) });
assert.equal(replay.status, 200); assert.equal((await replay.json()).replayed, true);
const current = await request("/api/documents/item/endor/ci-api-first", { headers: readHeaders });
assert.equal(current.status, 200); assert.equal((await current.json()).revision, 1);
const page = await request("/api/documents?limit=1", { headers: readHeaders });
assert.equal(page.status, 200);
const listing = await page.json();
assert.equal(listing.documents.length, 1); assert.equal(listing.documents[0].content, undefined);
const bad = { documents: [{ ...payload.documents[0], revision: 1, content: "Must roll back" }, payload.documents[1]] };
const conflict = await request("/api/documents/batch", { method: "POST", headers: { ...headers, "Idempotency-Key": "smoke-conflict" }, body: JSON.stringify(bad) });
assert.equal(conflict.status, 409);
const unchanged = await request("/api/documents/item/endor/ci-api-first", { headers: readHeaders });
assert.equal((await unchanged.json()).content, "# API smoke");
const history = await request("/api/documents/item/endor/ci-api-first?history=1", { headers });
assert.equal(history.status, 200); assert.equal((await history.json()).total, 1);
revokeDocumentKey(writer.id);
assert.equal((await request("/api/documents", { headers })).status, 401);
} finally { revokeDocumentKey(writer.id); revokeDocumentKey(reader.id); }
console.log("Document API smoke passed: scoped keys, batch rollback, pagination, history, replay and revocation; no secrets displayed");
}
+29 -3
View File
@@ -1,11 +1,37 @@
import { readFileSync, writeFileSync } from "node:fs";
import { readFileSync, writeFileSync, openSync, closeSync, unlinkSync } from "node:fs";
import { resolve, dirname, sep } from "node:path";
import { createDocumentKey, listDocumentKeys, revokeDocumentKey, type DocumentKeyScope } from "../src/lib/document-keys";
import { openSiteDatabase } from "../src/lib/site-database";
import { allDocuments, readDocument, saveDocument, importDocuments, documentHistory } from "../src/lib/docs";
process.umask(0o077);
const [command, argument] = process.argv.slice(2);
const [command, argument, ...extra] = process.argv.slice(2);
try {
switch (command) {
case "key-create": {
const authFile = process.env.AUTH_CONFIG_FILE;
if (!authFile || !argument || !extra[0]) throw Error("Usage: key-create <name> <private-file> [read|read-write] [days]");
const file = resolve(extra[0]);
const directory = resolve(dirname(authFile));
if (!file.startsWith(directory + sep)) throw Error("Key output must be inside the private authentication directory");
const fd = openSync(file, "wx", 0o600);
let created: { id: string } | undefined;
try {
const key = createDocumentKey(argument, (extra[1] ?? "read-write") as DocumentKeyScope, Number(extra[2] ?? 90));
created = key;
writeFileSync(fd, JSON.stringify(key) + "\n");
console.log(JSON.stringify({ id: key.id, scope: key.scope, expiresAt: key.expiresAt, saved: file }));
} catch (error) {
if (created) revokeDocumentKey(created.id);
unlinkSync(file);
throw error;
} finally { closeSync(fd); }
break;
}
case "key-list": console.log(JSON.stringify(listDocumentKeys(), null, 2)); break;
case "key-revoke":
if (!argument) throw Error("Provide key id");
revokeDocumentKey(argument); console.log("Document key revoked"); break;
case "backup": {
if (!argument) throw Error("Provide a private backup filename");
const db = openSiteDatabase();
@@ -36,7 +62,7 @@ try {
{ flag: "wx", mode: 0o600 });
console.log("Document export created; existing files are never overwritten"); break;
}
default: throw Error("Usage: docs backup <private-file> | list | get <slug> | save < input.json | history <sourceSlug> | import [--if-empty] < export.json | export <private-file>");
default: throw Error("Usage: docs key-create <name> <private-file> [read|read-write] [days] | key-list | key-revoke <id> | backup <private-file> | list | get <slug> | save < input.json | history <sourceSlug> | import [--if-empty] < export.json | export <private-file>");
}
} catch (error) {
console.error(error instanceof Error ? error.message : "Document operation failed");
+3 -4
View File
@@ -6,9 +6,8 @@ import { spawnSync } from "node:child_process";
const dir = await mkdtemp(join(tmpdir(), "tabularium-docs-tests-"));
try {
const file = join(dir, "docs.test.cjs");
await build({ entryPoints: ["tests/docs.test.ts"], bundle: true,
platform: "node", target: "node22", format: "cjs", outfile: file });
const result = spawnSync(process.execPath, ["--test", file], { stdio: "inherit" });
await build({ entryPoints: ["tests/docs.test.ts", "tests/documents-api.test.ts"], bundle: true,
platform: "node", target: "node22", format: "cjs", outdir: dir, outExtension: { ".js": ".cjs" } });
const result = spawnSync(process.execPath, ["--test", join(dir, "docs.test.cjs"), join(dir, "documents-api.test.cjs")], { stdio: "inherit" });
process.exitCode = result.status ?? 1;
} finally { await rm(dir, { recursive: true, force: true }); }
+178 -18
View File
@@ -1,5 +1,9 @@
import { randomUUID, createHash } from "node:crypto";
import { openSiteDatabase } from "./site-database";
import { DocumentValidationError, DocumentConflictError } from "./document-errors";
import { sectionSchema, loadSections, reconcileSections, persistSections, splitSections, outline,
applySectionOperations, parseSectionEdit, type DocumentSection } from "./document-sections";
export { DocumentValidationError, DocumentConflictError } from "./document-errors";
export type DocLink = {
slug: string; sourceSlug: string; category: string; title: string; searchText?: string;
@@ -9,6 +13,7 @@ export type DocLink = {
export type Doc = DocLink & { content: string };
export type DocInput = Omit<Doc, "revision" | "searchText">;
export type DocumentExport = { format: "tabularium-documents-v1"; documents: Doc[] };
export type DocumentReceipt = Pick<Doc, "slug" | "sourceSlug" | "revision">;
export const validSlug = /^[a-z0-9]+(?:-[a-z0-9]+)*(?:\/[a-z0-9]+(?:-[a-z0-9]+)*)*$/;
function database() {
@@ -25,58 +30,131 @@ function database() {
document_id TEXT NOT NULL REFERENCES documents(id), revision INTEGER NOT NULL,
payload TEXT NOT NULL, created_at TEXT NOT NULL, PRIMARY KEY(document_id, revision)
);
CREATE TABLE IF NOT EXISTS document_write_requests (
namespace TEXT NOT NULL, request_id TEXT NOT NULL, digest TEXT NOT NULL,
response TEXT NOT NULL, PRIMARY KEY(namespace, request_id)
);
CREATE TABLE IF NOT EXISTS document_imports (
digest TEXT PRIMARY KEY, document_count INTEGER NOT NULL, created_at TEXT NOT NULL
);
`);
sectionSchema(db);
return db;
}
export function validateDocument(input: DocInput): void {
if (!input || typeof input !== "object") throw Error("Invalid document");
if (!input || typeof input !== "object") throw new DocumentValidationError("Invalid document");
for (const key of ["slug", "sourceSlug"] as const)
if (typeof input[key] !== "string" || !validSlug.test(input[key])) throw Error(`Invalid ${key}`);
if (input.slug.split("/").some((part) => /^\d+-/.test(part))) throw Error("Use a semantic public slug");
if (typeof input[key] !== "string" || !validSlug.test(input[key])) throw new DocumentValidationError(`Invalid ${key}`);
if (input.slug.split("/").some((part) => /^\d+-/.test(part))) throw new DocumentValidationError("Use a semantic public slug");
for (const key of ["title", "category", "collection", "group"] as const)
if (typeof input[key] !== "string" || !input[key].trim()) throw Error(`Missing ${key}`);
if (typeof input[key] !== "string" || !input[key].trim()) throw new DocumentValidationError(`Missing ${key}`);
if (typeof input.content !== "string" || typeof input.description !== "string" ||
typeof input.draft !== "boolean" || typeof input.featured !== "boolean") throw Error("Invalid document fields");
if (input.content.length > 2_000_000) throw Error("Document exceeds content limit");
typeof input.draft !== "boolean" || typeof input.featured !== "boolean") throw new DocumentValidationError("Invalid document fields");
if (input.content.length > 2_000_000) throw new DocumentValidationError("Document exceeds content limit");
const images = [
...input.content.matchAll(/!\[[^\]]*\]\(<?([^\s)>]+)/g),
...input.content.matchAll(/<img\b[^>]*\bsrc=["']([^"']+)/gi),
];
if (images.some(([, source]) => /\.(?:png|jpe?g|tiff?|bmp)(?:[?#]|$)/i.test(source)))
throw Error("Use compressed WebP images instead of original rasters");
throw new DocumentValidationError("Use compressed WebP images instead of original rasters");
}
function write(db: ReturnType<typeof database>, input: DocInput, expectedRevision: number): Doc {
function write(db: ReturnType<typeof database>, input: DocInput, expectedRevision: number, editedSections?: DocumentSection[]): Doc {
validateDocument(input);
const existing = db.prepare("SELECT id, revision FROM documents WHERE source_slug=?")
.get(input.sourceSlug) as { id: string; revision: number } | undefined;
if ((existing?.revision ?? 0) !== expectedRevision) throw Error("Document revision conflict");
const existing = db.prepare("SELECT id, revision, payload FROM documents WHERE source_slug=?")
.get(input.sourceSlug) as { id: string; revision: number; payload: string } | undefined;
if ((existing?.revision ?? 0) !== expectedRevision) throw new DocumentConflictError("Document revision conflict");
const id = existing?.id ?? randomUUID();
const previous = existing ? loadSections(db, id, (JSON.parse(existing.payload) as Doc).content) : [];
const doc: Doc = { slug: input.slug, sourceSlug: input.sourceSlug, category: input.category,
title: input.title, group: input.group, collection: input.collection, description: input.description,
draft: input.draft, featured: input.featured, content: input.content, revision: expectedRevision + 1 };
const payload = JSON.stringify(doc);
// Aliases remain reserved after renaming; collision rolls back the whole write.
db.prepare(`INSERT INTO documents VALUES (?,?,?,?,?) ON CONFLICT(id) DO UPDATE SET
slug=excluded.slug, revision=excluded.revision, payload=excluded.payload`)
.run(id, doc.slug, doc.sourceSlug, doc.revision, payload);
// Reserve both canonical and historical names before updating any rows.
for (const alias of new Set([doc.slug, doc.sourceSlug])) {
const prior = db.prepare("SELECT document_id FROM document_aliases WHERE alias=?")
.get(alias) as { document_id: string } | undefined;
if (prior && prior.document_id !== id) throw Error(`Document alias collision: ${alias}`);
db.prepare("INSERT OR IGNORE INTO document_aliases VALUES (?,?)").run(alias, id);
if (prior && prior.document_id !== id) throw new DocumentConflictError(`Document alias collision: ${alias}`);
}
db.prepare(`INSERT INTO documents VALUES (?,?,?,?,?) ON CONFLICT(id) DO UPDATE SET
slug=excluded.slug, revision=excluded.revision, payload=excluded.payload`)
.run(id, doc.slug, doc.sourceSlug, doc.revision, payload);
for (const alias of new Set([doc.slug, doc.sourceSlug]))
db.prepare("INSERT OR IGNORE INTO document_aliases VALUES (?,?)").run(alias, id);
db.prepare("INSERT INTO document_revisions VALUES (?,?,?,?)")
.run(id, doc.revision, payload, new Date().toISOString());
persistSections(db, id, doc.revision, editedSections ?? reconcileSections(doc.content, previous));
return doc;
}
// Internal summaries never enter Doc, catalog, search or browser page props.
export function readDocumentSections(slug: string, revision?: number) {
if (!validSlug.test(slug)) throw new DocumentValidationError("Invalid document slug");
if (revision !== undefined && (!Number.isSafeInteger(revision) || revision < 1)) throw new DocumentValidationError("Invalid revision");
const db = database();
try {
db.exec("BEGIN");
const row = db.prepare(`SELECT d.id,d.payload FROM documents d JOIN document_aliases a
ON a.document_id=d.id WHERE a.alias=?`).get(slug) as { id: string; payload: string } | undefined;
if (!row) { db.exec("COMMIT"); return null; }
let doc = JSON.parse(row.payload) as Doc;
let sections: DocumentSection[];
if (revision === undefined || revision === doc.revision) sections = loadSections(db, row.id, doc.content);
else {
const prior = db.prepare("SELECT payload FROM document_revisions WHERE document_id=? AND revision=?")
.get(row.id, revision) as { payload: string } | undefined;
if (!prior) { db.exec("COMMIT"); return null; }
doc = JSON.parse(prior.payload) as Doc;
const snapshot = db.prepare("SELECT payload FROM document_section_revisions WHERE document_id=? AND revision=?")
.get(row.id, revision) as { payload: string } | undefined;
sections = snapshot ? JSON.parse(snapshot.payload) as DocumentSection[] : splitSections(doc.content);
}
db.exec("COMMIT");
return { slug: doc.slug, sourceSlug: doc.sourceSlug, title: doc.title, revision: doc.revision, sections };
} finally { db.close(); }
}
export function editDocumentSections(slug: string, value: unknown, request?: { namespace: string; id: string }) {
if (!validSlug.test(slug)) throw new DocumentValidationError("Invalid document slug");
const edit = parseSectionEdit(value);
if (request && (!request.namespace || !/^[A-Za-z0-9._:-]{1,128}$/.test(request.id))) throw new DocumentValidationError("Invalid Idempotency-Key");
const digest = createHash("sha256").update(JSON.stringify({ slug, edit })).digest("hex");
const namespace = request ? request.namespace + ":sections" : undefined;
const db = database();
try {
db.exec("BEGIN IMMEDIATE");
try {
if (request) {
const prior = db.prepare("SELECT digest,response FROM document_write_requests WHERE namespace=? AND request_id=?")
.get(namespace!, request.id) as { digest: string; response: string } | undefined;
if (prior) {
if (prior.digest !== digest) throw new DocumentConflictError("Idempotency-Key was used with different content");
db.exec("COMMIT");
return { ...JSON.parse(prior.response) as { slug: string; sourceSlug: string; revision: number; sections: ReturnType<typeof outline> }, replayed: true };
}
}
const row = db.prepare(`SELECT d.id,d.payload FROM documents d JOIN document_aliases a
ON a.document_id=d.id WHERE a.alias=?`).get(slug) as { id: string; payload: string } | undefined;
if (!row) { db.exec("COMMIT"); return null; }
const doc = JSON.parse(row.payload) as Doc;
if (doc.revision !== edit.revision) throw new DocumentConflictError("Document revision conflict");
const previous = loadSections(db, row.id, doc.content);
const sections = applySectionOperations(previous, edit.operations);
// Preserve the initial outline when upgrading an existing document for the first time.
db.prepare("INSERT OR IGNORE INTO document_section_revisions VALUES (?,?,?)").run(row.id, doc.revision, JSON.stringify(previous));
const saved = write(db, { ...doc, content: sections.map((s) => s.content).join("") }, doc.revision, sections);
const receipt = { slug: saved.slug, sourceSlug: saved.sourceSlug, revision: saved.revision, sections: outline(sections) };
if (request) db.prepare("INSERT INTO document_write_requests VALUES (?,?,?,?)")
.run(namespace!, request.id, digest, JSON.stringify(receipt));
db.exec("COMMIT");
return { ...receipt, replayed: false };
} catch (error) { db.exec("ROLLBACK"); throw error; }
} finally { db.close(); }
}
export function saveDocument(input: DocInput, expectedRevision: number): Doc {
if (!Number.isSafeInteger(expectedRevision) || expectedRevision < 0) throw Error("Provide expected revision");
if (!Number.isSafeInteger(expectedRevision) || expectedRevision < 0) throw new DocumentValidationError("Provide expected revision");
const db = database();
try {
db.exec("BEGIN IMMEDIATE");
@@ -145,3 +223,85 @@ export function documentHistory(sourceSlug: string): Doc[] {
.map((row) => JSON.parse(row.payload) as Doc);
} finally { db.close(); }
}
// API batches commit entirely or not at all; receipts make lost-response retries safe.
export function saveDocumentBatch(documents: Doc[], request?: { namespace: string; id: string }) {
if (!Array.isArray(documents) || !documents.length || documents.length > 100)
throw new DocumentValidationError("Provide 1 to 100 documents");
const seen = new Set<string>();
for (const doc of documents) {
validateDocument(doc);
if (!Number.isSafeInteger(doc.revision) || doc.revision < 0)
throw new DocumentValidationError("Provide expected revision");
if (seen.has(doc.sourceSlug)) throw new DocumentValidationError("Duplicate sourceSlug in batch");
seen.add(doc.sourceSlug);
}
if (request && (!request.namespace || !/^[A-Za-z0-9._:-]{1,128}$/.test(request.id)))
throw new DocumentValidationError("Invalid Idempotency-Key");
const digest = createHash("sha256").update(JSON.stringify(documents)).digest("hex");
const db = database();
try {
db.exec("BEGIN IMMEDIATE");
try {
if (request) {
const prior = db.prepare("SELECT digest,response FROM document_write_requests WHERE namespace=? AND request_id=?")
.get(request.namespace, request.id) as { digest: string; response: string } | undefined;
if (prior) {
if (prior.digest !== digest) throw new DocumentConflictError("Idempotency-Key was used with different content");
db.exec("COMMIT");
return { documents: JSON.parse(prior.response) as DocumentReceipt[], replayed: true };
}
}
const receipts = documents.map((input) => {
const { slug, sourceSlug, revision } = write(db, input, input.revision);
return { slug, sourceSlug, revision };
});
if (request) db.prepare("INSERT INTO document_write_requests VALUES (?,?,?,?)")
.run(request.namespace, request.id, digest, JSON.stringify(receipts));
db.exec("COMMIT");
return { documents: receipts, replayed: false };
} catch (error) { db.exec("ROLLBACK"); throw error; }
} finally { db.close(); }
}
export function documentPage(limit = 50, offset = 0) {
if (!Number.isSafeInteger(limit) || limit < 1 || limit > 100 || !Number.isSafeInteger(offset) || offset < 0)
throw new DocumentValidationError("Invalid pagination");
const db = database();
try {
db.exec("BEGIN");
const total = (db.prepare("SELECT COUNT(*) AS n FROM documents").get() as { n: number }).n;
const documents = (db.prepare("SELECT json_remove(payload,'$.content') AS payload FROM documents ORDER BY source_slug LIMIT ? OFFSET ?")
.all(limit, offset) as { payload: string }[]).map((row) => JSON.parse(row.payload) as DocLink);
db.exec("COMMIT");
return { documents, total, limit, offset, nextOffset: offset + documents.length < total ? offset + documents.length : null };
} finally { db.close(); }
}
export function documentHistoryPage(sourceSlug: string, limit = 50, offset = 0) {
if (!Number.isSafeInteger(limit) || limit < 1 || limit > 100 || !Number.isSafeInteger(offset) || offset < 0)
throw new DocumentValidationError("Invalid pagination");
const db = database();
try {
db.exec("BEGIN");
const total = (db.prepare(`SELECT COUNT(*) AS n FROM document_revisions r JOIN documents d
ON d.id=r.document_id WHERE d.source_slug=?`).get(sourceSlug) as { n: number }).n;
const revisions = (db.prepare(`SELECT json_remove(r.payload,'$.content') AS payload, r.created_at AS createdAt
FROM document_revisions r JOIN documents d ON d.id=r.document_id
WHERE d.source_slug=? ORDER BY r.revision DESC LIMIT ? OFFSET ?`)
.all(sourceSlug, limit, offset) as { payload: string; createdAt: string }[])
.map((row) => ({ ...JSON.parse(row.payload) as DocLink, createdAt: row.createdAt }));
db.exec("COMMIT");
return { revisions, total, limit, offset, nextOffset: offset + revisions.length < total ? offset + revisions.length : null };
} finally { db.close(); }
}
export function documentRevision(sourceSlug: string, revision: number): Doc | null {
if (!Number.isSafeInteger(revision) || revision < 1) throw new DocumentValidationError("Invalid revision");
const db = database();
try {
const row = db.prepare(`SELECT r.payload FROM document_revisions r JOIN documents d
ON d.id=r.document_id WHERE d.source_slug=? AND r.revision=?`).get(sourceSlug, revision) as { payload: string } | undefined;
return row ? JSON.parse(row.payload) as Doc : null;
} finally { db.close(); }
}
+2
View File
@@ -0,0 +1,2 @@
export class DocumentValidationError extends Error {}
export class DocumentConflictError extends Error {}
+56
View File
@@ -0,0 +1,56 @@
import { createHash, randomBytes, randomUUID } from "node:crypto";
import { openSiteDatabase } from "./site-database";
export type DocumentKeyScope = "read" | "read-write";
export type DocumentKey = { id: string; name: string; scope: DocumentKeyScope;
createdAt: number; expiresAt: number; revokedAt: number | null; lastUsedAt: number | null };
function database() {
const db = openSiteDatabase();
db.exec(`CREATE TABLE IF NOT EXISTS document_api_keys (
id TEXT PRIMARY KEY, name TEXT NOT NULL, digest TEXT NOT NULL UNIQUE,
scope TEXT NOT NULL CHECK(scope IN ('read','read-write')), created_at INTEGER NOT NULL,
expires_at INTEGER NOT NULL, revoked_at INTEGER, last_used_at INTEGER
);`);
return db;
}
export function createDocumentKey(name: string, scope: DocumentKeyScope = "read-write", days = 90) {
if (!name.trim() || name.length > 100 || !["read", "read-write"].includes(scope) ||
!Number.isSafeInteger(days) || days < 1 || days > 365) throw Error("Invalid key name, scope or lifetime");
const id = randomUUID();
const token = "tdoc_" + randomBytes(32).toString("base64url");
const createdAt = Date.now();
const expiresAt = createdAt + days * 86400000;
const db = database();
try {
db.prepare("INSERT INTO document_api_keys(id,name,digest,scope,created_at,expires_at) VALUES (?,?,?,?,?,?)")
.run(id, name.trim(), createHash("sha256").update(token).digest("hex"), scope, createdAt, expiresAt);
return { id, token, name: name.trim(), scope, createdAt, expiresAt };
} finally { db.close(); }
}
export function authenticateDocumentKey(token: string | undefined, now = Date.now()) {
if (!token || !/^tdoc_[A-Za-z0-9_-]{43}$/.test(token)) return null;
const db = database();
try {
const row = db.prepare("SELECT id,scope,last_used_at AS lastUsedAt FROM document_api_keys WHERE digest=? AND revoked_at IS NULL AND expires_at>?")
.get(createHash("sha256").update(token).digest("hex"), now) as { id: string; scope: DocumentKeyScope; lastUsedAt: number | null } | undefined;
if (!row) return null;
if (row.lastUsedAt === null || row.lastUsedAt < now - 60000)
db.prepare("UPDATE document_api_keys SET last_used_at=? WHERE id=? AND (last_used_at IS NULL OR last_used_at<?)")
.run(now, row.id, now - 60000);
return { id: row.id, scope: row.scope };
} finally { db.close(); }
}
export function listDocumentKeys(): DocumentKey[] {
const db = database();
try {
return db.prepare(`SELECT id,name,scope,created_at AS createdAt,expires_at AS expiresAt,
revoked_at AS revokedAt,last_used_at AS lastUsedAt FROM document_api_keys ORDER BY created_at,id`).all() as DocumentKey[];
} finally { db.close(); }
}
export function revokeDocumentKey(id: string) {
const db = database();
try {
const result = db.prepare("UPDATE document_api_keys SET revoked_at=COALESCE(revoked_at,?) WHERE id=?").run(Date.now(), id);
if (!result.changes) throw Error("Document key not found");
} finally { db.close(); }
}
+144
View File
@@ -0,0 +1,144 @@
import { createHash, randomUUID } from "node:crypto";
import type { DatabaseSync } from "node:sqlite";
import { DocumentValidationError } from "./document-errors";
export type DocumentSection = {
id: string; title: string; content: string; summary: string;
};
export type SectionOperation =
| { op: "update"; id: string; content?: string; summary?: string }
| { op: "insert"; after: string; content: string; summary: string }
| { op: "delete"; id: string }
| { op: "move"; id: string; after: string };
export type SectionEdit = { revision: number; operations: SectionOperation[] };
const hash = (text: string) => createHash("sha256").update(text).digest("hex");
// Preserve every byte, including CRLF and separators. H3-H6 belong to their H2 chapter.
export function splitSections(content: string): DocumentSection[] {
const starts: { offset: number; title: string }[] = [{ offset: 0, title: "导言" }];
let offset = 0;
let fence: { marker: string; length: number } | undefined;
for (const line of content.match(/[^\n]*\n|[^\n]+$/g) ?? []) {
const opening = line.match(/^ {0,3}(`{3,}|~{3,})(.*?)(?:\r?\n)?$/);
if (fence) {
if (new RegExp(`^ {0,3}${fence.marker}{${fence.length},}[ \\t]*\\r?\\n?$`).test(line)) fence = undefined;
} else if (opening && !(opening[1][0] === "`" && opening[2].includes("`"))) {
fence = { marker: opening[1][0], length: opening[1].length };
} else {
const heading = line.match(/^##(?:[ \t]+(.*?))?[ \t]*\r?\n?$/);
if (heading) starts.push({ offset, title: (heading[1] ?? "").replace(/[ \t]+#+[ \t]*$/, "").trim() });
}
offset += line.length;
}
const seen = new Map<string, number>();
return starts.map((start, index) => {
const occurrence = seen.get(start.title) ?? 0;
seen.set(start.title, occurrence + 1);
return { id: index === 0 ? "preamble" : "s_" + hash(`${start.title}\0${occurrence}`).slice(0, 24),
title: start.title, content: content.slice(start.offset, starts[index + 1]?.offset), summary: "" };
});
}
export function sectionSchema(db: DatabaseSync) {
db.exec(`CREATE TABLE IF NOT EXISTS document_sections (
document_id TEXT PRIMARY KEY REFERENCES documents(id), payload TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS document_section_revisions (
document_id TEXT NOT NULL REFERENCES documents(id), revision INTEGER NOT NULL,
payload TEXT NOT NULL, PRIMARY KEY(document_id, revision)
);`);
}
export function loadSections(db: DatabaseSync, documentId: string, content: string): DocumentSection[] {
const row = db.prepare("SELECT payload FROM document_sections WHERE document_id=?")
.get(documentId) as { payload: string } | undefined;
if (!row) return splitSections(content);
const sections = JSON.parse(row.payload) as DocumentSection[];
// A legacy writer must not leave an apparently current outline over different Markdown.
return sections.map((s) => s.content).join("") === content ? sections : reconcileSections(content, sections);
}
export function reconcileSections(content: string, previous: DocumentSection[]): DocumentSection[] {
const unused = new Set(previous);
return splitSections(content).map((section) => {
const prior = section.id === "preamble" ? previous.find((p) => p.id === "preamble") :
previous.find((p) => p.id !== "preamble" && unused.has(p) && p.content === section.content) ??
previous.find((p) => p.id !== "preamble" && unused.has(p) && p.title === section.title);
if (prior) unused.delete(prior);
return { ...section, id: prior?.id ?? (section.id === "preamble" ? "preamble" : "s_" + randomUUID()),
summary: prior?.content === section.content ? prior.summary : "" };
});
}
export function persistSections(db: DatabaseSync, documentId: string, revision: number, sections: DocumentSection[]) {
const payload = JSON.stringify(sections);
db.prepare(`INSERT INTO document_sections VALUES (?,?) ON CONFLICT(document_id)
DO UPDATE SET payload=excluded.payload`).run(documentId, payload);
db.prepare("INSERT INTO document_section_revisions VALUES (?,?,?)").run(documentId, revision, payload);
}
export function outline(sections: DocumentSection[]) {
return sections.map(({ content, ...section }, position) => ({ ...section, position,
contentHash: hash(content), summaryStatus: section.summary.trim() ? "current" : "missing" }));
}
export function parseSectionEdit(value: unknown): SectionEdit {
if (!value || typeof value !== "object" || Array.isArray(value)) throw new DocumentValidationError("Expected section edit object");
const row = value as Record<string, unknown>;
if (Object.keys(row).some((key) => !["revision", "operations"].includes(key)) ||
!Number.isSafeInteger(row.revision) || (row.revision as number) < 1 || !Array.isArray(row.operations) ||
!row.operations.length || row.operations.length > 100) throw new DocumentValidationError("Provide revision and 1 to 100 operations");
for (const raw of row.operations) {
if (!raw || typeof raw !== "object" || Array.isArray(raw)) throw new DocumentValidationError("Invalid operation");
const op = raw as Record<string, unknown>;
const fields: Record<string, string[]> = { update: ["op", "id", "content", "summary"],
insert: ["op", "after", "content", "summary"], delete: ["op", "id"], move: ["op", "id", "after"] };
if (typeof op.op !== "string" || !Object.hasOwn(fields, op.op) || Object.keys(op).some((key) => !fields[op.op as string].includes(key)))
throw new DocumentValidationError("Unknown operation or field");
for (const key of fields[op.op].filter((key) => !["content", "summary", "op"].includes(key)))
if (typeof op[key] !== "string" || !(op[key] as string).length) throw new DocumentValidationError("Missing section identifier");
for (const key of ["content", "summary"])
if (op[key] !== undefined && typeof op[key] !== "string") throw new DocumentValidationError(`Invalid ${key}`);
if (op.op === "update" && op.content === undefined && op.summary === undefined) throw new DocumentValidationError("Empty update");
if (op.op === "insert" && (typeof op.content !== "string" || typeof op.summary !== "string" || !op.summary.trim()))
throw new DocumentValidationError("Insert requires content and summary");
if (typeof op.summary === "string" && (!op.summary.trim() || op.summary.length > 8000))
throw new DocumentValidationError("Summary must contain 1 to 8000 characters");
}
return { revision: row.revision as number, operations: row.operations as SectionOperation[] };
}
export function applySectionOperations(previous: DocumentSection[], operations: SectionOperation[]) {
const sections = previous.map((section) => ({ ...section }));
function index(id: string) {
const result = sections.findIndex((s) => s.id === id);
if (result < 0) throw new DocumentValidationError("Unknown section id");
return result;
}
for (const op of operations) {
if (op.op === "insert") {
sections.splice(index(op.after) + 1, 0, { id: "s_" + randomUUID(), title: "", content: op.content, summary: op.summary });
} else {
const at = index(op.id);
if (op.op === "update") {
if (op.content !== undefined && op.content !== sections[at].content) sections[at].summary = "";
if (op.content !== undefined) sections[at].content = op.content;
if (op.summary !== undefined) sections[at].summary = op.summary;
} else {
if (op.id === "preamble") throw new DocumentValidationError("Cannot delete or move the preamble");
if (op.op === "delete") sections.splice(at, 1);
else {
if (op.id === op.after) throw new DocumentValidationError("Cannot move section after itself");
index(op.after);
const [section] = sections.splice(at, 1);
sections.splice(index(op.after) + 1, 0, section);
}
}
}
}
const content = sections.map((s) => s.content).join("");
const parsed = splitSections(content);
if (parsed.length !== sections.length || parsed.some((s, i) => s.content !== sections[i].content))
throw new DocumentValidationError("Each chapter must start with one H2 heading; preserve newline boundaries and balanced fences");
return sections.map((s, i) => ({ ...s, title: parsed[i].title }));
}
+51
View File
@@ -0,0 +1,51 @@
import type { NextApiRequest, NextApiResponse } from "next";
import { authenticateDocumentKey } from "./document-keys";
import { DocumentConflictError, DocumentValidationError, validateDocument, type Doc } from "./docs";
export function authorizeDocuments(request: NextApiRequest, response: NextApiResponse, write = false) {
response.setHeader("Cache-Control", "private, no-store");
response.setHeader("X-Content-Type-Options", "nosniff");
const header = request.headers.authorization;
const token = typeof header === "string" && header.startsWith("Bearer ") ? header.slice(7) : undefined;
const key = authenticateDocumentKey(token);
if (!key) {
response.status(401).json({ error: "需要有效的文档 API Key", code: "unauthorized" });
return null;
}
if (write && key.scope !== "read-write") {
response.status(403).json({ error: "该 API Key 仅允许读取文档", code: "read_only" });
return null;
}
return key;
}
export function documentApiError(response: NextApiResponse, error: unknown) {
if (error instanceof DocumentConflictError)
return response.status(409).json({ error: error.message, code: "conflict" });
if (error instanceof DocumentValidationError)
return response.status(400).json({ error: error.message, code: "invalid_document" });
return response.status(503).json({ error: "文档数据库暂时不可用", code: "unavailable" });
}
export function parseDocument(value: unknown): Doc {
if (!value || typeof value !== "object" || Array.isArray(value)) throw new DocumentValidationError("Expected a document object");
const row = value as Record<string, unknown>;
const fields = ["slug", "sourceSlug", "title", "category", "collection", "group", "description", "draft", "featured", "content", "revision"];
if (Object.keys(row).some((key) => !fields.includes(key))) throw new DocumentValidationError("Unknown document field");
const doc = Object.fromEntries(fields.map((key) => [key, row[key]])) as Doc;
validateDocument(doc);
if (!Number.isSafeInteger(doc.revision) || doc.revision < 0) throw new DocumentValidationError("Provide expected revision");
return doc;
}
export function pagination(request: NextApiRequest) {
const number = (value: string | string[] | undefined, fallback: number) => {
if (value === undefined) return fallback;
if (typeof value !== "string" || !/^\d+$/.test(value)) throw new DocumentValidationError("Invalid pagination");
return Number(value);
};
return { limit: number(request.query.limit, 50), offset: number(request.query.offset, 0) };
}
export function idempotency(request: NextApiRequest, namespace: string) {
const id = request.headers["idempotency-key"];
if (id === undefined) return undefined;
if (typeof id !== "string" || !/^[A-Za-z0-9._:-]{1,128}$/.test(id)) throw new DocumentValidationError("Invalid Idempotency-Key");
return { namespace, id };
}
@@ -0,0 +1,22 @@
import type { NextApiRequest, NextApiResponse } from "next";
import { DocumentValidationError, saveDocumentBatch } from "@/lib/docs";
import { authorizeDocuments, documentApiError, idempotency, parseDocument } from "@/lib/documents-api";
export const config = { api: { bodyParser: { sizeLimit: "8mb" } } };
export default function handler(request: NextApiRequest, response: NextApiResponse) {
try {
const key = authorizeDocuments(request, response, true);
if (!key) return;
if (request.method !== "POST") {
response.setHeader("Allow", "POST");
return response.status(405).end();
}
const body = request.body;
if (!body || typeof body !== "object" || Array.isArray(body) ||
Object.keys(body).length !== 1 || !Array.isArray(body.documents) ||
!body.documents.length || body.documents.length > 100)
throw new DocumentValidationError("Expected { documents: [...] } with 1 to 100 documents");
const documents = body.documents.map((row: unknown) => parseDocument(row));
return response.status(200).json(saveDocumentBatch(documents, idempotency(request, key.id)));
} catch (error) { return documentApiError(response, error); }
}
@@ -0,0 +1,21 @@
import type { NextApiRequest, NextApiResponse } from "next";
import { documentPage, saveDocumentBatch } from "@/lib/docs";
import { authorizeDocuments, documentApiError, idempotency, pagination, parseDocument } from "@/lib/documents-api";
export const config = { api: { bodyParser: { sizeLimit: "8mb" } } };
export default function handler(request: NextApiRequest, response: NextApiResponse) {
try {
const key = authorizeDocuments(request, response, request.method === "POST");
if (!key) return;
if (request.method === "GET") {
const { limit, offset } = pagination(request);
return response.status(200).json(documentPage(limit, offset));
}
if (request.method === "POST") {
const result = saveDocumentBatch([parseDocument(request.body)], idempotency(request, key.id));
return response.status(200).json({ document: result.documents[0], replayed: result.replayed });
}
response.setHeader("Allow", "GET, POST");
return response.status(405).end();
} catch (error) { return documentApiError(response, error); }
}
@@ -0,0 +1,30 @@
import type { NextApiRequest, NextApiResponse } from "next";
import { DocumentValidationError, documentHistoryPage, documentRevision, readDocument } from "@/lib/docs";
import { authorizeDocuments, documentApiError, pagination } from "@/lib/documents-api";
export default function handler(request: NextApiRequest, response: NextApiResponse) {
try {
if (!authorizeDocuments(request, response)) return;
if (request.method !== "GET") {
response.setHeader("Allow", "GET");
return response.status(405).end();
}
const slug = request.query.slug;
if (!Array.isArray(slug)) throw new DocumentValidationError("Invalid document slug");
const doc = readDocument(slug.join("/"));
if (!doc) return response.status(404).json({ error: "文档不存在", code: "not_found" });
if (request.query.revision !== undefined) {
if (request.query.history !== undefined || typeof request.query.revision !== "string" || !/^\d+$/.test(request.query.revision))
throw new DocumentValidationError("Invalid revision query");
const prior = documentRevision(doc.sourceSlug, Number(request.query.revision));
if (!prior) return response.status(404).json({ error: "修订不存在", code: "not_found" });
return response.status(200).json(prior);
}
if (request.query.history !== undefined) {
if (request.query.history !== "1") throw new DocumentValidationError("Use history=1");
const { limit, offset } = pagination(request);
return response.status(200).json(documentHistoryPage(doc.sourceSlug, limit, offset));
}
return response.status(200).json(doc);
} catch (error) { return documentApiError(response, error); }
}
@@ -0,0 +1,37 @@
import type { NextApiRequest, NextApiResponse } from "next";
import { DocumentValidationError, readDocumentSections, editDocumentSections } from "@/lib/docs";
import { outline } from "@/lib/document-sections";
import { authorizeDocuments, documentApiError, idempotency } from "@/lib/documents-api";
export const config = { api: { bodyParser: { sizeLimit: "8mb" } } };
export default function handler(request: NextApiRequest, response: NextApiResponse) {
try {
const key = authorizeDocuments(request, response, request.method === "POST");
if (!key) return;
if (request.method !== "GET" && request.method !== "POST") {
response.setHeader("Allow", "GET, POST");
return response.status(405).end();
}
if (!Array.isArray(request.query.slug)) throw new DocumentValidationError("Invalid document slug");
const slug = request.query.slug.join("/");
if (Object.keys(request.query).some((k) => !["slug", "section", "revision"].includes(k)))
throw new DocumentValidationError("Unknown section query");
if (request.method === "POST") {
if (request.query.section !== undefined || request.query.revision !== undefined)
throw new DocumentValidationError("POST uses revision and operations in the body");
const saved = editDocumentSections(slug, request.body, idempotency(request, key.id));
return saved ? response.status(200).json(saved) : response.status(404).json({ code: "not_found" });
}
const revision = request.query.revision;
if (revision !== undefined && (typeof revision !== "string" || !/^[1-9]\d*$/.test(revision)))
throw new DocumentValidationError("Invalid revision");
const doc = readDocumentSections(slug, revision === undefined ? undefined : Number(revision));
if (!doc) return response.status(404).json({ code: "not_found" });
const selected = request.query.section;
if (selected === undefined) return response.status(200).json({ ...doc, sections: outline(doc.sections) });
const ids = typeof selected === "string" ? [selected] : selected;
if (!ids.length || ids.length > 100 || ids.some((id) => !id)) throw new DocumentValidationError("Select 1 to 100 sections");
if (ids.some((id) => !doc.sections.some((s) => s.id === id))) return response.status(404).json({ code: "not_found" });
return response.status(200).json({ ...doc, sections: doc.sections.filter((s) => ids.includes(s.id)) });
} catch (error) { return documentApiError(response, error); }
}
+9
View File
@@ -1,5 +1,6 @@
import { NextResponse, type NextRequest } from "next/server";
import { authConfig, SESSION_COOKIE, validSession } from "./lib/auth";
import { authenticateDocumentKey } from "./lib/document-keys";
export function proxy(request: NextRequest) {
const path = request.nextUrl.pathname;
const headers = {
@@ -31,6 +32,14 @@ export function proxy(request: NextRequest) {
? authorization.slice(7)
: undefined;
const token = bearer ?? request.cookies.get(SESSION_COOKIE)?.value;
// Document keys are scoped to the document API; they never authenticate other pages or APIs.
if (bearer && (path === "/api/documents" || path.startsWith("/api/documents/"))) {
try {
if (authenticateDocumentKey(bearer)) return NextResponse.next({ headers });
} catch {
return NextResponse.json({ error: "文档认证暂时不可用" }, { status: 503, headers });
}
}
if (validSession(token, config)) {
if (!["GET", "HEAD", "OPTIONS"].includes(request.method) && !bearer) {
try {
+258
View File
@@ -0,0 +1,258 @@
import assert from "node:assert/strict";
import { test } from "node:test";
import { mkdtempSync, rmSync, writeFileSync, readFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import type { NextApiRequest, NextApiResponse } from "next";
import documents from "../src/pages/api/documents/index";
import batch from "../src/pages/api/documents/batch";
import item from "../src/pages/api/documents/item/[...slug]";
import sections from "../src/pages/api/documents/sections/[...slug]";
import { splitSections } from "../src/lib/document-sections";
import { openSiteDatabase } from "../src/lib/site-database";
import { createDocumentKey, authenticateDocumentKey, revokeDocumentKey, listDocumentKeys } from "../src/lib/document-keys";
import { allDocuments, documentHistory, readDocument, saveDocument, readDocumentSections, type Doc } from "../src/lib/docs";
import { newAuthConfig, signSession } from "../src/lib/auth";
const input: Doc = { slug: "endor/example", sourceSlug: "00-example", title: "Example", category: "Endor",
collection: "Endor 文档", group: "讨论", description: "Description", draft: true, featured: false,
content: "# Example\n\nOriginal body", revision: 0 };
type Options = { method?: string; body?: unknown; headers?: Record<string, string>;
query?: Record<string, string | string[]> };
function call(handler: (req: NextApiRequest, res: NextApiResponse) => unknown, options: Options = {}) {
let status = 200;
let body: unknown;
const headers: Record<string, string> = {};
const response = {
status(value: number) { status = value; return this; },
setHeader(name: string, value: string) { headers[name.toLowerCase()] = value; return this; },
json(value: unknown) { body = value; return this; },
end() { return this; },
};
handler({ method: options.method ?? "GET", body: options.body, headers: options.headers ?? {},
query: options.query ?? {} } as unknown as NextApiRequest, response as unknown as NextApiResponse);
return { status, body, headers };
}
function isolated(run: (context: { directory: string; headers: Record<string, string> }) => void) {
const directory = mkdtempSync(join(tmpdir(), "tabularium-doc-api-"));
const previous = process.env.AUTH_CONFIG_FILE;
process.env.AUTH_CONFIG_FILE = join(directory, "config.json");
writeFileSync(process.env.AUTH_CONFIG_FILE, JSON.stringify(newAuthConfig()), { mode: 0o600 });
try {
const key = createDocumentKey("test uploader");
run({ directory, headers: { authorization: "Bearer " + key.token } });
} finally {
if (previous === undefined) delete process.env.AUTH_CONFIG_FILE;
else process.env.AUTH_CONFIG_FILE = previous;
rmSync(directory, { recursive: true, force: true });
}
}
test("new API requires document keys, rejects browser cookies and broad web sessions", () => isolated(() => {
assert.equal(call(documents).status, 401);
assert.equal(call(documents, { headers: { authorization: "Bearer invalid" } }).status, 401);
const config = JSON.parse(readFileSync(process.env.AUTH_CONFIG_FILE!, "utf8"));
const token = signSession(config);
assert.equal(call(documents, { headers: { authorization: "Bearer " + token } }).status, 401);
assert.equal(call(documents, { headers: { cookie: "tabularium_session=" + token } }).status, 401);
assert.deepEqual(allDocuments(), []);
}));
test("key scopes, revocation, expiry and secret-free listing are persistent", () => isolated(({ directory }) => {
const key = createDocumentKey("reader", "read", 1);
const headers = { authorization: "Bearer " + key.token };
assert.equal(call(documents, { headers }).status, 200);
assert.equal(call(documents, { method: "POST", headers, body: input }).status, 403);
assert.equal(call(batch, { method: "POST", headers, body: { documents: [input] } }).status, 403);
assert.ok(!JSON.stringify(listDocumentKeys()).includes(key.token));
assert.ok(!readFileSync(join(directory, "login-state.sqlite")).includes(Buffer.from(key.token)));
assert.equal(authenticateDocumentKey(key.token, key.expiresAt), null);
revokeDocumentKey(key.id);
assert.equal(call(documents, { headers }).status, 401);
const revoked = listDocumentKeys().find((row) => row.id === key.id);
assert.ok(revoked); assert.equal(typeof revoked.revokedAt, "number");
}));
test("single save, aliases, metadata and historical content are visible without rebuilding", () => isolated(({ headers }) => {
assert.equal(call(documents, { method: "POST", headers, body: input }).status, 200);
assert.equal((call(item, { headers, query: { slug: input.slug.split("/") } }).body as Doc).revision, 1);
const changed = { ...input, revision: 1, slug: "endor/renamed", content: "# Changed", collection: "新合集" };
assert.equal(call(documents, { method: "POST", headers, body: changed }).status, 200);
assert.equal((call(item, { headers, query: { slug: input.slug.split("/") } }).body as Doc).slug, changed.slug);
assert.equal((call(item, { headers, query: { slug: [input.sourceSlug], revision: "1" } }).body as Doc).content, input.content);
const history = call(item, { headers, query: { slug: [input.sourceSlug], history: "1", limit: "1" } });
const page = history.body as { revisions: Doc[]; total: number; nextOffset: number };
assert.equal(page.total, 2);
assert.equal(page.nextOffset, 1);
assert.equal(page.revisions[0].revision, 2);
assert.equal(page.revisions[0].content, undefined);
assert.equal(call(item, { headers, query: { slug: [input.sourceSlug], revision: "99" } }).status, 404);
assert.equal(call(item, { headers, query: { slug: [input.sourceSlug], revision: "0" } }).status, 400);
}));
test("pagination returns metadata only with strict bounds", () => isolated(({ headers }) => {
const second = { ...input, slug: "endor/second", sourceSlug: "01-second" };
assert.equal(call(batch, { method: "POST", headers, body: { documents: [input, second] } }).status, 200);
const result = call(documents, { headers, query: { limit: "1" } });
const page = result.body as { documents: Doc[]; total: number; nextOffset: number };
assert.equal(page.total, 2); assert.equal(page.nextOffset, 1);
assert.equal(page.documents[0].content, undefined);
assert.equal((call(documents, { headers, query: { limit: "1", offset: "1" } }).body as typeof page).documents[0].slug, second.slug);
const invalidQueries: Record<string, string | string[]>[] = [
{ limit: "101" }, { offset: "-1" }, { limit: ["1", "2"] }, { limit: "1e2" },
];
for (const query of invalidQueries) assert.equal(call(documents, { headers, query }).status, 400);
assert.equal(result.headers["cache-control"], "private, no-store");
}));
test("late batch conflict rolls back earlier rows and their revision history", () => isolated(({ headers }) => {
saveDocument({ ...input, slug: "endor/existing", sourceSlug: "existing" }, 0);
const conflict = { ...input, slug: "endor/existing", sourceSlug: "existing", revision: 0 };
assert.equal(call(batch, { method: "POST", headers, body: { documents: [input, conflict] } }).status, 409);
assert.equal(readDocument(input.slug), null);
assert.equal(documentHistory(input.sourceSlug).length, 0);
assert.equal(documentHistory("existing").length, 1);
assert.equal(allDocuments().length, 1);
}));
test("idempotent retries return the original receipt without creating another revision", () => isolated(({ headers }) => {
const keyed = { ...headers, "idempotency-key": "upload-1" };
const first = call(documents, { method: "POST", headers: keyed, body: input });
const second = call(documents, { method: "POST", headers: keyed, body: input });
assert.equal(first.status, 200); assert.equal(second.status, 200);
assert.equal((second.body as { replayed: boolean }).replayed, true);
assert.equal(documentHistory(input.sourceSlug).length, 1);
assert.equal(call(documents, { method: "POST", headers: keyed, body: { ...input, content: "Different" } }).status, 409);
assert.equal(call(documents, { method: "POST", headers, body: input }).status, 409);
}));
test("invalid batch members, duplicate identifiers, raw images and malformed fields never write", () => isolated(({ headers }) => {
for (const body of [{ documents: [] }, { documents: [input, input] },
{ documents: [input], unexpected: true }, { documents: Array.from({ length: 101 }, () => input) }])
assert.equal(call(batch, { method: "POST", headers, body }).status, 400);
for (const body of [{ ...input, revision: undefined }, { ...input, typo: true },
{ ...input, slug: "../private" }, { ...input, content: "![raw](/figures/original.png)" }, null])
assert.equal(call(documents, { method: "POST", headers, body }).status, 400);
assert.deepEqual(allDocuments(), []);
assert.equal(call(documents, { method: "DELETE", headers }).status, 405);
assert.equal(call(item, { headers, query: { slug: ["missing"] } }).status, 404);
}));
const chapterContent = "# Example\n\nIntroduction.\n\n## First\n\nFirst body.\n\n### Detail\n\nChild stays in chapter.\n\n## Second\n\nSecond body.\n";
type SectionResponse = { revision: number; replayed?: boolean;
sections: { id: string; title: string; summary: string; summaryStatus?: string; content?: string }[] };
const chapterQuery = { slug: input.slug.split("/") };
test("chapter parser preserves bytes, fences, duplicate headings and CRLF", () => {
const content = "# Title\r\n\r\n## Repeated\r\n~~~text\r\n## Inside fence\r\n~~~\r\n### Nested\r\n```js\r\n## Also inside\r\n````\r\n\r\n## Repeated\r\nTail";
const parts = splitSections(content);
assert.equal(parts.length, 3);
assert.notEqual(parts[1].id, parts[2].id);
assert.equal(parts.map((p) => p.content).join(""), content);
assert.equal(parts[1].title, "Repeated");
assert.ok(parts[1].content.includes("### Nested"));
assert.equal(splitSections("").length, 1);
assert.equal(splitSections("# Title\n\n- Item\n ## List detail\n\n> ## Quoted\n").length, 1);
});
test("outline reads existing Markdown without changing it and returns only selected bodies", () => isolated(({ headers }) => {
saveDocument({ ...input, content: chapterContent }, 0);
const db = openSiteDatabase();
db.exec("DELETE FROM document_sections; DELETE FROM document_section_revisions;"); db.close();
const result = call(sections, { headers, query: chapterQuery });
assert.equal(result.status, 200);
const outline = result.body as SectionResponse;
assert.equal(outline.revision, 1); assert.equal(outline.sections.length, 3);
assert.deepEqual(outline.sections.map((s) => s.title), ["导言", "First", "Second"]);
assert.ok(outline.sections.every((s) => s.content === undefined && s.summaryStatus === "missing"));
const selected = call(sections, { headers, query: { ...chapterQuery, section: outline.sections[1].id } }).body as SectionResponse;
assert.equal(selected.sections.length, 1);
assert.ok(selected.sections[0].content!.includes("### Detail"));
assert.equal(readDocument(input.slug)!.content, chapterContent);
assert.equal(call(sections, { headers, query: { ...chapterQuery, section: "missing" } }).status, 404);
assert.equal(call(sections, { headers, query: { ...chapterQuery, revision: "0" } }).status, 400);
}));
test("internal summaries persist separately and chapter updates leave other bytes untouched", () => isolated(({ headers }) => {
saveDocument({ ...input, content: chapterContent }, 0);
const before = readDocumentSections(input.slug)!;
const first = before.sections[1]; const second = before.sections[2];
const changed = first.content.replace("First body.", "Revised body.");
const edited = call(sections, { method: "POST", headers, query: chapterQuery, body: { revision: 1, operations: [
{ op: "update", id: first.id, content: changed, summary: "INTERNAL_SUMMARY_FIRST" },
{ op: "update", id: second.id, summary: "INTERNAL_SUMMARY_SECOND" },
] } });
assert.equal(edited.status, 200);
const after = readDocumentSections(input.slug)!;
assert.equal(after.revision, 2); assert.equal(after.sections[1].id, first.id);
assert.equal(after.sections[0].content, before.sections[0].content);
assert.equal(after.sections[2].content, before.sections[2].content);
assert.equal(after.sections[1].summary, "INTERNAL_SUMMARY_FIRST");
assert.equal(readDocument(input.slug)!.content, chapterContent.replace("First body.", "Revised body."));
assert.ok(!JSON.stringify(allDocuments()).includes("INTERNAL_SUMMARY"));
assert.ok(!JSON.stringify(call(item, { headers, query: chapterQuery }).body).includes("INTERNAL_SUMMARY"));
assert.ok(!JSON.stringify(call(documents, { headers }).body).includes("INTERNAL_SUMMARY"));
const history = readDocumentSections(input.slug, 1)!;
assert.equal(history.sections[1].content, first.content); assert.equal(history.sections[1].summary, "");
assert.equal(readDocumentSections(input.slug, 2)!.sections[1].summary, "INTERNAL_SUMMARY_FIRST");
}));
test("insert, move and delete chapters commit together with replay and conflict protection", () => isolated(({ headers }) => {
saveDocument({ ...input, content: chapterContent }, 0);
const before = readDocumentSections(input.slug)!;
const first = before.sections[1]; const second = before.sections[2];
const body = { revision: 1, operations: [
{ op: "move", id: second.id, after: "preamble" },
{ op: "delete", id: first.id },
{ op: "insert", after: second.id, content: "## New chapter\n\nNew body.\n", summary: "New material" },
] };
const options = { method: "POST", headers: { ...headers, "idempotency-key": "chapters-1" }, query: chapterQuery, body };
assert.equal(call(sections, options).status, 200);
const repeated = call(sections, options);
assert.equal(repeated.status, 200); assert.equal((repeated.body as SectionResponse).replayed, true);
const after = readDocumentSections(input.slug)!;
assert.deepEqual(after.sections.map((s) => s.title), ["导言", "Second", "New chapter"]);
assert.equal(after.sections[1].content, second.content);
assert.equal(after.revision, 2); assert.equal(documentHistory(input.sourceSlug).length, 2);
assert.equal(call(sections, { ...options, body: { ...body, revision: 2 } }).status, 409);
assert.equal(call(sections, { ...options, headers }).status, 409);
assert.equal(saveDocument({ ...readDocument(input.slug)!, description: "Changed metadata" }, 2).revision, 3);
assert.equal(readDocumentSections(input.slug)!.sections[2].summary, "New material");
}));
test("invalid late operation rolls back content, summary and history; read-only keys cannot edit", () => isolated(({ headers }) => {
saveDocument({ ...input, content: chapterContent }, 0);
const before = readDocumentSections(input.slug)!;
const invalid = [
[{ op: "update", id: before.sections[1].id, summary: "Should roll back" }, { op: "delete", id: "missing" }],
[{ op: "delete", id: "preamble" }],
[{ op: "move", id: before.sections[1].id, after: before.sections[1].id }],
[{ op: "update", id: before.sections[1].id, content: "## First\n\n## Extra\n" }],
[{ op: "update", id: before.sections[1].id, content: "## First\n![raw](/figures/raw.png)\n" }],
[{ op: "insert", after: "preamble", content: "## Missing summary\n" }],
[{ op: "delete", id: before.sections[1].id, unexpected: true }],
];
for (const operations of invalid) {
assert.equal(call(sections, { method: "POST", headers, query: chapterQuery, body: { revision: 1, operations } }).status, 400);
assert.deepEqual(readDocumentSections(input.slug), before);
assert.equal(documentHistory(input.sourceSlug).length, 1);
}
assert.equal(call(sections, { query: chapterQuery }).status, 401);
const reader = createDocumentKey("chapter reader", "read");
const readHeaders = { authorization: "Bearer " + reader.token };
assert.equal(call(sections, { headers: readHeaders, query: chapterQuery }).status, 200);
assert.equal(call(sections, { method: "POST", headers: readHeaders, query: chapterQuery, body: { revision: 1, operations: [] } }).status, 403);
}));
test("full-document saves invalidate changed chapter summaries and preserve unaffected summaries and ids", () => isolated(({ headers }) => {
saveDocument({ ...input, content: chapterContent }, 0);
const before = readDocumentSections(input.slug)!;
assert.equal(call(sections, { method: "POST", headers, query: chapterQuery, body: { revision: 1, operations:
before.sections.map((s) => ({ op: "update", id: s.id, summary: "Summary of " + s.title })) } }).status, 200);
saveDocument({ ...readDocument(input.slug)!, content: chapterContent.replace("First body.", "Changed externally.") }, 2);
const after = readDocumentSections(input.slug)!;
assert.equal(after.sections[1].id, before.sections[1].id);
assert.equal(after.sections[1].summary, "");
assert.equal(after.sections[2].summary, "Summary of Second");
assert.equal(readDocumentSections(input.slug, 2)!.sections[1].summary, "Summary of First");
}));
+4 -4
View File
@@ -24,7 +24,7 @@ test("research revisions share the authentication database and preserve replay l
assert.equal(consumeCode(config, "invalid"), "limited");
const initial = readResearchSnapshot();
assert.equal(initial.companies.length, 1023);
assert.equal(initial.evidence.length, 1259);
assert.equal(initial.evidence.length, 1318);
assert.equal(initial.observations.length, 189);
assert.deepEqual(initial.candidateBatches, seed.candidateBatches, "candidate gaps persist in the same database metadata");
assert.deepEqual(initial.directoryRechecks, seed.directoryRechecks, "follow-up classifications persist without rewriting the discovery audit");
@@ -65,8 +65,8 @@ test("research validation rejects broken evidence and unsafe source links", () =
test("reviews link explicit companies without turning global or excluded cases into companies", () => {
const snapshot = structuredClone(seed) as ResearchSnapshot;
assert.equal(snapshot.reviews.filter((r) => r.scope === "companies").length, 60);
assert.equal(snapshot.reviews.filter((r) => r.scope === "dataset").length, 9);
assert.equal(snapshot.reviews.filter((r) => r.scope === "excluded-candidate").length, 29);
assert.equal(snapshot.reviews.filter((r) => r.scope === "dataset").length, 10);
assert.equal(snapshot.reviews.filter((r) => r.scope === "excluded-candidate").length, 31);
const review = snapshot.reviews.find((r) => r.id === "R4")!;
assert.deepEqual(reviewCompanyIds(snapshot, review), ["AI007", "AI013", "AI021", "AI034"]);
assert.deepEqual(reviewCompanyIds(snapshot, snapshot.reviews.find((r) => r.id === "CN-R-PSI")!), []);
@@ -307,7 +307,7 @@ test("unreconciled directory financials remain source reviews instead of income
assert.ok(review.evidenceIds.includes("PH-TANA-DR"));
assert.equal(reviewCompanyIds(snapshot, review).length, 0);
assert.ok(!snapshot.observations.some((o) => o.evidence === "PH-TANA-DR"));
assert.equal(snapshot.candidateBatches!.flatMap((b) => b.records).filter((r) => r.status === "pending-year").length, 8);
assert.equal(snapshot.candidateBatches!.flatMap((b) => b.records).filter((r) => r.status === "pending-year").length, 11);
});
test("product pivots and founder histories do not become new company years or current revenue", () => {
File diff suppressed because it is too large Load Diff
+3 -3
View File
@@ -1,5 +1,5 @@
{
"version": "2026-10-30-19",
"version": "2026-10-30-22",
"scopeConfirmedAt": "2026-10-30",
"acceptance": "以确认的公开来源清单为边界,逐条核对并标明缺项。",
"provenance": "S01—S12恢复自已提交用户的《AI公司研究_来源与分类模板》来源清单;S13—S16单列后续已使用的补充证据渠道,不将使用次数当作全量覆盖。",
@@ -28,8 +28,8 @@
"techcrunch.com"
],
"initialAccess": "AI栏目可读取",
"boundary": "栏目首页选择的3条公司线索已追溯原始资料:Wabi据公开名录2025年份、Thrive Holdings据公司主页2025自述关联公司,DreamHQ成立年仍待核。Thrive控股改造与收购/有机收入边界单列;非首页全部条目或两年档案,公开分类接口与RSS普通HTTP请求返回403。 已补读2025年度档案入口、AI第二页,并从2025企业竞赛报道定向核5条:复用1条、范围外1条、待核3条;不是完整32项名单或两年全量。",
"nextCheck": "沿已验证可读的普通2025档案及AI分页继续补台账;记录未读分页,补DreamHQ、Billow、Cashew Research和Zinnia成立与主体依据,区分竞赛年、批次、产品和公司成立年。",
"boundary": "2026首页定向3项已登记。另已读2025年度档案入口与AI第二页,并完整登记一篇2025企业竞赛名单32项:复用1项、范围外25项、待核6项;每项均尝试原始链接,未知与不可读保留。单篇32项不等全部200参赛者、全部分页或两年全量;RSS/API普通请求受限。",
"nextCheck": "继续核本篇6项的成立、主体及AI主营,优先解决Sponstar原始年份冲突、域名更名与不可读项;沿普通2025档案和AI分页扩展其余来源台账,区分参赛年、批次与公司成立年。",
"limitation": "需追溯原始披露,部分文章受访问限制",
"accessRecheckedAt": "2026-10-30"
},
+1 -1
View File
@@ -1,5 +1,5 @@
{
"sourceVersion": "2026-10-30-archives-1",
"sourceVersion": "2026-10-30-battlefield-3",
"asOf": "2026-10-30",
"metric": "platform-mrr-usd",
"sampleUnit": "product-or-business",
+2 -2
View File
@@ -3,7 +3,7 @@
set -eu
cd /opt/endorphin-tabularium-docs/current
case "${1:-}" in
list|get|save|history|export|backup|import)
list|get|save|history|export|backup|import|key-create|key-list|key-revoke)
docker compose -p endorphin-tabularium-docs --env-file /etc/endorphin-tabularium-docs/production.env --env-file release.env -f compose.yml exec -T docs node apps/docs/docs-cli.cjs "$@" ;;
*) echo 'Usage: docs list | get <slug> | save < JSON | history <sourceSlug> | export <private-file> | backup <private-file> | import < JSON' >&2; exit 1 ;;
*) echo 'Usage: docs key-create <name> <private-file> [read|read-write] [days] | key-list | key-revoke <id> | list | get <slug> | save < JSON | history <sourceSlug> | export <private-file> | backup <private-file> | import < JSON' >&2; exit 1 ;;
esac
+196
View File
@@ -0,0 +1,196 @@
"""Upload documents through the database API; credentials stay in a private file."""
import argparse
import json
import os
from pathlib import Path
import stat
import sys
import time
import urllib.error
import urllib.parse
import urllib.request
import uuid
class DocumentClientError(ValueError):
pass
class NoRedirect(urllib.request.HTTPRedirectHandler):
def redirect_request(self, req, fp, code, msg, headers, newurl):
# Never forward a bearer credential to a redirected host or protocol.
return None
def server_url(value):
parsed = urllib.parse.urlsplit(value)
if (parsed.username or parsed.password or parsed.query or parsed.fragment
or parsed.path not in ('', '/') or not parsed.hostname):
raise ValueError('Provide a server origin without credentials, path, query or fragment')
if parsed.scheme != 'https' and not (
parsed.scheme == 'http' and parsed.hostname in ('localhost', '127.0.0.1', '::1')):
raise ValueError('Use HTTPS or an HTTP loopback connection through a trusted SSH tunnel')
return value.rstrip('/')
def read_token(filename):
path = Path(filename)
if stat.S_IMODE(path.stat().st_mode) & 0o077:
raise ValueError('API key file must be private (chmod 600)')
value = path.read_text().strip()
if value.startswith('{'):
value = json.loads(value)['token']
if not isinstance(value, str) or not value.startswith('tdoc_') or len(value) != 48:
raise ValueError('Invalid document API key file')
return value
def request(base, token, method, path, payload=None, request_id=None):
data = None if payload is None else json.dumps(payload, ensure_ascii=False).encode()
if data is not None and len(data) > 8 * 1024 * 1024:
raise ValueError('Request exceeds 8 MiB; split into smaller atomic batches')
headers = {'Authorization': 'Bearer ' + token, 'Accept': 'application/json'}
if data is not None:
headers['Content-Type'] = 'application/json'
headers['Idempotency-Key'] = request_id
opener = urllib.request.build_opener(NoRedirect())
for attempt in range(2):
req = urllib.request.Request(base + path, data=data, method=method, headers=headers)
try:
with opener.open(req, timeout=30) as response:
return json.load(response)
except urllib.error.HTTPError as error:
if error.code in (502, 503, 504) and attempt == 0:
error.close()
time.sleep(1)
continue
status = error.code
error.close()
explanations = {400: 'Invalid document or pagination', 401: 'Key missing, expired or revoked',
403: 'Read-only key', 404: 'Document or revision not found',
409: 'Revision or request-id conflict; read current state before retrying',
413: 'Request too large', 503: 'Database unavailable'}
raise DocumentClientError(f'HTTP {status}: ' + explanations.get(status, 'Request failed')) from None
except (urllib.error.URLError, TimeoutError):
if attempt == 0:
time.sleep(1)
continue
raise DocumentClientError('Connection failed; retry using the same request id and unchanged input') from None
def document_batches(documents, max_bytes=8 * 1024 * 1024):
if not isinstance(documents, list) or not documents:
raise DocumentClientError('Provide a nonempty documents array')
batches, chunk, size, seen = [], [], 20, set()
for document in documents:
if not isinstance(document, dict) or not isinstance(document.get('sourceSlug'), str):
raise DocumentClientError('Every document needs a sourceSlug')
if document['sourceSlug'] in seen:
raise DocumentClientError('Duplicate sourceSlug in upload')
seen.add(document['sourceSlug'])
encoded = len(json.dumps(document, ensure_ascii=False).encode()) + 2
if encoded + 20 > max_bytes:
raise DocumentClientError('A document exceeds the request limit')
if chunk and (len(chunk) == 100 or size + encoded > max_bytes):
batches.append(chunk)
chunk, size = [], 20
chunk.append(document)
size += encoded
batches.append(chunk)
return batches
def main():
os.umask(0o077)
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--url', default=os.environ.get('TABULARIUM_DOCS_URL'))
parser.add_argument('--key-file', default=os.environ.get('TABULARIUM_DOCS_KEY_FILE'))
sub = parser.add_subparsers(dest='command', required=True)
listing = sub.add_parser('list')
listing.add_argument('--limit', type=int, default=50)
listing.add_argument('--offset', type=int, default=0)
reading = sub.add_parser('get')
reading.add_argument('slug')
mode = reading.add_mutually_exclusive_group()
mode.add_argument('--history', action='store_true')
mode.add_argument('--revision', type=int)
reading.add_argument('--limit', type=int, default=50)
reading.add_argument('--offset', type=int, default=0)
sections = sub.add_parser('outline', help='Read chapter titles and internal summaries without bodies')
sections.add_argument('slug')
sections.add_argument('--revision', type=int)
section = sub.add_parser('sections', help='Read selected chapter bodies')
section.add_argument('slug')
section.add_argument('ids', nargs='+')
section.add_argument('--revision', type=int)
editing = sub.add_parser('edit-sections', help='Atomically apply chapter operations from private JSON')
editing.add_argument('slug')
editing.add_argument('file')
editing.add_argument('--request-id')
saving = sub.add_parser('save')
saving.add_argument('file', help='Private JSON file or - for stdin; one document or {documents: [...]}')
saving.add_argument('--request-id', help='Reuse this id only with the same input when retrying')
args = parser.parse_args()
if not args.url or not args.key_file:
parser.error('Set --url and --key-file or TABULARIUM_DOCS_URL and TABULARIUM_DOCS_KEY_FILE')
try:
base = server_url(args.url)
token = read_token(args.key_file)
if args.command == 'list':
result = request(base, token, 'GET', '/api/documents?' + urllib.parse.urlencode(
{'limit': args.limit, 'offset': args.offset}))
elif args.command == 'get':
path = '/api/documents/item/' + urllib.parse.quote(args.slug, safe='/')
if args.history:
path += '?' + urllib.parse.urlencode({'history': 1, 'limit': args.limit, 'offset': args.offset})
elif args.revision is not None:
path += '?' + urllib.parse.urlencode({'revision': args.revision})
result = request(base, token, 'GET', path)
elif args.command in ('outline', 'sections', 'edit-sections'):
path = '/api/documents/sections/' + urllib.parse.quote(args.slug, safe='/')
if args.command == 'edit-sections':
payload = json.load(sys.stdin) if args.file == '-' else json.loads(Path(args.file).read_text())
request_id = args.request_id or str(uuid.uuid4())
if not request_id or len(request_id) > 100 or any(c not in 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789._:-' for c in request_id):
raise ValueError('Invalid request id')
print('Request id: ' + request_id, file=sys.stderr)
result = request(base, token, 'POST', path, payload, request_id)
else:
query = []
if args.revision is not None:
query.append(('revision', args.revision))
if args.command == 'sections':
query.extend(('section', section_id) for section_id in args.ids)
if query:
path += '?' + urllib.parse.urlencode(query)
result = request(base, token, 'GET', path)
else:
payload = json.load(sys.stdin) if args.file == '-' else json.loads(Path(args.file).read_text())
request_id = args.request_id or str(uuid.uuid4())
if not request_id or len(request_id) > 100 or any(c not in 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789._:-' for c in request_id):
raise ValueError('Invalid request id')
print('Request id: ' + request_id, file=sys.stderr)
if isinstance(payload, dict) and 'documents' in payload:
if set(payload) != {'documents'}:
raise DocumentClientError('Batch payload must contain only documents')
chunks = document_batches(payload['documents'])
receipts, replayed = [], 0
for index, chunk in enumerate(chunks, 1):
batch_id = f'{request_id}:{index}'
saved = request(base, token, 'POST', '/api/documents/batch', {'documents': chunk}, batch_id)
receipts.extend(saved['documents'])
replayed += int(saved['replayed'])
print(f'Batch {index}/{len(chunks)} committed; {len(receipts)} documents total; request id: {batch_id}', file=sys.stderr)
result = {'documents': receipts, 'batches': len(chunks), 'replayedBatches': replayed, 'requestId': request_id}
else:
result = request(base, token, 'POST', '/api/documents', payload, request_id)
print(json.dumps(result, ensure_ascii=False, indent=2))
except DocumentClientError as error:
raise SystemExit(str(error)) from None
except (ValueError, OSError, KeyError, TypeError):
# Never print exception objects that could contain credentials or document data.
raise SystemExit('Document request failed. Check the URL, private key file, JSON schema and server response; credentials omitted.') from None
if __name__ == '__main__':
main()
+110
View File
@@ -0,0 +1,110 @@
"""Lightweight client tests: no HTTP server, compilation or real credentials."""
import importlib.util
import io
import json
from pathlib import Path
import tempfile
import sys
import unittest
from unittest.mock import patch
import urllib.error
import urllib.request
spec = importlib.util.spec_from_file_location('documents_client', Path(__file__).with_name('documents-client.py'))
client = importlib.util.module_from_spec(spec)
spec.loader.exec_module(client)
class ClientTests(unittest.TestCase):
def test_section_reads_select_bodies_and_revision_without_full_document_fetch(self):
argv = ['client', '--url', 'https://docs.example', '--key-file', '/private/key',
'sections', 'endor/example', 'preamble', 's_one', '--revision', '3']
with patch.object(sys, 'argv', argv), patch.object(client, 'read_token', return_value='test-token'), \
patch.object(client, 'request', return_value={'revision': 3, 'sections': []}) as request, \
patch.object(sys, 'stdout', io.StringIO()):
client.main()
request.assert_called_once_with('https://docs.example', 'test-token', 'GET',
'/api/documents/sections/endor/example?revision=3&section=preamble&section=s_one')
def test_section_edit_keeps_payload_and_retry_id(self):
payload = {'revision': 3, 'operations': [{'op': 'update', 'id': 's_one', 'summary': 'Internal summary'}]}
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / 'edit.json'
path.write_text(json.dumps(payload))
argv = ['client', '--url', 'https://docs.example', '--key-file', '/private/key',
'edit-sections', 'endor/example', str(path), '--request-id', 'edit-1']
with patch.object(sys, 'argv', argv), patch.object(client, 'read_token', return_value='test-token'), \
patch.object(client, 'request', return_value={'revision': 4}) as request, \
patch.object(sys, 'stdout', io.StringIO()), patch.object(sys, 'stderr', io.StringIO()):
client.main()
request.assert_called_once_with('https://docs.example', 'test-token', 'POST',
'/api/documents/sections/endor/example', payload, 'edit-1')
def test_only_tls_or_loopback_origins_are_accepted(self):
self.assertEqual(client.server_url('https://docs.example/'), 'https://docs.example')
self.assertEqual(client.server_url('http://127.0.0.1:18135'), 'http://127.0.0.1:18135')
for url in ['http://docs.example', 'https://user:secret@docs.example', 'file:///tmp/data',
'https://docs.example/api', 'https://docs.example/?token=secret']:
with self.assertRaises(ValueError):
client.server_url(url)
def test_key_file_permissions_and_formats(self):
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / 'key.json'
token = 'tdoc_' + 'x' * 43
path.write_text(json.dumps({'token': token}))
path.chmod(0o600)
self.assertEqual(client.read_token(path), token)
path.chmod(0o644)
with self.assertRaises(ValueError):
client.read_token(path)
def test_redirects_never_forward_bearer_credentials(self):
request = urllib.request.Request('https://docs.example', headers={'Authorization': 'Bearer test-only'})
self.assertIsNone(client.NoRedirect().redirect_request(request, None, 302, '', {}, 'https://other.example'))
def test_network_retry_preserves_idempotency_and_body(self):
calls = []
class Opener:
def open(self, request, timeout):
calls.append(request)
if len(calls) == 1:
raise urllib.error.URLError('temporary network failure')
return io.BytesIO(b'{"replayed":true}')
with patch.object(client.urllib.request, 'build_opener', return_value=Opener()), patch.object(client.time, 'sleep'):
result = client.request('https://docs.example', 'test-token', 'POST', '/api/documents', {'revision': 0}, 'request-1')
self.assertTrue(result['replayed'])
self.assertEqual(calls[0].data, calls[1].data)
self.assertEqual(calls[0].get_header('Idempotency-key'), 'request-1')
self.assertEqual(calls[1].get_header('Idempotency-key'), 'request-1')
def test_large_uploads_are_split_by_count_and_bytes(self):
documents = [{'sourceSlug': str(i), 'content': 'a' * 20} for i in range(201)]
batches = client.document_batches(documents)
self.assertEqual([len(batch) for batch in batches], [100, 100, 1])
self.assertEqual([doc for batch in batches for doc in batch], documents)
bounded = client.document_batches(documents[:5], max_bytes=150)
self.assertTrue(all(len(json.dumps({'documents': batch}, ensure_ascii=False).encode()) <= 150 for batch in bounded))
def test_duplicate_identifiers_and_oversized_items_fail_before_upload(self):
with self.assertRaises(ValueError):
client.document_batches([{'sourceSlug': 'same'}, {'sourceSlug': 'same'}])
with self.assertRaises(ValueError):
client.document_batches([{'sourceSlug': 'one', 'content': 'a' * 500}], max_bytes=100)
def test_conflict_is_reported_without_printing_server_content_or_secret(self):
class Opener:
def open(self, request, timeout):
raise urllib.error.HTTPError(request.full_url, 409, 'Conflict', {}, io.BytesIO(b'{"error":"private-content"}'))
with patch.object(client.urllib.request, 'build_opener', return_value=Opener()):
with self.assertRaisesRegex(client.DocumentClientError, 'HTTP 409') as result:
client.request('https://docs.example', 'private-token', 'POST', '/api/documents', {}, 'request-1')
self.assertNotIn('private-token', str(result.exception))
self.assertNotIn('private-content', str(result.exception))
if __name__ == '__main__':
unittest.main()