Compare commits

..

1 Commits

16 changed files with 672 additions and 2512 deletions
+1
View File
@@ -4,6 +4,7 @@
- Web 技术栈为 TypeScript、React、Tailwind CSS、Next.js Pages Router;不要切换到 App Router。 - Web 技术栈为 TypeScript、React、Tailwind CSS、Next.js Pages Router;不要切换到 App Router。
- 使用 pnpm 管理 JavaScript workspace,使用 uv 管理 `tools/figures` 的 Python 项目,保留两套锁文件。 - 使用 pnpm 管理 JavaScript workspace,使用 uv 管理 `tools/figures` 的 Python 项目,保留两套锁文件。
- 文档正文、目录元数据与修订记录以站点现有 SQLite 数据库为唯一权威源,使用文档 API 或 docs 管理工具直接更新;不得把新正文、目录 JSON、数据库文件或日常内容快照提交到仓库。历史 Git 文档只用于显式的一次性迁移,不作为运行时种子。Python 图形源码放在 `tools/figures/src/endor_figures`,可复现的 SVG 输出放在 `apps/docs/public/figures`。 - 文档正文、目录元数据与修订记录以站点现有 SQLite 数据库为唯一权威源,使用文档 API 或 docs 管理工具直接更新;不得把新正文、目录 JSON、数据库文件或日常内容快照提交到仓库。历史 Git 文档只用于显式的一次性迁移,不作为运行时种子。Python 图形源码放在 `tools/figures/src/endor_figures`,可复现的 SVG 输出放在 `apps/docs/public/figures`。
- 编辑已有文章时优先使用章节 API:先读取章节目录与内部摘要,据此选择需要读取正文的章节,再提交单章或多章操作。摘要缺失时先读对应正文并补充;修改观点前必须核对该章原文,检查读取版本与提交 revision 一致。摘要只写入内部章节数据,不放入网页正文、目录摘要或 Git;改写正文时同步更新摘要。完整操作与 Python 客户端用法见 README 的「先读摘要,再按章节编辑」。
- 未确认的架构提议明确标注为草案,不能表述为现有实现。 - 未确认的架构提议明确标注为草案,不能表述为现有实现。
- 本地开发与预览只监听 `127.0.0.1`。部署容器监听 `0.0.0.0:3000`,服务器入口默认 18135,可通过独立 production.env 配置。 - 本地开发与预览只监听 `127.0.0.1`。部署容器监听 `0.0.0.0:3000`,服务器入口默认 18135,可通过独立 production.env 配置。
- 对 gitea.entroia.com 仅使用 codex-bot,提交身份为 Codex Bot <codex-bot@gitea.entroia.local>。SSH 使用隔离配置:`ssh -F /dev/null -o IdentitiesOnly=yes -i /Users/zzyang/.codex/credentials/gitea-entroia/codex_entroia_ed25519 -p 22222`。 - 对 gitea.entroia.com 仅使用 codex-bot,提交身份为 Codex Bot <codex-bot@gitea.entroia.local>。SSH 使用隔离配置:`ssh -F /dev/null -o IdentitiesOnly=yes -i /Users/zzyang/.codex/credentials/gitea-entroia/codex_entroia_ed25519 -p 22222`。
+50
View File
@@ -75,6 +75,10 @@ sudo /opt/endorphin-tabularium-docs/docs save < "$doc_work_dir/document.json"
| `GET /api/documents/item/<slug>?revision=1` | 读取指定修订的完整正文 | | `GET /api/documents/item/<slug>?revision=1` | 读取指定修订的完整正文 |
| `POST /api/documents` | 保存一个完整文档 JSON;字段与 `docs save` 相同 | | `POST /api/documents` | 保存一个完整文档 JSON;字段与 `docs save` 相同 |
| `POST /api/documents/batch` | 保存 `{ "documents": [...] }`,每个请求 1—100 篇 | | `POST /api/documents/batch` | 保存 `{ "documents": [...] }`,每个请求 1—100 篇 |
| `GET /api/documents/sections/<slug>` | 读取章节 ID、标题和内部摘要,不返回正文 |
| `GET /api/documents/sections/<slug>?section=<id>` | 读取选中章节正文;重复 `section` 可选多章 |
| `GET /api/documents/sections/<slug>?revision=2` | 读取对应修订的章节目录;也可搭配 `section` |
| `POST /api/documents/sections/<slug>` | 原子执行章节新增、更新、删除和移动 |
写入使用 `Authorization: Bearer <文档 API Key>`。新建传 `revision: 0`;更新先读取当前全文,再使用读到的 revision。每篇正文最多 2,000,000 个 JavaScript 字符单位;每次请求最多 8 MiB。目录和修订分页的 `limit` 范围为 1—100,响应包含 `total` 和 `nextOffset`。 写入使用 `Authorization: Bearer <文档 API Key>`。新建传 `revision: 0`;更新先读取当前全文,再使用读到的 revision。每篇正文最多 2,000,000 个 JavaScript 字符单位;每次请求最多 8 MiB。目录和修订分页的 `limit` 范围为 1—100,响应包含 `total` 和 `nextOffset`。
@@ -82,6 +86,52 @@ sudo /opt/endorphin-tabularium-docs/docs save < "$doc_work_dir/document.json"
建议写入时提供 `Idempotency-Key`,由 1—128 个字母、数字或 `._:-` 组成。同一个 Key、相同请求 ID 和相同输入重试时,返回原回执并标记 `replayed: true`,不追加修订;相同 ID 改换输入会冲突。回执描述原请求,不代表之后没有其他修改;需要当前内容时重新读取文档。改过正文或 revision 后,应使用新的请求 ID。 建议写入时提供 `Idempotency-Key`,由 1—128 个字母、数字或 `._:-` 组成。同一个 Key、相同请求 ID 和相同输入重试时,返回原回执并标记 `replayed: true`,不追加修订;相同 ID 改换输入会冲突。回执描述原请求,不代表之后没有其他修改;需要当前内容时重新读取文档。改过正文或 revision 后,应使用新的请求 ID。
### 先读摘要,再按章节编辑
章节接口使用相同的文档 Key 和读写权限。编辑流程为:先读章节目录及摘要,选择需要深入的章节,再读取这些章节的正文,最后提交一组操作。不要只凭摘要修改未读正文;目录与正文响应的 `revision` 应一致,否则重新读取。接口不调用模型生成摘要:由编辑 Agent 阅读正文、撰写摘要,并在编辑请求中保存。
章节按行首 Markdown ATX 二级标题(`##`)划分,三级至六级标题随所属章节一起编辑;代码围栏内的标题不会切章。二级标题之前的文档标题及导言属于固定 ID `preamble`,可以修改但不能删除或移动。Setext 标题、带缩进或引用、列表标记的标题不作为章节边界。正文无行首二级标题时,全文作为导言处理。拆分及组合保持换行和未编辑内容原样,不自动格式化 Markdown。
目录中的每章包含 `id`、`title`、`position`、`summary`、`summaryStatus` 和 `contentHash`。摘要为编辑内部数据,单独保存在 SQLite 的 `document_sections` 和 `document_section_revisions` 中,不加入普通文档响应、网页 props、搜索索引或页面正文。完整数据库备份包含这些表;`docs export` 仍只导出当前文档,不包含内部摘要和修订。
旧文章无需重写,首次读取即可按原文返回目录,初始摘要为空,`summaryStatus` 为 `missing`。Agent 应按需读取这些章节,生成摘要并通过 `update` 保存;保存后状态为 `current`。这里的 `current` 仅表示摘要与该版正文一起保存,不保证模型的总结正确。全文接口仍兼容;未变化章节保留摘要与 ID,正文变化的章节清空摘要,等待更新。全文改名且改写章节时可能产生新 ID,因此全文保存后应重新读取目录。
章节编辑请求示例(示例 ID 需要替换为目录实际返回的值):
```json
{
"revision": 3,
"operations": [
{
"op": "update",
"id": "s_existing",
"content": "## 原有章节\n\n修改后的正文。\n\n",
"summary": "本章说明修改后的核心观点与边界。"
},
{
"op": "insert",
"after": "s_existing",
"content": "## 新章节\n\n新增论证。\n\n",
"summary": "补充上一章尚未覆盖的论证。"
}
]
}
```
`update` 可只改 `summary`,也可改正文;正文改变时若不提供新摘要,旧摘要会清空。`insert` 必须提供正文和非空摘要;摘要最多 8000 个 JavaScript 字符单位。`delete` 使用 `{ "op": "delete", "id": "s_existing" }`;`move` 使用 `{ "op": "move", "id": "s_existing", "after": "preamble" }`。按数组顺序执行,最多 100 个操作;新章 ID 由服务器生成,在回执目录中返回。章正文应从自己的 `##` 标题开始,只包含一个二级标题,并保留与下一章之间的换行;跨章代码围栏或章节边界改变会被拒绝,拆章请使用多个操作。
章节变更、内部摘要、组合后的全文和历史快照在同一事务内提交。每批操作只增加一次文章 `revision`;摘要单独更新也增加版本号。后续操作失败或全文版本过期时全部回滚。写入沿用幂等规则,回执包含新版本、章节目录和 `replayed`;章节接口与全文接口的幂等记录分开。修改章节不会改变其 ID,移动亦保留 ID;删除后的正文和摘要仍可从对应的旧修订读取。启用前的历史修订只有原文,摘要显示缺失。
Python 客户端对应命令:
```sh
python3 scripts/documents-client.py outline endor/endor-purpose-draft
python3 scripts/documents-client.py sections endor/endor-purpose-draft preamble s_existing
python3 scripts/documents-client.py edit-sections endor/endor-purpose-draft /private/path/edit.json --request-id chapter-edit-1
```
`outline` 和 `sections` 支持 `--revision`;请求文件只包含 `revision` 和 `operations`,不要把内部摘要文件提交到 Git。
### 创建、查看与撤销 Key ### 创建、查看与撤销 Key
管理员只需通过已验证的服务器连接进行 Key 管理: 管理员只需通过已验证的服务器连接进行 Key 管理:
+3 -3
View File
@@ -182,10 +182,10 @@ async function main() {
assert.equal(researchResponse.status, 200, "authenticated research API"); assert.equal(researchResponse.status, 200, "authenticated research API");
assert.ok(researchResponse.headers.get("cache-control")?.includes("no-store")); assert.ok(researchResponse.headers.get("cache-control")?.includes("no-store"));
const research = await researchResponse.json(); const research = await researchResponse.json();
assert.equal(research.companies.length, 1031); assert.equal(research.companies.length, 1023);
assert.equal(research.categories.length, 11); assert.equal(research.categories.length, 11);
assert.equal(research.evidence.length, 1424); assert.equal(research.evidence.length, 1318);
assert.equal(research.observations.length, 192); assert.equal(research.observations.length, 189);
assert.equal((await request("/api/research/ai-companies", { method: "POST", headers })).status, 405); assert.equal((await request("/api/research/ai-companies", { method: "POST", headers })).status, 405);
const researchPage = await request("/docs/other-research/ai-company-research/overview", { headers }); const researchPage = await request("/docs/other-research/ai-company-research/overview", { headers });
assert.equal(researchPage.status, 200); assert.equal(researchPage.status, 200);
@@ -7,9 +7,9 @@ import {
const pageSize = 12; const pageSize = 12;
const evidencePageSize = 30; const evidencePageSize = 30;
const revenueLabels = { self: "企业披露,待核口径", estimate: "仅第三方估算", platform: "仅支付平台记录", unknown: "暂无可用收入数字" }; const revenueLabels = { self: "公司披露,待核口径", estimate: "仅第三方估算", platform: "仅支付平台记录", unknown: "暂无可用收入数字" };
const useLabels: Record<string, string> = { const useLabels: Record<string, string> = {
platform: "支付平台记录", signal: "企业披露信号", estimate: "第三方估算", historical: "历史里程碑", platform: "支付平台记录", signal: "公司披露信号", estimate: "第三方估算", historical: "历史里程碑",
"historical-estimate": "其他时点估算", excluded: "排除收入比较", scale: "团队规模", adoption: "采用规模", "historical-estimate": "其他时点估算", excluded: "排除收入比较", scale: "团队规模", adoption: "采用规模",
funding: "融资", technical: "第三方技术评测", crosscheck: "同一里程碑的其他资料", funding: "融资", technical: "第三方技术评测", crosscheck: "同一里程碑的其他资料",
}; };
@@ -138,7 +138,7 @@ export default function AICompanyResearch() {
<label>资料深度<select value={depth} onChange={(e) => { setDepth(e.target.value); setPage(0); }}><option value="">全部资料</option><option value="case">原有个案研究</option><option value="profile">目录基础资料</option></select></label> <label>资料深度<select value={depth} onChange={(e) => { setDepth(e.target.value); setPage(0); }}><option value="">全部资料</option><option value="case">原有个案研究</option><option value="profile">目录基础资料</option></select></label>
<label>记录团队规模<select value={teamBand} onChange={(e) => { setTeamBand(e.target.value); setPage(0); }}><option value="">全部规模</option><option value="small">1—5 人</option><option value="medium">6—20 人</option><option value="large">21—50 人</option><option value="xl">至少 51 人</option><option value="unknown">未披露或分组不确定</option></select></label> <label>记录团队规模<select value={teamBand} onChange={(e) => { setTeamBand(e.target.value); setPage(0); }}><option value="">全部规模</option><option value="small">1—5 人</option><option value="medium">6—20 人</option><option value="large">21—50 人</option><option value="xl">至少 51 人</option><option value="unknown">未披露或分组不确定</option></select></label>
<label>技术验证证据<select value={technicalLevel} onChange={(e) => { setTechnicalLevel(e.target.value); setPage(0); }}><option value="">全部验证层级</option>{Object.entries(technicalReviewLabels).map(([value, label]) => <option key={value} value={value}>{label}</option>)}</select></label> <label>技术验证证据<select value={technicalLevel} onChange={(e) => { setTechnicalLevel(e.target.value); setPage(0); }}><option value="">全部验证层级</option>{Object.entries(technicalReviewLabels).map(([value, label]) => <option key={value} value={value}>{label}</option>)}</select></label>
<label>收入资料<select value={income} onChange={(e) => { setIncome(e.target.value); setPage(0); }}><option value="">全部状态</option><option value="self">找到企业披露</option><option value="estimate">仅第三方估算</option><option value="platform">仅支付平台记录</option><option value="unknown">暂无可用数字</option></select></label> <label>收入资料<select value={income} onChange={(e) => { setIncome(e.target.value); setPage(0); }}><option value="">全部状态</option><option value="self">找到公司披露</option><option value="estimate">仅第三方估算</option><option value="platform">仅支付平台记录</option><option value="unknown">暂无可用数字</option></select></label>
</div> </div>
<p className="research-muted">技术验证层级记录我们查到了什么,不是能力强弱评分。技术专项评审与业务分类复核分别记录。</p> <p className="research-muted">技术验证层级记录我们查到了什么,不是能力强弱评分。技术专项评审与业务分类复核分别记录。</p>
<div className="research-result-count" role="status">{filtered.length} 条匹配记录<button onClick={() => { setCategory(""); setQuery(""); setCohort(""); setIncome(""); setDepth(""); setTeamBand(""); setTechnicalLevel(""); setPage(0); }}>清除筛选</button></div> <div className="research-result-count" role="status">{filtered.length} 条匹配记录<button onClick={() => { setCategory(""); setQuery(""); setCohort(""); setIncome(""); setDepth(""); setTeamBand(""); setTechnicalLevel(""); setPage(0); }}>清除筛选</button></div>
@@ -154,14 +154,14 @@ export default function AICompanyResearch() {
</>} </>}
{tab === "revenue" && <> {tab === "revenue" && <>
<IncomeDistribution data={data} onSelect={setSelected} /> <IncomeDistribution data={data} onSelect={setSelected} />
<h3>企业披露与估算的历史里程碑</h3> <h3>公司披露与估算的历史里程碑</h3>
<p className="research-notice">这一组包含 {disclosedCompanies} 家公司的披露或估算信号;企业披露含公司或交易当事方公开材料,披露主体与审计状态见证据。多为异时点历史里程碑,期间收入不直接年化,ARR 与年化收入口径仍待核对,暂不计算其同期领域均值。币种未确认的记录仅显示原值,不绘制美元长度条。上面的平台MRR统计与这一组分开。</p> <p className="research-notice">这一组包含 {disclosedCompanies} 家公司的披露或估算信号,多为异时点历史里程碑,ARR 与年化收入口径仍待核对,暂不计算其同期领域均值。币种未确认的记录仅显示原值,不绘制美元长度条。上面的平台MRR统计与这一组分开。</p>
<div className="research-revenue-chart" aria-label="不同时间的公开收入里程碑,线性长度,非同行排名">{observed.map((o) => <div key={o.id}> <div className="research-revenue-chart" aria-label="不同时间的公开收入里程碑,线性长度,非同行排名">{observed.map((o) => <div key={o.id}>
<div><button onClick={() => setSelected(o.company)}>{o.name}</button><strong>{o.bound !== "=" && o.bound}{money(o.value, o.unit)}</strong></div> <div><button onClick={() => setSelected(o.company)}>{o.name}</button><strong>{o.bound !== "=" && o.bound}{money(o.value, o.unit)}</strong></div>
{o.unit === "USD" && <span className="research-bar" data-estimate={o.selection === "estimate" || undefined} aria-hidden="true"><i style={{width: `${o.value / Math.max(1, ...observed.filter((v) => v.unit === "USD").map((v) => v.value)) * 100}%`}} /></span>} {o.unit === "USD" && <span className="research-bar" data-estimate={o.selection === "estimate" || undefined} aria-hidden="true"><i style={{width: `${o.value / Math.max(1, ...observed.filter((v) => v.unit === "USD").map((v) => v.value)) * 100}%`}} /></span>}
<small>{o.kind} · {useLabels[o.selection]} · {o.timing}</small> <small>{o.kind} · {useLabels[o.selection]} · {o.timing}</small>
</div>)}</div> </div>)}</div>
<div className="research-table-wrap"><table><caption>当前研究数据中的收入可得性,目录项未逐一检索;未建立同期、同龄、同口径比较组</caption><thead><tr><th>业务分类</th><th>企业披露数</th><th>仅估算数</th><th>仅平台数</th><th>暂无可用数字</th><th>同期均值 / 离散度</th></tr></thead><tbody>{summaries.map((c) => <tr key={c.id}><td><button onClick={() => chooseCategory(c.id)}>{c.name}</button></td><td>{c.self}</td><td>{c.estimate}</td><td>{c.platform}</td><td>{c.unknown}</td><td>条件不足</td></tr>)}</tbody></table></div> <div className="research-table-wrap"><table><caption>当前研究数据中的收入可得性,目录项未逐一检索;未建立同期、同龄、同口径比较组</caption><thead><tr><th>业务分类</th><th>公司披露数</th><th>仅估算数</th><th>仅平台数</th><th>暂无可用数字</th><th>同期均值 / 离散度</th></tr></thead><tbody>{summaries.map((c) => <tr key={c.id}><td><button onClick={() => chooseCategory(c.id)}>{c.name}</button></td><td>{c.self}</td><td>{c.estimate}</td><td>{c.platform}</td><td>{c.unknown}</td><td>条件不足</td></tr>)}</tbody></table></div>
<p className="research-muted">离散度用于筛选值得核查的异常,还需排除公司年龄、口径和披露偏差;不能据此认定财务造假。</p> <p className="research-muted">离散度用于筛选值得核查的异常,还需排除公司年龄、口径和披露偏差;不能据此认定财务造假。</p>
</>} </>}
{tab === "sources" && <><p className="research-muted">来源链接保留原始位置。相同 URL 的记录可能属于同一披露,不能按记录数计算独立证据数量。</p><ul className="research-evidence-list">{data.evidence.slice(sourcePage * evidencePageSize, (sourcePage + 1) * evidencePageSize).map((e) => <Evidence key={e.id} item={e} />)}</ul>{data.evidence.length > evidencePageSize && <nav className="research-pagination" aria-label="原始证据分页"><button disabled={sourcePage === 0} onClick={() => setSourcePage((n) => n - 1)}>上一页</button><span>{sourcePage + 1} / {Math.ceil(data.evidence.length / evidencePageSize)}</span><button disabled={(sourcePage + 1) * evidencePageSize >= data.evidence.length} onClick={() => setSourcePage((n) => n + 1)}>下一页</button></nav>}</>} {tab === "sources" && <><p className="research-muted">来源链接保留原始位置。相同 URL 的记录可能属于同一披露,不能按记录数计算独立证据数量。</p><ul className="research-evidence-list">{data.evidence.slice(sourcePage * evidencePageSize, (sourcePage + 1) * evidencePageSize).map((e) => <Evidence key={e.id} item={e} />)}</ul>{data.evidence.length > evidencePageSize && <nav className="research-pagination" aria-label="原始证据分页"><button disabled={sourcePage === 0} onClick={() => setSourcePage((n) => n - 1)}>上一页</button><span>{sourcePage + 1} / {Math.ceil(data.evidence.length / evidencePageSize)}</span><button disabled={(sourcePage + 1) * evidencePageSize >= data.evidence.length} onClick={() => setSourcePage((n) => n + 1)}>下一页</button></nav>}</>}
+75 -5
View File
@@ -1,5 +1,9 @@
import { randomUUID, createHash } from "node:crypto"; import { randomUUID, createHash } from "node:crypto";
import { openSiteDatabase } from "./site-database"; import { openSiteDatabase } from "./site-database";
import { DocumentValidationError, DocumentConflictError } from "./document-errors";
import { sectionSchema, loadSections, reconcileSections, persistSections, splitSections, outline,
applySectionOperations, parseSectionEdit, type DocumentSection } from "./document-sections";
export { DocumentValidationError, DocumentConflictError } from "./document-errors";
export type DocLink = { export type DocLink = {
slug: string; sourceSlug: string; category: string; title: string; searchText?: string; slug: string; sourceSlug: string; category: string; title: string; searchText?: string;
@@ -9,8 +13,6 @@ export type DocLink = {
export type Doc = DocLink & { content: string }; export type Doc = DocLink & { content: string };
export type DocInput = Omit<Doc, "revision" | "searchText">; export type DocInput = Omit<Doc, "revision" | "searchText">;
export type DocumentExport = { format: "tabularium-documents-v1"; documents: Doc[] }; export type DocumentExport = { format: "tabularium-documents-v1"; documents: Doc[] };
export class DocumentValidationError extends Error {}
export class DocumentConflictError extends Error {}
export type DocumentReceipt = Pick<Doc, "slug" | "sourceSlug" | "revision">; export type DocumentReceipt = Pick<Doc, "slug" | "sourceSlug" | "revision">;
export const validSlug = /^[a-z0-9]+(?:-[a-z0-9]+)*(?:\/[a-z0-9]+(?:-[a-z0-9]+)*)*$/; export const validSlug = /^[a-z0-9]+(?:-[a-z0-9]+)*(?:\/[a-z0-9]+(?:-[a-z0-9]+)*)*$/;
@@ -36,6 +38,7 @@ function database() {
digest TEXT PRIMARY KEY, document_count INTEGER NOT NULL, created_at TEXT NOT NULL digest TEXT PRIMARY KEY, document_count INTEGER NOT NULL, created_at TEXT NOT NULL
); );
`); `);
sectionSchema(db);
return db; return db;
} }
@@ -57,12 +60,13 @@ export function validateDocument(input: DocInput): void {
throw new DocumentValidationError("Use compressed WebP images instead of original rasters"); throw new DocumentValidationError("Use compressed WebP images instead of original rasters");
} }
function write(db: ReturnType<typeof database>, input: DocInput, expectedRevision: number): Doc { function write(db: ReturnType<typeof database>, input: DocInput, expectedRevision: number, editedSections?: DocumentSection[]): Doc {
validateDocument(input); validateDocument(input);
const existing = db.prepare("SELECT id, revision FROM documents WHERE source_slug=?") const existing = db.prepare("SELECT id, revision, payload FROM documents WHERE source_slug=?")
.get(input.sourceSlug) as { id: string; revision: number } | undefined; .get(input.sourceSlug) as { id: string; revision: number; payload: string } | undefined;
if ((existing?.revision ?? 0) !== expectedRevision) throw new DocumentConflictError("Document revision conflict"); if ((existing?.revision ?? 0) !== expectedRevision) throw new DocumentConflictError("Document revision conflict");
const id = existing?.id ?? randomUUID(); const id = existing?.id ?? randomUUID();
const previous = existing ? loadSections(db, id, (JSON.parse(existing.payload) as Doc).content) : [];
const doc: Doc = { slug: input.slug, sourceSlug: input.sourceSlug, category: input.category, const doc: Doc = { slug: input.slug, sourceSlug: input.sourceSlug, category: input.category,
title: input.title, group: input.group, collection: input.collection, description: input.description, title: input.title, group: input.group, collection: input.collection, description: input.description,
draft: input.draft, featured: input.featured, content: input.content, revision: expectedRevision + 1 }; draft: input.draft, featured: input.featured, content: input.content, revision: expectedRevision + 1 };
@@ -80,9 +84,75 @@ function write(db: ReturnType<typeof database>, input: DocInput, expectedRevisio
db.prepare("INSERT OR IGNORE INTO document_aliases VALUES (?,?)").run(alias, id); db.prepare("INSERT OR IGNORE INTO document_aliases VALUES (?,?)").run(alias, id);
db.prepare("INSERT INTO document_revisions VALUES (?,?,?,?)") db.prepare("INSERT INTO document_revisions VALUES (?,?,?,?)")
.run(id, doc.revision, payload, new Date().toISOString()); .run(id, doc.revision, payload, new Date().toISOString());
persistSections(db, id, doc.revision, editedSections ?? reconcileSections(doc.content, previous));
return doc; return doc;
} }
// Internal summaries never enter Doc, catalog, search or browser page props.
export function readDocumentSections(slug: string, revision?: number) {
if (!validSlug.test(slug)) throw new DocumentValidationError("Invalid document slug");
if (revision !== undefined && (!Number.isSafeInteger(revision) || revision < 1)) throw new DocumentValidationError("Invalid revision");
const db = database();
try {
db.exec("BEGIN");
const row = db.prepare(`SELECT d.id,d.payload FROM documents d JOIN document_aliases a
ON a.document_id=d.id WHERE a.alias=?`).get(slug) as { id: string; payload: string } | undefined;
if (!row) { db.exec("COMMIT"); return null; }
let doc = JSON.parse(row.payload) as Doc;
let sections: DocumentSection[];
if (revision === undefined || revision === doc.revision) sections = loadSections(db, row.id, doc.content);
else {
const prior = db.prepare("SELECT payload FROM document_revisions WHERE document_id=? AND revision=?")
.get(row.id, revision) as { payload: string } | undefined;
if (!prior) { db.exec("COMMIT"); return null; }
doc = JSON.parse(prior.payload) as Doc;
const snapshot = db.prepare("SELECT payload FROM document_section_revisions WHERE document_id=? AND revision=?")
.get(row.id, revision) as { payload: string } | undefined;
sections = snapshot ? JSON.parse(snapshot.payload) as DocumentSection[] : splitSections(doc.content);
}
db.exec("COMMIT");
return { slug: doc.slug, sourceSlug: doc.sourceSlug, title: doc.title, revision: doc.revision, sections };
} finally { db.close(); }
}
export function editDocumentSections(slug: string, value: unknown, request?: { namespace: string; id: string }) {
if (!validSlug.test(slug)) throw new DocumentValidationError("Invalid document slug");
const edit = parseSectionEdit(value);
if (request && (!request.namespace || !/^[A-Za-z0-9._:-]{1,128}$/.test(request.id))) throw new DocumentValidationError("Invalid Idempotency-Key");
const digest = createHash("sha256").update(JSON.stringify({ slug, edit })).digest("hex");
const namespace = request ? request.namespace + ":sections" : undefined;
const db = database();
try {
db.exec("BEGIN IMMEDIATE");
try {
if (request) {
const prior = db.prepare("SELECT digest,response FROM document_write_requests WHERE namespace=? AND request_id=?")
.get(namespace!, request.id) as { digest: string; response: string } | undefined;
if (prior) {
if (prior.digest !== digest) throw new DocumentConflictError("Idempotency-Key was used with different content");
db.exec("COMMIT");
return { ...JSON.parse(prior.response) as { slug: string; sourceSlug: string; revision: number; sections: ReturnType<typeof outline> }, replayed: true };
}
}
const row = db.prepare(`SELECT d.id,d.payload FROM documents d JOIN document_aliases a
ON a.document_id=d.id WHERE a.alias=?`).get(slug) as { id: string; payload: string } | undefined;
if (!row) { db.exec("COMMIT"); return null; }
const doc = JSON.parse(row.payload) as Doc;
if (doc.revision !== edit.revision) throw new DocumentConflictError("Document revision conflict");
const previous = loadSections(db, row.id, doc.content);
const sections = applySectionOperations(previous, edit.operations);
// Preserve the initial outline when upgrading an existing document for the first time.
db.prepare("INSERT OR IGNORE INTO document_section_revisions VALUES (?,?,?)").run(row.id, doc.revision, JSON.stringify(previous));
const saved = write(db, { ...doc, content: sections.map((s) => s.content).join("") }, doc.revision, sections);
const receipt = { slug: saved.slug, sourceSlug: saved.sourceSlug, revision: saved.revision, sections: outline(sections) };
if (request) db.prepare("INSERT INTO document_write_requests VALUES (?,?,?,?)")
.run(namespace!, request.id, digest, JSON.stringify(receipt));
db.exec("COMMIT");
return { ...receipt, replayed: false };
} catch (error) { db.exec("ROLLBACK"); throw error; }
} finally { db.close(); }
}
export function saveDocument(input: DocInput, expectedRevision: number): Doc { export function saveDocument(input: DocInput, expectedRevision: number): Doc {
if (!Number.isSafeInteger(expectedRevision) || expectedRevision < 0) throw new DocumentValidationError("Provide expected revision"); if (!Number.isSafeInteger(expectedRevision) || expectedRevision < 0) throw new DocumentValidationError("Provide expected revision");
const db = database(); const db = database();
+2
View File
@@ -0,0 +1,2 @@
export class DocumentValidationError extends Error {}
export class DocumentConflictError extends Error {}
+144
View File
@@ -0,0 +1,144 @@
import { createHash, randomUUID } from "node:crypto";
import type { DatabaseSync } from "node:sqlite";
import { DocumentValidationError } from "./document-errors";
export type DocumentSection = {
id: string; title: string; content: string; summary: string;
};
export type SectionOperation =
| { op: "update"; id: string; content?: string; summary?: string }
| { op: "insert"; after: string; content: string; summary: string }
| { op: "delete"; id: string }
| { op: "move"; id: string; after: string };
export type SectionEdit = { revision: number; operations: SectionOperation[] };
const hash = (text: string) => createHash("sha256").update(text).digest("hex");
// Preserve every byte, including CRLF and separators. H3-H6 belong to their H2 chapter.
export function splitSections(content: string): DocumentSection[] {
const starts: { offset: number; title: string }[] = [{ offset: 0, title: "导言" }];
let offset = 0;
let fence: { marker: string; length: number } | undefined;
for (const line of content.match(/[^\n]*\n|[^\n]+$/g) ?? []) {
const opening = line.match(/^ {0,3}(`{3,}|~{3,})(.*?)(?:\r?\n)?$/);
if (fence) {
if (new RegExp(`^ {0,3}${fence.marker}{${fence.length},}[ \\t]*\\r?\\n?$`).test(line)) fence = undefined;
} else if (opening && !(opening[1][0] === "`" && opening[2].includes("`"))) {
fence = { marker: opening[1][0], length: opening[1].length };
} else {
const heading = line.match(/^##(?:[ \t]+(.*?))?[ \t]*\r?\n?$/);
if (heading) starts.push({ offset, title: (heading[1] ?? "").replace(/[ \t]+#+[ \t]*$/, "").trim() });
}
offset += line.length;
}
const seen = new Map<string, number>();
return starts.map((start, index) => {
const occurrence = seen.get(start.title) ?? 0;
seen.set(start.title, occurrence + 1);
return { id: index === 0 ? "preamble" : "s_" + hash(`${start.title}\0${occurrence}`).slice(0, 24),
title: start.title, content: content.slice(start.offset, starts[index + 1]?.offset), summary: "" };
});
}
export function sectionSchema(db: DatabaseSync) {
db.exec(`CREATE TABLE IF NOT EXISTS document_sections (
document_id TEXT PRIMARY KEY REFERENCES documents(id), payload TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS document_section_revisions (
document_id TEXT NOT NULL REFERENCES documents(id), revision INTEGER NOT NULL,
payload TEXT NOT NULL, PRIMARY KEY(document_id, revision)
);`);
}
export function loadSections(db: DatabaseSync, documentId: string, content: string): DocumentSection[] {
const row = db.prepare("SELECT payload FROM document_sections WHERE document_id=?")
.get(documentId) as { payload: string } | undefined;
if (!row) return splitSections(content);
const sections = JSON.parse(row.payload) as DocumentSection[];
// A legacy writer must not leave an apparently current outline over different Markdown.
return sections.map((s) => s.content).join("") === content ? sections : reconcileSections(content, sections);
}
export function reconcileSections(content: string, previous: DocumentSection[]): DocumentSection[] {
const unused = new Set(previous);
return splitSections(content).map((section) => {
const prior = section.id === "preamble" ? previous.find((p) => p.id === "preamble") :
previous.find((p) => p.id !== "preamble" && unused.has(p) && p.content === section.content) ??
previous.find((p) => p.id !== "preamble" && unused.has(p) && p.title === section.title);
if (prior) unused.delete(prior);
return { ...section, id: prior?.id ?? (section.id === "preamble" ? "preamble" : "s_" + randomUUID()),
summary: prior?.content === section.content ? prior.summary : "" };
});
}
export function persistSections(db: DatabaseSync, documentId: string, revision: number, sections: DocumentSection[]) {
const payload = JSON.stringify(sections);
db.prepare(`INSERT INTO document_sections VALUES (?,?) ON CONFLICT(document_id)
DO UPDATE SET payload=excluded.payload`).run(documentId, payload);
db.prepare("INSERT INTO document_section_revisions VALUES (?,?,?)").run(documentId, revision, payload);
}
export function outline(sections: DocumentSection[]) {
return sections.map(({ content, ...section }, position) => ({ ...section, position,
contentHash: hash(content), summaryStatus: section.summary.trim() ? "current" : "missing" }));
}
export function parseSectionEdit(value: unknown): SectionEdit {
if (!value || typeof value !== "object" || Array.isArray(value)) throw new DocumentValidationError("Expected section edit object");
const row = value as Record<string, unknown>;
if (Object.keys(row).some((key) => !["revision", "operations"].includes(key)) ||
!Number.isSafeInteger(row.revision) || (row.revision as number) < 1 || !Array.isArray(row.operations) ||
!row.operations.length || row.operations.length > 100) throw new DocumentValidationError("Provide revision and 1 to 100 operations");
for (const raw of row.operations) {
if (!raw || typeof raw !== "object" || Array.isArray(raw)) throw new DocumentValidationError("Invalid operation");
const op = raw as Record<string, unknown>;
const fields: Record<string, string[]> = { update: ["op", "id", "content", "summary"],
insert: ["op", "after", "content", "summary"], delete: ["op", "id"], move: ["op", "id", "after"] };
if (typeof op.op !== "string" || !Object.hasOwn(fields, op.op) || Object.keys(op).some((key) => !fields[op.op as string].includes(key)))
throw new DocumentValidationError("Unknown operation or field");
for (const key of fields[op.op].filter((key) => !["content", "summary", "op"].includes(key)))
if (typeof op[key] !== "string" || !(op[key] as string).length) throw new DocumentValidationError("Missing section identifier");
for (const key of ["content", "summary"])
if (op[key] !== undefined && typeof op[key] !== "string") throw new DocumentValidationError(`Invalid ${key}`);
if (op.op === "update" && op.content === undefined && op.summary === undefined) throw new DocumentValidationError("Empty update");
if (op.op === "insert" && (typeof op.content !== "string" || typeof op.summary !== "string" || !op.summary.trim()))
throw new DocumentValidationError("Insert requires content and summary");
if (typeof op.summary === "string" && (!op.summary.trim() || op.summary.length > 8000))
throw new DocumentValidationError("Summary must contain 1 to 8000 characters");
}
return { revision: row.revision as number, operations: row.operations as SectionOperation[] };
}
export function applySectionOperations(previous: DocumentSection[], operations: SectionOperation[]) {
const sections = previous.map((section) => ({ ...section }));
function index(id: string) {
const result = sections.findIndex((s) => s.id === id);
if (result < 0) throw new DocumentValidationError("Unknown section id");
return result;
}
for (const op of operations) {
if (op.op === "insert") {
sections.splice(index(op.after) + 1, 0, { id: "s_" + randomUUID(), title: "", content: op.content, summary: op.summary });
} else {
const at = index(op.id);
if (op.op === "update") {
if (op.content !== undefined && op.content !== sections[at].content) sections[at].summary = "";
if (op.content !== undefined) sections[at].content = op.content;
if (op.summary !== undefined) sections[at].summary = op.summary;
} else {
if (op.id === "preamble") throw new DocumentValidationError("Cannot delete or move the preamble");
if (op.op === "delete") sections.splice(at, 1);
else {
if (op.id === op.after) throw new DocumentValidationError("Cannot move section after itself");
index(op.after);
const [section] = sections.splice(at, 1);
sections.splice(index(op.after) + 1, 0, section);
}
}
}
}
const content = sections.map((s) => s.content).join("");
const parsed = splitSections(content);
if (parsed.length !== sections.length || parsed.some((s, i) => s.content !== sections[i].content))
throw new DocumentValidationError("Each chapter must start with one H2 heading; preserve newline boundaries and balanced fences");
return sections.map((s, i) => ({ ...s, title: parsed[i].title }));
}
@@ -0,0 +1,37 @@
import type { NextApiRequest, NextApiResponse } from "next";
import { DocumentValidationError, readDocumentSections, editDocumentSections } from "@/lib/docs";
import { outline } from "@/lib/document-sections";
import { authorizeDocuments, documentApiError, idempotency } from "@/lib/documents-api";
export const config = { api: { bodyParser: { sizeLimit: "8mb" } } };
export default function handler(request: NextApiRequest, response: NextApiResponse) {
try {
const key = authorizeDocuments(request, response, request.method === "POST");
if (!key) return;
if (request.method !== "GET" && request.method !== "POST") {
response.setHeader("Allow", "GET, POST");
return response.status(405).end();
}
if (!Array.isArray(request.query.slug)) throw new DocumentValidationError("Invalid document slug");
const slug = request.query.slug.join("/");
if (Object.keys(request.query).some((k) => !["slug", "section", "revision"].includes(k)))
throw new DocumentValidationError("Unknown section query");
if (request.method === "POST") {
if (request.query.section !== undefined || request.query.revision !== undefined)
throw new DocumentValidationError("POST uses revision and operations in the body");
const saved = editDocumentSections(slug, request.body, idempotency(request, key.id));
return saved ? response.status(200).json(saved) : response.status(404).json({ code: "not_found" });
}
const revision = request.query.revision;
if (revision !== undefined && (typeof revision !== "string" || !/^[1-9]\d*$/.test(revision)))
throw new DocumentValidationError("Invalid revision");
const doc = readDocumentSections(slug, revision === undefined ? undefined : Number(revision));
if (!doc) return response.status(404).json({ code: "not_found" });
const selected = request.query.section;
if (selected === undefined) return response.status(200).json({ ...doc, sections: outline(doc.sections) });
const ids = typeof selected === "string" ? [selected] : selected;
if (!ids.length || ids.length > 100 || ids.some((id) => !id)) throw new DocumentValidationError("Select 1 to 100 sections");
if (ids.some((id) => !doc.sections.some((s) => s.id === id))) return response.status(404).json({ code: "not_found" });
return response.status(200).json({ ...doc, sections: doc.sections.filter((s) => ids.includes(s.id)) });
} catch (error) { return documentApiError(response, error); }
}
+123 -1
View File
@@ -7,8 +7,11 @@ import type { NextApiRequest, NextApiResponse } from "next";
import documents from "../src/pages/api/documents/index"; import documents from "../src/pages/api/documents/index";
import batch from "../src/pages/api/documents/batch"; import batch from "../src/pages/api/documents/batch";
import item from "../src/pages/api/documents/item/[...slug]"; import item from "../src/pages/api/documents/item/[...slug]";
import sections from "../src/pages/api/documents/sections/[...slug]";
import { splitSections } from "../src/lib/document-sections";
import { openSiteDatabase } from "../src/lib/site-database";
import { createDocumentKey, authenticateDocumentKey, revokeDocumentKey, listDocumentKeys } from "../src/lib/document-keys"; import { createDocumentKey, authenticateDocumentKey, revokeDocumentKey, listDocumentKeys } from "../src/lib/document-keys";
import { allDocuments, documentHistory, readDocument, saveDocument, type Doc } from "../src/lib/docs"; import { allDocuments, documentHistory, readDocument, saveDocument, readDocumentSections, type Doc } from "../src/lib/docs";
import { newAuthConfig, signSession } from "../src/lib/auth"; import { newAuthConfig, signSession } from "../src/lib/auth";
const input: Doc = { slug: "endor/example", sourceSlug: "00-example", title: "Example", category: "Endor", const input: Doc = { slug: "endor/example", sourceSlug: "00-example", title: "Example", category: "Endor",
@@ -134,3 +137,122 @@ test("invalid batch members, duplicate identifiers, raw images and malformed fie
assert.equal(call(documents, { method: "DELETE", headers }).status, 405); assert.equal(call(documents, { method: "DELETE", headers }).status, 405);
assert.equal(call(item, { headers, query: { slug: ["missing"] } }).status, 404); assert.equal(call(item, { headers, query: { slug: ["missing"] } }).status, 404);
})); }));
const chapterContent = "# Example\n\nIntroduction.\n\n## First\n\nFirst body.\n\n### Detail\n\nChild stays in chapter.\n\n## Second\n\nSecond body.\n";
type SectionResponse = { revision: number; replayed?: boolean;
sections: { id: string; title: string; summary: string; summaryStatus?: string; content?: string }[] };
const chapterQuery = { slug: input.slug.split("/") };
test("chapter parser preserves bytes, fences, duplicate headings and CRLF", () => {
const content = "# Title\r\n\r\n## Repeated\r\n~~~text\r\n## Inside fence\r\n~~~\r\n### Nested\r\n```js\r\n## Also inside\r\n````\r\n\r\n## Repeated\r\nTail";
const parts = splitSections(content);
assert.equal(parts.length, 3);
assert.notEqual(parts[1].id, parts[2].id);
assert.equal(parts.map((p) => p.content).join(""), content);
assert.equal(parts[1].title, "Repeated");
assert.ok(parts[1].content.includes("### Nested"));
assert.equal(splitSections("").length, 1);
assert.equal(splitSections("# Title\n\n- Item\n ## List detail\n\n> ## Quoted\n").length, 1);
});
test("outline reads existing Markdown without changing it and returns only selected bodies", () => isolated(({ headers }) => {
saveDocument({ ...input, content: chapterContent }, 0);
const db = openSiteDatabase();
db.exec("DELETE FROM document_sections; DELETE FROM document_section_revisions;"); db.close();
const result = call(sections, { headers, query: chapterQuery });
assert.equal(result.status, 200);
const outline = result.body as SectionResponse;
assert.equal(outline.revision, 1); assert.equal(outline.sections.length, 3);
assert.deepEqual(outline.sections.map((s) => s.title), ["导言", "First", "Second"]);
assert.ok(outline.sections.every((s) => s.content === undefined && s.summaryStatus === "missing"));
const selected = call(sections, { headers, query: { ...chapterQuery, section: outline.sections[1].id } }).body as SectionResponse;
assert.equal(selected.sections.length, 1);
assert.ok(selected.sections[0].content!.includes("### Detail"));
assert.equal(readDocument(input.slug)!.content, chapterContent);
assert.equal(call(sections, { headers, query: { ...chapterQuery, section: "missing" } }).status, 404);
assert.equal(call(sections, { headers, query: { ...chapterQuery, revision: "0" } }).status, 400);
}));
test("internal summaries persist separately and chapter updates leave other bytes untouched", () => isolated(({ headers }) => {
saveDocument({ ...input, content: chapterContent }, 0);
const before = readDocumentSections(input.slug)!;
const first = before.sections[1]; const second = before.sections[2];
const changed = first.content.replace("First body.", "Revised body.");
const edited = call(sections, { method: "POST", headers, query: chapterQuery, body: { revision: 1, operations: [
{ op: "update", id: first.id, content: changed, summary: "INTERNAL_SUMMARY_FIRST" },
{ op: "update", id: second.id, summary: "INTERNAL_SUMMARY_SECOND" },
] } });
assert.equal(edited.status, 200);
const after = readDocumentSections(input.slug)!;
assert.equal(after.revision, 2); assert.equal(after.sections[1].id, first.id);
assert.equal(after.sections[0].content, before.sections[0].content);
assert.equal(after.sections[2].content, before.sections[2].content);
assert.equal(after.sections[1].summary, "INTERNAL_SUMMARY_FIRST");
assert.equal(readDocument(input.slug)!.content, chapterContent.replace("First body.", "Revised body."));
assert.ok(!JSON.stringify(allDocuments()).includes("INTERNAL_SUMMARY"));
assert.ok(!JSON.stringify(call(item, { headers, query: chapterQuery }).body).includes("INTERNAL_SUMMARY"));
assert.ok(!JSON.stringify(call(documents, { headers }).body).includes("INTERNAL_SUMMARY"));
const history = readDocumentSections(input.slug, 1)!;
assert.equal(history.sections[1].content, first.content); assert.equal(history.sections[1].summary, "");
assert.equal(readDocumentSections(input.slug, 2)!.sections[1].summary, "INTERNAL_SUMMARY_FIRST");
}));
test("insert, move and delete chapters commit together with replay and conflict protection", () => isolated(({ headers }) => {
saveDocument({ ...input, content: chapterContent }, 0);
const before = readDocumentSections(input.slug)!;
const first = before.sections[1]; const second = before.sections[2];
const body = { revision: 1, operations: [
{ op: "move", id: second.id, after: "preamble" },
{ op: "delete", id: first.id },
{ op: "insert", after: second.id, content: "## New chapter\n\nNew body.\n", summary: "New material" },
] };
const options = { method: "POST", headers: { ...headers, "idempotency-key": "chapters-1" }, query: chapterQuery, body };
assert.equal(call(sections, options).status, 200);
const repeated = call(sections, options);
assert.equal(repeated.status, 200); assert.equal((repeated.body as SectionResponse).replayed, true);
const after = readDocumentSections(input.slug)!;
assert.deepEqual(after.sections.map((s) => s.title), ["导言", "Second", "New chapter"]);
assert.equal(after.sections[1].content, second.content);
assert.equal(after.revision, 2); assert.equal(documentHistory(input.sourceSlug).length, 2);
assert.equal(call(sections, { ...options, body: { ...body, revision: 2 } }).status, 409);
assert.equal(call(sections, { ...options, headers }).status, 409);
assert.equal(saveDocument({ ...readDocument(input.slug)!, description: "Changed metadata" }, 2).revision, 3);
assert.equal(readDocumentSections(input.slug)!.sections[2].summary, "New material");
}));
test("invalid late operation rolls back content, summary and history; read-only keys cannot edit", () => isolated(({ headers }) => {
saveDocument({ ...input, content: chapterContent }, 0);
const before = readDocumentSections(input.slug)!;
const invalid = [
[{ op: "update", id: before.sections[1].id, summary: "Should roll back" }, { op: "delete", id: "missing" }],
[{ op: "delete", id: "preamble" }],
[{ op: "move", id: before.sections[1].id, after: before.sections[1].id }],
[{ op: "update", id: before.sections[1].id, content: "## First\n\n## Extra\n" }],
[{ op: "update", id: before.sections[1].id, content: "## First\n![raw](/figures/raw.png)\n" }],
[{ op: "insert", after: "preamble", content: "## Missing summary\n" }],
[{ op: "delete", id: before.sections[1].id, unexpected: true }],
];
for (const operations of invalid) {
assert.equal(call(sections, { method: "POST", headers, query: chapterQuery, body: { revision: 1, operations } }).status, 400);
assert.deepEqual(readDocumentSections(input.slug), before);
assert.equal(documentHistory(input.sourceSlug).length, 1);
}
assert.equal(call(sections, { query: chapterQuery }).status, 401);
const reader = createDocumentKey("chapter reader", "read");
const readHeaders = { authorization: "Bearer " + reader.token };
assert.equal(call(sections, { headers: readHeaders, query: chapterQuery }).status, 200);
assert.equal(call(sections, { method: "POST", headers: readHeaders, query: chapterQuery, body: { revision: 1, operations: [] } }).status, 403);
}));
test("full-document saves invalidate changed chapter summaries and preserve unaffected summaries and ids", () => isolated(({ headers }) => {
saveDocument({ ...input, content: chapterContent }, 0);
const before = readDocumentSections(input.slug)!;
assert.equal(call(sections, { method: "POST", headers, query: chapterQuery, body: { revision: 1, operations:
before.sections.map((s) => ({ op: "update", id: s.id, summary: "Summary of " + s.title })) } }).status, 200);
saveDocument({ ...readDocument(input.slug)!, content: chapterContent.replace("First body.", "Changed externally.") }, 2);
const after = readDocumentSections(input.slug)!;
assert.equal(after.sections[1].id, before.sections[1].id);
assert.equal(after.sections[1].summary, "");
assert.equal(after.sections[2].summary, "Summary of Second");
assert.equal(readDocumentSections(input.slug, 2)!.sections[1].summary, "Summary of First");
}));
+12 -12
View File
@@ -23,9 +23,9 @@ test("research revisions share the authentication database and preserve replay l
for (let i = 0; i < 20; i++) consumeCode(config, "invalid"); for (let i = 0; i < 20; i++) consumeCode(config, "invalid");
assert.equal(consumeCode(config, "invalid"), "limited"); assert.equal(consumeCode(config, "invalid"), "limited");
const initial = readResearchSnapshot(); const initial = readResearchSnapshot();
assert.equal(initial.companies.length, 1031); assert.equal(initial.companies.length, 1023);
assert.equal(initial.evidence.length, 1424); assert.equal(initial.evidence.length, 1318);
assert.equal(initial.observations.length, 192); assert.equal(initial.observations.length, 189);
assert.deepEqual(initial.candidateBatches, seed.candidateBatches, "candidate gaps persist in the same database metadata"); assert.deepEqual(initial.candidateBatches, seed.candidateBatches, "candidate gaps persist in the same database metadata");
assert.deepEqual(initial.directoryRechecks, seed.directoryRechecks, "follow-up classifications persist without rewriting the discovery audit"); assert.deepEqual(initial.directoryRechecks, seed.directoryRechecks, "follow-up classifications persist without rewriting the discovery audit");
assert.deepEqual(readResearchSnapshot(), initial, "reopening does not duplicate records"); assert.deepEqual(readResearchSnapshot(), initial, "reopening does not duplicate records");
@@ -64,9 +64,9 @@ test("research validation rejects broken evidence and unsafe source links", () =
test("reviews link explicit companies without turning global or excluded cases into companies", () => { test("reviews link explicit companies without turning global or excluded cases into companies", () => {
const snapshot = structuredClone(seed) as ResearchSnapshot; const snapshot = structuredClone(seed) as ResearchSnapshot;
assert.equal(snapshot.reviews.filter((r) => r.scope === "companies").length, 76); assert.equal(snapshot.reviews.filter((r) => r.scope === "companies").length, 60);
assert.equal(snapshot.reviews.filter((r) => r.scope === "dataset").length, 10); assert.equal(snapshot.reviews.filter((r) => r.scope === "dataset").length, 10);
assert.equal(snapshot.reviews.filter((r) => r.scope === "excluded-candidate").length, 38); assert.equal(snapshot.reviews.filter((r) => r.scope === "excluded-candidate").length, 31);
const review = snapshot.reviews.find((r) => r.id === "R4")!; const review = snapshot.reviews.find((r) => r.id === "R4")!;
assert.deepEqual(reviewCompanyIds(snapshot, review), ["AI007", "AI013", "AI021", "AI034"]); assert.deepEqual(reviewCompanyIds(snapshot, review), ["AI007", "AI013", "AI021", "AI034"]);
assert.deepEqual(reviewCompanyIds(snapshot, snapshot.reviews.find((r) => r.id === "CN-R-PSI")!), []); assert.deepEqual(reviewCompanyIds(snapshot, snapshot.reviews.find((r) => r.id === "CN-R-PSI")!), []);
@@ -85,11 +85,11 @@ test("reviews link explicit companies without turning global or excluded cases i
test("disclosure counts exclude client income, biographies, targets and duplicate milestones", () => { test("disclosure counts exclude client income, biographies, targets and duplicate milestones", () => {
const snapshot = seed as ResearchSnapshot; const snapshot = seed as ResearchSnapshot;
const summary = categorySummary(snapshot); const summary = categorySummary(snapshot);
assert.equal(summary.reduce((n, c) => n + c.count, 0), 1031); assert.equal(summary.reduce((n, c) => n + c.count, 0), 1023);
assert.equal(summary.reduce((n, c) => n + c.self, 0), 28); assert.equal(summary.reduce((n, c) => n + c.self, 0), 26);
assert.equal(summary.reduce((n, c) => n + c.estimate, 0), 2); assert.equal(summary.reduce((n, c) => n + c.estimate, 0), 1);
assert.equal(summary.reduce((n, c) => n + c.platform, 0), 16); assert.equal(summary.reduce((n, c) => n + c.platform, 0), 16);
assert.equal(summary.reduce((n, c) => n + c.unknown, 0), 985); assert.equal(summary.reduce((n, c) => n + c.unknown, 0), 980);
assert.equal(snapshot.companies.find((c) => c.name === "Rork")?.revenueStatus, "unknown"); assert.equal(snapshot.companies.find((c) => c.name === "Rork")?.revenueStatus, "unknown");
assert.equal(snapshot.companies.find((c) => c.name === "Result")?.revenueStatus, "unknown"); assert.equal(snapshot.companies.find((c) => c.name === "Result")?.revenueStatus, "unknown");
assert.ok(snapshot.observations.filter((o) => o.selection === "signal").every((o) => o.scope === "公司整体")); assert.ok(snapshot.observations.filter((o) => o.selection === "signal").every((o) => o.scope === "公司整体"));
@@ -97,8 +97,8 @@ test("disclosure counts exclude client income, biographies, targets and duplicat
test("technical reviews keep accessible materials separate from tested capabilities", () => { test("technical reviews keep accessible materials separate from tested capabilities", () => {
const snapshot = structuredClone(seed) as ResearchSnapshot; const snapshot = structuredClone(seed) as ResearchSnapshot;
assert.equal(snapshot.companies.filter((c) => c.technicalReview).length, 30); assert.equal(snapshot.companies.filter((c) => c.technicalReview).length, 16);
assert.equal(snapshot.companies.filter((c) => c.technicalReview?.level === "public-artifact").length, 7); assert.equal(snapshot.companies.filter((c) => c.technicalReview?.level === "public-artifact").length, 6);
assert.equal(snapshot.companies.filter((c) => c.technicalReview?.level === "independent-test").length, 1); assert.equal(snapshot.companies.filter((c) => c.technicalReview?.level === "independent-test").length, 1);
assert.equal(snapshot.companies.find((c) => c.id === "AI051")?.revenueStatus, "unknown"); assert.equal(snapshot.companies.find((c) => c.id === "AI051")?.revenueStatus, "unknown");
assert.match(snapshot.companies.find((c) => c.id === "AI051")!.technicalReview!.testScope!, /Sep 2026/); assert.match(snapshot.companies.find((c) => c.id === "AI051")!.technicalReview!.testScope!, /Sep 2026/);
@@ -307,7 +307,7 @@ test("unreconciled directory financials remain source reviews instead of income
assert.ok(review.evidenceIds.includes("PH-TANA-DR")); assert.ok(review.evidenceIds.includes("PH-TANA-DR"));
assert.equal(reviewCompanyIds(snapshot, review).length, 0); assert.equal(reviewCompanyIds(snapshot, review).length, 0);
assert.ok(!snapshot.observations.some((o) => o.evidence === "PH-TANA-DR")); assert.ok(!snapshot.observations.some((o) => o.evidence === "PH-TANA-DR"));
assert.equal(snapshot.candidateBatches!.flatMap((b) => b.records).filter((r) => r.status === "pending-year").length, 13); assert.equal(snapshot.candidateBatches!.flatMap((b) => b.records).filter((r) => r.status === "pending-year").length, 11);
}); });
test("product pivots and founder histories do not become new company years or current revenue", () => { test("product pivots and founder histories do not become new company years or current revenue", () => {
File diff suppressed because it is too large Load Diff
+13 -16
View File
@@ -1,5 +1,5 @@
{ {
"version": "2026-10-30-30", "version": "2026-10-30-22",
"scopeConfirmedAt": "2026-10-30", "scopeConfirmedAt": "2026-10-30",
"acceptance": "以确认的公开来源清单为边界,逐条核对并标明缺项。", "acceptance": "以确认的公开来源清单为边界,逐条核对并标明缺项。",
"provenance": "S01—S12恢复自已提交用户的《AI公司研究_来源与分类模板》来源清单;S13—S16单列后续已使用的补充证据渠道,不将使用次数当作全量覆盖。", "provenance": "S01—S12恢复自已提交用户的《AI公司研究_来源与分类模板》来源清单;S13—S16单列后续已使用的补充证据渠道,不将使用次数当作全量覆盖。",
@@ -15,7 +15,7 @@
], ],
"initialAccess": "官网目录可访问", "initialAccess": "官网目录可访问",
"boundary": "已读取八个2025—2026批次镜像候选池1428条,再核YC官方页面;不声称覆盖全部YC历年或未公开公司。", "boundary": "已读取八个2025—2026批次镜像候选池1428条,再核YC官方页面;不声称覆盖全部YC历年或未公开公司。",
"nextCheck": "原新增目录中22条深化为个案并保留来源;当前基础项175条主营分类人工复核。继续核其余资料与财务缺项,估算、卖家收益展示和数据供给方估价不当公司ARR。", "nextCheck": "当前快照三条原待分类经官网方向补证归类;继续复核规则初分、具体产品与财务缺项,不把领域粗分类当独立能力或收入验证。",
"limitation": "批次不等于成立年;不代表全部公司" "limitation": "批次不等于成立年;不代表全部公司"
}, },
{ {
@@ -28,8 +28,8 @@
"techcrunch.com" "techcrunch.com"
], ],
"initialAccess": "AI栏目可读取", "initialAccess": "AI栏目可读取",
"boundary": "2026首页定向3项已登记,另从AI第3—4页登记5项:3项关联公司、1项范围外、1项待核;不等全部卡片核验。另已读2025年度档案入口与AI第二页,并完整登记一篇2025企业竞赛名单32项:复用1项、范围外29项、待核2项;每项均尝试原始链接,未知与不可读保留。单篇32项不等全部200参赛者、全部分页或两年全量;RSS/API普通请求受限。 另从AI第5—6页定向4项:2项原始资料归范围外,2项原始成立/年份待核;不是全部卡片、参赛者或档案。", "boundary": "2026首页定向3项已登记。另已读2025年度档案入口与AI第二页,并完整登记一篇2025企业竞赛名单32项:复用1项、范围外25项、待核6项;每项均尝试原始链接,未知与不可读保留。单篇32项不等全部200参赛者、全部分页或两年全量;RSS/API普通请求受限。",
"nextCheck": "继续核本篇2项的成立、主体及AI主营,优先解决Billow批次与发现年份时序、Sponstar原始年份冲突;既有业务的法律沿革及不可读项仍留缺;沿普通2025档案和AI分页扩展其余来源台账,区分参赛年、批次与公司成立年。 继续核Laplace原始年份及Eka精确注册主体;Recursive的2025/2026年份冲突保留。 继续核Honeyjar主字段与转载的年份差异、Blueprints原始可读资料与旧建筑业务主体;不把包价、演示金额或建设项目价值当ARR。", "nextCheck": "继续核本篇6项的成立、主体及AI主营,优先解决Sponstar原始年份冲突、域名更名与不可读项;沿普通2025档案和AI分页扩展其余来源台账,区分参赛年、批次与公司成立年。",
"limitation": "需追溯原始披露,部分文章受访问限制", "limitation": "需追溯原始披露,部分文章受访问限制",
"accessRecheckedAt": "2026-10-30" "accessRecheckedAt": "2026-10-30"
}, },
@@ -44,7 +44,7 @@
"app.dealroom.co" "app.dealroom.co"
], ],
"initialAccess": "官网可读取", "initialAccess": "官网可读取",
"boundary": "已读Tana、Wabi及Laplace公开公司档案:Tana财务字段不一致留缺,Wabi2025字段与官方业务关联入表,Laplace未取得成立字段而保留候选;尚未取得两年成立筛选结果集,三个个案不等目录全量。", "boundary": "已读Tana与Wabi公开公司档案:前者财务字段不一致留缺,后者2025成立字段与官方业务关联入表;尚未取得两年成立筛选结果集,两个个案不等于目录全量。",
"nextCheck": "确认公开可取得的两年成立筛选清单;逐项追溯财务原始凭证及字段口径,受权限限制保留缺项。", "nextCheck": "确认公开可取得的两年成立筛选清单;逐项追溯财务原始凭证及字段口径,受权限限制保留缺项。",
"limitation": "完整档案需权限;概览及财务表不能未经勾稽直接导入ARR", "limitation": "完整档案需权限;概览及财务表不能未经勾稽直接导入ARR",
"accessRecheckedAt": "2026-10-30" "accessRecheckedAt": "2026-10-30"
@@ -214,16 +214,15 @@
"name": "36氪", "name": "36氪",
"url": "https://www.36kr.com/", "url": "https://www.36kr.com/",
"role": "reporting", "role": "reporting",
"status": "targeted", "status": "access-limited",
"hosts": [ "hosts": [
"www.36kr.com", "www.36kr.com",
"36kr.com", "36kr.com"
"eu.36kr.com"
], ],
"initialAccess": "安全检测,内容未核验", "initialAccess": "安全检测,内容未核验",
"boundary": "此前首页安全检测仍保留;本轮4项定向检索及相关段落补查,3项据政府/公司/交易当事方资料关联、1项按原始登记归范围外。清析检索与直达正文不一致未确认关联,其他文章为平台发布/授权转载,不代表两年全量。 另登记合木、昆腾动力、寰宇心生3项报道相关段落,原始成立/主体留缺,未新增主公司表;不是全量。", "boundary": "2026-10-30仍返回安全检测页面,未把首页作为已读文章。",
"nextCheck": "墨甲已由交易当事方原始公告补足2025成立及历史期间收入,母公司项目沿革和当前财务仍待核;继续补清析发现文章关联及未读正文,建两年公开文章台账,不绕过访问保护。 补本组三项公司/孵化/投资原始资料,避免融资与研究院、母体业绩归属混淆。", "nextCheck": "使用可公开读取的具体文章或其他原始来源;不绕过访问保护。",
"limitation": "仅核具体相关段落,检索和直达差异、转载与原始披露分别记录。", "limitation": "暂不作为已读取内容证据",
"accessRecheckedAt": "2026-10-30" "accessRecheckedAt": "2026-10-30"
}, },
{ {
@@ -234,7 +233,7 @@
"status": "per-company", "status": "per-company",
"hosts": [], "hosts": [],
"initialAccess": "按公司逐一核验", "initialAccess": "按公司逐一核验",
"boundary": "公司官网和官方材料逐条绑定证据;不是可预先列尽的全站候选池。 定向新增企智芯与九昇智能的2025成立自述及交付资料,精确注册、实际客户与收入未核;两个官网不代表中文公司全集。", "boundary": "公司官网和官方材料逐条绑定证据;不是可预先列尽的全站候选池。",
"nextCheck": "核定价、主体、交付与收入定义,自述与独立验证分开。", "nextCheck": "核定价、主体、交付与收入定义,自述与独立验证分开。",
"limitation": "自述不等于独立审计" "limitation": "自述不等于独立审计"
}, },
@@ -286,11 +285,9 @@
"url": null, "url": null,
"role": "supplement", "role": "supplement",
"status": "targeted", "status": "targeted",
"hosts": [ "hosts": [],
"getlatka.com"
],
"initialAccess": "见具体证据访问记录", "initialAccess": "见具体证据访问记录",
"boundary": "未归入上述来源的公开材料按原始证据链接逐条保留;不视为新增已批准的完整候选池。 本轮读取GetLatka单页历史年收入估算,方法底表未取得,不视为新增完整候选池。", "boundary": "未归入上述来源的公开材料按原始证据链接逐条保留;不视为新增已批准的完整候选池。",
"nextCheck": "逐项核来源归属、可靠性及可读正文,不能将搜索摘要当全文。", "nextCheck": "逐项核来源归属、可靠性及可读正文,不能将搜索摘要当全文。",
"limitation": "来源可读不等于声明已独立验证。" "limitation": "来源可读不等于声明已独立验证。"
} }
+1 -1
View File
@@ -1,5 +1,5 @@
{ {
"sourceVersion": "2026-10-30-cn-sme-1", "sourceVersion": "2026-10-30-battlefield-3",
"asOf": "2026-10-30", "asOf": "2026-10-30",
"metric": "platform-mrr-usd", "metric": "platform-mrr-usd",
"sampleUnit": "product-or-business", "sampleUnit": "product-or-business",
+2 -2
View File
@@ -191,8 +191,8 @@ def render():
f"| 其他目录基础项 | {depth-reviewed} | 规则初分或资料不足,继续待复核 |", f"| 其他目录基础项 | {depth-reviewed} | 规则初分或资料不足,继续待复核 |",
f"| 有团队数字 | {sum(c['team'] is not None for c in companies)} | 多为自报Team Size或Employees,时点与口径不统一 |", f"| 有团队数字 | {sum(c['team'] is not None for c in companies)} | 多为自报Team Size或Employees,时点与口径不统一 |",
f"| 专项技术证据记录 | {len(companies)-technical['unreviewed']} | 其中{technical['description']}条自述、{technical['public-artifact']}条公开材料、{technical['independent-test']}条第三方评测 |", f"| 专项技术证据记录 | {len(companies)-technical['unreviewed']} | 其中{technical['description']}条自述、{technical['public-artifact']}条公开材料、{technical['independent-test']}条第三方评测 |",
f"| 企业披露收入线索 | {income['self']} | 含公司或交易当事方披露、历史、下界及币种不明,不等于可比ARR |", f"| 公司披露收入线索 | {income['self']} | 含历史、下界、币种不明及产品业务声明,不等于可比ARR |",
f"| 仅平台收入线索 | {income['platform']} | 同时有企业披露者计入上一行,不重复计数 |", f"| 仅平台收入线索 | {income['platform']} | 同时有公司披露者计入上一行,不重复计数 |",
f"| 仅第三方估算 | {income['estimate']} | 不与直接披露混合 |", f"| 仅第三方估算 | {income['estimate']} | 不与直接披露混合 |",
f"| 暂无可用收入数字 | {income['unknown']} | 包含尚未逐家搜索的目录项,不代表没有收入 |", '', f"| 暂无可用收入数字 | {income['unknown']} | 包含尚未逐家搜索的目录项,不代表没有收入 |", '',
'## 业务分类与数据入口', '', '## 业务分类与数据入口', '',
+29
View File
@@ -116,6 +116,17 @@ def main():
mode.add_argument('--revision', type=int) mode.add_argument('--revision', type=int)
reading.add_argument('--limit', type=int, default=50) reading.add_argument('--limit', type=int, default=50)
reading.add_argument('--offset', type=int, default=0) reading.add_argument('--offset', type=int, default=0)
sections = sub.add_parser('outline', help='Read chapter titles and internal summaries without bodies')
sections.add_argument('slug')
sections.add_argument('--revision', type=int)
section = sub.add_parser('sections', help='Read selected chapter bodies')
section.add_argument('slug')
section.add_argument('ids', nargs='+')
section.add_argument('--revision', type=int)
editing = sub.add_parser('edit-sections', help='Atomically apply chapter operations from private JSON')
editing.add_argument('slug')
editing.add_argument('file')
editing.add_argument('--request-id')
saving = sub.add_parser('save') saving = sub.add_parser('save')
saving.add_argument('file', help='Private JSON file or - for stdin; one document or {documents: [...]}') saving.add_argument('file', help='Private JSON file or - for stdin; one document or {documents: [...]}')
saving.add_argument('--request-id', help='Reuse this id only with the same input when retrying') saving.add_argument('--request-id', help='Reuse this id only with the same input when retrying')
@@ -135,6 +146,24 @@ def main():
elif args.revision is not None: elif args.revision is not None:
path += '?' + urllib.parse.urlencode({'revision': args.revision}) path += '?' + urllib.parse.urlencode({'revision': args.revision})
result = request(base, token, 'GET', path) result = request(base, token, 'GET', path)
elif args.command in ('outline', 'sections', 'edit-sections'):
path = '/api/documents/sections/' + urllib.parse.quote(args.slug, safe='/')
if args.command == 'edit-sections':
payload = json.load(sys.stdin) if args.file == '-' else json.loads(Path(args.file).read_text())
request_id = args.request_id or str(uuid.uuid4())
if not request_id or len(request_id) > 100 or any(c not in 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789._:-' for c in request_id):
raise ValueError('Invalid request id')
print('Request id: ' + request_id, file=sys.stderr)
result = request(base, token, 'POST', path, payload, request_id)
else:
query = []
if args.revision is not None:
query.append(('revision', args.revision))
if args.command == 'sections':
query.extend(('section', section_id) for section_id in args.ids)
if query:
path += '?' + urllib.parse.urlencode(query)
result = request(base, token, 'GET', path)
else: else:
payload = json.load(sys.stdin) if args.file == '-' else json.loads(Path(args.file).read_text()) payload = json.load(sys.stdin) if args.file == '-' else json.loads(Path(args.file).read_text())
request_id = args.request_id or str(uuid.uuid4()) request_id = args.request_id or str(uuid.uuid4())
+25
View File
@@ -4,6 +4,7 @@ import io
import json import json
from pathlib import Path from pathlib import Path
import tempfile import tempfile
import sys
import unittest import unittest
from unittest.mock import patch from unittest.mock import patch
import urllib.error import urllib.error
@@ -15,6 +16,30 @@ spec.loader.exec_module(client)
class ClientTests(unittest.TestCase): class ClientTests(unittest.TestCase):
def test_section_reads_select_bodies_and_revision_without_full_document_fetch(self):
argv = ['client', '--url', 'https://docs.example', '--key-file', '/private/key',
'sections', 'endor/example', 'preamble', 's_one', '--revision', '3']
with patch.object(sys, 'argv', argv), patch.object(client, 'read_token', return_value='test-token'), \
patch.object(client, 'request', return_value={'revision': 3, 'sections': []}) as request, \
patch.object(sys, 'stdout', io.StringIO()):
client.main()
request.assert_called_once_with('https://docs.example', 'test-token', 'GET',
'/api/documents/sections/endor/example?revision=3&section=preamble&section=s_one')
def test_section_edit_keeps_payload_and_retry_id(self):
payload = {'revision': 3, 'operations': [{'op': 'update', 'id': 's_one', 'summary': 'Internal summary'}]}
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / 'edit.json'
path.write_text(json.dumps(payload))
argv = ['client', '--url', 'https://docs.example', '--key-file', '/private/key',
'edit-sections', 'endor/example', str(path), '--request-id', 'edit-1']
with patch.object(sys, 'argv', argv), patch.object(client, 'read_token', return_value='test-token'), \
patch.object(client, 'request', return_value={'revision': 4}) as request, \
patch.object(sys, 'stdout', io.StringIO()), patch.object(sys, 'stderr', io.StringIO()):
client.main()
request.assert_called_once_with('https://docs.example', 'test-token', 'POST',
'/api/documents/sections/endor/example', payload, 'edit-1')
def test_only_tls_or_loopback_origins_are_accepted(self): def test_only_tls_or_loopback_origins_are_accepted(self):
self.assertEqual(client.server_url('https://docs.example/'), 'https://docs.example') self.assertEqual(client.server_url('https://docs.example/'), 'https://docs.example')
self.assertEqual(client.server_url('http://127.0.0.1:18135'), 'http://127.0.0.1:18135') self.assertEqual(client.server_url('http://127.0.0.1:18135'), 'http://127.0.0.1:18135')