foundry-core/docs/research-api.md

Shared research API (in progress)

This work belongs to Thesisly 研迹 and uses Foundry Core rather than a separate application backend. The corpus is public; account sessions and future private collections remain project scoped.

Current implementation boundary

internal/research implements the HTTP contract and a PostgreSQL repository based on the fixed Paper Atlas a3b677b schema (TEXT/BIGINT columns, native English title/abstract search index). cmd/foundry enables it only when FOUNDRY_RESEARCH_DATABASE_URL is configured. No live research connection has been configured during the ongoing migration. The existing SQLite implementation was withdrawn. Wait for the separate Paper Atlas PostgreSQL migration to finish, inspect the published schema and indexes, then verify the queries, indexes and full-corpus performance with a least-privilege read-only connection before enabling it in the existing server. Do not add FOUNDRY_RESEARCH_DATABASE.

The server-side deployment may load a protected /opt/endoleap-infra/foundry/research.env containing FOUNDRY_RESEARCH_DATABASE_URL and FOUNDRY_RESEARCH_SCHEMA (default research). Keep the connection string out of Git, Actions variables, browser configuration and command arguments. Use a dedicated role with schema USAGE and table SELECT only. Core uses a separate two-connection pool, read-only connection defaults and read-only transactions. Startup checks the published schema/SELECT permissions; an explicitly enabled but invalid connection fails candidate readiness.

Deployment migration defaults now follow the shared foundry-postgres container name. FOUNDRY_POSTGRES_CONTAINER or the existing --container option can select a different container without changing database/table names.

Routes

• GET /projects/{project}/research/papers
• GET /projects/{project}/research/papers/{id}

Only configured Foundry projects are accepted. Reading the public corpus does not require a session; these routes do not expose private data or accept writes.

List parameters are q (200 Unicode characters, 20 words), searchIn (text or author), topic (an exact arXiv category), sort (recent or title), page (1–1000), and pageSize (1–50, default 8). Unknown, malformed or repeated parameters return 400. recent means descending stable paper ID, not publication date. Search terms must become bound query parameters, never raw SQL or user-supplied full-text syntax.

Title/abstract search uses the migration's English vector with plainto_tsquery; author search uses simple-language terms over the stored author JSON. Both treat input as terms, never full-text operators. Author search and topic filtering still need full-corpus execution-plan/performance checks after publication; the migration's title/abstract index does not index authors.

List responses contain items, total, topics, page, and pageSize. Each paper preserves original title/abstract, authors, source dates, topics and source links. Publication decisions need source evidence; absent decisions stay unknown. No invented repositories, metrics or generated research conclusions.

The repository must count and read each list in one read-only snapshot. The HTTP layer permits two in-flight queries with an eight-second context budget. Busy requests return 429 with Retry-After: 1; deadlines return 504; storage errors return a generic 503 without database details. Missing papers return 404.

Validation

go test -race -count=1 ./internal/research covers routing, query limits, Unicode, anonymous reads, rejected mutations, project lookup, errors, saturation and timeout slot release. Run compilation/tests only on the user-confirmed ECS runner. These transport tests do not prove a database connection, full-corpus query performance or production availability.

The PostgreSQL repository batches source/tag/identifier/submission queries in the same read-only repeatable-read transaction. Author tuples are displayed as given name, family name and suffix; null abstracts/dates remain null. Publication status comes from the latest observed decision, using decision date and ID as tie-breakers, and stays unknown when no decision exists. Unit checks cover author shapes, bound user values and schema qualification; real PostgreSQL corpus checks remain pending the migration.

The existing deploy/check-schema.sh also loads the synthetic fixture in internal/research/testdata/corpus.sql inside its disposable PostgreSQL container (--network none, no host port). The research test process receives a separate SELECT-only fixture role via RESEARCH_TEST_DATABASE_URL. These integration checks execute real SQL for filtering, stable paging, nullable content, source/tag/identifier joins, latest decision tie-breakers, permission denial and SQL-backed HTTP routes. They never connect to the shared migration instance and do not establish full-corpus performance or live-data correctness.