Read the published research corpus and avoid expensive paging plans [skip ci]

docs/research-api.md

@@ -9,12 +9,12 @@ collections remain project scoped.
 `internal/research` implements the HTTP contract and a PostgreSQL repository
 based on the fixed Paper Atlas `a3b677b` schema (TEXT/BIGINT columns, native
 English title/abstract search index). `cmd/foundry` enables it only when
-`FOUNDRY_RESEARCH_DATABASE_URL` is configured. No live research connection has
-been configured during the ongoing migration.
-The existing SQLite implementation was withdrawn. Wait for the separate Paper
-Atlas PostgreSQL migration to finish, inspect the published schema and indexes,
-then verify the queries, indexes and full-corpus performance with a
-least-privilege read-only connection before enabling it in the existing server.
+`FOUNDRY_RESEARCH_DATABASE_URL` is configured. The Paper Atlas migration is now published: 21 tables passed its independent
+count/content-hash acceptance. A dedicated `foundry_research_reader` role has
+SELECT on the ten tables consumed here, schema USAGE, and read-only defaults.
+Its connection is stored only in the protected server file below.
+The existing SQLite implementation was withdrawn. Full-corpus API performance
+and frontend integration still require verification before enabling production.
 Do not add `FOUNDRY_RESEARCH_DATABASE`.
 
 The server-side deployment may load a protected
@@ -23,7 +23,10 @@ The server-side deployment may load a protected
 Keep the connection string out of Git, Actions variables, browser configuration
 and command arguments. Use a dedicated role with schema USAGE and table SELECT
 only. Core uses a separate two-connection pool, read-only connection defaults and
-read-only transactions. Startup checks the published schema/SELECT permissions;
+read-only transactions. Each research transaction disables JIT locally: the
+measured count plan spent about 2.85 seconds compiling JIT functions, and its
+end-to-end execution exceeded eight seconds. This does not alter shared
+PostgreSQL settings or replace exact counts with estimates. Startup checks the published schema/SELECT permissions;
 an explicitly enabled but invalid connection fails candidate readiness.
 
 Deployment migration defaults now follow the shared `foundry-postgres`
@@ -47,16 +50,25 @@ raw SQL or user-supplied full-text syntax.
 
 Title/abstract search uses the migration's English vector with
 `plainto_tsquery`; author search uses simple-language terms over the stored
-author JSON. Both treat input as terms, never full-text operators. Author search
-and topic filtering still need full-corpus execution-plan/performance checks
-after publication; the migration's title/abstract index does not index authors.
+author JSON. Both treat input as terms, never full-text operators. Initial full-corpus measurements showed eight-second timeouts for text,
+author and topic counts; the unfiltered count took 7.25 seconds. Execution plans
+confirmed sequential scans for authors and category joins.
+`deploy/research/query-indexes.sql` supplies operator-run concurrent indexes for
+those two missing access paths. It is separate from Core startup and auth
+migrations; do not grant DDL to the runtime reader. Both indexes completed on the confirmed ECS (category 35.4 seconds, author
+436.47 seconds); end-to-end timings are still under verification.
 
 List responses contain `items`, `total`, `topics`, `page`, and `pageSize`.
 Each paper preserves original title/abstract, authors, source dates, topics and
 source links. Publication decisions need source evidence; absent decisions stay
 unknown. No invented repositories, metrics or generated research conclusions.
 
-The repository must count and read each list in one read-only snapshot. The
+The repository counts and reads each list in one read-only snapshot.
+Paging first materializes matching IDs, then joins only that page to the wide
+paper rows. Filtered searches first materialize matching IDs without an
+ORDER BY/LIMIT, then sort/page those IDs, preventing sparse terms from choosing
+a backward heap scan with repeated text-vector computation. Activity sampling showed that selecting full rows before LIMIT
+caused sparse category queries to read large portions of the paper heap. The
 HTTP layer permits two in-flight queries with an eight-second context budget.
 Busy requests return 429 with `Retry-After: 1`; deadlines return 504; storage
 errors return a generic 503 without database details. Missing papers return 404.
@@ -71,11 +83,21 @@ query performance or production availability.
 
 The PostgreSQL repository batches source/tag/identifier/submission queries in
 the same read-only repeatable-read transaction. Author tuples are displayed as
-given name, family name and suffix; null abstracts/dates remain null. Publication
+given name, family name and suffix; an optional fourth affiliation field
+is excluded from the display name (observed in real arXiv records); null abstracts/dates remain null. Publication
 status comes from the latest observed decision, using decision date and ID as
 tie-breakers, and stays unknown when no decision exists. Unit checks cover
 author shapes, bound user values and schema qualification; real PostgreSQL
-corpus checks remain pending the migration.
+the real Core detail and homepage HTTP routes have returned 200.
+The frontend has also rendered the actual 3,526,999-paper corpus through a
+loopback SSH tunnel: eight rows, matching detail title/source, desktop and
+mobile layout, no page exceptions or horizontal overflow in that check.
+Author search returned 146 records in 0.336 seconds. After paging/author fixes, category `cs.AR` returned 9,457 records in 4.906
+seconds, `CartoonGAN` returned three in 0.332 seconds, and the multi-term search
+`attention is all you need` returned 8,653 in 6.325 seconds. These were sequential
+manual HTTP checks, not a load test. Very broad text searches can still exceed
+the eight-second budget and need further performance work. This is not a
+production deployment or an authentication acceptance result.
 
 The existing `deploy/check-schema.sh` also loads the synthetic fixture in
 `internal/research/testdata/corpus.sql` inside its disposable PostgreSQL
@@ -85,3 +107,21 @@ integration checks execute real SQL for filtering, stable paging, nullable
 content, source/tag/identifier joins, latest decision tie-breakers, permission
 denial and SQL-backed HTTP routes. They never connect to the shared migration
 instance and do not establish full-corpus performance or live-data correctness.
+
+## Text storage compatibility (repair 4e62710)
+
+The Paper Atlas repair uses `nul-soh-v1`: NUL is stored as SOH/SOH, and a
+literal SOH as SOH/STX, preserving binary text ordering. Core reads the schema
+comment `paper-atlas:text-codec=nul-soh-v1` through `pg_catalog` at startup,
+restores SQL text before JSON parsing/API responses, and encodes bound search
+terms. An absent/unrelated schema comment keeps legacy text unchanged; an
+unsupported marked codec or malformed encoded text fails explicitly.
+
+Schema comments are public metadata, so consumers need no privileges on the
+private migration controller. The research role still needs only schema USAGE
+and table SELECT. The codec adaptation remains local without CI or branch pushes;
+the published schema marker has been verified with the runtime reader.
+All research tests passed manually against disposable PostgreSQL on the
+confirmed ECS, including the encoded SQL fixture for schema detection, raw TEXT restoration,
+JSON author/metadata decoding, search and the HTTP round trip. Passing isolated
+fixtures does not establish full-corpus API performance.