Page research without exact counts and bound candidate scans [skip ci]

docs/research-api.md

@@ -46,7 +46,8 @@ not require a session; these routes do not expose private data or accept writes.
 
 List parameters are `q` (200 Unicode characters, 20 words), `searchIn`
 (`text` or `author`), `topic` (an exact arXiv category), `sort` (`recent` or
-`title`), `page` (1–1000), and `pageSize` (1–50, default 8). Unknown, malformed
+`title`), `cursor` (optional, opaque, at most 32768 characters), and
+`pageSize` (1–50, default 8). Unknown, malformed
 or repeated parameters return 400. `recent` means descending stable paper ID,
 not publication date. Search terms must become bound query parameters, never
 raw SQL or user-supplied full-text syntax.
@@ -61,17 +62,35 @@ those two missing access paths. It is separate from Core startup and auth
 migrations; do not grant DDL to the runtime reader. Both indexes completed on the confirmed ECS (category 35.4 seconds, author
 436.47 seconds); measured end-to-end checks are documented below.
 
-List responses contain `items`, `total`, `topics`, `page`, and `pageSize`.
+List responses contain `items`, `nextCursor`, `previousCursor`, `topics`, and
+`pageSize`. Empty cursor strings mean that direction is unavailable. There is
+no `total`, page number, total page count or deep-page cap. The old `page`
+parameter is rejected; coordinate the frontend/backend contract update.
 Each paper preserves original title/abstract, authors, source dates, topics and
 source links. Publication decisions need source evidence; absent decisions stay
 unknown. No invented repositories, metrics or generated research conclusions.
 
-The repository counts and reads each list in one read-only snapshot.
-Paging first materializes matching IDs, then joins only that page to the wide
-paper rows. Filtered searches first materialize matching IDs without an
-ORDER BY/LIMIT, then sort/page those IDs, preventing sparse terms from choosing
-a backward heap scan with repeated text-vector computation. Activity sampling showed that selecting full rows before LIMIT
-caused sparse category queries to read large portions of the paper heap. The
+Lists use a read-only repeatable-read snapshot without `count(*)` or `OFFSET`.
+Keyset paging uses ID descending for recent order and `(title_key, id)` ascending
+for title order, including a reverse query for previous pages. The title key is
+kept in its stored codec and C collation. Cursors bind normalized query, search
+scope, topic, sort and page size; changing filters requires a fresh cursor. They
+are public row positions, not authorization tokens or guaranteed frozen snapshots
+across requests. Concurrent corpus edits may alter later pages.
+
+Only pageSize+1 positions are selected; the extra position indicates more data,
+then full rows and batched metadata are loaded for at most pageSize papers.
+For recent order with only a topic filter, page directly through the existing
+`(tag_id,paper_id)` index and deduplicate evidence before LIMIT. For text/author
+search in recent order, try an ID interval of 512 near the cursor, expanding
+once to 4096 only if there are some matches but fewer than pageSize+1.
+Only a full page plus one proves that interval contains the next page. If neither bounded attempt fills the page,
+all local candidates are discarded and the complete indexed search is used;
+rare or older matches are never silently truncated. The fallback materializes
+matching IDs before sorting, avoiding an unbounded ordered heap scan for sparse
+terms. Title-order filtered queries still use that complete search and can time
+out for broad terms. No global database or proxy resource settings are changed.
+The
 HTTP layer permits two in-flight queries with an eight-second context budget.
 Busy requests return 429 with `Retry-After: 1`; deadlines return 504; storage
 errors return a generic 503 without database details. Missing papers return 404.
@@ -95,7 +114,7 @@ and homepage HTTP routes have returned 200.
 The frontend has also rendered the actual 3,526,999-paper corpus through a
 loopback SSH tunnel: eight rows, matching detail title/source, desktop and
 mobile layout, no page exceptions or horizontal overflow in that check.
-Author search returned 146 records in 0.336 seconds. After paging/author fixes, category `cs.AR` returned 9,457 records in 4.906
+The earlier count-based API returned 146 author records in 0.336 seconds. Before the cursor optimization, after paging/author fixes, category `cs.AR` returned 9,457 records in 4.906
 seconds, `CartoonGAN` returned three in 0.332 seconds, and the multi-term search
 `attention is all you need` returned 8,653 in 6.325 seconds. These were sequential
 manual HTTP checks, not a load test. Very broad text searches can still exceed
@@ -143,3 +162,35 @@ reload/new-tab persistence and logout behavior. Browser control was unavailable,
 so these browser results are attributed to user acceptance rather than an
 automated real-account test. This does not deploy the research work branch;
 mail delivery was not exercised.
+
+
+## Cursor regression coverage
+
+The disposable PostgreSQL fixture covers forward/backward traversal, identical
+title keys with different IDs, published control-character encoding, sparse
+matches separated by a large ID gap, and fallback from the bounded interval.
+Invalid, oversized, cross-filter and wrong-version cursors are rejected before
+repository execution. The runtime reader retains SELECT-only privileges; no DDL
+or writes are added. Run tests manually on the confirmed ECS; skip CI for this
+iteration and retain `[skip ci]` in commit and PR titles.
+
+
+## Manual cursor measurements (2026-11-07)
+
+On the confirmed ECS, after this task's compilation and disposable fixture had
+finished, sequential direct requests to the temporary Core returned: homepage
+65 ms, next/previous 39/21 ms, `graph` 345 ms, `CartoonGAN` 731 ms (3 papers),
+`cs.AR` 145 ms, title order 180 ms, author `Buhrman` 769 ms, and
+`attention is all you need` 1312 ms. Recent, topic and title cursor round trips
+returned the same first page without overlap. These are individual manual
+samples, not production latency guarantees; concurrent compilation earlier
+produced multi-second cold measurements. No proxy, database memory, CPU or
+production configuration was changed.
+
+The final source compiled manually on the ECS; package unit checks and all ten
+selected cursor/PostgreSQL integration tests passed, including interval expansion
+and complete fallback. No CI or local build ran. The local authorized Next dev
+uses the temporary API through the existing SSH tunnel. Independent Chrome also
+verified real homepage/search navigation, a preserved document, one mobile
+header/main, no page exceptions and no horizontal overflow. The production Core
+was not replaced by these temporary preview updates.