Endoleap/headless/README.md

# Headless Canvas: portable ESM preview

Build from the Endoleap checkout with `npm run build:headless`. The generated
`.headless-dist` directory is a private package; nothing is published to a registry.
Copy it elsewhere, or use `npm pack --offline --ignore-scripts` inside that directory.

Public exports:

| Import | Responsibility |
| --- | --- |
| `@endoleap/headless-canvas` or `/core` | Document, registry, capabilities, transactions, history, sessions, geometry and queries |
| `/runtime` | Project/Workspace/Host, storage and service port types, recovery, scoped loader and compute/display pools |
| `/service` | Foundry protocol codecs and transport adapters; credentials remain in the host |
| `/browser` | Explicit IndexedDB/Web Locks and module Worker adapters |
| `/bindings` | Portable projection, renderer descriptors, caches, connectors and hit queries |
| `/react` | React Provider and selector, element, action, mode, gesture, operation, asset and sync hooks; no React Flow runtime dependency |
| `/react-flow` | Explicit client adapter, renderer command context and framework node/edge types |

The portable entries have no React, Next, React Flow or other runtime package
dependency. They use standard platform APIs such as AbortController, Blob, URL,
TextEncoder and Web Crypto. The sample runs on Node 22+. Browser adapters require
their browser APIs; importing `/runtime` does not import the browser adapter.

Run `node consumer.mjs` inside the generated package. It uses a custom counter
component, rejects an atomic invalid transaction, commits edits, undoes/redoes,
cancels a gesture preview, deep-copies through the runtime, and reopens the engine.
The storage and protocol-2 transport are in-memory examples. Reopening demonstrates
the codec and host contract, not cross-process crash durability or a deployed server.

`npm run test:headless:package` builds, packs/unpacks into an isolated directory,
blocks framework imports and browser-only globals, runs the sample, checks Node
TypeScript consumers without the DOM library, and exercises compiled browser adapters
and module Worker URLs in isolated Chromium. The browser checks load the emitted JS
directly, without TypeScript stripping or a Next bundler.

The React adapter has an explicit entry and optional peer dependencies; portable
consumers do not load it. No registry publication, release support promise, server
deployment or full-goal completion is implied by this local package.

Experimental partial-model runtime: pass a validated prefix plus `loading` in
`EngineOptions`. Version 1 uses recovery format 7; version 2 carries snapshot-bound direct canvas
membership counts and uses format 8. Both keep the scheduler held. Version 3
uses format 9 and retains `runtime.loadingRead` separately from the moving ACK base.
Loaded non-reference data fields and node geometry can be edited and undone; creation,
deletions, ownership/reference changes, uploads, imports and AI admission wait.
`engine.loading.accept(document, versions, state)` extends the captured prefix without
replacing local edits. `engine.loading.complete(document, versions, {sourceToken})`
validates the full capture and releases scheduling only after its cut is durable.
Use `{fresh:true}` only for an explicitly fresh complete read; conflicts require the
existing reviewed recovery flow. `engine.loading.resume()` coalesces a fresh complete
read through the project's persistence adapter and persists completion before resolving.
`workspace.resumeLoading(projectId)` delegates to that owner and reports current-identity
failures. Closing the project aborts the read and saves the recoverable prefix; detaching
a view does not close its engine.

Endoleap's page now opens the same project engine for a validated prefix, allows the
restricted field/geometry edits above, coalesces incoming prefixes, and preserves its
textarea, history and viewport when the full model arrives. App navigation cancels the
view's stream and asks the retained project owner to complete through a fresh read.
Reload recovers the durable prefix when that project is opened again. This is not a
persistent HTTP cursor or a workspace-wide startup scan. Version-3 scoped ACK/chunk
coordination and later scope-based structure/import permissions are described below.

Scope-aware queries are obtained from `engine.editor.getQueries()` and share the
same publication boundary as the root. `lookupElement(id)` returns `present`,
`absent` (complete project), or `unloaded`; a missing ID in a partial model is not
proof of absence. `membersResult(canvasId)` adds a completeness flag for direct
members; `childCanvasesResult` covers published topology, while `incidentResult`
requires the complete project because another owner may contain an incoming edge.
Legacy array-returning methods enumerate loaded records only. Hit-query completeness
also includes scope coverage. Derived indexes and coverage views are not persisted.

An optional `canvasElementCounts` field in the version-1 stream header allows new
clients to certify individual scopes. Missing metadata retains legacy conservative
coverage. Counts include nodes and connections; they must cover exactly the declared
active canvases, sum to the total, and never be exceeded by received records. A scope
is ready only after its full count has entered the dependency-closed published prefix.
Version-2 loading permits `connection.path` edits and history in a certified owner
scope; endpoint, ownership, creation and deletion edits still wait. Scope completion never replaces terminal/EOF validation for the whole project.
Versions 1/2 continue to hold network scheduling. In version 3, mutable operations
wait in the scheduler's derived admission set until every participant scope is
complete and its root is durable; unrelated certified scopes can send meanwhile.
The normal frozen-body/key, writer claims and pre-dispatch durability barrier apply.

`runtime.loadingRead` retains the original source document and version vector. Incoming
prefixes and same-source completion validate against that capture; scopes with advanced
ACK versions retain their newer acknowledged records, then pending intents replay.
`Workspace.open(document, versions, {sourceToken})` completes that original stream,
even when its captured version is now older than a scoped ACK. An unrelated fresh
read waits for outstanding receipts and uses a scheduling hold while capturing and
persisting its base; edits during the read remain local and are replayed.

Recovery format 9 encodes both baselines and original frozen requests. Unknown results
still query/retry the same body/key after reopening. Partial-source completion clears
the retained read baseline and forces a full checkpoint without older partial journal
dependencies. Creation, deletion, owner/endpoint/reference changes, uploads and AI
remain gated during partial loading; broader structural scope admission and real
Foundry deployment/integration remain unfinished.


Loading version 4 adds an optional stream-header `elementOwners` manifest and uses
recovery format 10. It covers every active source ID, including unloaded records,
and must match both per-owner counts and each received record. New IDs must also
avoid the captured tombstones; new canvas IDs avoid all captured version keys.
Without this manifest the page retains version-3 field-only admission.

In version 4, complete owners can create elements and a frame/child-canvas pair,
undo/redo them, redirect loaded connections, and move/remove objects born after the
capture when the affected owners are ready. Existing-element removal/reparenting
requires the entire captured source closure; a missing incoming edge is not proof
that no edge exists. Declared references are checked against the transaction's final
model. Shared loaded asset references are permitted; new uploads, generation,
complete export remain gated in version 4. Version 5 adds import/copy admission as described below.

ACK validation reconstructs the exact protocol-2 effects of applied logical intents
on the original source; late source blocks cannot erase births, restorations or tree
changes. Both the source capture and exact request identities recover through format
10. Pure intent proofs and applied prefixes are cached by immutable source/registry/
intent identities; changing the source invalidates those proofs.

Query coverage also exposes an immutable captured ID set. A missing ID outside that
complete namespace can be reported absent before all payloads arrive; an ID inside it
remains unloaded until received. An old tombstone alone cannot classify a captured
restored ID as absent. Absence does not make an old ID available for fresh creation.
The identity manifest costs O(N) header bytes and memory and still obeys the existing
8 MiB line / 64 MiB response limits; this does not make database acquisition lazy.

Loading version 5 (recovery format 11) also retains the complete captured
`sourceCanvases` topology from the existing stream header, even for partitions
whose entry payloads are not published yet. With counts and `elementOwners`, the
page now selects this version. Existing v1–v4 recovery images retain their original
contracts; future format 12 is rejected.

`engine.imports.canStart(canvasId)` and `canDuplicate(selection, {viewCanvasId})`
return availability for UI controls, including policy and lifecycle checks. The
actual methods recheck admission. Native NDJSON import requires a complete target
owner. Deep copy requires complete selected subtrees and captured ancestor owners,
so upper-canvas incoming edges are copied from the same original endpoint into
the new partition. A declared but unpublished child is not an empty partition.
Unrelated incomplete partitions do not prevent these jobs from starting.

The existing ImportJob/source Blob/runner, history and scheduler are reused.
`source.loadingAdmission` records the source token, latest captured logical
operation sequence, and sorted required canvas IDs. On partial recovery it must
match the retained read and proved scopes. Before copy planning/resume, the source
topology is checked against that historical logical cut, independently of newer
source chunks and local edits. Source SHA, stable mappings and generation checks
continue to protect payloads and late callbacks. This is provenance metadata,
not another queue or a lock held during parsing.

Pause/reopen/resume retains source bytes and identities. Cancellation keeps already
published content and does not restart; undo remains independent of the source
parser. All content sends through the normal durability and frozen-request barriers.
Full-source completion removes the loading marker and read baseline without
replacing job/history state. In version 5, new image uploads and generation remain gated while
loading; loaded shared resource references obey the existing structural rules.

Loading version 6 (recovery format 12) admits local images in complete owners.
`engine.canStageFiles(canvasId)` supplies UI availability; `stageFiles` rechecks
actual transaction changes and policy. Existing field/geometry and import/copy
contracts continue to apply, including copying a pending image by the same
ResourceId rather than starting another upload.

`ResourceRecord.loadingAdmission` records `{version:1, sourceToken, sourceURL,
canvasIds}`. The resource's original upload OperationId precedes its first document
use. Validation checks that first use's exact owner set, complete captured/born
scopes, and the adapter's URL interpretation on reopening. New resource references
are admitted from these proved identities and the current staging transaction;
arbitrary resource IDs/URLs remain rejected. The proof and file bytes use the
existing project recovery store, not a second queue or workspace-wide AssetDepot.

Working content, history, upload intent and Blob bytes retain their existing atomic
durable cut. An upload has no canvas participants, so its own admission does not
claim a fake canvas writer; it still waits for the root/file/request barriers.
Document requests depend on stored upload results and keep normal owner/version
claims. Upload receipt validation does not advance document versions. A lost ACK
reopens the same frozen key/body and queries it before another document send.

Whole-source completion preserves pending/stored resources and history. Default
image sizing and display handles reuse the existing decoder/cache lifecycle.
Undo/redo, source failure, quota retry and late chunks do not re-upload bytes or
restore deleted incarnations. Earlier loading/recovery versions retain their old
restrictions; future format 13 is rejected. Generation, cross-project durable depot
sharing and broader lifecycle/acceptance work remain separate unfinished goals.

The package now exposes `/bindings` for pure projection, renderer descriptors,
geometry/projection caches, hit queries, connectors and resource-view handles.
`projectCanvas` consumes the captured Document/Registry/queries and optional gesture
geometry. It returns view DTOs; it does not own another document, history or queue.
Projection DTOs and compatibility card/edge hints no longer import React Flow types
or the legacy element classes. Legacy views re-export the same shared data types.

`/react-flow` is the explicit client/framework entry. It exports renderer factories,
the command context/hook, and `ReactFlowProjectionNode`/`ReactFlowProjectionEdge`
structural adapter types. React, React DOM and React Flow are optional peers at the
package level, required when a host opts into this entry. Core/runtime/service and
pure bindings remain usable without them. The declared floors are React/React DOM
19.3.0 and React Flow 12.12.1; tests use the locally installed compatible versions,
not a claim to have tested every allowed version.

A host retains the immutable renderer catalog and creates view-local projection
caches; installed components submit commands through its scoped Provider. Attaching
or removing UI does not replace the ProjectEngine or terminate durable jobs.
Compatibility DTOs are rendering hints, not the model schema or a migration format.
The existing application now uses the same explicit framework entry.

The packed Node consumer projects and hits its custom counter using only public
exports, with DOM globals and external runtime dependency resolution blocked. Its
NodeNext type check excludes DOM libraries. A separate peer consumer renders a
registered component, verifies its command context, and checks typed NodeProps and
EdgeProps with the declared peers. Browser tests still execute the compiled ESM
storage and Worker artifacts independently of Next.js/TypeScript stripping.


Workspace asset depot and file storage:

`createWorkspaceRuntime` owns one depot when `StoragePort.assetDepot` is provided.
`await engine.adoptResource(id, (resource, tx) => { ... }, {signal})` obtains fresh
server permission, verifies the original bytes, and runs a synchronous editor
transaction. The callback uses the existing `resource.id` / canonical `resource.src`
in a newly allocated element; it also receives server dimensions in `resource.asset`.
The project stores a `SharedAssetBinding`, not a second upload. Binding, document,
history, ledger, shared bytes and pins cross the same durable barrier. The returned
local `Result` does not imply durability or backend completion. Async or failed
callbacks leave no partial binding. The Foundry authority requires the matching
permission-checked asset metadata endpoint; cached metadata is not an authorization.

Bindings use recovery envelope 13 (including partial loading v6); readers retain
formats 1–12 and reject unknown future formats. A reopened binding can display
shared local bytes, and undo/redo reuse the asset without uploading again. The
workspace closes projects before its depot; retiring the identity fences late work.

The browser adapter uses IndexedDB version 4. After a proven stored-upload cut, a
project's `files` row holds an opaque pointer to digest-addressed `asset-bytes`.
Unconfirmed uploads and import source files keep their original Blob rows. Use
`readFile` / `scanFiles`; direct assumptions that every raw files-store row is a
Blob are invalid. Logical reads preserve the original local MIME, even if the
server detected another MIME. Existing v3 databases upgrade without dropping
checkpoints or original files and promote on a subsequent matching Root commit
(or an idempotent reindex). Older DB readers reject v4. Blob URLs and already
returned Blob objects live independently of a later persistent eviction; actual
disk reclamation remains the browser's responsibility.

Resource identities retain immutable metadata after byte eviction. Complete,
revision-matched durable manifests from every project in the workspace protect
closed editors; missing/stale/opaque coverage prevents shared deletion. Distinct
resource IDs may own the same digest; the shared byte row disappears only when
its last live identity is retired. An older Root cannot release temporary bytes
that belong to a newer adoption revision. Collection is explicit and local; this
preview does not enable background deletion on Foundry or claim the full retention,
permission-race and product acceptance matrix is finished.


React bindings:

Wrap descendants in `CanvasProvider` with an existing `engine` and, for session
hooks, an already attached `viewId`. The Provider borrows both; unmounting it does
not close the project or detach a view owned by the host. Explicitly unmount the
surface, detach its view, and close the engine at their respective lifetimes.
Import from `/react` for React only, or the re-export in `/react-flow`.

- `useCanvas(selector, isEqual?)` observes the one immutable project Root. Default
  equality is `Object.is`; choose an appropriate comparator for allocated results.
  Snapshots are cached per selection and unchanged selections do not notify React.
- `useElement(idOrTarget)` returns `present`, `absent`, `unloaded`, or `stale`.
  A Target includes incarnation; a restored element does not silently retarget it.
- `useActions(target)` exposes stable `rename`, `setData`, `resize`, `remove`, and
  `moveTo` facades over normal editor commands. Optional command settings retain
  label/grouping/history semantics. Call actions from events, not during render.
- `useMode(target)` returns the current view's mode and `setMode`; it never writes
  document/history state. Missing view context is an explicit error.
- `useGestureBindings(target)` returns effective geometry, active ownership and
  `begin/update/commit/cancel`. A token protects the lease; cleanup cancels only
  its own lease. Host input bindings still handle pointer capture and events.
- `useOperation(id)` observes both the logical record and its frozen request.
- `useAsset(id)` resolves a releasable display handle, returns metadata plus
  loading/ready/error/url/retry, and releases on ref/owner change or unmount.
  Ready means the display handle resolved; it does not prove upload ACK, image
  decoding or a successful subsequent fetch of a remote fallback URL.
- `useSyncStatus(canvasId?)` reports pending/blocked/unknown work plus loading,
  dirty and persistence errors. Durability is project-wide even when filtering
  operations to one canvas. It does not call an undurable Root synced. Pending
  counts logical ledger work; running generation/import job progress is observed
  separately from document synchronization.

`engine.getDurability()` now returns a stable readonly snapshot including closing,
closed and an optional error. `subscribeDurability` observes successful cuts,
failures and closure without creating another business Root. Hooks and the Provider
own no second document, ledger, network queue or storage lifecycle. Source selectors
still run on Root notifications; this is not a claim of complete per-field indexing.

A React renderer may use these hooks once its real engine exists. For the early
read-only preview before an engine opens, declare a separate `preview` renderer in
its `ReactRendererDefinition`. The assembly exposes `previewNodeTypes` and
`previewEdgeTypes`. An omitted preview gets a passive preparing placeholder; the
app never constructs a fake editor merely to satisfy the hooks. Preview functions
must render the supplied records without requiring CanvasProvider. Built-in preview
renderers are registered explicitly. Framework definitions never enter core data.

Presentation scheduling:

`createFrameQueue(clock, publish)` accumulates dirty keys in a Set and requests
one frame. `flush()` publishes now; reentrant invalidations wait for the next
frame. `dispose()` cancels pending work, including late canceled callbacks.
This queue never holds commands, undo units or requests. `createSessionFrameSource`
wraps a session source with a per-subscription queue; reads still return the
current session immediately, while preview notifications coalesce per frame.
The application uses it for CanvasView sessions. Pointerup reads its final event
and commits through the engine synchronously; persistence and Root notifications
are not frame-throttled. This does not implement paint extent indexing or selective
per-element drawing: dirty keys currently schedule a view notification.

Node paint overflow:

`createPaintIndex` owns a rebuildable view-space grid of rectangles, with bounded
cell expansion and oversized fallbacks. `projectedPaintOverflow` asks node geometry
for the `paint` role separately from logical/hit geometry, transforms its bounds
into scene space, and reports failed handlers conservatively. CanvasView queries
the overflow index against the live viewport; visible overflow or unknown bounds
suspend native culling so extended paint is retained. This currently disables
culling for the view rather than selectively retaining one node. Geometry
extensions must declare their paint extent; undeclared DOM overflow cannot be
inferred. Edge/overlay paint indexing and region-based drawing remain separate work.

`createProjectedPaintIndex` retains the grid per engine/view and caches each node's
paint result by projected node identity, immutable record identity and canvas
transform. Unchanged nodes reuse their geometry result; removed nodes release
cache entries. Viewport queries reuse the grid. `clear()` discards it. The cache
still scans projected nodes and synchronizes rectangle membership; it is not a
claim of constant-time full-frame updates or complete region-based drawing.