Endoleap/headless/README.md

Headless Canvas: portable ESM preview

Build from the Endoleap checkout with npm run build:headless. The generated .headless-dist directory is a private package; nothing is published to a registry. Copy it elsewhere, or use npm pack --offline --ignore-scripts inside that directory.

Public exports:

Import	Responsibility
@endoleap/headless-canvas or /core	Document, registry, capabilities, transactions, history, sessions, geometry and queries
/runtime	Project/Workspace/Host, storage and service port types, recovery, scoped loader and compute/display pools
/service	Foundry protocol codecs and transport adapters; credentials remain in the host
/browser	Explicit IndexedDB/Web Locks and module Worker adapters
/bindings	Portable projection, renderer descriptors, caches, connectors and hit queries
/react-flow	Explicit client adapter, renderer command context and framework node/edge types

The portable entries have no React, Next, React Flow or other runtime package dependency. They use standard platform APIs such as AbortController, Blob, URL, TextEncoder and Web Crypto. The sample runs on Node 22+. Browser adapters require their browser APIs; importing /runtime does not import the browser adapter.

Run node consumer.mjs inside the generated package. It uses a custom counter component, rejects an atomic invalid transaction, commits edits, undoes/redoes, cancels a gesture preview, deep-copies through the runtime, and reopens the engine. The storage and protocol-2 transport are in-memory examples. Reopening demonstrates the codec and host contract, not cross-process crash durability or a deployed server.

npm run test:headless:package builds, packs/unpacks into an isolated directory, blocks framework imports and browser-only globals, runs the sample, checks Node TypeScript consumers without the DOM library, and exercises compiled browser adapters and module Worker URLs in isolated Chromium. The browser checks load the emitted JS directly, without TypeScript stripping or a Next bundler.

The React adapter has an explicit entry and optional peer dependencies; portable consumers do not load it. No registry publication, release support promise, server deployment or full-goal completion is implied by this local package.

Experimental partial-model runtime: pass a validated prefix plus loading in EngineOptions. Version 1 uses recovery format 7; version 2 carries snapshot-bound direct canvas membership counts and uses format 8. Both keep the scheduler held. Version 3 uses format 9 and retains runtime.loadingRead separately from the moving ACK base. Loaded non-reference data fields and node geometry can be edited and undone; creation, deletions, ownership/reference changes, uploads, imports and AI admission wait. engine.loading.accept(document, versions, state) extends the captured prefix without replacing local edits. engine.loading.complete(document, versions, {sourceToken}) validates the full capture and releases scheduling only after its cut is durable. Use {fresh:true} only for an explicitly fresh complete read; conflicts require the existing reviewed recovery flow. engine.loading.resume() coalesces a fresh complete read through the project's persistence adapter and persists completion before resolving. workspace.resumeLoading(projectId) delegates to that owner and reports current-identity failures. Closing the project aborts the read and saves the recoverable prefix; detaching a view does not close its engine.

Endoleap's page now opens the same project engine for a validated prefix, allows the restricted field/geometry edits above, coalesces incoming prefixes, and preserves its textarea, history and viewport when the full model arrives. App navigation cancels the view's stream and asks the retained project owner to complete through a fresh read. Reload recovers the durable prefix when that project is opened again. This is not a persistent HTTP cursor or a workspace-wide startup scan. Version-3 scoped ACK/chunk coordination and later scope-based structure/import permissions are described below.

Scope-aware queries are obtained from engine.editor.getQueries() and share the same publication boundary as the root. lookupElement(id) returns present, absent (complete project), or unloaded; a missing ID in a partial model is not proof of absence. membersResult(canvasId) adds a completeness flag for direct members; childCanvasesResult covers published topology, while incidentResult requires the complete project because another owner may contain an incoming edge. Legacy array-returning methods enumerate loaded records only. Hit-query completeness also includes scope coverage. Derived indexes and coverage views are not persisted.

An optional canvasElementCounts field in the version-1 stream header allows new clients to certify individual scopes. Missing metadata retains legacy conservative coverage. Counts include nodes and connections; they must cover exactly the declared active canvases, sum to the total, and never be exceeded by received records. A scope is ready only after its full count has entered the dependency-closed published prefix. Version-2 loading permits connection.path edits and history in a certified owner scope; endpoint, ownership, creation and deletion edits still wait. Scope completion never replaces terminal/EOF validation for the whole project. Versions 1/2 continue to hold network scheduling. In version 3, mutable operations wait in the scheduler's derived admission set until every participant scope is complete and its root is durable; unrelated certified scopes can send meanwhile. The normal frozen-body/key, writer claims and pre-dispatch durability barrier apply.

runtime.loadingRead retains the original source document and version vector. Incoming prefixes and same-source completion validate against that capture; scopes with advanced ACK versions retain their newer acknowledged records, then pending intents replay. Workspace.open(document, versions, {sourceToken}) completes that original stream, even when its captured version is now older than a scoped ACK. An unrelated fresh read waits for outstanding receipts and uses a scheduling hold while capturing and persisting its base; edits during the read remain local and are replayed.

Recovery format 9 encodes both baselines and original frozen requests. Unknown results still query/retry the same body/key after reopening. Partial-source completion clears the retained read baseline and forces a full checkpoint without older partial journal dependencies. Creation, deletion, owner/endpoint/reference changes, uploads and AI remain gated during partial loading; broader structural scope admission and real Foundry deployment/integration remain unfinished.

Loading version 4 adds an optional stream-header elementOwners manifest and uses recovery format 10. It covers every active source ID, including unloaded records, and must match both per-owner counts and each received record. New IDs must also avoid the captured tombstones; new canvas IDs avoid all captured version keys. Without this manifest the page retains version-3 field-only admission.

In version 4, complete owners can create elements and a frame/child-canvas pair, undo/redo them, redirect loaded connections, and move/remove objects born after the capture when the affected owners are ready. Existing-element removal/reparenting requires the entire captured source closure; a missing incoming edge is not proof that no edge exists. Declared references are checked against the transaction's final model. Shared loaded asset references are permitted; new uploads, generation, complete export remain gated in version 4. Version 5 adds import/copy admission as described below.

ACK validation reconstructs the exact protocol-2 effects of applied logical intents on the original source; late source blocks cannot erase births, restorations or tree changes. Both the source capture and exact request identities recover through format 10. Pure intent proofs and applied prefixes are cached by immutable source/registry/ intent identities; changing the source invalidates those proofs.

Query coverage also exposes an immutable captured ID set. A missing ID outside that complete namespace can be reported absent before all payloads arrive; an ID inside it remains unloaded until received. An old tombstone alone cannot classify a captured restored ID as absent. Absence does not make an old ID available for fresh creation. The identity manifest costs O(N) header bytes and memory and still obeys the existing 8 MiB line / 64 MiB response limits; this does not make database acquisition lazy.

Loading version 5 (recovery format 11) also retains the complete captured sourceCanvases topology from the existing stream header, even for partitions whose entry payloads are not published yet. With counts and elementOwners, the page now selects this version. Existing v1–v4 recovery images retain their original contracts; future format 12 is rejected.

engine.imports.canStart(canvasId) and canDuplicate(selection, {viewCanvasId}) return availability for UI controls, including policy and lifecycle checks. The actual methods recheck admission. Native NDJSON import requires a complete target owner. Deep copy requires complete selected subtrees and captured ancestor owners, so upper-canvas incoming edges are copied from the same original endpoint into the new partition. A declared but unpublished child is not an empty partition. Unrelated incomplete partitions do not prevent these jobs from starting.

The existing ImportJob/source Blob/runner, history and scheduler are reused. source.loadingAdmission records the source token, latest captured logical operation sequence, and sorted required canvas IDs. On partial recovery it must match the retained read and proved scopes. Before copy planning/resume, the source topology is checked against that historical logical cut, independently of newer source chunks and local edits. Source SHA, stable mappings and generation checks continue to protect payloads and late callbacks. This is provenance metadata, not another queue or a lock held during parsing.

Pause/reopen/resume retains source bytes and identities. Cancellation keeps already published content and does not restart; undo remains independent of the source parser. All content sends through the normal durability and frozen-request barriers. Full-source completion removes the loading marker and read baseline without replacing job/history state. In version 5, new image uploads and generation remain gated while loading; loaded shared resource references obey the existing structural rules.

Loading version 6 (recovery format 12) admits local images in complete owners. engine.canStageFiles(canvasId) supplies UI availability; stageFiles rechecks actual transaction changes and policy. Existing field/geometry and import/copy contracts continue to apply, including copying a pending image by the same ResourceId rather than starting another upload.

ResourceRecord.loadingAdmission records {version:1, sourceToken, sourceURL, canvasIds}. The resource's original upload OperationId precedes its first document use. Validation checks that first use's exact owner set, complete captured/born scopes, and the adapter's URL interpretation on reopening. New resource references are admitted from these proved identities and the current staging transaction; arbitrary resource IDs/URLs remain rejected. The proof and file bytes use the existing project recovery store, not a second queue or workspace-wide AssetDepot.

Working content, history, upload intent and Blob bytes retain their existing atomic durable cut. An upload has no canvas participants, so its own admission does not claim a fake canvas writer; it still waits for the root/file/request barriers. Document requests depend on stored upload results and keep normal owner/version claims. Upload receipt validation does not advance document versions. A lost ACK reopens the same frozen key/body and queries it before another document send.

Whole-source completion preserves pending/stored resources and history. Default image sizing and display handles reuse the existing decoder/cache lifecycle. Undo/redo, source failure, quota retry and late chunks do not re-upload bytes or restore deleted incarnations. Earlier loading/recovery versions retain their old restrictions; future format 13 is rejected. Generation, cross-project durable depot sharing and broader lifecycle/acceptance work remain separate unfinished goals.

The package now exposes /bindings for pure projection, renderer descriptors, geometry/projection caches, hit queries, connectors and resource-view handles. projectCanvas consumes the captured Document/Registry/queries and optional gesture geometry. It returns view DTOs; it does not own another document, history or queue. Projection DTOs and compatibility card/edge hints no longer import React Flow types or the legacy element classes. Legacy views re-export the same shared data types.

/react-flow is the explicit client/framework entry. It exports renderer factories, the command context/hook, and ReactFlowProjectionNode/ReactFlowProjectionEdge structural adapter types. React, React DOM and React Flow are optional peers at the package level, required when a host opts into this entry. Core/runtime/service and pure bindings remain usable without them. The declared floors are React/React DOM 19.3.0 and React Flow 12.12.1; tests use the locally installed compatible versions, not a claim to have tested every allowed version.

A host retains the immutable renderer catalog and creates view-local projection caches; installed components submit commands through its scoped Provider. Attaching or removing UI does not replace the ProjectEngine or terminate durable jobs. Compatibility DTOs are rendering hints, not the model schema or a migration format. The existing application now uses the same explicit framework entry.

The packed Node consumer projects and hits its custom counter using only public exports, with DOM globals and external runtime dependency resolution blocked. Its NodeNext type check excludes DOM libraries. A separate peer consumer renders a registered component, verifies its command context, and checks typed NodeProps and EdgeProps with the declared peers. Browser tests still execute the compiled ESM storage and Worker artifacts independently of Next.js/TypeScript stripping.