test(canvas): verify crash cuts and pending image replacement lifetimes

This commit is contained in:
2026-11-11 19:05:33 +08:00
parent dcbbdad25d
commit d3eef78431
4 changed files with 126 additions and 6 deletions
+44 -6
View File
@@ -6,13 +6,20 @@ are not complete.
## Verified revisions and environment ## Verified revisions and environment
- Current delivered Endoleap consumer: `24c45920602378d52a42cda7e8ec2840864c8dbd`. - Current verified Endoleap service consumer: `dcbbdad25de7c3b1d6bee317df6074da2c0642eb`
- Current Foundry: `5db2bf3c98f2627139cbb6d47afefa686b4c6547`. (product implementation `15ee917`; receipt diagnostics added by `dcbbdad`).
- Current Foundry: `506f778ac33d47fbd6f38412995a10218d59fe0a`.
- [Foundry CI #1186](https://gitea.entroia.com/Zayden-Jung/foundry-core/actions/runs/1186):
job 1283 completed successfully, all seven steps successful. All prior flags
and `dissolveCanvas`, `dissolveLostAckRecovered`, `dissolveUndoRedo` are true;
`aiExecutionCount: 3`, 84 transport calls. The new canonical-frame rotation
test also passed. Same-commit PR CI #1187/job 1284 subsequently passed all seven
steps, with the same flags/execution count and 83 transport calls.
- [Foundry CI #1182](https://gitea.entroia.com/Zayden-Jung/foundry-core/actions/runs/1182): - [Foundry CI #1182](https://gitea.entroia.com/Zayden-Jung/foundry-core/actions/runs/1182):
job 1279 completed successfully, all seven steps successful, pinned to the job 1279 completed successfully, all seven steps successful, pinned to the
current delivered consumer. Its report has all document/resource/AI and browser delivery baseline `24c4592`, Foundry `5db2bf3`. Its report has all document/resource/AI and
migration flags true, `aiExecutionCount: 3`, and 75 transport calls. migration flags true, `aiExecutionCount: 3`, and 75 transport calls.
Same-commit PR CI #1183/job 1280 was still running at this audit. Same-commit PR CI #1183/job 1280 subsequently passed all seven steps too.
- [Foundry CI #1180](https://gitea.entroia.com/Zayden-Jung/foundry-core/actions/runs/1180) - [Foundry CI #1180](https://gitea.entroia.com/Zayden-Jung/foundry-core/actions/runs/1180)
and [PR CI #1181](https://gitea.entroia.com/Zayden-Jung/foundry-core/actions/runs/1181): and [PR CI #1181](https://gitea.entroia.com/Zayden-Jung/foundry-core/actions/runs/1181):
completed successfully for consumer `26da326` and Foundry `3268c57`; completed successfully for consumer `26da326` and Foundry `3268c57`;
@@ -21,7 +28,7 @@ are not complete.
- Existing `ces-control` runner executes Go checks/build, schema migration and - Existing `ces-control` runner executes Go checks/build, schema migration and
isolated PostgreSQL integration, then the pinned Node 24 consumer and HTTP smoke isolated PostgreSQL integration, then the pinned Node 24 consumer and HTTP smoke
checks. No local backend build or Docker execution was used. checks. No local backend build or Docker execution was used.
- Local frontend: 802 Node tests and full TypeScript checking passed. Tests use - Local frontend: 821 Node tests and full TypeScript checking passed. Tests use
the source modules; packaged SDK and application delivery remain separate gates. the source modules; packaged SDK and application delivery remain separate gates.
## What the checks actually prove ## What the checks actually prove
@@ -32,6 +39,8 @@ are not complete.
| Migration admission and late rollback | `TestCanonicalMigrationPermissionVersionAndRunGuards` checks viewer denial, stale source versions, queued/running work and stable negative receipts after conditions change. `TestCanonicalMigrationLateCancellationRollsBack` observes the actual identity-map INSERT blocked after canonical/material/subgraph writes, cancels it, then checks all related rows, original legacy JSON and version before retrying the original key successfully. | | Migration admission and late rollback | `TestCanonicalMigrationPermissionVersionAndRunGuards` checks viewer denial, stale source versions, queued/running work and stable negative receipts after conditions change. `TestCanonicalMigrationLateCancellationRollsBack` observes the actual identity-map INSERT blocked after canonical/material/subgraph writes, cancels it, then checks all related rows, original legacy JSON and version before retrying the original key successfully. |
| Incremental writes and receipt identity | `TestCanonicalModelMigrationAndIncrementalLifecycle` verifies exact retries, unknown receipt semantics, logical ID coverage, participant versions, and zero writes/logical-ID consumption after a later batch condition fails. It checks canonical DB rows, not just HTTP status. | | Incremental writes and receipt identity | `TestCanonicalModelMigrationAndIncrementalLifecycle` verifies exact retries, unknown receipt semantics, logical ID coverage, participant versions, and zero writes/logical-ID consumption after a later batch condition fails. It checks canonical DB rows, not just HTTP status. |
| Canonical concurrent writers | `TestCanonicalConcurrentMigrationAndOwnershipWrites` races four identical migrations, then four identical edits, requiring identical receipts and one version advance. Two different requests then compete to move alternate endpoints and their common edge: exactly one wins, the loser has a queryable terminal version-conflict receipt, both canvas versions advance once, and the full ownership graph remains valid. Reusing the winning key with changed bytes is rejected without changing the model. | | Canonical concurrent writers | `TestCanonicalConcurrentMigrationAndOwnershipWrites` races four identical migrations, then four identical edits, requiring identical receipts and one version advance. Two different requests then compete to move alternate endpoints and their common edge: exactly one wins, the loser has a queryable terminal version-conflict receipt, both canvas versions advance once, and the full ownership graph remains valid. Reusing the winning key with changed bytes is rejected without changing the model. |
| Dissolve without deleting members | The real consumer creates nested child/grandchild canvases, dissolves the child and loses its successful response. Reopen queries the original request without a second POST. Direct members and the nested entry move to the parent, the leaf retains its grandchild owner/geometry, and the original entry/edge remain. Undo/redo/undo restores the original bindings. This uses the existing atomic edit protocol and the real HTTP/DB service. |
| Canonical frame geometry | `TestCanonicalFramesRetainRotationAcrossOwnerChanges` preserves nonzero frame rotation through owner changes, projection and an unrelated canonical AI patch, while rejecting 360 degrees. The legacy snapshot writer still rejects rotated frames; its restriction no longer rejects canonical coordinate conversion. |
| Real frontend coordination | `TestHeadlessClientHTTPIntegration` runs `tests/headless/foundry.integration.mjs` against authenticated HTTP and PostgreSQL. It verifies lost document ACK recovery without duplicate POST, atomic partition edits, deep copy with a parent-owned incoming edge, partial import and old stream-tail coordination. | | Real frontend coordination | `TestHeadlessClientHTTPIntegration` runs `tests/headless/foundry.integration.mjs` against authenticated HTTP and PostgreSQL. It verifies lost document ACK recovery without duplicate POST, atomic partition edits, deep copy with a parent-owned incoming edge, partial import and old stream-tail coordination. |
| Resources and authorization | The consumer verifies lost upload ACK, shared pending-image upload, original bytes, metadata, cross-project adoption and reopen. `TestOnboardingAndSharing` additionally revokes source access after warm reads, verifies no target model/version/binding changes, preserves the rejected receipt after permission restoration, and checks independent target access. | | Resources and authorization | The consumer verifies lost upload ACK, shared pending-image upload, original bytes, metadata, cross-project adoption and reopen. `TestOnboardingAndSharing` additionally revokes source access after warm reads, verifies no target model/version/binding changes, preserves the rejected receipt after permission restoration, and checks independent target access. |
| AI execution identity and controls | `TestCanonicalAIAdmissionAndLifecycle` covers admission/query/retry, zero-effect rejection, old incarnation callbacks, cancellation, managed image output, owner relocation, archived admission routes and execution after permission revocation. | | AI execution identity and controls | `TestCanonicalAIAdmissionAndLifecycle` covers admission/query/retry, zero-effect rejection, old incarnation callbacks, cancellation, managed image output, owner relocation, archived admission routes and execution after permission revocation. |
@@ -65,6 +74,12 @@ identity/abort fences and terminal rejection followed by explicit preparation.
asset adapter now accepts a host `displayURL(id)` callback. Canonical source asset adapter now accepts a host `displayURL(id)` callback. Canonical source
identity and authority remain separate; preview bytes are not used as original identity and authority remain separate; preview bytes are not used as original
file digest evidence. Authentication stays in the host fetch/transport. file digest evidence. Authentication stays in the host fetch/transport.
3. CI #1184/#1185 rejected the nested creation before reaching dissolution:
canonical validation had copied the legacy frame's zero-rotation rule. A
rotated owner can require a frame to retain nonzero rotation after conversion.
Foundry `506f778` removes that canonical-only restriction; existing finite/range
checks and legacy normalization remain. The consumer now checks the creation
receipt before reading descendants, so a rejection reports its actual state.
The test harness also needed working controlled timers, a cancelled run's final The test harness also needed working controlled timers, a cancelled run's final
event drain before the next admission, and actual copy-source file persistence. event drain before the next admission, and actual copy-source file persistence.
@@ -73,6 +88,29 @@ copy stage permits source checkpoints but rejects bitmap file duplication and
asserts no extra upload or generation. Those fixes do not bypass busy, cursor, asserts no extra upload or generation. Those fixes do not bypass busy, cursor,
durability or resource-sharing checks. durability or resource-sharing checks.
## Final-audit recovery cuts
`crash-windows.test.mjs` selects actual immutable cuts produced by the engine:
before local durability, durable pending intent, frozen request, dispatch-marked
before server commit, dispatch-marked after server commit, and durable receipt/base.
Each independent recovery pairs that cut with its corresponding server/receipt
state, then checks document/history/logical IDs, exact frozen request bytes,
POST/query counts and final durability. Frozen recovery may conservatively query
before its exact retry; a missing receipt never creates a new identity. These are
memory StoragePort/protocol fixtures, not physical browser-process termination.
Actual IndexedDB atomic rollback, fencing/checksum/journal-gap and reopening
evidence remains the separate browser suite; this is not every possible cross-product.
`engine.test.mjs` additionally commits a create, loses its ACK, deletes locally,
reopens, queries the original create and finally sends the dependent deletion.
The server has no residual element; both original history units remain and the
creation is not submitted twice. The focused engine/crash suite passes 36 tests;
the full Node suite passes 821. `resources.test.mjs` also verifies replacement of
an existing image while the new upload is gated: the old server reference stays
valid, undo prevents a late replacement, and redo/reopen reuse the two uploaded
resources without a new upload or element incarnation. Its 32 checks pass.
The final acceptance matrix is still in progress.
## Boundaries and remaining audit ## Boundaries and remaining audit
The recovery store is an in-memory durable-batch fixture, and response loss is The recovery store is an in-memory durable-batch fixture, and response loss is
@@ -115,7 +153,7 @@ are not proved by that fixture. `TestCanvasAIRealWorker` remains opt-in and skip
The CI log includes an early skip of `TestHeadlessClientHTTPIntegration` without The CI log includes an early skip of `TestHeadlessClientHTTPIntegration` without
its opt-in environment and a later successful execution in the explicit Node/DB its opt-in environment and a later successful execution in the explicit Node/DB
phase. Those are different invocations. The latest log has 157 distinct PASS names phase. Those are different invocations. The latest successful log has 158 distinct PASS names
including subtests; that count is not 154 independent end-to-end scenarios. including subtests; that count is not 154 independent end-to-end scenarios.
The service audit covers the original stage requirements: target incremental and The service audit covers the original stage requirements: target incremental and
+53
View File
@@ -0,0 +1,53 @@
import {test} from 'node:test';
import assert from 'node:assert/strict';
import {createDocument,createRegistry,builtinComponents,applyChanges} from '../../src/lib/canvas-core/index.ts';
import {openProjectEngine} from '../../src/lib/canvas-runtime/engine.ts';
import {decodeRecovery} from '../../src/lib/canvas-runtime/recovery.ts';
import {createFoundryPersistence} from '../../src/lib/canvas-service/foundry.ts';
const root='10000000-0000-4000-8000-000000000001',workspace='10000000-0000-4000-8000-000000000002',account='10000000-0000-4000-8000-000000000003';
const registry=createRegistry(builtinComponents()),target={id:'a',incarnation:1},original=createDocument(root,[{id:root,parent:null}],[{...target,ownerCanvasId:root,componentId:'text',dataVersion:1,structure:'node',geometry:{x:0,y:0,width:100,height:80,rotation:0},data:{title:'Original',text:''}}]);
function fixture(image,initialServer=original,initialReceipt){
let owner,saved=image&&structuredClone(image),server=initialServer,version=initialReceipt?2:1,now=1000,next=0;
const cuts=[],calls=[],timers=new Map(),receipts=new Map(initialReceipt?[[initialReceipt.request_id,structuredClone(initialReceipt)]]:[]);
const model=()=>({documentFormatVersion:1,projectId:root,complete:false,canvases:[...server.canvases.values()],elements:[...server.elements.values()],tombstones:Object.fromEntries(server.tombstones),versions:{[root]:version},removedCanvasIds:[]});
const transport=async input=>{
calls.push(input);
if(input.method==='GET'){const receipt=receipts.get(input.path.split('/').at(-1));return receipt?{status:200,body:receipt}:{status:404,body:{status:'unknown',request_id:input.path.split('/').at(-1)}};}
if(receipts.has(input.requestId))return{status:200,body:receipts.get(input.requestId)};
const body=JSON.parse(input.body);assert.equal(body.expected_versions[root],version);server=applyChanges(server,body.operations.flatMap(operation=>operation.changes));version++;
const receipt={protocol_version:2,request_id:input.requestId,status:'applied',operation_ids:body.operations.map(operation=>operation.id),model:model()};receipts.set(input.requestId,receipt);return{status:200,body:receipt};
};
const storage={async claim(scope){assert.equal(owner,undefined);return owner={scope,active:true,fence:1,async release(){owner=undefined;this.active=false;}};},async recover(){return saved&&structuredClone(saved);},async commit(leader,batch){assert.strictEqual(leader,owner);saved=structuredClone(batch.image);cuts.push(saved);return{durableRootRevision:saved.rootRevision};},async readFile(){},close(){}};
const options={scope:{accountId:account,workspaceId:workspace,projectId:root},registry,document:original,versions:new Map([[root,1]]),storage,persistence:createFoundryPersistence({registry,workspaceId:workspace,rootId:root,transport}),clock:{now:()=>now,setTimer(callback,delay){const id=++next;timers.set(id,{callback,due:now+delay});return id;},clearTimer:id=>timers.delete(id)}};
return{cuts,calls,receipts,server:()=>server,open:()=>openProjectEngine(options),async drain(engine){for(let step=0;step<8;step++){await engine.settle();if([...engine.getSnapshot().runtime.ledger.operations.values()].every(operation=>operation.phase==='resolved'))return;now+=100000;for(const[id,timer]of[...timers])if(timer.due<=now){timers.delete(id);timer.callback();}}assert.fail('recovered operations did not settle');}};
}
test('each durable crash cut restores its own document/history/ledger and external side-effect state',async t=>{
const producer=fixture(),engine=await producer.open();await engine.persist();const initial=structuredClone(producer.cuts.at(-1));
const edited=engine.editor.dispatch({type:'element.data',target,patch:{title:'Changed'}});assert.equal(engine.getSnapshot().document.elements.get('a').data.title,'Changed');assert.ok(engine.getDurability().rootRevision>engine.getDurability().durableRootRevision);
await producer.drain(engine);const decoded=producer.cuts.map(image=>({image,state:decodeRecovery(image,registry)})),request=[...engine.getSnapshot().runtime.ledger.requests.values()][0],receipt=producer.receipts.get(request.id),changed=producer.server();
const pending=decoded.find(({state})=>state.runtime.ledger.operations.size===1&&state.runtime.ledger.requests.size===0),frozen=decoded.find(({state})=>state.runtime.ledger.requests.get(request.id)?.phase==='frozen'),dispatch=decoded.find(({state})=>state.runtime.ledger.requests.get(request.id)?.phase==='dispatch-marked'),resolved=decoded.findLast(({state})=>state.runtime.ledger.requests.get(request.id)?.phase==='resolved');
assert.ok(pending&&frozen&&dispatch&&resolved,'producer did not expose every required durable transition');await engine.close();
for(const scenario of[
{name:'visible edit before durability is absent after crash and was never safe to leave',image:initial,edited:false,posts:0,queries:0},
{name:'durable pending intent dispatches once with its original logical identity',image:pending.image,edited:true,posts:1,queries:0},
// Recovery conservatively queries this cut too; unknown permits the exact
// retry, not a new identity or a claim that no effect could have happened.
{name:'durable frozen request is reconciled then sends its original bytes only once',image:frozen.image,edited:true,posts:1,queries:1,exact:true},
{name:'dispatch-marked before server commit queries unknown then retries the exact request',image:dispatch.image,edited:true,posts:1,queries:1,exact:true},
{name:'server commit before local receipt queries without repeating the write',image:dispatch.image,edited:true,posts:0,queries:1,server:changed,receipt,exact:true},
{name:'durable receipt and baseline do not replay the acknowledged edit or history',image:resolved.image,edited:true,posts:0,queries:0,server:changed,receipt},
])await t.test(scenario.name,async()=>{
// The server state is matched to the cut: it is never rewound after a
// dispatched effect merely because an older local image was selected.
const f=fixture(scenario.image,scenario.server,scenario.receipt),recovered=await f.open();
try{
await f.drain(recovered);const state=recovered.getSnapshot();assert.equal(state.document.elements.get('a').data.title,scenario.edited?'Changed':'Original');assert.equal(f.server().elements.get('a').data.title,scenario.edited?'Changed':'Original');assert.equal(state.history.undo.length,scenario.edited?1:0);assert.equal(state.operations.size,scenario.edited?1:0);
assert.equal(f.calls.filter(call=>call.method==='POST').length,scenario.posts);assert.equal(f.calls.filter(call=>call.method==='GET').length,scenario.queries);
if(scenario.edited)assert.ok(state.runtime.ledger.operations.has(edited.operationIds[0]));
if(scenario.exact)for(const call of f.calls){assert.equal(call.requestId??call.path.split('/').at(-1),request.id);if(call.method==='POST')assert.equal(call.body,request.body);}
assert.equal(recovered.getDurability().rootRevision,recovered.getDurability().durableRootRevision);assert.equal(recovered.sessions.getSnapshot().views.size,0);
}finally{await recovered.close();}
});
});
+15
View File
@@ -86,6 +86,21 @@ test('closing a view keeps jobs alive; closing/reopening the engine restores his
await recovered.engine.close(); await recovered.engine.close();
}); });
test('create applied with a lost ACK followed by delete recovers the original request and leaves no server element',async()=>{
const storage=storageFixture(),port=persistenceFixture(),submit=port.submit;let lost=true,now=1000,next=0;
port.submit=async request=>{const status=await submit(request);if(lost){lost=false;return{outcome:'unresolved'};}return status;};
const options={clock:{now:()=>now,setTimer:()=>0,clearTimer(){}},createId:()=>`lifetime-${++next}`};let{engine}=await open(storage,port,options);
try{
const element={...original.elements.get('a'),id:'temporary'},created=engine.editor.dispatch({type:'element.create',element});await engine.settle();
const request=[...engine.getSnapshot().runtime.ledger.requests.values()][0];assert.equal(request.outcome,'unresolved');assert.equal(port.receipts.get(request.id).document.elements.has(element.id),true);
const removed=engine.editor.dispatch({type:'element.remove',target:{id:element.id,incarnation:1}});await engine.settle();assert.equal(port.calls.length,1);assert.equal(engine.getSnapshot().runtime.ledger.operations.get(removed.operationIds[0]).phase,'pending');assert.ok(engine.getSnapshot().runtime.ledger.operations.get(removed.operationIds[0]).intent.dependencies.includes(created.operationIds[0]));
await engine.close();now=10000;({engine}=await open(storage,port,options));await engine.settle();
assert.deepEqual(port.calls.map(([kind])=>kind),['submit','query','submit']);assert.equal(port.calls[1][1].id,request.id);assert.equal(port.calls[1][1].body,request.body);assert.notEqual(port.calls[2][1].id,request.id);
const final=port.receipts.get(port.calls[2][1].id).document;assert.equal(final.elements.has(element.id),false);assert.equal(final.tombstones.get(element.id),1);assert.equal(engine.getSnapshot().document.elements.has(element.id),false);assert.equal(engine.getSnapshot().history.undo.length,2);
for(const id of[...created.operationIds,...removed.operationIds])assert.equal(engine.getSnapshot().runtime.ledger.operations.get(id).phase,'resolved');
}finally{await engine.close();}
});
test('failed local durability preserves editing, prevents sends, and failed close retains the recovery leader', async () => { test('failed local durability preserves editing, prevents sends, and failed close retains the recovery leader', async () => {
const storage = storageFixture(), port = persistenceFixture(), { engine, errors } = await open(storage, port); const storage = storageFixture(), port = persistenceFixture(), { engine, errors } = await open(storage, port);
await engine.persist(); storage.broken = true; await engine.persist(); storage.broken = true;
+14
View File
@@ -223,6 +223,20 @@ test('lost upload acknowledgment recovers original byte identity and query recei
await recovered.close(); await recovered.close();
}); });
test('replacing an image keeps the old server reference during upload and undo/reopen/redo never uploads again',async()=>{
const f=fixture();let engine=await f.open();const original=addImage(engine);await engine.settle();const upload=f.assets.upload,gate=deferred(),entered=deferred();let replacement;
f.assets.upload=async(...args)=>{entered.resolve();await gate.promise;return upload(...args);};
try{
const result=engine.stageFiles([{name:'replacement.png',bytes:new Blob(['different image bytes'],{type:'image/png'})}],([ref],tx)=>{replacement=ref;tx.dispatch({type:'element.data',target:{id:'picture',incarnation:1},patch:{assetId:ref.id,src:ref.src}});});assert.equal(result.status,'applied');
assert.equal(engine.getSnapshot().document.elements.get('picture').data.assetId,replacement.id);assert.equal(engine.getSnapshot().history.undo.length,2);await entered.promise;
assert.equal(f.writes.length,1);assert.equal(f.persistence.canonical(f.writes[0]).document.elements.get('picture').data.assetId,original.id);
assert.equal(engine.editor.history.undo().status,'applied');assert.equal(engine.getSnapshot().document.elements.get('picture').data.assetId,original.id);gate.resolve();await engine.settle();
const last=()=>f.persistence.canonical(f.writes.at(-1)).document.elements.get('picture');assert.equal(last().data.assetId,original.id);assert.equal(last().incarnation,1);assert.equal(f.uploads.length,2);assert.equal(engine.getSnapshot().history.undo.length,1);
assert.equal(engine.editor.history.redo().status,'applied');await engine.settle();assert.equal(last().data.assetId,replacement.id);assert.equal(f.uploads.length,2);assert.equal(engine.getSnapshot().history.undo.length,2);assert.ok(engine.getResourcePins().pins.has(original.id));
await engine.close();engine=await f.open();assert.equal(engine.editor.history.undo().status,'applied');await engine.settle();assert.equal(last().data.assetId,original.id);assert.equal(engine.editor.history.redo().status,'applied');await engine.settle();assert.equal(last().data.assetId,replacement.id);assert.equal(f.uploads.length,2);assert.equal(last().incarnation,1);
}finally{gate.resolve();await engine.close();}
});
test('rejected and empty staging do not retain file bytes or create upload work', async () => { test('rejected and empty staging do not retain file bytes or create upload work', async () => {
const f = fixture(), engine = await f.open(); const f = fixture(), engine = await f.open();
const files = [{ name: 'image.png', bytes: new Blob(['image'], { type: 'image/png' }) }]; const files = [{ name: 'image.png', bytes: new Blob(['image'], { type: 'image/png' }) }];