Slice detached element recovery encoding and audit progressive canvas

headless/README.md

@@ -1001,8 +1001,8 @@ is suspended; the old image still contains its captured state, and the next cut
 is saved after it. Request dispatch retains the durable-prefix barrier. Encoding
 or scheduling failures retain the pending Root for retry and produce no partial
 storage commit. Initial DTO membership copying, synchronous preparation, unknown
-subtrees, first-time immutable normalization and individual scalar operations
-remain synchronous. This is partial time slicing, not a strict whole-Root latency
+subtrees and individual scalar operations remain synchronous. First-time element
+normalization can defer only when its detached JSON provenance is established. This is partial time slicing, not a strict whole-Root latency
 bound. Grouped history entries now keep shape ownership for newly built changes and
 operation-ID arrays, including undo/redo lifecycle remaps. Import runners keep it
 for their own job envelopes, completed-key arrays, binding dictionaries and
@@ -1136,3 +1136,21 @@ also rechecks its immutable fact cache when assembling a prepared container:
 when the same source occurs twice in one image, the second occurrence can reuse
 the first completed encoding. Preparation still captures unsafe occurrences
 independently; getters and mutable descendants never qualify for this reuse.
+
+First-time element recovery encoding now uses the core's detached-JSON ownership
+proof instead of synchronously normalizing every field before the first yield.
+Only `detachElementData` results qualify: JSON-cloned, recursively frozen records
+with immutable JSON descendants. A shallow freeze, a document factory envelope,
+or an arbitrary host object does not qualify. Preparation requires the standard
+Object/Array prototype chain without inherited `toJSON`; otherwise it captures
+through the original normalizer. Every unknown normalization invalidates this
+prototype check because callbacks may change the environment. The cooperative
+walk reads the already detached own data and validates depth while building the
+same immutable JSON and byte facts. A callback later in capture cannot change an
+earlier record's meaning. Cold and cached subtrees both retain the depth limit.
+This moves the first full field/row walk into the shared budget; container-member
+capture and individual native/scalar operations still have synchronous work.
+
+Step 4's current acceptance evidence, commands and explicit remaining checkpoint
+encoding work are recorded in [STEP4_ACCEPTANCE.md](STEP4_ACCEPTANCE.md). Passing
+functional fixtures is not a claim that all performance limits are resolved.

headless/STEP4_ACCEPTANCE.md

@@ -0,0 +1,68 @@
+# Step 4：渐进功能与呈现验收
+
+状态：已完成最终功能入口复核，调度余项未完成，整体进度 3/7。此文件随实现提交；完整目标仍包含后续真实服务、迁移和集成验收。
+
+## 范围与直接证据
+
+| 原要求 | 代码与验证入口 | 本次结果 |
+| --- | --- | --- |
+| 导入、深复制逐步加入并允许编辑 | `imports.ts`、`canvas-runtime/duplicate.ts`；`imports.test.mjs`、`imports-engine.test.mjs`、`import-document-stream.test.mjs`、`duplicate*.test.mjs`；实际页面文件选择/工具栏 | Node 与页面检查通过。SHA 完成前可显示和编辑；完整校验完成且对应 Root 持久化后才发送。单项失败保留已加入内容并报告 partial。复制保留原图、新建身份，父层新增连线，复用合法资源且不自动启动 run。 |
+| loader 旧捕获与后续编辑/ACK 协调 | `loading.ts`、`engine.ts`；`loading-engine.test.mjs`、浏览器存储和实际页面 | 当前 loading v1–v6 路径、未读 tail、undo/redo、重开、取消及源校验检查通过。旧数据不覆盖后来的编辑/ACK；已声明的 partial 状态保留。 |
+| 公共 hooks 和局部订阅 | `canvas-bindings/react.ts`、`canvas-bindings`；`react-hooks-fixture.tsx`、`renderer.integration.mjs --hooks-only` | 同一最终源码的 StrictMode 入口 10 项通过；包括 owner 切换、engine close、资源句柄与 unloaded/absent。普通字段与 ACK 的无关投影复用由实际页面计数验证。 |
+| 输入、呈现与合作调度 | frame queue、scene/paint projection、retained projection；`frame-queue.test.mjs`、`retained-projection.test.mjs`、实际页面与生产规模入口 | 页面验证拖动最终位置、IME、Portal、detach、未知焦点与订阅取消。生产万元素早/中/尾段输入与副本捕获隔离通过；性能仍有限制，见下文。 |
+
+所有 `*.test.mjs` 均位于 `tests/headless/`，源码位于 `src/lib/` 或 `src/components/canvas/`。每个测试文件中的断言决定其覆盖范围；通过总数不代表 Draft 的 T01–T36 或整个项目已经完成。
+
+## 编码与发布边界
+
+1. `writePrefix` 是有序单 writer，逐个保存 Root，异步编码不会跳过中间 Root。
+2. 外部对象、getter、custom `toJSON` 和浅冻结值在第一次让出前按原 codec 捕获；私有容器只证明成员固定，内部未知值仍需捕获。
+3. core 的 JSON 脱离、深冻结元素可以延后逐字段编码。需要标准 Object/Array 原型且没有继承的 `toJSON`；每次未知值捕获后重新检查环境。外部回调不能改变已经捕获的前项，也不能让后项错误沿用旧环境。
+4. 标量归一化、UTF-8 计数、history/import/pending 容器与首次元素遍历共用软预算。已完成的深不可变编码可复用，但不能省略 unsafe occurrence 的求值。
+5. 编码、调度或存储失败保留待存 Root；持久化没有确认，不解除网络发送屏障。旧 Root 与编码期间的新编辑保持各自内容。
+6. journal 比较先判断子值是否相同，再为变化项分配路径，避免为共享的整个前缀创建临时数组。patch 顺序与字节语义不变。
+
+`recovery-encoding.test.mjs` 验证首次归一化确实在 yield 间发生、继承行为、外部回调、Proxy trap 次数、UTF-8、冷/热深度边界；`engine.test.mjs` 以 17000 元素验证暂停编码期间编辑、逐 Root 落盘和发送屏障。`journal-delta.test.mjs` 验证精确重放及属性顺序；浏览器检查验证真实 IndexedDB 重开。
+
+## 可复现命令
+
+```sh
+node --test tests/headless/*.test.mjs
+npm run typecheck -- --incremental false
+node tests/headless/renderer.integration.mjs
+node tests/headless/renderer.integration.mjs --turbopack
+node tests/headless/renderer.integration.mjs --hooks-only
+node tests/headless/browser-storage.mjs
+node tests/headless/package.integration.mjs
+HEADLESS_PERFORMANCE_COUNT=10000 HEADLESS_PERFORMANCE_EDIT_DURING_COPY=1 HEADLESS_PERFORMANCE_REPORT=/private/tmp/headless-step4-production-10000.json node tests/headless/renderer.integration.mjs --performance --production --turbopack
+```
+
+性能入口须在其它构建/测试结束后单独执行；功能检查可并行。这里验证真实 Chrome、IndexedDB、Web Locks 和实际前端，但 HTTP 为隔离 fixture，不能据此声称已完成真实 Foundry 集成。
+
+
+## 本次运行结果（2026-11-11）
+
+- Node：780 项通过；全量前端 TypeScript 检查通过。
+- 实际页面：webpack 51 项、Turbopack 51 项通过；公共 hooks 10 项通过。
+- 浏览器存储：57 组通过；SDK：263 个打包文件，类型、独立消费和编译后的浏览器/Worker 检查通过。
+- webpack 曾在全部断言结束后被 Chrome 日志管道阻止退出。`renderer.integration.mjs` 现于 Chrome 退出后释放 stdio；重新运行的 51 项检查自动正常退出。
+- 生产规模环境：本机 arm64、macOS 15.6、Node 23.10.0、Chrome 157.0.8103.39，Next production Turbopack；无并行构建/测试、无计时代码插桩。
+- 10000-child 展开分区深复制：首副本呈现 799.5ms、本地完成帧 36264.8ms、加入并同步 36469.9ms；帧间隔 P95 33.4ms/max316.6ms；25 次 longtask，max341ms/total1869ms。
+- early 五次 input→两次 rAF：50.5/31.3/22.6/20.9/41.3ms；middle（5255 项）21.5ms、tail（9131 项）39.4ms。全部发生于复制活跃期间，DOM/focus 保留；中尾 CDP 往返约18.1/27.4ms。它们不含精确的派发前排队延迟。
+- 服务端最终 20005 元素/3 canvas；原/新父层边、原文后续输入、副本旧快照、pan 后副本挂载通过。样本呈现时挂载35节点，全过程最大136，pan 后28；屏幕外 IME/Portal 各保留1，独立应用面板不额外保活，runStartsDuringScenario=0。
+
+原始报告 `/private/tmp/headless-step4-production-10000.json` 和同名 PNG 为本次本机观察；命令与断言已保存在仓库，可独立复现。单次运行不同 adaptive 批次和 GC 负载，不能比较前轮26.2秒而宣称确定改善或回退，更不能说所有停顿已解决。
+
+## 明确剩余项
+
+复制规划的 `createDuplicatePlanner().checkpoint()` 当前返回 `freeze(clone(state))`；它具有 JSON 脱离性质，但这项来源尚未传递给 codec。`preparation.planner` / `source.plan` 因而仍可能经未知父对象走同步 normalization。下一项是由 core 提供同元素一致的、不可伪造的 JSON 脱离来源事实，并在 runtime 自建 checkpoint envelope 中保留这项事实，补首次 checkpoint 编码、外部回调与恢复测试。不能仅以 `Object.freeze` 推定来源，也不能跳过完整 checkpoint/发送屏障。
+
+此项完成并复验前，Step 4 保持未完成。本文件只确认已列入口和观察，未完成项不能转移到后端阶段来虚增进度。
+
+## 调度限制与后续分工
+
+6ms 是合作调度软目标。捕获成员引用、未知值的同步 JSON 语义、单个字符串/native 操作、原子模型发布和 GC 仍可能超过一个时间片。未知调用不能假冒不可变值以换取响应速度；也不能把普通循环停顿一概称为 GC。测试测量 input 到两次 rAF；CDP 往返另含传输和驱动时间，均不是精确的输入派发前排队延迟。
+
+未登记的第三方 Portal/未知输入 owner 保留明确的兼容回退；任意全局 Root selector、结构变化和未知 ACK 不保证 O(1)。renderer 不保存第二份业务文档，不为裁剪丢弃完整模型。
+
+下一阶段由服务/集成工作负责真实 Foundry 协议、权限、迁移、冻结请求幂等与恢复联调，并沿用现有远程 CI。分支推送不等于主线合并或部署，仍需单独报告。

src/lib/canvas-runtime/journal-delta.ts

@@ -9,11 +9,17 @@ export function diffJournal(before:Json,after:Json):JournalPatch[]|null {
   function visit(a:Json,b:Json,path:(string|number)[]){
     if(a===b||patches.length>10000)return;
     if(Array.isArray(a)&&Array.isArray(b)){
-      const common=Math.min(a.length,b.length);for(let i=0;i<common;i++)visit(a[i],b[i],[...path,i]);
+      const common=Math.min(a.length,b.length);for(let i=0;i<common;i++){
+        const previous=a[i],next=b[i];
+        // Shared immutable rows need no path allocation or recursive call.
+        if(previous!==next)visit(previous,next,[...path,i]);
+      }
       if(a.length!==b.length)patches.push({op:'splice',path,start:common,deleteCount:a.length-common,items:b.slice(common)});
     }else if(a&&b&&typeof a==='object'&&typeof b==='object'&&!Array.isArray(a)&&!Array.isArray(b)){
       const left=Object.keys(a),right=Object.keys(b);
-      if(left.length===right.length&&left.every((key,i)=>key===right[i]))for(const key of right)visit(a[key],b[key],[...path,key]);
+      if(left.length===right.length&&left.every((key,i)=>key===right[i]))for(const key of right){
+        const previous=a[key],next=b[key];if(previous!==next)visit(previous,next,[...path,key]);
+      }
       else patches.push({op:'set',path,value:b});
     }else patches.push({op:'set',path,value:b});
   }

src/lib/canvas-runtime/recovery-encoding.ts

@@ -1,3 +1,5 @@
+import {hasOwnedElementData} from '../canvas-core/element-data.ts';
+import type {ElementRecord} from '../canvas-core/types.ts';
 import type { Json } from '../canvas-core/types.ts';
 import { fault, freeze, validJson } from '../canvas-core/values.ts';
 import {isDataContainer} from '../canvas-core/data-containers.ts';
@@ -72,12 +74,20 @@ export function sealRecoveryImage(formatVersion:ImageVersion,rootRevision:number
 }
 
 type Scalar = null|string|number|boolean|undefined|symbol;
-type Prepared = Scalar | Encoded | {source:object;depth:number;children:Prepared[];keys?:string[]};
+type Prepared = Scalar | Encoded | {source:object;depth:number;children?:Prepared[];keys?:string[]};
 // Read unknown objects synchronously, before the first suspension. Shape-owned
 // containers can include external values; each such occurrence is captured now.
-// Only our fresh container shape and previously proven encoded values defer.
+// Detached element JSON can also defer after checking its prototype environment.
 function prepareEncoding(input:unknown) {
   const normalize=createEncoder(),visiting=new Set<object>();
+  let plainEnvironment:boolean|undefined;
+  const capture=(value:unknown,depth:number,key:string)=>{
+    try{return normalize(value,depth,key);}finally{plainEnvironment=undefined;}
+  };
+  const detached=(value:object)=>hasOwnedElementData(value as ElementRecord)&&(plainEnvironment??=(
+    Object.getPrototypeOf(Object.prototype)===null&&Object.getPrototypeOf(Array.prototype)===Object.prototype&&
+    !Object.getOwnPropertyDescriptor(Object.prototype,'toJSON')&&!Object.getOwnPropertyDescriptor(Array.prototype,'toJSON')
+  ));
   function prepare(value:unknown,depth:number,key:string):Prepared {
     const kind=typeof value;
     if(value===null||kind==='string'||kind==='number'||kind==='boolean'||kind==='undefined'||kind==='symbol'){
@@ -86,15 +96,18 @@ function prepareEncoding(input:unknown) {
       // as well as allocation of their Encoded facts to the budgeted traversal.
       return value as Scalar;
     }
-    if(typeof value!=='object')return normalize(value,depth,key);
+    if(typeof value!=='object')return capture(value,depth,key);
     const cached=immutable.get(value);
     if(cached){if(depth+cached.height>64)throw fault('invalid_recovery_payload');return cached;}
-    if(!isDataContainer(value))return normalize(value,depth,key);
+    // Detached element JSON has no accessors, custom prototypes or mutable
+    // descendants. Its full walk can wait; foreign freezes cannot grant this.
+    if(detached(value))return {source:value,depth};
+    if(!isDataContainer(value))return capture(value,depth,key);
     // Prototype toJSON changes must retain ordinary JSON semantics, including
     // accessors. Inspect descriptors without evaluating them an extra time.
     let prototype:object|null=value,hasToJSON=false;
     while(prototype){if(Object.getOwnPropertyDescriptor(prototype,'toJSON')){hasToJSON=true;break;}prototype=Object.getPrototypeOf(prototype);}
-    if(hasToJSON)return normalize(value,depth,key);
+    if(hasToJSON)return capture(value,depth,key);
     if(depth>64||visiting.has(value))throw fault('invalid_recovery_payload');
     visiting.add(value);
     try {
@@ -123,17 +136,23 @@ export async function sealRecoveryImageAsync(formatVersion:ImageVersion,rootRevi
     // can reuse that fact without rebuilding the same JSON containers.
     const cached=immutable.get(plan.source);
     if(cached){if(depth+cached.height>64)throw fault('invalid_recovery_payload');return cached;}
+    if(depth>64)throw fault('invalid_recovery_payload');
     let height=0,bytes=2,reusable=true,properties=0;
-    const array=plan.keys===undefined,result:Json[]|Record<string,Json>=array?[]:{};
-    for(let index=0;index<plan.children.length;index++){
-      const entry=plan.children[index],name=plan.keys?.[index]??String(index);
+    const array=Array.isArray(plan.source),keys=plan.children?plan.keys:array?undefined:Object.keys(plan.source);
+    const length=plan.children?.length??(array?(plan.source as unknown[]).length:keys!.length),result:Json[]|Record<string,Json>=array?[]:{};
+    for(let index=0;index<length;index++){
+      const name=keys?.[index]??String(index);
+      // No children array means detached JSON. Read only its immutable own data
+      // properties, with the prototype behavior captured during preparation.
+      const value=plan.children?undefined:(plan.source as Record<string,unknown>)[name];
+      const entry=plan.children?plan.children[index]:value&&typeof value==='object'?{source:value,depth:plan.depth+1}:value as Scalar;
       const child=entry===null||typeof entry!=='object'?normalize(entry,plan.depth+1,name):'height' in entry?entry:yield* encode(entry,plan.depth+1,name);
       reusable&&=child.reusable;
       if(array){
         if(child.value===undefined&&plan.depth+1>64)throw fault('invalid_recovery_payload');
         (result as Json[]).push(child.value??null);bytes+=(index?1:0)+(child.value===undefined?4:child.bytes);height=Math.max(height,1+child.height);
       }else if(child.value!==undefined){
-        const name=plan.keys![index];bytes+=(properties++?1:0)+scalarBytes(name)+1+child.bytes;
+        const name=keys![index];bytes+=(properties++?1:0)+scalarBytes(name)+1+child.bytes;
         Object.defineProperty(result,name,{value:child.value,enumerable:true});height=Math.max(height,1+child.height);
       }
       const pause=budget.step();if(pause)yield pause;

tests/headless/recovery-encoding.test.mjs

@@ -112,3 +112,53 @@ test('repeated owned containers reuse facts assembled earlier in the same asynch
  const captured=await sealRecoveryImageAsync(17,1,1,dataArray([unsafe,unsafe]));
  assert.deepEqual(captured.payload,[{getValue:{value:1}},{getValue:{value:2}}]);
 });
+
+const ownedRecord=async()=>{
+ const {detachElementData}=await import('../../src/lib/canvas-core/element-data.ts');
+ return detachElementData({id:'large',incarnation:1,structure:'node',ownerCanvasId:'canvas',componentId:'text',dataVersion:1,geometry:{x:0,y:0,width:100,height:100,rotation:0},data:{text:'',rows:Array.from({length:1000},(_,i)=>({index:i,text:'owned-中文😀-'+i}))}});
+};
+test('first-time detached element JSON is normalized inside slices rather than captured in one walk',async()=>{
+ const {sealRecoveryImageAsync,sealedRecoveryByteLength}=await import('../../src/lib/canvas-runtime/recovery-encoding.ts');
+ const element=await ownedRecord(),stringify=JSON.stringify;let strings=0,first,tick=0;
+ JSON.stringify=function(value,...args){if(typeof value==='string'&&value.startsWith('owned-'))strings++;return Reflect.apply(stringify,this,[value,...args]);};
+ try{
+  const image=await sealRecoveryImageAsync(17,1,1,element,{now:()=>tick++,frameBudgetMs:1,yieldTask:async()=>{first??=strings;}});
+  assert.ok(first>=0&&first<128,'detached data was fully encoded before the first yield');assert.equal(strings,1000);
+  assert.deepEqual(image.payload,element);assert.equal(sealedRecoveryByteLength(image),new TextEncoder().encode(stringify(image)).byteLength);
+ }finally{JSON.stringify=stringify;}
+});
+
+test('inherited toJSON accessors at admission keep detached elements on the synchronous capture path',async()=>{
+ const {sealRecoveryImageAsync}=await import('../../src/lib/canvas-runtime/recovery-encoding.ts');
+ const left=await ownedRecord(),right=await ownedRecord(),previous=Object.getOwnPropertyDescriptor(Object.prototype,'toJSON');let calls=0,yields=0,tick=0;
+ try{
+  Object.defineProperty(Object.prototype,'toJSON',{configurable:true,get(){calls++;return undefined;}});
+  const expected=sealRecoveryImage(17,1,1,left),syncCalls=calls;calls=0;
+  const image=await sealRecoveryImageAsync(17,1,1,right,{now:()=>tick++,frameBudgetMs:1,yieldTask:async()=>{yields++;}});
+  assert.equal(calls,syncCalls);assert.equal(yields,0);assert.deepEqual(image.payload,expected.payload);
+ }finally{if(previous)Object.defineProperty(Object.prototype,'toJSON',previous);else delete Object.prototype.toJSON;}
+});
+
+test('unknown callbacks invalidate the detached JSON environment without changing earlier captures',async()=>{
+ const {dataRecord}=await import('../../src/lib/canvas-core/data-containers.ts');
+ const {sealRecoveryImageAsync}=await import('../../src/lib/canvas-runtime/recovery-encoding.ts');
+ const previous=Object.getOwnPropertyDescriptor(Array.prototype,'toJSON');
+ const cleanup=()=>{if(previous)Object.defineProperty(Array.prototype,'toJSON',previous);else delete Array.prototype.toJSON;};
+ const make=async()=>dataRecord({left:await ownedRecord(),bridge:{get value(){Object.defineProperty(Array.prototype,'toJSON',{configurable:true,value(){return ['changed-by-callback'];}});return 'bridge';}},right:await ownedRecord()});
+ try{
+  const expected=sealRecoveryImage(17,1,1,await make());cleanup();
+  const payload=await make();let tick=0,yields=0;
+  const pending=sealRecoveryImageAsync(17,1,1,payload,{now:()=>tick++,frameBudgetMs:1,yieldTask:async()=>{yields++;}});
+  cleanup();const image=await pending;assert.ok(yields>0);assert.deepEqual(image.payload,expected.payload);
+  assert.equal(image.payload.left.data.rows.length,1000);assert.deepEqual(image.payload.right.data.rows,['changed-by-callback']);
+ }finally{cleanup();}
+});
+
+test('deferred detached subtrees enforce the depth bound both cold and after cache reuse',async()=>{
+ const {dataRecord}=await import('../../src/lib/canvas-core/data-containers.ts');
+ const {sealRecoveryImageAsync}=await import('../../src/lib/canvas-runtime/recovery-encoding.ts');
+ const nest=leaf=>{for(let i=0;i<61;i++)leaf=dataRecord({child:leaf});return leaf;};
+ const cold=await ownedRecord();await assert.rejects(sealRecoveryImageAsync(17,1,1,nest(cold)),/invalid_recovery_payload/);
+ const warm=await ownedRecord();await sealRecoveryImageAsync(17,1,1,warm);
+ await assert.rejects(sealRecoveryImageAsync(17,1,1,nest(warm)),/invalid_recovery_payload/);
+});

tests/headless/renderer.integration.mjs

@@ -591,7 +591,7 @@ module.exports=function(source){return nativeTransform.call(this,source).replace
   }
 } finally {
   releaseScopeStream();releaseAbandoned();releaseFirstModel();releaseImport();releaseApply();releaseUpload(); socket?.close();
-  if (chrome) { const exit = new Promise(resolve => { if (chrome.exitCode !== null) resolve(); else chrome.once('exit', resolve); }); chrome.kill(); await Promise.race([exit,pause(3000)]);if(chrome.exitCode===null)chrome.kill('SIGKILL');await exit; }
+  if (chrome) { const exit = new Promise(resolve => { if (chrome.exitCode !== null) resolve(); else chrome.once('exit', resolve); }); chrome.kill(); await Promise.race([exit,pause(3000)]);if(chrome.exitCode===null)chrome.kill('SIGKILL');await exit;chrome.stdin?.destroy();chrome.stdout?.destroy();chrome.stderr?.destroy(); }
   if (next?.pid) { const exit = new Promise(resolve => { if (next.exitCode !== null) resolve(); else next.once('exit', resolve); }); try { process.kill(-next.pid, 'SIGTERM'); } catch {} await Promise.race([exit, pause(3000)]); if (next.exitCode === null) try { process.kill(-next.pid, 'SIGKILL'); } catch {} }
   upstream.closeAllConnections(); await new Promise(resolve => upstream.close(resolve));
   await rm(directory, { recursive: true, force: true, maxRetries: 5 }); await rm(profile, { recursive: true, force: true, maxRetries: 5 });