Measure CPU block activation sparsity and energy concentration

Reviewed-on: #4

README.md

@@ -38,3 +38,7 @@ block scales are alternative partitions and their savings must not be summed.
 
 Dependencies are pinned in `requirements.txt`; install only prebuilt wheels on the remote host.
 Full resolved versions and run provenance accompany each result. See the SOP for commands.
+
+## CPU block activation measurements
+
+See [measurement definitions](activation/README.md) and [observed activation report](results/block-activation/REPORT.md) for all 24 layers, contiguous output-channel blocks, exact/near-zero rates and output energy concentration. This is an observational SiLU-gated study, not a pruning or reuse quality claim.

activation/README.md

@@ -0,0 +1,45 @@
+# Observed block activation, CPU only
+
+Measures the retained Qwen3.5-2B checkpoint without training or substitutions.
+This model uses SiLU-gated MLPs and sigmoid attention gates, not ReLU.
+Leaky ReLU also does not generally create exact zero on negative inputs.
+
+The probe records every linear projection in all 24 layers, plus MLP SiLU
+and gate-times-up intermediates and full-attention sigmoid gates. It does not
+record every internal recurrent state, convolution output or attention probability.
+A block means contiguous **output channels** (64 or 128), hence contiguous
+rows of a projection matrix; it is not a Transformer layer or arbitrary 2D tile.
+MLP gated-product channels correspond to columns consumed by down_proj.
+It is incorrect to call an individual weight active/inactive based only on its value.
+
+For every sample, module and block, metrics.csv reports:
+- exact zero and negative element fractions;
+- near-zero fractions at |x| <= 0.01 RMS and <= 0.1 RMS;
+- fraction of channels ever above 0.01 RMS over the 32 tokens;
+- minimum fraction of channels carrying 90% squared-output energy.
+
+RMS is computed within each observed sample/module/block; width=0 records the
+whole module. Thus thresholds are explicitly relative and are not shared absolute
+cutoffs across layers or blocks. Activation rate at a threshold is 1-near-zero rate.
+A small value or an energy-concentrated output does not prove that a block can
+be skipped, removed or shared without affecting LM quality. Exact zero is measured
+in this FP32 streaming implementation of BF16 source weights; other execution dtypes
+can have different underflow rates.
+
+Four prior authored Chinese/English texts, 32 tokens each, descriptive measurements
+only. No training, fresh-generalization claim, GPU, weight download or cleanup.
+The existing streaming implementation was checked against the official implementation
+in the earlier CPU study; this experiment changes only observation, not model equations.
+
+Remote commands (authorized server 8.211.154.134):
+```
+OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -B -m activation.run --source runs/cpu-qwen35-2b --out runs/NEW_ACTIVATION_DIR
+OPENBLAS_NUM_THREADS=1 venv/bin/python -B -m unittest tests.test_activation -v
+venv/bin/python -B activation/report.py runs/NEW_ACTIVATION_DIR
+```
+
+run.py measures; report.py plots saved rows and derives aggregates;
+__init__.py marks the package; tests/test_activation.py checks exact sparsity,
+ever-active behavior, all-zero inputs and scale invariance. Raw activations are
+not saved or needed for these descriptive statistics. Existing caches and weights
+are retained untouched. Full sample token IDs and CPU time/RSS appear in summary.json.

activation/audit.py

@@ -0,0 +1,28 @@
+"""Audit recorded scope, metrics and reproducibility files; no model execution."""
+import csv,json,hashlib
+from pathlib import Path
+import sys
+p=Path(sys.argv[1]);r=list(csv.DictReader((p/'metrics.csv').open()));s=json.loads((p/'summary.json').read_text())
+assert s['completed'] and len(r)==s['rows']==4*20904
+assert len({x['sample'] for x in r})==4
+assert {int(x['module'].split('.')[1]) for x in r}==set(range(24))
+for x in r:
+    for k in ['zero_rate','negative_rate','near_001_rms','near_01_rms','ever_active_001','energy90_channel_fraction']:assert 0<=float(x[k])<=1,(x,k)
+    assert float(x['zero_rate'])<=float(x['near_001_rms'])<=float(x['near_01_rms'])
+    assert int(x['channels'])==int(x['stop'])-int(x['start'])
+a=json.loads((p/'aggregates.json').read_text());lines=['# 实测激活率：SiLU 门控模型','', '覆盖24层、4条中英文文本，每条32 token；所有投影输出及 MLP 门控中间量，按64/128输出通道分块。','', '| 输出 | 精确零比例均值 | 0.01 RMS以下均值 | 0.1 RMS以下均值 | 32 token内曾活跃通道比例均值 | 90%能量所需通道比例均值 |','|---|---:|---:|---:|---:|---:|']
+for k,v in a.items():
+    cols=[v[m]['mean']*100 for m in ['zero_rate','near_001_rms','near_01_rms','ever_active_001','energy90_channel_fraction']]
+    lines.append('| '+k+' | '+' | '.join(f'{z:.4f}%' for z in cols)+' |')
+blocks=[x for x in r if x['width']=='128']; dead=sum(float(x['zero_rate'])==1 for x in blocks)
+lines+=['',f'128通道块×文本观测共{len(blocks)}条，整个32 token窗口全为精确零的块观测数为{dead}。', '', '表中是整模块逐文本统计再取均值；每个分块的独立统计见 metrics.csv。阈值是各自RMS的比例；阈值下激活率=1−近零比例。能量集中不等于可删除或可替换。', '', '仅测该模型、四条短文本；不推断所有语言/长上下文。SiLU不把负值整体截零，负数不等于静默。未执行删块/跳算的质量验证。权重和既有缓存全部保留。', '',f"CPU测量耗时{s['elapsed_seconds']:.1f}秒，峰值RSS{s['peak_rss_kib']/1024:.1f} MiB。"]
+(p/'REPORT.md').write_text('\n'.join(lines)+'\n')
+lines_extra=[]
+for suffix in ['mlp.gated_product','self_attn.q_proj.output','self_attn.k_proj.output','self_attn.v_proj.output']:
+    rr=[x for x in blocks if x['module'].endswith(suffix)]
+    act=sorted(1-float(x['near_01_rms']) for x in rr)
+    lines_extra.append(f'{suffix}: 128通道块在0.1 RMS阈值上的激活率范围 {100*act[0]:.2f}%–{100*act[-1]:.2f}%。')
+with (p/'REPORT.md').open('a') as f:f.write('\n'+'\n'.join(lines_extra)+'\n')
+(p/'audit.json').write_text(json.dumps({'passed':True,'rows':len(r),'layers':24,'samples':4,'all_zero_128_blocks':dead},indent=2))
+files=[x for x in p.rglob('*') if x.is_file() and x.name!='checksums.json'];(p/'checksums.json').write_text(json.dumps({str(x.relative_to(p)):hashlib.sha256(x.read_bytes()).hexdigest() for x in sorted(files)},indent=2))
+print((p/'REPORT.md').read_text())

activation/report.py

@@ -0,0 +1,38 @@
+"""Render scientific figures from recorded CPU measurements."""
+import csv,json
+from pathlib import Path
+import numpy as np
+import matplotlib
+matplotlib.use('Agg')
+import matplotlib.pyplot as plt
+import sys
+root=Path(sys.argv[1]); rows=list(csv.DictReader((root/'metrics.csv').open()))
+full=[r for r in rows if r['width']=='0']
+fig,axs=plt.subplots(1,3,figsize=(15,4.5),layout='constrained')
+for suffix,label in [('gate_proj.output','gate preactivation'),('silu_gate','SiLU gate'),('gated_product','gated product')]:
+    selected=[r for r in full if r['module'].endswith('mlp.'+suffix)]
+    for ax,key in zip(axs,['zero_rate','near_001_rms','near_01_rms']):
+        xs=[];ys=[]
+        for layer in range(24):
+            z=[float(r[key]) for r in selected if r['module'].startswith(f'layers.{layer}.')];xs.append(layer);ys.append(np.mean(z))
+        ax.plot(xs,ys,label=label)
+for ax,title in zip(axs,['Exact zero fraction','Near zero: |x| <= 0.01 RMS','Near zero: |x| <= 0.1 RMS']):
+    ax.set(title=title,xlabel='Layer index',ylabel='Fraction');ax.set_ylim(bottom=0);ax.grid(alpha=.2)
+axs[-1].legend();fig.suptitle('Observed MLP activation sparsity: four texts, 32 tokens each; no training')
+root.joinpath('figures').mkdir(exist_ok=True)
+for ext in ('png','pdf','svg'):fig.savefig(root/'figures'/('mlp_activation.'+ext),dpi=180)
+plt.close(fig)
+blocks=[r for r in rows if r['width']=='128' and r['module'].endswith('mlp.gated_product')]
+fig,axs=plt.subplots(1,2,figsize=(12,4.5),layout='constrained')
+for ax,key,title in zip(axs,['near_01_rms','energy90_channel_fraction'],['Near-zero fraction at 0.1 block RMS','Fraction of channels carrying 90% energy']):
+    matrix=[]
+    for layer in range(24):
+        rr=[r for r in blocks if r['module'].startswith(f'layers.{layer}.')]; starts=sorted(set(int(r['start']) for r in rr));matrix.append([np.mean([float(r[key]) for r in rr if int(r['start'])==s]) for s in starts])
+    im=ax.imshow(matrix,aspect='auto',vmin=0,vmax=1);ax.set(title=title,xlabel='Contiguous 128-channel block',ylabel='Layer index');fig.colorbar(im,ax=ax)
+fig.suptitle('MLP gated-product blocks: average over four observed texts')
+for ext in ('png','pdf','svg'):fig.savefig(root/'figures'/('block_activity.'+ext),dpi=180)
+agg={}
+for suffix in ['mlp.gate_proj.output','mlp.silu_gate','mlp.gated_product','self_attn.q_proj.output','self_attn.k_proj.output','self_attn.v_proj.output']:
+    rr=[r for r in full if r['module'].endswith(suffix)]
+    agg[suffix]={'observations':len(rr),**{key:{'min':min(float(r[key]) for r in rr),'mean':float(np.mean([float(r[key]) for r in rr])),'max':max(float(r[key]) for r in rr)} for key in ['zero_rate','negative_rate','near_001_rms','near_01_rms','ever_active_001','energy90_channel_fraction']}}
+(root/'aggregates.json').write_text(json.dumps(agg,indent=2))

activation/run.py

@@ -0,0 +1,48 @@
+"""CPU-only observed activation sparsity; no checkpoint changes or training."""
+import json,csv,time,resource,hashlib
+from pathlib import Path
+import numpy as np
+from tokenizers import Tokenizer
+from functional.model import StreamingLM,silu,sigmoid
+
+
+def metrics(x):
+    x=np.asarray(x,dtype=np.float64).reshape(-1,x.shape[-1]); a=np.abs(x); energy=np.square(x).sum(0); total=energy.sum()
+    rms=float(np.sqrt(np.mean(x*x))); cutoff=.01*rms
+    ordered=np.sort(energy)[::-1]; n90=int(np.searchsorted(np.cumsum(ordered),.9*total)+1) if total else 0
+    return dict(elements=x.size,channels=x.shape[-1],rms=rms,zero_rate=float(np.mean(x==0)),negative_rate=float(np.mean(x<0)),near_001_rms=float(np.mean(a<=cutoff)),near_01_rms=float(np.mean(a<=.1*rms)),ever_active_001=float(np.mean(np.any(a>cutoff,axis=0))),channels_for_90pct_energy=n90,energy90_channel_fraction=n90/x.shape[-1])
+
+class Probe(StreamingLM):
+    def record(self,x,name):
+        for width in (0,64,128):
+            for start in range(0,x.shape[-1],width or x.shape[-1]):
+                stop=min(start+(width or x.shape[-1]),x.shape[-1]); z=x[...,start:stop]
+                self.rows.append(dict(sample=self.sample,module=name,width=width,start=start,stop=stop,**metrics(z)))
+    def linear(self,x,name,rows=256):
+        y=super().linear(x,name,rows); self.record(y,name+'.output')
+        if name.endswith('self_attn.q_proj'):
+            dim=self.config['head_dim']; mixed=y.reshape(*y.shape[:-1],-1,2*dim)
+            self.record(sigmoid(mixed[...,dim:]).reshape(*y.shape[:-1],-1),name+'.sigmoid_gate')
+        return y
+    def mlp(self,h,layer):
+        p=f'layers.{layer}.mlp.'; pre=self.linear(h,p+'gate_proj'); g=silu(pre); u=self.linear(h,p+'up_proj')
+        self.record(g,p+'silu_gate'); self.record(g*u,p+'gated_product')
+        return self.linear(g*u,p+'down_proj')
+
+def main():
+    import argparse
+    p=argparse.ArgumentParser();p.add_argument('--source',type=Path,required=True);p.add_argument('--out',type=Path,required=True);a=p.parse_args();a.out.mkdir(parents=True,exist_ok=False)
+    started=time.monotonic();cfg=json.loads((a.source/'model_config.json').read_text()); lm=Probe(a.source/'weights',cfg,seconds=3600);lm.rows=[]
+    samples=json.loads(Path('crossqkv/fresh_samples.json').read_text())['samples'];tok=Tokenizer.from_file(str(a.source/'tokenizer.json'))
+    for s in samples:
+        lm.sample=s['id']; ids=tok.encode(s['text']).ids[:32];s['token_ids']=ids
+        lm.forward(np.array([ids],dtype=np.int64))
+        first=not (a.out/'metrics.csv').exists()
+        with (a.out/'metrics.csv').open('a') as f:
+            w=csv.DictWriter(f,fieldnames=list(lm.rows[0]))
+            if first:w.writeheader()
+            w.writerows(lm.rows)
+        print('completed',s['id'],len(lm.rows),flush=True);lm.rows.clear()
+    summary={'completed':True,'elapsed_seconds':time.monotonic()-started,'peak_rss_kib':resource.getrusage(resource.RUSAGE_SELF).ru_maxrss,'config_activation':lm.config.get('hidden_act'),'layer_count':len(lm.config['layer_types']),'samples':samples,'rows':sum(1 for _ in (a.out/'metrics.csv').open())-1,'block_widths':[64,128],'threshold_reference':'RMS within each sample/module/block, thresholds 0.01 and 0.1; exact zero measured separately','training':False,'weights_deleted':False,'limits':['4 short texts, 32 tokens each, descriptive only','Block is contiguous output channels, not Transformer block','Small output does not prove removable computation or parameter interchangeability','All samples previously used; no generalization claim','Relative threshold differs between blocks; width=0 is whole module reference']}
+    (a.out/'summary.json').write_text(json.dumps(summary,indent=2))
+if __name__=='__main__':main()

results/block-activation/REPORT.md

@@ -0,0 +1,25 @@
+# 实测激活率：SiLU 门控模型
+
+覆盖24层、4条中英文文本，每条32 token；所有投影输出及 MLP 门控中间量，按64/128输出通道分块。
+
+| 输出 | 精确零比例均值 | 0.01 RMS以下均值 | 0.1 RMS以下均值 | 32 token内曾活跃通道比例均值 | 90%能量所需通道比例均值 |
+|---|---:|---:|---:|---:|---:|
+| mlp.gate_proj.output | 0.0000% | 0.7811% | 7.8005% | 100.0000% | 63.3063% |
+| mlp.silu_gate | 0.0000% | 0.5031% | 5.1789% | 100.0000% | 70.1228% |
+| mlp.gated_product | 0.0000% | 3.1636% | 21.0460% | 100.0000% | 54.8421% |
+| self_attn.q_proj.output | 0.0000% | 1.2760% | 12.4831% | 100.0000% | 44.0277% |
+| self_attn.k_proj.output | 0.0000% | 1.2245% | 12.0834% | 100.0000% | 53.4587% |
+| self_attn.v_proj.output | 0.0000% | 1.3741% | 13.4814% | 100.0000% | 59.9447% |
+
+128通道块×文本观测共27600条，整个32 token窗口全为精确零的块观测数为0。
+
+表中是整模块逐文本统计再取均值；每个分块的独立统计见 metrics.csv。阈值是各自RMS的比例；阈值下激活率=1−近零比例。能量集中不等于可删除或可替换。
+
+仅测该模型、四条短文本；不推断所有语言/长上下文。SiLU不把负值整体截零，负数不等于静默。未执行删块/跳算的质量验证。权重和既有缓存全部保留。
+
+CPU测量耗时268.6秒，峰值RSS151.6 MiB。
+
+mlp.gated_product: 128通道块在0.1 RMS阈值上的激活率范围 10.47%–89.04%。
+self_attn.q_proj.output: 128通道块在0.1 RMS阈值上的激活率范围 75.49%–100.00%。
+self_attn.k_proj.output: 128通道块在0.1 RMS阈值上的激活率范围 81.20%–93.95%。
+self_attn.v_proj.output: 128通道块在0.1 RMS阈值上的激活率范围 76.66%–91.89%。

results/block-activation/aggregates.json

@@ -0,0 +1,200 @@
+{
+  "mlp.gate_proj.output": {
+    "observations": 96,
+    "zero_rate": {
+      "min": 0.0,
+      "mean": 0.0,
+      "max": 0.0
+    },
+    "negative_rate": {
+      "min": 0.6825663248697916,
+      "mean": 0.8063652250501846,
+      "max": 0.9117279052734375
+    },
+    "near_001_rms": {
+      "min": 0.005289713541666667,
+      "mean": 0.007811493343777125,
+      "max": 0.010442097981770834
+    },
+    "near_01_rms": {
+      "min": 0.053070068359375,
+      "mean": 0.07800536685519749,
+      "max": 0.10291544596354167
+    },
+    "ever_active_001": {
+      "min": 1.0,
+      "mean": 1.0,
+      "max": 1.0
+    },
+    "energy90_channel_fraction": {
+      "min": 0.5314127604166666,
+      "mean": 0.6330634223090278,
+      "max": 0.78955078125
+    }
+  },
+  "mlp.silu_gate": {
+    "observations": 96,
+    "zero_rate": {
+      "min": 0.0,
+      "mean": 0.0,
+      "max": 0.0
+    },
+    "negative_rate": {
+      "min": 0.6825663248697916,
+      "mean": 0.8063652250501846,
+      "max": 0.9117279052734375
+    },
+    "near_001_rms": {
+      "min": 0.0018768310546875,
+      "mean": 0.0050310028923882385,
+      "max": 0.0082244873046875
+    },
+    "near_01_rms": {
+      "min": 0.020375569661458332,
+      "mean": 0.05178917778862848,
+      "max": 0.09897359212239583
+    },
+    "ever_active_001": {
+      "min": 1.0,
+      "mean": 1.0,
+      "max": 1.0
+    },
+    "energy90_channel_fraction": {
+      "min": 0.04736328125,
+      "mean": 0.7012278238932291,
+      "max": 0.8186848958333334
+    }
+  },
+  "mlp.gated_product": {
+    "observations": 96,
+    "zero_rate": {
+      "min": 0.0,
+      "mean": 0.0,
+      "max": 0.0
+    },
+    "negative_rate": {
+      "min": 0.4942118326822917,
+      "mean": 0.5002292527092828,
+      "max": 0.5078125
+    },
+    "near_001_rms": {
+      "min": 0.020960489908854168,
+      "mean": 0.03163560231526693,
+      "max": 0.07338460286458333
+    },
+    "near_01_rms": {
+      "min": 0.16080220540364584,
+      "mean": 0.21046002705891928,
+      "max": 0.4075215657552083
+    },
+    "ever_active_001": {
+      "min": 1.0,
+      "mean": 1.0,
+      "max": 1.0
+    },
+    "energy90_channel_fraction": {
+      "min": 0.06380208333333333,
+      "mean": 0.5484212239583334,
+      "max": 0.7263997395833334
+    }
+  },
+  "self_attn.q_proj.output": {
+    "observations": 24,
+    "zero_rate": {
+      "min": 0.0,
+      "mean": 0.0,
+      "max": 0.0
+    },
+    "negative_rate": {
+      "min": 0.6834945678710938,
+      "mean": 0.7360480626424154,
+      "max": 0.7523574829101562
+    },
+    "near_001_rms": {
+      "min": 0.01090240478515625,
+      "mean": 0.012759526570638021,
+      "max": 0.01556396484375
+    },
+    "near_01_rms": {
+      "min": 0.106048583984375,
+      "mean": 0.1248311996459961,
+      "max": 0.148590087890625
+    },
+    "ever_active_001": {
+      "min": 1.0,
+      "mean": 1.0,
+      "max": 1.0
+    },
+    "energy90_channel_fraction": {
+      "min": 0.413330078125,
+      "mean": 0.440277099609375,
+      "max": 0.4931640625
+    }
+  },
+  "self_attn.k_proj.output": {
+    "observations": 24,
+    "zero_rate": {
+      "min": 0.0,
+      "mean": 0.0,
+      "max": 0.0
+    },
+    "negative_rate": {
+      "min": 0.47601318359375,
+      "mean": 0.49783579508463544,
+      "max": 0.52325439453125
+    },
+    "near_001_rms": {
+      "min": 0.00909423828125,
+      "mean": 0.01224517822265625,
+      "max": 0.01629638671875
+    },
+    "near_01_rms": {
+      "min": 0.095458984375,
+      "mean": 0.1208343505859375,
+      "max": 0.15106201171875
+    },
+    "ever_active_001": {
+      "min": 1.0,
+      "mean": 1.0,
+      "max": 1.0
+    },
+    "energy90_channel_fraction": {
+      "min": 0.41015625,
+      "mean": 0.5345865885416666,
+      "max": 0.65234375
+    }
+  },
+  "self_attn.v_proj.output": {
+    "observations": 24,
+    "zero_rate": {
+      "min": 0.0,
+      "mean": 0.0,
+      "max": 0.0
+    },
+    "negative_rate": {
+      "min": 0.49267578125,
+      "mean": 0.5059967041015625,
+      "max": 0.51666259765625
+    },
+    "near_001_rms": {
+      "min": 0.0084228515625,
+      "mean": 0.01374053955078125,
+      "max": 0.02447509765625
+    },
+    "near_01_rms": {
+      "min": 0.087890625,
+      "mean": 0.1348139444986979,
+      "max": 0.22296142578125
+    },
+    "ever_active_001": {
+      "min": 1.0,
+      "mean": 1.0,
+      "max": 1.0
+    },
+    "energy90_channel_fraction": {
+      "min": 0.291015625,
+      "mean": 0.5994466145833334,
+      "max": 0.732421875
+    }
+  }
+}

results/block-activation/audit.json

@@ -0,0 +1,7 @@
+{
+  "passed": true,
+  "rows": 83616,
+  "layers": 24,
+  "samples": 4,
+  "all_zero_128_blocks": 0
+}

results/block-activation/checksums.json

@@ -0,0 +1,18 @@
+{
+  "REPORT.md": "f7f05bdbf9c9d096fe685ebb49ee15ff9866a99071302f9d7710b99b1a2e7db4",
+  "aggregates.json": "57dfef79b14e73dafd39840661abacd8b44447884e640bb40b86f90ec441ee4d",
+  "audit.json": "2a67fde20f6765d0dbfa783ff31c548f8135af5d74b6fc0e27a3e71c9cbdffb1",
+  "figures/block_activity.pdf": "354cbc89dfc6da6d6611c6effc9dfefc6762e3da1d52bfba89cc11be8c8a9247",
+  "figures/block_activity.png": "ddcfa1f0de30f4f8bdece16b3451ca1b5bf419fac5a5bb9bb944301983efaeec",
+  "figures/block_activity.svg": "da85920dd0fc32ba439b650c8b695757d42eb72768b4017a78109cb8a2f24216",
+  "figures/mlp_activation.pdf": "6eef102d1ecc7c21206da862f79d4edaed77a695f7c46d53550c3dc6f9087ee5",
+  "figures/mlp_activation.png": "0e1a01ea5b21274eaa757c1c9cd493524003220eadcf7aa378e0eb39a37a7c40",
+  "figures/mlp_activation.svg": "d7b0a67d2d1d501e6fa70aa4a38010bead93a450423e5cd6b3b1284794353c8f",
+  "final-tests.txt": "7b52cfeffca483471297e30aa9658bc2c5c6585b682f5ee5fbbcc2e1b2b8abe4",
+  "first-attempt.txt": "7a1f686db3b7412b681d516bdd83bb8ed7ab2a758a44a2d84c4c7968017d9c91",
+  "full-tests.txt": "55fe658a01d15ed380f2b5cd01058f2374997ffde945025262e92fd966f0dc3b",
+  "metrics.csv": "8f6205e7f446f83567485275f57f63ce64233eee711a4be89c30b37182772e4d",
+  "run.txt": "9fe01020a3ac92db51a17042eef6af73f854dd2e81220d7a04cd5a843c6ff9ba",
+  "source-hashes.json": "34d1c164e787ec6ec5b77d4bc056da2e72365e1cd4b5089741f2b979a3c8beb3",
+  "summary.json": "1e7c3bace11673d10cb927d689348bf00790a9b074ccafed6ce44d898410ef98"
+}

results/block-activation/final-tests.txt

@@ -0,0 +1,7 @@
+test_known_activity (tests.test_activation.ActivationTests.test_known_activity) ... ok
+test_zero_and_scale (tests.test_activation.ActivationTests.test_zero_and_scale) ... ok
+
+----------------------------------------------------------------------
+Ran 2 tests in 0.002s
+
+OK

results/block-activation/first-attempt.txt

@@ -0,0 +1,24 @@
+completed fresh-zh-1 20904
+completed fresh-zh-2 41808
+completed fresh-en-1 62712
+Traceback (most recent call last):
+  File "/opt/exp-model-parameter-similarity/activation_probe.py", line 44, in <module>
+    if __name__=='__main__':main()
+                            ~~~~^^
+  File "/opt/exp-model-parameter-similarity/activation_probe.py", line 39, in main
+    lm.forward(np.array([ids],dtype=np.int64)); print('completed',s['id'],len(lm.rows),flush=True)
+    ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/opt/exp-model-parameter-similarity/functional/model.py", line 220, in forward
+    h = h + self.mlp(n,donor)
+            ~~~~~~~~^^^^^^^^^
+  File "/opt/exp-model-parameter-similarity/activation_probe.py", line 30, in mlp
+    return self.linear(g*u,p+'down_proj')
+           ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
+  File "/opt/exp-model-parameter-similarity/activation_probe.py", line 22, in linear
+    y=super().linear(x,name,rows); self.record(y,name+'.output')
+  File "/opt/exp-model-parameter-similarity/functional/model.py", line 130, in linear
+    self.guard()
+    ~~~~~~~~~~^^
+  File "/opt/exp-model-parameter-similarity/functional/model.py", line 111, in guard
+    raise MemoryError(f'Sustained host swap pressure plus constrained memory: available={available} KiB, process_swap={own_swap_kib} KiB')
+MemoryError: Sustained host swap pressure plus constrained memory: available=989040 KiB, process_swap=132360 KiB

results/block-activation/full-tests.txt

@@ -0,0 +1,33 @@
+test_known_activity (test_activation.ActivationTests.test_known_activity) ... ok
+test_zero_and_scale (test_activation.ActivationTests.test_zero_and_scale) ... ok
+test_assignment_against_exhaustive_and_inverse (test_crossqkv.CrossQKVTests.test_assignment_against_exhaustive_and_inverse) ... ok
+test_cached_mode_rejects_future_state_reuse (test_crossqkv.CrossQKVTests.test_cached_mode_rejects_future_state_reuse) ... ok
+test_controls_cover_multiple_projection_layers (test_crossqkv.CrossQKVTests.test_controls_cover_multiple_projection_layers) ... ok
+test_head_matrix_metrics_and_gram_are_diagnostic (test_crossqkv.CrossQKVTests.test_head_matrix_metrics_and_gram_are_diagnostic) ... ok
+test_probability_identity_and_nontrivial_queries (test_crossqkv.CrossQKVTests.test_probability_identity_and_nontrivial_queries) ... ok
+test_profile_matches_canonical_and_head_sum (test_crossqkv.CrossQKVTests.test_profile_matches_canonical_and_head_sum) ... ok
+test_recipient_value_gate_output_and_gqa_group_stay_fixed (test_crossqkv.CrossQKVTests.test_recipient_value_gate_output_and_gqa_group_stay_fixed) ... ok
+test_topk_tail_rank_can_change_with_tiny_probability_difference (test_crossqkv.CrossQKVTests.test_topk_tail_rank_can_change_with_tiny_probability_difference) ... ok
+test_causal_convolution_and_norm_boundary (test_functional.FunctionalTests.test_causal_convolution_and_norm_boundary) ... ok
+test_causality_and_identical_function_metrics (test_functional.FunctionalTests.test_causality_and_identical_function_metrics) ... ok
+test_neuron_function_metrics_and_targeted_replacement (test_functional.FunctionalTests.test_neuron_function_metrics_and_targeted_replacement) ... ok
+test_nonrotary_gain_identity_and_rotary_negative_control (test_functional.FunctionalTests.test_nonrotary_gain_identity_and_rotary_negative_control) ... ok
+test_streamed_projection_embedding_and_output_head (test_functional.FunctionalTests.test_streamed_projection_embedding_and_output_head) ... ok
+test_uniform_rmsnorm_function_bound_is_tight (test_functional.FunctionalTests.test_uniform_rmsnorm_function_bound_is_tight) ... ok
+test_3d_reader_and_exact_block (test_pilot.PilotTests.test_3d_reader_and_exact_block) ... ok
+test_bfloat16 (test_pilot.PilotTests.test_bfloat16) ... ok
+test_directional_replacement_error_and_scaling (test_pilot.PilotTests.test_directional_replacement_error_and_scaling) ... ok
+test_end_to_end_synthetic_groups_and_savings (test_pilot.PilotTests.test_end_to_end_synthetic_groups_and_savings) ... Inventory: 4 tensors, 33792 bytes
+tensor: sampling 4 objects
+block: sampling 22 objects
+block: verified 100/109
+{"model": "synthetic", "revision": "test", "tensor_count": 4, "parameter_count": 8448, "model_weight_bytes": 33792, "ndim_counts": {"2": 2, "3": 2}, "elapsed_seconds": 0.3087193910032511, "analysis_budget_seconds": 60, "peak_rss_kib": 57700, "python": "3.14.4", "numpy": "2.5.4", "seed": 20261109, "limitations": ["No inference or functional equivalence measured", "No exhaustive pair search", "Block search uses per-role/shape/scale reservoirs of 256", "Different block scales overlap; savings must not be added", "Block savings count only verified disjoint sampled blocks, no extrapolation", "Spectral probes use aligned bounded subvolumes, not whole tensors", "Shared encoding is theoretical, not a runnable compressed checkpoint", "No cross-role grouping, neuron permutation alignment, or training"], "completed": true}
+ok
+test_no_transitive_chaining (test_pilot.PilotTests.test_no_transitive_chaining) ... ok
+test_ranged_download_preserves_prefix (test_pilot.PilotTests.test_ranged_download_preserves_prefix) ... Weight download 36/36 bytes (100.0%)
+ok
+
+----------------------------------------------------------------------
+Ran 22 tests in 33.253s
+
+OK

results/block-activation/run.txt

@@ -0,0 +1,4 @@
+completed fresh-zh-1 20904
+completed fresh-zh-2 20904
+completed fresh-en-1 20904
+completed fresh-en-2 20904

results/block-activation/source-hashes.json

@@ -0,0 +1,8 @@
+{
+  "activation/report.py": "e67bfa4af69e43cdceb803b77fe6a6dcc151163e72b3ff2bb7e6b69e7c6fa1fc",
+  "activation/audit.py": "7dc8af7c31de379a50bdcbd3c41243eb190e6a50e8ec00a30182e82a34e73627",
+  "activation/run.py": "cba62a2d8b6344625f233309b166b091028154890d93e48278eb338cbcd19b18",
+  "activation/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
+  "tests/test_activation.py": "dee1756b63cd03c2051cdc92d678d2e27aff639fbfd3d3cb3169140760da4c68",
+  "functional/model.py": "48a2a97d0445dc6e08ac7f27a935637a7553417d0982eb94122d6cb02bbb499c"
+}

results/block-activation/summary.json

@@ -0,0 +1,180 @@
+{
+  "completed": true,
+  "elapsed_seconds": 268.55615069903433,
+  "peak_rss_kib": 155288,
+  "config_activation": "silu",
+  "layer_count": 24,
+  "samples": [
+    {
+      "id": "fresh-zh-1",
+      "language": "zh",
+      "text": "\u690d\u7269\u5b66\u5bb6\u5728\u4e0d\u540c\u6d77\u62d4\u91c7\u96c6\u4e86\u53f6\u7247\u6837\u672c\uff0c\u5e76\u4e3a\u6bcf\u4e00\u4efd\u6807\u672c\u8bb0\u5f55\u6e29\u5ea6\u3001\u5149\u7167\u548c\u571f\u58e4\u6e7f\u5ea6\u3002\u56de\u5230\u5b9e\u9a8c\u5ba4\u540e\uff0c\u5979\u6309\u5730\u70b9\u6574\u7406\u6807\u7b7e\uff0c\u5148\u6838\u5bf9\u7f3a\u5931\u8bb0\u5f55\uff0c\u518d\u6bd4\u8f83\u4e0d\u540c\u73af\u5883\u4e2d\u7684\u53f6\u7247\u5f62\u72b6\u3002",
+      "token_ids": [
+        99951,
+        101071,
+        115802,
+        111685,
+        102099,
+        95759,
+        111583,
+        109928,
+        3709,
+        115026,
+        134176,
+        116599,
+        98151,
+        99472,
+        5205,
+        114180,
+        95793,
+        104074,
+        102454,
+        1710,
+        103744,
+        103911,
+        95946,
+        3709,
+        96343,
+        96656,
+        100560,
+        99946,
+        101659,
+        3709,
+        96218,
+        113089
+      ]
+    },
+    {
+      "id": "fresh-zh-2",
+      "language": "zh",
+      "text": "\u5982\u679c\u4e00\u4e2a\u76d2\u5b50\u91cc\u6709\u5341\u4e8c\u679a\u84dd\u8272\u68cb\u5b50\u548c\u516b\u679a\u767d\u8272\u68cb\u5b50\uff0c\u968f\u673a\u53d6\u51fa\u4e00\u679a\u540e\u4e0d\u653e\u56de\uff0c\u7b2c\u4e8c\u6b21\u53d6\u5230\u84dd\u8272\u7684\u6982\u7387\u9700\u8981\u6839\u636e\u7b2c\u4e00\u6b21\u7684\u7ed3\u679c\u5206\u522b\u8ba1\u7b97\u3002\u8bf7\u5148\u5217\u51fa\u5404\u4e2a\u4e8b\u4ef6\uff0c\u518d\u68c0\u67e5\u5b83\u4eec\u7684\u6982\u7387\u4e4b\u548c\u3002",
+      "token_ids": [
+        127356,
+        98996,
+        98970,
+        95762,
+        98175,
+        101184,
+        104145,
+        99357,
+        111324,
+        96702,
+        101184,
+        103828,
+        129730,
+        3709,
+        102038,
+        96284,
+        98938,
+        101184,
+        95946,
+        122271,
+        96120,
+        3709,
+        109387,
+        96284,
+        95839,
+        116981,
+        104514,
+        122501,
+        103720,
+        109146,
+        99446,
+        97792
+      ]
+    },
+    {
+      "id": "fresh-en-1",
+      "language": "en",
+      "text": "A technician recorded the vibration of a small motor before and after replacing its bearings. She used the same mounting position and sampling interval for both measurements. The final notebook included the raw traces, instrument settings, and a brief explanation of unusual observations.",
+      "token_ids": [
+        32,
+        51715,
+        12076,
+        279,
+        46219,
+        314,
+        264,
+        2526,
+        8604,
+        1518,
+        321,
+        1238,
+        24101,
+        1141,
+        33309,
+        13,
+        2838,
+        1429,
+        279,
+        1788,
+        32525,
+        2234,
+        321,
+        24102,
+        9581,
+        364,
+        2107,
+        20934,
+        13,
+        561,
+        1534,
+        35935
+      ]
+    },
+    {
+      "id": "fresh-en-2",
+      "language": "en",
+      "text": "The function walks through a list of integers and stores each running total in a new list. An empty input should return an empty result. Before changing the implementation, compare a short example by hand and make sure negative numbers preserve the intended behavior.",
+      "token_ids": [
+        760,
+        709,
+        21780,
+        1472,
+        264,
+        1103,
+        314,
+        24959,
+        321,
+        10220,
+        1754,
+        4162,
+        2702,
+        303,
+        264,
+        491,
+        1103,
+        13,
+        1473,
+        4147,
+        1879,
+        1220,
+        460,
+        449,
+        4147,
+        1067,
+        13,
+        12851,
+        9722,
+        279,
+        7873,
+        11
+      ]
+    }
+  ],
+  "rows": 83616,
+  "block_widths": [
+    64,
+    128
+  ],
+  "threshold_reference": "RMS within each sample/module/block, thresholds 0.01 and 0.1; exact zero measured separately",
+  "training": false,
+  "weights_deleted": false,
+  "limits": [
+    "4 short texts, 32 tokens each, descriptive only",
+    "Block is contiguous output channels, not Transformer block",
+    "Small output does not prove removable computation or parameter interchangeability",
+    "All samples previously used; no generalization claim",
+    "Relative threshold differs between blocks; width=0 is whole module reference"
+  ]
+}

tests/test_activation.py

@@ -0,0 +1,15 @@
+import unittest
+import numpy as np
+from activation.run import metrics
+class ActivationTests(unittest.TestCase):
+    def test_known_activity(self):
+        m=metrics(np.array([[0.,1.,-1.,0.],[0.,0.,0.,0.]]))
+        self.assertEqual(m['zero_rate'],.75)
+        self.assertEqual(m['ever_active_001'],.5)
+        self.assertEqual(m['negative_rate'],.125)
+        self.assertEqual(m['channels_for_90pct_energy'],2)
+    def test_zero_and_scale(self):
+        m=metrics(np.zeros((3,4)));self.assertEqual(m['zero_rate'],1);self.assertEqual(m['ever_active_001'],0);self.assertEqual(m['channels_for_90pct_energy'],0)
+        x=np.array([[1.,.001,0.,-2.]])
+        for key in ['near_001_rms','near_01_rms','energy90_channel_fraction']:
+            self.assertEqual(metrics(x)[key],metrics(100*x)[key])