Compare commits
21 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 2f1113323e | |||
| 35090663e2 | |||
| e1514f4952 | |||
| 9e0e659287 | |||
| 52e6d13fe5 | |||
| d6d985487c | |||
| 098ea8650c | |||
| cd260a2483 | |||
| 419c7cb195 | |||
| 0934079ed9 | |||
| 9efbf0b3b1 | |||
| 5031aa604e | |||
| 6d1acbc26a | |||
| 573cbedd98 | |||
| d4012592b6 | |||
| c718e8bd9a | |||
| 24b3c97fa5 | |||
| 70b8be25b4 | |||
| 727b05eb3f | |||
| 6121016a26 | |||
| e98c5d0936 |
@@ -0,0 +1,9 @@
|
||||
# Scientific artifacts must retain the bytes recorded in checksums.json.
|
||||
results/** -text
|
||||
# Generated CSV uses RFC 4180 line endings; Matplotlib SVG paths contain spaces.
|
||||
results/**/*.csv -whitespace
|
||||
results/**/*.svg -whitespace
|
||||
results/**/*.log -whitespace
|
||||
|
||||
# Preserve captured scientific stdout byte-for-byte, including blank final lines.
|
||||
results/head-dominance/*-output.txt -whitespace
|
||||
+10
@@ -0,0 +1,10 @@
|
||||
__pycache__/
|
||||
.venv/
|
||||
weights/
|
||||
*.partial
|
||||
*.log
|
||||
|
||||
# Large reproducible functional activations remain on the experiment server.
|
||||
results/cpu-qwen35-2b/baseline-activations.npz
|
||||
results/cpu-qwen35-2b/neuron-reuse-activations.npz
|
||||
results/cpu-qwen35-2b/norm-repetition-input-profiles.npz
|
||||
@@ -0,0 +1,61 @@
|
||||
# Model parameter similarity experiments
|
||||
|
||||
**[研究图册与固定在线副本](docs/FIGURE_GALLERY.md)**:连续分布、联合消融及原始PDF/SVG。
|
||||
|
||||
[两个实际2B检查点的参数球壳结论](docs/MODEL_SHELL_FINDINGS.md) · [完整数值报告](results/model-shell/REPORT.md) · [固定分组和均匀参照协议](model_shell/PROTOCOL.md)。
|
||||
|
||||
Research on repeated learned functions in **Qwen3.5-2B**: a completed static weight baseline
|
||||
and a training-free CPU functional pilot. The CPU study found similar existing RMSNorm
|
||||
functions at layer indices 12/13, with a uniform analytic error bound and heldout substitution checks.
|
||||
Cost is not a success condition. The completed static pilot was inference-free and its
|
||||
weights were deleted after archival; the functional pilot executes streaming FP32 text forwards.
|
||||
The current CPU functional checkpoint is retained until the user explicitly requests deletion.
|
||||
No model weights are committed here.
|
||||
|
||||
The latest cross-layer QKV study covers all 15 full-attention layer pairs. Its primary
|
||||
protocol had no passing candidate; an exploratory H23-conditioned self-token retrieval
|
||||
pattern is reported separately with its fresh-test failures.
|
||||
|
||||
- [Cross-layer QKV findings and evidence limits](docs/CROSS_LAYER_QKV_FINDINGS.md)
|
||||
- [Cross-layer QKV fixed protocol](docs/CROSS_LAYER_QKV_PROTOCOL.md)
|
||||
- [CPU learned-function repetition findings](docs/CPU_FUNCTIONAL_FINDINGS.md)
|
||||
- [Current learned-function repetition criterion](docs/LEARNED_FUNCTION_REPETITION.md)
|
||||
- [CPU-first feasibility and validation](docs/CPU_FIRST_VALIDATION.md)
|
||||
- [Training-free functional reuse plan (proposed; not executed)](docs/TRAINING_FREE_FUNCTIONAL_PLAN.md)
|
||||
- [Primary-source literature and applicability](docs/FUNCTIONAL_REUSE_LITERATURE.md)
|
||||
- [Procedure and limits](docs/SOP.md)
|
||||
- [2B pilot findings and limitations](docs/PILOT_FINDINGS.md)
|
||||
- [Raw result report, data and figures](results/qwen35-2b/REPORT.md)
|
||||
- `pilot/download.py`: revision-pinned official checkpoint download and verification.
|
||||
- `pilot/tensors.py`: memory-mapped safetensors/BF16 reader and metrics.
|
||||
- `pilot/analyze.py`: bounded tensor/block search, exact verification, representative groups.
|
||||
- `pilot/plot.py`: scientific PNG/PDF/SVG figures and report from saved data.
|
||||
- `pilot/validate.py`: result/group validation, checksums and guarded weight cleanup.
|
||||
- `pilot/audit.py`: finite-metric checks, test evidence and archive provenance.
|
||||
- `pilot/run.py`: one-model supervisor with bounded download wait and analysis timeouts.
|
||||
- `tests/test_pilot.py`: synthetic numerical and 3D correctness tests.
|
||||
|
||||
The static baseline reports **verified net byte savings versus direct relative weight error**.
|
||||
The functional study separately reports same-input error, an all-input RMSNorm function bound
|
||||
and tiny heldout NLL changes; these are not broad task-accuracy guarantees. Different static
|
||||
block scales are alternative partitions and their savings must not be summed.
|
||||
|
||||
Dependencies are pinned in `requirements.txt`; install only prebuilt wheels on the remote host.
|
||||
Full resolved versions and run provenance accompany each result. See the SOP for commands.
|
||||
|
||||
## CPU block activation measurements
|
||||
|
||||
See [measurement definitions](activation/README.md) and [observed activation report](results/block-activation/REPORT.md) for all 24 layers, contiguous output-channel blocks, exact/near-zero rates and output energy concentration. This is an observational SiLU-gated study, not a pruning or reuse quality claim.
|
||||
|
||||
## Adjacent-layer head influence concentration
|
||||
|
||||
The CPU study measures all 23 adjacent transitions, observation-wise effective
|
||||
head counts and Q/K/V influence shares, with equal-size weak/strong/random joint
|
||||
removal controls. See [definitions and reproduction](dominance/README.md),
|
||||
[fixed protocol](dominance/PROTOCOL.md), and [measured report](results/head-dominance/REPORT.md).
|
||||
These are descriptive intervention effects, not additive contribution probabilities
|
||||
or validated final-LM pruning gains.
|
||||
|
||||
## Existing GSM8K task-finetuning comparison
|
||||
|
||||
[Paired study and reproduction](finetune_compare/README.md) · [Fixed protocol](finetune_compare/PROTOCOL.md) · [Results](results/finetune-comparison/REPORT.md). This compares an existing pure-LoRA GRPO checkpoint to exactly the same original release; no training is performed here.
|
||||
@@ -0,0 +1,45 @@
|
||||
# Observed block activation, CPU only
|
||||
|
||||
Measures the retained Qwen3.5-2B checkpoint without training or substitutions.
|
||||
This model uses SiLU-gated MLPs and sigmoid attention gates, not ReLU.
|
||||
Leaky ReLU also does not generally create exact zero on negative inputs.
|
||||
|
||||
The probe records every linear projection in all 24 layers, plus MLP SiLU
|
||||
and gate-times-up intermediates and full-attention sigmoid gates. It does not
|
||||
record every internal recurrent state, convolution output or attention probability.
|
||||
A block means contiguous **output channels** (64 or 128), hence contiguous
|
||||
rows of a projection matrix; it is not a Transformer layer or arbitrary 2D tile.
|
||||
MLP gated-product channels correspond to columns consumed by down_proj.
|
||||
It is incorrect to call an individual weight active/inactive based only on its value.
|
||||
|
||||
For every sample, module and block, metrics.csv reports:
|
||||
- exact zero and negative element fractions;
|
||||
- near-zero fractions at |x| <= 0.01 RMS and <= 0.1 RMS;
|
||||
- fraction of channels ever above 0.01 RMS over the 32 tokens;
|
||||
- minimum fraction of channels carrying 90% squared-output energy.
|
||||
|
||||
RMS is computed within each observed sample/module/block; width=0 records the
|
||||
whole module. Thus thresholds are explicitly relative and are not shared absolute
|
||||
cutoffs across layers or blocks. Activation rate at a threshold is 1-near-zero rate.
|
||||
A small value or an energy-concentrated output does not prove that a block can
|
||||
be skipped, removed or shared without affecting LM quality. Exact zero is measured
|
||||
in this FP32 streaming implementation of BF16 source weights; other execution dtypes
|
||||
can have different underflow rates.
|
||||
|
||||
Four prior authored Chinese/English texts, 32 tokens each, descriptive measurements
|
||||
only. No training, fresh-generalization claim, GPU, weight download or cleanup.
|
||||
The existing streaming implementation was checked against the official implementation
|
||||
in the earlier CPU study; this experiment changes only observation, not model equations.
|
||||
|
||||
Remote commands (authorized server 8.211.154.134):
|
||||
```
|
||||
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -B -m activation.run --source runs/cpu-qwen35-2b --out runs/NEW_ACTIVATION_DIR
|
||||
OPENBLAS_NUM_THREADS=1 venv/bin/python -B -m unittest tests.test_activation -v
|
||||
venv/bin/python -B activation/report.py runs/NEW_ACTIVATION_DIR
|
||||
```
|
||||
|
||||
run.py measures; report.py plots saved rows and derives aggregates;
|
||||
__init__.py marks the package; tests/test_activation.py checks exact sparsity,
|
||||
ever-active behavior, all-zero inputs and scale invariance. Raw activations are
|
||||
not saved or needed for these descriptive statistics. Existing caches and weights
|
||||
are retained untouched. Full sample token IDs and CPU time/RSS appear in summary.json.
|
||||
@@ -0,0 +1,28 @@
|
||||
"""Audit recorded scope, metrics and reproducibility files; no model execution."""
|
||||
import csv,json,hashlib
|
||||
from pathlib import Path
|
||||
import sys
|
||||
p=Path(sys.argv[1]);r=list(csv.DictReader((p/'metrics.csv').open()));s=json.loads((p/'summary.json').read_text())
|
||||
assert s['completed'] and len(r)==s['rows']==4*20904
|
||||
assert len({x['sample'] for x in r})==4
|
||||
assert {int(x['module'].split('.')[1]) for x in r}==set(range(24))
|
||||
for x in r:
|
||||
for k in ['zero_rate','negative_rate','near_001_rms','near_01_rms','ever_active_001','energy90_channel_fraction']:assert 0<=float(x[k])<=1,(x,k)
|
||||
assert float(x['zero_rate'])<=float(x['near_001_rms'])<=float(x['near_01_rms'])
|
||||
assert int(x['channels'])==int(x['stop'])-int(x['start'])
|
||||
a=json.loads((p/'aggregates.json').read_text());lines=['# 实测激活率:SiLU 门控模型','', '覆盖24层、4条中英文文本,每条32 token;所有投影输出及 MLP 门控中间量,按64/128输出通道分块。','', '| 输出 | 精确零比例均值 | 0.01 RMS以下均值 | 0.1 RMS以下均值 | 32 token内曾活跃通道比例均值 | 90%能量所需通道比例均值 |','|---|---:|---:|---:|---:|---:|']
|
||||
for k,v in a.items():
|
||||
cols=[v[m]['mean']*100 for m in ['zero_rate','near_001_rms','near_01_rms','ever_active_001','energy90_channel_fraction']]
|
||||
lines.append('| '+k+' | '+' | '.join(f'{z:.4f}%' for z in cols)+' |')
|
||||
blocks=[x for x in r if x['width']=='128']; dead=sum(float(x['zero_rate'])==1 for x in blocks)
|
||||
lines+=['',f'128通道块×文本观测共{len(blocks)}条,整个32 token窗口全为精确零的块观测数为{dead}。', '', '表中是整模块逐文本统计再取均值;每个分块的独立统计见 metrics.csv。阈值是各自RMS的比例;阈值下激活率=1−近零比例。能量集中不等于可删除或可替换。', '', '仅测该模型、四条短文本;不推断所有语言/长上下文。SiLU不把负值整体截零,负数不等于静默。未执行删块/跳算的质量验证。权重和既有缓存全部保留。', '',f"CPU测量耗时{s['elapsed_seconds']:.1f}秒,峰值RSS{s['peak_rss_kib']/1024:.1f} MiB。"]
|
||||
(p/'REPORT.md').write_text('\n'.join(lines)+'\n')
|
||||
lines_extra=[]
|
||||
for suffix in ['mlp.gated_product','self_attn.q_proj.output','self_attn.k_proj.output','self_attn.v_proj.output']:
|
||||
rr=[x for x in blocks if x['module'].endswith(suffix)]
|
||||
act=sorted(1-float(x['near_01_rms']) for x in rr)
|
||||
lines_extra.append(f'{suffix}: 128通道块在0.1 RMS阈值上的激活率范围 {100*act[0]:.2f}%–{100*act[-1]:.2f}%。')
|
||||
with (p/'REPORT.md').open('a') as f:f.write('\n'+'\n'.join(lines_extra)+'\n')
|
||||
(p/'audit.json').write_text(json.dumps({'passed':True,'rows':len(r),'layers':24,'samples':4,'all_zero_128_blocks':dead},indent=2))
|
||||
files=[x for x in p.rglob('*') if x.is_file() and x.name!='checksums.json'];(p/'checksums.json').write_text(json.dumps({str(x.relative_to(p)):hashlib.sha256(x.read_bytes()).hexdigest() for x in sorted(files)},indent=2))
|
||||
print((p/'REPORT.md').read_text())
|
||||
@@ -0,0 +1,38 @@
|
||||
"""Render scientific figures from recorded CPU measurements."""
|
||||
import csv,json
|
||||
from pathlib import Path
|
||||
import numpy as np
|
||||
import matplotlib
|
||||
matplotlib.use('Agg')
|
||||
import matplotlib.pyplot as plt
|
||||
import sys
|
||||
root=Path(sys.argv[1]); rows=list(csv.DictReader((root/'metrics.csv').open()))
|
||||
full=[r for r in rows if r['width']=='0']
|
||||
fig,axs=plt.subplots(1,3,figsize=(15,4.5),layout='constrained')
|
||||
for suffix,label in [('gate_proj.output','gate preactivation'),('silu_gate','SiLU gate'),('gated_product','gated product')]:
|
||||
selected=[r for r in full if r['module'].endswith('mlp.'+suffix)]
|
||||
for ax,key in zip(axs,['zero_rate','near_001_rms','near_01_rms']):
|
||||
xs=[];ys=[]
|
||||
for layer in range(24):
|
||||
z=[float(r[key]) for r in selected if r['module'].startswith(f'layers.{layer}.')];xs.append(layer);ys.append(np.mean(z))
|
||||
ax.plot(xs,ys,label=label)
|
||||
for ax,title in zip(axs,['Exact zero fraction','Near zero: |x| <= 0.01 RMS','Near zero: |x| <= 0.1 RMS']):
|
||||
ax.set(title=title,xlabel='Layer index',ylabel='Fraction');ax.set_ylim(bottom=0);ax.grid(alpha=.2)
|
||||
axs[-1].legend();fig.suptitle('Observed MLP activation sparsity: four texts, 32 tokens each; no training')
|
||||
root.joinpath('figures').mkdir(exist_ok=True)
|
||||
for ext in ('png','pdf','svg'):fig.savefig(root/'figures'/('mlp_activation.'+ext),dpi=180)
|
||||
plt.close(fig)
|
||||
blocks=[r for r in rows if r['width']=='128' and r['module'].endswith('mlp.gated_product')]
|
||||
fig,axs=plt.subplots(1,2,figsize=(12,4.5),layout='constrained')
|
||||
for ax,key,title in zip(axs,['near_01_rms','energy90_channel_fraction'],['Near-zero fraction at 0.1 block RMS','Fraction of channels carrying 90% energy']):
|
||||
matrix=[]
|
||||
for layer in range(24):
|
||||
rr=[r for r in blocks if r['module'].startswith(f'layers.{layer}.')]; starts=sorted(set(int(r['start']) for r in rr));matrix.append([np.mean([float(r[key]) for r in rr if int(r['start'])==s]) for s in starts])
|
||||
im=ax.imshow(matrix,aspect='auto',vmin=0,vmax=1);ax.set(title=title,xlabel='Contiguous 128-channel block',ylabel='Layer index');fig.colorbar(im,ax=ax)
|
||||
fig.suptitle('MLP gated-product blocks: average over four observed texts')
|
||||
for ext in ('png','pdf','svg'):fig.savefig(root/'figures'/('block_activity.'+ext),dpi=180)
|
||||
agg={}
|
||||
for suffix in ['mlp.gate_proj.output','mlp.silu_gate','mlp.gated_product','self_attn.q_proj.output','self_attn.k_proj.output','self_attn.v_proj.output']:
|
||||
rr=[r for r in full if r['module'].endswith(suffix)]
|
||||
agg[suffix]={'observations':len(rr),**{key:{'min':min(float(r[key]) for r in rr),'mean':float(np.mean([float(r[key]) for r in rr])),'max':max(float(r[key]) for r in rr)} for key in ['zero_rate','negative_rate','near_001_rms','near_01_rms','ever_active_001','energy90_channel_fraction']}}
|
||||
(root/'aggregates.json').write_text(json.dumps(agg,indent=2))
|
||||
@@ -0,0 +1,48 @@
|
||||
"""CPU-only observed activation sparsity; no checkpoint changes or training."""
|
||||
import json,csv,time,resource,hashlib
|
||||
from pathlib import Path
|
||||
import numpy as np
|
||||
from tokenizers import Tokenizer
|
||||
from functional.model import StreamingLM,silu,sigmoid
|
||||
|
||||
|
||||
def metrics(x):
|
||||
x=np.asarray(x,dtype=np.float64).reshape(-1,x.shape[-1]); a=np.abs(x); energy=np.square(x).sum(0); total=energy.sum()
|
||||
rms=float(np.sqrt(np.mean(x*x))); cutoff=.01*rms
|
||||
ordered=np.sort(energy)[::-1]; n90=int(np.searchsorted(np.cumsum(ordered),.9*total)+1) if total else 0
|
||||
return dict(elements=x.size,channels=x.shape[-1],rms=rms,zero_rate=float(np.mean(x==0)),negative_rate=float(np.mean(x<0)),near_001_rms=float(np.mean(a<=cutoff)),near_01_rms=float(np.mean(a<=.1*rms)),ever_active_001=float(np.mean(np.any(a>cutoff,axis=0))),channels_for_90pct_energy=n90,energy90_channel_fraction=n90/x.shape[-1])
|
||||
|
||||
class Probe(StreamingLM):
|
||||
def record(self,x,name):
|
||||
for width in (0,64,128):
|
||||
for start in range(0,x.shape[-1],width or x.shape[-1]):
|
||||
stop=min(start+(width or x.shape[-1]),x.shape[-1]); z=x[...,start:stop]
|
||||
self.rows.append(dict(sample=self.sample,module=name,width=width,start=start,stop=stop,**metrics(z)))
|
||||
def linear(self,x,name,rows=256):
|
||||
y=super().linear(x,name,rows); self.record(y,name+'.output')
|
||||
if name.endswith('self_attn.q_proj'):
|
||||
dim=self.config['head_dim']; mixed=y.reshape(*y.shape[:-1],-1,2*dim)
|
||||
self.record(sigmoid(mixed[...,dim:]).reshape(*y.shape[:-1],-1),name+'.sigmoid_gate')
|
||||
return y
|
||||
def mlp(self,h,layer):
|
||||
p=f'layers.{layer}.mlp.'; pre=self.linear(h,p+'gate_proj'); g=silu(pre); u=self.linear(h,p+'up_proj')
|
||||
self.record(g,p+'silu_gate'); self.record(g*u,p+'gated_product')
|
||||
return self.linear(g*u,p+'down_proj')
|
||||
|
||||
def main():
|
||||
import argparse
|
||||
p=argparse.ArgumentParser();p.add_argument('--source',type=Path,required=True);p.add_argument('--out',type=Path,required=True);a=p.parse_args();a.out.mkdir(parents=True,exist_ok=False)
|
||||
started=time.monotonic();cfg=json.loads((a.source/'model_config.json').read_text()); lm=Probe(a.source/'weights',cfg,seconds=3600);lm.rows=[]
|
||||
samples=json.loads(Path('crossqkv/fresh_samples.json').read_text())['samples'];tok=Tokenizer.from_file(str(a.source/'tokenizer.json'))
|
||||
for s in samples:
|
||||
lm.sample=s['id']; ids=tok.encode(s['text']).ids[:32];s['token_ids']=ids
|
||||
lm.forward(np.array([ids],dtype=np.int64))
|
||||
first=not (a.out/'metrics.csv').exists()
|
||||
with (a.out/'metrics.csv').open('a') as f:
|
||||
w=csv.DictWriter(f,fieldnames=list(lm.rows[0]))
|
||||
if first:w.writeheader()
|
||||
w.writerows(lm.rows)
|
||||
print('completed',s['id'],len(lm.rows),flush=True);lm.rows.clear()
|
||||
summary={'completed':True,'elapsed_seconds':time.monotonic()-started,'peak_rss_kib':resource.getrusage(resource.RUSAGE_SELF).ru_maxrss,'config_activation':lm.config.get('hidden_act'),'layer_count':len(lm.config['layer_types']),'samples':samples,'rows':sum(1 for _ in (a.out/'metrics.csv').open())-1,'block_widths':[64,128],'threshold_reference':'RMS within each sample/module/block, thresholds 0.01 and 0.1; exact zero measured separately','training':False,'weights_deleted':False,'limits':['4 short texts, 32 tokens each, descriptive only','Block is contiguous output channels, not Transformer block','Small output does not prove removable computation or parameter interchangeability','All samples previously used; no generalization claim','Relative threshold differs between blocks; width=0 is whole module reference']}
|
||||
(a.out/'summary.json').write_text(json.dumps(summary,indent=2))
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,42 @@
|
||||
# Cross-layer QKV CPU study
|
||||
|
||||
This phase targets learned QKV functions across different layers, independent of cost.
|
||||
The six full-attention layers (indices 3/7/11/15/19/23) form 15 unordered pairs.
|
||||
No within-layer Q/K/V substitution, training, continuous R fitting, GPU, or new model download.
|
||||
|
||||
Read [fixed primary protocol](../docs/CROSS_LAYER_QKV_PROTOCOL.md) and
|
||||
[findings with evidence limits](../docs/CROSS_LAYER_QKV_FINDINGS.md).
|
||||
The original main protocol had no passing candidate. A later fixed one-source audit
|
||||
showed near-self-token retrieval and small readout errors, but failed strict fresh
|
||||
mean-JS/row-JS-P95/row-TV-P95 criteria; it is not relabeled as a primary pass.
|
||||
|
||||
- `core.py`: exact discrete head assignment, actual attention parts, common-space head
|
||||
decomposition, fixed recipient V/gate/O transplants and temporal-cache controls.
|
||||
- `protocol.py`, `fresh_samples.json`: predeclared layer/unit/gate scope and untouched fresh texts.
|
||||
- `run.py`, `fresh.py`: calibration, frozen selection, old validation and gated fresh/network steps.
|
||||
- `conditional.py`: separately registered post-screen metric audit on one fixed H23 candidate;
|
||||
no fallback selection or later threshold changes.
|
||||
- `audit_case.py`: recompute selected fresh probabilities, self mass, shared top5 mass and
|
||||
actual-H3 versus same-H23 difference; no full-network intervention after failure.
|
||||
- `report.py`, `audit.py`: saved-data scientific plots, raw-count/matching/probability audits,
|
||||
SHA256 and explicit small Git delivery with external large-cache metadata.
|
||||
|
||||
Use the authorized remote CPU environment with the retained source checkpoint/cache:
|
||||
|
||||
```sh
|
||||
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -B -m unittest tests.test_crossqkv -v
|
||||
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 timeout 7200 venv/bin/python -B -m crossqkv.run --source runs/cpu-qwen35-2b --root runs/NEW_QKV_DIRECTORY
|
||||
```
|
||||
|
||||
The main run requires a fresh directory and never overwrites prior protocol evidence.
|
||||
The recorded execution used `runs/cross-layer-qkv-v2` after fixing a controls-variable
|
||||
shadowing error in the first attempt; old logs and its profile cache are retained.
|
||||
The first attempt took about 10.3 seconds, then a 60-second reserve was deducted from
|
||||
the next 7140-second run limit. The main study finished in about 74.8 seconds.
|
||||
The fixed conditional audit used only remaining shared budget and took about 105.3 seconds.
|
||||
Final tests/plots/audits fit comfortably within the two-hour computation budget.
|
||||
|
||||
All weights and caches stay on the server until explicit user deletion. Git contains
|
||||
all figure tables and the selected small probability matrices; large caches have exact
|
||||
paths/sizes/hashes in `external-caches.json`. Old validation texts were used in earlier
|
||||
experiments; the four fresh texts were fixed before the new cross-layer measurements.
|
||||
@@ -0,0 +1 @@
|
||||
"""Cross-layer learned QKV function comparisons, CPU only."""
|
||||
@@ -0,0 +1,95 @@
|
||||
"""Audit and package only science outputs; retain all weights/profile caches."""
|
||||
import argparse
|
||||
import csv
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import time
|
||||
import numpy as np
|
||||
from functional.run import write_json
|
||||
from .protocol import LAYERS,PAIRS
|
||||
from .core import assignment
|
||||
|
||||
|
||||
def digest(path):
|
||||
h=hashlib.sha256()
|
||||
with path.open('rb') as f:
|
||||
while b:=f.read(4*1024*1024):h.update(b)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def main():
|
||||
p=argparse.ArgumentParser();p.add_argument('--root',type=Path,required=True);p.add_argument('--source',type=Path,required=True);p.add_argument('--output',type=Path,required=True);a=p.parse_args();root=a.root.resolve();source=a.source.resolve();out=a.output.resolve();assert root!=out
|
||||
status=json.loads((root/'status.json').read_text());assert status['state']=='completed' and status['last_source_layer']==23 and status['last_pair']==[19,23]
|
||||
expected={'full_attention':15,'qkv_head':960,'qk_map':30,'qk_head':960,'v_group':60};assert status['candidate_counts']==expected
|
||||
summary=json.loads((root/'calibration-summary.json').read_text());assert len(summary)==2025
|
||||
assert not status['selected'] and all(not x['strict_pass'] and not x['approximate_pass'] for x in summary)
|
||||
selection=json.loads((root/'selection.json').read_text());assert selection['qk_head_prequalified']==selection['v_group_prequalified']==0
|
||||
from collections import defaultdict
|
||||
rawout=defaultdict(list)
|
||||
for r in csv.DictReader((root/'calibration-output-metrics.csv').open()):rawout[r['candidate']].append(r)
|
||||
for c in summary:
|
||||
if c['output_verified']:
|
||||
rr=rawout[c['id']];assert len(rr)==c['output_observations']
|
||||
worst=max(max(float(r['error_a']),float(r['error_b'])) for r in rr)
|
||||
assert abs(worst-c['worst_output_error'])<1e-12 and worst>.20
|
||||
assert min(c['worst_mean_js'] for c in summary if c['kind']=='qk_head')>.05
|
||||
assert min(c['worst_value_error'] for c in summary if c['kind']=='v_group')>.20
|
||||
|
||||
rows=list(csv.DictReader((root/'head-assignment-costs.csv').open()));assert len(rows)==960
|
||||
for i,j in PAIRS:
|
||||
r=[x for x in rows if int(x['i'])==i and int(x['j'])==j];cost=np.empty((8,8));chosen=[]
|
||||
for x in r:
|
||||
ha,hb=int(x['head_i']),int(x['head_j']);cost[ha,hb]=float(x['calibration_mean_js'])
|
||||
if x['chosen']=='True':chosen.append((ha,hb))
|
||||
assert len(chosen)==8;assert [b for _,b in sorted(chosen)]==assignment(cost)
|
||||
tests=(root/'final-tests.txt').read_text();assert 'Ran 20 tests' in tests and '\nOK\n' in tests
|
||||
cond=json.loads((root/'conditional-status.json').read_text());assert cond['state']=='completed' and cond['fresh_function_pass'] is False
|
||||
case=json.loads((root/'conditional-case-audit.json').read_text());assert case['passed_recomputation'] and case['no_success_relabeling']
|
||||
assert len(case['strict_failures'])==3 and case['network_intervention_not_run_due_to_fresh_gate_failure']
|
||||
assert cond['shared_budget_seconds_including_60s_early_reserve']+600<7200
|
||||
prob=np.load(root/'conditional-fresh-probabilities.npz');target=prob['target_layer23_head6'];donor=prob['donor_layer3_head7_on_H23']
|
||||
from .core import distribution_metrics
|
||||
from .conditional import tv_metrics
|
||||
metric={**distribution_metrics(target,donor),**tv_metrics(target,donor)}
|
||||
fresh=[r for r in csv.DictReader((root/'conditional-function-metrics.csv').open()) if r['split']=='fresh_test']
|
||||
assert len(fresh)==4
|
||||
for k,values in metric.items():
|
||||
if k in fresh[0] and values[0] is not None:np.testing.assert_allclose(values,[float(r[k]) for r in fresh],atol=1e-12,rtol=1e-12)
|
||||
for file in root.glob('*.csv'):
|
||||
with file.open() as f:
|
||||
for r in csv.DictReader(f):
|
||||
for value in r.values():
|
||||
if value in ('',None):continue
|
||||
try:x=float(value)
|
||||
except ValueError:continue
|
||||
assert np.isfinite(x),(file,value)
|
||||
weights=json.loads((source/'manifest.json').read_text());assert weights['revision']=='15852e8c16360a2fea060d615a32b45270f8a8fc'
|
||||
for entry in weights['files']:
|
||||
if entry['name'].endswith('.safetensors'):assert (source/'weights'/entry['name']).stat().st_size==entry['bytes'] and entry['sha256']==entry['upstream_sha256']
|
||||
assert json.loads((source/'retention-policy.json').read_text())['automatic_cleanup'] is False
|
||||
reference=json.loads((source/'reference-check.json').read_text());assert reference['implementation_sha256']==digest(Path('functional/model.py'))
|
||||
write_json(root/'final-source-hashes.json',{str(file):digest(file) for file in [*sorted(Path('crossqkv').glob('*.py')),Path('crossqkv/fresh_samples.json'),Path('tests/test_crossqkv.py'),Path('functional/model.py')]})
|
||||
caches=[]
|
||||
for file in sorted((root/'cache').glob('*')):
|
||||
if file.is_file():caches.append({'server_path':str(file),'bytes':file.stat().st_size,'sha256':digest(file),'retained':True})
|
||||
old_cache=root.parent/'cross-layer-qkv'/'cache'/'calibration-source-3.npz'
|
||||
if old_cache.exists():caches.append({'server_path':str(old_cache),'bytes':old_cache.stat().st_size,'sha256':digest(old_cache),'retained':True,'failed_attempt_cache':True})
|
||||
write_json(root/'external-caches.json',{'host':'8.211.154.134','source_weights_directory':str(source/'weights'),'source_baseline_cache':str(source/'baseline-activations.npz'),
|
||||
'files':caches,'automatic_deletion':False,'all_figure_source_tables_and_selected_probabilities_in_git':True})
|
||||
(root/'DELIVERY.md').write_text('# Cross-layer QKV scientific delivery\n\nAll plot CSV/JSON, the selected fresh probability matrices, protocols, reference tests and failure logs are included.\nLarge intermediate profiles remain on 8.211.154.134 with locations/sizes/SHA256 in external-caches.json.\nThe fixed checkpoint and all caches are retained until explicit user deletion. No weights are committed.\nThe primary protocol has no passing candidate. The later fixed conditional audit is exploratory and its strict fresh checks failed; observed readout overlap is reported without relabeling it a complete pass.\n')
|
||||
out.mkdir(exist_ok=True,parents=True)
|
||||
files=[f for f in root.iterdir() if f.is_file() and f.name not in ('checksums.json','audit-validation.json','archive-files.json')]
|
||||
files += list((root/'figures').glob('*'))
|
||||
for file in files:
|
||||
dest=out/file.relative_to(root);dest.parent.mkdir(exist_ok=True,parents=True);shutil.copy2(file,dest)
|
||||
checks={str(f.relative_to(out)):digest(f) for f in sorted(out.rglob('*')) if f.is_file() and f.name not in ('checksums.json','audit-validation.json','archive-files.json')}
|
||||
write_json(out/'checksums.json',checks)
|
||||
write_json(out/'archive-files.json',[*checks,'checksums.json','audit-validation.json','archive-files.json'])
|
||||
write_json(out/'audit-validation.json',{'passed':True,'validated_unix':time.time(),'checksummed_files':len(checks),'primary_counts_rechecked':expected,
|
||||
'fresh_probabilities_independently_recomputed':True,'no_unrun_network_quality_claims':True,'all_weights_and_caches_retained':True})
|
||||
print(json.dumps({'passed':True,'checksummed_files':len(checks),'external_cache_files':len(caches),'weights_retained':True}))
|
||||
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,62 @@
|
||||
"""Recompute the frozen exploratory case for transparent plotted probabilities.
|
||||
|
||||
No new selection, no threshold changes, no full-network intervention after failure.
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
import numpy as np
|
||||
from functional.run import save_csv,write_json
|
||||
from .core import QKVModel,distribution_metrics
|
||||
from .conditional import tv_metrics
|
||||
|
||||
|
||||
def ranks_and_mass(a,b):
|
||||
result=[]
|
||||
for bi in range(a.shape[0]):
|
||||
d={'target_self_mass':[],'donor_self_mass':[],'argmax_agreement':[],'shared_top5_mass_target':[],'shared_top5_mass_donor':[]}
|
||||
for t in range(7,32):
|
||||
x,y=a[bi,0,t,:t+1],b[bi,0,t,:t+1]
|
||||
common=np.intersect1d(np.argsort(x)[-5:],np.argsort(y)[-5:])
|
||||
d['target_self_mass'].append(float(x[t]));d['donor_self_mass'].append(float(y[t]))
|
||||
d['argmax_agreement'].append(float(np.argmax(x)==np.argmax(y)))
|
||||
d['shared_top5_mass_target'].append(float(x[common].sum()));d['shared_top5_mass_donor'].append(float(y[common].sum()))
|
||||
result.append({k:float(np.mean(v)) for k,v in d.items()})
|
||||
return result
|
||||
|
||||
|
||||
def main():
|
||||
p=argparse.ArgumentParser();p.add_argument('--source',type=Path,required=True);p.add_argument('--root',type=Path,required=True);args=p.parse_args();root=args.root;source=args.source
|
||||
status=json.loads((root/'conditional-status.json').read_text());assert status['state']=='completed'
|
||||
data=np.load(root/'cache'/'conditional-fresh-inputs.npz')
|
||||
samples=json.loads((root/'conditional-fresh-samples.json').read_text())['samples']
|
||||
lm=QKVModel(source/'weights',json.loads((source/'model_config.json').read_text()),seconds=300)
|
||||
own=lm.parts(data['attn_23'],23);same=lm.parts(data['attn_23'],3);actual=lm.parts(data['attn_3'],3)
|
||||
target=own['p'][:,6:7];donor=same['p'][:,7:8];cached=actual['p'][:,7:8]
|
||||
np.savez(root/'conditional-fresh-probabilities.npz',target_layer23_head6=target,donor_layer3_head7_on_H23=donor,donor_layer3_head7_on_H3=cached)
|
||||
rows=[]
|
||||
from .run import outrows
|
||||
for mode,profile,prob in [('same_H23',same,donor),('actual_H3_to_H23',actual,cached)]:
|
||||
stats={**distribution_metrics(target,prob),**tv_metrics(target,prob)}
|
||||
mass=ranks_and_mass(target,prob)
|
||||
changed=lm.head_transplant_output(own,profile,23,6,7)
|
||||
out=outrows(own['heads'][:,:,6],changed,samples,{},1e-6,1e-6)
|
||||
for n,s in enumerate(samples):rows.append({'mode':mode,'sample':s['id'],**mass[n],**{k:(None if v[n] is None else float(v[n])) for k,v in stats.items()},'head_output_error':max(out[n]['error_a'],out[n]['error_b'])})
|
||||
save_csv(root/'conditional-fresh-source-diagnostic.csv',rows)
|
||||
with (root/'conditional-function-metrics.csv').open() as f:import csv;allrows=list(csv.DictReader(f))
|
||||
failures=[]
|
||||
for r in allrows:
|
||||
if r['split']!='fresh_test':continue
|
||||
for field,limit in [('js',.01),('js_p95',.05),('total_variation',.10),('total_variation_p95',.20)]:
|
||||
if float(r[field])>limit:failures.append({'sample':r['sample'],'field':field,'value':float(r[field]),'limit':limit})
|
||||
same_rows=[r for r in rows if r['mode']=='same_H23']
|
||||
classification='self_token_dominated' if min(min(r['target_self_mass'],r['donor_self_mass']) for r in same_rows)>.85 else 'inspect detailed source diagnostic'
|
||||
write_json(root/'conditional-case-audit.json',{'passed_recomputation':True,'strict_failures':failures,'pattern_classification':classification,
|
||||
'minimum_same_input_target_self_mass':min(r['target_self_mass'] for r in same_rows),'minimum_same_input_donor_self_mass':min(r['donor_self_mass'] for r in same_rows),
|
||||
'minimum_same_input_argmax_agreement':min(r['argmax_agreement'] for r in same_rows),
|
||||
'minimum_shared_top5_probability_mass':min(min(r['shared_top5_mass_target'],r['shared_top5_mass_donor']) for r in same_rows),
|
||||
'maximum_same_input_head_output_error':max(r['head_output_error'] for r in same_rows),
|
||||
'no_success_relabeling':True,'network_intervention_not_run_due_to_fresh_gate_failure':True})
|
||||
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,113 @@
|
||||
"""One explicitly exploratory, fixed-candidate audit of unweighted top-k rejection.
|
||||
|
||||
Primary protocol remains unchanged/negative. A new confirmation uses untouched
|
||||
fresh texts; no further candidate selection, threshold tuning, weights or R fitting.
|
||||
"""
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import resource
|
||||
import time
|
||||
import traceback
|
||||
import numpy as np
|
||||
from tokenizers import Tokenizer
|
||||
from functional.run import write_json,save_csv
|
||||
from .core import InterventionLM,distribution_metrics,output_metrics
|
||||
from .run import head_floor,rows_for
|
||||
|
||||
CANDIDATE={'id':'conditional_qk_head_3_23_7_6','kind':'qk_head','i':3,'j':23,'a':7,'b':6,'perm':None}
|
||||
PROTOCOL={'classification':'post-screen exploratory metric audit; not a primary-protocol positive',
|
||||
'candidate':CANDIDATE,'conditioning_input_layer':23,'donor_projection_layer':3,'target_head':6,'donor_head':7,
|
||||
'selection_basis':'one minimum worst-text JS among the already measured source-conditioned pairs; no raw readout/fresh data used',
|
||||
'js_max':.01,'js_row_p95_max':.05,'mean_total_variation_max':.10,'row_total_variation_p95_max':.20,
|
||||
'target_head_output_relative_l2_max':.20,'target_head_output_cosine_min':.98,
|
||||
'primary_query_positions':list(range(7,32)),'top5_is_diagnostic_only':True,
|
||||
'active_head_required':True,'no_candidate_fallback':True,'fresh_text_file':'crossqkv/fresh_samples.json',
|
||||
'same_input_network_direction':'layer23 head6 uses QK probability of layer3 head7 recomputed on H23',
|
||||
'actual_cache_separate':'layer3 probability on H3 reused at23; different-input diagnostic, not function-equivalence evidence',
|
||||
'quality_gate':{'mean_nll_delta_max':.01,'per_text_p95_nll_delta_max':.05},'training':False,'R_fitting':False,'cost_required':False}
|
||||
|
||||
|
||||
def tv_metrics(a,b):
|
||||
tv=.5*np.sum(np.abs(a.astype(np.float64)-b.astype(np.float64)),axis=-1)[:,:,7:]
|
||||
return {'total_variation':tv.mean((1,2)),'total_variation_p95':np.quantile(tv.reshape(tv.shape[0],-1),.95,axis=1)}
|
||||
|
||||
|
||||
def evaluate(lm,h,samples,split,donor_h=None):
|
||||
target=lm.parts(h,23);donor=lm.parts(h if donor_h is None else donor_h,3)
|
||||
a,b=target['p'][:,6:7],donor['p'][:,7:8]
|
||||
pm={**distribution_metrics(a,b),**tv_metrics(a,b)}
|
||||
changed=lm.head_transplant_output(target,donor,23,6,7)
|
||||
om=output_metrics(target['heads'][:,7:,6],changed[:,7:]);floor=head_floor(target)
|
||||
uniform=np.zeros_like(a)
|
||||
for t in range(a.shape[-1]):uniform[:,:,t,:t+1]=1/(t+1)
|
||||
null=distribution_metrics(a,uniform)
|
||||
rows=rows_for({**pm,**{f'output_{k}':v for k,v in om.items()},'target_uniform_js':null['js']},samples,{'split':split,'mode':'same_H23' if donor_h is None else 'actual_H3_to_H23'})
|
||||
for n,r in enumerate(rows):r['active']=bool(om['rms_a'][n]>=floor[n] and om['rms_b'][n]>=floor[n])
|
||||
passed=all(r['js']<=.01 and r['js_p95']<=.05 and r['total_variation']<=.10 and r['total_variation_p95']<=.20 and
|
||||
max(r['output_error_a'],r['output_error_b'])<=.20 and r['output_cosine']>=.98 and r['active'] for r in rows)
|
||||
return rows,passed
|
||||
|
||||
|
||||
def main():
|
||||
p=argparse.ArgumentParser();p.add_argument('--source',type=Path,required=True);p.add_argument('--root',type=Path,required=True);a=p.parse_args();source=a.source.resolve();root=a.root.resolve()
|
||||
assert not (root/'conditional-protocol.json').exists(),'Fresh conditional audit protocol required'
|
||||
write_json(root/'conditional-protocol.json',{**PROTOCOL,'registered_unix':time.time(),'fresh_text_sha256':hashlib.sha256(Path(__file__).with_name('fresh_samples.json').read_bytes()).hexdigest()})
|
||||
prior=json.loads((root/'status.json').read_text());assert prior['state']=='completed' and not prior['selected']
|
||||
reserve=prior['elapsed_seconds']+60
|
||||
remaining=int(7200-reserve-180)
|
||||
if remaining<120:raise TimeoutError('Insufficient shared two-hour budget')
|
||||
started=time.monotonic();status={'state':'running','phase':'calibration_readout','primary_protocol_unchanged_and_negative':True}
|
||||
rows=[]
|
||||
try:
|
||||
lm=InterventionLM(source/'weights',json.loads((source/'model_config.json').read_text()),seconds=remaining)
|
||||
old=json.loads((source/'samples-tokenized.json').read_text())['samples'];data=np.load(source/'baseline-activations.npz')
|
||||
for split in ('calibration','heldout'):
|
||||
index=np.array([i for i,s in enumerate(old) if s['split']==split]);samples=[old[i] for i in index]
|
||||
rr,passed=evaluate(lm,data['attn_23'][index],samples,split)
|
||||
rows.extend(rr);save_csv(root/'conditional-function-metrics.csv',rows)
|
||||
status[split+'_pass']=passed
|
||||
write_json(root/'conditional-status.json',status)
|
||||
if not passed:
|
||||
status['stop_reason']=f'{split} probability-mass/readout/activity screen failed';break
|
||||
else:
|
||||
fresh=json.loads(Path(__file__).with_name('fresh_samples.json').read_text())['samples'];tokenizer=Tokenizer.from_file(str(source/'tokenizer.json'));ids=[]
|
||||
for s in fresh:
|
||||
tokens=tokenizer.encode(s['text'],add_special_tokens=False).ids;assert len(tokens)>=32
|
||||
s.update(token_ids=tokens[:32],full_token_count=len(tokens),text_sha256=hashlib.sha256(s['text'].encode()).hexdigest());ids.append(tokens[:32])
|
||||
ids=np.array(ids,dtype=np.int64);write_json(root/'conditional-fresh-samples.json',{'samples':fresh,'candidate_already_fixed':True})
|
||||
status['phase']='fresh_baseline';write_json(root/'conditional-status.json',status)
|
||||
h,inputs=lm.forward(ids,capture=[('attn',3),('attn',23)])
|
||||
np.savez(root/'cache'/'conditional-fresh-inputs.npz',**inputs,final=h)
|
||||
base=lm.nll(h,ids);np.save(root/'conditional-fresh-baseline-token-nll.npy',base)
|
||||
rr,passed=evaluate(lm,inputs['attn_23'],fresh,'fresh_test');rows.extend(rr);save_csv(root/'conditional-function-metrics.csv',rows)
|
||||
status['fresh_function_pass']=passed
|
||||
if passed:
|
||||
status['phase']='one_direction_same_input_network';write_json(root/'conditional-status.json',status)
|
||||
lm.configure(CANDIDATE,reverse=True,cached=False);changed,_=lm.forward(ids);nll=lm.nll(changed,ids);np.save(root/'conditional-same-input-network-token-nll.npy',nll)
|
||||
delta=nll.mean(1)-base.mean(1)
|
||||
network={'mode':'layer23_head6_uses_layer3_head7_on_H23','sequence_nll_deltas':delta.tolist(),'mean_nll_delta':float(delta.mean()),'sequence_p95_nll_delta':float(np.quantile(delta,.95)),
|
||||
'quality_pass':bool(delta.mean()<=.01 and np.quantile(delta,.95)<=.05),'one_direction_only':True}
|
||||
write_json(root/'conditional-network.json',network);status['same_input_network_quality_pass']=network['quality_pass']
|
||||
# Check the actual-cache probability/readout separately, then run its network experiment.
|
||||
cached_rows,cached_function_pass=evaluate(lm,inputs['attn_23'],fresh,'fresh_test_actual_cache',donor_h=inputs['attn_3'])
|
||||
save_csv(root/'conditional-actual-cache-metrics.csv',cached_rows)
|
||||
status['actual_cache_function_screen_pass']=cached_function_pass
|
||||
lm.configure(CANDIDATE,reverse=True,cached=True);changed,_=lm.forward(ids);nll=lm.nll(changed,ids);np.save(root/'conditional-actual-cache-network-token-nll.npy',nll)
|
||||
delta=nll.mean(1)-base.mean(1)
|
||||
cache={'mode':'actual_layer3_H3_probability_reused_at23_H23','sequence_nll_deltas':delta.tolist(),'mean_nll_delta':float(delta.mean()),'sequence_p95_nll_delta':float(np.quantile(delta,.95)),
|
||||
'quality_pass':bool(delta.mean()<=.01 and np.quantile(delta,.95)<=.05),'different_hidden_inputs':True}
|
||||
write_json(root/'conditional-actual-cache-network.json',cache);status['actual_cache_network_quality_pass']=cache['quality_pass']
|
||||
status.update(state='completed',phase='done',conditional_function_positive=bool(status.get('fresh_function_pass')),
|
||||
position_pattern_classification='near_uniform' if rows and max(r['target_uniform_js'] for r in rows)<=.01 else 'inspect first-key/self/recent3 masses',
|
||||
claim_limit='Single-source H23-conditioned, single-head QK probability/readout similarity; not bidirectional, not V similarity, not full-QKV-head equivalence; exploratory addendum independently fresh-tested.')
|
||||
except Exception as exc:
|
||||
status.update(state='stopped',error_type=type(exc).__name__,error=str(exc));(root/'conditional-error.txt').write_text(traceback.format_exc());raise
|
||||
finally:
|
||||
status.update(elapsed_seconds=time.monotonic()-started,peak_rss_kib=resource.getrusage(resource.RUSAGE_SELF).ru_maxrss,
|
||||
shared_budget_seconds_including_60s_early_reserve=reserve+time.monotonic()-started)
|
||||
write_json(root/'conditional-status.json',status)
|
||||
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,171 @@
|
||||
"""Attention functions and discrete diagnostics, no continuous alignment/fitting."""
|
||||
import numpy as np
|
||||
from functional.model import StreamingLM,PREFIX,attention,rope,sigmoid
|
||||
|
||||
|
||||
def assignment(cost):
|
||||
"""Exact minimum-cost bijection via O(n^2 2^n) dynamic programming."""
|
||||
cost=np.asarray(cost,dtype=np.float64);n=cost.shape[0]
|
||||
assert cost.shape==(n,n) and n<=12 and np.isfinite(cost).all()
|
||||
states={0:(0.,())}
|
||||
for i in range(n):
|
||||
nxt={}
|
||||
for mask,(value,perm) in states.items():
|
||||
for j in range(n):
|
||||
if mask&(1<<j):continue
|
||||
key=mask|(1<<j);candidate=(value+float(cost[i,j]),perm+(j,))
|
||||
if key not in nxt or candidate<nxt[key]:nxt[key]=candidate
|
||||
states=nxt
|
||||
return list(states[(1<<n)-1][1])
|
||||
|
||||
|
||||
def inverse(perm):
|
||||
out=[0]*len(perm)
|
||||
for i,j in enumerate(perm):out[j]=i
|
||||
return out
|
||||
|
||||
|
||||
def output_metrics(a,b):
|
||||
"""Per-text directional error, cosine and output RMS. No fitted scaling."""
|
||||
x=a.reshape(a.shape[0],-1).astype(np.float64)
|
||||
y=b.reshape(b.shape[0],-1).astype(np.float64)
|
||||
xx=np.sum(x*x,-1);yy=np.sum(y*y,-1);diff=np.sum((x-y)**2,-1)
|
||||
return {'error_a':np.sqrt(diff/np.maximum(xx,1e-30)),
|
||||
'error_b':np.sqrt(diff/np.maximum(yy,1e-30)),
|
||||
'cosine':np.sum(x*y,-1)/np.maximum(np.sqrt(xx*yy),1e-30),
|
||||
'rms_a':np.sqrt(xx/x.shape[1]),'rms_b':np.sqrt(yy/y.shape[1])}
|
||||
|
||||
|
||||
def distribution_metrics(a,b):
|
||||
"""Aligned heads; primary queries have at least 8 legal keys (t>=7)."""
|
||||
a,b=a.astype(np.float64),b.astype(np.float64)
|
||||
assert a.shape==b.shape and a.shape[-1]>=8
|
||||
m=(a+b)*.5
|
||||
def term(x):return np.where(x>0,x*np.log(np.maximum(x,1e-300)/np.maximum(m,1e-300)),0)
|
||||
js=np.maximum(.5*np.sum(term(a)+term(b),axis=-1),0)
|
||||
eligible=js[...,7:]
|
||||
overlaps=[]
|
||||
for t in range(7,a.shape[-2]):
|
||||
aa=np.argsort(a[:,:,t,:t+1],axis=-1)[...,-5:]
|
||||
bb=np.argsort(b[:,:,t,:t+1],axis=-1)[...,-5:]
|
||||
overlaps.append(np.mean((aa[...,None]==bb[...,None,:]).any(-1),-1))
|
||||
# First-key ablation is diagnostic, not an unreported filter of positional repetitions.
|
||||
ar,br=a[...,1:],b[...,1:];ma,mb=ar.sum(-1),br.sum(-1)
|
||||
valid=(ma>1e-8)&(mb>1e-8)
|
||||
ar=ar/np.maximum(ma[...,None],1e-300);br=br/np.maximum(mb[...,None],1e-300)
|
||||
mr=(ar+br)*.5
|
||||
cond=.5*np.sum(np.where(ar>0,ar*np.log(np.maximum(ar,1e-300)/np.maximum(mr,1e-300)),0)+np.where(br>0,br*np.log(np.maximum(br,1e-300)/np.maximum(mr,1e-300)),0),-1)
|
||||
sink=[]
|
||||
for bi in range(a.shape[0]):
|
||||
which=valid[bi,:,7:];values=cond[bi,:,7:]
|
||||
sink.append(float(np.maximum(values[which],0).mean()) if which.any() else None)
|
||||
diag=np.stack([a[:,:,t,t] for t in range(7,a.shape[-2])],-1).mean((1,2))
|
||||
local=np.stack([a[:,:,t,max(0,t-2):t+1].sum(-1) for t in range(7,a.shape[-2])],-1).mean((1,2))
|
||||
return {'js':eligible.mean((1,2)),'js_p95':np.quantile(eligible.reshape(a.shape[0],-1),.95,axis=1),
|
||||
'top5':np.stack(overlaps,-1).mean((1,2)),'js_all_queries':js.mean((1,2)),
|
||||
'first_key_mass_a':a[:,:,7:,0].mean((1,2)),'first_key_mass_b':b[:,:,7:,0].mean((1,2)),
|
||||
'self_mass_a':diag,'recent3_mass_a':local,'js_without_first_key':sink}
|
||||
|
||||
|
||||
def cross_head_output_metrics(a,b):
|
||||
"""All head pairs, per text. a/b: [B,T,H,Dhidden]."""
|
||||
aa=a.transpose(0,2,1,3).reshape(a.shape[0],a.shape[2],-1).astype(np.float64)
|
||||
bb=b.transpose(0,2,1,3).reshape(b.shape[0],b.shape[2],-1).astype(np.float64)
|
||||
xx=np.sum(aa*aa,-1);yy=np.sum(bb*bb,-1)
|
||||
dot=np.einsum('bif,bjf->bij',aa,bb,optimize=True)
|
||||
diff=np.maximum(xx[:,:,None]+yy[:,None,:]-2*dot,0)
|
||||
return {'error_a':np.sqrt(diff/np.maximum(xx[:,:,None],1e-30)),
|
||||
'error_b':np.sqrt(diff/np.maximum(yy[:,None,:],1e-30)),
|
||||
'cosine':dot/np.maximum(np.sqrt(xx[:,:,None]*yy[:,None,:]),1e-30),
|
||||
'rms_a':np.broadcast_to(np.sqrt(xx/aa.shape[-1])[:,:,None],dot.shape),
|
||||
'rms_b':np.broadcast_to(np.sqrt(yy/bb.shape[-1])[:,None,:],dot.shape)}
|
||||
|
||||
|
||||
def value_gram_cka(a,b):
|
||||
"""Centered token Gram agreement, diagnostic only; not interchangeability."""
|
||||
a=a.astype(np.float64)-a.mean(axis=1,keepdims=True,dtype=np.float64)
|
||||
b=b.astype(np.float64)-b.mean(axis=1,keepdims=True,dtype=np.float64)
|
||||
ga=np.einsum('btd,bsd->bts',a,a);gb=np.einsum('btd,bsd->bts',b,b)
|
||||
return np.sum(ga*gb,(1,2))/np.maximum(np.sqrt(np.sum(ga*ga,(1,2))*np.sum(gb*gb,(1,2))),1e-30)
|
||||
|
||||
|
||||
class QKVModel(StreamingLM):
|
||||
def project(self,context,gate,layer):
|
||||
return self.linear((context*sigmoid(gate)).reshape(*context.shape[:2],-1),f'layers.{layer}.self_attn.o_proj')
|
||||
|
||||
def project_heads(self,context,gate,layer):
|
||||
"""Each head's contribution in the common residual output coordinates."""
|
||||
c=context*sigmoid(gate);b,t,h,d=c.shape
|
||||
weight=self.tensors[PREFIX+f'layers.{layer}.self_attn.o_proj.weight']
|
||||
outdim,indim=weight.shape;assert indim==h*d
|
||||
out=np.empty((b,t,h,outdim),dtype=np.float32)
|
||||
for start in range(0,outdim,256):
|
||||
self.guard();end=min(start+256,outdim)
|
||||
w=weight.read(slice(start*indim,end*indim)).reshape(end-start,indim)
|
||||
for head in range(h):out[:,:,head,start:end]=c[:,:,head,:]@w[:,head*d:(head+1)*d].T
|
||||
del w;weight.evict()
|
||||
return out
|
||||
|
||||
def parts(self,h,layer,heads=True):
|
||||
p=f'layers.{layer}.self_attn.';c=self.config;d=c['head_dim'];nh=c['num_attention_heads'];nk=c['num_key_value_heads']
|
||||
mixed=self.linear(h,p+'q_proj').reshape(*h.shape[:2],nh,2*d)
|
||||
q,gate=mixed[...,:d],mixed[...,d:].copy()
|
||||
k=self.linear(h,p+'k_proj').reshape(*h.shape[:2],nk,d)
|
||||
v=self.linear(h,p+'v_proj').reshape(*h.shape[:2],nk,d)
|
||||
q=self.normalization(q,p+'q_norm');k=self.normalization(k,p+'k_norm')
|
||||
r=c['rope_parameters'];q,k=(rope(x,r['rope_theta'],r['partial_rotary_factor']) for x in (q,k))
|
||||
context,probs=attention(q,k,v)
|
||||
out={'q':q,'k':k,'p':probs,'v':v,'gate':gate,'context':context,'output':self.project(context,gate,layer)}
|
||||
if heads:out['heads']=self.project_heads(context,gate,layer)
|
||||
return out
|
||||
|
||||
def transplant(self,own,donor,layer,kind,target_head=None,donor_head=None,perm=None):
|
||||
repeats=self.config['num_attention_heads']//self.config['num_key_value_heads']
|
||||
if kind=='full_attention':return donor['output']
|
||||
if kind=='qkv_head':return own['output']-own['heads'][:,:,target_head]+donor['heads'][:,:,donor_head]
|
||||
probs=own['p'].copy();v=own['v'].copy()
|
||||
if kind=='qk_map':probs=donor['p'][:,perm]
|
||||
elif kind=='qk_head':probs[:,target_head]=donor['p'][:,donor_head]
|
||||
elif kind=='v_group':v[:,:,target_head]=donor['v'][:,:,donor_head]
|
||||
else:raise ValueError(kind)
|
||||
context=np.einsum('bhts,bshd->bthd',probs,np.repeat(v,repeats,axis=2))
|
||||
return self.project(context,own['gate'],layer)
|
||||
|
||||
def head_transplant_output(self,own,donor,layer,target_head,donor_head):
|
||||
"""QK-only transplanted target head contribution; all other heads omitted."""
|
||||
repeats=self.config['num_attention_heads']//self.config['num_key_value_heads']
|
||||
c=own['context'].copy()
|
||||
c[:,:,target_head]=np.einsum('bts,bsd->btd',donor['p'][:,donor_head],own['v'][:,:,target_head//repeats])
|
||||
# Zero remaining head contexts before projection to isolate this learned subfunction.
|
||||
for head in range(c.shape[2]):
|
||||
if head!=target_head:c[:,:,head]=0
|
||||
return self.project(c,own['gate'],layer)
|
||||
|
||||
def value_group_output(self,own,donor,layer,target_group,donor_group):
|
||||
repeats=self.config['num_attention_heads']//self.config['num_key_value_heads'];c=np.zeros_like(own['context'])
|
||||
for head in range(target_group*repeats,(target_group+1)*repeats):
|
||||
c[:,:,head]=np.einsum('bts,bsd->btd',own['p'][:,head],donor['v'][:,:,donor_group])
|
||||
return self.project(c,own['gate'],layer)
|
||||
|
||||
|
||||
class InterventionLM(QKVModel):
|
||||
def configure(self,candidate,reverse=False,cached=False):
|
||||
self.candidate=candidate;self.reverse=reverse;self.use_cache=cached;self.donor_cache=None
|
||||
self.target_layer=candidate['j'] if reverse else candidate['i']
|
||||
self.donor_layer=candidate['i'] if reverse else candidate['j']
|
||||
self.target_head=candidate.get('b') if reverse else candidate.get('a')
|
||||
self.donor_head=candidate.get('a') if reverse else candidate.get('b')
|
||||
self.perm=inverse(candidate['perm']) if reverse and candidate.get('perm') else candidate.get('perm')
|
||||
if cached:assert self.donor_layer<self.target_layer,'Cannot reuse future layer state'
|
||||
|
||||
def full_attention(self,h,layer,shared_p=None):
|
||||
if not hasattr(self,'candidate'):return super().full_attention(h,layer,shared_p)
|
||||
if self.use_cache and layer==self.donor_layer:
|
||||
self.donor_cache=self.parts(h,layer,heads=self.candidate['kind']=='qkv_head')
|
||||
return self.donor_cache['output'],self.donor_cache['p']
|
||||
if layer!=self.target_layer:return super().full_attention(h,layer,shared_p)
|
||||
own=self.parts(h,layer,heads=self.candidate['kind']=='qkv_head')
|
||||
donor=self.donor_cache if self.use_cache else self.parts(h,self.donor_layer,heads=self.candidate['kind']=='qkv_head')
|
||||
assert donor is not None
|
||||
out=self.transplant(own,donor,layer,self.candidate['kind'],self.target_head,self.donor_head,self.perm)
|
||||
return out,own['p']
|
||||
@@ -0,0 +1,56 @@
|
||||
"""Fresh heldout confirmation and same-input versus actual-state interventions."""
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import time
|
||||
import numpy as np
|
||||
from functional.run import write_json,save_csv
|
||||
from .protocol import LAYERS
|
||||
|
||||
|
||||
def run_fresh(source,root,c,tier,lm,status):
|
||||
from tokenizers import Tokenizer
|
||||
from .run import validate_selected,candidate_pass,summarize
|
||||
samples=json.loads(Path(__file__).with_name('fresh_samples.json').read_text())['samples']
|
||||
tokenizer=Tokenizer.from_file(str(source/'tokenizer.json'));ids=[]
|
||||
for s in samples:
|
||||
encoded=tokenizer.encode(s['text'],add_special_tokens=False).ids
|
||||
assert len(encoded)>=32,s['id']
|
||||
s.update(token_ids=encoded[:32],full_token_count=len(encoded),text_sha256=hashlib.sha256(s['text'].encode()).hexdigest())
|
||||
ids.append(encoded[:32])
|
||||
ids=np.array(ids,dtype=np.int64)
|
||||
write_json(root/'fresh-samples-tokenized.json',{'samples':samples,'selection_already_frozen':True})
|
||||
status['phase']='fresh_test_baseline';write_json(root/'status.json',status)
|
||||
h,inputs=lm.forward(ids,capture=[('attn',i) for i in LAYERS])
|
||||
np.savez(root/'cache'/'fresh-baseline-activations.npz',final=h,**inputs)
|
||||
nll=lm.nll(h,ids);np.save(root/'fresh-baseline-token-nll.npy',nll)
|
||||
checked=validate_selected(lm,c,inputs,samples,np.arange(len(samples)))
|
||||
passed=candidate_pass(checked,tier)
|
||||
write_json(root/'fresh-function-summary.json',summarize(checked))
|
||||
save_csv(root/'fresh-output-metrics.csv',checked['output_rows']);save_csv(root/'fresh-probability-metrics.csv',checked['probability_rows']);save_csv(root/'fresh-value-metrics.csv',checked['value_rows'])
|
||||
status.update(fresh_function_pass=passed,fresh_confirmation_tier=tier)
|
||||
if not passed:return
|
||||
status['phase']='same_input_network_substitution';write_json(root/'status.json',status)
|
||||
interventions=[]
|
||||
for reverse in (False,True):
|
||||
lm.configure(c,reverse=reverse,cached=False)
|
||||
t=time.monotonic();changed,_=lm.forward(ids);loss=lm.nll(changed,ids)
|
||||
label=f'{lm.target_layer}_uses_{lm.donor_layer}'
|
||||
np.save(root/f'network-{label}-token-nll.npy',loss)
|
||||
delta=loss.mean(1)-nll.mean(1)
|
||||
rec={'mode':'same_input_donor_function','target_layer':lm.target_layer,'donor_layer':lm.donor_layer,
|
||||
'target_head_or_group':lm.target_head,'donor_head_or_group':lm.donor_head,
|
||||
'sequence_nll_deltas':delta.tolist(),'mean_nll_delta':float(delta.mean()),'sequence_p95_nll_delta':float(np.quantile(delta,.95)),
|
||||
'quality_pass':bool(delta.mean()<=.01 and np.quantile(delta,.95)<=.05),'elapsed_seconds':time.monotonic()-t}
|
||||
interventions.append(rec);write_json(root/'network-interventions.json',interventions)
|
||||
# Temporal reuse is a separate intervention and only goes from earlier to later.
|
||||
status['phase']='earlier_to_later_actual_cache';write_json(root/'status.json',status)
|
||||
lm.configure(c,reverse=True,cached=True)
|
||||
changed,_=lm.forward(ids);loss=lm.nll(changed,ids);np.save(root/'network-earlier-actual-cache-token-nll.npy',loss)
|
||||
delta=loss.mean(1)-nll.mean(1)
|
||||
cache={'mode':'earlier_actual_state_reused_later','donor_layer':lm.donor_layer,'target_layer':lm.target_layer,
|
||||
'sequence_nll_deltas':delta.tolist(),'mean_nll_delta':float(delta.mean()),'sequence_p95_nll_delta':float(np.quantile(delta,.95)),
|
||||
'quality_pass':bool(delta.mean()<=.01 and np.quantile(delta,.95)<=.05),
|
||||
'not_same_input_comparison':True,'no_reverse_future_state_claim':True}
|
||||
write_json(root/'actual-cache-intervention.json',cache)
|
||||
status.update(bidirectional_same_input_network_pass=all(r['quality_pass'] for r in interventions),actual_cache_quality_pass=cache['quality_pass'])
|
||||
@@ -0,0 +1,9 @@
|
||||
{
|
||||
"origin":"Four experiment-authored fresh plain texts, fixed for this cross-layer QKV goal before measurements; no chat template, first 32 tokenizer tokens.",
|
||||
"samples":[
|
||||
{"id":"fresh-zh-1","language":"zh","text":"植物学家在不同海拔采集了叶片样本,并为每一份标本记录温度、光照和土壤湿度。回到实验室后,她按地点整理标签,先核对缺失记录,再比较不同环境中的叶片形状。"},
|
||||
{"id":"fresh-zh-2","language":"zh","text":"如果一个盒子里有十二枚蓝色棋子和八枚白色棋子,随机取出一枚后不放回,第二次取到蓝色的概率需要根据第一次的结果分别计算。请先列出各个事件,再检查它们的概率之和。"},
|
||||
{"id":"fresh-en-1","language":"en","text":"A technician recorded the vibration of a small motor before and after replacing its bearings. She used the same mounting position and sampling interval for both measurements. The final notebook included the raw traces, instrument settings, and a brief explanation of unusual observations."},
|
||||
{"id":"fresh-en-2","language":"en","text":"The function walks through a list of integers and stores each running total in a new list. An empty input should return an empty result. Before changing the implementation, compare a short example by hand and make sure negative numbers preserve the intended behavior."}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
"""Fixed protocol, constants frozen before any new cross-layer measurements."""
|
||||
from itertools import combinations
|
||||
|
||||
LAYERS=[3,7,11,15,19,23]
|
||||
PAIRS=list(combinations(LAYERS,2))
|
||||
TIERS={
|
||||
'strict':{'output_error':.05,'output_cosine':.99,'js':.01,'js_p95':.05,'top5':.8,'value_error':.05,'value_cosine':.99},
|
||||
'approximate':{'output_error':.20,'output_cosine':.98,'js':.05,'js_p95':.15,'top5':.6,'value_error':.20,'value_cosine':.98}}
|
||||
PROTOCOL={'layers':LAYERS,'layer_pairs':PAIRS,'heads':8,'kv_heads':2,'head_dim':256,
|
||||
'calibration_samples':['cal-zh-1','cal-zh-2','cal-en-1','cal-en-2'],
|
||||
'validation_samples':['test-zh-1','test-zh-2','test-en-1','test-en-2'],
|
||||
'fresh_test_file':'crossqkv/fresh_samples.json','fresh_test_count':4,'tokens_per_text':32,
|
||||
'query_positions_for_primary_metrics':list(range(7,32)),
|
||||
'candidate_types':['full_attention','qkv_head','qk_map','qk_head','v_group'],
|
||||
'discrete_head_matching':'Exact 8-head minimum mean-JS bijection, two source-layer calibration input distributions; frozen before validation; P injection only',
|
||||
'continuous_alignment':False,'training':False,'scale_fitting':False,'gpu':False,'cost_required':False,
|
||||
'tiers':TIERS,'max_qk_head_output_candidates':32,'max_v_group_output_candidates':16,
|
||||
'all_qk_map_output_candidates':30,'max_final_candidate':1,
|
||||
'selection':'strict before approximate; type priority full_attention/qkv_head/qk_map/qk_head/v_group; then worst output error, JS, ID; no validation-driven fallback',
|
||||
'head_output_activity_floor':'max(1e-6, 0.1 * within-layer median head RMS) for every calibration/validation/test text',
|
||||
'fresh_test_run_only_if_frozen_candidate_passes_validation':True,
|
||||
'network_interventions':'two same-input donor-function substitution directions; plus one earlier-to-later actual cached-state reuse test reported separately',
|
||||
'network_quality':{'mean_token_nll_delta_max':.01,'per_text_p95_nll_delta_max':.05},
|
||||
'retention':'all weights and caches retained until explicit user deletion request',
|
||||
'seconds':7200,'controls':['identity','uniform causal P','seeded within-causal-row key shuffle','first-key ablation','self/recent3 mass','value Gram to input','value Gram of two seeded Gaussian projections'],
|
||||
'positional_repeat_labels':{'sink_dominated':.90,'self_dominated':.90,'recent3_dominated':.90},
|
||||
'limitations':['Old validation texts were used in prior experiments; four fresh texts are reserved for this new goal',
|
||||
'Direct V-coordinate disagreement does not disprove V/O functional equivalence; centered value Gram CKA is diagnostic only',
|
||||
'Arbitrary P-head assignment does not authorize arbitrary GQA KV-weight permutation',
|
||||
'Short text-only pilot does not establish broad task equivalence or linear-attention reuse']}
|
||||
@@ -0,0 +1,126 @@
|
||||
"""Scientific plots from saved tables/probabilities only; no model access."""
|
||||
import argparse
|
||||
import csv
|
||||
import json
|
||||
from pathlib import Path
|
||||
import numpy as np
|
||||
import matplotlib
|
||||
matplotlib.use('Agg')
|
||||
import matplotlib.pyplot as plt
|
||||
from matplotlib.patches import Rectangle
|
||||
from .protocol import LAYERS,PAIRS
|
||||
|
||||
|
||||
def read_csv(path):
|
||||
with path.open() as f:return list(csv.DictReader(f))
|
||||
|
||||
|
||||
def main():
|
||||
p=argparse.ArgumentParser();p.add_argument('--root',type=Path,required=True);args=p.parse_args();root=args.root;out=root/'figures';out.mkdir(exist_ok=True)
|
||||
plt.rcParams.update({'font.size':9,'pdf.fonttype':42,'svg.fonttype':'none','savefig.bbox':'tight'})
|
||||
def save(fig,name):
|
||||
for ext in ('png','pdf','svg'):fig.savefig(out/f'{name}.{ext}',dpi=180)
|
||||
plt.close(fig)
|
||||
status=json.loads((root/'status.json').read_text());summary=json.loads((root/'calibration-summary.json').read_text())
|
||||
matrices=[]
|
||||
for mode in ('qk_head','identity','matched'):
|
||||
matrix=np.full((6,6),np.nan)
|
||||
for i,j in PAIRS:
|
||||
rows=[r for r in summary if r['i']==i and r['j']==j and ((mode=='qk_head' and r['kind']=='qk_head') or (mode!='qk_head' and r['id']==f'qk_map_{i}_{j}_{mode}'))]
|
||||
value=min(r['worst_mean_js'] for r in rows)
|
||||
a,b=LAYERS.index(i),LAYERS.index(j);matrix[a,b]=matrix[b,a]=value
|
||||
matrices.append(matrix)
|
||||
fig,axes=plt.subplots(1,3,figsize=(13,4.7));vmax=max(np.nanmax(m) for m in matrices)
|
||||
for ax,m,title in zip(axes,matrices,['Best single head pair','All heads: original order','All heads: calibration bijection']):
|
||||
im=ax.imshow(m,cmap='YlOrRd',vmin=0,vmax=vmax)
|
||||
for a in range(6):
|
||||
ax.add_patch(Rectangle((a-.5,a-.5),1,1,facecolor='#eeeeee',edgecolor='#aaaaaa',hatch='///',lw=.3))
|
||||
for b in range(6):
|
||||
if a!=b:ax.text(b,a,f'{m[a,b]:.3f}',ha='center',va='center',fontsize=7,color='black' if m[a,b]<vmax*.65 else 'white')
|
||||
ax.set_xticks(range(6),LAYERS);ax.set_yticks(range(6),LAYERS);ax.set_xlabel('Layer index');ax.set_ylabel('Layer index');ax.set_title(title)
|
||||
fig.suptitle('QK retrieval divergence: worst-text mean JS over both source input sets\nCalibration only; single-head panel minimizes over 64 cross-layer head pairs; hatched diagonal unused')
|
||||
fig.subplots_adjust(top=.77,bottom=.18,right=.9,wspace=.35);cax=fig.add_axes([.92,.2,.012,.5]);fig.colorbar(im,cax=cax,label='JS (natural log); lower is closer');save(fig,'cross_layer_qk_divergence')
|
||||
order=['full_attention','qkv_head','qk_map','qk_head','v_group'];labels=['Whole attention','QKV+gate/O head','QK map transplant','QK head transplant','V group transplant']
|
||||
fig,ax=plt.subplots(figsize=(10,4.9));rng=np.random.default_rng(0)
|
||||
for idx,(kind,label) in enumerate(zip(order,labels)):
|
||||
rows=[r for r in summary if r['kind']==kind and r['output_verified']]
|
||||
if rows:
|
||||
x=idx+rng.uniform(-.15,.15,len(rows));y=[r['worst_output_error'] for r in rows]
|
||||
ax.scatter(x,y,s=12,alpha=.35,color='#606060')
|
||||
ax.text(idx,max(y)*1.1,f'n={len(rows)}',ha='center',fontsize=8)
|
||||
else:ax.text(idx,.3,'not output-probed\ninitial gate failed',ha='center',va='center',fontsize=8)
|
||||
ax.axhline(.05,color='#b2182b',ls='--',label='Strict output error = 0.05')
|
||||
ax.axhline(.2,color='#d95f02',ls=':',label='Approximate output error = 0.20')
|
||||
ax.set_yscale('log');ax.set_xlim(-.5,4.5);ax.set_xticks(range(5),labels,rotation=20,ha='right');ax.set_ylabel('Worst directional relative L2, queries 7–31')
|
||||
ax.set_title('Primary protocol: calibrated cross-layer output comparisons\nEach point is one unit; no candidate passed all predeclared gates')
|
||||
ax.legend(loc='upper left',bbox_to_anchor=(0,-.28),fontsize=8);fig.tight_layout();save(fig,'cross_layer_output_errors')
|
||||
values=read_csv(root/'calibration-value-metrics.csv');unit={}
|
||||
for r in values:unit.setdefault(r['candidate'],[]).append(r)
|
||||
xx=[np.mean([float(r['centered_value_gram_cka']) for r in rows]) for rows in unit.values()]
|
||||
yy=[max(max(float(r['error_a']),float(r['error_b'])) for r in rows) for rows in unit.values()]
|
||||
controls=read_csv(root/'controls.csv');random_cka=[float(r['gram_cka']) for r in controls if r['control']=='value_gram_random_projection_pair']
|
||||
fig,ax=plt.subplots(figsize=(7,4.5));ax.scatter(xx,yy,s=24,color='#377eb8',label='60 cross-layer KV-head pairs')
|
||||
ax.axvspan(min(random_cka),max(random_cka),color='gray',alpha=.15,label='Random-projection Gram CKA range')
|
||||
ax.axvline(np.median(random_cka),color='gray',ls=':')
|
||||
ax.axhline(.2,color='#b2182b',ls='--',label='Approximate direct V error = 0.20')
|
||||
ax.set_xlabel('Mean centered token-Gram CKA');ax.set_ylabel('Worst direct V relative L2');ax.set_title('V geometry is a diagnostic, not substitution evidence')
|
||||
ax.legend(loc='upper left',bbox_to_anchor=(0,-.19),fontsize=8);fig.tight_layout();save(fig,'value_geometry_and_direct_error')
|
||||
conditional_path=root/'conditional-status.json'
|
||||
cond=json.loads(conditional_path.read_text()) if conditional_path.exists() else {}
|
||||
if (root/'conditional-function-metrics.csv').exists():
|
||||
rows=read_csv(root/'conditional-function-metrics.csv');labels=[r['sample'] for r in rows];x=np.arange(len(rows))
|
||||
fig,axes=plt.subplots(1,2,figsize=(13,5))
|
||||
axes[0].plot(x,[float(r['top5']) for r in rows],'o-',label='Unweighted top-5 overlap')
|
||||
axes[0].plot(x,[1-float(r['total_variation']) for r in rows],'o-',label='Probability-mass overlap (1−TV)')
|
||||
axes[0].set_ylim(0,1.03);axes[0].set_ylabel('Overlap');axes[0].legend(loc='lower left',fontsize=8)
|
||||
axes[1].plot(x,[max(float(r['output_error_a']),float(r['output_error_b'])) for r in rows],'o-',color='#238b45')
|
||||
axes[1].set_ylim(0,max(.05,max(max(float(r['output_error_a']),float(r['output_error_b'])) for r in rows)*1.15))
|
||||
axes[1].set_ylabel('Head readout relative L2');axes[1].set_title('Fixed layer23 V/gate/O',fontsize=9)
|
||||
for ax in axes:
|
||||
for pos in (3.5,7.5):ax.axvline(pos,color='gray',lw=.6)
|
||||
ax.set_xticks(x,labels,rotation=50,ha='right');ax.set_xlabel('Calibration / old validation / fresh tests, 4 each')
|
||||
fig.suptitle('Exploratory fixed case: layer3 head7 vs layer23 head6, both applied to H23\nFresh JS / tail-mass limits not fully passed; network substitution not run')
|
||||
fig.tight_layout();save(fig,'conditional_probability_mass_and_readout')
|
||||
if (root/'conditional-fresh-probabilities.npz').exists():
|
||||
p=np.load(root/'conditional-fresh-probabilities.npz');a=p['target_layer23_head6'][0,0];b=p['donor_layer3_head7_on_H23'][0,0];d=p['donor_layer3_head7_on_H3'][0,0]
|
||||
fig,axes=plt.subplots(2,2,figsize=(9,8));images=[a,b,np.abs(a-b),d]
|
||||
titles=['Layer23 head6 on H23','Layer3 head7 on the same H23','Absolute same-input probability difference','Layer3 head7 on its actual H3 (different input)']
|
||||
for idx,(ax,z,title) in enumerate(zip(axes.flat,images,titles)):
|
||||
im=ax.imshow(z,cmap='magma',vmin=0,vmax=(max(float(z.max()),1e-8) if idx==2 else 1.));fig.colorbar(im,ax=ax,fraction=.045,pad=.03)
|
||||
ax.set_xticks([0,7,15,23,31]);ax.set_yticks([0,7,15,23,31]);ax.set_xlabel('Key position');ax.set_ylabel('Query position');ax.set_title(title,fontsize=9)
|
||||
fig.suptitle('One fresh Chinese text: cross-layer conditional self-token retrieval\nRows 0–6 shown for context; primary metrics use queries 7–31')
|
||||
fig.tight_layout();save(fig,'conditional_cross_layer_probability_maps')
|
||||
text=['# Cross-layer QKV CPU study','',
|
||||
'Primary protocol and exploratory addendum are reported separately; no post-hoc success relabeling.',
|
||||
f'- Primary state: {status["state"]}; selected candidate: {status.get("selected",[])}.',
|
||||
f'- Candidate counts: {status.get("candidate_counts")}.',
|
||||
f'- Passing counts (approximate tier, all gates): {status.get("passing_counts")}.',
|
||||
f'- Primary computation seconds: {status.get("elapsed_seconds")}; peak RSS KiB: {status.get("peak_rss_kib")}.','',
|
||||
'## What was measured','',
|
||||
'Six full-attention layers, indices 3/7/11/15/19/23; all 15 cross-layer pairs. No within-layer Q/K/V interchange.',
|
||||
'Same normalized hidden inputs from both source layers; frozen Q/K norm, partial RoPE, GQA, V, output gate and O.',
|
||||
'QK head probabilities, whole-map injection in original/frozen matched head order, direct V functions, complete head contributions in residual coordinates, and complete branches were separated.',
|
||||
'Primary metrics use queries 7–31. Discrete matching uses calibration only; no continuous alignment, scale fitting or training.',
|
||||
'No primary candidate passed, so primary validation/fresh/network intervention stages were not entered.','']
|
||||
for kind in order:
|
||||
group=[r for r in summary if r['kind']==kind];outrows=[r for r in group if r['worst_output_error'] is not None];p=[r for r in group if r['worst_mean_js'] is not None];v=[r for r in group if r['worst_value_error'] is not None]
|
||||
text += [f'- {kind}: n={len(group)}, best worst output error={min((r["worst_output_error"] for r in outrows),default=None)}, minimum worst-text mean JS={min((r["worst_mean_js"] for r in p),default=None)}, best worst direct V error={min((r["worst_value_error"] for r in v),default=None)}.']
|
||||
if cond:
|
||||
text+=['','## Exploratory metric audit: frozen one-source candidate','',
|
||||
'After inspecting primary calibration results, one candidate was frozen: layer3 head7 vs layer23 head6 on H23.',
|
||||
'This addendum audited the instability of unweighted top-k tail ranking; original primary results were not changed.',
|
||||
'Four old validation and four previously unused fresh texts were evaluated with frozen extra gates and no candidate fallback.',
|
||||
f'- Addendum fresh function pass: {cond.get("fresh_function_pass")}.',
|
||||
f'- Overall addendum state: {cond.get("state")}; primary unchanged/negative: {cond.get("primary_protocol_unchanged_and_negative")}.',
|
||||
'Fresh probability-mass and fixed V/gate/O readout measurements show conditional behavioral overlap; strict fresh mean-JS, row-JS-P95 and row-TV-P95 limits still failed.',
|
||||
'No full-network substitution was run after that failure. V similarity, bidirectionality and complete QKV-head equivalence are not established.',
|
||||
'The actual-H3 cache probability is a separate different-input diagnostic, not the same-input function comparison.']
|
||||
text+=['','## Limits and retention','',
|
||||
'This is a short text-only pilot, not proof of absence of repetition elsewhere. Exact arithmetic, matrix coordinates, retrieval behavior and final residual output are distinct questions.',
|
||||
'Centered value Gram CKA can be high even for independent Gaussian projections of common inputs; it is not substitution evidence.',
|
||||
'All weights and large profile caches remain on the authorized server. Git delivery includes all plot tables, probabilities, protocols and cache location/hash metadata.',
|
||||
'No GPU, new checkpoint download, model training, continuous R fitting or automatic weight/cache deletion.']
|
||||
(root/'REPORT.md').write_text('\n'.join(text)+'\n')
|
||||
|
||||
|
||||
if __name__=='__main__':main()
|
||||
+279
@@ -0,0 +1,279 @@
|
||||
"""Bounded 15-pair CPU study, calibration selection before any heldout feedback."""
|
||||
import argparse
|
||||
from collections import defaultdict
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import resource
|
||||
import time
|
||||
import traceback
|
||||
import numpy as np
|
||||
from functional.run import save_csv,write_json
|
||||
from .core import (QKVModel,InterventionLM,assignment,inverse,output_metrics,
|
||||
cross_head_output_metrics,distribution_metrics,value_gram_cka)
|
||||
from .protocol import LAYERS,PAIRS,PROTOCOL,TIERS
|
||||
|
||||
PRIMARY=slice(7,None)
|
||||
TYPE_ORDER={'full_attention':0,'qkv_head':1,'qk_map':2,'qk_head':3,'v_group':4}
|
||||
|
||||
|
||||
def candidate_id(kind,i,j,a=None,b=None,suffix=''):
|
||||
return f'{kind}_{i}_{j}'+(f'_{a}_{b}' if a is not None else '')+suffix
|
||||
|
||||
|
||||
def rows_for(metrics,samples,meta):
|
||||
rows=[]
|
||||
for n,s in enumerate(samples):
|
||||
r={**meta,'sample':s['id']}
|
||||
for key,value in metrics.items():
|
||||
v=value[n]
|
||||
r[key]=None if v is None else float(v)
|
||||
rows.append(r)
|
||||
return rows
|
||||
|
||||
|
||||
def outrows(a,b,samples,meta,floor_a=1e-6,floor_b=1e-6):
|
||||
m=output_metrics(a[:,PRIMARY],b[:,PRIMARY]);rows=rows_for(m,samples,meta)
|
||||
for n,r in enumerate(rows):r['active']=bool(m['rms_a'][n]>=np.broadcast_to(floor_a,m['rms_a'].shape)[n] and m['rms_b'][n]>=np.broadcast_to(floor_b,m['rms_b'].shape)[n])
|
||||
return rows
|
||||
|
||||
|
||||
def head_floor(profile):
|
||||
rms=np.sqrt(np.mean(profile['heads'][:,7:].astype(np.float64)**2,axis=(1,3)))
|
||||
return np.maximum(1e-6,.1*np.median(rms,axis=1))
|
||||
|
||||
|
||||
def candidate_pass(c,tier):
|
||||
t=TIERS[tier];out=c.get('output_rows',[])
|
||||
if not out or not all(r['active'] and max(r['error_a'],r['error_b'])<=t['output_error'] and r['cosine']>=t['output_cosine'] for r in out):return False
|
||||
if c['kind'] in ('qk_head','qk_map'):
|
||||
p=c.get('probability_rows',[])
|
||||
if not p or not all(r['js']<=t['js'] and r['js_p95']<=t['js_p95'] and r['top5']>=t['top5'] for r in p):return False
|
||||
if c['kind']=='v_group':
|
||||
v=c.get('value_rows',[])
|
||||
if not v or not all(max(r['error_a'],r['error_b'])<=t['value_error'] and r['cosine']>=t['value_cosine'] and r['active'] for r in v):return False
|
||||
return True
|
||||
|
||||
|
||||
def preliminary_probability(c):
|
||||
t=TIERS['approximate'];r=c['probability_rows']
|
||||
return bool(r) and all(x['js']<=t['js'] and x['js_p95']<=t['js_p95'] and x['top5']>=t['top5'] for x in r)
|
||||
|
||||
|
||||
def preliminary_value(c):
|
||||
t=TIERS['approximate'];r=c['value_rows']
|
||||
return bool(r) and all(x['active'] and max(x['error_a'],x['error_b'])<=t['value_error'] and x['cosine']>=t['value_cosine'] for x in r)
|
||||
|
||||
|
||||
def load_profile(root,source,layer):
|
||||
with np.load(root/'cache'/f'calibration-source-{source}.npz') as data:
|
||||
prefix=f'layer{layer}_'
|
||||
return {key[len(prefix):]:data[key] for key in data.files if key.startswith(prefix)}
|
||||
|
||||
|
||||
def map_profile_values(p):return {k:v for k,v in p.items()}
|
||||
|
||||
|
||||
def new_bank():return {}
|
||||
|
||||
|
||||
def get_candidate(bank,kind,i,j,a=None,b=None,suffix='',perm=None):
|
||||
name=candidate_id(kind,i,j,a,b,suffix)
|
||||
if name not in bank:
|
||||
bank[name]={'id':name,'kind':kind,'i':i,'j':j,'a':a,'b':b,'perm':perm,
|
||||
'output_rows':[],'probability_rows':[],'value_rows':[]}
|
||||
return bank[name]
|
||||
|
||||
|
||||
def add_output(c,a,b,samples,source,target=None,floor_a=1e-6,floor_b=1e-6):
|
||||
c['output_rows'].extend(outrows(a,b,samples,{'candidate':c['id'],'source_input_layer':source,'target':target},floor_a,floor_b))
|
||||
|
||||
|
||||
def observe_pair(lm,bank,pi,pj,i,j,source,samples,diagnostics):
|
||||
fi,fj=head_floor(pi),head_floor(pj)
|
||||
whole=get_candidate(bank,'full_attention',i,j)
|
||||
add_output(whole,pi['output'],pj['output'],samples,source)
|
||||
hm=cross_head_output_metrics(pi['heads'][:,7:],pj['heads'][:,7:])
|
||||
qm=cross_head_output_metrics(pi['q'][:,7:],pj['q'][:,7:])
|
||||
repeats=lm.config['num_attention_heads']//lm.config['num_key_value_heads']
|
||||
ki,kj=np.repeat(pi['k'],repeats,axis=2),np.repeat(pj['k'],repeats,axis=2)
|
||||
km=cross_head_output_metrics(ki[:,7:],kj[:,7:])
|
||||
for a in range(pi['p'].shape[1]):
|
||||
for b in range(pj['p'].shape[1]):
|
||||
qk=get_candidate(bank,'qk_head',i,j,a,b);qkv=get_candidate(bank,'qkv_head',i,j,a,b)
|
||||
pm=distribution_metrics(pi['p'][:,a:a+1],pj['p'][:,b:b+1])
|
||||
rr=rows_for(pm,samples,{'candidate':qk['id'],'source_input_layer':source})
|
||||
qk['probability_rows'].extend(rr);qkv['probability_rows'].extend(rr)
|
||||
for n,s in enumerate(samples):
|
||||
rec={'candidate':qkv['id'],'source_input_layer':source,'sample':s['id'],**{key:float(v[n,a,b]) for key,v in hm.items()}}
|
||||
rec['active']=bool(rec['rms_a']>=fi[n] and rec['rms_b']>=fj[n]);qkv['output_rows'].append(rec)
|
||||
diagnostics.append({'i':i,'j':j,'head_i':a,'head_j':b,'source_input_layer':source,'sample':s['id'],
|
||||
'q_error':float(max(qm['error_a'][n,a,b],qm['error_b'][n,a,b])),
|
||||
'q_cosine':float(qm['cosine'][n,a,b]),'k_error':float(max(km['error_a'][n,a,b],km['error_b'][n,a,b])),
|
||||
'k_cosine':float(km['cosine'][n,a,b]),'qkv_head_output_error':max(rec['error_a'],rec['error_b'])})
|
||||
vm=cross_head_output_metrics(pi['v'][:,7:],pj['v'][:,7:])
|
||||
vfi=np.maximum(1e-6,.1*np.median(np.sqrt(np.mean(pi['v'][:,7:].astype(np.float64)**2,axis=(1,3))),axis=1))
|
||||
vfj=np.maximum(1e-6,.1*np.median(np.sqrt(np.mean(pj['v'][:,7:].astype(np.float64)**2,axis=(1,3))),axis=1))
|
||||
for a in range(pi['v'].shape[2]):
|
||||
for b in range(pj['v'].shape[2]):
|
||||
c=get_candidate(bank,'v_group',i,j,a,b)
|
||||
gram=value_gram_cka(pi['v'][:,7:,a],pj['v'][:,7:,b])
|
||||
for n,s in enumerate(samples):
|
||||
rec={'candidate':c['id'],'source_input_layer':source,'sample':s['id'],**{key:float(v[n,a,b]) for key,v in vm.items()},'centered_value_gram_cka':float(gram[n])}
|
||||
rec['active']=bool(rec['rms_a']>=vfi[n] and rec['rms_b']>=vfj[n]);c['value_rows'].append(rec)
|
||||
|
||||
|
||||
def probe_candidate(lm,c,pi,pj,source,samples):
|
||||
i,j=c['i'],c['j'];kind=c['kind'];fi,fj=head_floor(pi),head_floor(pj)
|
||||
if kind=='qk_map':
|
||||
perm=c['perm'];p=distribution_metrics(pi['p'],pj['p'][:,perm])
|
||||
c['probability_rows'].extend(rows_for(p,samples,{'candidate':c['id'],'source_input_layer':source}))
|
||||
changed_i=lm.transplant(pi,pj,i,kind,perm=perm)
|
||||
changed_j=lm.transplant(pj,pi,j,kind,perm=inverse(perm))
|
||||
add_output(c,pi['output'],changed_i,samples,source,i)
|
||||
add_output(c,pj['output'],changed_j,samples,source,j)
|
||||
elif kind=='qk_head':
|
||||
a,b=c['a'],c['b']
|
||||
changed_i=lm.head_transplant_output(pi,pj,i,a,b)
|
||||
changed_j=lm.head_transplant_output(pj,pi,j,b,a)
|
||||
add_output(c,pi['heads'][:,:,a],changed_i,samples,source,i,fi,fi)
|
||||
add_output(c,pj['heads'][:,:,b],changed_j,samples,source,j,fj,fj)
|
||||
elif kind=='v_group':
|
||||
a,b=c['a'],c['b'];rep=lm.config['num_attention_heads']//lm.config['num_key_value_heads']
|
||||
own_i=pi['heads'][:,:,a*rep:(a+1)*rep].sum(2);own_j=pj['heads'][:,:,b*rep:(b+1)*rep].sum(2)
|
||||
changed_i=lm.value_group_output(pi,pj,i,a,b);changed_j=lm.value_group_output(pj,pi,j,b,a)
|
||||
add_output(c,own_i,changed_i,samples,source,i,fi,fi);add_output(c,own_j,changed_j,samples,source,j,fj,fj)
|
||||
else:raise ValueError(kind)
|
||||
|
||||
|
||||
def summarize(c):
|
||||
out=c['output_rows'];p=c['probability_rows'];v=c['value_rows']
|
||||
m={'id':c['id'],'kind':c['kind'],'i':c['i'],'j':c['j'],'a':c['a'],'b':c['b'],'perm':c['perm'],
|
||||
'output_verified':bool(out),'output_observations':len(out),'probability_observations':len(p),'value_observations':len(v),
|
||||
'strict_pass':candidate_pass(c,'strict'),'approximate_pass':candidate_pass(c,'approximate')}
|
||||
m['worst_output_error']=max((max(r['error_a'],r['error_b']) for r in out),default=None)
|
||||
m['minimum_output_cosine']=min((r['cosine'] for r in out),default=None)
|
||||
m['worst_mean_js']=max((r['js'] for r in p),default=None)
|
||||
m['minimum_top5']=min((r['top5'] for r in p),default=None)
|
||||
m['worst_value_error']=max((max(r['error_a'],r['error_b']) for r in v),default=None)
|
||||
return m
|
||||
|
||||
|
||||
def controls_for_profiles(profiles,source,samples,rows,h):
|
||||
rng=np.random.default_rng(20261109+source)
|
||||
ra=(h@rng.standard_normal((h.shape[-1],256),dtype=np.float32))/np.float32(np.sqrt(h.shape[-1]))
|
||||
rb=(h@rng.standard_normal((h.shape[-1],256),dtype=np.float32))/np.float32(np.sqrt(h.shape[-1]))
|
||||
random_cka=value_gram_cka(ra[:,7:],rb[:,7:])
|
||||
for n,s in enumerate(samples):rows.append({'control':'value_gram_random_projection_pair','source_input_layer':source,'sample':s['id'],'gram_cka':float(random_cka[n])})
|
||||
for layer,p in profiles.items():
|
||||
for kv in range(p['v'].shape[2]):
|
||||
for label,other in [('value_gram_to_input',h),('value_gram_to_random_projection',ra)]:
|
||||
vcka=value_gram_cka(p['v'][:,7:,kv],other[:,7:])
|
||||
for n,s in enumerate(samples):rows.append({'control':label,'source_input_layer':source,'projection_layer':layer,'kv_head':kv,'sample':s['id'],'gram_cka':float(vcka[n])})
|
||||
probs=p['p'];uniform=np.zeros_like(probs);shuffled=probs.copy()
|
||||
for t in range(probs.shape[-1]):
|
||||
uniform[:,:,t,:t+1]=1/(t+1)
|
||||
for bi in range(probs.shape[0]):
|
||||
for head in range(probs.shape[1]):shuffled[bi,head,t,:t+1]=probs[bi,head,t,rng.permutation(t+1)]
|
||||
for name,other in [('identity',probs),('uniform_causal',uniform),('shuffled_legal_keys',shuffled)]:
|
||||
for head in range(probs.shape[1]):
|
||||
m=distribution_metrics(probs[:,head:head+1],other[:,head:head+1])
|
||||
rows.extend(rows_for(m,samples,{'control':name,'source_input_layer':source,'projection_layer':layer,'head':head}))
|
||||
|
||||
|
||||
def validate_selected(lm,selected,inputs,samples,indices):
|
||||
i,j=selected['i'],selected['j'];c={k:v for k,v in selected.items() if k not in ('output_rows','probability_rows','value_rows')}
|
||||
c.update(output_rows=[],probability_rows=[],value_rows=[])
|
||||
for source in (i,j):
|
||||
h=inputs[f'attn_{source}'][indices];pi,pj=lm.parts(h,i),lm.parts(h,j)
|
||||
temporary={};diag=[];observe_pair(lm,temporary,pi,pj,i,j,source,samples,diag)
|
||||
if c['kind'] in ('full_attention','qkv_head','qk_head','v_group'):
|
||||
original=temporary[candidate_id(c['kind'],i,j,c['a'],c['b'])]
|
||||
c['probability_rows'].extend(original['probability_rows']);c['value_rows'].extend(original['value_rows'])
|
||||
if c['kind'] in ('full_attention','qkv_head'):c['output_rows'].extend(original['output_rows'])
|
||||
if c['kind'] in ('qk_map','qk_head','v_group'):probe_candidate(lm,c,pi,pj,source,samples)
|
||||
return c
|
||||
|
||||
|
||||
def main():
|
||||
p=argparse.ArgumentParser();p.add_argument('--source',required=True,type=Path);p.add_argument('--root',required=True,type=Path);p.add_argument('--seconds',type=int,default=7200)
|
||||
args=p.parse_args();source=args.source.resolve();root=args.root.resolve();root.mkdir(parents=True,exist_ok=True);(root/'cache').mkdir(exist_ok=True)
|
||||
assert not (root/'protocol.json').exists(),'Fresh experiment directory required; do not overwrite evidence'
|
||||
protocol={**PROTOCOL,'registered_unix':time.time(),'source_run':str(source),'seconds':args.seconds,
|
||||
'fresh_sample_sha256':hashlib.sha256(Path(__file__).with_name('fresh_samples.json').read_bytes()).hexdigest()}
|
||||
write_json(root/'protocol.json',protocol)
|
||||
write_json(root/'retention-policy.json',{'automatic_cleanup':False,'weights_and_caches':'retain_until_explicit_user_deletion'})
|
||||
start=time.monotonic();status={'state':'running','phase':'calibration_profiles','started_unix':time.time()}
|
||||
bank={};diagnostics=[];controls=[]
|
||||
try:
|
||||
manifest=json.loads((source/'manifest.json').read_text());assert manifest['revision']=='15852e8c16360a2fea060d615a32b45270f8a8fc'
|
||||
reference=json.loads((source/'reference-check.json').read_text());assert reference['passed']
|
||||
assert reference['implementation_sha256']==hashlib.sha256(Path('functional/model.py').read_bytes()).hexdigest()
|
||||
config=json.loads((source/'model_config.json').read_text());lm=InterventionLM(source/'weights',config,seconds=args.seconds-180)
|
||||
assert [i for i,t in enumerate(lm.config['layer_types']) if t=='full_attention']==LAYERS
|
||||
samples=json.loads((source/'samples-tokenized.json').read_text())['samples'];cal=[s for s in samples if s['split']=='calibration'];valid=[s for s in samples if s['split']=='heldout']
|
||||
ci=np.array([i for i,s in enumerate(samples) if s['split']=='calibration']);vi=np.array([i for i,s in enumerate(samples) if s['split']=='heldout'])
|
||||
write_json(root/'source-samples.json',{'samples':samples,'note':'old validation samples used in earlier studies; fresh test separately fixed'})
|
||||
inputs=np.load(source/'baseline-activations.npz')
|
||||
write_json(root/'source-provenance.json',{'revision':manifest['revision'],'weights_manifest':str(source/'manifest.json'),
|
||||
'baseline_activation_sha256':hashlib.sha256((source/'baseline-activations.npz').read_bytes()).hexdigest(),
|
||||
'source_hashes':{str(f):hashlib.sha256(f.read_bytes()).hexdigest() for f in sorted(Path('crossqkv').glob('*')) if f.is_file()}})
|
||||
for src in LAYERS:
|
||||
lm.guard();h=inputs[f'attn_{src}'][ci];profiles={layer:lm.parts(h,layer) for layer in LAYERS}
|
||||
np.savez(root/'cache'/f'calibration-source-{src}.npz',**{f'layer{layer}_{key}':value for layer,profile in profiles.items() for key,value in profile.items()})
|
||||
controls_for_profiles(profiles,src,cal,controls,h)
|
||||
for i,j in PAIRS:
|
||||
if src in (i,j):observe_pair(lm,bank,profiles[i],profiles[j],i,j,src,cal,diagnostics)
|
||||
del profiles
|
||||
status.update(last_source_layer=src,elapsed_seconds=time.monotonic()-start);write_json(root/'status.json',status)
|
||||
save_csv(root/'raw-qk-diagnostics.csv',diagnostics);save_csv(root/'controls.csv',controls)
|
||||
print(json.dumps(status),flush=True)
|
||||
status['phase']='discrete_head_matching_and_output_probes';write_json(root/'status.json',status)
|
||||
mapping_rows=[]
|
||||
qk_pre=[c for c in bank.values() if c['kind']=='qk_head' and preliminary_probability(c)]
|
||||
qk_probe=sorted(qk_pre,key=lambda c:(max(r['js'] for r in c['probability_rows']),c['id']))[:32]
|
||||
v_pre=[c for c in bank.values() if c['kind']=='v_group' and preliminary_value(c)]
|
||||
v_probe=sorted(v_pre,key=lambda c:(max(max(r['error_a'],r['error_b']) for r in c['value_rows']),c['id']))[:16]
|
||||
for i,j in PAIRS:
|
||||
cost=np.empty((8,8),dtype=np.float64)
|
||||
for a in range(8):
|
||||
for b in range(8):cost[a,b]=np.mean([r['js'] for r in bank[candidate_id('qk_head',i,j,a,b)]['probability_rows']])
|
||||
perm=assignment(cost)
|
||||
for a in range(8):
|
||||
for b in range(8):mapping_rows.append({'i':i,'j':j,'head_i':a,'head_j':b,'calibration_mean_js':float(cost[a,b]),'chosen':perm[a]==b})
|
||||
maps=[get_candidate(bank,'qk_map',i,j,suffix='_identity',perm=list(range(8))),get_candidate(bank,'qk_map',i,j,suffix='_matched',perm=perm)]
|
||||
for src in (i,j):
|
||||
pi,pj=load_profile(root,src,i),load_profile(root,src,j)
|
||||
for c in [*maps,*[c for c in qk_probe+v_probe if c['i']==i and c['j']==j]]:
|
||||
lm.guard();probe_candidate(lm,c,pi,pj,src,cal)
|
||||
status.update(last_pair=[i,j],elapsed_seconds=time.monotonic()-start);write_json(root/'status.json',status)
|
||||
save_csv(root/'head-assignment-costs.csv',mapping_rows)
|
||||
summaries=[summarize(c) for c in bank.values()];write_json(root/'calibration-summary.json',summaries)
|
||||
save_csv(root/'calibration-output-metrics.csv',[r for c in bank.values() for r in c['output_rows']])
|
||||
save_csv(root/'calibration-probability-metrics.csv',[r for c in bank.values() if c['kind']!='qkv_head' for r in c['probability_rows']])
|
||||
save_csv(root/'calibration-value-metrics.csv',[r for c in bank.values() for r in c['value_rows']])
|
||||
eligible=[c for c in summaries if c['strict_pass'] or c['approximate_pass']]
|
||||
eligible.sort(key=lambda c:(not c['strict_pass'],TYPE_ORDER[c['kind']],c['worst_output_error'],c['worst_mean_js'] or 0,c['id']))
|
||||
selected=eligible[:1]
|
||||
write_json(root/'selection.json',{'selected':selected,'qk_head_prequalified':len(qk_pre),'qk_head_output_verified':len(qk_probe),
|
||||
'v_group_prequalified':len(v_pre),'v_group_output_verified':len(v_probe),'selection_uses_validation':False})
|
||||
status['selected']=selected
|
||||
if selected:
|
||||
c=selected[0];status['phase']='old_validation';write_json(root/'status.json',status)
|
||||
checked=validate_selected(lm,c,inputs,valid,vi);tier='strict' if c['strict_pass'] else 'approximate'
|
||||
write_json(root/'validation-summary.json',summarize(checked));save_csv(root/'validation-output-metrics.csv',checked['output_rows']);save_csv(root/'validation-probability-metrics.csv',checked['probability_rows']);save_csv(root/'validation-value-metrics.csv',checked['value_rows'])
|
||||
passed=candidate_pass(checked,tier);status['validation_pass']=passed
|
||||
if passed:
|
||||
from .fresh import run_fresh
|
||||
run_fresh(source,root,c,tier,lm,status)
|
||||
status.update(state='completed',phase='done',elapsed_seconds=time.monotonic()-start,peak_rss_kib=resource.getrusage(resource.RUSAGE_SELF).ru_maxrss,
|
||||
candidate_counts={kind:sum(c['kind']==kind for c in summaries) for kind in TYPE_ORDER},
|
||||
passing_counts={kind:sum(c['kind']==kind and c['approximate_pass'] for c in summaries) for kind in TYPE_ORDER},
|
||||
original_11_and_norm_positive_not_used_as_qkv_evidence=True)
|
||||
except Exception as exc:
|
||||
status.update(state='stopped',error_type=type(exc).__name__,error=str(exc),elapsed_seconds=time.monotonic()-start,peak_rss_kib=resource.getrusage(resource.RUSAGE_SELF).ru_maxrss)
|
||||
(root/'error.txt').write_text(traceback.format_exc());raise
|
||||
finally:write_json(root/'status.json',status)
|
||||
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,75 @@
|
||||
# CPU 先导:先找值得继续验证的信号
|
||||
|
||||
更新:2026-11-09。目的不是在 CPU 上完成广泛能力及高吞吐评测,
|
||||
而是低成本判断实际输入分布上是否存在值得 GPU 进一步验证的复用候选。
|
||||
全程不训练,不求 R。
|
||||
|
||||
## 已执行的容量检查
|
||||
|
||||
服务器 8.211.154.134:2 核 x86_64,约 1.7 GiB RAM;检查时可用约 892 MiB,
|
||||
swap 已用约 2.2 GiB,根盘可用约 73 GiB。共享主机快照不是实验独占资源。
|
||||
NumPy 2.5.4,OPENBLAS_NUM_THREADS=1、OMP_NUM_THREADS=1,远程 timeout 60 秒。
|
||||
随机种子 42、float32、X 为 32×2048;每种 W 随机生成后连续执行 4 次 X@W,
|
||||
第 1 次作为冷调用,其余 3 次单独报告。随机输入仅用于容量测量。
|
||||
|
||||
| W 形状 | W 大小 | 首次乘法 | 后续三次乘法(秒) |
|
||||
|---|---|---|---|
|
||||
| 2048×2048 | 16 MiB | 0.027816 | 0.027035、0.027207、0.025878 |
|
||||
| 2048×6144 | 48 MiB | 0.086039 | 0.082543、0.110806、0.075336 |
|
||||
|
||||
输出均为有限数,进程峰值 RSS 88,220 KiB(约 86.2 MiB)。
|
||||
耗时只包含矩阵乘法,不包含权重生成、磁盘读取、BF16 转换、模型其余操作及框架开销。
|
||||
它支持继续评估小批量 CPU 投影计算,不足以外推完整模型时延或宣称发现可复用性。
|
||||
没有模型权重参与,没有编译、下载或模型前向。
|
||||
|
||||
## 执行路线与证据等级
|
||||
|
||||
1. **实现与资源可行性**:核对固定 revision 的位置编码、归一化、GQA 和门控,
|
||||
使用受信任的算子实现及小数组正确性检查;模型仓库代码不直接执行。
|
||||
先确认首个短样本能以受控内存运行。不用不正确的“简化 Qwen”产生看似有价值的结果。
|
||||
2. **真实分布上的候选**:batch=1、32 tokens,校准和留出各 4 段真实文本。
|
||||
按顺序加载、计算、释放每层,保留很小的入口激活;探测预先固定的 3 个 full-attention 层对、2 个 MLP 对,以及 6 个 full-attention 层内的 Q/K 归一化模块对,共 11 对。
|
||||
每对在两个来源层的隐藏输入上分别测试,记录 JS、top-k、完整分支输出误差。
|
||||
3. **最小干预证据**:最多对 1 个候选做单处替换;冻结参数,在留出文本上看输出变化,
|
||||
条件允许时流式计算 token NLL。词表输出头分块归约 logsumexp,避免一次物化大 logits。
|
||||
若没有正确且可承受的干预实现,就止于“候选”,不称为验证成功。
|
||||
4. **GPU 细化**:候选通过才扩展长度、样本、线性注意力、多处共享及质量—成本曲线。
|
||||
|
||||
先导主证据来自真实文本产生的隐藏状态,而不是高维空间上的均匀随机向量。
|
||||
直接按真实样本出现频率聚合,就能近似按目标使用分布加权,不先引入昂贵的高维密度估计。
|
||||
保留逐文本及尾部误差,少量低频但关键用例不被平均值掩盖。
|
||||
|
||||
## CPU 内存与停止约束
|
||||
|
||||
- 不同时加载完整模型的 BF16/FP32 副本;模型原始权重约 4.55 GB,大于当前 RAM。
|
||||
- 单层/子矩阵流式读取,及时解除映射或释放页面;先设本进程 RSS 512 MiB 软停止线,
|
||||
系统可用内存低于 256 MiB 或持续 swap 活动就停止并保存进度,不改全局 swap 配置。
|
||||
软停止线不替代实际峰值检测,分配前须计算临时内存。
|
||||
- 先实现最小正确路径;线性状态递推、长上下文和生成速度不塞入首轮。
|
||||
- 总计算 120 分钟,阶段预算见功能方案;无候选、超时或数值/实现核对失败均停止。
|
||||
- 容量探测已完成,真实模型阶段尚未执行。后续需要恢复本实验自有的固定 revision 权重,
|
||||
权重持续保留,只有用户明确要求删除时才清理;不增加模型规模或自动租用 GPU。
|
||||
|
||||
CPU 上“找到输出接近的候选”能支持继续投入 GPU;
|
||||
“在指定小样本中单处复用保持质量且节省明确资源”才是初步正例。
|
||||
两者都不能替代大规模能力验证。CPU 时延不外推为 GPU 加速率。
|
||||
|
||||
## 当前权重保留规则
|
||||
|
||||
用户于 2026-11-09 明确要求:本轮模型权重保留,只有用户明确要求删除时才删除。
|
||||
试验完成、超时、失败、归档或结果校验均不得自动触发清理。该要求覆盖先前逐模型分析后删除的约定。
|
||||
上一轮已删除的历史记录保留;本轮恢复下载的权重受新规则保护。
|
||||
|
||||
## 本轮执行补充
|
||||
|
||||
实际实验分支为 `codex/cpu-reuse-experiment`。在读取真实权重/功能指标前,新增固定的
|
||||
6 个层内 Q/K RMSNorm 增益共享候选,以检查较小粒度的功能复用;不是看过留出结果后扩充搜索。
|
||||
最多选择一个候选,必须分别验证两个替换方向;完整模块失败不会被小模块通过所掩盖。
|
||||
归一化共享最多节省少量字节,不预设能显著减少计算。
|
||||
采集真实隐藏输入必须正确执行前置线性注意力;它不进入首轮复用候选,但其递推计算纳入官方参考对照。
|
||||
|
||||
## 用户最新目标修订
|
||||
|
||||
不再以成本/加速作为成功条件;主问题是现有学习部分的功能重复。
|
||||
新增一次固定四层的 MLP 神经元函数搜索、独立留出和双向替换,
|
||||
见 [功能重复判据](LEARNED_FUNCTION_REPETITION.md)。解析增益合并单独归类,不冒充学习部分重复。
|
||||
@@ -0,0 +1,86 @@
|
||||
# CPU 先导结论:存在已训练归一化模块的相似功能重复
|
||||
|
||||
本轮主目标按用户最新要求为功能重复的存在性,不以成本/加速为成功条件。
|
||||
全部计算在 2 核、约 1.7 GiB RAM 的 CPU 服务器完成,无训练、缩放拟合或旋转搜索。
|
||||
层编号均为从 0 开始的索引。
|
||||
|
||||
## 一个具体正例
|
||||
|
||||
固定官方 Qwen3.5-2B checkpoint 的两个独立已训练模块:
|
||||
|
||||
- `model.language_model.layers.12.input_layernorm`
|
||||
- `model.language_model.layers.13.input_layernorm`
|
||||
|
||||
二者均为 2048→2048 的 RMSNorm,eps=1e-6,使用训练后的有效增益 gamma=1+weight。
|
||||
增益标准差分别约 0.1901、0.1685,最小值约 0.5547、0.5703;不是零函数或恒定增益。
|
||||
|
||||
| 证据 | 结果 |
|
||||
|---|---|
|
||||
| 全输入域数学界(相对于层12输出) | 输出相对 L2 误差≤10.1852% |
|
||||
| 反方向全输入域数学界(相对于层13输出) | 输出相对 L2 误差≤11.3402% |
|
||||
| 4 条留出文本、两个来源层输入的最坏双向误差 | 4.5285% |
|
||||
| 相同留出比较中的最低余弦 | 0.999487 |
|
||||
| 层12使用层13,留出平均 token NLL 增量 | +0.0000325651 nat/token |
|
||||
| 层13使用层12,留出平均 token NLL 增量 | −0.0003639730 nat/token |
|
||||
|
||||
两方向逐文本 NLL 增量 P95 分别约 0.00098464、0.00070972,均通过原定小样本质量门槛。
|
||||
所有数值来自保存的逐文本、逐 token 原始数据。留出为 4 条不同的实验原创短文本,
|
||||
每条截取 32 tokens;校准另有 4 条。它们不是广泛任务能力评测。
|
||||
|
||||
## 为什么不只是“权重看起来接近”
|
||||
|
||||
对相同 eps 和维数,f_a(x)=diag(a)x/sqrt(mean(x²)+eps)。任意非零实数输入满足:
|
||||
|
||||
||f_a(x)-f_b(x)|| / ||f_a(x)|| ≤ max_j |a_j-b_j|/|a_j|。
|
||||
|
||||
共同归一化分母消去;坐标基向量可达到最大值,因而该界是精确的最坏情况。
|
||||
反向分母换成 |b_j|。本轮先穷举 24 层同角色 input/post-attention norm 的 552 对,
|
||||
仅按最小双向最坏界选定一对,未使用任何输入样本作选择。≤20% 界有 18 对,≤10% 无配对;
|
||||
本轮只验证最优的一对,不根据留出结果换选。
|
||||
|
||||
这个确定性界直接给出两个已训练函数整体接近的证据。真实输入上的误差更小,
|
||||
与只关心目标输入分布的目标一致。浮点实测与实数数学界分别报告。
|
||||
重新采集输入时,未修改的完整前向逐 bit 复现了最初保存的基线输出。
|
||||
输入探测分别使用两个来源层,对相同输入同时应用两套增益,避免比较不同输入的输出。
|
||||
从已加增益的激活还原共同归一化输入时有额外 FP32 舍入,未将其称为 bitwise 等价。
|
||||
|
||||
结论限定为两个已有 RMSNorm 模块实现相似映射;不把共同包含 RMS 操作本身当作证据。
|
||||
二者学习增益不同且非恒定,输出非零。完整 attention/MLP 主体的重复仍未得到正例。
|
||||
|
||||
## 已完成的阴性搜索
|
||||
|
||||
1. 原定 3 对完整 attention、2 对完整 MLP、6 对层内 Q/K norm 均未通过预设门槛。
|
||||
attention 检索分布 JS 范围约 0.145~0.318,也未通过 0.01 的门槛;
|
||||
未进行头排列、连续坐标变换或训练对齐,不能外推为不存在其他可复用表示。
|
||||
2. 在层 5、6、20、21 各 6144 个 MLP scalar 子函数中,排除低活跃和近常量单元后,
|
||||
共精确筛查 7,450,139 个唯一同层配对。要求双向相对误差≤20%、余弦≥0.98、
|
||||
Pearson≥0.95,无通过者。没有针对留出集不断更换候选或放宽门槛。
|
||||
3. 这些结果只覆盖声明的模块、输入和阈值,不否定其他粒度/角色的功能重复。
|
||||
|
||||
## 单独归类的解析恒等式
|
||||
|
||||
Q/K 未参与 RoPE 的 192 个维度,其增益只以乘积影响点积,可合并到任一侧。
|
||||
96 组真实输入检查最大分数差 3.8147e-6、最大概率差 8.0466e-7;固定第3层两个方向
|
||||
留出 NLL 变化接近 FP32 舍入。详见 [推导](QK_GAIN_FACTORIZATION.md)。
|
||||
这是重参数化等价,不作为“两个已有学习部分功能重复”的正例。先前性能测试原始数据
|
||||
保留用于溯源,成本不属于当前结论的成功条件。
|
||||
|
||||
## 正确性、资源与保留规则
|
||||
|
||||
- 固定模型 revision:15852e8c16360a2fea060d615a32b45270f8a8fc,原始权重 SHA256 校验通过。
|
||||
- 流式读取原始 BF16,逐块转为 FP32,不同时载入全模型。
|
||||
- 与固定 Transformers 官方定义比较的 22 项检查通过,最大差约 5.4e-7。
|
||||
- 保留早期参考适配器错误、临时分区空间不足及两次过于保守的换页停止记录。
|
||||
只归还自身临时堆页,未改变系统 swap 设置;后续有界前向完成。
|
||||
- 计算预算跨重启累计,初始保守计入 180 秒先前检查预留;具体时长及峰值见归档日志。
|
||||
- 当前权重按用户最新指令持续保留,只在用户明确要求删除时清理。
|
||||
|
||||
原始表、样本/token IDs、激活、NLL、证明界与图表见 `results/cpu-qwen35-2b/`。
|
||||
后续 GPU 阶段可增加长度、语言和任务来验证正例的稳健性;本轮不自动启动新模型或扩展搜索。
|
||||
|
||||
## Git 交付与保留缓存
|
||||
|
||||
全部图表源 CSV/JSON、逐 token NLL、证明界和日志已交付。三份较大的中间激活缓存
|
||||
保留在原服务器及本地完整归档,路径、大小、SHA256 见结果目录的 external-activation-cache.json。
|
||||
其完整原实验校验表为 checksums-full-server.json;Git交付包校验表为 checksums-functional.json。
|
||||
没有删除模型或缓存;这些缓存可由固定代码和样本重新生成,也可从服务器按哈希取回。
|
||||
@@ -0,0 +1,90 @@
|
||||
# 跨层 QKV:主检验阴性,发现条件性的自 token 检索重合
|
||||
|
||||
全部在 CPU 上运行,冻结 Qwen3.5-2B 权重,无训练、连续 R 拟合或缩放拟合。
|
||||
层索引从0开始。成本不作为判据;归一化相似不作为本轮 QKV 正例。
|
||||
|
||||
## 结论分级
|
||||
|
||||
**主协议没有候选通过全部门槛。** 覆盖六个 full-attention 层的全部15个跨层配对,
|
||||
同时测试两个来源层的真实输入,不能将该结论扩大为模型不存在任何功能重复。
|
||||
|
||||
**探索性补充有一条具体行为观察,但严格新测试仍未全部通过:**
|
||||
把层23的同一隐藏输入 H23 分别交给层3头7、层23头6的 QK 检索函数,
|
||||
它们主要都读取当前 token;在4条新文本上,query位置7~31的 argmax 完全相同。
|
||||
这不是同层 Q/K/V 互换,不是两个完整 attention 头等价,也没有证明 V 相似。
|
||||
|
||||
| 新文本证据(逐文本平均或最坏值) | 结果 |
|
||||
|---|---:|
|
||||
| 层23头6分配给当前token的平均概率,四文本最小值 | 95.71% |
|
||||
| 层3头7在同一H23上分配给当前token的平均概率,最小值 | 92.07% |
|
||||
| 两者最高概率key一致率 | 100% |
|
||||
| 两套top5交集覆盖各自概率质量的最小值 | 97.97% |
|
||||
| 分布总变差的最大文本平均值 | 6.96% |
|
||||
| 保留层23 V/gate/O后的最坏双向头读出相对误差 | 3.82% |
|
||||
| 最坏文本平均JS | 0.018921 |
|
||||
|
||||
新英文文本 fresh-en-2 超过补充协议的三条严格线:平均 JS 0.018921>0.01、
|
||||
行 JS P95 0.072943>0.05、行总变差 P95 0.238731>0.20。
|
||||
因此补充试验的 `fresh_function_pass=false`,**没有重标为完全通过,也未运行替换后的全LM NLL**。
|
||||
已计算新文本原模型基线,不能把该基线文件当作替换实验结果。
|
||||
可支持的观察是特定输入条件下的主要检索行为和实际读出接近;严格可互换尚未确认。
|
||||
|
||||
## 主协议实际覆盖
|
||||
|
||||
| 对象 | 初筛/直接比较数量 | 后续输出核验 |
|
||||
|---|---:|---|
|
||||
| 完整 attention 分支 | 15 | 全部测量 |
|
||||
| QKV+gate/O 的完整头函数 | 960 | 全部测量,输出在共同residual坐标中 |
|
||||
| 全8头P图映射(原序/冻结匹配) | 30 | 全部保留接收层V/gate/O做替换读出 |
|
||||
| 单头联合QK检索 | 960 | 无候选通过初筛,主协议未进入单头读出 |
|
||||
| 跨层KV组的直接V函数 | 60 | 无候选通过原始V初筛,未进入V组替换 |
|
||||
|
||||
主协议最接近的若干项,指标来自不同候选,不能拼接成同一个正例:
|
||||
|
||||
- QK两来源检索:层7头0↔层11头6,最坏文本平均JS约0.08609,最低top5约70.4%。
|
||||
- 完整头输出:同一对头的最坏方向相对误差约134.4%。检索接近不意味着V/O组成的输出接近。
|
||||
- 直接V函数:层7 KV头0↔层11 KV头1,最坏方向误差约150.3%。
|
||||
- 完整分支:层7↔层11的最坏方向误差约185.3%。
|
||||
- 保留目标V/O的完整P图映射:最小最坏输出误差约70.3%(层15↔19、冻结头匹配)。
|
||||
|
||||
V token-Gram CKA另与输入Gram和两个独立高斯投影比较。共同输入几何可以造成高Gram相似,
|
||||
不能据此宣布V可替换;完整头输出检验已经包含各自的O/gate,避免仅因内部坐标差异就漏掉输出等价。
|
||||
|
||||
## 为什么增加单独的条件性审计
|
||||
|
||||
主协议要求两个来源层的输入分布均通过,并把无权重top5重合率作为硬门槛。
|
||||
复核全部已保存的单来源指标后,固定了唯一一对:层3头7与层23头6,条件为H23。
|
||||
它在校准集的JS很小,却因概率很低的尾部排名不同而top5重合低。
|
||||
|
||||
无权重top-k在近似并列处不连续:可以构造总变差趋近0而top5集合仍不同的例子,
|
||||
本轮已加入独立回归测试。补充协议在观察该候选保留接收端V/O的读出、旧验证或新测试之前固定。
|
||||
它改为单一H23输入条件,将top5作为诊断,并固定概率质量/实际读出/活跃度约束;
|
||||
不换候选、不重新拟合参数。
|
||||
它使用4条原验证文本,再使用本轮事先冻结的4条新文本,保留全部未通过项。
|
||||
这是探索性补充,不改写主协议,不是事后声称原协议成功。
|
||||
|
||||
原主检验的图与条件性图使用不同输入集合:主检验取两个来源分布的最坏情况;
|
||||
条件性图只研究H23。条件性相似不能推广到H3,也不能把早层在H3上已算好的P直接当作P_3(H23)。
|
||||
实际H3缓存与H23接收端的差异另有概率/读出诊断,未做失败后追加的全模型替换。
|
||||
|
||||
## 图表阅读
|
||||
|
||||
- `cross_layer_qk_divergence`:15对层的JS,左为各配对最佳单头,中为原头序,右为冻结双射;
|
||||
颜色越浅分布越近,斜线对角不作经验数据。全部仅为校准结果。
|
||||
- `cross_layer_output_errors`:各类输出误差。QK单头和V组未通过初筛时,不画成零误差,而标未输出核验。
|
||||
- `value_geometry_and_direct_error`:V Gram几何与直接V误差,灰区为随机投影对照,仅诊断。
|
||||
- `conditional_probability_mass_and_readout`:同一条件候选在校准、旧验证和新测试上的top5、质量重合与实际读出差异。
|
||||
- `conditional_cross_layer_probability_maps`:一条新文本的具体P图;层3在H23上的结果与层3在实际H3上的结果分开。
|
||||
|
||||
## 复现、预算与保留
|
||||
|
||||
20项完整远程测试通过。主计算约74.8秒,补充条件试验约105.3秒;峰值进程RSS分别约168/174 MiB。
|
||||
另有出图、审计和测试,仍在两小时预算内。早期对照生成器的变量遮蔽错误及10.3秒失败记录保留,
|
||||
修复后在新目录按相同配对和门槛重跑;没有删除原缓存。
|
||||
|
||||
模型revision 15852e8c16360a2fea060d615a32b45270f8a8fc,与此前官方参考校验的流式CPU实现一致。
|
||||
全部权重和大型profile缓存继续保留;Git只上传代码、全部图表源表、选定的新文本概率矩阵、
|
||||
日志、协议和缓存位置/SHA。没有新模型下载或自动清理。
|
||||
|
||||
后续若继续,应重新预注册“单向输入条件”和概率质量/读出判据,使用新文本及更长上下文;
|
||||
当前结果不能证明跨上下文稳定,也不应把归一化正例或代数重参数化当作QKV主体验证。
|
||||
@@ -0,0 +1,59 @@
|
||||
# 跨层 QKV 功能重复:CPU 预注册方案
|
||||
|
||||
目标:两个不同层中的学习部分是否实现相似检索、内容变换或完整头函数。不以成本为条件。
|
||||
冻结 Qwen3.5-2B 权重;不训练、不拟合连续 R 或缩放,不使用 GPU。权重与缓存持续保留。
|
||||
|
||||
## 配对与数据
|
||||
|
||||
仅研究完整注意力层索引 3/7/11/15/19/23(从0开始)的全部15个跨层配对。
|
||||
18个线性注意力层不套用 softmax 概率指标,不在本轮自动扩大搜索。
|
||||
复用已采集的同一隐藏输入,四条原校准文本选对,四条旧留出文本作验证;
|
||||
后者曾用于其他试验,不称完全未见的最终测试。另固定4条新文本作新测试,32 tokens,中文英文各半。
|
||||
仅对冻结候选通过验证时采集新测试输入,不根据测试结果重选。
|
||||
|
||||
## 拆开回答的问题
|
||||
|
||||
- QK:比较实际 Q/K norm、部分 RoPE 和 GQA 后产生的逐头检索概率,保留接收层 V/gate/O 时替换检索分布。
|
||||
- V:比较跨层两个 KV 头的 V(H);保留接收层 P/gate/O,再替换一个 KV 组,覆盖该组4个Q头。
|
||||
token Gram/CKA仅用于诊断坐标表示问题,不能代替实际输出替换;另与输入Gram、固定seed的两个高斯投影Gram比较,以控制共同输入结构造成的高相似。
|
||||
- 完整头:比较 P V、gate、O 共同形成的每个头在 residual 坐标中的输出函数。
|
||||
它不要求内部 V 坐标一致,能够避免仅比较内部元素漏掉表示差异。
|
||||
- 完整分支:比较全部8个头组成的完整 attention 输出。
|
||||
|
||||
因此原始 Q/K/V 输出相似、QK 检索相似、V 可替换、完整头可替换分别报告,归一化单独相似不作主正例。
|
||||
同一层 Q、K、V 互相比较不进入主实验。
|
||||
|
||||
## 头匹配与位置对照
|
||||
|
||||
15对层各在校准集选择一次最小平均JS的8头双射(精确离散DP,不求连续R)。
|
||||
该排列只用于给接收Q头注入 donor P;接收层V/gate/O保持顺序,不宣称任意GQA权重排列合法。
|
||||
全部960个跨层Q头配对、60个跨层KV头配对和960个完整头函数配对均作初筛。
|
||||
完整P映射考察原顺序及冻结排列,共30项;完整分支共15项。
|
||||
|
||||
主指标只用 query 位置7~31,避免第一行概率必为1、早期合法key过少的平凡相似。
|
||||
记录均匀因果P、固定seed的合法key打乱、去首key后重归一化、首key/self/最近3key质量。
|
||||
首key或局部位置主导的真实重复单独标为位置型,不冒充内容语义重复。
|
||||
|
||||
## 固定门槛与预算
|
||||
|
||||
两档门槛在读取新跨层指标前固定,不依据结果放宽:
|
||||
|
||||
| 档位 | 输出双向相对L2 | 输出余弦 | 每文本平均JS | JS行P95 | top5重合 |
|
||||
|---|---:|---:|---:|---:|---:|
|
||||
| strict | ≤0.05 | ≥0.99 | ≤0.01 | ≤0.05 | ≥0.8 |
|
||||
| approximate | ≤0.20 | ≥0.98 | ≤0.05 | ≤0.15 | ≥0.6 |
|
||||
|
||||
V另需同档原始V直接误差/余弦;QK候选另需同档检索门槛。
|
||||
每个来源层、每条文本、两个替换方向分别检查,输出不含共同 residual 输入。
|
||||
头输出需达到该层头RMS中位数的10%,且绝对RMS≥1e-6,避免近零头制造假正例。
|
||||
|
||||
QK单头最多精确核验32个初筛通过者,V组最多16个;完整P的30个输出均核验。
|
||||
只按校准选定1个最终候选:先strict,再approximate;同档优先完整分支、完整头、完整P、QK单头、V组。
|
||||
之后按最坏输出误差/JS/ID排序。不按验证或新测试结果换选。
|
||||
|
||||
同输入函数验证通过后,做两个方向的真实网络干预,计算新测试token NLL(均值增量≤0.01、
|
||||
逐文本增量P95≤0.05)。另外只做一次时间顺序合法的“早层实际缓存→晚层”复用,单独报告;
|
||||
同输入函数可互换不代表来自不同实际输入的已计算激活可以直接复制。
|
||||
|
||||
CPU总计算限两小时,保留阶段覆盖率和停止原因;没有候选即停止,不自动扩模型、层数或阈值。
|
||||
较大的中间profile缓存在服务器,Git交付全部图表源表、协议、NLL和缓存位置/SHA。
|
||||
@@ -0,0 +1,57 @@
|
||||
# 研究图册:Qwen3.5-2B 参数几何与头贡献
|
||||
|
||||
这份图册与原始PNG/PDF/SVG一同保存在Git中。头贡献基线图片已随PR #6合并到main;其固定提交链接不依赖聊天附件或本机路径。后续实验分别列于下方,可通过图册所在的Git提交查看对应版本。
|
||||
|
||||
原始结果提交:`098ea8650c7d0e06339c66e3a0509e38b85b3d7d`;模型revision:`15852e8c16360a2fea060d615a32b45270f8a8fc`。
|
||||
|
||||
| 基线指标(逐观测中位数) | QKV复合变化 | √JS检索距离 |
|
||||
|---|---:|---:|
|
||||
| 有效头数÷总头数 | 76.0% | 55.8% |
|
||||
| 最强25%头的影响份额 | 44.6% | 56.5% |
|
||||
| 最弱25%头的影响份额 | 10.6% | 6.8% |
|
||||
|
||||
**√JS有效头数:8.931822/16;P10=5.784081,P90=11.672658。** 这是消融影响的集中程度,不是注意力概率质量。原基线为4条文本各32 token,普通注意力接收层的query7–31。
|
||||
|
||||
## 连续分布与尺度敏感性
|
||||
|
||||

|
||||
|
||||
[PNG固定提交副本](https://gitea.entroia.com/Zayden-Jung/exp-model-parameter-similarity/raw/commit/098ea8650c7d0e06339c66e3a0509e38b85b3d7d/results/head-dominance/figures/retrieval_scale_sensitivity.png)
|
||||
[PDF固定提交副本](https://gitea.entroia.com/Zayden-Jung/exp-model-parameter-similarity/raw/commit/098ea8650c7d0e06339c66e3a0509e38b85b3d7d/results/head-dominance/figures/retrieval_scale_sensitivity.pdf)
|
||||
[SVG固定提交副本](https://gitea.entroia.com/Zayden-Jung/exp-model-parameter-similarity/raw/commit/098ea8650c7d0e06339c66e3a0509e38b85b3d7d/results/head-dominance/figures/retrieval_scale_sensitivity.svg)
|
||||
|
||||
## 联合强弱头消融
|
||||
|
||||

|
||||
|
||||
[PNG固定提交副本](https://gitea.entroia.com/Zayden-Jung/exp-model-parameter-similarity/raw/commit/098ea8650c7d0e06339c66e3a0509e38b85b3d7d/results/head-dominance/figures/joint_removal_effects.png)
|
||||
[PDF固定提交副本](https://gitea.entroia.com/Zayden-Jung/exp-model-parameter-similarity/raw/commit/098ea8650c7d0e06339c66e3a0509e38b85b3d7d/results/head-dominance/figures/joint_removal_effects.pdf)
|
||||
[SVG固定提交副本](https://gitea.entroia.com/Zayden-Jung/exp-model-parameter-similarity/raw/commit/098ea8650c7d0e06339c66e3a0509e38b85b3d7d/results/head-dominance/figures/joint_removal_effects.svg)
|
||||
|
||||
[完整基线报告](../results/head-dominance/REPORT.md) · [全部图表目录](../results/head-dominance/figures) · [图册SHA清单](figure-gallery-manifest.json)
|
||||
|
||||
使用√JS距离进行微调前后配对比较;保留JS散度作为尺度敏感性参考,避免混淆平方尺度与距离尺度。
|
||||
|
||||
## GSM8K微调前后配对图
|
||||
|
||||
[配对报告](../results/finetune-comparison/REPORT.md) · [CDF](../results/finetune-comparison/figures/paired_effective_heads_cdf.png) · [逐层分布](../results/finetune-comparison/figures/paired_effective_heads_by_layer.png) · [配对变化](../results/finetune-comparison/figures/paired_effective_heads_delta.png)
|
||||
|
||||
同目录保留PDF/SVG。图表随代码、原始数据和SHA清单一同提交,不依赖聊天附件。
|
||||
|
||||
## 高维壳与稀疏性对照
|
||||
|
||||

|
||||
|
||||
[条件定理、反例与数据说明](../results/sparse-shell/REPORT.md) · [PDF](../results/sparse-shell/sparse_shell_controls.pdf) · [SVG](../results/sparse-shell/sparse_shell_controls.svg)
|
||||
|
||||
## 两个实际2B检查点的参数球壳检验
|
||||
|
||||
[结论与解释](MODEL_SHELL_FINDINGS.md) · [实测报告、分组数量与限制](../results/model-shell/REPORT.md) · [固定协议](../model_shell/PROTOCOL.md) · [指标源表](../results/model-shell/metrics.csv) · [校验清单](../results/model-shell/audit.json)
|
||||
|
||||

|
||||
|
||||

|
||||
|
||||

|
||||
|
||||
[微调前后配对图](../results/model-shell/figures/finetune_paired_radii.png) · [有限样本参与率](../results/model-shell/figures/participation_reference.png)。每幅图同目录均保存PDF/SVG,完整CDF及参照数据保存在`profiles.json.gz`,实测范数保存在`measured_radii.npz`;均随本次成果进入Git。
|
||||
@@ -0,0 +1,23 @@
|
||||
# 功能复用文献核对
|
||||
|
||||
核对日期:2026-11-09。以下为原始论文摘要及相关章节核对,不是复现实验。
|
||||
本项目约束:不训练任何参数;已有静态试验没有执行模型前向。下面的方法不自动获得执行授权。
|
||||
|
||||
| 原始来源 | 研究的共享对象 / 证据 | 训练条件与适用限制 | 对本项目的用途 |
|
||||
|---|---|---|---|
|
||||
| [LISA, 2408.01890v2](https://arxiv.org/html/2408.01890v2), §3–5 | 跨层 token 注意力分布;考察头对齐、浅层敏感性;基准能力和吞吐验证 | 学习头对齐网络及低秩差异修正;不能归类为免训练替换。论文的 attention weights 在此主要指注意力分布,不是 W_Q/W_K 权重 | 借鉴逐头比较、首 token 对照和替换敏感性验证;不采用训练模块 |
|
||||
| [CLA, 2405.12981](https://arxiv.org/abs/2405.12981) | 相邻层共享 K/V heads,降低 KV cache;语言模型能力验证 | 摘要明确实验为从头训练 1B/3B 模型;不是现成 Qwen 的直接合并证据 | 区分 KV 缓存收益和参数/计算收益;作为架构可行性依据 |
|
||||
| [Systematic Study, NAACL 2025](https://aclanthology.org/2025.naacl-short.34/) | 系统比较跨层 KV 共享配置;语言建模、下游任务及生成吞吐 | 摘要报告激进配置涉及额外训练及 prefill 代价;不能由摘要推断零训练可迁移性 | 复用方向、共享拓扑及 prefill/decode 成本须分别记录 |
|
||||
| [MiniLM, 2002.10957v2](https://arxiv.org/abs/2002.10957) | 注意力分布及 value-value 关系蒸馏 | 明确训练学生模型;本项目不进行蒸馏 | 提醒检索分布和 value 内容需要分开考察 |
|
||||
| [CKA, ICML 2019](https://proceedings.mlr.press/v97/kornblith19a.html) | 表征之间的关系相似性,不是模块替换实验 | 论文讨论表征比较及可逆变换不变性的局限;高 CKA 不是可替换保证 | 可作为候选诊断,不能取代输出误差、干预和成本评测 |
|
||||
| [ALBERT, 1909.11942v6](https://arxiv.org/html/1909.11942v6), §3.1/4.5 | 跨层参数共享及训练后能力验证 | 共享架构参与训练;不能直接外推为已训练 checkpoint 可无损合并 | 参数存储减少与前向运算减少分别记账 |
|
||||
|
||||
## 对当前试验的推论
|
||||
|
||||
以上文献给出研究动机,没有证明 Qwen3.5-2B 已存在免训练的有效复用组。
|
||||
静态权重试验也没有否定该可能性。应使用同输入功能探测筛选,再对实际计算图实施单处干预。
|
||||
Qwen 的 full attention 与线性注意力需分别定义探测量;不能把 softmax attention 的结论照搬到状态递推模块。
|
||||
|
||||
数学工具参考:[SciPy orthogonal_procrustes](https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.orthogonal_procrustes.html)。
|
||||
纯正交最小二乘有直接的矩阵分解解法,无须枚举旋转矩阵;但求解一个对齐并不自动带来可省略的计算。
|
||||
本项目下一阶段默认不求 R,直接测目标函数和干预效果。
|
||||
@@ -0,0 +1,58 @@
|
||||
# 当前目标:已有学习部分是否重复实现相似函数
|
||||
|
||||
用户于 2026-11-09 明确:不以成本或加速作为成功条件,只关心相似功能重复是否存在。
|
||||
权重继续保留,只有用户明确要求删除才清理;不训练、不拟合旋转或缩放。
|
||||
|
||||
## 什么可以作为正例
|
||||
|
||||
两个不同的已训练部分,在同一真实输入上产生接近、且并非恒定/接近零的输出;
|
||||
独立留出输入仍满足预先固定的门槛。实际双向替换只作补充验证。
|
||||
不能仅因替换一小部分后整模型 NLL 几乎不变,就认定它们实现相同函数。
|
||||
|
||||
Q/K 增益的解析合并证明同一计算的重参数化自由度,不等于两个现成学习部分功能重复。
|
||||
前一轮 11 对完整/归一化模块未通过门槛,也不能否定更小粒度的重复。
|
||||
不将两个模块都包含 SiLU、矩阵乘法等共同运算本身作为重复功能证据。
|
||||
|
||||
## 新增一次有界搜索:MLP 学习神经元
|
||||
|
||||
在已保存的第 5、6、20、21 层真实 MLP 入口输入上,研究
|
||||
|
||||
f_(l,j)(h) = SiLU(w_gate_(l,j)^T h) × (w_up_(l,j)^T h)。
|
||||
|
||||
每层 6,144 个神经元具有不同学习参数。比较同一层内两个 scalar 子函数,
|
||||
不是比较来自不同输入的两个观测值,也不是把整层或整 tensor 称作重复。
|
||||
下游 down-projection 的列保持各自原样;替换发生在两个投影和 SiLU 之后的激活值处。
|
||||
|
||||
1. 4 条校准文本、4 条留出文本,每条 32 tokens,中文/英文各半,沿用先前冻结样本。
|
||||
2. 校准阶段只保留 RMS 至少为该层神经元中位数、std/RMS≥0.25、能量有效 token 数≥16
|
||||
的非平凡单元。能量有效样本数定义为 (sum f²)² / sum f⁴,不是独立统计样本量。
|
||||
3. 对这些单元做分块穷举;实际配对数必须记录。不用随机 hidden 向量作主要证据。
|
||||
4. 预先固定门槛:双向直接相对 L2≤20%、余弦≥0.98、Pearson≥0.95;同时输出 5/10/20%
|
||||
三档校准配对数量。没有拟合线性缩放、偏置或 R。
|
||||
5. 全部层中仅按校准的最小最坏方向误差选择 1 对,冻结选择;无通过者停止。
|
||||
6. 留出集要求同样的汇总函数门槛及活跃度条件,逐文本双向误差≤35%。不根据留出失败换候选。
|
||||
7. 只有函数验证通过,才做两个方向的独立网络替换;留出平均 NLL 增量≤0.01 nat/token,
|
||||
逐文本增量 P95≤0.05。它不是主要的功能重复判据,防止因单元贡献很小而得到假正例。
|
||||
|
||||
本阶段使用先前两小时累计预算的剩余额度,复用现有隐藏输入;不增加模型、样本规模或成本测试。
|
||||
神经元层面的经验正例不证明完整层可以互换、不证明全输入空间等价,也不能推断独立任务能力。
|
||||
留出只有 4 条文本,结论必须限定到此短文本先导分布;后续扩展文本才能检验稳健性。
|
||||
|
||||
## 同角色学习 RMSNorm:确定性函数误差界
|
||||
|
||||
静态试验已经发现部分语言层 post-attention norm 的参数接近。新增一次独立检验:
|
||||
比较 24 层各自的 input/post-attention RMSNorm(同角色内共 552 对),
|
||||
仅按训练后增益的全输入最坏误差界选 1 对,完全不依赖校准或留出输入。
|
||||
增益需非零且不是常量向量(std≥0.001),不把没有学习参数的共同归一化运算当作正例。
|
||||
|
||||
相同 eps、相同形状下,f_a(x)=diag(a)x/sqrt(mean(x²)+eps)。对任意非零 x,
|
||||
|
||||
||f_a(x)-f_b(x)|| / ||f_a(x)|| ≤ max_j |a_j-b_j|/|a_j|。
|
||||
|
||||
反方向分母换为 |b_j|。该上界由相应坐标基向量达到,是实数算术下的精确最坏界。
|
||||
因此可先证明两个现成学习模块的函数整体接近,而不只比较原始参数 w;实际增益为 1+w。
|
||||
选择门槛:最坏双向界≤20%;同时记录≤5%、≤10%、≤20% 配对数量。只选最优界的一对。
|
||||
随后重新采集其实际输入,要求留出逐文本直接双向输出误差≤5%,再测两个方向网络替换。
|
||||
函数接近与替换后网络 NLL 分开报告;后者沿用已有 0.01/0.05 门槛,不决定函数存在性本身。
|
||||
选定后不根据留出结果换配对;不搜索视觉模块、不增加模型、不以成本为条件。
|
||||
归一化层正例不能推广为 attention 或 MLP 主体存在同样重复。
|
||||
@@ -0,0 +1,50 @@
|
||||
# 实际2B参数:比均匀参照更靠里吗?
|
||||
|
||||
**在明确的“校准最大半径以内”口径下,是;但数据同时显示更宽的径向分布,不能概括为整团参数向内收缩。此次GSM8K微调几乎没有改变这些几何指标。**
|
||||
|
||||
这轮直接读取了保留的Qwen3.5-2B权重和同一底座的已有纯LoRA GRPO检查点,在8.211.154.134上使用CPU完成。没有训练、推理或删除权重。
|
||||
|
||||
## 一个模型有多少组高维分布?
|
||||
|
||||
按事先固定的参数矩阵行/列与语言注意力头视图规则,每个模型有 **1,558组**:554个矩阵视图、1,004个按头视图。它们有重叠,不是模型固有的唯一分布数量。
|
||||
|
||||
主要汇总使用1,521个样本充足、未抽取坐标的视图。另有36个小样本视图和1个坐标抽样视图,全部单列并保留数据。整个检查点包括语言、视觉与MTP,共632个参数张量;319个一维张量和18个低维卷积核不满足这次“高维向量云”的规则。
|
||||
|
||||
## 有更多内侧质量,但也有更长的外尾
|
||||
|
||||
每组最多512个参数向量,一半校准中心及半径,另一半测量。对照也使用同样的校准规则;没有用所有向量的最大范数事后包住全部数据。
|
||||
|
||||
以原点为中心,比较归一化半径 `r / R_cal <= 0.9`:
|
||||
|
||||
|视图|组数|相对均匀球的内侧质量差中位数|相对匹配均值、逐坐标方差的均匀盒的质量差中位数|
|
||||
|---|---:|---:|---:|
|
||||
|矩阵行/列|517|+98.0个百分点|+97.7个百分点|
|
||||
|按头|1,004|+97.3个百分点|+97.3个百分点|
|
||||
|
||||
改用校准均值作为中心,结果仍有约96.9–97.7个百分点的差异。因此,这个现象并非仅来自中心放在原点。
|
||||
|
||||
但最大半径很容易受到少数大范数向量影响。换成各自的校准RMS尺度,并比较完整CDF后,**93.8%的矩阵视图和87.8%的头视图同时存在超过10个百分点的内尾、外尾差异**。去均值后分别为94.4%和86.4%。这说明两尾同时增加是普遍现象,比“所有向量整体更靠内”更符合这些数据。
|
||||
|
||||

|
||||
|
||||
这三组示例按模块角色固定:GDN的Q头、全注意力的Q头、MLP上投影行。上排用校准最大半径,下排用RMS;它们只是帮助读图,不代替全量统计。
|
||||
|
||||
## 是否说明稀疏或低维?
|
||||
|
||||
还不能如此推断。采样视图的逐组精确零值率中位数为0;点坐标能量参与率除以维度的中位数约0.31。径向内侧质量、坐标能量集中和样本间的有效维数是不同的测量对象。
|
||||
|
||||
本轮同时记录了有限样本Gram参与率,并与同样本数的均匀参照比较。它受校准样本秩上限限制,不能作为真实内在维度的直接估计。盒参照也只匹配边际范围或逐坐标矩,并未匹配相关结构。因此,本轮没有单独识别“稀疏支持”导致的效应。
|
||||
|
||||
## 微调前后改变了吗?
|
||||
|
||||
LoRA实际改变72个投影矩阵,影响972个视图,其中936个属于主要汇总。只对这些受影响的主要视图配对,以免大量未改动参数稀释差异。
|
||||
|
||||
原点中心下,逐组中位 `r/R_cal` 的配对变化中位数为 **6.30×10⁻⁷**,最大绝对变化 **4.03×10⁻⁴**。去均值后的结论相近。90%半径质量差在原点中心口径下未跨越当前经验CDF的分辨率;这不表示两套权重完全相同。
|
||||
|
||||
因此,这两个检查点不能支持“微调后、更好的模型更靠内”的结论;本轮也没有测任务质量。参数几何中的宽径向结构可以继续挖掘,但它本身还不是功能重复或可互相替代的证据。
|
||||
|
||||
## 数据与核验
|
||||
|
||||
[完整数值报告](../results/model-shell/REPORT.md) · [冻结协议](../model_shell/PROTOCOL.md) · [复现方式](../model_shell/README.md) · [指标CSV](../results/model-shell/metrics.csv) · [摘要JSON](../results/model-shell/summary.json) · [审计SHA](../results/model-shell/audit.json)
|
||||
|
||||
6,232个模型×中心×视图组合全部核验;CDF从保留的校准/留出范数复算,检查有限样本秩上限,原模型和适配器SHA均与既有记录一致。远程4项测试通过,覆盖索引、LoRA读取、中心与尺度、均匀球校准和稀疏贴壳反例。图表、完整CDF及对照摘要、实测范数一同保存。结论为描述性统计:视图相关且重叠,没有将其当作独立重复实验进行显著性检验。
|
||||
@@ -0,0 +1,132 @@
|
||||
# Qwen3.5-2B 有界试验结果
|
||||
|
||||
本轮已完成约定时间预算内的试验。整 tensor 精确比较 2,000 对;子块在内部截止时
|
||||
完成 14,778 / 20,000 对(73.89%),其余 5,222 对未计算。不能把本轮写成全量搜索。
|
||||
|
||||
结论:在本次候选、坐标对齐及误差阈值范围内,没有获得显著的直接权重共享收益。
|
||||
这不等于证明该模型不能通过其他方法压缩,也不代表 MoE 模型具有同样结果。
|
||||
|
||||
## 已确认的试验范围
|
||||
|
||||
- 模型:官方 `Qwen/Qwen3.5-2B`,revision `15852e8c16360a2fea060d615a32b45270f8a8fc`。
|
||||
- 原始 safetensors 文件为 4,548,221,488 字节,已核对官方 LFS SHA256。
|
||||
- 共 632 个存储 tensor;权重数据本身为 4,548,144,832 字节。
|
||||
- 实际存储参数 2,274,069,824 个,包含官方 checkpoint 中的视觉部分。
|
||||
- 维数分布:319 个一维、294 个二维、18 个三维、1 个五维。
|
||||
- 三维权重是 `[6144, 1, 4]` 的卷积核,包含长度为 1 的轴;不能把它解释为一般三维体积。
|
||||
- 五维视觉权重纳入完整清单和全模型字节分母,本轮不扩展五维子块搜索。
|
||||
- 不执行前向计算、能力评测、训练或蒸馏。
|
||||
|
||||
## 整 tensor 的结果
|
||||
|
||||
已完成 2,000 对同角色、同形状候选的精确比较。它们是经过近邻筛选的候选,
|
||||
不是所有可能 tensor 对的穷举。
|
||||
|
||||
| 最大直接替换相对误差 | 非单成员共享组 | 最大组大小 | 净节省字节 | 占模型权重 |
|
||||
|---|---:|---:|---:|---:|
|
||||
| 1% | 0 | 1 | 0 | 0% |
|
||||
| 2% | 0 | 1 | 0 | 0% |
|
||||
| 5% | 1 | 2 | 0 | 0% |
|
||||
| 10% | 11 | 6 | 32,704 | 0.000719% |
|
||||
| 20% | 16 | 15 | 119,424 | 0.002626% |
|
||||
|
||||
20% 阈值下找到的共享组均为一维权重。组数增加不代表显著的字节收益。
|
||||
这里的百分比是权重误差,绝不是任务能力损失百分比。
|
||||
|
||||
净节省按本轮声明的理论布局计算:保留组代表,每个分析对象 16 字节映射,
|
||||
每个共享组 64 字节描述;若大于原存储,则保持原表示。
|
||||
|
||||
这部分结果不支持在本次搜索范围内通过直接共享整 tensor 显著压缩该 checkpoint。
|
||||
它不排除其他未比较组合、子块共享、重参数化或带残差的压缩方法。
|
||||
|
||||
## 子块结果
|
||||
|
||||
共筛选 13,328 个抽样子块。20% 直接替换误差阈值下:
|
||||
|
||||
这里的 tensor 指完整命名权重数组,block 指其局部子张量,不是 Transformer block。
|
||||
三维原始权重为 6144×1×4,本轮切片实际是 1×1×4(4 个数),局部块是
|
||||
4×1×4(16 个数);小块的高余弦不能推广为整份三维权重相似。
|
||||
|
||||
| 表示 | 非单成员共享组 | 最大组 | 映射开销前节省 | 当前布局净节省 |
|
||||
|---|---:|---:|---:|---:|
|
||||
| 64×64 二维块 | 3 | 3 | 40,960 字节 | 0 |
|
||||
| 128×128 二维块 | 1 | 2 | 32,768 字节 | 0 |
|
||||
| 三维 axis-0 切片 | 45 | 12 | 872 字节 | 0 |
|
||||
| 三维局部块 | 2 | 2 | 64 字节 | 0 |
|
||||
|
||||
二维共享候选均来自 `model.visual.pos_embed.weight`,未在已复核候选中形成大规模
|
||||
投影矩阵共享。不同块尺度是替代方案,不能相加。
|
||||
|
||||
净收益为零与声明的映射成本有关,不应解读为完全没有相似子块。更稀疏或更紧凑的
|
||||
索引格式可能降低开销,但本轮找到的二维毛节省也仅为几十 KB,尚不足以支持大幅
|
||||
减容。三维切片组数量较多,但整组三维权重合计只占模型字节的约 0.01945%。
|
||||
|
||||
模型字节占比:一维 0.01972%、二维 99.89167%、三维 0.01945%、五维 0.06917%。
|
||||
因此,应按字节贡献评估价值,不能按 tensor 数量或共享组数量推断收益。
|
||||
|
||||
全部分组是满足代表半径约束的可行方案,不保证达到最少组数或最优压缩率。
|
||||
|
||||
## 运行经验
|
||||
|
||||
1. 实际服务器内存约 1.7 GiB,新增的 60 GB 是磁盘扩容,不是内存。
|
||||
使用分块 BF16 解码和 mmap 页面释放完成了完整权重扫描与整 tensor 比较。
|
||||
2. 只看进程 RSS 不足以解释运行成本。`vmstat` 快照观察到 75%~85% 的 I/O 等待,
|
||||
同时存在交换活动;这些是共享主机的快照,不能全部归因于本进程。
|
||||
3. 与权重读取并行的绘图合成测试触发过 300 秒超时;观察到测试处于磁盘等待。
|
||||
后续验证应在权重读取结束后串行执行,避免把资源争用误判为数值算法失败。
|
||||
4. Matplotlib 图例改为固定位置,避免自动定位引入额外布局开销。
|
||||
5. 公平地选择各角色的候选,与高效地安排读取顺序,是两个不同问题。
|
||||
下一轮应评估“先固定公平候选集合,再按文件/源 tensor 分批执行”的效果,
|
||||
比较是否得到相同指标及更低 I/O 开销;本轮不在中途改变候选集合。
|
||||
6. 从已保存元数据计算,13,328 个抽样子块的原始数据合计 256,387,072 字节
|
||||
(约 244.5 MiB),对应指纹约 100.1 MiB。这是下一轮原始子块缓存的容量估计,
|
||||
不是本轮已经实现的缓存或实测加速。应先用小样本验证数值一致性、峰值内存和
|
||||
I/O 改善,再决定是否作为默认方案;不能仅凭容量估计保证运行效率。
|
||||
|
||||
参数分析及分组汇总耗时 6,661.13 秒(约 111.0 分钟),分析进程峰值 RSS 约
|
||||
201.3 MiB;这不包含系统文件缓存和同机服务内存。内部比较截止后保存覆盖率与
|
||||
分组,整个参数分析未触及 7,200 秒进程硬上限。块比较阶段将本进程 IO 优先级
|
||||
调整为 best-effort 7,未改变服务器全局设置。
|
||||
|
||||
最终修图版 6 项串行合成测试通过(约 77.2 秒),六类图表的 PNG/PDF/SVG 已生成,
|
||||
并对全部六类 PNG 做视觉检查;程序还核验了分组成员与代表之间的精确误差边。
|
||||
|
||||
## 三张诊断图的修订与解读
|
||||
|
||||
- 热力图只展示 `linear_attn.in_proj_a.weight`(16×2048),不是整个层。18 个相关层
|
||||
的 153 种配对中核验了 74 对;已测非对角余弦约 0.011~0.298。
|
||||
对角线 1 是身份比较,不是跨层发现。未测格子改用斜线,避免与接近零的底色混淆。
|
||||
- 原混合层距曲线受维数组成影响:层距 1 含 183 对一维、19 对二维;层距 8
|
||||
含 52 对一维、75 对二维。一维余弦中位数约从 0.941 变为 0.883,
|
||||
不能将混合曲线的陡降全部归因于层距。修订版按维数/块尺度分开,模块和筛选偏差仍存在。
|
||||
- 随机对照分为保留分布的元素打乱、匹配范数的零均值高斯向量、真实同角色随机配对。
|
||||
元素打乱后仍高余弦的情况主要出现在一维归一化、偏置和 A_log,提示均值/分布影响。
|
||||
修订版按维数分面、按密度归一化,并显示样本数,避免将不同样本量的峰高当概率比较。
|
||||
它们是诊断对照,不是显著性检验,也没有完全匹配每个模块的采样构成。
|
||||
三维对照分层仅有约 3~6 个样本,尤其不能据其峰形作总体推断。
|
||||
|
||||
这次只从已有 CSV/JSON 重绘并导出诊断表,模型权重已删除,没有增加参数比较或重新运行模型。
|
||||
|
||||
## 限制及后续决策
|
||||
|
||||
- 只研究一个 checkpoint;近邻筛选、同角色分组和子块抽样均可能漏掉其他共享机会。
|
||||
- 直方图和层距曲线存在候选筛选与模块混合偏差,不能用作总体分布或独立样本推断。
|
||||
- 光谱/子空间分析仅使用有界子体积;分布或子空间相似不等于可直接替换。
|
||||
- 本轮没有测量能力损失、困惑度、运行内存或推理速度,也没有产出可运行的压缩模型。
|
||||
- 大幅减少推理计算量需要进一步的计算图和路由分析,单纯共享存储不能保证加速。
|
||||
|
||||
下一轮先讨论候选选择与执行局部性、原始子块缓存、映射格式和阶段预算,
|
||||
再修订 SOP;不自动下载第二个模型。只有获得额外授权后,才进行校准前向及
|
||||
留出任务测试,以验证替换造成的能力损失。
|
||||
|
||||
数据与标准化报告见 `../results/qwen35-2b/`;具体阈值、分组、覆盖率、来源和校验记录
|
||||
随原始图表数据归档。权重在交付副本校验后按授权删除,重现实验需要按固定 revision
|
||||
重新下载。
|
||||
|
||||
## 功能复用假设仍待验证
|
||||
|
||||
本轮坐标对齐权重结果不能排除 Q/K 的联合功能接近或 MLP 分支可复用。
|
||||
下一阶段采用同一隐藏输入下的函数比较,不训练参数、不默认求正交 R;
|
||||
注意力分布、完整分支输出、真实网络单处替换和运行收益分别验证。
|
||||
详见 [免训练有界方案](TRAINING_FREE_FUNCTIONAL_PLAN.md) 与
|
||||
[文献核对](FUNCTIONAL_REUSE_LITERATURE.md)。这些是方案,尚未执行。
|
||||
@@ -0,0 +1,46 @@
|
||||
# Q/K 未旋转维度的确定性增益合并(已数值验证,归为重参数化)
|
||||
|
||||
这是从已核对架构直接推导的恒等式,不是看留出指标后扩大近邻搜索。
|
||||
原定 11 对候选保留原协议;本项是独立的解析对照,不训练、不求 R,也不选择层。
|
||||
预先固定完整网络干预层为第 3 层,分别验证两个移动方向;不根据数据换层。
|
||||
|
||||
## 恒等式与前提
|
||||
|
||||
以完成 RMS 单位化、尚未乘可学习增益的 q/k 为 u/v,Q/K 增益向量为 a/b。
|
||||
固定 checkpoint 的 norm 参数实际使用 a=1+w_Q,b=1+w_K。
|
||||
模型 head_dim=256,partial_rotary_factor=0.25:前 64 维旋转,后 192 维不旋转。
|
||||
逐头共享的增益向量与 GQA 结构已由官方代码确认。
|
||||
|
||||
对于后 192 维,任意 query/key 位置 p/s:
|
||||
|
||||
sum_j (a_j u_pj)(b_j v_sj) = sum_j u_pj ((a_j b_j) v_sj)
|
||||
= sum_j ((a_j b_j) u_pj) v_sj。
|
||||
|
||||
前 64 维保留原增益和位置旋转。因而完整 QK 点积在实数算术中相等,softmax、V、
|
||||
输出门控和 O 不变时,完整 attention 分支也是同一个函数。这个结论不依赖某一输入分布。
|
||||
浮点乘法次序改变,不能承诺逐 bit 相等;需报告实际误差并验证留出 NLL。
|
||||
不能把这一规则用于旋转维度,或未经核对的逐头独立增益架构。
|
||||
|
||||
## 验证与成本边界
|
||||
|
||||
1. 在已有真实 q/k 输入上,检查 6 个 full-attention 层的原始/两个合并方向分数、概率及输出。
|
||||
2. 固定第 3 层进行两个方向的独立网络替换,不叠加,使用原留出文本和质量阈值。
|
||||
3. 记录预先缓存增益后、相同 CPU 环境下原版与合并版局部归一化耗时;不外推 GPU 性能。
|
||||
4. 双精度数学推导和 FP32 实测分开。没有消除 Q/K 投影或 RMS 方差计算。
|
||||
|
||||
将 Q 的未旋转增益移入 K 时,每 token、每层少做 8×192=1,536 次逐元素增益乘法;
|
||||
反方向少做 2×192=384 次。它们只是归一化里的小部分操作,不是整个模型的百分比加速。
|
||||
NumPy 切片/拼接可能抵消这点收益,必须报告测得的局部时延。
|
||||
|
||||
参数量从两个 256 维增益变为两个 64 维旋转增益及一个 192 维联合增益,即 512→320 个数。
|
||||
但原增益以 BF16 存储、联合乘积以 FP32 保留时,载荷仍是 1,024 字节,没有 checkpoint
|
||||
净压缩;元数据还可能增加。FP32 运行时缓存增益的理论载荷 2,048→1,280 字节,
|
||||
不等于已测得整个模型 RSS 降低。另行 BF16 量化乘积会增加误差,不属于本次恒等式验证。
|
||||
|
||||
这是同一注意力计算内部的重参数化与冗余消除,不是两个现成 attention 层能互换的证明。
|
||||
若得到正例,应将它作为小粒度确定性优化机会;较大模块互换仍由原候选试验单独回答。
|
||||
|
||||
## 完成状态
|
||||
|
||||
96 组真实输入分数/概率检查及固定第3层双向网络检查均通过。
|
||||
该结果独立归类;主目标的已训练函数重复正例见 [CPU 功能结论](CPU_FUNCTIONAL_FINDINGS.md)。
|
||||
+140
@@ -0,0 +1,140 @@
|
||||
# Parameter similarity SOP — 2B pilot draft
|
||||
|
||||
Status: v0.1, executed for the bounded Qwen3.5-2B pilot. See [findings](PILOT_FINDINGS.md).
|
||||
Changes proposed from this experience require a small equivalence/performance check before
|
||||
the next model; no second model is started automatically.
|
||||
|
||||
## Scope and decision
|
||||
|
||||
Analyze official, unquantized safetensors without instantiating a model, running inference,
|
||||
training, or executing repository-supplied code. Start with Qwen/Qwen3.5-2B only.
|
||||
The question is the number of representative-centered sharing groups under direct weight
|
||||
replacement error limits, and the net byte savings of a theoretical shared representation.
|
||||
Weight error is not task-accuracy loss, and parameter sharing does not imply fewer FLOPs.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. Confirm authorized remote host, free disk, memory, CPU, existing work, and access.
|
||||
2. Install prebuilt wheels only. Never compile locally or silently use another runner.
|
||||
3. Freeze the official repository revision and download file manifest including SHA256.
|
||||
4. Allocate checkpoint size plus 3 GB disk margin. Use an experiment-owned directory,
|
||||
not a shared Hugging Face cache. Never delete pre-existing weights.
|
||||
|
||||
## Bounded procedure
|
||||
|
||||
1. Run synthetic correctness tests on the remote host: BF16, 3D indexing, directional
|
||||
replacement error, and non-transitive groups.
|
||||
2. Download one checkpoint sequentially and verify upstream sizes and LFS SHA256.
|
||||
3. Scan each tensor in 262,144-element chunks. Record shape, dtype, axis role,
|
||||
layer, bytes, moments, max magnitude, and finite-value checks. Memory-map weights;
|
||||
evict scanned pages. Do not search for exactly equal weight tensors.
|
||||
4. Whole-tensor screening: same role, same shape, same dtype. Sample 2,048 aligned,
|
||||
evenly spaced elements. Select up to 10 neighbors by each of cosine and relative
|
||||
error. Fairly distribute the 2,000-pair exact-verification cap across role groups.
|
||||
5. Block screening: non-overlapping 64x64 and 128x128 2D partitions. For 3D tensors,
|
||||
use axis-0 slices and bounded volumes adapted to actual dimensions. Keep up to
|
||||
256 uniformly reservoir-sampled blocks per role/shape/scale for the 1.7 GiB pilot host
|
||||
(below the planned maximum of 512). Record edge bytes
|
||||
and sampling coverage. Verify at most 20,000 block pairs. Never extrapolate sampled
|
||||
discoveries to unseen blocks or add savings across alternative scales.
|
||||
6. Exact metrics: full-element cosine, Pearson, directional replacement errors,
|
||||
symmetric error, optimal-scalar residual. Tail diagnostics use samples and say so.
|
||||
Controls include random actual comparable pairs, shuffled sampled elements,
|
||||
and norm-matched Gaussian vectors. Save sampled versus exact error discrepancies.
|
||||
Bounded mode-unfolding SVD probes are diagnostic only, not a replacement test.
|
||||
7. Form deterministic star groups at 1%, 2%, 5%, 10%, 20% relative weight error.
|
||||
Every member requires an exact verified edge to its representative. Unknown
|
||||
pairs remain independent; never use graph connected components as sharing groups.
|
||||
8. Compute original bytes minus representatives, 16-byte mapping per analyzed object,
|
||||
and 64-byte descriptor per shared group. Fall back to unchanged storage if larger.
|
||||
Count savings against the entire model, preserving unsampled blocks and edge regions.
|
||||
9. Save raw data, groups, thresholds, coverage, timings, resource usage, figures
|
||||
(PNG/PDF/SVG), model metadata and report. Validate every group against exact edges.
|
||||
10. Copy results to the deliverable workspace and verify checksums before deleting
|
||||
the experiment-owned checkpoint. Save a cleanup record. Do not delete the venv,
|
||||
result data or any unrelated cache.
|
||||
11. Review full diff, commit and push a work branch with Codex Bot identity, then
|
||||
create a PR for manual merge. Never silently initialize/push main or auto-merge.
|
||||
|
||||
## Budget and stop conditions
|
||||
|
||||
- Real 2B analysis computation: hard wall-time limit 7,200 seconds. Reserve final time
|
||||
for summaries; download and report-generation time are recorded separately.
|
||||
- Use one BLAS thread to avoid interference with existing services.
|
||||
- Save exact-pair tables every 100 pairs; stage deadlines prevent whole-tensor work
|
||||
from consuming the complete block budget.
|
||||
- On timeout or failure, retain source weights until partial results are recoverable
|
||||
and report coverage/failure truthfully. An internal planned cutoff may produce a complete
|
||||
bounded report with partial candidate coverage; always state verified/selected counts.
|
||||
An external kill or missing required report is an incomplete run, not a completed experiment.
|
||||
- Empty or negligible low-error sharing is a valid negative result. Do not add scales,
|
||||
increase error thresholds, align permutations, train or expand to other models automatically.
|
||||
- No runtime capability/performance claim without separately authorized evaluation.
|
||||
|
||||
## Commands (remote, from project root)
|
||||
|
||||
```sh
|
||||
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 timeout 300 venv/bin/python -B -m unittest discover -s tests -v
|
||||
venv/bin/python -B -m pilot.download --root runs/qwen35-2b
|
||||
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 timeout 7200 venv/bin/python -B -m pilot.analyze --root runs/qwen35-2b --seconds 6600
|
||||
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -B -m pilot.plot --root runs/qwen35-2b
|
||||
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 timeout 300 venv/bin/python -B -m unittest discover -s tests -v > runs/qwen35-2b/tests.log 2>&1
|
||||
venv/bin/python -B -m pilot.audit --root runs/qwen35-2b --analysis-code-commit 6121016
|
||||
venv/bin/python -B -m pilot.validate --root runs/qwen35-2b
|
||||
# Copy archived results, verify checksums at the destination, then:
|
||||
venv/bin/python -B -m pilot.validate --root runs/qwen35-2b --cleanup-weights
|
||||
```
|
||||
|
||||
## Follow-up decisions after the pilot
|
||||
|
||||
- The pilot verified 2,000 tensor pairs and 14,778/20,000 selected block pairs before cutoff.
|
||||
It took about 111 minutes; the final six-test suite including revised figure checks passed in about 77 seconds.
|
||||
- Hatch missing heatmap cells and label identity values as constructed. Separate layer-distance
|
||||
curves by dimension/scale; retain warnings about within-stratum module/selection bias.
|
||||
Normalize control histograms, display sample counts, and verify their areas equal one.
|
||||
Keep these diagnostics distinct from task-equivalence or statistical-significance claims.
|
||||
- Avoid concurrent plotting tests and heavy weight IO on the shared host. Record IO wait,
|
||||
swap, CPU and RSS; a low process RSS is not proof of low total resource cost.
|
||||
- Distinguish tensor/group counts from byte contributions: 1D tensors were about half the
|
||||
inventory count but only 0.0197% of weight bytes in this checkpoint.
|
||||
- Keep fair candidate selection separate from IO-local execution order. Benchmark batching
|
||||
by source file/tensor without changing the selected pairs or metrics.
|
||||
- The sampled raw blocks total about 244.5 MiB. Test an explicitly bounded raw BF16/F32
|
||||
cache against the uncached implementation before adopting it; include fingerprint and
|
||||
temporary conversion memory in the budget. This was not enabled in the current run.
|
||||
- Report both pre-metadata and net savings. Tiny blocks can have groups yet lose all savings
|
||||
to mapping overhead. Any alternative sparse encoding needs its own explicit accounting.
|
||||
- Separate low-byte normalization groups from matrix/expert compression opportunities.
|
||||
- Decide whether approximate-neighbor recall and block coverage justify another bounded pass.
|
||||
- If meaningful candidates exist, request a small calibration-forward experiment to measure
|
||||
output distortion, then held-out loss/tasks, then implementation-specific memory and latency.
|
||||
- For MoE, analyze router behavior and joint gate/up/down expert consistency before treating
|
||||
independently similar matrix blocks as interchangeable experts.
|
||||
- Use these observations to revise this SOP before downloading a second model.
|
||||
|
||||
## Training-free functional follow-up (design only)
|
||||
|
||||
Direct coordinate-aligned parameter similarity is a baseline, not a test of Q/K/V or MLP
|
||||
functional interchangeability. The follow-up uses same-input module probes without solving
|
||||
rotations or training any parameters, then single-site intervention and held-out quality/cost
|
||||
validation. See the [bounded plan](TRAINING_FREE_FUNCTIONAL_PLAN.md) and
|
||||
[primary-source literature](FUNCTIONAL_REUSE_LITERATURE.md). No forward pass, new download,
|
||||
or new model is started by adopting this document. The original static pilot remains
|
||||
inference-free; the proposed follow-up requires a separately confirmed execution environment.
|
||||
|
||||
## Current functional-run retention override
|
||||
|
||||
On 2026-11-09 the user explicitly changed retention: keep the CPU functional pilot
|
||||
checkpoint until the user explicitly requests deletion. Completion, timeout, failure
|
||||
and archive validation do not authorize cleanup. The earlier pilot cleanup remains
|
||||
a historical action; do not apply its automatic cleanup step to the current run.
|
||||
|
||||
## CPU functional follow-up outcome
|
||||
|
||||
The executed CPU study found a learned RMSNorm function-repetition example at text
|
||||
layer indices 12/13. Use same-input function error and a uniform analytic bound as
|
||||
primary evidence, with separate heldout network substitution checks. Costs are not
|
||||
a success condition under the latest user instruction. See the
|
||||
[executed findings](CPU_FUNCTIONAL_FINDINGS.md) and
|
||||
[function-repetition criterion](LEARNED_FUNCTION_REPETITION.md).
|
||||
Keep whole-module, neuron, normalization and algebraic-reparameterization results distinct.
|
||||
@@ -0,0 +1,125 @@
|
||||
# 免训练功能复用:下一阶段有界方案
|
||||
|
||||
状态:研究设计,真实模型功能探测尚未执行。原始参数试验已完成;另已完成单线程合成 CPU 容量探测,未新增模型前向、下载或训练。
|
||||
目标:寻找并验证至少一个在留出数据上保持约定质量、同时降低明确运行成本的复用实例;
|
||||
若没有通过者,报告有界搜索的阴性结果,不保证一定能找到。
|
||||
|
||||
## 1. 硬约束与研究对象
|
||||
|
||||
- 冻结所有模型参数。不进行优化器更新、微调、蒸馏、LoRA、学习投影或低秩修正。
|
||||
- 默认不求正交 R。校准集只用于选择有限候选和阈值,留出集不参与选择。
|
||||
- 同类型模块配对;Q/K/V 联合作为功能系统,MLP 独立验证。保留归一化、位置处理、门控及头结构。
|
||||
- 已存配置为 24 个文本层,其中 6 个 full attention、18 个线性注意力;执行前按固定 revision 的实际实现复核。
|
||||
full attention 的 15 个无序层对可穷举;线性注意力首轮只检查 6 个预先固定的相邻同类型层对。
|
||||
MLP 首轮检查 6 个预先固定、覆盖浅中深层的相邻层对。具体名单在运行前写入 manifest。
|
||||
- 之前热力图的 in_proj_a 是线性注意力控制投影示例,不是 Q/K/V 功能相似性的测量。
|
||||
|
||||
## 2. 为什么不用先找 R
|
||||
|
||||
给定相同隐藏输入 H,且位置、mask 和缓存/递推状态一致,比较两套模块的实际检索函数:
|
||||
|
||||
S_i(H) = Q_i(H) K_i(H)^T / sqrt(d),P_i(H) = softmax(S_i(H) + mask)。
|
||||
|
||||
这里 Q/K 指完成该架构实际归一化和位置处理后的量。直接比较 P 可避开内部坐标差异;
|
||||
logit 每行加常数不改变 softmax,因此同时报告去行均值的分数误差。
|
||||
|
||||
这不是“存在共同正交 R”的数学证明。反例 Q_2=2Q_1、K_2=K_1/2 保持所有点积,
|
||||
但非零 Q 的范数改变,不能由正交变换产生。功能等价允许比共同正交旋转更广的关系。
|
||||
相反,在单套 Q/K 上共同右乘 R 保持点积,只说明坐标不唯一,并未说明两个已训练层可复用。
|
||||
实际 RoPE、GQA、归一化、门控也必须包含在函数边界内。
|
||||
|
||||
有限探针只能提供相应输入分布上的经验证据,不是对所有 H 的形式证明。
|
||||
在无位置处理/归一化的纯线性特例中,可研究 W_Q W_K^T 的代数相等性;
|
||||
一般模块不据此简化,首轮不物化所有大算子。
|
||||
|
||||
## 3. 同输入与真实输入分开
|
||||
|
||||
对候选 i/j,分别从原模型 i 和 j 的入口收集 H_i、H_j;分别比较 f_i(H_i) 与 f_j(H_i),
|
||||
以及 f_i(H_j) 与 f_j(H_j)。避免只选有利于某一层的输入。模块前归一化如位于边界外,
|
||||
必须明确分别测“归一化后核心”和“包含归一化的完整分支”。
|
||||
|
||||
这检验模块函数接近程度。实际网络的 f_i(H_i) 与 f_j(H_j) 使用不同输入,是另一项量。
|
||||
权重共享仍需对每个输入计算;缓存上游输出用于下游则改变了输入,必须实际干预后检验。
|
||||
MLP 测残差相加前的分支输出,防止共同输入 H 掩盖分支差异。
|
||||
|
||||
## 4. 指标和对照
|
||||
|
||||
| 对象 | 首轮指标 | 不能据此宣称的结论 |
|
||||
|---|---|---|
|
||||
| full attention 逐头 P | 平均及 P95 行 JS(自然对数,范围 0–ln2)、top-5 重合率;去行均值 S 相对误差 | P 相似不保证 V/O 输出相似 |
|
||||
| 完整 attention 分支 | 相对 L2、逐 token 误差 P50/P95、余弦;包含 V/O/门控 | 平均低误差不保证长尾任务能力 |
|
||||
| 线性注意力 | 同输入、同合法初始状态的分支输出及最终状态误差;后续状态继续使用测试 | 不构造不存在的 softmax 图;状态数值相似不等于长期稳定 |
|
||||
| MLP | 同输入分支输出相对 L2、逐 token P95、余弦 | 不以各矩阵单独相似代替 gate/up/down 联合验证 |
|
||||
|
||||
相对误差统一为 ||candidate-reference|| / max(||reference||, epsilon),并记录低范数样本数量。
|
||||
首轮筛选门槛预设:P 平均 JS ≤0.01 且 top-5 重合 ≥0.8;完整分支相对 L2 ≤0.05、
|
||||
逐 token 相对误差 P95 ≤0.1。它们是筛选约定,不是文献保证或能力损失界。
|
||||
线性模块和 MLP 使用输出门槛;只通过 P 的候选不能自动通过完整模块门槛。
|
||||
|
||||
对照:同模块重复测量、随机同类型层对;首 token 保留/移除并重新归一化两个版本;
|
||||
排除自注意与局部窗口主导后再测;按序列/位置/浅中深层分层。零分母与无效行单独标记。
|
||||
逐头记录,不先平均头。默认只用原头序;可选的离散头排列仅在校准集选定后冻结,
|
||||
须满足 GQA 分组和 V/O 一致性,不允许把最优头匹配分数当作可执行替换结果。
|
||||
CKA 仅作辅助筛选;共享位置模板、attention sink 和共同残差都可能制造高相似。
|
||||
|
||||
## 5. 资源与阶段预算
|
||||
|
||||
采用 CPU 先导、GPU 后续细化的两阶段路径;GPU 不作为寻找初步证据的前置条件。
|
||||
当前服务器为 2 核 CPU、约 1.7 GiB RAM;60 GB 是磁盘扩容。详见
|
||||
[CPU 先导设计及实测](CPU_FIRST_VALIDATION.md)。
|
||||
|
||||
CPU 阶段仅处理文本,batch=1、长度 32,校准/留出各 4 条真实且去重的文本;
|
||||
仅检查 3 个预先固定、覆盖浅中深的 full-attention 层对和 2 个 MLP 层对。
|
||||
完整保留合法 key,逐头比较,不通过采样 key 改变 softmax。
|
||||
模型依赖实现一致性检查和容量探测完成后,才实施真实前向。
|
||||
权重按层/子矩阵流式读取并释放,禁止全模型转换成 FP32;不把依赖换页的完整框架加载视为可行方案。
|
||||
数据来源、许可、tokenizer revision、截断、样本 hash 和 seed 在运行前写入 manifest。
|
||||
|
||||
| CPU 阶段 | 上限 | 完成条件 / 停止条件 |
|
||||
|---|---|---|
|
||||
| 实现一致性及短样本容量探测 | 20 分钟 | 模块计算语义核对,内存峰值可控;OOM、持续换页或实现核对失败即停止 |
|
||||
| 真实隐藏输入采集 | 40 分钟 | 流式文本前向采集固定入口,不保存所有层权重/注意力图;超时保存覆盖率 |
|
||||
| 同输入功能比较 | 30 分钟 | 固定 5 对候选;无通过者停止;最多留下 1 个干预候选 |
|
||||
| 单处干预及归档 | 30 分钟 | 在 4 条留出文本上检查输出或流式 token NLL;不足以完成则只交付候选证据 |
|
||||
|
||||
CPU 首轮计算上限 120 分钟,下载单独计时。当前仅执行了约 88 MiB 峰值 RSS 的
|
||||
合成矩阵乘法探测,没有重下权重或执行真实前向。容量探测不能保证完整模型实现的峰值内存。
|
||||
线性注意力需另行完成正确性核对,首轮先不加入,避免状态递推实现拖延初步判断。
|
||||
短文本小样本无法覆盖长上下文或低频关键能力;阴性结果也不能排除更广泛候选。
|
||||
|
||||
只有发现值得跟进的候选,GPU 阶段才扩展到原定 27 对候选、长度 256、
|
||||
校准/留出各 16 条及更多独立任务,细化单处/多处干预与真实性能。
|
||||
届时由用户提供并确认 GPU 执行环境;当前不租用或假定已有 GPU。
|
||||
|
||||
## 6. 干预必须对应可节省的对象
|
||||
|
||||
| 干预 | 可以争取的收益 | 必须额外确认 |
|
||||
|---|---|---|
|
||||
| 跨层绑定权重 | 权重存储/加载内存 | 各层输入仍不同,通常不减少投影 FLOPs |
|
||||
| 重用上游 attention 分布,保留下游 V/O | 部分 Q/K 投影与 QK/softmax 计算 | 保留分布的内存及实现开销;decode 未必值得缓存完整图 |
|
||||
| 跨层共享 K/V cache | KV 内存,可能减少 K/V 投影 | GQA、位置、缓存布局与层依赖合法性 |
|
||||
| 重用完整 attention/MLP 输出 | 对应分支计算 | 下游输入不同,必须测试整个原模型中的替换效应 |
|
||||
|
||||
选择的每个候选首先给出一种具体可实现干预;只有同输入指标通过却无实际可省操作,
|
||||
标记“功能相似但暂无复用实现”。不把整模块可替换和其中 Q/K 可替换混为一谈。
|
||||
|
||||
首轮探索性质量门槛:留出平均 token NLL 增量 ≤0.01 nat/token,逐序列增量 P95 ≤0.05,
|
||||
输出有限;同时保存全部逐序列差值,不将该小样本阈值称为统计显著性或普适能力保证。
|
||||
候选只在校准集筛选;留出失败不回调阈值、不反复选择。
|
||||
|
||||
性能分别测 prefill 和固定 32-token decode;固定 batch/长度/dtype/backend,预热 3 次、
|
||||
交错运行基线与干预各 10 次,报告原始时延和中位数/P95、峰值内存、KV/权重字节及理论运算变化。
|
||||
插桩诊断时延不作为优化实现时延。若速度差落在运行波动内,不宣称加速;
|
||||
若只减内存,则结论限定为内存收益。计入缓存、索引与数据移动开销。
|
||||
|
||||
## 7. 分组与最终证据
|
||||
|
||||
单处通过后,才讨论多成员组:每个成员必须通过对同一代表的定向替换检查;
|
||||
不能用 A≈B、B≈C 推出 A 可代表 C。多处同时替换须再测累积误差和性能,
|
||||
单处收益不能相加为最终模型收益。首轮不自动执行多处替换。
|
||||
|
||||
输出每对候选/每序列/每头指标 CSV、覆盖率和失败原因、质量—成本散点图、逐层热图、
|
||||
基线与干预原始计时、环境及代码版本。结论分为:功能接近、单处干预通过、测得实际收益三档。
|
||||
“存在一个免训练复用点”的操作性证据必须同时具有后两项;当前静态试验尚未提供这类证据。
|
||||
|
||||
相关依据与训练限制见 [文献核对](FUNCTIONAL_REUSE_LITERATURE.md)。
|
||||
@@ -0,0 +1,48 @@
|
||||
{
|
||||
"source_commit": "098ea8650c7d0e06339c66e3a0509e38b85b3d7d",
|
||||
"base_model_revision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
|
||||
"sqrt_js_effective_heads": {
|
||||
"median": 8.93182211792415,
|
||||
"p10": 5.7840811612328125,
|
||||
"p90": 11.67265834734814,
|
||||
"donor_heads": 16
|
||||
},
|
||||
"files": [
|
||||
{
|
||||
"path": "results/head-dominance/figures/retrieval_scale_sensitivity.png",
|
||||
"sha256": "4f2dac5fdb5506ee5282d459dcb86cc6b58fac05d494d6409c29ea6924365add",
|
||||
"immutable_url": "https://gitea.entroia.com/Zayden-Jung/exp-model-parameter-similarity/raw/commit/098ea8650c7d0e06339c66e3a0509e38b85b3d7d/results/head-dominance/figures/retrieval_scale_sensitivity.png",
|
||||
"http_status": 200,
|
||||
"remote_sha_verified": true,
|
||||
"content_type": "image/png"
|
||||
},
|
||||
{
|
||||
"path": "results/head-dominance/figures/retrieval_scale_sensitivity.pdf",
|
||||
"sha256": "56a2f5655196018f4310dbccb29cb55c314111007598491496c9a36256958525",
|
||||
"immutable_url": "https://gitea.entroia.com/Zayden-Jung/exp-model-parameter-similarity/raw/commit/098ea8650c7d0e06339c66e3a0509e38b85b3d7d/results/head-dominance/figures/retrieval_scale_sensitivity.pdf"
|
||||
},
|
||||
{
|
||||
"path": "results/head-dominance/figures/retrieval_scale_sensitivity.svg",
|
||||
"sha256": "f51803e68df40df6dda91bce7236c999f405ccf30be56e7819242147a5e9a5b8",
|
||||
"immutable_url": "https://gitea.entroia.com/Zayden-Jung/exp-model-parameter-similarity/raw/commit/098ea8650c7d0e06339c66e3a0509e38b85b3d7d/results/head-dominance/figures/retrieval_scale_sensitivity.svg"
|
||||
},
|
||||
{
|
||||
"path": "results/head-dominance/figures/joint_removal_effects.png",
|
||||
"sha256": "7dc26da9fda80e62c6445c5dee29fe52c06146dea0cd65d444755cf0eb5bb729",
|
||||
"immutable_url": "https://gitea.entroia.com/Zayden-Jung/exp-model-parameter-similarity/raw/commit/098ea8650c7d0e06339c66e3a0509e38b85b3d7d/results/head-dominance/figures/joint_removal_effects.png",
|
||||
"http_status": 200,
|
||||
"remote_sha_verified": true,
|
||||
"content_type": "image/png"
|
||||
},
|
||||
{
|
||||
"path": "results/head-dominance/figures/joint_removal_effects.pdf",
|
||||
"sha256": "b35132d3e1cf3d202bc6695e64f64c9e495196bc6c9a92659ccf65aceb566040",
|
||||
"immutable_url": "https://gitea.entroia.com/Zayden-Jung/exp-model-parameter-similarity/raw/commit/098ea8650c7d0e06339c66e3a0509e38b85b3d7d/results/head-dominance/figures/joint_removal_effects.pdf"
|
||||
},
|
||||
{
|
||||
"path": "results/head-dominance/figures/joint_removal_effects.svg",
|
||||
"sha256": "03e6b299c1dbfa703ac72cbc7e9749e3173b649a22d829316dd5b5dccbe8728d",
|
||||
"immutable_url": "https://gitea.entroia.com/Zayden-Jung/exp-model-parameter-similarity/raw/commit/098ea8650c7d0e06339c66e3a0509e38b85b3d7d/results/head-dominance/figures/joint_removal_effects.svg"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,76 @@
|
||||
# Head influence concentration: measurement protocol
|
||||
|
||||
Target: quantify the statistical distribution of large and small upstream head
|
||||
influences on the next layer's Q/K/V; identity of dominant heads may vary at each
|
||||
sample, query token and recipient head. No training, GPU, deletion or model edits.
|
||||
|
||||
Scope: all 23 adjacent layer transitions of the retained Qwen3.5-2B text model,
|
||||
including 16-head linear attention and 8-head full attention. Measure effective
|
||||
Q/K/V, and retrieval probability changes where the recipient uses full attention.
|
||||
The intervening source-layer residual/MLP and recipient input normalization must
|
||||
be recomputed after interventions. Residual stream is preserved: removing an
|
||||
attention head is not removing all information reaching the recipient.
|
||||
|
||||
For each observation, remove one upstream attention head's projected contribution
|
||||
at a time. Record nonnegative influence magnitude d_i and its absolute scale.
|
||||
When sum(d_i)>0, p_i=d_i/sum(d_i) is normalized intervention influence, NOT an
|
||||
additive causal attribution or a native attention probability over upstream heads.
|
||||
All-zero influence has undefined concentration and must be flagged, not assigned
|
||||
uniform or concentrated values. Record small total influences explicitly.
|
||||
|
||||
Primary continuous measures:
|
||||
- Effective head count: N_eff=1/sum(p_i^2), range [1,H].
|
||||
- Effective head fraction: N_eff/H, supports different H.
|
||||
- Normalized concentration: C=(sum(p_i^2)-1/H)/(1-1/H), range [0,1].
|
||||
These are related summaries, not independent evidence. Report distributions over
|
||||
texts/tokens/recipient heads/layers, not only concentration of globally averaged
|
||||
influences. Primary query positions 7 onward avoid trivial short-prefix attention.
|
||||
|
||||
Secondary intuitive measures: top1/top25%/top50% influence share, bottom25%
|
||||
share, and number of heads reaching 90% influence. Preserve raw influence tables.
|
||||
No RMS activity threshold is used to define influence concentration.
|
||||
|
||||
Joint interventions are required: per observation's weak-head set and strong-head
|
||||
set must be removed together and measured, because individual ablations are not
|
||||
additive and may conceal cancellation or synergy. Rankings may be input-specific;
|
||||
these experiments cannot imply a universally removable fixed head set. Distinguish
|
||||
local next-layer QKV/probability effects from unmeasured final LM quality.
|
||||
|
||||
CPU measurements, scientific plots, tests and audits execute on the previously
|
||||
authorized server 8.211.154.134. All weights and existing caches remain retained.
|
||||
|
||||
## Fixed execution details (before real-model measurements)
|
||||
|
||||
Four texts are reused from crossqkv/fresh_samples.json (two Chinese/two English),
|
||||
32 tokens each; no independent generalization claim. All tokens are computed,
|
||||
queries7..31 are primary statistical observations. The observer caches source
|
||||
residual inputs and exact per-head projected output components; their sum is
|
||||
checked against the native branch output. First text's full forward is checked
|
||||
against the canonical streaming forward, and every transition has a null-mask
|
||||
control. No cached tensor is deleted.
|
||||
|
||||
For each query/recipient query-head, individual influence is the L2 change divided
|
||||
by that same baseline vector's L2 (floor1e-12). Q,K,V are analyzed separately.
|
||||
A declared balanced score sqrt((relative_Q^2+relative_K^2+relative_V^2)/3) is also
|
||||
analyzed; it does not treat their different physical scales as directly additive.
|
||||
Full attention: Q/K are measured after learned norm/partial RoPE, V before output
|
||||
mixing, and actual softmax retrieval JS is measured. Linear attention: Q/K/V are
|
||||
after causal convolution+SiLU; Q/K include the delta-rule L2 normalization and Q
|
||||
scale. It has no ordinary softmax retrieval distribution. Full-attention GQA K/V
|
||||
are repeated to query-head slots for aligned joint tests; these slots are not
|
||||
independent KV heads, and statistical observations are not independent samples.
|
||||
|
||||
Joint weak-half/strong-half/random-half/all-head removal uses a separate coherent
|
||||
sequence for each recipient query-head. At each source token, choose the donor
|
||||
half by that recipient query-head's QKV influence ranking at the same token.
|
||||
Recompute full recipient sequence; convolution and QK retrieval can therefore
|
||||
propagate changes between positions. Only the intended recipient head is scored
|
||||
for each scenario. This is an input-adaptive diagnostic, not a computationally
|
||||
free head-selection policy. Random uses fixed seed31000+sample_index*100+layer.
|
||||
Compare joint effects to sums of corresponding single effects but do not assume
|
||||
additivity. For JS use the same joint sets (selected by QKV, not by JS).
|
||||
|
||||
No success criterion assumes that dominance exists. Report effective-head and
|
||||
absolute-effect distributions even if contributions are diffuse. No final-LM
|
||||
NLL or deployable pruning quality claim. Scientific output includes all raw
|
||||
single/joint effects and deletion masks; large baseline caches remain external.
|
||||
@@ -0,0 +1,75 @@
|
||||
# Statistical head influence dominance (CPU)
|
||||
|
||||
This study answers how concentrated the **distribution of measured upstream-head
|
||||
influences** is, without assuming that dominant head identities are fixed across
|
||||
texts, positions or recipient heads. Read [registered protocol](PROTOCOL.md).
|
||||
|
||||
- `core.py`: normalized effective Q/K/V extraction for both attention types,
|
||||
source projected head decomposition, exact nonlinear downstream recomputation,
|
||||
coherent masks, continuous concentration and JS.
|
||||
- `run.py`: four fixed prior texts, all23 adjacent connections, single-head and
|
||||
weak/strong/random-half/all-head interventions, per-transition checkpointing,
|
||||
canonical-forward/null controls, preserved baseline caches.
|
||||
- `report.py`: full saved-data distributions, per-transition CSV, continuous
|
||||
effective head plots, influence rank curves, joint tests and Chinese report.
|
||||
- `audit.py`: independently recomputes concentration/masks from raw effects,
|
||||
validates scope/null/head sums, and hashes retained caches, source and results.
|
||||
- `tests/test_dominance.py`: continuous limiting cases, undefined zero-total,
|
||||
token-dependent masks, JS identity, both canonical mixed-layer paths, and joint
|
||||
interventions on a small checkpoint with known arrays.
|
||||
|
||||
Influence for Q/K/V is relative L2 change after deleting one whole donor attention
|
||||
head (at all sequence positions), preserving the prior residual and recomputing
|
||||
source MLP + recipient normalization. Source **output/value heads** are the units:
|
||||
16 for linear attention,8 for full attention. These are not arbitrary individual
|
||||
Q/K/V weight blocks. For a recipient query-head at one token, normalize donor
|
||||
influences to p. Primary continuous summary is N_eff=1/sum(p²), and N_eff/H.
|
||||
C=(sum(p²)-1/H)/(1-1/H) is a related0..1 concentration score. These are not
|
||||
independent confirmations. No RMS near-zero threshold or R fitting is used.
|
||||
|
||||
Q/K/V are separate primary scales; balanced QKV score is RMS of their three
|
||||
relative changes. It is a declared summary, not a norm in a single physical space.
|
||||
Zero total implies undefined concentration, not a uniform distribution. Single-head relative effects before head-share normalization and true joint
|
||||
effects are retained as well as shares. The saved metric named max_absolute is
|
||||
the largest unsigned relative-L2 effect, not an unnormalized hidden-space norm.
|
||||
The total field sums individual relative effects; it is not the measured joint
|
||||
effect of removing the whole attention branch.
|
||||
|
||||
Full-attention recipients additionally have real token-key probability JS changes;
|
||||
linear attention does not have an ordinary softmax attention probability matrix.
|
||||
Full-attention K/V repeat their GQA groups to query-head slots for aligned joint
|
||||
experiments, not as extra independent KV observations. Every headline statistic
|
||||
is descriptive: query/token/head observations are dependent, and four short prior
|
||||
texts do not establish universal dominance or independent generalization.
|
||||
|
||||
Joint masks vary per source token and recipient head. Recompute a coherent full
|
||||
sequence for each recipient head, scoring that intended head afterward. Rank by
|
||||
single QKV influence, remove weakest half/strongest half/a fixed-seed random half,
|
||||
or all attention heads. This investigates interactions and cross-token effects;
|
||||
it is not a validated free selector, fixed removable head subset or LM pruning.
|
||||
|
||||
Only run model work and scientific plotting on the authorized CPU server:
|
||||
```
|
||||
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -B -m unittest tests.test_dominance -v
|
||||
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -B -m dominance.run --source runs/cpu-qwen35-2b --out runs/NEW_DOMINANCE_DIR --seconds 7200
|
||||
venv/bin/python -B -m dominance.report runs/NEW_DOMINANCE_DIR
|
||||
venv/bin/python -B -m dominance.audit runs/NEW_DOMINANCE_DIR runs/cpu-qwen35-2b
|
||||
```
|
||||
|
||||
No training, GPU, new model download or automatic deletion. Raw intervention
|
||||
arrays and masks are delivered to Git; large baseline caches stay on the server
|
||||
with path/size/SHA metadata. Existing checkpoints/caches are never removed.
|
||||
|
||||
The saved sum of selected individual effects is only a query-wise mask-weighted
|
||||
reference: individual ablations remove a head at all positions, whereas joint
|
||||
masks can vary across positions. Their difference cannot isolate pure nonlinear
|
||||
interaction, since temporal intervention supports also differ. Use the measured
|
||||
weak/strong/random joint effects directly; do not call this reference an additive
|
||||
causal decomposition or a Shapley attribution.
|
||||
|
||||
Scale sensitivity: after reviewing the first complete summaries, add sqrt(JS)
|
||||
(Jensen–Shannon distance) alongside primary JS divergence. Divergence is roughly
|
||||
quadratic for small changes, so shares and effective head counts depend on this
|
||||
scale. All original primary statistics and raw interventions remain unchanged;
|
||||
primary-analysis-summary.json preserves the pre-sensitivity aggregate for exact
|
||||
audit comparison. Retrieval-scale plots and source tables show both definitions.
|
||||
@@ -0,0 +1,90 @@
|
||||
"""Recompute saved concentration, masks, scale and scope checks; preserve caches."""
|
||||
import json,hashlib,csv,sys
|
||||
from pathlib import Path
|
||||
import numpy as np
|
||||
from .core import concentration
|
||||
|
||||
|
||||
def main(root,source):
|
||||
root=Path(root);source=Path(source);status=json.loads((root/'status.json').read_text());assert status['state']=='completed'
|
||||
samples=json.loads((root/'samples.json').read_text());assert len(samples)==4 and all(len(s['token_ids'])==32 for s in samples)
|
||||
expected={(s['id'],i) for s in samples for i in range(23)};actual={(x['sample'],x['source']) for x in status['completed_links']};assert actual==expected and len(status['completed_links'])==92
|
||||
cfg=json.loads((source/'model_config.json').read_text())['text_config'];obs=dict(np.load(root/'observations.npz'));lookup=sorted(s['id'] for s in samples);counts={};undefined={};full_js_links=0;joint_values={}
|
||||
for sample,layer in sorted(actual):
|
||||
f=root/'raw'/f'{sample}-link-{layer:02d}.npz';d=dict(np.load(f));hs=8 if cfg['layer_types'][layer]=='full_attention' else 16;hr=8 if cfg['layer_types'][layer+1]=='full_attention' else 16
|
||||
if hr==8:
|
||||
full_js_links+=1;d['single_sqrtJS']=np.sqrt(d['single_JS'])
|
||||
for label in ('weak_half','strong_half','random_half','all_heads'):
|
||||
d['joint_'+label+'_sqrtJS']=np.sqrt(d['joint_'+label+'_JS'])
|
||||
mask=d.get('mask_'+label,np.ones((hr,32,hs),dtype=np.float32))
|
||||
d['individual_sum_'+label+'_sqrtJS']=np.sum(d['single_sqrtJS'].transpose(2,1,0)*mask,-1).T
|
||||
d['null_sqrtJS']=np.sqrt(d['null_JS'])
|
||||
masks={label:d['mask_'+label] for label in ('weak_half','strong_half','random_half')}
|
||||
for label,m in masks.items():
|
||||
assert m.shape==(hr,32,hs);assert np.isin(m,[0,1]).all();np.testing.assert_array_equal(m.sum(-1),hs//2)
|
||||
np.testing.assert_array_equal(masks['weak_half']+masks['strong_half'],1)
|
||||
ranked=np.argsort(d['single_QKV'].transpose(2,1,0),axis=-1,kind='stable')
|
||||
np.testing.assert_array_equal(np.take_along_axis(masks['weak_half'],ranked[...,:hs//2],-1),1)
|
||||
for key in ('Q','K','V','QKV')+(('JS','sqrtJS') if hr==8 else ()):
|
||||
single=d['single_'+key];assert single.shape==(hs,32,hr) and np.isfinite(single).all() and np.min(single)>=0
|
||||
effect=single[:,7:,:].transpose(1,2,0);stats=concentration(effect);counts[key]=counts.get(key,0)+effect.shape[0]*hr;undefined[key]=undefined.get(key,0)+int((~stats['valid']).sum())
|
||||
choose=(obs[key+'_source']==layer)&(obs[key+'_sample_index']==lookup.index(sample))
|
||||
assert int(choose.sum())==25*hr
|
||||
for metric,values in stats.items():
|
||||
if metric=='valid':continue
|
||||
np.testing.assert_allclose(obs[key+'_'+metric][choose],values.ravel(),rtol=1e-6,atol=1e-7,equal_nan=True)
|
||||
for label in ('weak_half','strong_half','random_half','all_heads'):
|
||||
mask=masks.get(label,np.ones((hr,32,hs),dtype=np.float32));expected_sum=np.sum(single.transpose(2,1,0)*mask,-1).T
|
||||
np.testing.assert_allclose(d['individual_sum_'+label+'_'+key],expected_sum,rtol=1e-6,atol=1e-8)
|
||||
assert d['joint_'+label+'_'+key].shape==(32,hr) and np.isfinite(d['joint_'+label+'_'+key]).all()
|
||||
joint_values.setdefault((key,label),[]).extend(d['joint_'+label+'_'+key][7:].ravel())
|
||||
assert float(np.max(d['null_'+key]))<=2e-5
|
||||
assert full_js_links==24
|
||||
summary=json.loads((root/'analysis-summary.json').read_text())
|
||||
def check_stats(values,record):
|
||||
v=np.asarray(values,dtype=np.float64);v=v[np.isfinite(v)];assert len(v)==record['count']
|
||||
if len(v):
|
||||
for name,x in zip(['p10','median','p90','mean'],[*np.quantile(v,[.1,.5,.9]),v.mean()]):np.testing.assert_allclose(record[name],x,rtol=2e-6,atol=1e-8)
|
||||
for key in counts:
|
||||
for metric,record in summary['concentration'][key].items():check_stats(obs[key+'_'+metric],record)
|
||||
primary=json.loads((root/'primary-analysis-summary.json').read_text())
|
||||
for section in ('concentration','joint','paired_joint','by_recipient_type'):
|
||||
for key,value in primary[section].items():assert summary[section][key]==value,(section,key)
|
||||
full_sources=sorted({x['source'] for x in status['completed_links'] if x['full_attention_recipient']})
|
||||
assert summary['full_attention_source_indices']==full_sources
|
||||
for key,groups in summary['by_recipient_type'].items():
|
||||
src=obs[key+'_source']
|
||||
for label,metrics in groups.items():
|
||||
choose=np.isin(src,full_sources) if label=='full_attention' else ~np.isin(src,full_sources)
|
||||
for metric,record in metrics.items():check_stats(obs[key+'_'+metric][choose],record)
|
||||
for (key,label),values in joint_values.items():check_stats(values,summary['joint'][key][label])
|
||||
for key in counts:
|
||||
weak=np.asarray(joint_values[(key,'weak_half')]);strong=np.asarray(joint_values[(key,'strong_half')]);random=np.asarray(joint_values[(key,'random_half')]);valid=weak>0;record=summary['paired_joint'][key]
|
||||
assert record['zero_weak_effect_count']==int((~valid).sum())
|
||||
np.testing.assert_allclose(record['weak_less_than_strong_fraction'],np.mean(weak<strong))
|
||||
np.testing.assert_allclose(record['weak_less_than_random_fraction'],np.mean(weak<random))
|
||||
check_stats(strong[valid]/weak[valid],record['strong_over_weak']);check_stats(random[valid]/weak[valid],record['random_over_weak'])
|
||||
controls=json.loads((root/'head-sum-controls.json').read_text());assert len(controls)==96 and max(x['relative_l2'] for x in controls)<2e-5
|
||||
assert status['canonical_forward_relative_error']<2e-5
|
||||
caches=[]
|
||||
for f in sorted((root/'cache').rglob('*')):
|
||||
if f.is_file():caches.append({'path':str(f.resolve()),'bytes':f.stat().st_size,'sha256':hashlib.sha256(f.read_bytes()).hexdigest()})
|
||||
weights=list((source/'weights').rglob('*.safetensors'));assert weights and sum(f.stat().st_size for f in weights)==4548221488
|
||||
manifest=json.loads((source/'manifest.json').read_text());weight_records=[]
|
||||
for f in weights:
|
||||
with f.open('rb') as stream:sha=hashlib.file_digest(stream,'sha256').hexdigest()
|
||||
expected=next(x['upstream_sha256'] for x in manifest['files'] if x['name']==f.name);assert sha==expected
|
||||
weight_records.append({'path':str(f.resolve()),'bytes':f.stat().st_size,'sha256':sha,'matches_upstream':True})
|
||||
external={'host':'8.211.154.134','checkpoint_directory':str((source/'weights').resolve()),'checkpoint_bytes':sum(f.stat().st_size for f in weights),'checkpoint_files':weight_records,'revision':'15852e8c16360a2fea060d615a32b45270f8a8fc','baseline_caches':caches,'automatic_deletion':False,'prior_caches_untouched':True}
|
||||
(root/'external-caches.json').write_text(json.dumps(external,indent=2)+'\n')
|
||||
reference=json.loads((source/'reference-check.json').read_text())
|
||||
assert reference['passed'] and all(x['passed'] for x in reference['tests'])
|
||||
assert hashlib.sha256(Path('functional/model.py').read_bytes()).hexdigest()==reference['implementation_sha256']
|
||||
(root/'prior-official-reference-check.json').write_bytes((source/'reference-check.json').read_bytes())
|
||||
files=list(Path('dominance').glob('*.py'))+[Path('dominance/PROTOCOL.md'),Path('dominance/README.md'),Path('tests/test_dominance.py'),Path('functional/model.py'),Path('crossqkv/fresh_samples.json')]
|
||||
(root/'source-hashes.json').write_text(json.dumps({str(f):hashlib.sha256(f.read_bytes()).hexdigest() for f in files if not f.name.startswith('._')},indent=2)+'\n')
|
||||
audit={'passed':True,'transitions':23,'texts':4,'link_observations':92,'primary_observations_per_metric':counts,'undefined_zero_total_per_metric':undefined,'full_attention_js_links':full_js_links,'head_sum_controls':96,'canonical_forward_relative_error':status['canonical_forward_relative_error'],'all_raw_concentration_and_masks_recomputed':True,'pooled_concentration_and_joint_summary_recomputed':True,'original_primary_summaries_unchanged':True,'sqrt_JS_scale_sensitivity_recomputed':True,'retained_baseline_cache_files':len(caches),'checkpoint_sha_matches_upstream':True,'prior_official_reference_check_passed_same_implementation':True,'prior_official_reference_tests':len(reference['tests'])}
|
||||
(root/'audit.json').write_text(json.dumps(audit,indent=2)+'\n');print(json.dumps(audit,indent=2),flush=True)
|
||||
included=[f for f in root.rglob('*') if f.is_file() and 'cache' not in f.relative_to(root).parts and f.name!='checksums.json']
|
||||
(root/'checksums.json').write_text(json.dumps({str(f.relative_to(root)):hashlib.sha256(f.read_bytes()).hexdigest() for f in sorted(included)},indent=2)+'\n')
|
||||
if __name__=='__main__':main(sys.argv[1],sys.argv[2])
|
||||
@@ -0,0 +1,85 @@
|
||||
"""Frozen-model head interventions and continuous influence concentration."""
|
||||
import numpy as np
|
||||
from functional.model import StreamingLM,PREFIX,rope,causal_conv,attention
|
||||
|
||||
|
||||
def concentration(d):
|
||||
"""Last axis donor head; zero total is undefined, represented by NaNs."""
|
||||
d=np.asarray(d,dtype=np.float64)
|
||||
if not np.isfinite(d).all() or np.any(d<0):raise ValueError('Finite nonnegative influences required')
|
||||
h=d.shape[-1];total=d.sum(-1);valid=total>0
|
||||
p=np.divide(d,total[...,None],out=np.zeros_like(d),where=valid[...,None]);square=np.sum(p*p,-1)
|
||||
neff=np.divide(1,square,out=np.full_like(total,np.nan),where=valid)
|
||||
ordered=np.sort(p,axis=-1)[...,::-1]
|
||||
return {'total':total,'max_absolute':d.max(-1),'valid':valid,'effective_heads':neff,'effective_fraction':neff/h,
|
||||
'concentration':np.where(valid,(square-1/h)/(1-1/h),np.nan),
|
||||
'top1':np.where(valid,ordered[...,0],np.nan),'top25':np.where(valid,ordered[...,:max(1,h//4)].sum(-1),np.nan),
|
||||
'top50':np.where(valid,ordered[...,:max(1,h//2)].sum(-1),np.nan),
|
||||
'bottom25':np.where(valid,ordered[...,-max(1,h//4):].sum(-1),np.nan),
|
||||
'heads90':np.where(valid,1+np.sum(np.cumsum(ordered,axis=-1)<.9,axis=-1),np.nan)}
|
||||
|
||||
|
||||
def masks_from_influences(d,weak=True):
|
||||
"""d [donor,T,recipient]; output masks [recipient,T,donor]."""
|
||||
ranked=np.argsort(d.transpose(2,1,0),axis=-1,kind='stable');n=d.shape[0]//2
|
||||
selected=ranked[...,:n] if weak else ranked[...,-n:]
|
||||
mask=np.zeros_like(ranked,dtype=np.float32);np.put_along_axis(mask,selected,1,axis=-1)
|
||||
return mask
|
||||
|
||||
|
||||
def js_rows(a,b):
|
||||
a=np.asarray(a,dtype=np.float64);b=np.asarray(b,dtype=np.float64);m=(a+b)/2
|
||||
def term(x):return np.where(x>0,x*np.log(np.maximum(x,1e-300)/np.maximum(m,1e-300)),0)
|
||||
return np.maximum(0,.5*np.sum(term(a)+term(b),axis=-1))
|
||||
|
||||
|
||||
class InfluenceLM(StreamingLM):
|
||||
capturing=False
|
||||
def linear(self,x,name,rows=256):
|
||||
y=super().linear(x,name,rows)
|
||||
if self.capturing and (name.endswith('self_attn.o_proj') or name.endswith('linear_attn.out_proj')):
|
||||
layer=int(name.split('.')[1]);nh=self.config['num_attention_heads'] if 'self_attn' in name else self.config['linear_num_value_heads']
|
||||
xx=x.reshape(*x.shape[:2],nh,-1);dim=xx.shape[-1];t=self.tensors[PREFIX+name+'.weight'];od,idim=t.shape
|
||||
out=np.empty((*x.shape[:2],nh,od),dtype=np.float32)
|
||||
for start in range(0,od,256):
|
||||
self.guard();stop=min(od,start+256);w=t.read(slice(start*idim,stop*idim)).reshape(stop-start,idim)
|
||||
for head in range(nh):out[:,:,head,start:stop]=xx[:,:,head]@w[:,head*dim:(head+1)*dim].T
|
||||
del w;t.evict()
|
||||
err=float(np.linalg.norm(out.sum(2).astype(np.float64)-y)/max(np.linalg.norm(y.astype(np.float64)),1e-30))
|
||||
if err>2e-5:raise AssertionError(('head sum mismatch',layer,err))
|
||||
self.head_outputs=out;self.head_sum_errors.append({'layer':layer,'relative_l2':err})
|
||||
return y
|
||||
|
||||
def baseline(self,ids,root):
|
||||
h=self.embedding(ids);self.head_sum_errors=[]
|
||||
for layer,kind in enumerate(self.config['layer_types']):
|
||||
raw=h.copy();n=self.normalization(h,f'layers.{layer}.input_layernorm');self.capturing=True
|
||||
out,_=self.full_attention(n,layer) if kind=='full_attention' else (self.linear_attention(n,layer),None)
|
||||
self.capturing=False;heads=self.head_outputs;self.head_outputs=None
|
||||
mid=raw+out;h=mid+self.mlp(self.normalization(mid,f'layers.{layer}.post_attention_layernorm'),layer)
|
||||
np.savez(root/f'layer-{layer}.npz',raw=raw,out=out,heads=heads,next_raw=h)
|
||||
print('baseline layer',layer,flush=True)
|
||||
return h
|
||||
|
||||
def recipient(self,raw,layer):
|
||||
h=self.normalization(raw,f'layers.{layer}.input_layernorm');c=self.config;b,t=h.shape[:2]
|
||||
if c['layer_types'][layer]=='full_attention':
|
||||
p=f'layers.{layer}.self_attn.';nh=c['num_attention_heads'];nk=c['num_key_value_heads'];d=c['head_dim']
|
||||
q=self.linear(h,p+'q_proj').reshape(b,t,nh,2*d)[...,:d];k=self.linear(h,p+'k_proj').reshape(b,t,nk,d);v=self.linear(h,p+'v_proj').reshape(b,t,nk,d)
|
||||
q=self.normalization(q,p+'q_norm');k=self.normalization(k,p+'k_norm');r=c['rope_parameters'];q,k=(rope(z,r['rope_theta'],r['partial_rotary_factor']) for z in (q,k))
|
||||
k=np.repeat(k,nh//nk,axis=2);v=np.repeat(v,nh//nk,axis=2)
|
||||
_,prob=attention(q,k,v)
|
||||
return {'Q':q,'K':k,'V':v,'P':prob}
|
||||
p=f'layers.{layer}.linear_attn.';kh=c['linear_num_key_heads'];vh=c['linear_num_value_heads'];kd=c['linear_key_head_dim'];vd=c['linear_value_head_dim']
|
||||
x=causal_conv(self.linear(h,p+'in_proj_qkv'),self.small(p+'conv1d.weight')[:,0,:]);q,k,v=np.split(x,[kh*kd,2*kh*kd],axis=-1)
|
||||
q=q.reshape(b,t,kh,kd);k=k.reshape(b,t,kh,kd);v=v.reshape(b,t,vh,vd)
|
||||
if vh!=kh:q,k=np.repeat(q,vh//kh,axis=2),np.repeat(k,vh//kh,axis=2)
|
||||
q=q/np.sqrt(np.sum(q*q,axis=-1,keepdims=True)+1e-6)/np.float32(np.sqrt(kd));k=k/np.sqrt(np.sum(k*k,axis=-1,keepdims=True)+1e-6)
|
||||
return {'Q':q,'K':k,'V':v}
|
||||
|
||||
def intervene(self,cache,layer,masks):
|
||||
# Each scenario has a full coherent token-dependent deletion mask.
|
||||
removed=np.einsum('bth,thd->btd',masks,cache['heads'][0],optimize=True)
|
||||
mid=cache['raw']+cache['out']-removed
|
||||
nxt=mid+self.mlp(self.normalization(mid,f'layers.{layer}.post_attention_layernorm'),layer)
|
||||
return self.recipient(nxt,layer+1)
|
||||
@@ -0,0 +1,140 @@
|
||||
"""Saved-data scientific analysis; no model invocation or fitting."""
|
||||
import csv,json
|
||||
from pathlib import Path
|
||||
import numpy as np
|
||||
import matplotlib
|
||||
matplotlib.use('Agg')
|
||||
import matplotlib.pyplot as plt
|
||||
from .core import concentration
|
||||
|
||||
|
||||
def stats(x):
|
||||
x=np.asarray(x,dtype=np.float64);valid=x[np.isfinite(x)]
|
||||
if not valid.size:return {'count':0,'p10':None,'median':None,'p90':None,'mean':None}
|
||||
q=np.quantile(valid,[.1,.5,.9]);return {'count':int(valid.size),'p10':float(q[0]),'median':float(q[1]),'p90':float(q[2]),'mean':float(valid.mean())}
|
||||
|
||||
|
||||
def csvwrite(p,rows):
|
||||
with p.open('w') as f:
|
||||
w=csv.DictWriter(f,fieldnames=list(rows[0]));w.writeheader();w.writerows(rows)
|
||||
|
||||
|
||||
def main(root):
|
||||
root=Path(root);figdir=root/'figures';figdir.mkdir(exist_ok=True);status=json.loads((root/'status.json').read_text());assert status['state']=='completed'
|
||||
rows=[];joint_rows=[];pool={};curves={};joint_pool={};obs={};winner=[];joint_by_layer={}
|
||||
for entry in status['completed_links']:
|
||||
sample=entry['sample'];source=entry['source'];data=dict(np.load(root/'raw'/f'{sample}-link-{source:02d}.npz'))
|
||||
if 'single_JS' in data:
|
||||
data['single_sqrtJS']=np.sqrt(data['single_JS'])
|
||||
for label in ('weak_half','strong_half','random_half','all_heads'):
|
||||
data['joint_'+label+'_sqrtJS']=np.sqrt(data['joint_'+label+'_JS'])
|
||||
masks=data.get('mask_'+label,np.ones_like(data['mask_weak_half']))
|
||||
data['individual_sum_'+label+'_sqrtJS']=np.sum(data['single_sqrtJS'].transpose(2,1,0)*masks,-1).T
|
||||
for key in ('Q','K','V','QKV','JS','sqrtJS'):
|
||||
if 'single_'+key not in data:continue
|
||||
d=data['single_'+key][:,7:,:].transpose(1,2,0);c=concentration(d);valid=c['valid'];h=d.shape[-1]
|
||||
metric={k:v.reshape(-1) for k,v in c.items() if k!='valid'}
|
||||
row={'sample':sample,'source':source,'recipient':source+1,'metric':key,'source_heads':h,'recipient_attention_type':'full_attention' if entry['full_attention_recipient'] else 'linear_attention','observations':d.shape[0]*d.shape[1],'undefined_zero_total':int((~valid).sum())}
|
||||
for k,v in metric.items():
|
||||
pool.setdefault(key,{}).setdefault(k,[]).extend(v)
|
||||
for s,z in stats(v).items():row[k+'_'+s]=z
|
||||
rows.append(row)
|
||||
# Raw per-observation statistics for independently reproducing figures.
|
||||
for k,v in metric.items():obs.setdefault(key+'_'+k,[]).append(v.astype(np.float32))
|
||||
n=d.shape[0]*d.shape[1]
|
||||
for k,v in [('source',np.full(n,source)),('sample_index',np.full(n,sorted({e['sample'] for e in status['completed_links']}).index(sample))),('token',np.repeat(np.arange(7,32),d.shape[1])),('recipient_head',np.tile(np.arange(d.shape[1]),25))]:obs.setdefault(key+'_'+k,[]).append(v.astype(np.int16))
|
||||
p=np.divide(d,d.sum(-1,keepdims=True),out=np.zeros_like(d),where=d.sum(-1,keepdims=True)>0);cum=np.concatenate([np.zeros((*d.shape[:2],1)),np.cumsum(np.sort(p,-1)[...,::-1],-1)],-1)
|
||||
grid=np.linspace(0,1,101);cc=np.array([np.interp(grid,np.linspace(0,1,h+1),v) for v in cum[valid]])
|
||||
curves.setdefault(key,[]).extend(cc)
|
||||
win=np.argmax(d,-1);counts=np.bincount(win[valid],minlength=h)
|
||||
winner.append({'sample':sample,'source':source,'metric':key,'most_frequent_winner_fraction':float(counts.max()/max(counts.sum(),1)),'distinct_winning_heads':int((counts>0).sum()),'head_counts':json.dumps(counts.tolist())})
|
||||
for label in ('weak_half','strong_half','random_half','all_heads'):
|
||||
values=data['joint_'+label+'_'+key][7:];summed=data['individual_sum_'+label+'_'+key][7:]
|
||||
jr={'sample':sample,'source':source,'recipient':source+1,'metric':key,'removed':label}
|
||||
for k,x in [('joint_effect',values),('sum_individual',summed)]:
|
||||
for s,z in stats(x).items():jr[k+'_'+s]=z
|
||||
joint_rows.append(jr);joint_pool.setdefault(key,{}).setdefault(label,[]).extend(values.ravel());joint_by_layer.setdefault((key,label,source),[]).extend(values.ravel())
|
||||
csvwrite(root/'concentration-by-link.csv',rows);csvwrite(root/'joint-by-link.csv',joint_rows);csvwrite(root/'winner-identities.csv',winner)
|
||||
np.savez_compressed(root/'observations.npz',**{k:np.concatenate(v) for k,v in obs.items()})
|
||||
pooled={k:{m:stats(v) for m,v in metrics.items()} for k,metrics in pool.items()};joint_stats={k:{m:stats(v) for m,v in metrics.items()} for k,metrics in joint_pool.items()}
|
||||
paired={}
|
||||
for key,labels in joint_pool.items():
|
||||
weak=np.asarray(labels['weak_half']);strong=np.asarray(labels['strong_half']);random=np.asarray(labels['random_half']);valid=weak>0
|
||||
paired[key]={'weak_less_than_strong_fraction':float(np.mean(weak<strong)),'weak_less_than_random_fraction':float(np.mean(weak<random)),'zero_weak_effect_count':int((~valid).sum()),'strong_over_weak':stats(strong[valid]/weak[valid]),'random_over_weak':stats(random[valid]/weak[valid])}
|
||||
full_sources=sorted({x['source'] for x in status['completed_links'] if x['full_attention_recipient']})
|
||||
by_type={}
|
||||
for key in pool:
|
||||
src=np.concatenate(obs[key+'_source']);by_type[key]={}
|
||||
for label,choose in [('full_attention',np.isin(src,full_sources)),('linear_attention',~np.isin(src,full_sources))]:
|
||||
if not choose.any():continue
|
||||
by_type[key][label]={m:stats(np.concatenate(obs[key+'_'+m])[choose]) for m in pool[key]}
|
||||
summary={'scale_sensitivity':'sqrt(JS) added after reviewing completed primary summaries; primary JS and all raw interventions unchanged; squared divergences cannot be equated to probability mass or linear-scale effects','by_recipient_type':by_type,'full_attention_source_indices':full_sources,'paired_joint':paired,'pooling':'query7..31, per text/token/recipient-query-head; descriptive dependent observations, not independent samples','concentration':pooled,'joint':joint_stats,'counts':{'links':len(status['completed_links']),'texts':len(json.loads((root/'samples.json').read_text())),'layer_transitions':23},'sample_index_key':sorted({e['sample'] for e in status['completed_links']}),'winner_identity_file':'winner-identities.csv'}
|
||||
(root/'analysis-summary.json').write_text(json.dumps(summary,indent=2,allow_nan=False)+'\n')
|
||||
def save(fig,name):
|
||||
for ext in ('png','pdf','svg'):fig.savefig(figdir/(name+'.'+ext),dpi=180)
|
||||
plt.close(fig)
|
||||
fig,axs=plt.subplots(1,2,figsize=(14,4.8),layout='constrained')
|
||||
colors={'Q':'C0','K':'C1','V':'C2','QKV':'C3'}
|
||||
for key in colors:
|
||||
for ax,metric in zip(axs,['effective_fraction','top1']):
|
||||
med=[];lo=[];hi=[]
|
||||
source= np.concatenate(obs[key+'_source']);vals=np.concatenate(obs[key+'_'+metric])
|
||||
for layer in range(23):
|
||||
z=stats(vals[source==layer]);lo.append(z['p10']);med.append(z['median']);hi.append(z['p90'])
|
||||
ax.plot(range(1,24),med,label=key,color=colors[key]);ax.fill_between(range(1,24),lo,hi,color=colors[key],alpha=.10)
|
||||
axs[0].set(title='Continuous effective-head fraction (median, p10–p90)',ylabel='N_eff / number of donor heads')
|
||||
axs[1].set(title='Largest donor-head influence share',ylabel='Top-1 normalized ablation effect')
|
||||
for ax in axs:
|
||||
for j,source in enumerate(full_sources):ax.axvspan(source+.6,source+1.4,color='gray',alpha=.08,label='Full-attention receiver' if j==0 else None)
|
||||
ax.set(xlabel='Recipient layer index',ylim=(0,1.02));ax.legend();ax.grid(alpha=.2)
|
||||
fig.suptitle('All 23 adjacent transitions; four texts, query positions 7–31; no training')
|
||||
save(fig,'continuous_dominance')
|
||||
fig,axs=plt.subplots(1,2,figsize=(13,4.8),layout='constrained');curve_rows=[]
|
||||
for key in colors:
|
||||
a=np.asarray(curves[key]);lo,med,hi=np.quantile(a,[.1,.5,.9],axis=0);x=np.linspace(0,1,101)
|
||||
axs[0].plot(x,med,label=key,color=colors[key]);axs[0].fill_between(x,lo,hi,color=colors[key],alpha=.1)
|
||||
for j,xx in enumerate(x):curve_rows.append({'metric':key,'fraction_ranked_heads':xx,'p10':lo[j],'median':med[j],'p90':hi[j]})
|
||||
v=np.sort(np.asarray(pool[key]['effective_fraction']));axs[1].plot(v,np.arange(1,len(v)+1)/len(v),label=key,color=colors[key])
|
||||
axs[0].plot([0,1],[0,1],':',color='gray',label='Uniform');axs[0].set(title='Influence concentration curve',xlabel='Fraction of donor heads, largest first',ylabel='Cumulative normalized single-head effects')
|
||||
axs[1].set(title='Continuous effective-head fraction: empirical CDF',xlabel='N_eff / donor heads',ylabel='Fraction of observed cases')
|
||||
for ax in axs:ax.set(xlim=(0,1),ylim=(0,1.02));ax.legend();ax.grid(alpha=.2)
|
||||
fig.suptitle('Observation-wise ranking; dominant identities can differ; shaded p10–p90')
|
||||
csvwrite(root/'concentration-curve.csv',curve_rows);save(fig,'ranked_influence_distribution')
|
||||
fig,axs=plt.subplots(1,2,figsize=(14,4.8),layout='constrained')
|
||||
for label,color in zip(['weak_half','strong_half','random_half','all_heads'],['C0','C3','C1','gray']):
|
||||
for ax,key in zip(axs,['QKV','JS']):
|
||||
selected=[r for r in joint_rows if r['metric']==key and r['removed']==label]
|
||||
xs=sorted(set(r['recipient'] for r in selected));ys=[np.median(joint_by_layer[(key,label,x-1)]) for x in xs]
|
||||
ax.plot(xs,np.maximum(ys,1e-12),'o-',label=label,color=color,markersize=4)
|
||||
axs[0].set(title='Joint removal: balanced QKV relative effect',ylabel='Median token/head relative L2 score')
|
||||
axs[1].set(title='Full-attention recipients: retrieval change',ylabel='Median token/head JS divergence (nats)')
|
||||
for ax in axs:ax.set(xlabel='Recipient layer index',yscale='log');ax.legend();ax.grid(alpha=.2)
|
||||
fig.suptitle('Equal-count weak/strong/random removal; token-wise masks; log display floor 1e-12')
|
||||
save(fig,'joint_removal_effects')
|
||||
fig,axs=plt.subplots(1,2,figsize=(13,4.8),layout='constrained')
|
||||
for key,label,color in [('JS','JS divergence (squared scale)','C4'),('sqrtJS','sqrt(JS) distance','C5')]:
|
||||
a=np.asarray(curves[key]);lo,med,hi=np.quantile(a,[.1,.5,.9],axis=0);x=np.linspace(0,1,101)
|
||||
axs[0].plot(x,med,label=label,color=color);axs[0].fill_between(x,lo,hi,color=color,alpha=.12)
|
||||
for j,xx in enumerate(x):curve_rows.append({'metric':key,'fraction_ranked_heads':xx,'p10':lo[j],'median':med[j],'p90':hi[j]})
|
||||
v=np.sort(np.asarray(pool[key]['effective_fraction']));axs[1].plot(v,np.arange(1,len(v)+1)/len(v),label=label,color=color)
|
||||
axs[0].plot([0,1],[0,1],':',color='gray',label='Uniform');axs[0].set(title='Retrieval influence rank curve',xlabel='Fraction of donor heads, largest first',ylabel='Cumulative normalized intervention effects')
|
||||
axs[1].set(title='Sensitivity to divergence versus distance scale',xlabel='Effective donor-head fraction',ylabel='Fraction of observations')
|
||||
for ax in axs:ax.set(xlim=(0,1),ylim=(0,1.02));ax.legend();ax.grid(alpha=.2)
|
||||
fig.suptitle('Full-attention recipients; sqrt(JS) is post-summary scale sensitivity, not a new model run')
|
||||
csvwrite(root/'concentration-curve.csv',curve_rows);save(fig,'retrieval_scale_sensitivity')
|
||||
lines=['# 上游注意力头对下一层 Q/K/V 的影响集中度','', '实测全部23个相邻层连接、4条既有中英文文本各32 token。主统计取query7–31,按文本/token/接收query头分别计算,再汇总。不是独立样本的显著性检验。','', '影响是移除上游一个头后,下一层Q/K/V向量的相对L2变化;重新计算中间MLP/残差/归一化。各头影响归一化仅构成消融影响分布,不是天然可相加的贡献概率。','', '| 对象 | 有效头比例中位数 [P10,P90] | 最大头份额中位数 | 前25%头份额中位数 | 最小25%头份额中位数 | 90%影响所需头数中位数 |','|---|---:|---:|---:|---:|---:|']
|
||||
for k in ('Q','K','V','QKV','JS','sqrtJS'):
|
||||
if k not in pooled:continue
|
||||
z=pooled[k];a=z['effective_fraction'];lines.append(f"| {k} | {a['median']:.3f} [{a['p10']:.3f},{a['p90']:.3f}] | {z['top1']['median']:.1%} | {z['top25']['median']:.1%} | {z['bottom25']['median']:.1%} | {z['heads90']['median']:.1f} |")
|
||||
lines+=['','sqrtJS为JS距离尺度敏感性分析:在初次完整汇总后增加,原始消融和原JS主结果不变。JS散度近似平方尺度,归一化后会放大强弱差距;JS影响份额不是注意力概率质量份额。','', '按接收层类型分层(避免总体分布混合不同机制):','', '| 接收层类型 | Q有效头比例中位数 | K | V | QKV |','|---|---:|---:|---:|---:|']
|
||||
for label in ('linear_attention','full_attention'):
|
||||
lines.append('| '+label+' | '+' | '.join(format(by_type[k][label]['effective_fraction']['median'],'.3f') for k in ('Q','K','V','QKV'))+' |')
|
||||
lines+=['','QKV是三个相对变化平方的均值再开根号,避免混加不同Q/K/V尺度;JS仅普通注意力接收层有。普通注意力GQA的K/V复制到query头索引用于配对,不是额外独立KV头。','', '| 联合移除 | QKV相对影响中位数 [P10,P90] | 检索JS中位数 [P10,P90] |','|---|---:|---:|']
|
||||
for label in ('weak_half','strong_half','random_half','all_heads'):
|
||||
a=joint_stats['QKV'][label];b=joint_stats['JS'][label];lines.append(f"| {label} | {a['median']:.5f} [{a['p10']:.5f},{a['p90']:.5f}] | {b['median']:.6f} [{b['p10']:.6f},{b['p90']:.6f}] |")
|
||||
lines+=['', '配对观测:QKV弱半数的实际联合影响小于强半数的比例为'+format(paired['QKV']['weak_less_than_strong_fraction'],'.1%')+';强/弱影响比中位数为'+format(paired['QKV']['strong_over_weak']['median'],'.3f')+'。这是逐观测比值的中位数,不是两个总体中位数之比。弱效应为零的观测另计,不做除法。', '', '联合实验的弱/强/随机半数头,在各token、各接收头上分别选择,形成完整序列再重算。排名依据单头QKV影响,同样用于JS;它是输入自适应诊断,不是免费可用的剪枝策略。线性注意力的卷积和普通注意力的QK可能传播跨token影响。所有上游头移除仍保留残差背景,不能把它称为删除全部上游信息。', '', '连续指标:N_eff=1/Σp²;有效头比例=N_eff/H;C=(Σp²−1/H)/(1−1/H)。全零总影响时集中度未定义,原始数组保留NaN并在统计时排除、另计数量。单头相对L2影响在做头间占比归一化之前的总和及最大值均保留,避免把微小效应中的集中误认作重要。字段max_absolute指最大无符号相对效应,并非隐藏坐标中的未归一化L2;total是单头效应之和,真正全头联合效果另测。', '', '联合实验中保存的单头影响之和仅为按query掩码加权的参考。单头消融移除整段序列的一个头,联合掩码则可跨位置变化;二者的差不能单独解释成纯非线性交互,因为时间支持也不同。应直接比较实际弱/强/随机联合效果。', '', '## 文件与限制','', '`raw/*.npz`保留每次单头影响、联合影响、相应单头影响之和、删除掩码及空操作对照;`observations.npz`保留逐观测连续指标及索引;CSV和JSON为全部图表源表。原模型权重和大基线缓存继续保留在服务器。','', '只测短文本上的局部下一层作用;不代表全LM质量、可直接删头、所有输入普遍集中,或相似头可互相替代。所有头贡献可能是广泛分布的;报告不以发现稀疏为成功条件。','',f"CPU测量耗时{status['elapsed_seconds']:.1f}秒,峰值RSS{status['peak_rss_kib']/1024:.1f} MiB;无训练、GPU或清理。"]
|
||||
(root/'REPORT.md').write_text('\n'.join(lines)+'\n')
|
||||
print((root/'REPORT.md').read_text())
|
||||
if __name__=='__main__':
|
||||
import sys
|
||||
main(sys.argv[1])
|
||||
@@ -0,0 +1,84 @@
|
||||
"""All adjacent layers, input-dependent influence distributions; CPU only."""
|
||||
import argparse,json,time,resource,hashlib,traceback
|
||||
from pathlib import Path
|
||||
import numpy as np
|
||||
from tokenizers import Tokenizer
|
||||
from functional.model import StreamingLM
|
||||
from .core import InfluenceLM,concentration,masks_from_influences,js_rows
|
||||
|
||||
|
||||
def dump(path,data):path.write_text(json.dumps(data,indent=2,allow_nan=False)+'\n')
|
||||
|
||||
|
||||
def distances(base,changed):
|
||||
out={}
|
||||
for key in ('Q','K','V'):
|
||||
a=base[key].astype(np.float64);b=changed[key].astype(np.float64)
|
||||
out[key]=np.linalg.norm(b-a,axis=-1)/np.maximum(np.linalg.norm(a,axis=-1),1e-12)
|
||||
out['QKV']=np.sqrt(sum(out[k]**2 for k in ('Q','K','V'))/3)
|
||||
if 'P' in base:out['JS']=js_rows(base['P'],changed['P']).transpose(0,2,1)
|
||||
return out
|
||||
|
||||
|
||||
def link(lm,cache,layer,path,seed):
|
||||
started=time.monotonic();base=lm.recipient(cache['next_raw'],layer+1);hs=cache['heads'].shape[2];t=cache['raw'].shape[1];hr=base['Q'].shape[2]
|
||||
# Whole donor head across the full sequence is the single-ablation intervention.
|
||||
singles={k:[] for k in ('Q','K','V','QKV')+ (('JS',) if 'P' in base else ())}
|
||||
for start in range(0,hs,4):
|
||||
index=list(range(start,min(start+4,hs)));masks=np.zeros((len(index),t,hs),dtype=np.float32)
|
||||
for n,h in enumerate(index):masks[n,:,h]=1
|
||||
scores=distances(base,lm.intervene(cache,layer,masks))
|
||||
for k,v in scores.items():singles[k].extend(v)
|
||||
singles={k:np.asarray(v,dtype=np.float32) for k,v in singles.items()}
|
||||
# Identical cardinality weak/strong/random removals, ranking by balanced QKV influence.
|
||||
# One coherent full-sequence mask per recipient head; only that head's output scored.
|
||||
rank=singles['QKV'];rng=np.random.default_rng(seed)
|
||||
weak=masks_from_influences(rank,True);strong=masks_from_influences(rank,False)
|
||||
random=np.zeros_like(weak)
|
||||
for r in range(hr):
|
||||
for q in range(t):random[r,q,rng.choice(hs,hs//2,replace=False)]=1
|
||||
joint={};sum_individual={}
|
||||
for label,masks in [('weak_half',weak),('strong_half',strong),('random_half',random),('all_heads',np.ones_like(weak))]:
|
||||
values={k:[] for k in singles}
|
||||
for start in range(0,hr,4):
|
||||
stop=min(start+4,hr);scores=distances(base,lm.intervene(cache,layer,masks[start:stop]))
|
||||
for k,v in scores.items():
|
||||
for n,r in enumerate(range(start,stop)):values[k].append(v[n,:,r])
|
||||
for k,v in values.items():
|
||||
joint[label+'_'+k]=np.asarray(v,dtype=np.float32).T
|
||||
sum_individual[label+'_'+k]=np.sum(singles[k].transpose(2,1,0)*masks,axis=-1).T
|
||||
null=distances(base,lm.intervene(cache,layer,np.zeros((1,t,hs),dtype=np.float32)))
|
||||
if max(float(np.max(v)) for v in null.values())>2e-5:raise AssertionError('Null intervention changed recipient')
|
||||
payload={'single_'+k:v for k,v in singles.items()};payload.update({'joint_'+k:v for k,v in joint.items()});payload.update({'individual_sum_'+k:v for k,v in sum_individual.items()})
|
||||
payload.update({'mask_weak_half':weak,'mask_strong_half':strong,'mask_random_half':random})
|
||||
payload.update({'null_'+k:v for k,v in null.items()})
|
||||
np.savez_compressed(path,**payload)
|
||||
return {'source':layer,'recipient':layer+1,'source_heads':hs,'recipient_query_heads':hr,'full_attention_recipient':'P' in base,'elapsed_seconds':time.monotonic()-started,'null_max':max(float(np.max(v)) for v in null.values())}
|
||||
|
||||
|
||||
def main():
|
||||
p=argparse.ArgumentParser();p.add_argument('--source',type=Path,required=True);p.add_argument('--out',type=Path,required=True);p.add_argument('--seconds',type=int,default=7200);a=p.parse_args()
|
||||
a.out.mkdir(parents=True,exist_ok=False);(a.out/'cache').mkdir();(a.out/'raw').mkdir();started=time.monotonic()
|
||||
protocol={'scope':'all 23 adjacent layer transitions; source attention head contribution to next effective Q/K/V','samples':'4 previously authored zh/en texts, first32 tokens, descriptive','primary_queries':list(range(7,32)), 'influence':'L2 change / baseline token recipient-head L2; QKV score sqrt(mean of three squared relative changes))','linear_QK':'after causal convolution/SiLU and delta-rule L2 normalization; Q additionally divided by sqrt(key dim)','full_QK':'QK norm and partial RoPE applied; K/V repeated to query heads for indexing (not independent KV observations)','single_ablation':'remove complete source head projected output at all tokens; recompute source postattention normalization, MLP, residual and next input normalization','joint_ablation':'weak/strong/random half and all attention heads; one coherent token-varying removal mask for each recipient query head; ranking uses QKV single effects; no fitting','continuous':'effective heads=1/sum(p^2); fraction=effective/H; concentration=(sum(p^2)-1/H)/(1-1/H)','zero_total':'undefined concentration (NaN arrays), count explicitly','cost_budget_seconds':a.seconds,'no_training':True,'keep_all_weights_and_caches':True,'limitations':['not final LM quality or pruning proof','individual effects not additive; normalized scores not true contribution probabilities','token-varying masks are input-specific diagnostic and not a deployable fixed head subset','linear attention does not have an ordinary softmax probability matrix','Q/K/V scales handled separately; combined QKV is a declared balanced relative-effect score']}
|
||||
dump(a.out/'protocol.json',protocol);status={'state':'running','completed_links':[]};dump(a.out/'status.json',status)
|
||||
try:
|
||||
lm=InfluenceLM(a.source/'weights',json.loads((a.source/'model_config.json').read_text()),seconds=a.seconds)
|
||||
samples=json.loads(Path('crossqkv/fresh_samples.json').read_text())['samples'];tok=Tokenizer.from_file(str(a.source/'tokenizer.json'));controls=[]
|
||||
for si,s in enumerate(samples):
|
||||
lm.guard();s['token_ids']=tok.encode(s['text']).ids[:32];ids=np.array([s['token_ids']],dtype=np.int64);cache=a.out/'cache'/s['id'];cache.mkdir()
|
||||
final=lm.baseline(ids,cache);np.save(cache/'final.npy',final)
|
||||
for x in lm.head_sum_errors:controls.append({'sample':s['id'],**x})
|
||||
if si==0:
|
||||
reference,_=StreamingLM.forward(lm,ids);normalized=lm.normalization(final,'norm');err=float(np.linalg.norm(normalized-reference)/max(np.linalg.norm(reference),1e-30));assert err<2e-5;status['canonical_forward_relative_error']=err
|
||||
dump(a.out/'head-sum-controls.json',controls);dump(a.out/'samples.json',samples)
|
||||
for layer in range(23):
|
||||
target=a.out/'raw'/f'{s["id"]}-link-{layer:02d}.npz'
|
||||
with np.load(cache/f'layer-{layer}.npz') as data:loaded={k:data[k] for k in data.files}
|
||||
info=link(lm,loaded,layer,target,31000+si*100+layer)
|
||||
del loaded
|
||||
status['completed_links'].append({'sample':s['id'],**info});status['elapsed_seconds']=time.monotonic()-started;dump(a.out/'status.json',status)
|
||||
print('completed',s['id'],layer,'seconds',round(info['elapsed_seconds'],2),flush=True)
|
||||
status.update(state='completed',elapsed_seconds=time.monotonic()-started,peak_rss_kib=resource.getrusage(resource.RUSAGE_SELF).ru_maxrss);dump(a.out/'status.json',status)
|
||||
except BaseException as e:
|
||||
status.update(state='failed',error=repr(e),elapsed_seconds=time.monotonic()-started);dump(a.out/'status.json',status);raise
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,42 @@
|
||||
# Paired task-finetuning comparison (registered before reading finetuned effects)
|
||||
|
||||
Primary checkpoint: Avifenesh/gdn-beta-gate-qwen3.5-2b-grpo,
|
||||
HF revision038d8be9bd0edd3883fa019651b02bdf65609468, pure lora/seed0,
|
||||
step150 after shared40-step warmup. GSM8K GRPO, not SFT. Selected by smallest
|
||||
seed index before observing concentration, not by benchmark score or our metric.
|
||||
No gate01/gate02 arm or architecture changes. Exact base revision matches retained
|
||||
Qwen/Qwen3.5-2B15852e8c16360a2fea060d615a32b45270f8a8fc.
|
||||
|
||||
Download existing18.9MB F32 LoRA tensors, verify official file SHA. All144 tensors
|
||||
must form72 A/B pairs on the4 declared GDN projections of18 linear-attention
|
||||
layers, rank16 alpha32. Apply FP32 effective W +2BA in streamed row chunks, also
|
||||
when decomposing source output heads. Do not quantize merged weights, mutate or
|
||||
save over base weights, execute foreign .pt files, train or use GPU. Validate
|
||||
against separate base-output +2B(Ax) forward and check head sums/null masks.
|
||||
|
||||
Primary comparison exactly matches the reported8.931822/16 statistic: all six
|
||||
ordinary attention receivers3/7/11/15/19/23, previous-layer16 donor output heads,
|
||||
all32 tokens computed, query7..31, original4 plain-text inputs and same token IDs.
|
||||
Reuse corresponding immutable base single-ablation arrays, recompute FT baselines
|
||||
and all16 single donor removals, source MLP/residual/recipient normalization exact.
|
||||
Use the same sqrt(JS) distance influence, normalized shares and N_eff=1/sum(p²).
|
||||
Preserve JS divergence as a sensitivity row; no change of ranking or threshold.
|
||||
All4800 per-token/query-head observations are paired. No new joint-mask search:
|
||||
the requested statistic depends on single removals, not the previous joint tests.
|
||||
|
||||
Secondary task-conditioned comparison: first4 original GSM8K test questions with
|
||||
at least32 base-tokenizer tokens (length selection only, before model effects),
|
||||
question-only plain text, first32 tokens. Freeze original author repository commit,
|
||||
JSONL SHA, item indices, question hashes and exact token IDs. No answers, generation,
|
||||
chat-template change or accuracy claim. Compute both base and FT on these same
|
||||
questions; never compare old general-text base to new GSM8K FT as if paired.
|
||||
|
||||
Report overall median/P10/P90, paired differences, per receiver layer and per text,
|
||||
strong/weak-quartile shares, and dominant donor identity agreement. No iid-token
|
||||
p-values, no claim over training seeds (one fixed trained seed), no proof of
|
||||
GSM8K accuracy improvement or long-context behavior. Positive, negative and near
|
||||
zero changes all count as results. Source cohort remains4+4 short texts.
|
||||
|
||||
All model work/tests/plots on authorized CPU server8.211.154.134, bounded7200s.
|
||||
Preserve all base/FT weights, adapters and caches; no automatic cleanup. Save raw
|
||||
arrays, source provenance, SHA manifests, fixed chart gallery and reviewable PR.
|
||||
@@ -0,0 +1,61 @@
|
||||
# Existing GSM8K fine-tune: paired head-concentration comparison
|
||||
|
||||
First restore the original charts through [the fixed figure gallery](../docs/FIGURE_GALLERY.md).
|
||||
The original 8.931822/16, P10=5.784081 and P90=11.672658 refer to sqrt(JS)
|
||||
influences of all16 prior-layer heads on the six full-attention receivers.
|
||||
|
||||
Selected release: [Avifenesh/gdn-beta-gate-qwen3.5-2b-grpo](https://huggingface.co/Avifenesh/gdn-beta-gate-qwen3.5-2b-grpo),
|
||||
pinned038d8be9bd0edd3883fa019651b02bdf65609468, pure lora/seed0. The publisher
|
||||
records GSM8K GRPO from a shared warmup, rank16 alpha32, exact same original
|
||||
Qwen3.5-2B revision. This is extra RL fine-tuning of the post-trained release,
|
||||
not pretraining-to-SFT, and not a beta-gate architecture modification. Our seed
|
||||
choice is fixed before seeing effects; no new training is performed.
|
||||
|
||||
`lora.py` validates all144 tensors/72 pairs, maps only declared GDN projections,
|
||||
and provides read-only FP32 effective row chunks W+2BA. Source head decomposition
|
||||
uses the same effective weights. `ExplicitLoRA` adds2B(Ax) independently to the
|
||||
base projection as a numerical reference. No BF16 re-rounding is introduced:
|
||||
the author's training code also warns that rounding merged increments can erase
|
||||
small LoRA changes. Author configuration commit/SHA and public HF metadata are
|
||||
saved alongside the protocol; foreign training code and pickle checkpoints are
|
||||
not executed.
|
||||
|
||||
`run.py` reuses the exact original single-ablation arrays for the original cohort,
|
||||
then computes matched finetuned states; it computes both models on4 question-only
|
||||
GSM8K test prefixes as a separate cohort. Every input has32 tokens. All16 donor
|
||||
heads, six receiver layers andquery7..31 remain identical to the original metric.
|
||||
The task corpus is the canonical author's test JSONL at a fixed Git commit, and
|
||||
selection uses only the first4 items with at least32 tokenizer tokens. All item
|
||||
indices, question hashes and token IDs are recorded; answers are not model inputs.
|
||||
|
||||
`report.py` generates paired CDFs, per-layer medians/quantiles, paired change
|
||||
histograms, raw observations and tables. `audit.py` independently reconstructs
|
||||
effective-head counts via(sum d)^2/sum(d^2), checks pairing, exact old baselines,
|
||||
head sums, null masks, adapter/dataset/base SHA and retained external caches.
|
||||
`tests/test_finetune_compare.py` checks streamed row arithmetic, an independent
|
||||
low-rank forward and head decomposition, plus exact zero-scale restoration.
|
||||
|
||||
No iid-token p-values, training-seed generalization, GSM8K accuracy result,
|
||||
long-context claim, pruning recommendation or functional-duplicate claim.
|
||||
The paired delta median is distinguished from the difference of pooled medians.
|
||||
All weights, adapters and caches remain on the authorized CPU server until user
|
||||
explicit deletion. Git stores all raw influence arrays, not the model/adapter.
|
||||
|
||||
Remote workflow on8.211.154.134:
|
||||
```
|
||||
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -B -m unittest tests.test_finetune_compare -v
|
||||
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -B -m finetune_compare.run --source runs/cpu-qwen35-2b --parent runs/head-dominance --assets runs/finetune-head-comparison --out runs/finetune-head-comparison/NEW_EXPERIMENT
|
||||
venv/bin/python -B -m finetune_compare.report runs/finetune-head-comparison/NEW_EXPERIMENT
|
||||
venv/bin/python -B -m finetune_compare.audit runs/finetune-head-comparison/NEW_EXPERIMENT runs/head-dominance runs/finetune-head-comparison runs/cpu-qwen35-2b
|
||||
```
|
||||
|
||||
The supplemental QKV field is measured only on the same six full-attention
|
||||
receivers, so its baseline must not be confused with the older76.0% pooled
|
||||
QKV effective-head fraction across all23 adjacent transitions. The gallery
|
||||
preserves that original table separately; the requested8.931822 metric uses
|
||||
exactly the matching six receivers in both studies.
|
||||
|
||||
The audit also records all72 low-rank update Frobenius norms (via small Gram
|
||||
matrices) and final pre-normalization residual changes per text. These separate
|
||||
a stable concentration statistic from an accidentally inactive adapter. Residual
|
||||
changes are not logit distributions, accuracy differences or capability scores.
|
||||
@@ -0,0 +1,74 @@
|
||||
"""Independent paired effective-head, source, preservation and delivery checks."""
|
||||
import json,hashlib,sys
|
||||
from pathlib import Path
|
||||
import numpy as np
|
||||
|
||||
def sha(path):
|
||||
with Path(path).open('rb') as f:return hashlib.file_digest(f,'sha256').hexdigest()
|
||||
def main(root,parent,assets,source):
|
||||
root,parent,assets,source=map(Path,[root,parent,assets,source]);status=json.loads((root/'status.json').read_text());assert status['state']=='completed' and len(status['links'])==96
|
||||
samples=json.loads((root/'samples.json').read_text());assert set(samples)=={'legacy','gsm8k'} and all(len(v)==4 for v in samples.values());assert all(len(s['token_ids'])==32 for ss in samples.values() for s in ss)
|
||||
assert status['explicit_lora_forward_relative_error']<1e-4;rawstats={};expected=set();obs=dict(np.load(root/'observations.npz'));summary=json.loads((root/'summary.json').read_text())
|
||||
def check(x,record):
|
||||
x=np.asarray(x,dtype=np.float64);x=x[np.isfinite(x)];assert record['count']==len(x)
|
||||
for name,value in zip(('p10','median','p90','mean'),[*np.quantile(x,[.1,.5,.9]),x.mean()]):np.testing.assert_allclose(record[name],value,atol=1e-8,rtol=1e-6)
|
||||
for cohort,ss in samples.items():
|
||||
for role in ('base','finetuned'):
|
||||
for sample in ss:
|
||||
for layer in (2,6,10,14,18,22):
|
||||
name=f'{cohort}-{role}-{sample["id"]}-{layer:02d}.npz';expected.add(name);d=dict(np.load(root/'raw'/name))
|
||||
for key in ('Q','K','V','QKV','JS'):
|
||||
assert d['single_'+key].shape==(16,32,8) and np.isfinite(d['single_'+key]).all() and np.min(d['single_'+key])>=0
|
||||
assert np.max(d['null_'+key])<2e-5
|
||||
if cohort=='legacy' and role=='base':
|
||||
with np.load(parent/'raw'/f'{sample["id"]}-link-{layer:02d}.npz') as original:
|
||||
for key in d:np.testing.assert_array_equal(d[key],original[key])
|
||||
for metric,impacts in [('distance',np.sqrt(d['single_JS'])),('divergence',d['single_JS']),('QKV',d['single_QKV'])]:
|
||||
x=impacts[:,7:,:].transpose(1,2,0).astype(np.float64);total=x.sum(-1);den=np.sum(x*x,-1);assert np.all(den>0)
|
||||
neff=total*total/den
|
||||
rawstats.setdefault((cohort,role,metric),[]).append(neff.ravel())
|
||||
assert expected=={f.name for f in (root/'raw').glob('*.npz')}
|
||||
for cohort in samples:
|
||||
for metric in ('distance','divergence','QKV'):
|
||||
for role in ('base','finetuned'):
|
||||
x=np.concatenate(rawstats[(cohort,role,metric)]);assert len(x)==4800
|
||||
np.testing.assert_allclose(obs['__'.join((cohort,role,metric,'effective_heads'))],x,rtol=1e-8,atol=1e-8)
|
||||
check(x,summary[cohort][metric][role]['effective_heads'])
|
||||
before=np.concatenate(rawstats[(cohort,'base',metric)]);after=np.concatenate(rawstats[(cohort,'finetuned',metric)]);check(after-before,summary[cohort][metric]['paired']['effective_heads_delta'])
|
||||
base=summary['legacy']['distance']['base']['effective_heads'];np.testing.assert_allclose([base[x] for x in ('median','p10','p90')],[8.93182211792415,5.7840811612328125,11.67265834734814],rtol=1e-10)
|
||||
controls=json.loads((root/'head-sum-controls.json').read_text());assert len(controls)==288 and all(c['relative_l2']<2e-5 for c in controls)
|
||||
refs=json.loads((root/'baseline-references.json').read_text());assert len(refs)==24 and all(sha(v['source'])==v['sha256'] for v in refs)
|
||||
assert sha(assets/'adapter.safetensors')=='93a3f43a4356f84da0d13ecc622fee13e41ba15d237cdaad6094f243899c62d6'
|
||||
dataset=json.loads((root/'gsm8k-source.json').read_text());assert sha(assets/'gsm8k-test.jsonl')==dataset['sha256']
|
||||
oldweights=list((source/'weights').glob('*.safetensors'));assert len(oldweights)==1
|
||||
base_sha=sha(oldweights[0]);assert base_sha=='aa33250c4fc64891ddfaba3a314fd9542ea371843c387178b425fbcc5ed680b1'
|
||||
from pilot.tensors import load_tensors
|
||||
import re
|
||||
tensors=load_tensors(assets);pairs={}
|
||||
for name,t in tensors.items():
|
||||
stem,which=name.split('.lora_');pairs.setdefault(stem,{})[which[0]]=t.read().reshape(t.shape).copy();t.evict()
|
||||
updates=[]
|
||||
for name,pair in pairs.items():
|
||||
a=pair['A'].astype(np.float64);b=pair['B'].astype(np.float64)
|
||||
norm2=4*float(np.sum((b.T@b)*(a@a.T).T))
|
||||
updates.append({'module':name,'delta_frobenius':float(np.sqrt(max(norm2,0))),'A_nonzero':int(np.count_nonzero(a)),'B_nonzero':int(np.count_nonzero(b))})
|
||||
assert len(updates)==72 and any(x['delta_frobenius']>0 for x in updates)
|
||||
(root/'weight-update-summary.json').write_text(json.dumps({'scale':2,'modules':updates,'nonzero_update_modules':sum(x['delta_frobenius']>0 for x in updates),'norm_identity':'||2BA||_F^2 = 4 trace((B^T B)(A A^T))'},indent=2)+'\n')
|
||||
changes=[]
|
||||
for cohort,ss in samples.items():
|
||||
for sample in ss:
|
||||
basefile=parent/'cache'/sample['id']/'final.npy' if cohort=='legacy' else root/'cache'/f"{cohort}-base-{sample['id']}"/'final.npy'
|
||||
ftfile=root/'cache'/f"{cohort}-finetuned-{sample['id']}"/'final.npy'
|
||||
before=np.load(basefile).astype(np.float64);after=np.load(ftfile).astype(np.float64)
|
||||
changes.append({'cohort':cohort,'sample':sample['id'],'final_pre_norm_residual_relative_l2':float(np.linalg.norm(after-before)/np.linalg.norm(before))})
|
||||
assert any(x['final_pre_norm_residual_relative_l2']>0 for x in changes)
|
||||
(root/'hidden-state-changes.json').write_text(json.dumps({'definition':'final residual before final RMSNorm, not logits, accuracy or LM quality','changes':changes},indent=2)+'\n')
|
||||
caches=[{'path':str(p.resolve()),'bytes':p.stat().st_size,'sha256':sha(p)} for p in sorted((root/'cache').rglob('*')) if p.is_file()]
|
||||
assert len(caches)==300
|
||||
(root/'external-caches.json').write_text(json.dumps({'host':'8.211.154.134','base_weights':str(oldweights[0].resolve()),'base_sha256':base_sha,'adapter_path':str((assets/'adapter.safetensors').resolve()),'adapter_sha256':sha(assets/'adapter.safetensors'),'caches':caches,'automatic_deletion':False},indent=2)+'\n')
|
||||
files=list(Path('finetune_compare').glob('*.py'))+[Path('finetune_compare/PROTOCOL.md'),Path('tests/test_finetune_compare.py'),Path('dominance/core.py'),Path('dominance/run.py'),Path('functional/model.py')]
|
||||
(root/'source-hashes.json').write_text(json.dumps({str(f):sha(f) for f in files},indent=2)+'\n')
|
||||
proof={'passed':True,'links':96,'paired_observations_per_cohort_metric':4800,'legacy_baseline_exact':True,'effective_head_formula_independently_recomputed':True,'head_sum_checks':288,'explicit_lora_forward_relative_error':status['explicit_lora_forward_relative_error'],'adapter_sha_verified':True,'base_sha_verified':True,'dataset_sha_verified':True,'retained_cache_files':len(caches),'nonzero_update_modules':sum(x['delta_frobenius']>0 for x in updates),'nonzero_hidden_state_changes_verified':True}
|
||||
(root/'audit.json').write_text(json.dumps(proof,indent=2)+'\n');print(json.dumps(proof,indent=2),flush=True)
|
||||
included=[p for p in root.rglob('*') if p.is_file() and 'cache' not in p.relative_to(root).parts and p.name!='checksums.json'];(root/'checksums.json').write_text(json.dumps({str(p.relative_to(root)):sha(p) for p in sorted(included)},indent=2)+'\n')
|
||||
if __name__=='__main__':main(*sys.argv[1:])
|
||||
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"source_repository": "https://github.com/avifenesh/gdn-beta-gate",
|
||||
"source_revision": "f12f61f24ebe5997aac58667f413dcd1dcd96f54",
|
||||
"lora_configuration_path": "src/gdn_gate/broad/arms.py",
|
||||
"lora_configuration_sha256": "fc5b8de096eaa259ab166b8083bb774f9e14859d566ce82ea15986cc8d6eb5b5",
|
||||
"verified_fields": [
|
||||
"LoraConfig r=rank",
|
||||
"lora_alpha=2*rank",
|
||||
"use_rslora not enabled"
|
||||
],
|
||||
"effective_scale": 2
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
93a3f43a4356f84da0d13ecc622fee13e41ba15d237cdaad6094f243899c62d6 lora/seed0/adapter.safetensors
|
||||
b8d4737a5b940bcdfed72258600227a89f883dc863e9724bf84e457b46a53703 lora/seed0/adapter_meta.json
|
||||
63d54e1c2f6bf7d07ca0f6e83a3299bd4b6d00740919bc7cae0b422af05a828e lora/seed1/adapter.safetensors
|
||||
f8be6c4218876b7b12a47945cafe18fcc27d59ff06447db6d3e5b21f0bb9fc29 lora/seed1/adapter_meta.json
|
||||
702b764e2c39b3529914dac1547148ef2f462d9ee04e256201e79f00b20ccc62 lora/seed2/adapter.safetensors
|
||||
9acee331b9383f49a81cfc057163e034f06ea1ba4b8167fea441f0fbfd490f94 lora/seed2/adapter_meta.json
|
||||
7dbf35899fe1186453e599ad8c7a85c56b7fad3d630e371ca026a1e95da2c5a7 lora/seed3/adapter.safetensors
|
||||
b5bfbefa8ce6b428feea761c39a148f003bf02396ca6e01b68abfba2ab3f71af lora/seed3/adapter_meta.json
|
||||
a5cb1e7fa74dff467bd7f163e1db23d30ef8b1a1c935cfa55dcc8cd1b39082a7 lora/seed4/adapter.safetensors
|
||||
23a6c3f198978e79014361a326af20ec364ea266cf9b5e9ced4c9da06feab30e lora/seed4/adapter_meta.json
|
||||
d513d499dfd409da88b3db7da025ba9872fb48ec92d6b29bc2ce7a092a69745d lora/seed5/adapter.safetensors
|
||||
9384d0b4ce4cec6b3dc15756fb4d46c5625a4dacd3805af728fc37b89627d0ed lora/seed5/adapter_meta.json
|
||||
fda9b4c15d283ef8ee778c4416d69b12e1922527ea330bfc67bab5cde1b6741a gate01/seed0/adapter.safetensors
|
||||
131e60bc768412f4d344e5b17cbc811f35d149aad0c328bc6503646513bbd8fc gate01/seed0/adapter_meta.json
|
||||
6ada79d07414641bed58d74fd23090937abcf52337e75e8f06483d502fa64c57 gate01/seed1/adapter.safetensors
|
||||
0350240ed838a93a26f406ef0d63f29ced21ea2284c9430d07aefccd30f8d924 gate01/seed1/adapter_meta.json
|
||||
a9359e3f3be68468604f9bf93f20f4b65114084f3fc6a3c13363e7062c4cf547 gate01/seed2/adapter.safetensors
|
||||
17fa2e876770ec636f8a0a0eb0b3f7b58c91a5f6f9910c09c38c593d856b035f gate01/seed2/adapter_meta.json
|
||||
a2d0c27fee0263c9a8152e0849975edefedc9d10061bbf789a3ead223db8703c gate01/seed3/adapter.safetensors
|
||||
4da32ea52fe6e7c2a0a8858e4d3255540fcc7adf18a7ab2ce9fd3164e4bb0a05 gate01/seed3/adapter_meta.json
|
||||
810b8f8d6147af3620cf08d2cf575c0eb2b0134ae0253a93ceeae5be9dca8695 gate01/seed4/adapter.safetensors
|
||||
f943fc6874af4e4b24763ee6e0cf6e737bc62720045190441d0b33dd561a439f gate01/seed4/adapter_meta.json
|
||||
b34e8d999612415c7395f341633ff3d2365a020b9e27b7e371c5f80f0105a486 gate01/seed5/adapter.safetensors
|
||||
98983ecbf303bf1b10e11c816f850db13af8a0ca5ff2e4efd121f7fd70262918 gate01/seed5/adapter_meta.json
|
||||
1fc5a4a4392c35820c26e24d0075824b3426c4a77f3119713b4a1e577d31df5b gate02/seed0/adapter.safetensors
|
||||
0a17ea0542fb52751d0e628985f8d07240a3ccc4b44a7e2c970e7029c00f6c3f gate02/seed0/adapter_meta.json
|
||||
b742f4ffed0e293f27e4646c364a85ae359a3eae1690d3840e531dca7fba9cd8 gate02/seed1/adapter.safetensors
|
||||
4b6c7bf77edf3049ca904863d59d4d3bfdbe1d0b1b3f563b7fd3bea7f179f97c gate02/seed1/adapter_meta.json
|
||||
dc36c27a98379535007f2df798953624b6a67f0677bd41dec42349f8ce868afa gate02/seed2/adapter.safetensors
|
||||
20009502b1bd6aa9c67b509b77eac0aa0d367af7f347caf4d05c7a51367b8f61 gate02/seed2/adapter_meta.json
|
||||
1dcb6132afeec8101daf9a3fd73d42141fb871bff11fff28069f238dcff6afd0 gate02/seed3/adapter.safetensors
|
||||
518289370e0eb9fc064c3226c2b3811f94cec180a9ae366a026f30cf1d3a3131 gate02/seed3/adapter_meta.json
|
||||
5b97b0601585925c19bf849ecb11ad68597d1b0d55551b917ec0f5091f85c2ff gate02/seed4/adapter.safetensors
|
||||
8419e8e22aa583f97e570950baf37e7c6370856b7cacefaf9c5ebeb0ea4f731c gate02/seed4/adapter_meta.json
|
||||
3f944a6d210958de5840e5b76e263f2105b9f49d15b9ab505950c2b29a5343af gate02/seed5/adapter.safetensors
|
||||
94d9c1e202f7962da61fa060d27941bbef7ff3d23ca5548f5d73fb6f5ac39aac gate02/seed5/adapter_meta.json
|
||||
9108bce2acfcc3ccd38d0dfae67ac33253b95bdc94133bd25deb4daa45d41700 warmup/seed0/adapter.safetensors
|
||||
d2d4346179104830410be9d2776d79983481503daeb5c8383e2a062de43714ef warmup/seed0/adapter_meta.json
|
||||
@@ -0,0 +1,379 @@
|
||||
{
|
||||
"_id": "6accb5a881b0c228f413b5e5",
|
||||
"id": "Avifenesh/gdn-beta-gate-qwen3.5-2b-grpo",
|
||||
"private": false,
|
||||
"pipeline_tag": "text-generation",
|
||||
"library_name": "pytorch",
|
||||
"tags": [
|
||||
"pytorch",
|
||||
"lora",
|
||||
"gated-deltanet",
|
||||
"qwen3.5",
|
||||
"beta-gate",
|
||||
"grpo",
|
||||
"reasoning",
|
||||
"math",
|
||||
"safetensors",
|
||||
"research",
|
||||
"text-generation",
|
||||
"en",
|
||||
"dataset:openai/gsm8k",
|
||||
"base_model:Qwen/Qwen3.5-2B",
|
||||
"base_model:adapter:Qwen/Qwen3.5-2B",
|
||||
"license:apache-2.0",
|
||||
"region:us"
|
||||
],
|
||||
"downloads": 0,
|
||||
"likes": 2,
|
||||
"modelId": "Avifenesh/gdn-beta-gate-qwen3.5-2b-grpo",
|
||||
"author": "Avifenesh",
|
||||
"sha": "038d8be9bd0edd3883fa019651b02bdf65609468",
|
||||
"lastModified": "2026-11-09T19:45:55.000Z",
|
||||
"gated": false,
|
||||
"disabled": false,
|
||||
"widgetData": [
|
||||
{
|
||||
"text": "Hi, what can you help me with?"
|
||||
},
|
||||
{
|
||||
"text": "What is 84 * 3 / 2?"
|
||||
},
|
||||
{
|
||||
"text": "Tell me an interesting fact about the universe!"
|
||||
},
|
||||
{
|
||||
"text": "Explain quantum computing in simple terms."
|
||||
}
|
||||
],
|
||||
"model-index": null,
|
||||
"config": {},
|
||||
"cardData": {
|
||||
"license": "apache-2.0",
|
||||
"base_model": "Qwen/Qwen3.5-2B",
|
||||
"base_model_relation": "adapter",
|
||||
"library_name": "pytorch",
|
||||
"pipeline_tag": "text-generation",
|
||||
"language": [
|
||||
"en"
|
||||
],
|
||||
"datasets": [
|
||||
"openai/gsm8k"
|
||||
],
|
||||
"tags": [
|
||||
"lora",
|
||||
"gated-deltanet",
|
||||
"qwen3.5",
|
||||
"beta-gate",
|
||||
"grpo",
|
||||
"reasoning",
|
||||
"math",
|
||||
"safetensors",
|
||||
"research"
|
||||
]
|
||||
},
|
||||
"siblings": [
|
||||
{
|
||||
"rfilename": ".gitattributes",
|
||||
"blobId": "a6344aac8c09253b3b630fb776ae94478aa0275b",
|
||||
"size": 1519
|
||||
},
|
||||
{
|
||||
"rfilename": "README.md",
|
||||
"blobId": "916419ce9530318d51d71325023fb2a5024475f0",
|
||||
"size": 6527
|
||||
},
|
||||
{
|
||||
"rfilename": "SHA256SUMS",
|
||||
"blobId": "cc8927482e300cb79fed740be333a8ac5639386c",
|
||||
"size": 3700
|
||||
},
|
||||
{
|
||||
"rfilename": "gate01/seed0/adapter.safetensors",
|
||||
"blobId": "86a05f3f118a88940a9122fab59a8e7db73d4b4f",
|
||||
"size": 21293968,
|
||||
"lfs": {
|
||||
"sha256": "fda9b4c15d283ef8ee778c4416d69b12e1922527ea330bfc67bab5cde1b6741a",
|
||||
"size": 21293968,
|
||||
"pointerSize": 133
|
||||
}
|
||||
},
|
||||
{
|
||||
"rfilename": "gate01/seed0/adapter_meta.json",
|
||||
"blobId": "0652fd0ab9647ed184e2c65a670da74de1a24a4e",
|
||||
"size": 661
|
||||
},
|
||||
{
|
||||
"rfilename": "gate01/seed1/adapter.safetensors",
|
||||
"blobId": "596ed265e665aea6f674b81eecfdaaada202be45",
|
||||
"size": 21293968,
|
||||
"lfs": {
|
||||
"sha256": "6ada79d07414641bed58d74fd23090937abcf52337e75e8f06483d502fa64c57",
|
||||
"size": 21293968,
|
||||
"pointerSize": 133
|
||||
}
|
||||
},
|
||||
{
|
||||
"rfilename": "gate01/seed1/adapter_meta.json",
|
||||
"blobId": "934875448487af7efe63e3eae775b24d2d388964",
|
||||
"size": 661
|
||||
},
|
||||
{
|
||||
"rfilename": "gate01/seed2/adapter.safetensors",
|
||||
"blobId": "6413c056f06232ad2ca3d513fa44cd01a90d251b",
|
||||
"size": 21293968,
|
||||
"lfs": {
|
||||
"sha256": "a9359e3f3be68468604f9bf93f20f4b65114084f3fc6a3c13363e7062c4cf547",
|
||||
"size": 21293968,
|
||||
"pointerSize": 133
|
||||
}
|
||||
},
|
||||
{
|
||||
"rfilename": "gate01/seed2/adapter_meta.json",
|
||||
"blobId": "abc653d6a65aa8b3b1f2a18aaf12dc1260409573",
|
||||
"size": 661
|
||||
},
|
||||
{
|
||||
"rfilename": "gate01/seed3/adapter.safetensors",
|
||||
"blobId": "f17c2715463cadb293f36a0fc2c15fb0542990a6",
|
||||
"size": 21293968,
|
||||
"lfs": {
|
||||
"sha256": "a2d0c27fee0263c9a8152e0849975edefedc9d10061bbf789a3ead223db8703c",
|
||||
"size": 21293968,
|
||||
"pointerSize": 133
|
||||
}
|
||||
},
|
||||
{
|
||||
"rfilename": "gate01/seed3/adapter_meta.json",
|
||||
"blobId": "f511293207bf322ce11a4c26dc83cc57b20edf52",
|
||||
"size": 661
|
||||
},
|
||||
{
|
||||
"rfilename": "gate01/seed4/adapter.safetensors",
|
||||
"blobId": "32503f87fd062bf0631c8eac7d8dbba09ef0e6cb",
|
||||
"size": 21293968,
|
||||
"lfs": {
|
||||
"sha256": "810b8f8d6147af3620cf08d2cf575c0eb2b0134ae0253a93ceeae5be9dca8695",
|
||||
"size": 21293968,
|
||||
"pointerSize": 133
|
||||
}
|
||||
},
|
||||
{
|
||||
"rfilename": "gate01/seed4/adapter_meta.json",
|
||||
"blobId": "3fef81591cd7a41c636c39e72e22d5b3394dc6a5",
|
||||
"size": 661
|
||||
},
|
||||
{
|
||||
"rfilename": "gate01/seed5/adapter.safetensors",
|
||||
"blobId": "63b9a6dd509a288e6095ed9531fe76a055e2e5b4",
|
||||
"size": 21293968,
|
||||
"lfs": {
|
||||
"sha256": "b34e8d999612415c7395f341633ff3d2365a020b9e27b7e371c5f80f0105a486",
|
||||
"size": 21293968,
|
||||
"pointerSize": 133
|
||||
}
|
||||
},
|
||||
{
|
||||
"rfilename": "gate01/seed5/adapter_meta.json",
|
||||
"blobId": "d0daa05e6348923237cdac83935118e4f020e9f7",
|
||||
"size": 661
|
||||
},
|
||||
{
|
||||
"rfilename": "gate02/seed0/adapter.safetensors",
|
||||
"blobId": "d566d86611edc89edb7cac62dc73d72619df55bd",
|
||||
"size": 21293968,
|
||||
"lfs": {
|
||||
"sha256": "1fc5a4a4392c35820c26e24d0075824b3426c4a77f3119713b4a1e577d31df5b",
|
||||
"size": 21293968,
|
||||
"pointerSize": 133
|
||||
}
|
||||
},
|
||||
{
|
||||
"rfilename": "gate02/seed0/adapter_meta.json",
|
||||
"blobId": "36173814d1d838077da330e92fa5f0b1b222ba17",
|
||||
"size": 660
|
||||
},
|
||||
{
|
||||
"rfilename": "gate02/seed1/adapter.safetensors",
|
||||
"blobId": "cba04a7ef1bcb48952a723b6aa80d1b6847ef8dc",
|
||||
"size": 21293968,
|
||||
"lfs": {
|
||||
"sha256": "b742f4ffed0e293f27e4646c364a85ae359a3eae1690d3840e531dca7fba9cd8",
|
||||
"size": 21293968,
|
||||
"pointerSize": 133
|
||||
}
|
||||
},
|
||||
{
|
||||
"rfilename": "gate02/seed1/adapter_meta.json",
|
||||
"blobId": "3c7751cefbe375871d0b99bfda0b3ba1e6ca896c",
|
||||
"size": 660
|
||||
},
|
||||
{
|
||||
"rfilename": "gate02/seed2/adapter.safetensors",
|
||||
"blobId": "af547c0fcaa9e69024e14818eb82c46f155dfa59",
|
||||
"size": 21293968,
|
||||
"lfs": {
|
||||
"sha256": "dc36c27a98379535007f2df798953624b6a67f0677bd41dec42349f8ce868afa",
|
||||
"size": 21293968,
|
||||
"pointerSize": 133
|
||||
}
|
||||
},
|
||||
{
|
||||
"rfilename": "gate02/seed2/adapter_meta.json",
|
||||
"blobId": "baaefc1b64b0dce0c87bbc2a7c4342f87111aba6",
|
||||
"size": 660
|
||||
},
|
||||
{
|
||||
"rfilename": "gate02/seed3/adapter.safetensors",
|
||||
"blobId": "21066e00dfc55fc619ca74202ff8ad8dd6017d4e",
|
||||
"size": 21293968,
|
||||
"lfs": {
|
||||
"sha256": "1dcb6132afeec8101daf9a3fd73d42141fb871bff11fff28069f238dcff6afd0",
|
||||
"size": 21293968,
|
||||
"pointerSize": 133
|
||||
}
|
||||
},
|
||||
{
|
||||
"rfilename": "gate02/seed3/adapter_meta.json",
|
||||
"blobId": "f675496569bbec647fcb31784c4a6d9d83872ee0",
|
||||
"size": 660
|
||||
},
|
||||
{
|
||||
"rfilename": "gate02/seed4/adapter.safetensors",
|
||||
"blobId": "bec0713c08368303a25e2ccef690d37a8cf2d54b",
|
||||
"size": 21293968,
|
||||
"lfs": {
|
||||
"sha256": "5b97b0601585925c19bf849ecb11ad68597d1b0d55551b917ec0f5091f85c2ff",
|
||||
"size": 21293968,
|
||||
"pointerSize": 133
|
||||
}
|
||||
},
|
||||
{
|
||||
"rfilename": "gate02/seed4/adapter_meta.json",
|
||||
"blobId": "8535e4793f089b37fa8bf5724cf49ac24fd2750c",
|
||||
"size": 660
|
||||
},
|
||||
{
|
||||
"rfilename": "gate02/seed5/adapter.safetensors",
|
||||
"blobId": "128821f8b3b0591990a3b18c70ca66a16d69aff3",
|
||||
"size": 21293968,
|
||||
"lfs": {
|
||||
"sha256": "3f944a6d210958de5840e5b76e263f2105b9f49d15b9ab505950c2b29a5343af",
|
||||
"size": 21293968,
|
||||
"pointerSize": 133
|
||||
}
|
||||
},
|
||||
{
|
||||
"rfilename": "gate02/seed5/adapter_meta.json",
|
||||
"blobId": "ae2076c513dc6898a8e30f2cf0fafc4486d8f7c3",
|
||||
"size": 660
|
||||
},
|
||||
{
|
||||
"rfilename": "lora/seed0/adapter.safetensors",
|
||||
"blobId": "c3d80937be6f8262769b08a53f5e19548c516420",
|
||||
"size": 18932336,
|
||||
"lfs": {
|
||||
"sha256": "93a3f43a4356f84da0d13ecc622fee13e41ba15d237cdaad6094f243899c62d6",
|
||||
"size": 18932336,
|
||||
"pointerSize": 133
|
||||
}
|
||||
},
|
||||
{
|
||||
"rfilename": "lora/seed0/adapter_meta.json",
|
||||
"blobId": "c67a9c7f3db4db8f73cc71d27d4a8a664967b624",
|
||||
"size": 573
|
||||
},
|
||||
{
|
||||
"rfilename": "lora/seed1/adapter.safetensors",
|
||||
"blobId": "5f0b88be4fa8d1a7fd586ecc640b7ac73fb8a68d",
|
||||
"size": 18932336,
|
||||
"lfs": {
|
||||
"sha256": "63d54e1c2f6bf7d07ca0f6e83a3299bd4b6d00740919bc7cae0b422af05a828e",
|
||||
"size": 18932336,
|
||||
"pointerSize": 133
|
||||
}
|
||||
},
|
||||
{
|
||||
"rfilename": "lora/seed1/adapter_meta.json",
|
||||
"blobId": "2f1ebca329c8579362d80f0e9e731b832a139349",
|
||||
"size": 573
|
||||
},
|
||||
{
|
||||
"rfilename": "lora/seed2/adapter.safetensors",
|
||||
"blobId": "d66df32e3e1e67c4c6b3eccf6229f223fa61a5f8",
|
||||
"size": 18932336,
|
||||
"lfs": {
|
||||
"sha256": "702b764e2c39b3529914dac1547148ef2f462d9ee04e256201e79f00b20ccc62",
|
||||
"size": 18932336,
|
||||
"pointerSize": 133
|
||||
}
|
||||
},
|
||||
{
|
||||
"rfilename": "lora/seed2/adapter_meta.json",
|
||||
"blobId": "7e4737b99685c9f9542f7d15ad4efebc45fc26f3",
|
||||
"size": 573
|
||||
},
|
||||
{
|
||||
"rfilename": "lora/seed3/adapter.safetensors",
|
||||
"blobId": "abb761e518f2bcd7285c7c2cb11579943b942d55",
|
||||
"size": 18932336,
|
||||
"lfs": {
|
||||
"sha256": "7dbf35899fe1186453e599ad8c7a85c56b7fad3d630e371ca026a1e95da2c5a7",
|
||||
"size": 18932336,
|
||||
"pointerSize": 133
|
||||
}
|
||||
},
|
||||
{
|
||||
"rfilename": "lora/seed3/adapter_meta.json",
|
||||
"blobId": "655fd5600b67568bc9da02e4480a14483e0a1067",
|
||||
"size": 573
|
||||
},
|
||||
{
|
||||
"rfilename": "lora/seed4/adapter.safetensors",
|
||||
"blobId": "8bd506ae2220c7d0b025d69a4fbeb9ddaf37198d",
|
||||
"size": 18932336,
|
||||
"lfs": {
|
||||
"sha256": "a5cb1e7fa74dff467bd7f163e1db23d30ef8b1a1c935cfa55dcc8cd1b39082a7",
|
||||
"size": 18932336,
|
||||
"pointerSize": 133
|
||||
}
|
||||
},
|
||||
{
|
||||
"rfilename": "lora/seed4/adapter_meta.json",
|
||||
"blobId": "d8d7b0df8c3d7d1e2f5ef2f2f6e02e22478e2055",
|
||||
"size": 573
|
||||
},
|
||||
{
|
||||
"rfilename": "lora/seed5/adapter.safetensors",
|
||||
"blobId": "487c771e11d8e7ca1fa5bd2670d060e8f77fbaee",
|
||||
"size": 18932336,
|
||||
"lfs": {
|
||||
"sha256": "d513d499dfd409da88b3db7da025ba9872fb48ec92d6b29bc2ce7a092a69745d",
|
||||
"size": 18932336,
|
||||
"pointerSize": 133
|
||||
}
|
||||
},
|
||||
{
|
||||
"rfilename": "lora/seed5/adapter_meta.json",
|
||||
"blobId": "26fc54919eeaffc92a453ddd31cec660c2ab7f0b",
|
||||
"size": 573
|
||||
},
|
||||
{
|
||||
"rfilename": "warmup/seed0/adapter.safetensors",
|
||||
"blobId": "378bc7e4f70afb2e947fc6f53ae822dc4bc422d2",
|
||||
"size": 18932344,
|
||||
"lfs": {
|
||||
"sha256": "9108bce2acfcc3ccd38d0dfae67ac33253b95bdc94133bd25deb4daa45d41700",
|
||||
"size": 18932344,
|
||||
"pointerSize": 133
|
||||
}
|
||||
},
|
||||
{
|
||||
"rfilename": "warmup/seed0/adapter_meta.json",
|
||||
"blobId": "20989af8c7f2ff5959625878167aca5e0fc7203a",
|
||||
"size": 576
|
||||
}
|
||||
],
|
||||
"spaces": [],
|
||||
"createdAt": "2026-10-12T10:25:44.000Z",
|
||||
"usedStorage": 388053976
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"record": "qwen3.5-2b-grpo",
|
||||
"arm": "lora",
|
||||
"receipt_arm": "ctrl",
|
||||
"seed": 0,
|
||||
"step": 150,
|
||||
"base_model": "Qwen/Qwen3.5-2B",
|
||||
"base_revision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
|
||||
"recipe": "GRPO, correctness-gated length penalty alpha 0.2, 150 steps from the shared warm-up (paper Section 4)",
|
||||
"lora_rank": 16,
|
||||
"lora_alpha": 32,
|
||||
"lora_targets": [
|
||||
"in_proj_qkv",
|
||||
"in_proj_a",
|
||||
"in_proj_b",
|
||||
"out_proj"
|
||||
],
|
||||
"gate": null,
|
||||
"n_tensors": 144,
|
||||
"n_params": 4727808,
|
||||
"source_sha256": "de2794c085f5096fea53374ba1cea31075071bb6de3f227bb3350b3035970447"
|
||||
}
|
||||
@@ -0,0 +1,47 @@
|
||||
"""Read-only FP32 effective weights for an existing pure-LoRA checkpoint."""
|
||||
import re
|
||||
import numpy as np
|
||||
from functional.model import PREFIX,StreamingLM
|
||||
from pilot.tensors import load_tensors
|
||||
BASE_REVISION='15852e8c16360a2fea060d615a32b45270f8a8fc'
|
||||
TARGETS={'in_proj_qkv','in_proj_a','in_proj_b','out_proj'}
|
||||
|
||||
class EffectiveWeight:
|
||||
def __init__(self,base,a,b,scale):
|
||||
assert a.ndim==b.ndim==2 and (b.shape[0],a.shape[1])==base.shape and a.shape[0]==b.shape[1]
|
||||
self.base,self.a,self.b,self.scale=base,a,b,scale
|
||||
self.shape,self.numel,self.dtype=base.shape,base.numel,'F32'
|
||||
def read(self,selector=slice(None)):
|
||||
if not isinstance(selector,slice):raise ValueError('Only contiguous row chunks supported')
|
||||
start,stop,step=selector.indices(self.numel);dim=self.shape[1]
|
||||
if step!=1 or start%dim or stop%dim:raise ValueError('Row aligned chunks required')
|
||||
w=self.base.read(slice(start,stop)).reshape(-1,dim)
|
||||
return (w+self.scale*(self.b[start//dim:stop//dim]@self.a)).astype(np.float32).reshape(-1)
|
||||
def evict(self):self.base.evict()
|
||||
|
||||
|
||||
def install(lm,directory,meta,multiplier=1.):
|
||||
assert meta['base_model']=='Qwen/Qwen3.5-2B' and meta['base_revision']==BASE_REVISION and meta['gate'] is None and meta['arm']=='lora'
|
||||
tensors=load_tensors(directory);pairs={};pattern=re.compile(r'^base_model\.model\.model\.(layers\.(\d+)\.linear_attn\.(in_proj_qkv|in_proj_a|in_proj_b|out_proj))\.lora_([AB])\.default\.weight$')
|
||||
for name,tensor in tensors.items():
|
||||
match=pattern.fullmatch(name)
|
||||
if not match:raise ValueError('Unexpected trainable tensor: '+name)
|
||||
target=PREFIX+match[1]+'.weight';assert lm.config['layer_types'][int(match[2])]=='linear_attention'
|
||||
assert target in lm.tensors
|
||||
value=tensor.read().reshape(tensor.shape).copy();tensor.evict();assert np.isfinite(value).all()
|
||||
pairs.setdefault(target,{})[match[4]]=value
|
||||
expected={PREFIX+f'layers.{layer}.linear_attn.{role}.weight' for layer,kind in enumerate(lm.config['layer_types']) if kind=='linear_attention' for role in TARGETS}
|
||||
assert set(pairs)==expected and len(tensors)==meta['n_tensors']
|
||||
scale=meta['lora_alpha']/meta['lora_rank']*multiplier
|
||||
for target,pair in pairs.items():
|
||||
assert set(pair)=={'A','B'} and pair['A'].shape[0]==meta['lora_rank']
|
||||
lm.tensors[target]=EffectiveWeight(lm.tensors[target],pair['A'],pair['B'],scale)
|
||||
return pairs,scale
|
||||
|
||||
|
||||
class ExplicitLoRA(StreamingLM):
|
||||
"""Independent reference: base projection + scaled low-rank output, no W merge."""
|
||||
def linear(self,x,name,rows=256):
|
||||
y=super().linear(x,name,rows);pair=self.pairs.get(PREFIX+name+'.weight')
|
||||
if pair is not None:y=y+self.scale*((x@pair['A'].T)@pair['B'].T)
|
||||
return y.astype(np.float32)
|
||||
@@ -0,0 +1,73 @@
|
||||
"""Paired concentration changes from saved immutable intervention arrays."""
|
||||
import csv,json,sys
|
||||
from pathlib import Path
|
||||
import numpy as np
|
||||
import matplotlib
|
||||
matplotlib.use('Agg')
|
||||
import matplotlib.pyplot as plt
|
||||
from dominance.core import concentration
|
||||
|
||||
def stats(x):
|
||||
x=np.asarray(x,dtype=np.float64);v=x[np.isfinite(x)];q=np.quantile(v,[.1,.5,.9])
|
||||
return {'count':int(v.size),'p10':float(q[0]),'median':float(q[1]),'p90':float(q[2]),'mean':float(v.mean())}
|
||||
def csvwrite(path,rows):
|
||||
with path.open('w') as f:w=csv.DictWriter(f,fieldnames=list(rows[0]));w.writeheader();w.writerows(rows)
|
||||
def main(root):
|
||||
root=Path(root);status=json.loads((root/'status.json').read_text());assert status['state']=='completed';samples=json.loads((root/'samples.json').read_text());pool={};observations={};summary={};rows=[];layer_rows=[]
|
||||
for cohort in ('legacy','gsm8k'):
|
||||
for role in ('base','finetuned'):
|
||||
for sample in samples[cohort]:
|
||||
for source in (2,6,10,14,18,22):
|
||||
data=np.load(root/'raw'/f'{cohort}-{role}-{sample["id"]}-{source:02d}.npz')
|
||||
for metric,key,transform in [('distance','JS',np.sqrt),('divergence','JS',lambda x:x),('QKV','QKV',lambda x:x)]:
|
||||
impacts=transform(data['single_'+key])[:,7:,:].transpose(1,2,0);c=concentration(impacts)
|
||||
record={'cohort':cohort,'role':role,'sample':sample['id'],'recipient':source+1,'metric':metric}
|
||||
for name,x in c.items():
|
||||
observations.setdefault((cohort,role,metric,name),[]).append(x.reshape(-1))
|
||||
if name!='valid':
|
||||
for stat,value in stats(x).items():record[name+'_'+stat]=value
|
||||
pool.setdefault((cohort,role,metric,'recipient'),[]).append(np.full(200,source+1))
|
||||
pool.setdefault((cohort,role,metric,'sample'),[]).append(np.full(200,sample['id']))
|
||||
observations.setdefault((cohort,role,metric,'winner'),[]).append(np.argmax(impacts,-1).reshape(-1))
|
||||
rows.append(record)
|
||||
arrays={k:np.concatenate(v) for k,v in observations.items()};np.savez_compressed(root/'observations.npz',**{'__'.join(k):v for k,v in arrays.items()})
|
||||
for cohort in ('legacy','gsm8k'):
|
||||
summary[cohort]={}
|
||||
for metric in ('distance','divergence','QKV'):
|
||||
item={role:{name:stats(arrays[(cohort,role,metric,name)]) for name in ('effective_heads','effective_fraction','concentration','top1','top25','bottom25','heads90','total')} for role in ('base','finetuned')}
|
||||
before=arrays[(cohort,'base',metric,'effective_heads')];after=arrays[(cohort,'finetuned',metric,'effective_heads')];delta=after-before
|
||||
item['paired']={'effective_heads_delta':stats(delta),'increased_fraction':float(np.mean(delta>0)),'decreased_fraction':float(np.mean(delta<0)),'winner_agreement':float(np.mean(arrays[(cohort,'base',metric,'winner')]==arrays[(cohort,'finetuned',metric,'winner')]))}
|
||||
summary[cohort][metric]=item
|
||||
recipients=np.concatenate(pool[(cohort,'base',metric,'recipient')]);names=np.concatenate(pool[(cohort,'base',metric,'sample')])
|
||||
for layer in (3,7,11,15,19,23):
|
||||
which=recipients==layer;a=stats(before[which]);b=stats(after[which]);d=stats(delta[which]);layer_rows.append({'cohort':cohort,'metric':metric,'recipient':layer,'base_median':a['median'],'base_p10':a['p10'],'base_p90':a['p90'],'finetuned_median':b['median'],'finetuned_p10':b['p10'],'finetuned_p90':b['p90'],'paired_delta_median':d['median']})
|
||||
for sample in samples[cohort]:
|
||||
which=names==sample['id'];item.setdefault('per_text',{})[sample['id']]={'base':stats(before[which]),'finetuned':stats(after[which]),'paired_delta':stats(delta[which])}
|
||||
expected=(8.93182211792415,5.7840811612328125,11.67265834734814);b=summary['legacy']['distance']['base']['effective_heads'];np.testing.assert_allclose([b[k] for k in ['median','p10','p90']],expected,rtol=1e-10)
|
||||
(root/'summary.json').write_text(json.dumps(summary,indent=2)+'\n');csvwrite(root/'per-text-layer.csv',rows);csvwrite(root/'per-layer.csv',layer_rows)
|
||||
figs=root/'figures';figs.mkdir(exist_ok=True)
|
||||
def save(fig,name):
|
||||
for ext in ('png','pdf','svg'):fig.savefig(figs/(name+'.'+ext),dpi=180)
|
||||
plt.close(fig)
|
||||
fig,axs=plt.subplots(1,2,figsize=(13,4.8),layout='constrained')
|
||||
for ax,cohort in zip(axs,['legacy','gsm8k']):
|
||||
for role,label,color in [('base','Original release','C0'),('finetuned','GSM8K GRPO LoRA seed0','C1')]:
|
||||
x=np.sort(arrays[(cohort,role,'distance','effective_heads')]);ax.plot(x,np.arange(1,len(x)+1)/len(x),label=label,color=color)
|
||||
ax.set(title='Original four texts' if cohort=='legacy' else 'Four GSM8K held-out question prefixes',xlabel='Effective donor heads: sqrt(JS) influence',ylabel='Empirical CDF',xlim=(1,16),ylim=(0,1));ax.grid(alpha=.2);ax.legend()
|
||||
fig.suptitle('Same inputs before/after existing task fine-tuning; 32 tokens; one training seed; no training here');save(fig,'paired_effective_heads_cdf')
|
||||
fig,axs=plt.subplots(1,2,figsize=(13,4.8),layout='constrained')
|
||||
for ax,cohort in zip(axs,['legacy','gsm8k']):
|
||||
for role,prefix,color in [('base','base','C0'),('finetuned','finetuned','C1')]:
|
||||
rr=[r for r in layer_rows if r['cohort']==cohort and r['metric']=='distance'];x=[r['recipient'] for r in rr];med=[r[prefix+'_median'] for r in rr];lo=[r[prefix+'_p10'] for r in rr];hi=[r[prefix+'_p90'] for r in rr];ax.plot(x,med,'o-',color=color,label=role);ax.fill_between(x,lo,hi,color=color,alpha=.12)
|
||||
ax.set(title=cohort,xlabel='Full-attention recipient layer index',ylabel='Effective donor heads: median / P10–P90',ylim=(1,16));ax.legend();ax.grid(alpha=.2)
|
||||
fig.suptitle('All six full-attention receivers, sixteen donor heads each; descriptive dependent observations');save(fig,'paired_effective_heads_by_layer')
|
||||
fig,axs=plt.subplots(1,2,figsize=(13,4.8),layout='constrained')
|
||||
for ax,cohort in zip(axs,['legacy','gsm8k']):
|
||||
delta=arrays[(cohort,'finetuned','distance','effective_heads')]-arrays[(cohort,'base','distance','effective_heads')];ax.hist(delta,bins=60,density=True,alpha=.7);ax.axvline(0,color='black',linestyle='--');ax.axvline(np.median(delta),color='C3',label='Paired median');ax.set(title=cohort,xlabel='Finetuned minus original effective head count',ylabel='Density');ax.legend()
|
||||
fig.suptitle('Paired changes; positive = more distributed, negative = more concentrated');save(fig,'paired_effective_heads_delta')
|
||||
lines=['# GSM8K任务微调前后的有效头数','', '比较原Qwen3.5-2B发布权重与同一revision上的纯LoRA seed0(额外GSM8K GRPO,非SFT)。没有训练新模型。主指标为√JS距离消融影响的N_eff;正变化表示更分散,负变化表示更集中。','', '| 输入组 | 原权重中位数 [P10,P90] | 微调后中位数 [P10,P90] | 配对变化中位数 | 增加的观测比例 |','|---|---:|---:|---:|---:|']
|
||||
for cohort,label in [('legacy','原四条文本'),('gsm8k','GSM8K测试题前缀')]:
|
||||
item=summary[cohort]['distance'];a=item['base']['effective_heads'];b=item['finetuned']['effective_heads'];d=item['paired']['effective_heads_delta'];lines.append(f"| {label} | {a['median']:.6f} [{a['p10']:.6f},{a['p90']:.6f}] | {b['median']:.6f} [{b['p10']:.6f},{b['p90']:.6f}] | {d['median']:+.6f} | {item['paired']['increased_fraction']:.1%} |")
|
||||
lines+=['', '每组4条输入×6个接收层×25个query位置×8个接收query头=4800个配对观测;它们不是4800个独立文本样本。前后使用同一原模型tokenizer、完全相同token IDs。原四条输入的基线8.931822/16已精确复现。','', 'GSM8K输入取原作者固定版本test split中前4道至少32token的题目,仅用题目前32token,不含答案,不套不同聊天模板。另算了同组GSM8K输入的原权重基线;没有把旧通用文本与新任务文本混作配对。','', '## 方法与限定','', '所有6个普通注意力接收层和16个上游输出头均保留;每次移除一个上游头在全序列的投影输出,重算中间MLP/残差/归一化和检索分布。微调只改变18个GDN层的指定投影,普通注意力权重不变;隐藏状态和上游贡献仍会变化。','', '采用read-only FP32有效权重W+2BA,未把增量舍入为BF16、未修改底座;另用base projection+2B(Ax)独立前向校验。头输出求和、空消融和源数组复用校验均记录。JS散度及QKV复合相对变化也保留,避免只展示有利度量。这里QKV补充指标仅覆盖相同六个普通注意力接收层,其原基线不能与旧研究全部23连接的76.0%混作同一统计量。','', '单个发布者训练seed、4条短文本/组,仅支持这些输入上的描述性变化;不推广为所有微调都会如此,不报告GSM8K准确率,不据此推断删头可用或最终LM能力。配对变化中位数与两个总体中位数之差不是同一个统计量。','', '[公开微调权重与训练说明](https://huggingface.co/Avifenesh/gdn-beta-gate-qwen3.5-2b-grpo);固定版本、参数SHA、dataset版本和题目索引见protocol.json/gsm8k-source.json/samples.json。所有权重、适配器和缓存继续保留。']
|
||||
(root/'REPORT.md').write_text('\n'.join(lines)+'\n');print((root/'REPORT.md').read_text(),end='')
|
||||
if __name__=='__main__':main(sys.argv[1])
|
||||
@@ -0,0 +1,72 @@
|
||||
"""Paired CPU-only existing-adapter comparison for the published sqrt(JS) metric."""
|
||||
import argparse,json,hashlib,time,resource,gc
|
||||
from pathlib import Path
|
||||
import numpy as np
|
||||
from tokenizers import Tokenizer
|
||||
from dominance.core import InfluenceLM,concentration
|
||||
from dominance.run import distances
|
||||
from .lora import install,ExplicitLoRA,BASE_REVISION
|
||||
SOURCES=(2,6,10,14,18,22)
|
||||
|
||||
def dump(p,d):p.write_text(json.dumps(d,indent=2,allow_nan=False)+'\n')
|
||||
def single(lm,cache,layer,path):
|
||||
base=lm.recipient(cache['next_raw'],layer+1);hs=cache['heads'].shape[2];t=cache['raw'].shape[1];assert hs==16
|
||||
scores={k:[] for k in ('Q','K','V','QKV','JS')}
|
||||
for start in range(0,hs,4):
|
||||
masks=np.zeros((4,t,hs),dtype=np.float32)
|
||||
for n in range(4):masks[n,:,start+n]=1
|
||||
effects=distances(base,lm.intervene(cache,layer,masks))
|
||||
for k,v in effects.items():scores[k].extend(v)
|
||||
null=distances(base,lm.intervene(cache,layer,np.zeros((1,t,hs),dtype=np.float32)))
|
||||
assert max(float(np.max(v)) for v in null.values())<2e-5
|
||||
payload={'single_'+k:np.asarray(v,dtype=np.float32) for k,v in scores.items()};payload.update({'null_'+k:v for k,v in null.items()});np.savez_compressed(path,**payload)
|
||||
|
||||
def main():
|
||||
p=argparse.ArgumentParser();p.add_argument('--source',type=Path,required=True);p.add_argument('--parent',type=Path,required=True);p.add_argument('--assets',type=Path,required=True);p.add_argument('--out',type=Path,required=True);a=p.parse_args();a.out.mkdir(parents=True,exist_ok=False);(a.out/'cache').mkdir();(a.out/'raw').mkdir()
|
||||
started=time.monotonic();meta=json.loads((a.assets/'adapter_meta.json').read_text());assert meta['base_revision']==BASE_REVISION
|
||||
adapter_sha=hashlib.sha256((a.assets/'adapter.safetensors').read_bytes()).hexdigest();assert adapter_sha=='93a3f43a4356f84da0d13ecc622fee13e41ba15d237cdaad6094f243899c62d6'
|
||||
tok=Tokenizer.from_file(str(a.source/'tokenizer.json'));legacy=json.loads((a.parent/'samples.json').read_text());gsm=[]
|
||||
for s in legacy:assert s['token_ids']==tok.encode(s['text']).ids[:32]
|
||||
for index,line in enumerate((a.assets/'gsm8k-test.jsonl').read_text().splitlines()):
|
||||
q=json.loads(line)['question'];ids=tok.encode(q).ids
|
||||
if len(ids)<32:continue
|
||||
gsm.append({'id':f'gsm8k-test-{index:04d}','row_index':index,'text':q,'question_sha256':hashlib.sha256(q.encode()).hexdigest(),'token_ids':ids[:32]})
|
||||
if len(gsm)==4:break
|
||||
assert len(gsm)==4
|
||||
cohorts={'legacy':legacy,'gsm8k':gsm};dump(a.out/'samples.json',cohorts);dump(a.out/'adapter_meta.json',meta);dump(a.out/'gsm8k-source.json',json.loads((a.assets/'gsm8k-source.json').read_text()))
|
||||
protocol={'primary':'same six full-attention receivers and original4x32 token cohort as published8.931822/16','adapter_repo':'Avifenesh/gdn-beta-gate-qwen3.5-2b-grpo','adapter_revision':'038d8be9bd0edd3883fa019651b02bdf65609468','adapter_file':'lora/seed0/adapter.safetensors','adapter_sha256':adapter_sha,'base_revision':BASE_REVISION,'scale':2,'rank':16,'arm':'pure LoRA','training':'publisher-reported GSM8K GRPO; we do no training','sample_choice':'four original texts; first4 canonical GSM8K test questions with >=32tokens, length selection only, first32tokens; question only','source_layers':SOURCES,'primary_queries':list(range(7,32)),'main_metric':'concentration(sqrt(single_JS)); same FP32 tensors and sequence-wide donor-head deletion as prior run','cohort_pairing':'each comparison uses same question/tokenIDs before and after; base legacy arrays reused with elementwise checks','runtime':'read-only FP32 effective W+2BA; no BF16 re-rounding or quantization','controls':'head sum, null mask, explicit low-rank versus effective-weight full forward relL2<1e-4','budget_seconds':7200,'keep_all_weights_and_caches':True,'limitations':['one existing training seed','4 texts per cohort, 32 tokens each','dependent query/head observations, not iid textual samples','no generation or accuracy measurement','GDN targets only; ordinary attention weights stay unchanged','no pruning or functional interchangeability claim']}
|
||||
dump(a.out/'protocol.json',protocol);status={'state':'running','links':[]};dump(a.out/'status.json',status);cfg=json.loads((a.source/'model_config.json').read_text());controls=[];references=[]
|
||||
try:
|
||||
for cohort,samples in cohorts.items():
|
||||
for role in ('base','finetuned'):
|
||||
reuse=cohort=='legacy' and role=='base';lm=None
|
||||
if not reuse:
|
||||
lm=InfluenceLM(a.source/'weights',cfg,seconds=max(1,7200-(time.monotonic()-started)))
|
||||
if role=='finetuned':pairs,scale=install(lm,a.assets,meta)
|
||||
for si,s in enumerate(samples):
|
||||
if time.monotonic()-started>7200:raise TimeoutError('Paired CPU budget')
|
||||
if not reuse:
|
||||
cache=a.out/'cache'/f'{cohort}-{role}-{s["id"]}';cache.mkdir();ids=np.array([s['token_ids']],dtype=np.int64);final=lm.baseline(ids,cache);np.save(cache/'final.npy',final)
|
||||
controls.extend({'cohort':cohort,'role':role,'sample':s['id'],**v} for v in lm.head_sum_errors)
|
||||
if cohort=='legacy' and role=='finetuned' and si==0:
|
||||
ref=ExplicitLoRA(a.source/'weights',cfg,seconds=max(1,7200-(time.monotonic()-started)));ref.pairs,ref.scale=pairs,scale;expected,_=ref.forward(ids);actual=lm.normalization(final,'norm');error=float(np.linalg.norm(actual.astype(np.float64)-expected)/np.linalg.norm(expected.astype(np.float64)));assert error<1e-4
|
||||
status['explicit_lora_forward_relative_error']=error;del ref;gc.collect()
|
||||
for layer in SOURCES:
|
||||
out=a.out/'raw'/f'{cohort}-{role}-{s["id"]}-{layer:02d}.npz'
|
||||
if reuse:
|
||||
old=a.parent/'raw'/f'{s["id"]}-link-{layer:02d}.npz'
|
||||
with np.load(old) as d:payload={prefix+k:d[prefix+k] for prefix in ('single_','null_') for k in ('Q','K','V','QKV','JS')}
|
||||
np.savez_compressed(out,**payload)
|
||||
with np.load(out) as d:
|
||||
for k,v in payload.items():np.testing.assert_array_equal(d[k],v)
|
||||
references.append({'source':str(old.resolve()),'sha256':hashlib.sha256(old.read_bytes()).hexdigest(),'derived':out.name})
|
||||
else:
|
||||
with np.load(cache/f'layer-{layer}.npz') as d:loaded={k:d[k] for k in d.files}
|
||||
single(lm,loaded,layer,out);del loaded
|
||||
status['links'].append({'cohort':cohort,'role':role,'sample':s['id'],'source':layer,'path':str(out.relative_to(a.out))});status['elapsed_seconds']=time.monotonic()-started;dump(a.out/'status.json',status);print('completed',cohort,role,s['id'],layer,flush=True)
|
||||
dump(a.out/'head-sum-controls.json',controls);dump(a.out/'baseline-references.json',references)
|
||||
if lm is not None:del lm;gc.collect()
|
||||
status.update(state='completed',elapsed_seconds=time.monotonic()-started,peak_rss_kib=resource.getrusage(resource.RUSAGE_SELF).ru_maxrss);dump(a.out/'status.json',status)
|
||||
except BaseException as e:
|
||||
status.update(state='failed',error=repr(e),elapsed_seconds=time.monotonic()-started);dump(a.out/'status.json',status);raise
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,91 @@
|
||||
# Bounded CPU functional pilot
|
||||
|
||||
This is the execution implementation of the CPU-first plan, not the completed static
|
||||
parameter pilot. Frozen Qwen3.5-2B weights are decoded to FP32 a projection block at a
|
||||
time. CPU only, no autograd/optimization, no rotation fitting, no model-hub Python code.
|
||||
The user explicitly requires retaining this checkpoint until they request deletion.
|
||||
|
||||
## Fixed scope
|
||||
|
||||
- Batch of eight distinct experiment-authored plain texts, first 32 tokens each:
|
||||
four calibration and four heldout, two Chinese/two English in each split.
|
||||
- Full attention pairs: (3,7), (11,15), (19,23).
|
||||
- MLP pairs: (5,6), (20,21).
|
||||
- Smaller-granularity Q/K RMSNorm pairs inside layers 3,7,11,15,19,23.
|
||||
Added before observing real weights/functions for this pilot; these are small storage
|
||||
opportunities, not a claim of substantial compute savings.
|
||||
- At most one calibration-selected candidate, two separate substitution directions.
|
||||
Selection prefers component size, then worst symmetric output error. No heldout-driven retry.
|
||||
- Same-input relative L2 <=0.05 in both directions and token P95 <=0.1 in both directions;
|
||||
attention additionally requires mean JS <=0.01 and top-5 overlap >=0.8.
|
||||
- Heldout mean NLL increase <=0.01 nat/token and per-sequence P95 increase <=0.05.
|
||||
These are pilot decision criteria, not mathematical capability guarantees.
|
||||
|
||||
## Source and checks
|
||||
|
||||
`reference.json` pins the official Transformers Qwen3.5 source commit, URL and hash.
|
||||
`reference_check.py` extracts only named mathematical definitions from that exact file,
|
||||
removes optional fused-kernel decorators, and compares the NumPy implementation against
|
||||
Torch operators, full toy decoder layers and a two-layer text forward. Chunked delta and
|
||||
recurrent delta implementations are both checked, including sequence padding boundaries.
|
||||
The official source is Apache-2.0, copyright Qwen Team and HuggingFace; the reference file
|
||||
is fetched for testing and is not bundled here. FP32 equivalence is checked, not equality
|
||||
to every deployed BF16 kernel.
|
||||
|
||||
`tests/test_functional.py` independently checks streamed projection/embedding/NLL,
|
||||
causality, causal convolution, normalization conventions and basic metric identities.
|
||||
These tests use tiny synthetic arrays. Real model evidence is produced only by `run.py`.
|
||||
|
||||
## Remote execution
|
||||
|
||||
Use the previously authorized CPU host and existing NumPy/Matplotlib environment. Install
|
||||
only binary wheels: tokenizers==0.23.3 from HTTPS PyPI; torch==2.15.0+cpu from the official
|
||||
CPU wheel index for independent reference checks. Use an experiment-owned on-disk TMPDIR:
|
||||
the shared host's /tmp tmpfs was too small during the first wheel download. Do not change
|
||||
system swap or shared temporary files.
|
||||
|
||||
```sh
|
||||
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -B -m unittest tests.test_functional -v
|
||||
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -B -m functional.reference_check --source runs/cpu-qwen35-2b/reference.py --output runs/cpu-qwen35-2b/reference-check.json
|
||||
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -B -m functional.supervise --root runs/cpu-qwen35-2b
|
||||
```
|
||||
|
||||
The supervisor waits at most one hour for binary dependencies and four hours for the
|
||||
already started pinned, verified checkpoint download. It does not install, retry downloads,
|
||||
search extra candidates, mutate Git or delete weights. After reference checks, it runs one
|
||||
8-token real text capacity check, then the bounded functional pilot. Reference, preflight
|
||||
and functional compute together are capped at two hours with a reporting reserve;
|
||||
download wait is recorded separately. Projection memory is checked, RSS soft stop is
|
||||
512 MiB, available-system-memory floor 256 MiB; sustained host swap IO combined with low available memory or >128 MiB process swap also stops the run.
|
||||
A persistent ledger counts computation across restarts; early checks/preflight receive a conservative 180-second reserve.
|
||||
|
||||
`preflight.py` is a capacity smoke check, not a semantic oracle. `run.py` records exact
|
||||
sample/token IDs, fixed configuration, source hashes, activations, per-text/per-head metrics,
|
||||
selection, per-token NLL, intervention results and truthful stop coverage. `report.py` reads
|
||||
saved data to produce PNG/PDF/SVG figures without reopening weights.
|
||||
|
||||
No generation throughput is measured in this short-text pilot. Any parameter-tying storage
|
||||
benefit must be accounted for separately, including metadata. Neither same-input similarity
|
||||
nor favorable tiny heldout NLL demonstrates broad mutual interchangeability or GPU speedup.
|
||||
|
||||
## Current primary result and completed extensions
|
||||
|
||||
The latest user goal is learned-function repetition, independent of costs. Existing
|
||||
input RMSNorm modules at layer indices 12/13 have a proven worst two-direction real-arithmetic
|
||||
error bound of 11.3402%, worst observed heldout relative error 4.5285%, minimum cosine 0.999487,
|
||||
and small bidirectional heldout NLL changes. See [findings](../docs/CPU_FUNCTIONAL_FINDINGS.md).
|
||||
|
||||
- `neuron_reuse.py`: fixed four-layer scalar MLP function search, nontrivial activity
|
||||
gates, calibration-only selection and independent heldout/substitution logic; no pair passed.
|
||||
- `norm_repetition.py`: same-role learned RMSNorm bound across 552 pairs, one deterministic
|
||||
selection independent of any input samples, real probes and two full-network substitutions.
|
||||
- `gain_factorization.py`: architecture-derived identity on non-RoPE coordinates; a separate
|
||||
reparameterization result, not a learned-function repetition example.
|
||||
- `benchmark_factorization.py`: historical one-attempt CPU allocation optimization and
|
||||
bitwise equivalence check. Performance is no longer a success condition.
|
||||
- `storage.py`: explicit component layout accounting; the original candidate run selected
|
||||
no pair. This does not affect the primary function-repetition conclusion.
|
||||
|
||||
Do not expand the final positive claim to full attention or MLP bodies. Retain the
|
||||
fixed checkpoint until the user explicitly requests deletion. Full tests and audit are
|
||||
run remotely; no local build or model execution is required to inspect the archived results.
|
||||
@@ -0,0 +1 @@
|
||||
"""CPU-only, training-free functional reuse pilot."""
|
||||
@@ -0,0 +1,86 @@
|
||||
"""Audit saved functional evidence and archive only outputs, never checkpoint weights."""
|
||||
import argparse
|
||||
import csv
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import platform
|
||||
import subprocess
|
||||
import time
|
||||
import numpy as np
|
||||
from .model import StreamingLM
|
||||
from .norm_repetition import uniform_bounds
|
||||
from .run import write_json
|
||||
|
||||
EXCLUDE={'tokenizer.json','reference.py','checksums-functional.json','audit-validation.json','archive-files.json'}
|
||||
|
||||
|
||||
def archive_files(root):
|
||||
files=[p for p in root.iterdir() if p.is_file() and p.name not in EXCLUDE and not p.name.endswith(('.partial','.safetensors'))]
|
||||
files += [p for p in (root/'figures').glob('*') if p.is_file()]
|
||||
return sorted(files)
|
||||
|
||||
|
||||
def main():
|
||||
p=argparse.ArgumentParser();p.add_argument('--root',type=Path,required=True);a=p.parse_args();root=a.root
|
||||
manifest=json.loads((root/'manifest.json').read_text())
|
||||
assert manifest['revision']=='15852e8c16360a2fea060d615a32b45270f8a8fc'
|
||||
for f in manifest['files']:
|
||||
if f['name'].endswith('.safetensors'):
|
||||
path=root/'weights'/f['name']
|
||||
assert path.is_file() and path.stat().st_size==f['bytes']
|
||||
assert f['sha256']==f['upstream_sha256']
|
||||
retention=json.loads((root/'retention-policy.json').read_text());assert retention['automatic_cleanup'] is False
|
||||
reference=json.loads((root/'reference-check.json').read_text())
|
||||
assert reference['passed'] and len(reference['tests'])==22
|
||||
assert reference['implementation_sha256']==hashlib.sha256(Path('functional/model.py').read_bytes()).hexdigest()
|
||||
tests=(root/'final-tests.txt').read_text();assert 'Ran 12 tests' in tests and '\nOK\n' in tests
|
||||
result=json.loads((root/'norm-repetition-summary.json').read_text());assert result['state']=='completed'
|
||||
assert result['learned_function_repetition_positive'] and result['bidirectional_network_quality_pass']
|
||||
cfg=json.loads((root/'model_config.json').read_text());lm=StreamingLM(root/'weights',cfg,seconds=60)
|
||||
c=result['selected'];ga=1+lm.small(f'layers.{c["a"]}.{c["role"]}.weight');gb=1+lm.small(f'layers.{c["b"]}.{c["role"]}.weight')
|
||||
bounds=uniform_bounds(ga,gb)
|
||||
np.testing.assert_allclose(bounds,[c['a_to_b_uniform_bound'],c['b_to_a_uniform_bound']],atol=1e-14,rtol=1e-14)
|
||||
with (root/'norm-repetition-uniform-bounds.csv').open() as f:assert len(list(csv.DictReader(f)))==552
|
||||
baseline=np.load(root/'baseline-heldout-token-nll.npy');assert baseline.shape==(4,31)
|
||||
for rec in result['interventions']:
|
||||
nll=np.load(root/f'norm-repetition-{rec["target"]}-uses-{rec["donor"]}-token-nll.npy')
|
||||
delta=nll.mean(1)-baseline.mean(1)
|
||||
np.testing.assert_allclose(delta,rec['sequence_nll_deltas'],atol=1e-13,rtol=1e-13)
|
||||
assert abs(delta.mean()-rec['mean_nll_delta'])<1e-13
|
||||
with (root/'norm-repetition-function-metrics.csv').open() as f:rows=list(csv.DictReader(f))
|
||||
held=[r for r in rows if r['split']=='heldout'];assert len(held)==8
|
||||
worst=max(max(float(r['relative_l2']),float(r['reverse_relative_l2'])) for r in held)
|
||||
assert abs(worst-result['worst_heldout_relative_l2'])<1e-14
|
||||
neuron=json.loads((root/'neuron-reuse-search-summary.json').read_text());assert sum(x['eligible_unique_pairs_checked'] for x in neuron)==7450139
|
||||
ledger=json.loads((root/'compute-budget-ledger.json').read_text());assert ledger['spent_seconds']+600<7200
|
||||
environment={'python':platform.python_version(),'platform':platform.platform(),'numpy':np.__version__,
|
||||
'packages':subprocess.run([str(Path(__import__('sys').executable)),'-m','pip','freeze'],capture_output=True,text=True,check=True).stdout.splitlines(),
|
||||
'weights_retained':True,'no_automatic_cleanup':True,'ledger_seconds_including_initial_reserve':ledger['spent_seconds'],
|
||||
'final_tests_report_audit_additional_budget_upper_reserve_seconds':600,
|
||||
'historical_error_logs':'Early reference adapter errors and preflight pressure stops; latest summaries/tests are authoritative.'}
|
||||
write_json(root/'functional-environment.json',environment)
|
||||
source_paths=[*Path('functional').glob('*.py'),Path('functional/reference.json'),Path('pilot/tensors.py'),Path('tests/test_functional.py')]
|
||||
write_json(root/'final-source-hashes.json',{str(p):hashlib.sha256(p.read_bytes()).hexdigest() for p in sorted(source_paths)})
|
||||
files=archive_files(root)
|
||||
# Every numeric CSV value must be finite. Empty diagnostic fields stay unknown.
|
||||
for path in files:
|
||||
if path.suffix=='.csv':
|
||||
with path.open() as f:
|
||||
for row in csv.DictReader(f):
|
||||
for value in row.values():
|
||||
if value in (None,''):continue
|
||||
try:number=float(value)
|
||||
except ValueError:continue
|
||||
assert np.isfinite(number),(path,value)
|
||||
checksums={str(p.relative_to(root)):hashlib.sha256(p.read_bytes()).hexdigest() for p in files}
|
||||
write_json(root/'checksums-functional.json',checksums)
|
||||
selected=[*checksums,'checksums-functional.json','audit-validation.json','archive-files.json']
|
||||
write_json(root/'archive-files.json',selected)
|
||||
write_json(root/'audit-validation.json',{'passed':True,'validated_unix':time.time(),'checksummed_files':len(checksums),
|
||||
'proof_recomputed_from_retained_weights':True,'heldout_nll_recomputed_from_saved_arrays':True,
|
||||
'weights_present_and_retained':True,'archive_excludes_weights_and_tokenizer':True})
|
||||
print(json.dumps({'passed':True,'checksummed_files':len(checksums),'weights_retained':True}))
|
||||
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,83 @@
|
||||
"""One allocation-reduction attempt, with a similarly optimized baseline.
|
||||
|
||||
Does not change weights or run the network again. Checks bitwise equality to the
|
||||
network-tested factorization on every captured real Q/K input before timing.
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
import resource
|
||||
import time
|
||||
import numpy as np
|
||||
from .model import StreamingLM
|
||||
from .gain_factorization import consolidate,normalized,LAYERS
|
||||
from .run import save_csv,write_json
|
||||
|
||||
|
||||
def baseline_inplace(q,k,a,b):
|
||||
q=normalized(q);q*=a
|
||||
k=normalized(k);k*=b
|
||||
return q,k
|
||||
|
||||
|
||||
def factored_inplace(q,k,a,b,r,combined,move='q_to_k'):
|
||||
q,k=normalized(q),normalized(k)
|
||||
if move=='q_to_k':
|
||||
q[...,:r]*=a[:r];k*=combined
|
||||
else:
|
||||
q*=combined;k[...,:r]*=b[:r]
|
||||
return q,k
|
||||
|
||||
|
||||
def main():
|
||||
p=argparse.ArgumentParser();p.add_argument('--root',required=True,type=Path);a=p.parse_args();root=a.root
|
||||
ledger_path=root/'compute-budget-ledger.json';ledger=json.loads(ledger_path.read_text());remaining=int(7200-ledger['spent_seconds']-20)
|
||||
if remaining<60:raise TimeoutError('Cumulative budget insufficient')
|
||||
start=time.monotonic();result={'state':'running'}
|
||||
try:
|
||||
cfg=json.loads((root/'model_config.json').read_text());lm=StreamingLM(root/'weights',cfg,seconds=min(remaining,300))
|
||||
r=64;data=np.load(root/'baseline-activations.npz');checks=[]
|
||||
for layer in LAYERS:
|
||||
lm.guard();pfx=f'layers.{layer}.self_attn.'
|
||||
ag,bg=1+lm.small(pfx+'q_norm.weight'),1+lm.small(pfx+'k_norm.weight')
|
||||
q,k=data[f'qraw_{layer}'],data[f'kraw_{layer}']
|
||||
for move in ('q_to_k','k_to_q'):
|
||||
combined=np.concatenate([(bg if move=='q_to_k' else ag)[:r],ag[r:]*bg[r:]])
|
||||
expected=consolidate(q,k,ag,bg,r,move)
|
||||
actual=factored_inplace(q,k,ag,bg,r,combined,move)
|
||||
for x,y in zip(expected,actual):np.testing.assert_array_equal(x,y)
|
||||
checks.append({'layer':layer,'direction':move,'query_elements':int(q.size),'key_elements':int(k.size),'bitwise_equal':True})
|
||||
q,k=data['qraw_3'][:1],data['kraw_3'][:1];pfx='layers.3.self_attn.'
|
||||
ag,bg=1+lm.small(pfx+'q_norm.weight'),1+lm.small(pfx+'k_norm.weight')
|
||||
combined=np.concatenate([bg[:r],ag[r:]*bg[r:]])
|
||||
funcs={'baseline_legacy':lambda:(normalized(q)*ag,normalized(k)*bg),
|
||||
'baseline_inplace':lambda:baseline_inplace(q,k,ag,bg),
|
||||
'factored_inplace':lambda:factored_inplace(q,k,ag,bg,r,combined)}
|
||||
for _ in range(3):
|
||||
for fn in funcs.values():fn()
|
||||
timing=[];names=list(funcs)
|
||||
for repeat in range(30):
|
||||
order=names[repeat%3:]+names[:repeat%3]
|
||||
for name in order:
|
||||
lm.guard();t=time.perf_counter()
|
||||
for _ in range(50):funcs[name]()
|
||||
timing.append({'repeat':repeat,'implementation':name,'seconds_per_call':(time.perf_counter()-t)/50,'calls_per_block':50})
|
||||
save_csv(root/'gain-factorization-optimized-timing.csv',timing)
|
||||
values={name:np.array([row['seconds_per_call'] for row in timing if row['implementation']==name]) for name in names}
|
||||
ratio=values['baseline_inplace']/values['factored_inplace']
|
||||
medians={name:float(np.median(v)) for name,v in values.items()}
|
||||
result.update(state='completed',bitwise_checks=checks,medians_seconds=medians,
|
||||
paired_baseline_over_factored_median=float(np.median(ratio)),
|
||||
paired_ratio_p10=float(np.quantile(ratio,.1)),paired_ratio_p90=float(np.quantile(ratio,.9)),
|
||||
fraction_blocks_factored_faster=float(np.mean(ratio>1)),
|
||||
descriptive_local_gain=bool(np.median(ratio)>1.05 and np.mean(ratio>1)>=.8 and np.quantile(ratio,.1)>1),
|
||||
claim_limit='Hot-array CPU normalization microbenchmark against equally allocation-optimized baseline; no full-model speedup or independent statistical significance claim.')
|
||||
except Exception as exc:
|
||||
result.update(state='stopped',error_type=type(exc).__name__,error=str(exc));raise
|
||||
finally:
|
||||
elapsed=time.monotonic()-start;result.update(elapsed_seconds=elapsed,peak_rss_kib=resource.getrusage(resource.RUSAGE_SELF).ru_maxrss)
|
||||
ledger=json.loads(ledger_path.read_text());ledger['spent_seconds']+=elapsed;ledger['calls'].append({'module':'functional.benchmark_factorization','seconds':elapsed});write_json(ledger_path,ledger)
|
||||
write_json(root/'gain-factorization-optimized-summary.json',result)
|
||||
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,135 @@
|
||||
"""Architecture-derived Q/K gain consolidation on non-RoPE dimensions only.
|
||||
|
||||
Fixed layer3 interventions; no fitting, layer selection or learned alignment.
|
||||
Run only after baseline/candidate pilot finishes, within the same cumulative budget.
|
||||
"""
|
||||
import argparse
|
||||
import csv
|
||||
import json
|
||||
from pathlib import Path
|
||||
import resource
|
||||
import time
|
||||
import numpy as np
|
||||
from .model import StreamingLM,norm,rope
|
||||
from .run import probability_metrics,write_json,save_csv
|
||||
|
||||
LAYERS=[3,7,11,15,19,23]
|
||||
|
||||
|
||||
def normalized(x):
|
||||
return x/np.sqrt(np.mean(x*x,axis=-1,keepdims=True)+np.float32(1e-6))
|
||||
|
||||
|
||||
def consolidate(q,k,a,b,r,move,joint=None):
|
||||
q,k=normalized(q),normalized(k)
|
||||
c=a[r:]*b[r:] if joint is None else joint
|
||||
if move=='q_to_k':
|
||||
q=np.concatenate([q[...,:r]*a[:r],q[...,r:]],axis=-1)
|
||||
k=np.concatenate([k[...,:r]*b[:r],k[...,r:]*c],axis=-1)
|
||||
else:
|
||||
q=np.concatenate([q[...,:r]*a[:r],q[...,r:]*c],axis=-1)
|
||||
k=np.concatenate([k[...,:r]*b[:r],k[...,r:]],axis=-1)
|
||||
return q,k
|
||||
|
||||
|
||||
class FactoredLM(StreamingLM):
|
||||
def configure(self,layer,move):
|
||||
self.factor_layer=layer;self.move=move
|
||||
p=f'layers.{layer}.self_attn.'
|
||||
self.a=1+self.small(p+'q_norm.weight');self.b=1+self.small(p+'k_norm.weight')
|
||||
self.r=int(self.config['head_dim']*self.config['rope_parameters']['partial_rotary_factor'])
|
||||
self.joint=self.a[self.r:]*self.b[self.r:]
|
||||
|
||||
def normalization(self,x,name):
|
||||
p=f'layers.{self.factor_layer}.self_attn.'
|
||||
if name not in (p+'q_norm',p+'k_norm'):return super().normalization(x,name)
|
||||
# This path does not multiply the unit-gain tail, unlike a full-vector mask.
|
||||
unit=x/np.sqrt(np.mean(x*x,axis=-1,keepdims=True)+self.eps)
|
||||
isq=name==p+'q_norm';head=self.a if isq else self.b
|
||||
skip=(isq and self.move=='q_to_k') or (not isq and self.move=='k_to_q')
|
||||
tail=unit[...,self.r:] if skip else unit[...,self.r:]*self.joint
|
||||
return np.concatenate([unit[...,:self.r]*head[:self.r],tail],axis=-1)
|
||||
|
||||
|
||||
def main():
|
||||
p=argparse.ArgumentParser();p.add_argument('--root',required=True,type=Path);a=p.parse_args();root=a.root
|
||||
# Protocol is fixed before inspecting functional measurements.
|
||||
protocol={'fixed_intervention_layer':3,'directions':['q_to_k','k_to_q'],'diagnostic_layers':LAYERS,'no_selection':True,'no_training':True,'precision':'FP32 product; no BF16 requantization','quality_gate':{'mean_nll_delta':.01,'sequence_p95_nll_delta':.05}}
|
||||
protocol_path=root/'gain-factorization-protocol.json'
|
||||
if protocol_path.exists():
|
||||
old=json.loads(protocol_path.read_text());assert all(old[k]==v for k,v in protocol.items()), 'Protocol differs from preregistration'
|
||||
else:write_json(protocol_path,{**protocol,'registered_unix':time.time()})
|
||||
ledger_path=root/'compute-budget-ledger.json';ledger=json.loads(ledger_path.read_text())
|
||||
remaining=int(7200-ledger['spent_seconds']-30)
|
||||
if remaining<60:raise TimeoutError('No remaining cumulative budget for deterministic identity verification')
|
||||
start=time.monotonic()
|
||||
report={'state':'running','records':[],'interventions':[]}
|
||||
try:
|
||||
cfg=json.loads((root/'model_config.json').read_text());c=cfg['text_config'];r=int(c['head_dim']*c['rope_parameters']['partial_rotary_factor'])
|
||||
assert c['rms_norm_eps']==1e-6
|
||||
assert c['head_dim']==256 and r==64 and c['num_attention_heads']==8 and c['num_key_value_heads']==2
|
||||
lm=FactoredLM(root/'weights',cfg,seconds=remaining)
|
||||
data=np.load(root/'baseline-activations.npz');samples=json.loads((root/'samples-tokenized.json').read_text())['samples']
|
||||
rp=c['rope_parameters'];repeats=c['num_attention_heads']//c['num_key_value_heads']
|
||||
records=[]
|
||||
for layer in LAYERS:
|
||||
lm.guard()
|
||||
pfx=f'layers.{layer}.self_attn.'
|
||||
aw=lm.small(pfx+'q_norm.weight');bw=lm.small(pfx+'k_norm.weight');ag,bg=1+aw,1+bw
|
||||
q,k=data[f'qraw_{layer}'],data[f'kraw_{layer}']
|
||||
qr,kr=rope(norm(q,aw),rp['rope_theta'],rp['partial_rotary_factor']),rope(norm(k,bw),rp['rope_theta'],rp['partial_rotary_factor'])
|
||||
ref=np.einsum('bthd,bshd->bhts',qr,np.repeat(kr,repeats,axis=2))/np.float32(np.sqrt(c['head_dim']))
|
||||
mask=np.triu(np.ones(ref.shape[-2:],dtype=bool),1)
|
||||
def probs(s):
|
||||
s=np.where(mask,-np.inf,s);e=np.exp(s-s.max(-1,keepdims=True));return e/e.sum(-1,keepdims=True)
|
||||
pref=probs(ref)
|
||||
for move in ('q_to_k','k_to_q'):
|
||||
qq,kk=consolidate(q,k,ag,bg,r,move)
|
||||
qq,kk=rope(qq,rp['rope_theta'],rp['partial_rotary_factor']),rope(kk,rp['rope_theta'],rp['partial_rotary_factor'])
|
||||
changed=np.einsum('bthd,bshd->bhts',qq,np.repeat(kk,repeats,axis=2))/np.float32(np.sqrt(c['head_dim']))
|
||||
pp=probs(changed)
|
||||
for bi,s in enumerate(samples):
|
||||
diff=changed[bi].astype(np.float64)-ref[bi]
|
||||
records.append({'layer':layer,'direction':move,'sample_id':s['id'],'split':s['split'],
|
||||
'score_max_abs_error':float(np.abs(diff).max()),'score_relative_l2':float(np.linalg.norm(diff)/max(np.linalg.norm(ref[bi].astype(np.float64)),1e-12)),
|
||||
'probability_max_abs_error':float(np.abs(pp[bi]-pref[bi]).max()),**probability_metrics(pref[bi:bi+1],pp[bi:bi+1])})
|
||||
save_csv(root/'gain-factorization-score-metrics.csv',records)
|
||||
held=np.array([i for i,s in enumerate(samples) if s['split']=='heldout']);ids=np.array([s['token_ids'] for s in samples],dtype=np.int64)
|
||||
base=np.load(root/'baseline-heldout-token-nll.npy').mean(1)
|
||||
for move in ('q_to_k','k_to_q'):
|
||||
lm.configure(3,move)
|
||||
h,_=lm.forward(ids[held])
|
||||
nll=lm.nll(h,ids[held]);np.save(root/f'gain-factorization-{move}-token-nll.npy',nll)
|
||||
delta=nll.mean(1)-base
|
||||
report['interventions'].append({'layer':3,'direction':move,'sequence_nll_deltas':delta.tolist(),'mean_nll_delta':float(delta.mean()),'sequence_p95_delta':float(np.quantile(delta,.95)),'passed':bool(delta.mean()<=.01 and np.quantile(delta,.95)<=.05)})
|
||||
write_json(root/'gain-factorization-summary.json',report)
|
||||
# Local normalization timing uses precomputed gains for both implementations.
|
||||
q,k=data['qraw_3'][:1],data['kraw_3'][:1];pfx='layers.3.self_attn.'
|
||||
ag,bg=1+lm.small(pfx+'q_norm.weight'),1+lm.small(pfx+'k_norm.weight')
|
||||
def baseline():return normalized(q)*ag,normalized(k)*bg
|
||||
joint=ag[r:]*bg[r:]
|
||||
def factored():return consolidate(q,k,ag,bg,r,'q_to_k',joint=joint)
|
||||
for _ in range(3):baseline();factored()
|
||||
timing=[]
|
||||
for repeat in range(10):
|
||||
for label,fn in (('baseline',baseline),('factored',factored)) if repeat%2==0 else (('factored',factored),('baseline',baseline)):
|
||||
lm.guard();t=time.perf_counter()
|
||||
for _ in range(20):fn()
|
||||
timing.append({'repeat':repeat,'implementation':label,'seconds_per_call':(time.perf_counter()-t)/20})
|
||||
save_csv(root/'gain-factorization-kernel-timing.csv',timing)
|
||||
report.update(state='completed',maximum_score_abs_error=max(x['score_max_abs_error'] for x in records),maximum_probability_abs_error=max(x['probability_max_abs_error'] for x in records),
|
||||
bidirectional_heldout_pass=all(x['passed'] for x in report['interventions']),
|
||||
elementwise_gain_multiplies_saved_per_token_layer={'q_to_k':1536,'k_to_q':384},
|
||||
original_bf16_payload_bytes=1024,factored_lossless_mixed_payload_bytes=1024,
|
||||
fp32_cached_gain_payload_before=2048,fp32_cached_gain_payload_after=1280,
|
||||
claim_limit='Exact real-arithmetic identity only on non-RoPE coordinates; empirical FP32/NLL checks on tiny text pilot. Not evidence that two existing full layers interchange, not model-wide speedup.')
|
||||
except Exception as exc:
|
||||
report.update(state='stopped',error_type=type(exc).__name__,error=str(exc));raise
|
||||
finally:
|
||||
elapsed=time.monotonic()-start
|
||||
report.update(elapsed_seconds=elapsed,peak_rss_kib=resource.getrusage(resource.RUSAGE_SELF).ru_maxrss)
|
||||
ledger=json.loads(ledger_path.read_text());ledger['spent_seconds']+=elapsed;ledger['calls'].append({'module':'functional.gain_factorization','seconds':elapsed});write_json(ledger_path,ledger)
|
||||
write_json(root/'gain-factorization-summary.json',report)
|
||||
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,247 @@
|
||||
"""Streaming FP32 text-only Qwen3.5 reference, original BF16 weights.
|
||||
|
||||
Mathematical semantics checked against Transformers Qwen3.5, commit
|
||||
fd3cf0140de03f7619ac95b5cfed05b711ff9e22 (Apache-2.0, Qwen Team/HuggingFace).
|
||||
No cached decode, padding, images, generation, fused kernels or autograd.
|
||||
Full-sequence gated delta recurrence replaces the algebraically equivalent
|
||||
chunk algorithm; both are independently compared in remote tests.
|
||||
"""
|
||||
import ctypes
|
||||
import json
|
||||
import time
|
||||
from pathlib import Path
|
||||
import numpy as np
|
||||
from pilot.tensors import load_tensors, decode
|
||||
|
||||
PREFIX = 'model.language_model.'
|
||||
|
||||
|
||||
def sigmoid(x):
|
||||
return np.exp(-np.logaddexp(np.float32(0), -x))
|
||||
|
||||
|
||||
def silu(x):
|
||||
return x * sigmoid(x)
|
||||
|
||||
|
||||
def norm(x, weight, eps=1e-6, centered=True):
|
||||
return (x / np.sqrt(np.mean(x*x, axis=-1, keepdims=True) + eps)
|
||||
* (1 + weight if centered else weight)).astype(np.float32)
|
||||
|
||||
|
||||
def rope(x, theta=10000000., fraction=.25):
|
||||
# Text-only: all three MRoPE position axes coincide.
|
||||
dim = int(x.shape[-1] * fraction)
|
||||
inv = np.float32(1) / np.float32(theta) ** (np.arange(0, dim, 2, dtype=np.float32) / dim)
|
||||
freq = np.arange(x.shape[1], dtype=np.float32)[:, None] * inv[None, :]
|
||||
angle = np.concatenate([freq, freq], axis=-1)[None, :, None, :]
|
||||
a = x[..., :dim]
|
||||
rotated = np.concatenate([-a[..., dim//2:], a[..., :dim//2]], axis=-1)
|
||||
return np.concatenate([a*np.cos(angle) + rotated*np.sin(angle), x[..., dim:]], axis=-1)
|
||||
|
||||
|
||||
def delta_rule(q, k, v, g, beta):
|
||||
q = q / np.sqrt(np.sum(q*q, axis=-1, keepdims=True) + 1e-6)
|
||||
k = k / np.sqrt(np.sum(k*k, axis=-1, keepdims=True) + 1e-6)
|
||||
q = q / np.float32(np.sqrt(q.shape[-1]))
|
||||
state = np.zeros((q.shape[0], q.shape[2], q.shape[3], v.shape[3]), dtype=np.float32)
|
||||
out = np.empty_like(v)
|
||||
for t in range(q.shape[1]):
|
||||
state *= np.exp(g[:, t, :, None, None])
|
||||
predicted = np.sum(state * k[:, t, :, :, None], axis=-2)
|
||||
delta = (v[:, t] - predicted) * beta[:, t, :, None]
|
||||
state += k[:, t, :, :, None] * delta[:, :, None, :]
|
||||
out[:, t] = np.sum(state * q[:, t, :, :, None], axis=-2)
|
||||
return out, state
|
||||
|
||||
|
||||
def causal_conv(x, kernel):
|
||||
out = np.zeros_like(x)
|
||||
width = kernel.shape[-1]
|
||||
for j in range(width):
|
||||
lag = width - 1 - j
|
||||
if lag < x.shape[1]:
|
||||
out[:, lag:] += x[:, :x.shape[1]-lag] * kernel[:, j]
|
||||
return silu(out)
|
||||
|
||||
|
||||
def attention(q, k, v):
|
||||
repeats = q.shape[2] // k.shape[2]
|
||||
k, v = np.repeat(k, repeats, axis=2), np.repeat(v, repeats, axis=2)
|
||||
scores = np.einsum('bthd,bshd->bhts', q, k) / np.float32(np.sqrt(q.shape[-1]))
|
||||
mask = np.triu(np.ones(scores.shape[-2:], dtype=bool), 1)
|
||||
scores = np.where(mask, -np.inf, scores)
|
||||
exp = np.exp(scores - scores.max(axis=-1, keepdims=True))
|
||||
p = exp / exp.sum(axis=-1, keepdims=True)
|
||||
return np.einsum('bhts,bshd->bthd', p, v), p
|
||||
|
||||
|
||||
class StreamingLM:
|
||||
def __init__(self, weights, config, seconds=7200, enforce_memory=True):
|
||||
self.tensors = load_tensors(Path(weights))
|
||||
self.config = config.get('text_config', config)
|
||||
self.deadline = time.monotonic() + seconds
|
||||
self.enforce_memory = enforce_memory
|
||||
self.eps = self.config['rms_norm_eps']
|
||||
self.norm_replacement = None
|
||||
self.capture_qk = False
|
||||
self.qk_inputs = {}
|
||||
self.swap_observation = None
|
||||
self.swap_pressure_windows = 0
|
||||
self.malloc_trim = getattr(ctypes.CDLL(None), 'malloc_trim', None)
|
||||
|
||||
def guard(self):
|
||||
if time.monotonic() > self.deadline:
|
||||
raise TimeoutError('CPU computation deadline reached')
|
||||
if not self.enforce_memory:
|
||||
return
|
||||
status = dict(line.split(':', 1) for line in Path('/proc/self/status').read_text().splitlines() if ':' in line)
|
||||
rss_kib = int(status['VmRSS'].split()[0])
|
||||
available = int(next(line.split()[1] for line in Path('/proc/meminfo').read_text().splitlines() if line.startswith('MemAvailable:')))
|
||||
now = time.monotonic()
|
||||
if self.swap_observation is None or now-self.swap_observation[0] >= 5:
|
||||
counters = dict(line.split() for line in Path('/proc/vmstat').read_text().splitlines())
|
||||
count = int(counters['pswpin'])+int(counters['pswpout'])
|
||||
if self.swap_observation is not None:
|
||||
elapsed = now-self.swap_observation[0]
|
||||
rate = (count-self.swap_observation[1])*4096/elapsed
|
||||
self.swap_pressure_windows = self.swap_pressure_windows+1 if rate > 4*1024**2 else 0
|
||||
own_swap_kib = int(status.get('VmSwap', '0 kB').split()[0])
|
||||
if self.swap_pressure_windows >= 3 and (available < 384*1024 or own_swap_kib > 128*1024):
|
||||
raise MemoryError(f'Sustained host swap pressure plus constrained memory: available={available} KiB, process_swap={own_swap_kib} KiB')
|
||||
self.swap_observation = (now,count)
|
||||
if rss_kib > 512*1024 or available < 256*1024:
|
||||
raise MemoryError(f'Soft memory stop: RSS {rss_kib} KiB, available {available} KiB')
|
||||
|
||||
def small(self, name):
|
||||
t = self.tensors[PREFIX + name]
|
||||
if t.numel > 1048576:
|
||||
raise ValueError('Use blockwise linear for large tensors')
|
||||
x = t.read().reshape(t.shape)
|
||||
t.evict()
|
||||
return x
|
||||
|
||||
def linear(self, x, name, rows=256):
|
||||
t = self.tensors[PREFIX + name + '.weight']
|
||||
outdim, indim = t.shape
|
||||
flat = x.reshape(-1, indim)
|
||||
result = np.empty((flat.shape[0], outdim), dtype=np.float32)
|
||||
for start in range(0, outdim, rows):
|
||||
self.guard()
|
||||
end = min(start + rows, outdim)
|
||||
w = t.read(slice(start*indim, end*indim)).reshape(end-start, indim)
|
||||
result[:, start:end] = flat @ w.T
|
||||
del w
|
||||
t.evict()
|
||||
if self.malloc_trim is not None:
|
||||
self.malloc_trim(0) # Release only this process's freed temporary heap pages.
|
||||
return result.reshape(*x.shape[:-1], outdim)
|
||||
|
||||
def embedding(self, ids):
|
||||
t = self.tensors[PREFIX + 'embed_tokens.weight']
|
||||
raw = t.raw.reshape(t.shape)
|
||||
result = decode(raw[ids], t.dtype)
|
||||
t.evict()
|
||||
return result
|
||||
|
||||
def normalization(self, x, name):
|
||||
if getattr(self, 'norm_replacement', None) and name == self.norm_replacement[0]:
|
||||
name = self.norm_replacement[1]
|
||||
return norm(x, self.small(name + '.weight'), self.eps)
|
||||
|
||||
def mlp(self, h, layer):
|
||||
p = f'layers.{layer}.mlp.'
|
||||
gate = silu(self.linear(h, p+'gate_proj'))
|
||||
up = self.linear(h, p+'up_proj')
|
||||
return self.linear(gate*up, p+'down_proj')
|
||||
|
||||
def full_attention(self, h, layer, shared_p=None):
|
||||
p = f'layers.{layer}.self_attn.'
|
||||
c = self.config
|
||||
heads, kvheads, dim = c['num_attention_heads'], c['num_key_value_heads'], c['head_dim']
|
||||
# Q and output gate are interleaved per head, not globally split.
|
||||
mixed = self.linear(h, p+'q_proj').reshape(*h.shape[:-1], heads, 2*dim)
|
||||
q, gate = mixed[..., :dim], mixed[..., dim:]
|
||||
k = self.linear(h, p+'k_proj').reshape(*h.shape[:-1], kvheads, dim)
|
||||
v = self.linear(h, p+'v_proj').reshape(*h.shape[:-1], kvheads, dim)
|
||||
if getattr(self, 'capture_qk', False):
|
||||
self.qk_inputs[layer] = (q.copy(), k.copy())
|
||||
q = self.normalization(q, p+'q_norm')
|
||||
k = self.normalization(k, p+'k_norm')
|
||||
rp = c['rope_parameters']
|
||||
q, k = (rope(x, rp['rope_theta'], rp['partial_rotary_factor']) for x in (q,k))
|
||||
out, probs = attention(q,k,v)
|
||||
if shared_p is not None:
|
||||
out = np.einsum('bhts,bshd->bthd', shared_p, np.repeat(v, heads//kvheads, axis=2))
|
||||
out = (out*sigmoid(gate)).reshape(*h.shape[:-1], -1)
|
||||
return self.linear(out, p+'o_proj'), probs
|
||||
|
||||
def linear_attention(self, h, layer):
|
||||
p = f'layers.{layer}.linear_attn.'
|
||||
c = self.config
|
||||
kh, vh = c['linear_num_key_heads'], c['linear_num_value_heads']
|
||||
kd, vd = c['linear_key_head_dim'], c['linear_value_head_dim']
|
||||
qkv = self.linear(h, p+'in_proj_qkv')
|
||||
qkv = causal_conv(qkv, self.small(p+'conv1d.weight')[:,0,:])
|
||||
q, k, v = np.split(qkv, [kh*kd, 2*kh*kd], axis=-1)
|
||||
q = q.reshape(*h.shape[:-1], kh, kd)
|
||||
k = k.reshape(*h.shape[:-1], kh, kd)
|
||||
v = v.reshape(*h.shape[:-1], vh, vd)
|
||||
if vh != kh:
|
||||
q, k = np.repeat(q, vh//kh, axis=2), np.repeat(k, vh//kh, axis=2)
|
||||
z = self.linear(h, p+'in_proj_z').reshape(*h.shape[:-1], vh, vd)
|
||||
beta = sigmoid(self.linear(h, p+'in_proj_b'))
|
||||
a = self.linear(h, p+'in_proj_a')
|
||||
g = -np.exp(self.small(p+'A_log')) * np.logaddexp(np.float32(0), a+self.small(p+'dt_bias'))
|
||||
out, _ = delta_rule(q,k,v,g,beta)
|
||||
out = norm(out, self.small(p+'norm.weight'), self.eps, centered=False)*silu(z)
|
||||
return self.linear(out.reshape(*h.shape[:-1], -1), p+'out_proj')
|
||||
|
||||
def forward(self, ids, capture=None, replacement=None, progress=None):
|
||||
h = self.embedding(ids)
|
||||
saved = {}
|
||||
capture = set(capture or [])
|
||||
for layer, kind in enumerate(self.config['layer_types']):
|
||||
self.guard()
|
||||
raw = h
|
||||
n = self.normalization(h, f'layers.{layer}.input_layernorm')
|
||||
if ('attn', layer) in capture:
|
||||
saved[f'attn_{layer}'] = n.copy()
|
||||
donor = replacement[2] if replacement and replacement[:2] == ('attn',layer) else layer
|
||||
if kind == 'full_attention':
|
||||
out, _ = self.full_attention(n,donor)
|
||||
else:
|
||||
out = self.linear_attention(n,donor)
|
||||
h = raw + out
|
||||
n = self.normalization(h, f'layers.{layer}.post_attention_layernorm')
|
||||
if ('mlp',layer) in capture:
|
||||
saved[f'mlp_{layer}'] = n.copy()
|
||||
donor = replacement[2] if replacement and replacement[:2] == ('mlp',layer) else layer
|
||||
h = h + self.mlp(n,donor)
|
||||
if not np.isfinite(h).all():
|
||||
raise FloatingPointError(f'Nonfinite hidden states at layer {layer}')
|
||||
if progress:
|
||||
progress(layer)
|
||||
return self.normalization(h,'norm'), saved
|
||||
|
||||
def nll(self, h, ids, rows=512):
|
||||
# Tied output embedding; online logsumexp avoids full vocabulary logits.
|
||||
x = h[:,:-1].reshape(-1,h.shape[-1])
|
||||
targets = ids[:,1:].reshape(-1)
|
||||
t = self.tensors[PREFIX+'embed_tokens.weight']
|
||||
mx = np.full(x.shape[0],-np.inf,dtype=np.float64)
|
||||
total = np.zeros(x.shape[0],dtype=np.float64)
|
||||
correct = np.empty(x.shape[0],dtype=np.float64)
|
||||
for start in range(0,t.shape[0],rows):
|
||||
self.guard()
|
||||
end=min(start+rows,t.shape[0])
|
||||
w=t.read(slice(start*t.shape[1],end*t.shape[1])).reshape(end-start,t.shape[1])
|
||||
scores=x@w.T
|
||||
m=np.maximum(mx,scores.max(axis=-1))
|
||||
total=total*np.exp(mx-m)+np.exp(scores-m[:,None]).sum(axis=-1)
|
||||
mx=m
|
||||
which=(targets>=start)&(targets<end)
|
||||
correct[which]=scores[np.flatnonzero(which),targets[which]-start]
|
||||
del w,scores
|
||||
t.evict()
|
||||
return (mx+np.log(total)-correct).reshape(ids.shape[0],-1)
|
||||
@@ -0,0 +1,134 @@
|
||||
"""Find repeated learned scalar MLP functions on real hidden inputs, no training.
|
||||
|
||||
Four fixed layers; calibration-only selection; one frozen pair; independent heldout
|
||||
functional test and two actual network substitutions. No scale fitting or R.
|
||||
"""
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import resource
|
||||
import time
|
||||
import numpy as np
|
||||
from .model import StreamingLM,silu
|
||||
from .run import write_json,save_csv
|
||||
|
||||
LAYERS=[5,6,20,21]
|
||||
|
||||
|
||||
def pair_metrics(x,y):
|
||||
x,y=x.astype(np.float64),y.astype(np.float64)
|
||||
xx,yy=float(x@x),float(y@y);delta=float((x-y)@(x-y))
|
||||
xc,yc=x-x.mean(),y-y.mean()
|
||||
return {'relative_l2_a_to_b':float(np.sqrt(delta/max(xx,1e-30))),
|
||||
'relative_l2_b_to_a':float(np.sqrt(delta/max(yy,1e-30))),
|
||||
'cosine':float(x@y/max(np.sqrt(xx*yy),1e-30)),
|
||||
'pearson':float(xc@yc/max(np.linalg.norm(xc)*np.linalg.norm(yc),1e-30)),
|
||||
'rms_a':float(np.sqrt(xx/x.size)),'rms_b':float(np.sqrt(yy/y.size)),
|
||||
'std_over_rms_a':float(np.std(x)/max(np.sqrt(xx/x.size),1e-30)),
|
||||
'std_over_rms_b':float(np.std(y)/max(np.sqrt(yy/y.size),1e-30)),
|
||||
'effective_energy_tokens_a':float(xx*xx/max(float((x**4).sum()),1e-30)),
|
||||
'effective_energy_tokens_b':float(yy*yy/max(float((y**4).sum()),1e-30))}
|
||||
|
||||
|
||||
class NeuronReuseLM(StreamingLM):
|
||||
def configure(self,layer,target,donor):self.reused_layer,self.target,self.donor=layer,target,donor
|
||||
def mlp(self,h,layer):
|
||||
p=f'layers.{layer}.mlp.'
|
||||
z=silu(self.linear(h,p+'gate_proj'))*self.linear(h,p+'up_proj')
|
||||
if layer==self.reused_layer:z[...,self.target]=z[...,self.donor]
|
||||
return self.linear(z,p+'down_proj')
|
||||
|
||||
|
||||
def main():
|
||||
p=argparse.ArgumentParser();p.add_argument('--root',type=Path,required=True);a=p.parse_args();root=a.root
|
||||
protocol={'layers':LAYERS,'unit':'post_silu_gate_times_up_scalar_function; keep each output column',
|
||||
'thresholds':[.05,.10,.20],'maximum_directional_relative_l2':.20,'cosine_min':.98,'pearson_min':.95,
|
||||
'rms_floor':'at least within-layer median calibration neuron RMS','std_over_rms_min':.25,
|
||||
'effective_energy_tokens_min':16,'max_selected_pairs':1,'selection':'smallest worst directional calibration error; then layer/id',
|
||||
'heldout_per_text_error_max':.35,'heldout_quality_mean_nll_delta_max':.01,'heldout_quality_sequence_p95_delta_max':.05,
|
||||
'training':False,'scale_fitting':False,'cost_required':False}
|
||||
protocol_path=root/'neuron-reuse-protocol.json'
|
||||
if protocol_path.exists():
|
||||
old=json.loads(protocol_path.read_text());assert all(old[k]==v for k,v in protocol.items())
|
||||
else:write_json(protocol_path,{**protocol,'registered_unix':time.time()})
|
||||
ledger_path=root/'compute-budget-ledger.json';ledger=json.loads(ledger_path.read_text());remaining=int(7200-ledger['spent_seconds']-30)
|
||||
if remaining<120:raise TimeoutError('Remaining cumulative budget insufficient')
|
||||
started=time.monotonic();result={'state':'running','selected':None,'function_pass':False,'interventions':[]}
|
||||
try:
|
||||
cfg=json.loads((root/'model_config.json').read_text());lm=NeuronReuseLM(root/'weights',cfg,seconds=remaining)
|
||||
data=np.load(root/'baseline-activations.npz');samples=json.loads((root/'samples-tokenized.json').read_text())['samples']
|
||||
cal=np.array([i for i,s in enumerate(samples) if s['split']=='calibration']);held=np.array([i for i,s in enumerate(samples) if s['split']=='heldout'])
|
||||
activations={};summaries=[];best=None
|
||||
for layer in LAYERS:
|
||||
h=data[f'mlp_{layer}'];pfx=f'layers.{layer}.mlp.'
|
||||
z=silu(lm.linear(h,pfx+'gate_proj'))*lm.linear(h,pfx+'up_proj');activations[layer]=z
|
||||
# Profiles use all calibration tokens; never use heldout in selection.
|
||||
x=np.ascontiguousarray(z[cal].reshape(-1,z.shape[-1]).T,dtype=np.float64)
|
||||
n=x.shape[1];normsq=(x*x).sum(1);means=x.mean(1);variance=np.maximum(normsq-n*means**2,0)
|
||||
rms=np.sqrt(normsq/n);floor=float(np.median(rms));effective=normsq**2/np.maximum((x**4).sum(1),1e-30)
|
||||
active=(rms>=floor)&(np.sqrt(variance/np.maximum(normsq,1e-30))>=.25)&(effective>=16)
|
||||
counts={str(t):0 for t in protocol['thresholds']};tested=0;layerbest=None
|
||||
for start in range(0,x.shape[0],256):
|
||||
lm.guard();end=min(start+256,x.shape[0]);dot=x[start:end]@x.T
|
||||
delta=np.maximum(normsq[start:end,None]+normsq[None,:]-2*dot,0)
|
||||
error=np.sqrt(delta/np.maximum(np.minimum(normsq[start:end,None],normsq[None,:]),1e-30))
|
||||
cosine=dot/np.maximum(np.sqrt(normsq[start:end,None]*normsq[None,:]),1e-30)
|
||||
pearson=(dot-n*means[start:end,None]*means[None,:])/np.maximum(np.sqrt(variance[start:end,None]*variance[None,:]),1e-30)
|
||||
unique=np.arange(start,end)[:,None]<np.arange(x.shape[0])[None,:]
|
||||
allowed=unique&active[start:end,None]&active[None,:]&(cosine>=.98)&(pearson>=.95)
|
||||
tested+=int((unique&active[start:end,None]&active[None,:]).sum())
|
||||
for threshold in protocol['thresholds']:counts[str(threshold)]+=int((allowed&(error<=threshold)).sum())
|
||||
mask=allowed&(error<=.20)
|
||||
if mask.any():
|
||||
score=np.where(mask,error,np.inf);row,col=np.unravel_index(np.argmin(score),score.shape)
|
||||
candidate={'layer':layer,'a':int(start+row),'b':int(col),'calibration_worst_relative_l2':float(score[row,col])}
|
||||
key=lambda c:(c['calibration_worst_relative_l2'],c['layer'],c['a'],c['b'])
|
||||
if layerbest is None or key(candidate)<key(layerbest):layerbest=candidate
|
||||
if best is None or key(candidate)<key(best):best=candidate
|
||||
summaries.append({'layer':layer,'neurons':int(x.shape[0]),'eligible_neurons':int(active.sum()),'eligible_unique_pairs_checked':tested,'rms_floor':floor,'calibration_pair_counts':counts,'best':layerbest})
|
||||
write_json(root/'neuron-reuse-search-summary.json',summaries)
|
||||
print(json.dumps(summaries[-1]),flush=True)
|
||||
np.savez(root/'neuron-reuse-activations.npz',**{f'layer{k}':v for k,v in activations.items()})
|
||||
write_json(root/'neuron-reuse-selection.json',{'selected':best,'selection_uses_heldout':False})
|
||||
result['selected']=best
|
||||
if best is not None:
|
||||
layer,i,j=best['layer'],best['a'],best['b'];z=activations[layer]
|
||||
metrics=[]
|
||||
for split,index in [('calibration',cal),('heldout',held)]:
|
||||
metrics.append({'split':split,'scope':'pooled',**pair_metrics(z[index,:,i].reshape(-1),z[index,:,j].reshape(-1))})
|
||||
for idx in index:metrics.append({'split':split,'scope':samples[idx]['id'],**pair_metrics(z[idx,:,i],z[idx,:,j])})
|
||||
save_csv(root/'neuron-reuse-function-metrics.csv',metrics)
|
||||
pooled=next(r for r in metrics if r['split']=='heldout' and r['scope']=='pooled')
|
||||
floor=next(r['rms_floor'] for r in summaries if r['layer']==layer)
|
||||
function_pass=(max(pooled['relative_l2_a_to_b'],pooled['relative_l2_b_to_a'])<=.20 and pooled['cosine']>=.98 and pooled['pearson']>=.95 and
|
||||
min(pooled['std_over_rms_a'],pooled['std_over_rms_b'])>=.25 and min(pooled['effective_energy_tokens_a'],pooled['effective_energy_tokens_b'])>=16 and
|
||||
min(pooled['rms_a'],pooled['rms_b'])>=floor and
|
||||
all(max(r['relative_l2_a_to_b'],r['relative_l2_b_to_a'])<=.35 for r in metrics if r['split']=='heldout' and r['scope']!='pooled'))
|
||||
result.update(function_pass=bool(function_pass),heldout_function_metrics=pooled)
|
||||
# Show parameter similarity separately from actual functional similarity.
|
||||
wd=[]
|
||||
for role in ('gate_proj','up_proj'):
|
||||
t=lm.tensors[f'model.language_model.layers.{layer}.mlp.{role}.weight'];d=t.shape[1]
|
||||
wa,wb=t.read(slice(i*d,(i+1)*d)),t.read(slice(j*d,(j+1)*d))
|
||||
wd.append({'role':role,'weight_cosine':float(wa.astype(np.float64)@wb/max(np.linalg.norm(wa.astype(np.float64))*np.linalg.norm(wb.astype(np.float64)),1e-30))});t.evict()
|
||||
result['weight_diagnostics']=wd
|
||||
if function_pass:
|
||||
ids=np.array([s['token_ids'] for s in samples],dtype=np.int64);base=np.load(root/'baseline-heldout-token-nll.npy').mean(1)
|
||||
for target,donor in ((i,j),(j,i)):
|
||||
lm.configure(layer,target,donor);h,_=lm.forward(ids[held]);nll=lm.nll(h,ids[held])
|
||||
np.save(root/f'neuron-reuse-{target}-uses-{donor}-token-nll.npy',nll)
|
||||
delta=nll.mean(1)-base
|
||||
result['interventions'].append({'target':target,'donor':donor,'sequence_nll_deltas':delta.tolist(),'mean_nll_delta':float(delta.mean()),'sequence_p95_delta':float(np.quantile(delta,.95)),'passed':bool(delta.mean()<=.01 and np.quantile(delta,.95)<=.05)})
|
||||
write_json(root/'neuron-reuse-summary.json',result)
|
||||
# No fallback after looking at heldout.
|
||||
result.update(state='completed',empirical_repeated_function_positive=bool(result['function_pass'] and len(result['interventions'])==2 and all(r['passed'] for r in result['interventions'])),
|
||||
claim_limit='Distinct learned scalar MLP subfunctions on the specified short-text distribution; not whole-layer duplication, global equivalence or broad task-capability proof. No performance requirement.')
|
||||
except Exception as exc:result.update(state='stopped',error_type=type(exc).__name__,error=str(exc));raise
|
||||
finally:
|
||||
elapsed=time.monotonic()-started;result.update(elapsed_seconds=elapsed,peak_rss_kib=resource.getrusage(resource.RUSAGE_SELF).ru_maxrss)
|
||||
ledger=json.loads(ledger_path.read_text());ledger['spent_seconds']+=elapsed;ledger['calls'].append({'module':'functional.neuron_reuse','seconds':elapsed});write_json(ledger_path,ledger)
|
||||
write_json(root/'neuron-reuse-summary.json',result)
|
||||
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,108 @@
|
||||
"""Existing learned RMSNorm modules: uniform function bound + independent probes.
|
||||
|
||||
Deterministic exhaustive selection by a mathematical all-input error bound, not
|
||||
calibration/heldout fitting. Same role, shape and eps; one pair only.
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
import resource
|
||||
import time
|
||||
import numpy as np
|
||||
from .model import StreamingLM,norm
|
||||
from .run import write_json,save_csv,error_rows
|
||||
|
||||
ROLES=['input_layernorm','post_attention_layernorm']
|
||||
|
||||
|
||||
def uniform_bounds(a,b):
|
||||
if np.any(a==0) or np.any(b==0):return None
|
||||
diff=np.abs(a.astype(np.float64)-b.astype(np.float64))
|
||||
return float(np.max(diff/np.abs(a))),float(np.max(diff/np.abs(b)))
|
||||
|
||||
|
||||
def main():
|
||||
p=argparse.ArgumentParser();p.add_argument('--root',type=Path,required=True);args=p.parse_args();root=args.root
|
||||
protocol={'roles':ROLES,'layers':list(range(24)),'same_role_only':True,'same_eps':1e-6,
|
||||
'uniform_worst_error_limit':.20,'empirical_function_worst_relative_error_limit':.05,
|
||||
'selection':'smallest maximum two-direction uniform bound, then role/layer; no input data used',
|
||||
'minimum_gain_standard_deviation':.001,'max_selected_pairs':1,'heldout_nll_mean_delta_limit':.01,
|
||||
'heldout_nll_sequence_p95_delta_limit':.05,'training':False,'cost_required':False}
|
||||
path=root/'norm-repetition-protocol.json'
|
||||
if path.exists():
|
||||
existing=json.loads(path.read_text());assert all(existing[k]==v for k,v in protocol.items())
|
||||
else:write_json(path,{**protocol,'registered_unix':time.time()})
|
||||
ledger_path=root/'compute-budget-ledger.json';ledger=json.loads(ledger_path.read_text());remaining=int(7200-ledger['spent_seconds']-30)
|
||||
if remaining<120:raise TimeoutError('Insufficient cumulative computation budget')
|
||||
started=time.monotonic();result={'state':'running','selected':None,'interventions':[]}
|
||||
try:
|
||||
cfg=json.loads((root/'model_config.json').read_text());lm=StreamingLM(root/'weights',cfg,seconds=remaining)
|
||||
assert lm.eps==1e-6
|
||||
candidates=[];gains={}
|
||||
for role in ROLES:
|
||||
for layer in range(24):gains[(role,layer)]=1+lm.small(f'layers.{layer}.{role}.weight')
|
||||
for i in range(24):
|
||||
for j in range(i+1,24):
|
||||
a,b=gains[(role,i)],gains[(role,j)]
|
||||
bounds=uniform_bounds(a,b)
|
||||
if bounds is None:continue
|
||||
eligible=min(float(np.std(a)),float(np.std(b)))>=.001
|
||||
candidates.append({'role':role,'a':i,'b':j,'a_to_b_uniform_bound':bounds[0],'b_to_a_uniform_bound':bounds[1],
|
||||
'worst_uniform_bound':max(bounds),'eligible_nonconstant_gains':eligible,
|
||||
'gain_std_a':float(np.std(a)),'gain_std_b':float(np.std(b)),
|
||||
'gain_min_a':float(np.min(a)),'gain_min_b':float(np.min(b))})
|
||||
save_csv(root/'norm-repetition-uniform-bounds.csv',candidates)
|
||||
feasible=[c for c in candidates if c['eligible_nonconstant_gains'] and c['worst_uniform_bound']<=.20]
|
||||
counts={str(t):sum(c['eligible_nonconstant_gains'] and c['worst_uniform_bound']<=t for c in candidates) for t in (.05,.10,.20)}
|
||||
selected=sorted(feasible,key=lambda c:(c['worst_uniform_bound'],c['role'],c['a'],c['b']))[:1]
|
||||
write_json(root/'norm-repetition-selection.json',{'selected':selected,'uniform_bound_counts':counts,'selection_reads_no_input_data':True})
|
||||
result['uniform_bound_counts']=counts
|
||||
if selected:
|
||||
c=selected[0];result['selected']=c
|
||||
# Selection is frozen before reading tokenized inputs or their activations.
|
||||
samples=json.loads((root/'samples-tokenized.json').read_text())['samples'];ids=np.array([s['token_ids'] for s in samples],dtype=np.int64)
|
||||
held=np.array([i for i,s in enumerate(samples) if s['split']=='heldout'])
|
||||
kind='attn' if c['role']=='input_layernorm' else 'mlp'
|
||||
captures=[(kind,c['a']),(kind,c['b'])]
|
||||
h,saved=lm.forward(ids,capture=captures)
|
||||
# The untouched recomputation must reproduce the saved real baseline.
|
||||
original=np.load(root/'baseline-activations.npz')['final']
|
||||
np.testing.assert_array_equal(h,original)
|
||||
np.savez(root/'norm-repetition-input-profiles.npz',**saved)
|
||||
ga,gb=gains[(c['role'],c['a'])],gains[(c['role'],c['b'])]
|
||||
rows=[]
|
||||
for source in (c['a'],c['b']):
|
||||
n=saved[f'{kind}_{source}'];g=gains[(c['role'],source)]
|
||||
# n = u * gamma_source. Reconstruct common normalized input u;
|
||||
# source gains are nonzero. Additional rounding is ~FP32 only.
|
||||
unit=n/g
|
||||
fa,fb=unit*ga,unit*gb
|
||||
rec=error_rows(fa,fb,samples,{'source_layer':source,'role':c['role'],'a':c['a'],'b':c['b']})
|
||||
rows.extend(rec)
|
||||
save_csv(root/'norm-repetition-function-metrics.csv',rows)
|
||||
heldrows=[r for r in rows if r['split']=='heldout']
|
||||
function_pass=all(max(r['relative_l2'],r['reverse_relative_l2'])<=.05 for r in heldrows)
|
||||
result.update(uniform_function_similarity_proven=True,heldout_function_pass=bool(function_pass),
|
||||
worst_heldout_relative_l2=max(max(r['relative_l2'],r['reverse_relative_l2']) for r in heldrows),
|
||||
worst_heldout_cosine=min(r['cosine'] for r in heldrows),untouched_full_forward_bitwise_matches_baseline=True)
|
||||
if function_pass:
|
||||
baseline=np.load(root/'baseline-heldout-token-nll.npy').mean(1)
|
||||
for target,donor in ((c['a'],c['b']),(c['b'],c['a'])):
|
||||
lm.norm_replacement=(f'layers.{target}.{c["role"]}',f'layers.{donor}.{c["role"]}')
|
||||
changed,_=lm.forward(ids[held]);nll=lm.nll(changed,ids[held]);np.save(root/f'norm-repetition-{target}-uses-{donor}-token-nll.npy',nll)
|
||||
delta=nll.mean(1)-baseline
|
||||
result['interventions'].append({'target':target,'donor':donor,'sequence_nll_deltas':delta.tolist(),
|
||||
'mean_nll_delta':float(delta.mean()),'sequence_p95_nll_delta':float(np.quantile(delta,.95)),
|
||||
'passed_quality':bool(delta.mean()<=.01 and np.quantile(delta,.95)<=.05)})
|
||||
write_json(root/'norm-repetition-summary.json',result)
|
||||
result['learned_function_repetition_positive']=bool(function_pass)
|
||||
result['bidirectional_network_quality_pass']=bool(len(result['interventions'])==2 and all(x['passed_quality'] for x in result['interventions']))
|
||||
result.update(state='completed',claim_limit='Existing learned RMSNorm module functions with identical eps and shape; uniform real-arithmetic relative bound and tiny real-text probes. Does not establish attention/MLP-body duplication or broad task capability.')
|
||||
except Exception as exc:result.update(state='stopped',error_type=type(exc).__name__,error=str(exc));raise
|
||||
finally:
|
||||
elapsed=time.monotonic()-started;result.update(elapsed_seconds=elapsed,peak_rss_kib=resource.getrusage(resource.RUSAGE_SELF).ru_maxrss)
|
||||
ledger=json.loads(ledger_path.read_text());ledger['spent_seconds']+=elapsed;ledger['calls'].append({'module':'functional.norm_repetition','seconds':elapsed});write_json(ledger_path,ledger)
|
||||
write_json(root/'norm-repetition-summary.json',result)
|
||||
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,46 @@
|
||||
"""Package code-independent scientific outputs, retain large activation caches externally.
|
||||
|
||||
The experiment directory remains immutable and complete. The delivery directory
|
||||
contains all figure source tables/NLL arrays plus hashes/locations for three large
|
||||
reproducible activation caches. Never deletes model weights or cache files.
|
||||
"""
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
from .run import write_json
|
||||
|
||||
CACHES={'baseline-activations.npz','neuron-reuse-activations.npz','norm-repetition-input-profiles.npz'}
|
||||
|
||||
|
||||
def main():
|
||||
p=argparse.ArgumentParser();p.add_argument('--root',required=True,type=Path);p.add_argument('--output',required=True,type=Path);a=p.parse_args()
|
||||
root,out=a.root.resolve(),a.output.resolve();assert root!=out
|
||||
out.mkdir(parents=True,exist_ok=True)
|
||||
full=json.loads((root/'checksums-functional.json').read_text())
|
||||
files=json.loads((root/'archive-files.json').read_text())
|
||||
for name in files:
|
||||
src=root/name
|
||||
if name in CACHES:continue
|
||||
dest=out/name;dest.parent.mkdir(parents=True,exist_ok=True);shutil.copy2(src,dest)
|
||||
external=[]
|
||||
for name in sorted(CACHES):
|
||||
src=root/name;digest=hashlib.sha256(src.read_bytes()).hexdigest();assert digest==full[name]
|
||||
external.append({'name':name,'server_path':str(src),'bytes':src.stat().st_size,'sha256':digest,'retained':True,'reproducible_by':'functional.run / functional.neuron_reuse / functional.norm_repetition'})
|
||||
write_json(out/'external-activation-cache.json',{'host':'8.211.154.134','files':external,'not_model_weights':True,
|
||||
'reason':'Large intermediate caches retained on the authorized server; all figure source tables and token NLL arrays are delivered in Git.'})
|
||||
shutil.copy2(root/'checksums-functional.json',out/'checksums-full-server.json')
|
||||
(out/'DELIVERY.md').write_text('# Scientific delivery and retained caches\n\nAll figure CSV/JSON source data, per-token NLL, protocols, reports, metadata and historical logs are included.\nThree large reproducible intermediate activation caches remain on 8.211.154.134 and in the original full local archive;\n`external-activation-cache.json` records exact locations, sizes and SHA256. No weight or cache was deleted.\nThe original full experiment manifest is `checksums-full-server.json`; the delivery-only manifest is\n`checksums-functional.json`. No model checkpoint or tokenizer payload is in this Git delivery.\n')
|
||||
validation=json.loads((out/'audit-validation.json').read_text());validation.update(delivery_excludes_external_activation_caches=True,full_experiment_checksums='checksums-full-server.json',figure_source_tables_and_token_nll_complete=True)
|
||||
write_json(out/'audit-validation.json',validation)
|
||||
checks={}
|
||||
for file in sorted(out.rglob('*')):
|
||||
if file.is_file() and file.name not in ('checksums-functional.json','audit-validation.json','archive-files.json'):
|
||||
checks[str(file.relative_to(out))]=hashlib.sha256(file.read_bytes()).hexdigest()
|
||||
write_json(out/'checksums-functional.json',checks)
|
||||
write_json(out/'archive-files.json',[*checks,'checksums-functional.json','audit-validation.json','archive-files.json'])
|
||||
print(json.dumps({'delivery_files':len(checks)+3,'checksummed_files':len(checks),'external_caches':len(external),'weights_and_caches_retained':True}))
|
||||
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,39 @@
|
||||
"""One 8-token real text, bounded memory/semantic capacity smoke check."""
|
||||
import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
import resource
|
||||
import time
|
||||
import numpy as np
|
||||
from .model import StreamingLM
|
||||
from .run import tokenize,write_json
|
||||
|
||||
|
||||
def main():
|
||||
p=argparse.ArgumentParser();p.add_argument('--root',required=True,type=Path);a=p.parse_args()
|
||||
samples,ids=tokenize(a.root)
|
||||
start=time.time()
|
||||
lm=StreamingLM(a.root/'weights',json.loads((a.root/'model_config.json').read_text()),seconds=1200)
|
||||
snapshots=[]
|
||||
result={'passed':False,'sample_id':samples[0]['id'],'tokens':8}
|
||||
def progress(layer):
|
||||
status=dict(line.split(':',1) for line in Path('/proc/self/status').read_text().splitlines() if ':' in line)
|
||||
snapshots.append({'layer':layer,'seconds':time.time()-start,'rss_kib':int(status['VmRSS'].split()[0]),
|
||||
'process_swap_kib':int(status.get('VmSwap','0 kB').split()[0]),
|
||||
'available_kib':int(next(line.split()[1] for line in Path('/proc/meminfo').read_text().splitlines() if line.startswith('MemAvailable:')))})
|
||||
print(json.dumps(snapshots[-1]),flush=True)
|
||||
try:
|
||||
h,_=lm.forward(ids[:1,:8],progress=progress)
|
||||
nll=lm.nll(h,ids[:1,:8])
|
||||
if not np.isfinite(nll).all():raise FloatingPointError('Nonfinite NLL')
|
||||
result.update(passed=True,mean_nll=float(nll.mean()),token_nll=nll.tolist())
|
||||
except Exception as exc:
|
||||
result.update(error_type=type(exc).__name__,error=str(exc))
|
||||
raise
|
||||
finally:
|
||||
result.update(elapsed_seconds=time.time()-start,peak_rss_kib=resource.getrusage(resource.RUSAGE_SELF).ru_maxrss,
|
||||
resource_snapshots=snapshots,limitation='Real-weight capacity check, not an independent correctness oracle or quality benchmark')
|
||||
write_json(a.root/'preflight.json',result)
|
||||
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,5 @@
|
||||
{
|
||||
"commit": "fd3cf0140de03f7619ac95b5cfed05b711ff9e22",
|
||||
"url": "https://raw.githubusercontent.com/huggingface/transformers/fd3cf0140de03f7619ac95b5cfed05b711ff9e22/src/transformers/models/qwen3_5/modeling_qwen3_5.py",
|
||||
"sha256": "4463b8a108f678c166755101d95f188179f54143305953d78147de0389b80ebc"
|
||||
}
|
||||
@@ -0,0 +1,142 @@
|
||||
"""Remote-only numerical comparison with pinned official Torch definitions.
|
||||
|
||||
Reads only a known official Transformers source file; extracts named mathematical
|
||||
operators/classes via AST, removing optional kernel decorators. Never imports the
|
||||
complete model or executes checkpoint repository code. No compilation/build.
|
||||
"""
|
||||
import argparse
|
||||
import ast
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
import numpy as np
|
||||
from .model import StreamingLM, delta_rule, rope, norm
|
||||
|
||||
EXPECTED_COMMIT = 'fd3cf0140de03f7619ac95b5cfed05b711ff9e22'
|
||||
NAMES = {'apply_mask_to_padding_states', 'l2norm', 'torch_chunk_gated_delta_rule', 'torch_recurrent_gated_delta_rule',
|
||||
'causal_conv1d_fn', 'Qwen3_5RMSNormGated', 'Qwen3_5GatedDeltaNet',
|
||||
'rotate_half', 'apply_rotary_pos_emb', 'repeat_kv', 'eager_attention_forward',
|
||||
'Qwen3_5Attention', 'Qwen3_5MLP', 'Qwen3_5RMSNorm', 'Qwen3_5DecoderLayer',
|
||||
'Qwen3_5TextRotaryEmbedding'}
|
||||
|
||||
|
||||
def official_namespace(source):
|
||||
import torch
|
||||
import torch.nn.functional as F
|
||||
tree = ast.parse(source)
|
||||
nodes = []
|
||||
for node in tree.body:
|
||||
if isinstance(node, (ast.FunctionDef, ast.ClassDef)) and node.name in NAMES:
|
||||
for child in ast.walk(node):
|
||||
if isinstance(child, (ast.FunctionDef, ast.ClassDef)):
|
||||
child.decorator_list = [d for d in child.decorator_list if isinstance(d, ast.Name) and d.id in ('staticmethod','classmethod')]
|
||||
nodes.append(node)
|
||||
assert {x.name for x in nodes} == NAMES
|
||||
tree = ast.Module(body=[ast.ImportFrom(module='__future__',names=[ast.alias(name='annotations')],level=0),*nodes],type_ignores=[])
|
||||
ns = {'torch':torch, 'nn':torch.nn, 'F':F, 'ACT2FN':{'silu':F.silu},
|
||||
'GradientCheckpointingLayer':torch.nn.Module,
|
||||
'is_torchdynamo_exporting':lambda:False,
|
||||
'ALL_ATTENTION_FUNCTIONS':SimpleNamespace(get_interface=lambda name, fallback:fallback)}
|
||||
exec(compile(ast.fix_missing_locations(tree),'pinned_transformers_reference','exec'),ns)
|
||||
return ns
|
||||
|
||||
|
||||
class ArrayLM(StreamingLM):
|
||||
def __init__(self, arrays, config):
|
||||
self.arrays, self.config, self.eps = arrays,config,config['rms_norm_eps']
|
||||
def guard(self):
|
||||
pass
|
||||
def small(self,name):
|
||||
return self.arrays[name]
|
||||
def linear(self,x,name,rows=256):
|
||||
return x@self.arrays[name+'.weight'].T
|
||||
def embedding(self,ids):
|
||||
return self.arrays['embed_tokens.weight'][ids]
|
||||
|
||||
|
||||
def main():
|
||||
p=argparse.ArgumentParser();p.add_argument('--source',type=Path,required=True);p.add_argument('--output',type=Path,required=True)
|
||||
a=p.parse_args()
|
||||
import torch
|
||||
torch.set_num_threads(1)
|
||||
torch.manual_seed(42)
|
||||
provenance=json.loads(Path(__file__).with_name('reference.json').read_text())
|
||||
assert hashlib.sha256(a.source.read_bytes()).hexdigest()==provenance['sha256'], 'Reference source hash mismatch'
|
||||
ns=official_namespace(a.source.read_text())
|
||||
cfg={'hidden_size':16,'num_attention_heads':2,'num_key_value_heads':1,'head_dim':8,
|
||||
'intermediate_size':24,'hidden_act':'silu','attention_bias':False,'attention_dropout':0.,
|
||||
'linear_num_value_heads':2,'linear_num_key_heads':2,'linear_key_head_dim':4,
|
||||
'linear_value_head_dim':4,'linear_conv_kernel_dim':4,'rms_norm_eps':1e-6,
|
||||
'layer_types':['linear_attention','full_attention'],'max_position_embeddings':4096,
|
||||
'rope_parameters':{'rope_type':'default','rope_theta':10000000.,'partial_rotary_factor':.5,'mrope_section':[1,0,0]},
|
||||
'_attn_implementation':'eager'}
|
||||
config=SimpleNamespace(**cfg)
|
||||
layers=[ns['Qwen3_5DecoderLayer'](config,i).eval() for i in range(2)]
|
||||
for layer in layers:
|
||||
for name,t in layer.named_parameters():
|
||||
with torch.no_grad():
|
||||
t.uniform_(-.2,.2)
|
||||
if name=='linear_attn.norm.weight':t.add_(1.)
|
||||
arrays={f'layers.{i}.'+k:v.detach().numpy().copy() for i,l in enumerate(layers) for k,v in l.state_dict().items()}
|
||||
lm=ArrayLM(arrays,cfg)
|
||||
rng=np.random.default_rng(123)
|
||||
x=rng.standard_normal((2,7,16),dtype=np.float32)
|
||||
tx=torch.from_numpy(x)
|
||||
positions=torch.arange(7)[None,:]
|
||||
rotary=ns['Qwen3_5TextRotaryEmbedding'](config)
|
||||
cos,sin=rotary(tx,positions)
|
||||
mask=torch.triu(torch.full((7,7),-float('inf')),diagonal=1)[None,None]
|
||||
evidence=[]
|
||||
def check(name,actual,expected,atol=2e-5,rtol=2e-5):
|
||||
actual=np.asarray(actual);expected=expected.detach().numpy() if hasattr(expected,'detach') else np.asarray(expected)
|
||||
np.testing.assert_allclose(actual,expected,atol=atol,rtol=rtol)
|
||||
evidence.append({'test':name,'shape':list(actual.shape),'max_abs_error':float(np.max(np.abs(actual-expected))),'atol':atol,'rtol':rtol,'passed':True})
|
||||
with torch.no_grad():
|
||||
for i,layer in enumerate(layers):
|
||||
n=lm.normalization(x,f'layers.{i}.input_layernorm')
|
||||
tn=layer.input_layernorm(torch.from_numpy(x))
|
||||
check(f'layer{i}_rmsnorm',n,tn)
|
||||
if i==0:
|
||||
out=lm.linear_attention(n,i)
|
||||
tout=layer.linear_attn(tn)
|
||||
else:
|
||||
out,pr=lm.full_attention(n,i)
|
||||
tout,tpr=layer.self_attn(tn,(cos,sin),mask)
|
||||
check('attention_probabilities',pr,tpr)
|
||||
check(f'layer{i}_mixer',out,tout)
|
||||
h=x+out
|
||||
n=lm.normalization(h,f'layers.{i}.post_attention_layernorm')
|
||||
check(f'layer{i}_mlp',lm.mlp(n,i),layer.mlp(layer.post_attention_layernorm(torch.from_numpy(h))))
|
||||
actual=h+lm.mlp(n,i)
|
||||
expected=layer(torch.from_numpy(x),(cos,sin),attention_mask=mask if i else None)
|
||||
check(f'layer{i}_complete',actual,expected)
|
||||
# Independent recurrence/chunk equality, including nonmultiple sequence length.
|
||||
for length in (1,7,33):
|
||||
q=rng.standard_normal((2,length,2,4),dtype=np.float32)
|
||||
k=rng.standard_normal(q.shape,dtype=np.float32)
|
||||
v=rng.standard_normal((2,length,2,5),dtype=np.float32)
|
||||
g=-np.abs(rng.standard_normal((2,length,2),dtype=np.float32))
|
||||
beta=rng.uniform(size=g.shape).astype(np.float32)
|
||||
out,state=delta_rule(q,k,v,g,beta)
|
||||
args=[torch.from_numpy(z) for z in (q,k,v,g,beta)]
|
||||
for kind in ('torch_recurrent_gated_delta_rule','torch_chunk_gated_delta_rule'):
|
||||
ref,refstate=ns[kind](*args,output_final_state=True,use_qk_l2norm_in_kernel=True)
|
||||
check(f'{kind}_length{length}_output',out,ref)
|
||||
check(f'{kind}_length{length}_state',state,refstate)
|
||||
# Full two-layer residual forward, final norm and embedding lookup.
|
||||
arrays['embed_tokens.weight']=rng.standard_normal((31,16),dtype=np.float32)
|
||||
arrays['norm.weight']=rng.standard_normal((16,),dtype=np.float32)*.1
|
||||
ids=rng.integers(0,31,size=(2,7))
|
||||
h=torch.from_numpy(arrays['embed_tokens.weight'][ids])
|
||||
for i,l in enumerate(layers):h=l(h,(cos,sin),attention_mask=mask if i else None)
|
||||
expected=h*torch.rsqrt(h.square().mean(-1,keepdim=True)+1e-6)*(1+torch.from_numpy(arrays['norm.weight']))
|
||||
actual,_=lm.forward(ids)
|
||||
check('complete_two_layer_text_forward',actual,expected)
|
||||
result={'passed':True,'reference_commit':EXPECTED_COMMIT,'source_sha256':hashlib.sha256(a.source.read_bytes()).hexdigest(),
|
||||
'torch':torch.__version__,'numpy':np.__version__,'implementation_sha256':hashlib.sha256(Path(__file__).with_name('model.py').read_bytes()).hexdigest(),'tests':evidence,'limitations':'FP32 CPU semantics on tiny synthetic configurations; not full-checkpoint BF16 kernel equivalence'}
|
||||
a.output.write_text(json.dumps(result,indent=2))
|
||||
print(json.dumps({'passed':True,'checks':len(evidence),'maximum_absolute_discrepancy':max(x['max_abs_error'] for x in evidence)}))
|
||||
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,104 @@
|
||||
"""Render scientific pilot figures using saved data; never reopen model weights."""
|
||||
import argparse
|
||||
import csv
|
||||
import json
|
||||
from pathlib import Path
|
||||
import numpy as np
|
||||
import matplotlib
|
||||
matplotlib.use('Agg')
|
||||
import matplotlib.pyplot as plt
|
||||
|
||||
|
||||
def main():
|
||||
p=argparse.ArgumentParser();p.add_argument('--root',type=Path,required=True);a=p.parse_args();root=a.root
|
||||
out=root/'figures';out.mkdir(exist_ok=True)
|
||||
plt.rcParams.update({'font.size':10,'savefig.bbox':'tight','pdf.fonttype':42,'svg.fonttype':'none'})
|
||||
def save(fig,name):
|
||||
for ext in ('png','pdf','svg'):fig.savefig(out/f'{name}.{ext}',dpi=180)
|
||||
plt.close(fig)
|
||||
status=json.loads((root/'functional-status.json').read_text())
|
||||
if (root/'candidate-summary.json').exists():
|
||||
c=json.loads((root/'candidate-summary.json').read_text())
|
||||
fig,ax=plt.subplots(figsize=(10,4.5))
|
||||
ax.bar(np.arange(len(c)),[max(v['maximum_symmetric_relative_error'],1e-8) for v in c],color=['#238b45' if v['passed'] else '#9e9e9e' for v in c])
|
||||
ax.axhline(.05,color='#b2182b',linestyle='--',label='Relative L2 screen = 0.05; other gates also apply')
|
||||
ax.set_yscale('log');ax.set_xticks(np.arange(len(c)),[v['id'] for v in c],rotation=45,ha='right')
|
||||
ax.set_ylabel('Worst calibration directional relative L2')
|
||||
ax.set_title('Fixed candidates; same-input function comparison\n4 calibration texts, 32 tokens; no training')
|
||||
ax.legend(loc='upper left',bbox_to_anchor=(0,-.38),fontsize=8);fig.tight_layout();save(fig,'candidate_function_errors')
|
||||
if (root/'interventions.json').exists():
|
||||
interventions=json.loads((root/'interventions.json').read_text())
|
||||
fig,ax=plt.subplots(figsize=(8,4.5))
|
||||
for idx,r in enumerate(interventions):
|
||||
values=r['sequence_nll_deltas'];x=np.arange(len(values))
|
||||
ax.plot(x,values,'o-',label=r['direction'])
|
||||
ax.axhline(0,color='gray',linewidth=.7);ax.axhline(.01,color='#b2182b',linestyle='--',label='Mean-delta budget (not per-sequence bound)')
|
||||
ax.set_xticks(np.arange(4),['heldout zh1','heldout zh2','heldout en1','heldout en2'])
|
||||
ax.set_ylabel('Token NLL change (nat/token)');ax.set_title('Single-site interventions; frozen candidate\n4 heldout texts, 32 tokens; limited pilot evidence')
|
||||
ax.legend(loc='upper left',bbox_to_anchor=(0,-.18),fontsize=8);fig.tight_layout();save(fig,'heldout_intervention_nll')
|
||||
norm_path=root/'norm-repetition-summary.json'
|
||||
repetition=json.loads(norm_path.read_text()) if norm_path.exists() else {}
|
||||
if (root/'norm-repetition-function-metrics.csv').exists():
|
||||
with (root/'norm-repetition-function-metrics.csv').open() as f:rows=list(csv.DictReader(f))
|
||||
sample_ids=list(dict.fromkeys(row['sample_id'] for row in rows))
|
||||
fig,ax=plt.subplots(figsize=(9,4.8))
|
||||
for source in sorted({row['source_layer'] for row in rows},key=int):
|
||||
rr=[r for r in rows if r['source_layer']==source]
|
||||
ax.plot(range(len(rr)),[max(float(r['relative_l2']),float(r['reverse_relative_l2'])) for r in rr],'o-',label=f'Input captured at layer {source}')
|
||||
ax.axhline(.05,color='#b2182b',ls='--',label='Heldout empirical screen = 0.05')
|
||||
chosen=repetition.get('selected')
|
||||
if chosen:ax.axhline(chosen['worst_uniform_bound'],color='#6a51a3',ls=':',label='Proven uniform all-input bound')
|
||||
ax.axvline(3.5,color='gray',linewidth=.8)
|
||||
ax.set_xticks(range(len(sample_ids)),sample_ids,rotation=30,ha='right');ax.set_ylabel('Worst of two directional relative L2 errors')
|
||||
ax.set_title('Learned input RMSNorm at layer indices 12/13: same-input comparison\nCalibration (left) / independent heldout texts (right), 32 tokens each')
|
||||
ax.legend(loc='upper left',bbox_to_anchor=(0,-.25),fontsize=8);fig.tight_layout();save(fig,'learned_norm_function_errors')
|
||||
if repetition.get('interventions'):
|
||||
fig,ax=plt.subplots(figsize=(8,4.5))
|
||||
for v in repetition['interventions']:
|
||||
ax.plot(range(4),v['sequence_nll_deltas'],'o-',label=f'Layer {v["target"]} uses {v["donor"]}; mean={v["mean_nll_delta"]:.5g}')
|
||||
ax.axhline(0,color='gray',linewidth=.8);ax.set_xticks(range(4),['zh1','zh2','en1','en2'])
|
||||
ax.set_xlabel('Independent heldout text (32 tokens)');ax.set_ylabel('Mean token NLL change (nat/token)')
|
||||
ax.set_title('Bidirectional single-site substitution of existing learned RMSNorm modules')
|
||||
ax.legend(loc='upper left',bbox_to_anchor=(0,-.18),fontsize=8);fig.tight_layout();save(fig,'learned_norm_network_substitution')
|
||||
if (root/'gain-factorization-score-metrics.csv').exists():
|
||||
with (root/'gain-factorization-score-metrics.csv').open() as f:rr=list(csv.DictReader(f))
|
||||
layers=sorted({int(v['layer']) for v in rr});fig,axes=plt.subplots(1,2,figsize=(10,4))
|
||||
for ax,key,label in zip(axes,['score_max_abs_error','probability_max_abs_error'],['Maximum absolute score error','Maximum absolute probability error']):
|
||||
for direction in ('q_to_k','k_to_q'):
|
||||
ax.plot(layers,[max(float(v[key]) for v in rr if int(v['layer'])==layer and v['direction']==direction) for layer in layers],'o-',label=direction)
|
||||
ax.set_xlabel('Full-attention layer index');ax.set_xticks(layers);ax.set_ylabel(label);ax.ticklabel_format(axis='y',style='sci',scilimits=(0,0));ax.legend(loc='upper left',fontsize=8)
|
||||
fig.suptitle('Supplement: algebraic reparameterization, separate from learned-function repetition');fig.tight_layout();save(fig,'algebraic_gain_identity')
|
||||
neuron_path=root/'neuron-reuse-summary.json'
|
||||
neuron=json.loads(neuron_path.read_text()) if neuron_path.exists() else {}
|
||||
text=['# CPU learned-function repetition pilot','',
|
||||
'Primary question: do two distinct trained parts implement similar functions? Cost is not a success condition.',
|
||||
f'- Existing learned RMSNorm function positive: {repetition.get("learned_function_repetition_positive",False)}',
|
||||
f'- Bidirectional network quality screen passed: {repetition.get("bidirectional_network_quality_pass",False)}',
|
||||
f'- Learned scalar MLP neuron search positive: {neuron.get("empirical_repeated_function_positive",False)}',
|
||||
f'- Original 11 candidate run state: {status["state"]}; selected: {status.get("selected_candidate",[])}',
|
||||
'- All model weights are retained on the server until the user explicitly requests deletion.','']
|
||||
if repetition.get('selected'):
|
||||
c=repetition['selected'];text += ['## Existing learned module evidence','',
|
||||
f'Role: {c["role"]}; layers {c["a"]} and {c["b"]}; each maps 2048-dimensional input to output.',
|
||||
f'Uniform real-arithmetic relative-error bound, first-to-second: {c["a_to_b_uniform_bound"]:.12g}; reverse: {c["b_to_a_uniform_bound"]:.12g}.',
|
||||
f'Maximum heldout directional output relative L2: {repetition.get("worst_heldout_relative_l2")}; minimum cosine: {repetition.get("worst_heldout_cosine")}.',
|
||||
'Same epsilon and nonzero gains: the common RMS denominator cancels. For any nonzero input x,',
|
||||
'`||f_a(x)-f_b(x)|| / ||f_a(x)|| <= max_j |a_j-b_j|/|a_j|`; swap a/b for the reverse direction.',
|
||||
'Selection used only the tight all-input bound across 552 same-role pairs, before inspecting inputs.',
|
||||
'Real probes use four calibration and four heldout experiment-authored texts, 32 tokens each, and both source layers.',
|
||||
'These are learned, nonconstant gain vectors. The claim concerns RMSNorm modules; it does not establish attention/MLP-body duplication.','']
|
||||
for v in repetition.get('interventions',[]):text += [f'- Layer {v["target"]} uses {v["donor"]}: mean heldout NLL delta {v["mean_nll_delta"]:.12g}; sequence P95 delta {v["sequence_p95_nll_delta"]:.12g}; screen pass={v["passed_quality"]}.']
|
||||
text += ['', '## Negative results and separate algebraic result','',
|
||||
'The original whole-attention, whole-MLP and within-layer Q/K norm candidates did not pass their gates.',
|
||||
'The scalar neuron search excludes low-RMS/near-constant profiles and selects at most one pair without heldout feedback; consult its search summary for coverage.',
|
||||
'Q/K non-RoPE gain consolidation is a separately verified real-arithmetic identity, not evidence that two existing learned modules duplicate functions.',
|
||||
'Earlier local CPU timing records are preserved for provenance; performance is outside the current success criterion.','',
|
||||
'## Limits and provenance','',
|
||||
'Original BF16 weights were decoded into streaming FP32 CPU operators; no training, scale fitting, R fitting or GPU.',
|
||||
'The numerical reference includes 22 independent official Torch comparisons. Early adapter errors and two resource-guard stops are preserved as historical logs.',
|
||||
'The persistent computation ledger includes a conservative 180-second reserve for early checks; no budget reset after that reserve.',
|
||||
'The text sample is tiny and short. Network NLL checks are not broad task-accuracy validation.','']
|
||||
(root/'FUNCTIONAL_REPORT.md').write_text('\n'.join(text))
|
||||
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,227 @@
|
||||
"""Bounded CPU pilot. One fixed model; calibration selects, heldout only validates."""
|
||||
import argparse
|
||||
import csv
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import resource
|
||||
import time
|
||||
import traceback
|
||||
import urllib.request
|
||||
import numpy as np
|
||||
from .model import StreamingLM, norm
|
||||
|
||||
ATTN_PAIRS=[(3,7),(11,15),(19,23)]
|
||||
MLP_PAIRS=[(5,6),(20,21)]
|
||||
NORM_LAYERS=[3,7,11,15,19,23]
|
||||
REVISION='15852e8c16360a2fea060d615a32b45270f8a8fc'
|
||||
|
||||
|
||||
def write_json(path,obj):
|
||||
tmp=path.with_suffix('.tmp')
|
||||
tmp.write_text(json.dumps(obj,indent=2,allow_nan=False)+'\n')
|
||||
tmp.replace(path)
|
||||
|
||||
|
||||
def error_rows(a,b,samples,metadata):
|
||||
records=[]
|
||||
for x,y,sample in zip(a,b,samples):
|
||||
xf,yf=x.astype(np.float64).reshape(-1),y.astype(np.float64).reshape(-1)
|
||||
xn,yn=np.linalg.norm(xf),np.linalg.norm(yf)
|
||||
delta=np.linalg.norm(xf-yf)
|
||||
tokenerr=np.linalg.norm((x-y).reshape(x.shape[0],-1).astype(np.float64),axis=-1)/np.maximum(np.linalg.norm(x.reshape(x.shape[0],-1).astype(np.float64),axis=-1),1e-12)
|
||||
records.append({**metadata,'sample_id':sample['id'],'split':sample['split'],
|
||||
'relative_l2':float(delta/max(xn,1e-12)),
|
||||
'reverse_relative_l2':float(delta/max(yn,1e-12)),
|
||||
'token_relative_p95':float(np.quantile(tokenerr,.95)),
|
||||
'reverse_token_relative_p95':float(np.quantile(np.linalg.norm((x-y).reshape(x.shape[0],-1).astype(np.float64),axis=-1)/np.maximum(np.linalg.norm(y.reshape(x.shape[0],-1).astype(np.float64),axis=-1),1e-12),.95)),
|
||||
'cosine':float(xf@yf/max(xn*yn,1e-30)),
|
||||
'low_norm_tokens':int((np.linalg.norm(x.reshape(x.shape[0],-1),axis=-1)<1e-8).sum())})
|
||||
return records
|
||||
|
||||
|
||||
def probability_metrics(a,b):
|
||||
m=(a+b)*.5
|
||||
def term(p):
|
||||
return np.where(p>0,p*np.log(np.maximum(p,1e-30)/np.maximum(m,1e-30)),0)
|
||||
js=(term(a)+term(b)).sum(axis=-1)*.5
|
||||
overlap=[]
|
||||
for row in range(a.shape[-2]):
|
||||
k=min(5,row+1)
|
||||
ai=np.argsort(a[:,:,row,:row+1],axis=-1)[...,-k:]
|
||||
bi=np.argsort(b[:,:,row,:row+1],axis=-1)[...,-k:]
|
||||
overlap.append(np.mean((ai[...,None]==bi[...,None,:]).any(axis=-1),axis=-1))
|
||||
# Attention-sink control: remove key zero, renormalize valid remaining mass.
|
||||
aa,bb=a[...,1:],b[...,1:]
|
||||
mass_a,mass_b=aa.sum(-1),bb.sum(-1)
|
||||
valid=(mass_a>1e-12)&(mass_b>1e-12)
|
||||
aa=aa/np.maximum(mass_a[...,None],1e-30)
|
||||
bb=bb/np.maximum(mass_b[...,None],1e-30)
|
||||
mm=(aa+bb)*.5
|
||||
conditional_js=.5*(np.where(aa>0,aa*np.log(np.maximum(aa,1e-30)/np.maximum(mm,1e-30)),0)+np.where(bb>0,bb*np.log(np.maximum(bb,1e-30)/np.maximum(mm,1e-30)),0)).sum(-1)
|
||||
return {'first_key_mass_a':float(a[...,0].mean()),'first_key_mass_b':float(b[...,0].mean()),
|
||||
'js_without_first_key':float(conditional_js[valid].mean()) if valid.any() else None,
|
||||
'without_first_key_valid_rows':int(valid.sum()),'js_mean':float(js.mean()),'js_p95':float(np.quantile(js,.95)),
|
||||
'js_without_first_query':float(js[...,1:].mean()),
|
||||
'top5_overlap':float(np.stack(overlap,axis=-1).mean())}
|
||||
|
||||
|
||||
def save_csv(path,rows):
|
||||
if not rows:return
|
||||
keys=list(dict.fromkeys(k for r in rows for k in r))
|
||||
with path.open('w',newline='') as f:
|
||||
w=csv.DictWriter(f,fieldnames=keys);w.writeheader();w.writerows(rows)
|
||||
|
||||
|
||||
def tokenize(root):
|
||||
from tokenizers import Tokenizer
|
||||
path=root/'tokenizer.json'
|
||||
if not path.exists():
|
||||
url=f'https://huggingface.co/Qwen/Qwen3.5-2B/resolve/{REVISION}/tokenizer.json'
|
||||
with urllib.request.urlopen(url,timeout=180) as r:path.write_bytes(r.read())
|
||||
tokenizer=Tokenizer.from_file(str(path))
|
||||
samples=json.loads(Path(__file__).with_name('samples.json').read_text())['samples']
|
||||
ids=[]
|
||||
for s in samples:
|
||||
tokens=tokenizer.encode(s['text'],add_special_tokens=False).ids
|
||||
if len(tokens)<32:raise ValueError(f'Text too short: {s["id"]}')
|
||||
s['text_sha256']=hashlib.sha256(s['text'].encode()).hexdigest()
|
||||
s['full_token_count']=len(tokens);s['token_ids']=tokens[:32]
|
||||
ids.append(tokens[:32])
|
||||
write_json(root/'samples-tokenized.json',{'revision':REVISION,'tokenizer_sha256':hashlib.sha256(path.read_bytes()).hexdigest(),'samples':samples})
|
||||
return samples,np.array(ids,dtype=np.int64)
|
||||
|
||||
|
||||
def main():
|
||||
p=argparse.ArgumentParser();p.add_argument('--root',type=Path,required=True);p.add_argument('--seconds',type=int,default=7200)
|
||||
args=p.parse_args();root=args.root;root.mkdir(exist_ok=True,parents=True)
|
||||
write_json(root/'retention-policy.json',{'weights':'retain_until_explicit_user_delete_request','automatic_cleanup':False,'instruction_date':'2026-11-09'})
|
||||
samples,ids=tokenize(root)
|
||||
cal=np.array([i for i,s in enumerate(samples) if s['split']=='calibration'])
|
||||
held=np.array([i for i,s in enumerate(samples) if s['split']=='heldout'])
|
||||
# Freeze all candidates/thresholds before opening weights or measuring functions.
|
||||
experiment={'revision':REVISION,'attention_pairs':ATTN_PAIRS,'mlp_pairs':MLP_PAIRS,'norm_layers':NORM_LAYERS,
|
||||
'seconds':args.seconds,'output_relative_l2_limit':.05,'token_p95_limit':.1,
|
||||
'attention_js_limit':.01,'attention_top5_min':.8,'heldout_nll_mean_delta_limit':.01,
|
||||
'heldout_nll_sequence_p95_delta_limit':.05,'max_intervention_candidates':1,
|
||||
'directions_per_selected_candidate':2,'training':False,'search_R':False,'automatic_weight_cleanup':False,
|
||||
'dtype':'original weights decoded to FP32; not a BF16 backend benchmark',
|
||||
'blas_threads':os.environ.get('OPENBLAS_NUM_THREADS')}
|
||||
write_json(root/'experiment.json',experiment)
|
||||
started=time.time();status={'state':'running','started_unix':started,'phase':'baseline'}
|
||||
write_json(root/'functional-status.json',status)
|
||||
rows=[];headrows=[];interventions=[]
|
||||
try:
|
||||
config=json.loads((root/'model_config.json').read_text())
|
||||
lm=StreamingLM(root/'weights',config,seconds=args.seconds)
|
||||
overall_deadline=lm.deadline
|
||||
lm.deadline=min(overall_deadline,time.monotonic()+2400)
|
||||
lm.capture_qk=True
|
||||
capture=[('attn',i) for pair in ATTN_PAIRS for i in pair]+[('mlp',i) for pair in MLP_PAIRS for i in pair]
|
||||
def progress(layer):
|
||||
status.update(last_completed_layer=layer,elapsed_seconds=time.time()-started)
|
||||
write_json(root/'functional-status.json',status)
|
||||
print(json.dumps({'phase':status['phase'],'layer':layer,'elapsed':time.time()-started}),flush=True)
|
||||
h,saved=lm.forward(ids,capture=capture,progress=progress)
|
||||
np.savez(root/'baseline-activations.npz',**saved,**{f'qraw_{k}':v[0] for k,v in lm.qk_inputs.items()},**{f'kraw_{k}':v[1] for k,v in lm.qk_inputs.items()},final=h)
|
||||
# Baseline heldout NLL uses no selection information.
|
||||
nll=lm.nll(h[held],ids[held]);np.save(root/'baseline-heldout-token-nll.npy',nll)
|
||||
baseline_nll=nll.mean(axis=1)
|
||||
lm.capture_qk=False
|
||||
lm.deadline=min(overall_deadline,time.monotonic()+1800)
|
||||
status['phase']='calibration';write_json(root/'functional-status.json',status)
|
||||
candidates=[]
|
||||
for kind,pairs in [('attn',ATTN_PAIRS),('mlp',MLP_PAIRS)]:
|
||||
for i,j in pairs:
|
||||
pairid=f'{kind}_{i}_{j}';pairrows=[]
|
||||
for source in (i,j):
|
||||
x=saved[f'{kind}_{source}'][cal]
|
||||
if kind=='attn':
|
||||
a,pa=lm.full_attention(x,i);b,pb=lm.full_attention(x,j)
|
||||
pm={}
|
||||
for bi,si in enumerate(cal):
|
||||
for hi in range(pa.shape[1]):
|
||||
hpm=probability_metrics(pa[bi:bi+1,hi:hi+1],pb[bi:bi+1,hi:hi+1])
|
||||
headrows.append({'candidate':pairid,'source':source,'sample_id':samples[si]['id'],'head':hi,**hpm})
|
||||
save_csv(root/'attention-head-metrics.csv',headrows)
|
||||
else:
|
||||
a,b=lm.mlp(x,i),lm.mlp(x,j);pm={}
|
||||
rec=error_rows(a,b,[samples[t] for t in cal],{'candidate':pairid,'kind':kind,'source':source,**pm})
|
||||
if kind=='attn':
|
||||
for bi,r in enumerate(rec):r.update(probability_metrics(pa[bi:bi+1],pb[bi:bi+1]))
|
||||
rows.extend(rec);pairrows.extend(rec)
|
||||
save_csv(root/'calibration-metrics.csv',rows)
|
||||
passing=all(r['relative_l2']<=.05 and r['reverse_relative_l2']<=.05 and r['token_relative_p95']<=.1 and r['reverse_token_relative_p95']<=.1 for r in pairrows)
|
||||
if kind=='attn':passing=passing and all(r['js_mean']<=.01 and r['top5_overlap']>=.8 for r in pairrows)
|
||||
prefix=f'model.language_model.layers.{i}.'+('self_attn.' if kind=='attn' else 'mlp.')
|
||||
size=sum(t.bytes for name,t in lm.tensors.items() if name.startswith(prefix))
|
||||
candidates.append({'id':pairid,'kind':kind,'i':i,'j':j,'passed':passing,'maximum_symmetric_relative_error':max(max(r['relative_l2'],r['reverse_relative_l2']) for r in pairrows),'donor_component_bytes':size})
|
||||
for layer in NORM_LAYERS:
|
||||
pairid=f'qk_norm_{layer}';pfx=f'layers.{layer}.self_attn.'
|
||||
qw,kw=lm.small(pfx+'q_norm.weight'),lm.small(pfx+'k_norm.weight')
|
||||
pairrows=[]
|
||||
for source,x in zip(('q','k'),lm.qk_inputs[layer]):
|
||||
a,b=norm(x[cal],qw,lm.eps),norm(x[cal],kw,lm.eps)
|
||||
rec=error_rows(a,b,[samples[t] for t in cal],{'candidate':pairid,'kind':'qk_norm','source':source})
|
||||
pairrows.extend(rec);rows.extend(rec)
|
||||
passing=all(r['relative_l2']<=.05 and r['reverse_relative_l2']<=.05 and r['token_relative_p95']<=.1 and r['reverse_token_relative_p95']<=.1 for r in pairrows)
|
||||
candidates.append({'id':pairid,'kind':'qk_norm','i':layer,'j':layer,'passed':passing,'maximum_symmetric_relative_error':max(max(r['relative_l2'],r['reverse_relative_l2']) for r in pairrows),'donor_component_bytes':lm.tensors['model.language_model.'+pfx+'q_norm.weight'].bytes})
|
||||
save_csv(root/'calibration-metrics.csv',rows)
|
||||
write_json(root/'candidate-summary.json',candidates)
|
||||
passed=[c for c in candidates if c['passed']]
|
||||
# Prefer larger potential savings, then smaller worst calibration error; never consult heldout.
|
||||
selected=sorted(passed,key=lambda c:(-c['donor_component_bytes'],c['maximum_symmetric_relative_error'],c['id']))[:1]
|
||||
write_json(root/'selection.json',{'selected':selected,'rule':'calibration pass; descending component bytes, ascending maximum symmetric error; maximum one candidate'})
|
||||
if selected:
|
||||
c=selected[0]
|
||||
heldrows=[]
|
||||
if c['kind']=='qk_norm':
|
||||
pfx=f'layers.{c["i"]}.self_attn.'
|
||||
qw,kw=lm.small(pfx+'q_norm.weight'),lm.small(pfx+'k_norm.weight')
|
||||
for source,x in zip(('q','k'),lm.qk_inputs[c['i']]):
|
||||
heldrows.extend(error_rows(norm(x[held],qw,lm.eps),norm(x[held],kw,lm.eps),[samples[t] for t in held],{'candidate':c['id'],'kind':'qk_norm','source':source}))
|
||||
else:
|
||||
for source in (c['i'],c['j']):
|
||||
x=saved[f'{c["kind"]}_{source}'][held]
|
||||
if c['kind']=='attn':
|
||||
a,_=lm.full_attention(x,c['i']);b,_=lm.full_attention(x,c['j'])
|
||||
else:a,b=lm.mlp(x,c['i']),lm.mlp(x,c['j'])
|
||||
heldrows.extend(error_rows(a,b,[samples[t] for t in held],{'candidate':c['id'],'kind':c['kind'],'source':source}))
|
||||
save_csv(root/'heldout-function-metrics.csv',heldrows)
|
||||
held_function_pass=all(r['relative_l2']<=.05 and r['reverse_relative_l2']<=.05 and r['token_relative_p95']<=.1 and r['reverse_token_relative_p95']<=.1 for r in heldrows)
|
||||
status['heldout_function_pass']=held_function_pass
|
||||
lm.deadline=min(overall_deadline,time.monotonic()+1800)
|
||||
status['phase']='intervention';write_json(root/'functional-status.json',status)
|
||||
for reverse in (False,True):
|
||||
replacement=None;lm.norm_replacement=None
|
||||
if c['kind']=='qk_norm':
|
||||
pfx=f'layers.{c["i"]}.self_attn.'
|
||||
target,donor=('q_norm','k_norm') if not reverse else ('k_norm','q_norm')
|
||||
lm.norm_replacement=(pfx+target,pfx+donor)
|
||||
label=target+'_uses_'+donor
|
||||
else:
|
||||
target,donor=(c['i'],c['j']) if not reverse else (c['j'],c['i'])
|
||||
replacement=(c['kind'],target,donor);label=f'{target}_uses_{donor}'
|
||||
t=time.monotonic();modified,_=lm.forward(ids[held],replacement=replacement,progress=progress)
|
||||
changed=lm.nll(modified,ids[held]);np.save(root/f'intervention-{label}-token-nll.npy',changed)
|
||||
delta=changed.mean(axis=1)-baseline_nll
|
||||
rec={'candidate':c['id'],'direction':label,'elapsed_seconds':time.monotonic()-t,
|
||||
'nll_mean_delta':float(delta.mean()),'nll_sequence_p95_delta':float(np.quantile(delta,.95)),
|
||||
'sequence_nll_deltas':delta.tolist(),'passed_quality':bool(delta.mean()<=.01 and np.quantile(delta,.95)<=.05)}
|
||||
interventions.append(rec);write_json(root/'interventions.json',interventions)
|
||||
# No fallback candidate if heldout fails.
|
||||
status.update(state='completed',phase='done',selected_candidate=selected,
|
||||
positive_bidirectional_quality=bool(interventions and status.get('heldout_function_pass') and all(x['passed_quality'] for x in interventions)),
|
||||
elapsed_seconds=time.time()-started,peak_rss_kib=resource.getrusage(resource.RUSAGE_SELF).ru_maxrss,
|
||||
claim_limit='Small text-only heldout pilot; no global interchangeability proof, no measured GPU speedup; component storage savings require explicit layout accounting.')
|
||||
except Exception as exc:
|
||||
status.update(state='stopped',error_type=type(exc).__name__,error=str(exc),elapsed_seconds=time.time()-started,
|
||||
peak_rss_kib=resource.getrusage(resource.RUSAGE_SELF).ru_maxrss)
|
||||
(root/'functional-error.txt').write_text(traceback.format_exc())
|
||||
raise
|
||||
finally:
|
||||
write_json(root/'functional-status.json',status)
|
||||
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"origin": "Experiment-authored plain text, no external corpus; fixed before functional metrics. Chinese/English balanced in each split. No chat template. First 32 tokenizer tokens retained.",
|
||||
"samples": [
|
||||
{"id":"cal-zh-1","split":"calibration","language":"zh","text":"清晨,研究员打开实验记录本,检查昨天保存的数据。他先确认文件完整,再比较两组观测结果。为了避免偶然误差,他为每一步留下时间和条件,并将尚未验证的推测单独记录。"},
|
||||
{"id":"cal-zh-2","split":"calibration","language":"zh","text":"一辆列车从山间驶向海边,经过村庄、桥梁和开阔的田野。旅客望着窗外变化的景色,计划抵达后先步行到旅馆,再去市场购买晚餐需要的食材。天气预报说傍晚可能下雨。"},
|
||||
{"id":"cal-en-1","split":"calibration","language":"en","text":"A small library opens every morning beside the river. The librarian checks returned books, repairs damaged covers, and places new arrivals on a wooden table. Visitors often stop to read before continuing their journey across the old bridge."},
|
||||
{"id":"cal-en-2","split":"calibration","language":"en","text":"To compare two measurements, first record their units and the conditions under which they were collected. Compute the difference, inspect unusual observations, and repeat the procedure with independent examples. A short report should separate observations from explanations and unresolved questions."},
|
||||
{"id":"test-zh-1","split":"heldout","language":"zh","text":"午后,工程师检查温室中的传感器。温度记录出现一个异常值,她没有立刻删除,而是查看设备状态并补做一次测量。确认原因以后,她调整了采样计划,在报告中说明这次变化。"},
|
||||
{"id":"test-zh-2","split":"heldout","language":"zh","text":"周末,家人沿着湖边的小路散步。他们带了水和简单的午餐,打算在树林旁休息。途中遇到一处正在维修的桥,于是参考路标选择另一条道路,终于在日落前回到车站。"},
|
||||
{"id":"test-en-1","split":"heldout","language":"en","text":"The gardener planted several rows of beans behind the house. Each evening she checked the soil and wrote down how much water the plants received. After a week of warm weather, the first green shoots appeared near the fence beside the apple tree."},
|
||||
{"id":"test-en-2","split":"heldout","language":"en","text":"When a program reads a table, it should check column names and handle missing entries explicitly. The developer tested a small example, compared the output with a hand calculation, and saved the input alongside the result so another person could reproduce the check."}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
"""Measure an explicit two-component alias storage layout, not inference speed.
|
||||
|
||||
Payloads remain on the server under weights/ and must not be committed or cleaned
|
||||
without the user's explicit deletion request. Manifest/byte/hash evidence may be archived.
|
||||
"""
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import numpy as np
|
||||
from pilot.tensors import load_tensors,decode
|
||||
from .run import write_json
|
||||
|
||||
|
||||
def payload(t,path):
|
||||
digest=hashlib.sha256()
|
||||
with path.open('wb') as f:
|
||||
for start in range(0,t.raw.size,262144):
|
||||
b=t.raw[start:start+262144].tobytes();f.write(b);digest.update(b)
|
||||
t.evict()
|
||||
return {'file':path.name,'dtype':t.dtype,'shape':list(t.shape),'bytes':path.stat().st_size,'sha256':digest.hexdigest()}
|
||||
|
||||
|
||||
def main():
|
||||
p=argparse.ArgumentParser();p.add_argument('--root',type=Path,required=True);a=p.parse_args();root=a.root
|
||||
selected=json.loads((root/'selection.json').read_text())['selected']
|
||||
if not selected:
|
||||
write_json(root/'storage-evidence.json',{'measured':False,'reason':'No calibration candidate selected'});return
|
||||
c=selected[0]
|
||||
if c['kind']!='qk_norm':
|
||||
write_json(root/'storage-evidence.json',{'measured':False,'reason':'Only two-gain alias layout implemented in this bounded pilot; larger-module savings remain theoretical'});return
|
||||
tensors=load_tensors(root/'weights')
|
||||
prefix=f'model.language_model.layers.{c["i"]}.self_attn.'
|
||||
names=[prefix+'q_norm.weight',prefix+'k_norm.weight']
|
||||
# Original pair plus two alternative layouts; each is counted independently.
|
||||
path=root/'component-layout'/'weights';path.mkdir(parents=True,exist_ok=True)
|
||||
original={'format':'two_component_raw_v1','tensors':{}}
|
||||
for short,name in zip(('q','k'),names):original['tensors'][name]=payload(tensors[name],path/f'{short}.bin')
|
||||
original_bytes=json.dumps(original,separators=(',',':'),sort_keys=True).encode()
|
||||
(path/'original-layout.json').write_bytes(original_bytes)
|
||||
evidence=[]
|
||||
for target,donor in ((names[0],names[1]),(names[1],names[0])):
|
||||
entry=original['tensors'][donor]
|
||||
layout={'format':'two_component_raw_v1','tensors':{donor:entry},'aliases':{target:donor}}
|
||||
encoded=json.dumps(layout,separators=(',',':'),sort_keys=True).encode()
|
||||
file=path/('share-'+('k' if donor==names[1] else 'q')+'.json');file.write_bytes(encoded)
|
||||
t=tensors[donor]
|
||||
raw=np.fromfile(path/entry['file'],dtype=t.raw.dtype)
|
||||
reconstructed=decode(raw,t.dtype).reshape(t.shape)
|
||||
np.testing.assert_array_equal(reconstructed,t.read().reshape(t.shape));t.evict()
|
||||
before=sum(e['bytes'] for e in original['tensors'].values())+len(original_bytes)
|
||||
after=entry['bytes']+len(encoded)
|
||||
evidence.append({'target':target,'donor':donor,'original_payload_bytes':sum(e['bytes'] for e in original['tensors'].values()),
|
||||
'shared_payload_bytes':entry['bytes'],'original_descriptor_bytes':len(original_bytes),
|
||||
'shared_descriptor_bytes':len(encoded),'original_total_bytes':before,'shared_total_bytes':after,
|
||||
'net_saved_bytes':before-after,'shared_descriptor_sha256':hashlib.sha256(encoded).hexdigest(),
|
||||
'decoded_donor_exactly_matches_intervention_gain':True})
|
||||
write_json(root/'storage-evidence.json',{'measured':True,'candidate':c['id'],'layouts':evidence,
|
||||
'claim_limit':'Measured standalone component layout including descriptors, not an integrated compressed checkpoint, latency gain or full-model RSS reduction. Two sharing directions are alternatives and must not be added.'})
|
||||
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,95 @@
|
||||
"""One bounded CPU run; waits for verified download and binary dependencies.
|
||||
|
||||
No installation, retry search, cleanup, training, GPU selection or Git mutation.
|
||||
"""
|
||||
import argparse
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import json
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import traceback
|
||||
from .run import write_json
|
||||
|
||||
|
||||
def main():
|
||||
p=argparse.ArgumentParser();p.add_argument('--root',type=Path,required=True);p.add_argument('--download-wait-seconds',type=int,default=14400)
|
||||
a=p.parse_args();root=a.root.resolve();start=time.monotonic()
|
||||
state={'state':'waiting','phase':'dependencies','weight_retention':'until_explicit_user_delete_request'}
|
||||
def status(**kwargs):
|
||||
state.update(kwargs,supervisor_elapsed_seconds=time.monotonic()-start)
|
||||
write_json(root/'supervisor-status.json',state)
|
||||
def execute(module,args,timeout,log):
|
||||
ledger_path=root/'compute-budget-ledger.json'
|
||||
ledger=json.loads(ledger_path.read_text()) if ledger_path.exists() else {'spent_seconds':0.,'calls':[]}
|
||||
timeout=min(timeout,int(7200-ledger['spent_seconds']))
|
||||
if timeout<1:raise TimeoutError('Cumulative two-hour compute budget exhausted')
|
||||
t=time.monotonic()
|
||||
try:
|
||||
with (root/log).open('w') as out:
|
||||
r=subprocess.run([sys.executable,'-B','-m',module,*args],stdout=out,stderr=subprocess.STDOUT,timeout=timeout)
|
||||
finally:
|
||||
elapsed=time.monotonic()-t
|
||||
ledger['spent_seconds']+=elapsed
|
||||
ledger['calls'].append({'module':module,'seconds':elapsed})
|
||||
write_json(ledger_path,ledger)
|
||||
if r.returncode:raise RuntimeError(f'{module} failed: exit {r.returncode}, see {log}')
|
||||
status()
|
||||
try:
|
||||
while True:
|
||||
ready=False
|
||||
if all(importlib.util.find_spec(name) for name in ('torch','tokenizers')):
|
||||
try:
|
||||
r=subprocess.run([sys.executable,'-B','-c','import torch, tokenizers; torch.empty(1)'],capture_output=True,timeout=30)
|
||||
ready=r.returncode==0
|
||||
except subprocess.TimeoutExpired:pass
|
||||
if ready:break
|
||||
if time.monotonic()-start>3600:raise TimeoutError('Binary reference dependencies not ready within one hour')
|
||||
status()
|
||||
time.sleep(15)
|
||||
compute_start=time.monotonic()
|
||||
status(state='running',phase='reference_check')
|
||||
provenance={str(p):hashlib.sha256(p.read_bytes()).hexdigest() for p in sorted(Path('functional').glob('*')) if p.is_file()}
|
||||
write_json(root/'functional-code-hashes.json',provenance)
|
||||
execute('functional.reference_check',['--source',str(root/'reference.py'),'--output',str(root/'reference-check.json')],300,'reference-check.log')
|
||||
reference_seconds=time.monotonic()-compute_start
|
||||
status(state='waiting',phase='verified_weights',reference_seconds=reference_seconds)
|
||||
while True:
|
||||
path=root/'manifest.json'
|
||||
try:
|
||||
manifest=json.loads(path.read_text())
|
||||
weights=[f for f in manifest.get('files',[]) if f['name'].endswith('.safetensors')]
|
||||
ready=(manifest.get('completed_unix') is not None and weights and
|
||||
all((root/'weights'/f['name']).is_file() and (root/'weights'/f['name']).stat().st_size==f['bytes'] and f['sha256']==f['upstream_sha256'] for f in weights))
|
||||
except (OSError,ValueError,KeyError):ready=False
|
||||
if ready:break
|
||||
if time.monotonic()-start>a.download_wait_seconds:raise TimeoutError('Verified weight download not ready before wait deadline')
|
||||
status()
|
||||
time.sleep(15)
|
||||
status(state='running',phase='real_weight_preflight')
|
||||
compute_start=time.monotonic()
|
||||
execute('functional.preflight',['--root',str(root)],1200,'preflight.log')
|
||||
preflight_seconds=time.monotonic()-compute_start
|
||||
remaining=int(7200-json.loads((root/'compute-budget-ledger.json').read_text())['spent_seconds']-60)
|
||||
if remaining<60:raise TimeoutError('No remaining bounded functional computation budget')
|
||||
status(phase='functional_analysis',preflight_seconds=preflight_seconds,remaining_seconds=remaining)
|
||||
execute('functional.run',['--root',str(root),'--seconds',str(remaining)],remaining+15,'functional-run.log')
|
||||
status(phase='storage_accounting')
|
||||
execute('functional.storage',['--root',str(root)],120,'storage-accounting.log')
|
||||
status(phase='report')
|
||||
execute('functional.report',['--root',str(root)],120,'functional-report.log')
|
||||
status(state='completed',phase='done')
|
||||
except Exception as exc:
|
||||
status(state='stopped',error_type=type(exc).__name__,error=str(exc))
|
||||
(root/'supervisor-error.txt').write_text(traceback.format_exc())
|
||||
# Retain all weights/results. No automatic retries or candidate expansion.
|
||||
if not (root/'functional-status.json').exists():
|
||||
write_json(root/'functional-status.json',{'state':'stopped','phase':state['phase'],'error_type':type(exc).__name__,'error':str(exc)})
|
||||
try:execute('functional.report',['--root',str(root)],120,'functional-report.log')
|
||||
except Exception:pass
|
||||
raise
|
||||
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,83 @@
|
||||
# Actual-model parameter clouds: frozen grouping and null protocol
|
||||
|
||||
Assets: retained Qwen3.5-2B original15852e8c16360a2fea060d615a32b45270f8a8fc,
|
||||
and pure GSM8K GRPO LoRA seed0 at HF038d8be9bd0edd3883fa019651b02bdf65609468.
|
||||
Use the established read-only FP32 W+2BA wrapper. No training, GPU, new models,
|
||||
weight mutation, inference requirement or deletion. Every retained asset stays.
|
||||
|
||||
## What constitutes a distribution
|
||||
|
||||
A model has no unique canonical number of parameter distributions. Enumerate a
|
||||
fixed family, not "all possible distributions":
|
||||
1. Each ndim>=2 tensor flattened as [axis0, product(other axes)]: row vectors
|
||||
and column vectors, when point count>=16 and feature dimension>=32.
|
||||
2. Decoder language/MTP full-attention Q and interleaved gate rows per query
|
||||
head; K/V rows per KV head. GDN concatenated Q/K/V rows per corresponding head.
|
||||
No claim that these groups are disjoint or iid; whole-matrix views overlap.
|
||||
3. Primary sample-size stratum has >=64 available points;16..63 is small-sample
|
||||
descriptive only. One-dimensional norm/bias vectors have no cloud of vectors;
|
||||
GDN kernel row dimension4 is low dimensional and excluded with explicit reason.
|
||||
Vision parameters are covered by matrix/flattened views, not vision-head splits.
|
||||
|
||||
Include all checkpoint components (language, vision and MTP). Per view choose
|
||||
up to512 point indices without replacement, split deterministically into calibration
|
||||
and held-out halves. Use full vector coordinates up to dimension8192; only larger
|
||||
vectors use a fixed random coordinate subset, labelled projected rather than full.
|
||||
Use identical indices, coordinates and null random draws for original/FT.
|
||||
No overlap-group parameter savings or total distribution counts are additive.
|
||||
|
||||
## Centers and radius/scaling
|
||||
|
||||
Analyze origin and calibration-mean centers separately. Fit R_max using calibration
|
||||
norms only; do not silently clip held-out points outside R_max. Also fit calibration
|
||||
RMS radius. Store held-out maximum, outside-radius mass and radial spread.
|
||||
|
||||
Null A: uniform d-dimensional ball. Simulate the SAME center-fitting and calibration
|
||||
maximum/RMS fitting before held-out evaluation. For origin, also retain exact
|
||||
expected max-fitted null CDF a^d*n_cal/(n_cal+1), a<=1; log probabilities remain
|
||||
available and zero Monte Carlo hits are not exact zeros.
|
||||
Null B: independent coordinate-uniform box bounded by calibration min/max,
|
||||
including asymmetric coordinates. Fit its center/radii using its own calibration
|
||||
sample by exactly the same procedure. Record energy/range mismatch and held-out
|
||||
coordinate violations; this null is conditional on fitted bounds, not a statement
|
||||
that all real vectors lie inside that box.
|
||||
Null C: independent coordinate-uniform box with calibration mean mu and standard
|
||||
deviation sigma, bounds mu +/- sqrt(3)*sigma (sample variance ddof1). This matches
|
||||
first and diagonal second moments in its generating law. Use its own calibration
|
||||
and held-out center/max/RMS fits too; RMS-normalized comparisons target shape
|
||||
rather than a change of global scale. Matching diagonal moments is not covariance
|
||||
matching and does not preserve correlations.
|
||||
|
||||
Use512 validation null draws (not iid model observations), same calibration count
|
||||
as each data cloud. Reuse isotropic-ball nulls only for identical dimension/count
|
||||
and center-fitting rules. Monte Carlo resolution/error must be stated. Fit both
|
||||
centers from the same raw draws where applicable; paired FT uses common randomness.
|
||||
|
||||
## Metrics and interpretation
|
||||
|
||||
Retain measured norm arrays, empirical CDF grids, max-normalized interior fractions
|
||||
at0.5/0.75/0.9/0.95/0.99 and outside1, RMS-normalized CDF from0.5 to1.5,
|
||||
median radial ratios and radial spread. Prefer probability-point differences over
|
||||
astronomical ratios against tiny a^d. Inspect positive AND negative CDF gaps:
|
||||
more inner tail plus more outer tail is not global inward stochastic dominance.
|
||||
|
||||
Record exact zeros, coordinate energy participation (sum squares)^2/sum fourths,
|
||||
coordinate-variance participation, empirical covariance/second-moment participation
|
||||
via calibration Gram matrices. Label sample rank ceiling n_cal-1 (mean centered)
|
||||
or n_cal (origin); compare to equally sampled nulls, do not interpret the raw
|
||||
finite-sample participation as true intrinsic dimension.
|
||||
|
||||
Report both full-radius and moment-controlled results for each center, whole-view
|
||||
and head subgroups, primary versus small sample/projection strata, original and
|
||||
changed72-LoRA-target groups. Unchanged parameter views can reuse identical results
|
||||
with explicit references rather than re-read/recompute. Results describe parameters,
|
||||
not activations, model quality, functional duplicates, or pruning gains.
|
||||
|
||||
Execution budget: one CPU, 512 MiB RSS ceiling, 7200 seconds per invocation;
|
||||
checkpoint every group, resume without replacing completed groups. Any incomplete
|
||||
coverage is reported before interpretation. No numerical analysis runs locally.
|
||||
|
||||
Illustrative radial CDFs use these fixed original/origin views, selected by module
|
||||
role before the census completes, not by the resulting similarity/interior score:
|
||||
language layer0 linear-attention Q head0, layer3 full-attention Q head0, and layer0
|
||||
MLP up-projection rows. Their plots are illustrations, not independent replications.
|
||||
@@ -0,0 +1,42 @@
|
||||
# Actual-model shell census
|
||||
|
||||
Read [PROTOCOL.md](PROTOCOL.md) before interpreting the results. This analyzes
|
||||
retained parameter arrays, not model activations or task accuracy.
|
||||
|
||||
Run only on the authorized CPU server from the repository root:
|
||||
|
||||
```sh
|
||||
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -m unittest model_shell.test_analysis -v
|
||||
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -m model_shell.analyze
|
||||
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -m model_shell.report
|
||||
```
|
||||
|
||||
Inputs are the already retained `runs/cpu-qwen35-2b/weights` and
|
||||
`runs/finetune-head-comparison/adapter.safetensors` with their existing JSON
|
||||
configuration and metadata. The adapter is evaluated as FP32 W+2BA. No weights
|
||||
are modified, no training or inference is run, and no files are deleted.
|
||||
|
||||
`runs/model-shell/checkpoints` is a resumable group ledger. Re-running skips
|
||||
completed groups; do not reuse a ledger after changing its protocol/code/inputs.
|
||||
One invocation stops at 7,200 seconds, RSS above 512 MiB or available memory below
|
||||
128 MiB. Resume after addressing a resource stop; do not relax guards silently.
|
||||
|
||||
`results/model-shell` contains the report, group inventory, per-profile metric
|
||||
CSV, compressed complete CDF/control summaries, and measured calibration/held-out
|
||||
radii. The NPZ keys are `GROUPINDEX_MODEL_CENTER_cal` / `_eval`, with zero-based
|
||||
group indices into `inventory.json`. Unchanged fine-tune profiles explicitly
|
||||
reference the original profiles in `profiles.json.gz`; the CSV expands both.
|
||||
Raw null draws are reproducible from the recorded deterministic seeds; null CDFs,
|
||||
scales and participation summaries are stored. `audit.json` verifies coverage,
|
||||
recomputes empirical CDFs from the saved norms, checks finite-sample rank limits,
|
||||
verifies the original asset SHA256 values, and hashes the output artifacts.
|
||||
`example_radii.npz` also saves calibration/held-out radii for the three fixed role
|
||||
examples and their regenerated nulls; their grid CDFs are checked against the
|
||||
original ledger before plotting full empirical step curves from radius zero.
|
||||
|
||||
The count of views is defined by the protocol and is not a unique number of
|
||||
distributions inherent in a neural network. These overlapping views are equally
|
||||
weighted for description; they are not independent replicates. The box nulls
|
||||
match fitted marginal ranges or moments, not correlations. The maximum fitted
|
||||
radius is not the unknown support boundary. Quantities near numerical or Monte
|
||||
Carlo resolution cannot support a definitive population statement.
|
||||
@@ -0,0 +1,124 @@
|
||||
"""CPU-only, read-only parameter cloud census with paired uniform references."""
|
||||
import argparse, hashlib, json, math, os, time
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
import numpy as np
|
||||
from pilot.tensors import load_tensors
|
||||
from finetune_compare.lora import install
|
||||
from model_shell.groups import enumerate_groups
|
||||
|
||||
MAX_GRID=np.array([.5,.75,.9,.95,.99,1.,1.05,1.1,1.25])
|
||||
RMS_GRID=np.linspace(.5,1.5,41)
|
||||
|
||||
def seed(key):return int.from_bytes(hashlib.sha256(key.encode()).digest()[:8],'little')
|
||||
|
||||
def guard(start):
|
||||
mem=dict((s.split(':')[0],int(s.split()[1])) for s in Path('/proc/meminfo').read_text().splitlines() if s.startswith('MemAvailable:'))
|
||||
rss=int(Path('/proc/self/statm').read_text().split()[1])*os.sysconf('SC_PAGE_SIZE')
|
||||
if rss>512*1024**2 or mem['MemAvailable']<128*1024 or time.monotonic()-start>7200:
|
||||
raise RuntimeError(f'Resource guard: RSS={rss}, available_kB={mem}, elapsed={time.monotonic()-start:.0f}')
|
||||
|
||||
def sample(t,g):
|
||||
rng=np.random.default_rng(seed(g['id']+'/indices'))
|
||||
n=min(512,g['population']);d=min(8192,g['dimension'])
|
||||
points=np.sort(rng.choice(g['population'],n,replace=False))
|
||||
features=np.sort(rng.choice(g['dimension'],d,replace=False)) if d<g['dimension'] else np.arange(d)
|
||||
width=math.prod(t.shape[1:]);out=np.empty((n,d),np.float32)
|
||||
if g['orientation']=='rows':
|
||||
for j,i in enumerate(points):
|
||||
row=i+g['start'];out[j]=t.read(slice(row*width,(row+1)*width))[features]
|
||||
if j%32==0:t.evict()
|
||||
else:
|
||||
for j,i in enumerate(features):
|
||||
out[:,j]=t.read(slice(i*width,(i+1)*width))[points]
|
||||
if j%32==0:t.evict()
|
||||
t.evict();out=out[rng.permutation(n)]
|
||||
digest=hashlib.sha256(points.tobytes()+features.tobytes()).hexdigest()
|
||||
return out,digest
|
||||
|
||||
def participation(y):
|
||||
gram=y@y.T;energy=float(np.trace(gram));den=float(np.square(gram).sum())
|
||||
return energy**2/den if den else 0.
|
||||
|
||||
def geometry(x,ncal,center,raw=False):
|
||||
# Float64 arithmetic; bounded sample matrix, no model-wide conversion.
|
||||
z=x.astype(np.float64);mu=z[:ncal].mean(0) if center=='mean' else np.zeros(z.shape[1])
|
||||
y=z-mu;norm=np.linalg.norm(y,axis=1);cal=norm[:ncal];r=norm[ncal:]
|
||||
maximum=float(cal.max());rms=float(np.sqrt(np.mean(cal**2)))
|
||||
valid=maximum>0 and rms>0
|
||||
a=r/maximum if maximum else np.zeros_like(r);b=r/rms if rms else np.zeros_like(r)
|
||||
coordvar=z[:ncal].var(0,ddof=1);cv=float(coordvar.sum()**2/np.square(coordvar).sum()) if coordvar.any() else 0.
|
||||
e=np.square(y[ncal:]);den=np.square(e).sum(1)
|
||||
pe=np.divide(e.sum(1)**2,den,out=np.zeros(len(e)),where=den>0)
|
||||
out=dict(valid=bool(valid),ncal=ncal,neval=len(r),d=z.shape[1],rmax=maximum,rms=rms,
|
||||
median_max=float(np.median(a)),median_rms=float(np.median(b)),
|
||||
p10_rms=float(np.quantile(b,.1)),p90_rms=float(np.quantile(b,.9)),
|
||||
outside=float(np.mean(a>1)),cdf_max=[float(np.mean(a<=v)) for v in MAX_GRID],
|
||||
cdf_rms=[float(np.mean(b<=v)) for v in RMS_GRID],
|
||||
covariance_participation=participation(y[:ncal]),rank_ceiling=ncal-(center=='mean'),
|
||||
coordinate_variance_participation=cv,point_energy_participation_median=float(np.median(pe)),
|
||||
zero_fraction=float(np.mean(z==0)))
|
||||
if raw:out.update(cal_norm=cal.tolist(),eval_norm=r.tolist())
|
||||
return out
|
||||
|
||||
BALL_CACHE={}
|
||||
def references(x,ncal,key):
|
||||
d=x.shape[1];cachekey=(d,ncal)
|
||||
if cachekey not in BALL_CACHE:
|
||||
rng=np.random.default_rng(seed(f'ball/{d}/{ncal}'))
|
||||
y=rng.standard_normal((ncal+512,d),dtype=np.float32)
|
||||
y*=((rng.random(len(y))**(1/d))/np.linalg.norm(y,axis=1))[:,None]
|
||||
BALL_CACHE[cachekey]={c:geometry(y,ncal,c) for c in ('origin','mean')}
|
||||
cal=x[:ncal].astype(np.float64);lo=cal.min(0);hi=cal.max(0);mu=cal.mean(0);sd=cal.std(0,ddof=1)
|
||||
rng=np.random.default_rng(seed(key+'/uniform'))
|
||||
u=rng.random((ncal+512,d),dtype=np.float32)
|
||||
refs={'ball':BALL_CACHE[cachekey]}
|
||||
for name,left,right in [('range_box',lo,hi),('moment_box',mu-np.sqrt(3)*sd,mu+np.sqrt(3)*sd)]:
|
||||
y=(left+(right-left)*u).astype(np.float32)
|
||||
refs[name]={c:geometry(y,ncal,c) for c in ('origin','mean')}
|
||||
return refs,dict(heldout_outside_coordinate_range=float(np.mean((x[ncal:]<lo)|(x[ncal:]>hi))),
|
||||
range_box_expected_energy=float(np.sum((lo**2+lo*hi+hi**2)/3)),
|
||||
observed_cal_energy=float(np.mean(np.sum(cal**2,axis=1))))
|
||||
|
||||
def analyze(x,key):
|
||||
ncal=len(x)//2;refs,extra=references(x,ncal,key);out={}
|
||||
for center in ('origin','mean'):
|
||||
p=geometry(x,ncal,center,True);p['references']=refs; p.update(extra)
|
||||
p['comparisons']={}
|
||||
for name,bycenter in refs.items():
|
||||
ref=bycenter[center];diff=np.array(p['cdf_rms'])-ref['cdf_rms']
|
||||
p['comparisons'][name]=dict(interior90_gap=p['cdf_max'][2]-ref['cdf_max'][2],
|
||||
interior95_gap=p['cdf_max'][3]-ref['cdf_max'][3],
|
||||
rms_cdf_positive_gap=float(max(0,diff.max())),rms_cdf_negative_gap=float(max(0,-diff.min())),
|
||||
median_rms_delta=p['median_rms']-ref['median_rms'],
|
||||
participation_ratio=p['covariance_participation']/ref['covariance_participation'] if ref['covariance_participation'] else None)
|
||||
p['ball_origin_log_expected_cdf90']=float(x.shape[1]*np.log(.9)+np.log(ncal/(ncal+1)))
|
||||
out[center]=p
|
||||
return out
|
||||
|
||||
def main():
|
||||
ap=argparse.ArgumentParser();ap.add_argument('--root',default='runs');ap.add_argument('--out',default='runs/model-shell');args=ap.parse_args()
|
||||
root=Path(args.root);out=Path(args.out);out.mkdir(exist_ok=True,parents=True);(out/'checkpoints').mkdir(exist_ok=True)
|
||||
cfg=json.loads((root/'cpu-qwen35-2b/model_config.json').read_text())['text_config']
|
||||
base=load_tensors(root/'cpu-qwen35-2b/weights');ft=SimpleNamespace(tensors=dict(base),config=cfg)
|
||||
meta=json.loads((root/'finetune-head-comparison/adapter_meta.json').read_text())
|
||||
pairs,scale=install(ft,root/'finetune-head-comparison',meta)
|
||||
groups,skips=enumerate_groups(base,cfg)
|
||||
manifest=dict(groups=groups,skips=skips,total_tensors=len(base),parameters=sum(t.numel for t in base.values()),changed_tensors=sorted(pairs),adapter_meta=meta,
|
||||
max_grid=MAX_GRID.tolist(),rms_grid=RMS_GRID.tolist(),null_eval=512,point_cap=512,feature_cap=8192)
|
||||
(out/'inventory.json').write_text(json.dumps(manifest,indent=2));start=time.monotonic()
|
||||
for i,g in enumerate(groups):
|
||||
target=out/'checkpoints'/f'{i:04d}.json'
|
||||
if target.exists():continue
|
||||
guard(start);x,digest=sample(base[g['tensor']],g);p=analyze(x,g['id']);del x
|
||||
changed=g['tensor'] in pairs
|
||||
if changed:
|
||||
guard(start);x,digest2=sample(ft.tensors[g['tensor']],g);assert digest==digest2
|
||||
q=analyze(x,g['id']);del x
|
||||
else:q=None
|
||||
record=dict(group=g,sampling_sha256=digest,changed=changed,original=p,finetuned=q,finetuned_reuses_original=not changed)
|
||||
temp=target.with_suffix('.tmp');temp.write_text(json.dumps(record,allow_nan=False));temp.replace(target)
|
||||
if i%10==0:print(f'{i+1}/{len(groups)} elapsed={time.monotonic()-start:.1f}s',flush=True)
|
||||
print('COMPLETE',len(groups),flush=True)
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,29 @@
|
||||
"""Deterministic finite family of matrix/vector clouds; no data-dependent choice."""
|
||||
import math,re
|
||||
|
||||
def enumerate_groups(tensors,cfg):
|
||||
groups=[];skips=[]
|
||||
for name,t in sorted(tensors.items()):
|
||||
if len(t.shape)<2:skips.append({'name':name,'reason':'one vector, no vector cloud','shape':t.shape});continue
|
||||
rows=t.shape[0];cols=math.prod(t.shape[1:]);count=0
|
||||
for axis,n,d in [('rows',rows,cols),('columns',cols,rows)]:
|
||||
if n>=16 and d>=32:
|
||||
groups.append({'id':name+'::'+axis,'tensor':name,'kind':'matrix','orientation':axis,'start':0,'stop':rows,'population':n,'dimension':d,'stratum':'primary' if n>=64 else 'small_sample'});count+=1
|
||||
if not count:skips.append({'name':name,'reason':'insufficient points or dimension <32','shape':t.shape})
|
||||
if not (name.startswith('model.language_model.layers.') or name.startswith('mtp.layers.')):continue
|
||||
segments=[]
|
||||
if name.endswith('.linear_attn.in_proj_qkv.weight'):
|
||||
kh,kd,vh,vd=[cfg[k] for k in ('linear_num_key_heads','linear_key_head_dim','linear_num_value_heads','linear_value_head_dim')]
|
||||
for role,num,dim,offset in [('Q',kh,kd,0),('K',kh,kd,kh*kd),('V',vh,vd,2*kh*kd)]:
|
||||
for h in range(num):segments.append((role,h,offset+h*dim,offset+(h+1)*dim))
|
||||
elif name.endswith('.self_attn.q_proj.weight'):
|
||||
dim=cfg['head_dim'];assert rows==cfg['num_attention_heads']*2*dim
|
||||
for h in range(cfg['num_attention_heads']):
|
||||
for role,offset in [('Q',0),('gate',dim)]:segments.append((role,h,h*2*dim+offset,h*2*dim+offset+dim))
|
||||
elif name.endswith(('.self_attn.k_proj.weight','.self_attn.v_proj.weight')):
|
||||
role='K' if '.k_proj.' in name else 'V';dim=cfg['head_dim'];assert rows==cfg['num_key_value_heads']*dim
|
||||
for h in range(cfg['num_key_value_heads']):segments.append((role,h,h*dim,(h+1)*dim))
|
||||
for role,h,start,stop in segments:
|
||||
assert stop<=rows
|
||||
groups.append({'id':name+f'::{role}_head{h}','tensor':name,'kind':'head_rows','orientation':'rows','start':start,'stop':stop,'population':stop-start,'dimension':cols,'stratum':'primary' if stop-start>=64 else 'small_sample'})
|
||||
return groups,skips
|
||||
@@ -0,0 +1,176 @@
|
||||
"""Validate complete checkpoints; write compact scientific data, figures and report."""
|
||||
import argparse,csv,gzip,hashlib,json,platform,sys
|
||||
from pathlib import Path
|
||||
import numpy as np
|
||||
import matplotlib
|
||||
matplotlib.use('Agg')
|
||||
import matplotlib.pyplot as plt
|
||||
from model_shell.analyze import MAX_GRID,RMS_GRID,sample,seed
|
||||
from pilot.tensors import load_tensors
|
||||
|
||||
def sha(p):
|
||||
h=hashlib.sha256()
|
||||
with p.open('rb') as f:
|
||||
for b in iter(lambda:f.read(1024*1024),b''):h.update(b)
|
||||
return h.hexdigest()
|
||||
|
||||
def main():
|
||||
ap=argparse.ArgumentParser();ap.add_argument('--input',default='runs/model-shell');ap.add_argument('--out',default='results/model-shell');a=ap.parse_args()
|
||||
source=Path(a.input);out=Path(a.out);out.mkdir(parents=True,exist_ok=True);figdir=out/'figures';figdir.mkdir(exist_ok=True)
|
||||
inventory=json.loads((source/'inventory.json').read_text());groups=inventory['groups'];records=[];rows=[];radii={}
|
||||
for i,g in enumerate(groups):
|
||||
r=json.loads((source/'checkpoints'/f'{i:04d}.json').read_text());assert r['group']==g
|
||||
assert r['changed']==(g['tensor'] in inventory['changed_tensors'])
|
||||
for model in ('original','finetuned'):
|
||||
profiles=r[model] if r[model] is not None else r['original']
|
||||
for center,p in profiles.items():
|
||||
if center=='origin':
|
||||
p['ball_log_expected_max_cdf']=[float(p['d']*np.log(z)+np.log(p['ncal']/(p['ncal']+1))) for z in MAX_GRID if z<=1]
|
||||
assert p['valid'] and p['d']==min(8192,g['dimension'])
|
||||
assert p['covariance_participation']<=p['rank_ceiling']+1e-6
|
||||
v=np.asarray(p['eval_norm']);cal=np.asarray(p['cal_norm'])
|
||||
assert abs(p['rmax']-cal.max())<1e-12
|
||||
assert abs(p['outside']-np.mean(v>p['rmax']))<1e-12
|
||||
np.testing.assert_allclose(p['cdf_max'],[np.mean(v<=z*p['rmax']) for z in MAX_GRID])
|
||||
np.testing.assert_allclose(p['cdf_rms'],[np.mean(v<=z*p['rms']) for z in RMS_GRID])
|
||||
key=f'{i}_{model}_{center}';radii[key+'_cal']=cal;radii[key+'_eval']=v
|
||||
row=dict(group_index=i,group=g['id'],model=model,center=center,kind=g['kind'],stratum=g['stratum'],changed=r['changed'],projected=g['dimension']>8192,
|
||||
dimension=g['dimension'],analyzed_dimension=p['d'],population=g['population'],ncal=p['ncal'],neval=p['neval'],sampling_sha256=r['sampling_sha256'])
|
||||
for k in ('median_max','median_rms','p10_rms','p90_rms','outside','rmax','rms','covariance_participation','rank_ceiling','coordinate_variance_participation','point_energy_participation_median','zero_fraction','heldout_outside_coordinate_range'):
|
||||
row[k]=p[k]
|
||||
for name,c in p['comparisons'].items():
|
||||
for k,vv in c.items():row[name+'_'+k]=vv
|
||||
rows.append(row)
|
||||
# Compact profiles keep CDF and controls; raw measured radii live in NPZ.
|
||||
for model in ('original','finetuned'):
|
||||
if r[model] is not None:
|
||||
for p in r[model].values():p.pop('cal_norm');p.pop('eval_norm')
|
||||
records.append(r)
|
||||
with gzip.open(out/'profiles.json.gz','wt',encoding='utf-8') as f:json.dump(records,f,allow_nan=False)
|
||||
np.savez_compressed(out/'measured_radii.npz',**radii)
|
||||
with (out/'metrics.csv').open('w',newline='') as f:
|
||||
w=csv.DictWriter(f,fieldnames=list(rows[0]));w.writeheader();w.writerows(rows)
|
||||
(out/'inventory.json').write_text(json.dumps(inventory,indent=2))
|
||||
def subset(model='original',center='origin',kind=None,changed=None):
|
||||
return [r for r in rows if r['model']==model and r['center']==center and r['stratum']=='primary' and not r['projected'] and (kind is None or r['kind']==kind) and (changed is None or r['changed']==changed)]
|
||||
summary={}
|
||||
for center in ('origin','mean'):
|
||||
for kind in ('matrix','head_rows'):
|
||||
rs=subset(center=center,kind=kind);s={'n':len(rs),
|
||||
'median_zero_fraction':float(np.median([r['zero_fraction'] for r in rs])),
|
||||
'median_coordinate_energy_participation_fraction':float(np.median([r['point_energy_participation_median']/r['analyzed_dimension'] for r in rs])),
|
||||
'median_coordinate_variance_participation_fraction':float(np.median([r['coordinate_variance_participation']/r['analyzed_dimension'] for r in rs]))}
|
||||
for ref in ('ball','range_box','moment_box'):
|
||||
gap=np.array([r[ref+'_interior90_gap'] for r in rs]);pos=np.array([r[ref+'_rms_cdf_positive_gap'] for r in rs]);neg=np.array([r[ref+'_rms_cdf_negative_gap'] for r in rs])
|
||||
s[ref]=dict(median_interior90_gap=float(np.median(gap)),fraction_gap_gt10pp=float(np.mean(gap>.1)),median_positive_rms_gap=float(np.median(pos)),median_negative_rms_gap=float(np.median(neg)),
|
||||
fraction_two_sided_rms_gt10pp=float(np.mean((pos>.1)&(neg>.1))),median_participation_ratio=float(np.median([r[ref+'_participation_ratio'] for r in rs])))
|
||||
summary[center+'/'+kind]=s
|
||||
paired={}
|
||||
for center in ('origin','mean'):
|
||||
b=subset(center=center,changed=True);t=subset(model='finetuned',center=center,changed=True);assert [r['group'] for r in b]==[r['group'] for r in t]
|
||||
paired[center]={'n':len(b)}
|
||||
for field in ('median_max','median_rms','covariance_participation','moment_box_interior90_gap'):
|
||||
delta=np.array([q[field]-p[field] for p,q in zip(b,t)])
|
||||
paired[center][field]=dict(median=float(np.median(delta)),p10=float(np.quantile(delta,.1)),p90=float(np.quantile(delta,.9)),max_abs=float(np.max(abs(delta))))
|
||||
secondary=[]
|
||||
for model in ('original','finetuned'):
|
||||
for center in ('origin','mean'):
|
||||
for label in ('small_sample','projected'):
|
||||
ss=[r for r in rows if r['model']==model and r['center']==center and (r['stratum']=='small_sample' if label=='small_sample' else r['projected'])]
|
||||
secondary.append(dict(model=model,center=center,stratum=label,n=len(ss),median_max=float(np.median([r['median_max'] for r in ss])),moment_interior90_gap=float(np.median([r['moment_box_interior90_gap'] for r in ss]))))
|
||||
(out/'summary.json').write_text(json.dumps(dict(strata=summary,paired_changed=paired,secondary=secondary),indent=2))
|
||||
plt.rcParams.update({'font.size':10,'axes.spines.top':False,'axes.spines.right':False})
|
||||
def save(fig,name):
|
||||
fig.tight_layout()
|
||||
for ext in ('png','pdf','svg'):fig.savefig(figdir/(name+'.'+ext),dpi=180,bbox_inches='tight')
|
||||
plt.close(fig)
|
||||
fig,axs=plt.subplots(2,2,figsize=(11,7),sharex=True,sharey=True)
|
||||
for i,center in enumerate(('origin','mean')):
|
||||
for j,kind in enumerate(('matrix','head_rows')):
|
||||
rs=subset(center=center,kind=kind);ax=axs[i,j]
|
||||
for ref,label in [('ball','Uniform ball'),('range_box','Range-matched box'),('moment_box','Moment-matched box')]:
|
||||
v=np.sort([r[ref+'_interior90_gap'] for r in rs]);ax.plot(v,np.arange(1,len(v)+1)/len(v),label=label)
|
||||
ax.axvline(0,color='gray',lw=.8);ax.set_title(f'{center} center / {kind}, n={len(rs)}');ax.set_xlabel('F(data r/Rcal <= 0.9) - F(null)');ax.set_ylabel('Fraction of sampled groups')
|
||||
axs[0,0].legend(fontsize=8);fig.suptitle('Actual Qwen3.5-2B: interior mass depends on the reference');save(fig,'interior_reference_sensitivity')
|
||||
fig,axs=plt.subplots(1,2,figsize=(11,4.3))
|
||||
for ax,center in zip(axs,('origin','mean')):
|
||||
for kind,marker in [('matrix','o'),('head_rows','.')]:
|
||||
rs=subset(center=center,kind=kind);ax.scatter([r['moment_box_rms_cdf_positive_gap'] for r in rs],[r['moment_box_rms_cdf_negative_gap'] for r in rs],s=10,alpha=.25,label=kind,marker=marker)
|
||||
ax.axvline(.1,color='gray',lw=.7);ax.axhline(.1,color='gray',lw=.7);ax.set(xlabel='Largest positive CDF gap (more inner tail)',ylabel='Largest negative CDF gap (more outer tail)',title=f'{center} / moment-matched box',xlim=(-.02,1),ylim=(-.02,1));ax.legend()
|
||||
fig.suptitle('RMS-normalized radial shape: both tails can grow together');save(fig,'radial_shape_two_tails')
|
||||
fig,axs=plt.subplots(1,2,figsize=(11,4.3))
|
||||
for ax,center in zip(axs,('origin','mean')):
|
||||
b=subset(center=center,changed=True);t=subset(model='finetuned',center=center,changed=True)
|
||||
x=[r['median_max'] for r in b];y=[r['median_max'] for r in t];ax.scatter(x,y,s=10,alpha=.3);low=min(x+y);high=max(x+y);ax.plot([low,high],[low,high],color='gray',lw=1)
|
||||
ax.set(xlabel='Original median r/Rcal',ylabel='GRPO-LoRA median r/Rcal',title=f'{center}, changed primary groups n={len(b)}')
|
||||
fig.suptitle('Paired same-index comparison; unchanged groups excluded');save(fig,'finetune_paired_radii')
|
||||
fig,axs=plt.subplots(1,2,figsize=(11,4.3))
|
||||
for ax,center in zip(axs,('origin','mean')):
|
||||
for kind in ('matrix','head_rows'):
|
||||
rs=subset(center=center,kind=kind);v=np.sort([r['moment_box_participation_ratio'] for r in rs]);ax.plot(v,np.arange(1,len(v)+1)/len(v),label=kind)
|
||||
ax.set(xlabel='Empirical participation / equal-sample moment-box reference',ylabel='Fraction of groups',title=center);ax.axvline(1,color='gray',lw=.8);ax.legend()
|
||||
fig.suptitle('Finite-sample participation is not the true intrinsic dimension');save(fig,'participation_reference')
|
||||
examples=[('model.language_model.layers.0.linear_attn.in_proj_qkv.weight::Q_head0','L0 GDN Q head0'),
|
||||
('model.language_model.layers.3.self_attn.q_proj.weight::Q_head0','L3 full-attention Q head0'),
|
||||
('model.language_model.layers.0.mlp.up_proj.weight::rows','L0 MLP up rows')]
|
||||
fig,axs=plt.subplots(2,3,figsize=(13,7),sharey=True)
|
||||
byid={r['group']['id']:r for r in records}
|
||||
indices={g['id']:i for i,g in enumerate(groups)}
|
||||
base=load_tensors(Path('runs/cpu-qwen35-2b/weights'));example_radii={}
|
||||
for j,(gid,label) in enumerate(examples):
|
||||
p=byid[gid]['original']['origin']
|
||||
g=byid[gid]['group'];x,digest=sample(base[g['tensor']],g);assert digest==byid[gid]['sampling_sha256']
|
||||
ncal=len(x)//2;d=x.shape[1];cal=x[:ncal].astype(np.float64)
|
||||
rng=np.random.default_rng(seed(gid+'/uniform'));u=rng.random((ncal+512,d),dtype=np.float32)
|
||||
mu=cal.mean(0);sd=cal.std(0,ddof=1);lo=mu-np.sqrt(3)*sd;hi=mu+np.sqrt(3)*sd
|
||||
moment=(lo+(hi-lo)*u).astype(np.float32)
|
||||
rng=np.random.default_rng(seed(f'ball/{d}/{ncal}'));ball=rng.standard_normal((ncal+512,d),dtype=np.float32)
|
||||
ball*=((rng.random(len(ball))**(1/d))/np.linalg.norm(ball,axis=1))[:,None]
|
||||
arrays={'Actual weights':(radii[f'{indices[gid]}_original_origin_cal'],radii[f'{indices[gid]}_original_origin_eval'])}
|
||||
for name,ref,raw in [('Uniform ball','ball',ball),('Moment box','moment_box',moment)]:
|
||||
nr=np.linalg.norm(raw.astype(np.float64),axis=1);arrays[name]=(nr[:ncal],nr[ncal:])
|
||||
np.testing.assert_allclose([np.mean(nr[ncal:]/nr[:ncal].max()<=z) for z in MAX_GRID],p['references'][ref]['origin']['cdf_max'])
|
||||
np.testing.assert_allclose([np.mean(nr[ncal:]/np.sqrt(np.mean(nr[:ncal]**2))<=z) for z in RMS_GRID],p['references'][ref]['origin']['cdf_rms'])
|
||||
for name,(nc,ne) in arrays.items():example_radii[f'example{j}_{name.replace(" ","_")}_cal']=nc;example_radii[f'example{j}_{name.replace(" ","_")}_eval']=ne
|
||||
for i,(field,grid) in enumerate([('cdf_max',MAX_GRID),('cdf_rms',RMS_GRID)]):
|
||||
ax=axs[i,j]
|
||||
for name,(nc,ne) in arrays.items():
|
||||
scale=nc.max() if i==0 else np.sqrt(np.mean(nc**2));v=np.sort(ne/scale)
|
||||
ax.step(np.r_[0,v, max(1.5,float(v[-1]))],np.r_[0,np.arange(1,len(v)+1)/len(v),1],where='post',label=name,lw=1.8,ls='-' if name=='Actual weights' else '--')
|
||||
ax.set(title=label,xlabel='r / calibration '+('maximum' if i==0 else 'RMS'),ylabel='Held-out CDF',ylim=(-.03,1.03),xlim=(0,1.5));ax.legend(fontsize=8)
|
||||
np.savez_compressed(out/'example_radii.npz',**example_radii)
|
||||
fig.suptitle('Fixed role examples: maximum-radius and RMS normalization');save(fig,'fixed_role_radial_cdfs')
|
||||
lines=['# 两个实际2B检查点:参数云的球壳检验','',f'完整覆盖:{len(groups)}组/模型,{len(rows)}个模型×中心组合。632个参数张量,2,274,069,824个参数(包含语言、视觉和MTP)。',
|
||||
'','## 分布计数与范围','', '固定规则给出554个矩阵行/列视图、1,004个语言/MTP头视图;视图互相重叠,不能相加成参数节省量。1,522组有至少64个可用点,36组只有16个点;1组高维嵌入列视图使用8,192个抽样坐标。主表只汇总样本充足且没有投影的1,521组。319个一维张量和18个低维卷积核不构成此规则下的高维样本云。',
|
||||
'','## 原模型结果(逐组等权,非独立观测)','', '|中心|视图|组数|相对均匀球:90%半径内质量差中位数|相对矩匹配均匀盒:质量差中位数|矩匹配且RMS归一后,内外两尾差均>10个百分点的组占比|','|---|---|---:|---:|---:|---:|']
|
||||
for key,s in summary.items():
|
||||
c,k=key.split('/');lines.append(f'|{c}|{k}|{s["n"]}|{s["ball"]["median_interior90_gap"]:.3f}|{s["moment_box"]["median_interior90_gap"]:.3f}|{s["moment_box"]["fraction_two_sided_rms_gt10pp"]:.1%}|')
|
||||
lines+=['','这些是描述性质量差,不是显著性检验。正差表示所指定半径以内有更多样本;不能将其自动解释为整条径向分布向内平移。若RMS归一后内外尾均变多,更准确的解释是径向分布更宽。',
|
||||
'', '|中心/视图|逐组零值率中位数|点坐标能量参与率÷维度,中位数|坐标方差参与率÷维度,中位数|','|---|---:|---:|---:|']
|
||||
for key,s in summary.items():lines.append(f'|{key}|{s["median_zero_fraction"]:.6g}|{s["median_coordinate_energy_participation_fraction"]:.4f}|{s["median_coordinate_variance_participation_fraction"]:.4f}|')
|
||||
lines+=['','能量参与率衡量坐标能量集中程度,依赖坐标基;有限样本Gram参与率反映样本间几何,另见图。两者均不是精确非零坐标计数,也不是模型功能的有效维度。',
|
||||
'','## 微调前后','', '原模型是官方Qwen3.5-2B;第二个模型是在同一revision上进行GSM8K GRPO的已有纯LoRA检查点,不是另一个完整模型。72个投影矩阵使用只读W+2BA展开,其余参数共享。972个视图受影响;其中936个属于主分析。未改变视图复用完全相同的分析结果。','', '|中心|指标|配对变化中位数|P10|P90|最大绝对变化|','|---|---|---:|---:|---:|---:|']
|
||||
for c,p in paired.items():
|
||||
for k,v in p.items():
|
||||
if k!='n':lines.append(f'|{c}|{k}|{v["median"]:.6g}|{v["p10"]:.6g}|{v["p90"]:.6g}|{v["max_abs"]:.6g}|')
|
||||
lines+=['','## 单独描述的小样本与坐标抽样组','', '|模型|中心|类别|组数|中位r/Rcal的组间中位数|相对矩匹配盒的90%半径质量差中位数|','|---|---|---|---:|---:|---:|']
|
||||
for s in secondary:lines.append(f'|{s["model"]}|{s["center"]}|{s["stratum"]}|{s["n"]}|{s["median_max"]:.4f}|{s["moment_interior90_gap"]:.4f}|')
|
||||
lines+=['','## 如何读图','', '1. `interior_reference_sensitivity`:不同均匀参照下,90%校准半径内质量差的组间CDF。横轴向右表示更多内侧质量。','2. `radial_shape_two_tails`:匹配矩且按各自校准RMS归一后的最大正/负CDF差。右上角同时增加两尾,并非单纯向内。','3. `finetune_paired_radii`:同一视图的微调前后中位半径;对角线表示不变。','4. `participation_reference`:Gram矩阵参与率相对同样本量参照的比值。受样本秩上限影响,不能当作真实内在维度。',
|
||||
'5. `fixed_role_radial_cdfs`:预先按角色固定的GDN Q、全注意力Q和MLP上投影示例,上排按校准最大半径、下排按RMS归一化。不是按结果挑选最明显的组。',
|
||||
'','## 方法与限制','', '每组最多512个不同参数向量,一半校准、一半留出;模型固定,留出指未用于拟合中心/半径的参数向量,不是独立训练模型。对照每组512个验证向量,最坏单项二项标准误约2.21个百分点;模型留出样本的误差还取决于样本数,且参数向量不保证独立。10个百分点仅是描述性筛选尺度,没有多重检验或总体显著性结论。',
|
||||
'','均匀球、坐标范围均匀盒、匹配均值和逐坐标方差的均匀盒都按同一规则拟合自身中心、最大半径及RMS。均匀球原点中心的精确拟合半径CDF为a^d*ncal/(ncal+1),保留其对数以避免把模拟零命中误报成零概率。盒对照不是协方差匹配。校准最大半径不是已知总体支撑边界;留出越界率已记录。',
|
||||
'','`origin`保留坐标均值贡献;`mean`使用校准向量均值。没有使用所有数据来选边界、训练或推理;没有删除模型。坐标稀疏性、有限样本参与率与径向内侧质量分别测量,不能相互代替。该实验不能证明更好的模型更靠里,也不能证明功能重复、可剪枝或可复用。',
|
||||
'','复现:`OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -m model_shell.analyze`,完成后 `... python -m model_shell.report`。只在已授权CPU服务器运行。分组协议见 `model_shell/PROTOCOL.md`;全量指标在metrics.csv,完整CDF/参照在profiles.json.gz,实测范数在measured_radii.npz。']
|
||||
(out/'REPORT.md').write_text('\n'.join(lines)+'\n')
|
||||
sources={'base':Path('runs/cpu-qwen35-2b/weights/model.safetensors-00001-of-00001.safetensors'),'adapter':Path('runs/finetune-head-comparison/adapter.safetensors'),
|
||||
'config':Path('runs/cpu-qwen35-2b/model_config.json'),'adapter_meta':Path('runs/finetune-head-comparison/adapter_meta.json')}
|
||||
sourcehash={k:dict(path=str(p),bytes=p.stat().st_size,sha256=sha(p)) for k,p in sources.items()}
|
||||
(out/'environment.json').write_text(json.dumps(dict(python=sys.version,platform=platform.platform(),numpy=np.__version__,matplotlib=matplotlib.__version__,
|
||||
source_code_sha256={str(p):sha(p) for p in sorted(Path('model_shell').glob('*.py'))},protocol_sha256=sha(Path('model_shell/PROTOCOL.md')),seed='first 8 SHA256 bytes, little endian; group id plus purpose suffix',
|
||||
thread_limit=1,server='8.211.154.134',training=False,inference=False,gpu=False),indent=2))
|
||||
assert sourcehash['base']['sha256']=='aa33250c4fc64891ddfaba3a314fd9542ea371843c387178b425fbcc5ed680b1'
|
||||
assert sourcehash['adapter']['sha256']=='93a3f43a4356f84da0d13ecc622fee13e41ba15d237cdaad6094f243899c62d6'
|
||||
audit=dict(complete_groups=len(groups),profile_rows=len(rows),all_finite=True,cdf_recomputed=True,sample_rank_checked=True,sources=sourcehash,files={str(p.relative_to(out)):sha(p) for p in sorted(out.rglob('*')) if p.is_file() and p.name!='audit.json'})
|
||||
(out/'audit.json').write_text(json.dumps(audit,indent=2));print(json.dumps(dict(summary=summary,paired=paired),indent=2))
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,49 @@
|
||||
"""Run on the authorized CPU server: python -m unittest model_shell.test_analysis."""
|
||||
import unittest
|
||||
import numpy as np
|
||||
from model_shell.analyze import sample,geometry,analyze,seed
|
||||
from finetune_compare.lora import EffectiveWeight
|
||||
|
||||
class ArrayTensor:
|
||||
def __init__(self,x):self.x=x;self.shape=x.shape;self.numel=x.size;self.dtype='F32'
|
||||
def read(self,s=slice(None)):return self.x.reshape(-1)[s].copy()
|
||||
def evict(self):pass
|
||||
|
||||
class Checks(unittest.TestCase):
|
||||
def test_sampling_and_lora(self):
|
||||
rng=np.random.default_rng(17);w=rng.normal(size=(80,64)).astype('f');a=rng.normal(size=(2,64)).astype('f');b=rng.normal(size=(80,2)).astype('f')
|
||||
effective=EffectiveWeight(ArrayTensor(w),a,b,2);explicit=ArrayTensor(w+2*b@a)
|
||||
for axis,n,d,start,stop in [('rows',80,64,0,80),('columns',64,80,0,80),('rows',32,64,16,48)]:
|
||||
g=dict(id=axis+str(start),orientation=axis,population=n,dimension=d,start=start,stop=stop)
|
||||
x,h=sample(effective,g);y,k=sample(explicit,g);np.testing.assert_allclose(x,y,atol=2e-6,rtol=1e-5);self.assertEqual(h,k)
|
||||
self.assertEqual(x.shape,(n,d))
|
||||
rr=np.random.default_rng(seed(g['id']+'/indices'));indices=np.sort(rr.choice(n,n,replace=False));order=rr.permutation(n)
|
||||
cloud=explicit.x[start:stop] if axis=='rows' else explicit.x.T
|
||||
np.testing.assert_allclose(y,cloud[indices][order],atol=0,rtol=0)
|
||||
def test_center_scale_and_rank(self):
|
||||
rng=np.random.default_rng(5);x=rng.normal(size=(128,96)).astype('f')
|
||||
for center in ('origin','mean'):
|
||||
p=geometry(x,64,center);q=geometry(x*3,64,center)
|
||||
np.testing.assert_allclose(p['cdf_rms'],q['cdf_rms']);self.assertLessEqual(p['covariance_participation'],p['rank_ceiling']+1e-6)
|
||||
p=geometry(x,64,'mean');q=geometry(x+4,64,'mean');np.testing.assert_allclose(p['median_rms'],q['median_rms'],rtol=1e-6)
|
||||
self.assertFalse(geometry(np.zeros((32,64)),16,'mean')['valid'])
|
||||
def test_ball_fit_and_sparse_counterexample(self):
|
||||
rng=np.random.default_rng(33);d=32;ncal=256
|
||||
# Analytic expectation over independent calibration maxima.
|
||||
c=rng.random((20000,ncal))**(1/d);v=rng.random(20000)**(1/d)
|
||||
observed=np.mean(v<=.99*c.max(1));expected=.99**d*ncal/(ncal+1)
|
||||
self.assertLess(abs(observed-expected),.015)
|
||||
radii=rng.random(512)**(1/d);x=np.zeros((512,d),dtype='f');x[:,0]=radii
|
||||
p=geometry(x,256,'origin');self.assertGreater(p['median_max'],.9)
|
||||
# Sparse support alone retains the full-dimensional radial law.
|
||||
self.assertLess(p['cdf_max'][2],.08)
|
||||
x[:,0]=rng.random(512)**.5
|
||||
self.assertGreater(geometry(x,256,'origin')['cdf_max'][2],.65)
|
||||
sphere=rng.normal(size=(512,d));sphere/=np.linalg.norm(sphere,axis=1)[:,None]
|
||||
self.assertEqual(geometry(sphere,256,'origin')['cdf_max'][2],0.)
|
||||
def test_reproducible_controls(self):
|
||||
x=np.random.default_rng(7).normal(size=(32,40)).astype('f');a=analyze(x,'test');b=analyze(x,'test')
|
||||
self.assertEqual(a,b)
|
||||
self.assertTrue(a['mean']['valid'])
|
||||
|
||||
if __name__=='__main__':unittest.main()
|
||||
@@ -0,0 +1 @@
|
||||
"""Bounded, inference-free weight similarity pilot."""
|
||||
@@ -0,0 +1,396 @@
|
||||
"""Bounded CPU-only screening, exact verification and conservative sharing groups."""
|
||||
import argparse
|
||||
import collections
|
||||
import csv
|
||||
import hashlib
|
||||
import itertools
|
||||
import json
|
||||
import math
|
||||
import pathlib
|
||||
import platform
|
||||
import resource
|
||||
import time
|
||||
|
||||
import numpy as np
|
||||
|
||||
from .tensors import decode, finish, load_tensors, metrics
|
||||
|
||||
THRESHOLDS = [.01, .02, .05, .10, .20]
|
||||
|
||||
|
||||
def write_json(path, obj):
|
||||
path.write_text(json.dumps(obj, indent=2, allow_nan=False))
|
||||
|
||||
|
||||
def write_csv(path, rows):
|
||||
if not rows:
|
||||
path.write_text('')
|
||||
return
|
||||
with path.open('w', newline='') as f:
|
||||
writer = csv.DictWriter(f, fieldnames=list(rows[0]))
|
||||
writer.writeheader()
|
||||
writer.writerows(rows)
|
||||
|
||||
|
||||
def stable_seed(key, seed):
|
||||
return int.from_bytes(hashlib.sha256((str(seed) + str(key)).encode()).digest()[:8], 'little')
|
||||
|
||||
|
||||
def read_object(obj, tensors):
|
||||
t = tensors[obj['tensor']]
|
||||
if obj['kind'] == 'tensor':
|
||||
return t.read()
|
||||
slices = tuple(slice(s, s + b) for s, b in zip(obj['start'], obj['shape']))
|
||||
return decode(t.raw.reshape(t.shape)[slices], t.dtype).ravel()
|
||||
|
||||
|
||||
def sample_object(obj, tensors, sample_size=2048):
|
||||
t = tensors[obj['tensor']]
|
||||
n = math.prod(obj['shape'])
|
||||
indices = np.linspace(0, n - 1, min(n, sample_size), dtype=np.int64)
|
||||
if obj['kind'] == 'tensor':
|
||||
return t.read(indices)
|
||||
coords = np.unravel_index(indices, obj['shape'])
|
||||
global_coords = tuple(c + s for c, s in zip(coords, obj['start']))
|
||||
flat = np.ravel_multi_index(global_coords, t.shape)
|
||||
return t.read(flat)
|
||||
|
||||
|
||||
def exact_pair(a, b, tensors, deadline):
|
||||
if a['kind'] != 'tensor':
|
||||
result = metrics(read_object(a, tensors), read_object(b, tensors))
|
||||
tensors[a['tensor']].evict()
|
||||
tensors[b['tensor']].evict()
|
||||
return result
|
||||
ta, tb = tensors[a['tensor']], tensors[b['tensor']]
|
||||
xx = yy = xy = delta = sx = sy = 0.
|
||||
for start in range(0, ta.numel, 262144):
|
||||
if time.monotonic() > deadline:
|
||||
raise TimeoutError('Analysis deadline reached')
|
||||
sl = slice(start, min(start + 262144, ta.numel))
|
||||
x, y = ta.read(sl).astype(np.float64), tb.read(sl).astype(np.float64)
|
||||
xx += float(x @ x)
|
||||
yy += float(y @ y)
|
||||
xy += float(x @ y)
|
||||
d = x - y
|
||||
delta += float(d @ d)
|
||||
sx += float(x.sum())
|
||||
sy += float(y.sum())
|
||||
ta.evict()
|
||||
tb.evict()
|
||||
ta.evict()
|
||||
tb.evict()
|
||||
return finish(xx, yy, xy, delta, sx, sy, ta.numel)
|
||||
|
||||
|
||||
def sharing_groups(objects, pairs, threshold):
|
||||
"""Star groups; every member must have an exact verified edge to its representative."""
|
||||
adjacent = collections.defaultdict(dict)
|
||||
for pair in pairs:
|
||||
a, b = pair['a'], pair['b']
|
||||
if pair['error_member_to_rep'] <= threshold:
|
||||
adjacent[b][a] = pair['error_member_to_rep']
|
||||
if pair['error_rep_to_member'] <= threshold:
|
||||
adjacent[a][b] = pair['error_rep_to_member']
|
||||
unassigned = set(range(len(objects)))
|
||||
order = sorted(unassigned, key=lambda i: (-len(adjacent[i]), objects[i]['id']))
|
||||
groups = []
|
||||
for rep in order:
|
||||
if rep not in unassigned:
|
||||
continue
|
||||
members = sorted(set(adjacent[rep]) & unassigned)
|
||||
members.append(rep)
|
||||
unassigned.difference_update(members)
|
||||
groups.append({'representative': rep, 'members': members,
|
||||
'max_error': max([adjacent[rep][i] for i in members if i != rep] or [0.])})
|
||||
return groups
|
||||
|
||||
|
||||
def nearest_candidates(objects, features, cap, neighbors=10):
|
||||
buckets = collections.defaultdict(list)
|
||||
for i, obj in enumerate(objects):
|
||||
buckets[(obj['role'], tuple(obj['shape']), obj['scale'], obj['dtype'])].append(i)
|
||||
candidates = []
|
||||
eligible_pairs = 0
|
||||
for key, ids in sorted(buckets.items(), key=lambda item: str(item[0])):
|
||||
eligible_pairs += len(ids) * (len(ids) - 1) // 2
|
||||
if len(ids) < 2:
|
||||
continue
|
||||
x = np.stack([features[i] for i in ids]).astype(np.float64)
|
||||
norm = np.linalg.norm(x, axis=1)
|
||||
u = x / np.maximum(norm[:, None], 1e-30)
|
||||
# Cap each bucket's reservoir to bound the screening matrix.
|
||||
cos = u @ u.T
|
||||
sq = np.maximum(0., norm[:, None] ** 2 + norm[None, :] ** 2 - 2 * (x @ x.T))
|
||||
err = np.sqrt(sq / np.maximum(norm[:, None] ** 2, 1e-30))
|
||||
np.fill_diagonal(cos, -np.inf)
|
||||
np.fill_diagonal(err, np.inf)
|
||||
seen = set()
|
||||
for row, i in enumerate(ids):
|
||||
near = set(np.argsort(err[row])[:neighbors]) | set(np.argsort(-cos[row])[:neighbors])
|
||||
for col in near:
|
||||
j = ids[col]
|
||||
edge = (min(i, j), max(i, j))
|
||||
if i == j or edge in seen:
|
||||
continue
|
||||
seen.add(edge)
|
||||
candidates.append({'a': edge[0], 'b': edge[1], 'screen_error': float(min(err[row, col], err[col, row])),
|
||||
'screen_cosine': float(cos[row, col])})
|
||||
# Low-error candidates first, then guarantee role/scale diversity round-robin.
|
||||
queues = collections.defaultdict(list)
|
||||
for c in sorted(candidates, key=lambda c: (c['screen_error'], c['a'], c['b'])):
|
||||
o = objects[c['a']]
|
||||
queues[(o['role'], tuple(o['shape']), o['scale'])].append(c)
|
||||
queues = [collections.deque(q) for _, q in sorted(queues.items(), key=lambda x: str(x[0]))]
|
||||
chosen = []
|
||||
while queues and len(chosen) < cap:
|
||||
for q in queues:
|
||||
if q and len(chosen) < cap:
|
||||
chosen.append(q.popleft())
|
||||
queues = [q for q in queues if q]
|
||||
return chosen, {'eligible_pairs_in_screened_objects': eligible_pairs,
|
||||
'candidate_pairs_before_cap': len(candidates), 'selected_pairs': len(chosen)}
|
||||
|
||||
|
||||
def object_record(t, kind='tensor', shape=None, start=None, scale='whole'):
|
||||
shape = tuple(shape or t.shape)
|
||||
return {'id': t.name if kind == 'tensor' else f'{t.name}|{scale}|{start}',
|
||||
'tensor': t.name, 'role': t.role, 'layer': t.layer, 'kind': kind, 'dtype': t.dtype,
|
||||
'shape': shape, 'start': start, 'scale': scale,
|
||||
'bytes': math.prod(shape) * (t.bytes // t.numel)}
|
||||
|
||||
|
||||
def build_blocks(tensors, seed, deadline, reservoir_limit=256):
|
||||
reservoirs, counts, rngs = {}, collections.Counter(), {}
|
||||
totals = collections.defaultdict(lambda: {'blocks': 0, 'bytes': 0, 'edge_bytes': 0})
|
||||
for t in tensors.values():
|
||||
if len(t.shape) == 2:
|
||||
specs = [(f'2d-{s}', (s, s)) for s in (64, 128) if min(t.shape) >= s]
|
||||
elif len(t.shape) == 3:
|
||||
specs = [('3d-axis0-slice', (1, min(64, t.shape[1]), min(64, t.shape[2]))),
|
||||
('3d-volume', (min(4, t.shape[0]), min(32, t.shape[1]), min(32, t.shape[2])))]
|
||||
else:
|
||||
continue
|
||||
for scale, shape in specs:
|
||||
nblocks = math.prod(d // b for d, b in zip(t.shape, shape))
|
||||
totals[scale]['blocks'] += nblocks
|
||||
block_bytes = math.prod(shape) * (t.bytes // t.numel)
|
||||
totals[scale]['bytes'] += nblocks * block_bytes
|
||||
totals[scale]['edge_bytes'] += t.bytes - nblocks * block_bytes
|
||||
for grid in itertools.product(*(range(d // b) for d, b in zip(t.shape, shape))):
|
||||
if time.monotonic() > deadline:
|
||||
raise TimeoutError('Deadline during block enumeration')
|
||||
start = tuple(i * b for i, b in zip(grid, shape))
|
||||
obj = object_record(t, 'block', shape, start, scale)
|
||||
key = (t.role, shape, scale)
|
||||
if key not in reservoirs:
|
||||
reservoirs[key], rngs[key] = [], np.random.default_rng(stable_seed(key, seed))
|
||||
counts[key] += 1
|
||||
reservoir = reservoirs[key]
|
||||
if len(reservoir) < reservoir_limit:
|
||||
reservoir.append(obj)
|
||||
else:
|
||||
j = int(rngs[key].integers(counts[key]))
|
||||
if j < reservoir_limit:
|
||||
reservoir[j] = obj
|
||||
objects = [obj for key in sorted(reservoirs, key=str) for obj in reservoirs[key]]
|
||||
coverage = [{'role': key[0], 'shape': str(key[1]), 'scale': key[2],
|
||||
'total_blocks': counts[key], 'sampled_blocks': len(reservoirs[key])}
|
||||
for key in sorted(reservoirs, key=str)]
|
||||
return objects, coverage, dict(totals)
|
||||
|
||||
|
||||
def structural_probe(a, b, tensors):
|
||||
"""Bounded mode unfoldings of aligned subvolumes; exploratory, never a grouping criterion."""
|
||||
def small(obj):
|
||||
t = tensors[obj['tensor']]
|
||||
shape = obj['shape']
|
||||
starts = obj['start'] or [0] * len(shape)
|
||||
slices = tuple(slice(s, s + min(d, 32)) for s, d in zip(starts, shape))
|
||||
return decode(t.raw.reshape(t.shape)[slices], t.dtype)
|
||||
x, y = small(a), small(b)
|
||||
result = []
|
||||
if x.ndim < 2:
|
||||
return result
|
||||
for axis in range(x.ndim):
|
||||
mx = np.moveaxis(x, axis, 0).reshape(x.shape[axis], -1)
|
||||
my = np.moveaxis(y, axis, 0).reshape(y.shape[axis], -1)
|
||||
if min(mx.shape) < 2:
|
||||
result.append({'axis': axis, 'probe_shape': list(x.shape), 'trivial_mode_skipped': True})
|
||||
continue
|
||||
ux, sx, _ = np.linalg.svd(mx, full_matrices=False)
|
||||
uy, sy, _ = np.linalg.svd(my, full_matrices=False)
|
||||
k = min(4, max(1, min(len(sx), len(sy)) // 2))
|
||||
result.append({'axis': axis, 'probe_shape': list(x.shape),
|
||||
'k': k,
|
||||
'normalized_spectrum_distance': float(np.linalg.norm(sx / max(np.linalg.norm(sx), 1e-30) - sy / max(np.linalg.norm(sy), 1e-30))),
|
||||
'left_subspace_overlap': float(np.linalg.norm(ux[:, :k].T @ uy[:, :k]) ** 2 / k)})
|
||||
return result
|
||||
|
||||
|
||||
def analyze_mode(mode, objects, tensors, root, deadline, cap, seed, baseline_cap=100):
|
||||
print(f'{mode}: sampling {len(objects)} objects', flush=True)
|
||||
features = []
|
||||
for obj in objects:
|
||||
features.append(sample_object(obj, tensors))
|
||||
tensors[obj['tensor']].evict()
|
||||
candidates, coverage = nearest_candidates(objects, features, cap)
|
||||
write_json(root / f'{mode}_objects.json', objects)
|
||||
pairs, nulls, structures = [], [], []
|
||||
rng = np.random.default_rng(seed)
|
||||
random_pairs = []
|
||||
comparable = collections.defaultdict(list)
|
||||
for j, obj in enumerate(objects):
|
||||
comparable[(obj['role'], tuple(obj['shape']), obj['scale'], obj['dtype'])].append(j)
|
||||
for key, ids in sorted(comparable.items(), key=lambda item: str(item[0])):
|
||||
if len(ids) < 2:
|
||||
continue
|
||||
for _ in range(3):
|
||||
a, b = rng.choice(ids, size=2, replace=False).tolist()
|
||||
random_pairs.append({'a': a, 'b': b, 'role': key[0], 'sampled': True,
|
||||
**metrics(features[a], features[b])})
|
||||
write_json(root / f'{mode}_random_pairs.json', random_pairs)
|
||||
timeout = False
|
||||
structural_counts = collections.Counter()
|
||||
for i, candidate in enumerate(candidates):
|
||||
if time.monotonic() > deadline:
|
||||
timeout = True
|
||||
break
|
||||
a, b = candidate['a'], candidate['b']
|
||||
try:
|
||||
measured = exact_pair(objects[a], objects[b], tensors, deadline)
|
||||
except TimeoutError:
|
||||
timeout = True
|
||||
break
|
||||
row = {**candidate, **measured,
|
||||
'a_id': objects[a]['id'], 'b_id': objects[b]['id'],
|
||||
'role': objects[a]['role'], 'scale_name': objects[a]['scale'],
|
||||
'layer_distance': abs(objects[a]['layer'] - objects[b]['layer'])}
|
||||
row['screen_vs_exact_min_error_absolute_difference'] = abs(candidate['screen_error'] - min(measured['error_member_to_rep'], measured['error_rep_to_member']))
|
||||
# Tail diagnostics are sampled, explicitly labelled, not claimed exact.
|
||||
fx, fy = features[a].astype(np.float64), features[b].astype(np.float64)
|
||||
row['sampled_error_q99_over_member_rms'] = float(np.quantile(np.abs(fx - fy), .99) / max(np.sqrt(np.mean(fx * fx)), 1e-30))
|
||||
pairs.append(row)
|
||||
if i < baseline_cap:
|
||||
for method, y in [('shuffled', rng.permutation(fy)), ('gaussian_norm_matched', rng.normal(size=fy.size))]:
|
||||
if method == 'gaussian_norm_matched':
|
||||
y *= np.linalg.norm(fy) / max(np.linalg.norm(y), 1e-30)
|
||||
nulls.append({'a': a, 'b': b, 'method': method, 'sampled': True, **metrics(fx, y)})
|
||||
ndim = len(objects[a]['shape'])
|
||||
if ndim >= 2 and structural_counts[ndim] < 20:
|
||||
structures.append({'a': a, 'b': b, 'modes': structural_probe(objects[a], objects[b], tensors)})
|
||||
tensors[objects[a]['tensor']].evict()
|
||||
tensors[objects[b]['tensor']].evict()
|
||||
structural_counts[ndim] += 1
|
||||
if (i + 1) % 100 == 0:
|
||||
write_csv(root / f'{mode}_pairs.csv', pairs)
|
||||
print(f'{mode}: verified {i + 1}/{len(candidates)}', flush=True)
|
||||
write_csv(root / f'{mode}_pairs.csv', pairs)
|
||||
write_json(root / f'{mode}_nulls.json', nulls)
|
||||
write_json(root / f'{mode}_structure.json', structures)
|
||||
coverage.update({'objects_screened': len(objects), 'pairs_verified': len(pairs), 'timed_out': timeout,
|
||||
'screen_sample_elements': 2048, 'candidate_neighbors_per_metric': 10})
|
||||
write_json(root / f'{mode}_coverage.json', coverage)
|
||||
return pairs
|
||||
|
||||
|
||||
def summarize(mode, objects, pairs, model_bytes, root, partition_totals=None):
|
||||
rows = []
|
||||
scales = sorted({o['scale'] for o in objects})
|
||||
for scale in scales:
|
||||
ids = [i for i, o in enumerate(objects) if o['scale'] == scale]
|
||||
selected = set(ids)
|
||||
for threshold in THRESHOLDS:
|
||||
groups = sharing_groups(objects, [p for p in pairs if p['a'] in selected], threshold)
|
||||
groups = [g for g in groups if g['representative'] in selected]
|
||||
original = sum(objects[i]['bytes'] for i in ids)
|
||||
rep_bytes = sum(objects[g['representative']]['bytes'] for g in groups)
|
||||
# Concrete theoretical layout: 16 bytes per object mapping, plus 64 bytes per shared group.
|
||||
# Identity case falls back to original checkpoint at zero additional cost.
|
||||
mapping_bytes = 16 * len(ids) + 64 * sum(len(g['members']) > 1 for g in groups)
|
||||
proposed = rep_bytes + mapping_bytes
|
||||
saved = max(0, original - proposed)
|
||||
partition_count = partition_totals[scale]['blocks'] if partition_totals else len(ids)
|
||||
rows.append({'mode': mode, 'scale': scale, 'threshold': threshold,
|
||||
'objects': len(ids), 'groups': len(groups),
|
||||
'total_partition_objects': partition_count,
|
||||
'unexamined_partition_objects': partition_count - len(ids),
|
||||
'conservative_partition_groups': partition_count - len(ids) + len(groups),
|
||||
'shared_groups': sum(len(g['members']) > 1 for g in groups),
|
||||
'max_group_size': max([len(g['members']) for g in groups] or [0]),
|
||||
'verified_shared_bytes': sum(sum(objects[i]['bytes'] for i in g['members']) for g in groups if len(g['members']) > 1),
|
||||
'examined_object_bytes': original, 'representative_bytes': rep_bytes,
|
||||
'mapping_bytes_if_encoded': mapping_bytes, 'net_saved_bytes': saved,
|
||||
'net_saved_model_fraction': saved / model_bytes,
|
||||
'max_replacement_error': max([g['max_error'] for g in groups] or [0.])})
|
||||
write_json(root / f'groups_{mode}_{scale}_{threshold:.2f}.json', groups)
|
||||
return rows
|
||||
|
||||
|
||||
def main():
|
||||
p = argparse.ArgumentParser()
|
||||
p.add_argument('--root', type=pathlib.Path, required=True)
|
||||
p.add_argument('--seconds', type=int, default=7200)
|
||||
p.add_argument('--seed', type=int, default=20261109)
|
||||
args = p.parse_args()
|
||||
started = time.monotonic()
|
||||
deadline = started + args.seconds
|
||||
root = args.root
|
||||
output = root / 'analysis'
|
||||
output.mkdir(exist_ok=True)
|
||||
tensors = load_tensors(root / 'weights')
|
||||
inventory = []
|
||||
for t in tensors.values():
|
||||
if time.monotonic() > deadline:
|
||||
raise TimeoutError('Deadline during inventory')
|
||||
n = 0
|
||||
total = squared = 0.
|
||||
max_abs = 0.
|
||||
for x in t.chunked():
|
||||
if not np.isfinite(x).all():
|
||||
raise ValueError(f'Nonfinite weights: {t.name}')
|
||||
z = x.astype(np.float64)
|
||||
n += z.size
|
||||
total += float(z.sum())
|
||||
squared += float(z @ z)
|
||||
max_abs = max(max_abs, float(np.abs(z).max()))
|
||||
t.evict()
|
||||
inventory.append({'name': t.name, 'shape': str(t.shape), 'ndim': len(t.shape), 'dtype': t.dtype,
|
||||
'numel': t.numel, 'bytes': t.bytes, 'role': t.role, 'layer': t.layer,
|
||||
'mean': total / n, 'rms': math.sqrt(squared / n), 'max_abs': max_abs})
|
||||
t.evict()
|
||||
write_csv(output / 'inventory.csv', inventory)
|
||||
print(f'Inventory: {len(tensors)} tensors, {sum(t.bytes for t in tensors.values())} bytes', flush=True)
|
||||
whole = [object_record(t) for t in tensors.values()]
|
||||
# Stage budgets prevent the whole-tensor stage from consuming block-analysis time.
|
||||
whole_deadline = min(deadline, started + args.seconds * .50)
|
||||
whole_pairs = analyze_mode('tensor', whole, tensors, output, whole_deadline, 2000, args.seed)
|
||||
blocks, block_coverage, block_totals = build_blocks(tensors, args.seed, deadline)
|
||||
write_csv(output / 'block_sampling.csv', block_coverage)
|
||||
write_json(output / 'block_partitions.json', block_totals)
|
||||
block_pairs = analyze_mode('block', blocks, tensors, output, deadline, 20000, args.seed)
|
||||
model_bytes = sum(t.bytes for t in tensors.values())
|
||||
rows = summarize('tensor', whole, whole_pairs, model_bytes, output)
|
||||
rows += summarize('block', blocks, block_pairs, model_bytes, output, block_totals)
|
||||
write_csv(output / 'compression_curve.csv', rows)
|
||||
summary = {'model': json.loads((root / 'manifest.json').read_text())['repo'],
|
||||
'revision': json.loads((root / 'manifest.json').read_text())['revision'],
|
||||
'tensor_count': len(tensors), 'parameter_count': sum(t.numel for t in tensors.values()),
|
||||
'model_weight_bytes': model_bytes, 'ndim_counts': dict(collections.Counter(len(t.shape) for t in tensors.values())),
|
||||
'elapsed_seconds': time.monotonic() - started, 'analysis_budget_seconds': args.seconds,
|
||||
'peak_rss_kib': resource.getrusage(resource.RUSAGE_SELF).ru_maxrss,
|
||||
'python': platform.python_version(), 'numpy': np.__version__, 'seed': args.seed,
|
||||
'limitations': ['No inference or functional equivalence measured', 'No exhaustive pair search',
|
||||
'Block search uses per-role/shape/scale reservoirs of 256',
|
||||
'Different block scales overlap; savings must not be added',
|
||||
'Block savings count only verified disjoint sampled blocks, no extrapolation',
|
||||
'Spectral probes use aligned bounded subvolumes, not whole tensors',
|
||||
'Shared encoding is theoretical, not a runnable compressed checkpoint',
|
||||
'No cross-role grouping, neuron permutation alignment, or training'],
|
||||
'completed': True}
|
||||
write_json(output / 'summary.json', summary)
|
||||
print(json.dumps(summary), flush=True)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,75 @@
|
||||
"""Audit saved numerical data and freeze provenance before archive/cleanup."""
|
||||
import argparse
|
||||
import csv
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import pathlib
|
||||
import platform
|
||||
import re
|
||||
import shutil
|
||||
import time
|
||||
|
||||
|
||||
def main():
|
||||
p = argparse.ArgumentParser()
|
||||
p.add_argument('--root', required=True, type=pathlib.Path)
|
||||
p.add_argument('--analysis-code-commit', help='Commit actually used for the weight analysis; omit if unknown')
|
||||
args = p.parse_args()
|
||||
root = args.root.resolve()
|
||||
if not __debug__:
|
||||
raise RuntimeError('Validation requires Python assertions enabled')
|
||||
checked = {}
|
||||
for mode in ('tensor', 'block'):
|
||||
with (root / 'analysis' / f'{mode}_pairs.csv').open() as f:
|
||||
rows = list(csv.DictReader(f))
|
||||
coverage = json.loads((root / 'analysis' / f'{mode}_coverage.json').read_text())
|
||||
if len(rows) != coverage['pairs_verified']:
|
||||
raise ValueError('Coverage count mismatch')
|
||||
for row in rows:
|
||||
for key in ('cosine', 'pearson'):
|
||||
if row[key] and (not math.isfinite(float(row[key])) or abs(float(row[key])) > 1 + 1e-7):
|
||||
raise ValueError(f'Invalid correlation: {mode}/{key}')
|
||||
for key in ('error_member_to_rep', 'error_rep_to_member', 'symmetric_error', 'scaled_error',
|
||||
'sampled_error_q99_over_member_rms', 'screen_vs_exact_min_error_absolute_difference'):
|
||||
value = float(row[key])
|
||||
if not math.isfinite(value) or value < 0:
|
||||
raise ValueError(f'Invalid error: {mode}/{key}')
|
||||
if not math.isfinite(float(row['scale'])):
|
||||
raise ValueError('Invalid scaling diagnostic')
|
||||
checked[mode] = {'rows': len(rows), 'finite_metrics': True, 'coverage_matches': True}
|
||||
test_log = (root / 'tests.log').read_text()
|
||||
match = re.search(r'Ran (\d+) tests in ([\d.]+)s', test_log)
|
||||
if not match or '\nOK\n' not in test_log:
|
||||
raise ValueError('Final tests did not pass')
|
||||
qc = {'passed': True, 'checked_unix': time.time(), 'numerical_tables': checked,
|
||||
'final_tests': int(match[1]), 'final_test_seconds': float(match[2]),
|
||||
'earlier_parallel_test_timeout_seconds': 300 if (root / 'tests-during-io-timeout.txt').exists() else None}
|
||||
(root / 'numerical-qc.json').write_text(json.dumps(qc, indent=2))
|
||||
for src, dest in [('tests.log', 'test-results.txt'), ('analysis.log', 'analysis-run.txt'),
|
||||
('plot.log', 'plot-run.txt'), ('download-ranged.log', 'download-progress.txt'),
|
||||
('supervisor.log', 'supervisor-run.txt')]:
|
||||
shutil.copy2(root / src, root / dest)
|
||||
mem = {}
|
||||
for line in pathlib.Path('/proc/meminfo').read_text().splitlines():
|
||||
key, value = line.split(':', 1)
|
||||
if key in ('MemTotal', 'MemAvailable', 'SwapTotal', 'SwapFree'):
|
||||
mem[key + '_KiB'] = int(value.split()[0])
|
||||
environment = {'recorded_unix': time.time(), 'python': platform.python_version(),
|
||||
'platform': platform.platform(), 'machine': platform.machine(),
|
||||
'logical_cpus': os.cpu_count(), 'memory_snapshot': mem,
|
||||
'blas_threads': 1, 'assertions_enabled': __debug__,
|
||||
'analysis_io_priority': 'best-effort 7, applied during block verification',
|
||||
'analysis_code_commit': args.analysis_code_commit,
|
||||
'note': 'Shared host; see resource-observation.txt. Memory snapshot is post-analysis.'}
|
||||
(root / 'execution-environment.json').write_text(json.dumps(environment, indent=2))
|
||||
project = pathlib.Path(__file__).resolve().parents[1]
|
||||
files = sorted((project / 'pilot').glob('*.py')) + [project / 'requirements.txt', project / 'tests/test_pilot.py']
|
||||
hashes = [f'{hashlib.sha256(file.read_bytes()).hexdigest()} {file.relative_to(project)}' for file in files]
|
||||
(root / 'code_checksums.txt').write_text('\n'.join(hashes) + '\n')
|
||||
print(json.dumps(qc))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,140 @@
|
||||
"""Download pinned official safetensors, no framework or shared cache."""
|
||||
import argparse
|
||||
import concurrent.futures
|
||||
import hashlib
|
||||
import json
|
||||
import pathlib
|
||||
import os
|
||||
import shutil
|
||||
import time
|
||||
import threading
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
|
||||
|
||||
def fetch_json(url):
|
||||
with urllib.request.urlopen(url, timeout=90) as response:
|
||||
return json.load(response)
|
||||
|
||||
|
||||
def ranged_download(url, part, total, workers=8):
|
||||
"""Resume a sequential prefix; journal completed byte ranges before final hashing."""
|
||||
journal = part.with_suffix(part.suffix + '.ranges.json')
|
||||
if journal.exists():
|
||||
state = json.loads(journal.read_text())
|
||||
if state['total'] != total:
|
||||
raise RuntimeError('Range journal size mismatch')
|
||||
else:
|
||||
state = {'total': total, 'prefix': part.stat().st_size if part.exists() else 0, 'done': []}
|
||||
journal.write_text(json.dumps(state))
|
||||
lock = threading.Lock()
|
||||
fd = os.open(part, os.O_CREAT | os.O_RDWR, 0o600)
|
||||
chunk_size = 64 * 1024 * 1024
|
||||
ranges = [(start, min(start + chunk_size, total) - 1)
|
||||
for start in range(state['prefix'], total, chunk_size)]
|
||||
|
||||
def one(bounds):
|
||||
start, end = bounds
|
||||
if start in state['done']:
|
||||
return
|
||||
for attempt in range(3):
|
||||
try:
|
||||
req = urllib.request.Request(url, headers={'Range': f'bytes={start}-{end}'})
|
||||
with urllib.request.urlopen(req, timeout=180) as response:
|
||||
expected = f'bytes {start}-{end}/{total}'
|
||||
if response.status != 206 or response.headers.get('Content-Range') != expected:
|
||||
raise RuntimeError('Server did not honor exact requested range')
|
||||
offset = start
|
||||
while chunk := response.read(min(4 * 1024 * 1024, end - offset + 1)):
|
||||
written = os.pwrite(fd, chunk, offset)
|
||||
if written != len(chunk):
|
||||
raise RuntimeError('Short positional write')
|
||||
offset += written
|
||||
if offset > end:
|
||||
break
|
||||
if offset != end + 1:
|
||||
raise RuntimeError('Incomplete range')
|
||||
with lock:
|
||||
os.fsync(fd)
|
||||
state['done'].append(start)
|
||||
tmp = journal.with_suffix('.tmp')
|
||||
tmp.write_text(json.dumps(state))
|
||||
tmp.replace(journal)
|
||||
done = state['prefix'] + sum(b - a + 1 for a, b in ranges if a in state['done'])
|
||||
print(f'Weight download {done}/{total} bytes ({100 * done / total:.1f}%)', flush=True)
|
||||
return
|
||||
except Exception:
|
||||
if attempt == 2:
|
||||
raise
|
||||
time.sleep(2 ** attempt)
|
||||
try:
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=workers) as pool:
|
||||
list(pool.map(one, ranges))
|
||||
finally:
|
||||
os.close(fd)
|
||||
journal.unlink()
|
||||
|
||||
|
||||
def main():
|
||||
p = argparse.ArgumentParser()
|
||||
p.add_argument('--root', type=pathlib.Path, required=True)
|
||||
p.add_argument('--repo', default='Qwen/Qwen3.5-2B')
|
||||
args = p.parse_args()
|
||||
root = args.root.resolve()
|
||||
root.mkdir(parents=True, exist_ok=True)
|
||||
weights = root / 'weights'
|
||||
weights.mkdir(exist_ok=True)
|
||||
marker = weights / '.pilot-owned'
|
||||
marker.write_text(args.repo)
|
||||
old_manifest = root / 'manifest.json'
|
||||
pinned = json.loads(old_manifest.read_text())['revision'] if old_manifest.exists() else None
|
||||
endpoint = f'https://huggingface.co/api/models/{args.repo}'
|
||||
if pinned:
|
||||
endpoint += f'/revision/{pinned}'
|
||||
info = fetch_json(endpoint + '?blobs=true')
|
||||
revision = info['sha']
|
||||
files = [f for f in info['siblings'] if f['rfilename'].endswith('.safetensors')
|
||||
or f['rfilename'] in ('config.json', 'model.safetensors.index.json', 'README.md')]
|
||||
total = sum(f.get('size', 0) for f in files)
|
||||
if shutil.disk_usage(root).free < total + 3_000_000_000:
|
||||
raise RuntimeError('Insufficient space for checkpoint plus 3 GB margin')
|
||||
manifest = {'repo': args.repo, 'revision': revision, 'expected_bytes': total,
|
||||
'started_unix': time.time(), 'files': [], 'license': info.get('cardData', {}).get('license')}
|
||||
(root / 'source_metadata.json').write_text(json.dumps(info, indent=2))
|
||||
for entry in files:
|
||||
name = entry['rfilename']
|
||||
if pathlib.PurePosixPath(name).name != name:
|
||||
raise RuntimeError(f'Unexpected nested path: {name}')
|
||||
dest = weights / name
|
||||
part = dest.with_suffix(dest.suffix + '.partial')
|
||||
digest = hashlib.sha256()
|
||||
url = f'https://huggingface.co/{args.repo}/resolve/{revision}/{urllib.parse.quote(name)}?download=true'
|
||||
print(f'Downloading {name}, expected {entry.get("size")} bytes', flush=True)
|
||||
if entry.get('size', 0) > 128 * 1024 * 1024:
|
||||
ranged_download(url, part, entry['size'])
|
||||
with part.open('rb') as stream:
|
||||
while chunk := stream.read(4 * 1024 * 1024):
|
||||
digest.update(chunk)
|
||||
else:
|
||||
with urllib.request.urlopen(url, timeout=180) as response, part.open('wb') as out:
|
||||
while chunk := response.read(4 * 1024 * 1024):
|
||||
out.write(chunk)
|
||||
digest.update(chunk)
|
||||
sha = digest.hexdigest()
|
||||
if entry.get('size') and part.stat().st_size != entry['size']:
|
||||
raise RuntimeError(f'Size mismatch: {name}')
|
||||
expected_sha = entry.get('lfs', {}).get('sha256')
|
||||
if expected_sha and sha != expected_sha:
|
||||
raise RuntimeError(f'SHA256 mismatch: {name}')
|
||||
part.rename(dest)
|
||||
manifest['files'].append({'name': name, 'bytes': dest.stat().st_size,
|
||||
'sha256': sha, 'upstream_sha256': expected_sha})
|
||||
(root / 'manifest.json').write_text(json.dumps(manifest, indent=2))
|
||||
manifest['completed_unix'] = time.time()
|
||||
(root / 'manifest.json').write_text(json.dumps(manifest, indent=2))
|
||||
shutil.copy2(weights / 'config.json', root / 'model_config.json')
|
||||
print(json.dumps({'revision': revision, 'bytes': sum(f['bytes'] for f in manifest['files'])}), flush=True)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
+261
@@ -0,0 +1,261 @@
|
||||
"""Scientific figures and a compact result report from saved data only."""
|
||||
import argparse
|
||||
import csv
|
||||
import json
|
||||
import pathlib
|
||||
|
||||
import matplotlib
|
||||
matplotlib.use('Agg')
|
||||
import matplotlib.pyplot as plt
|
||||
from matplotlib.patches import Patch, Rectangle
|
||||
import numpy as np
|
||||
|
||||
|
||||
def read_csv(path):
|
||||
with path.open() as f:
|
||||
return list(csv.DictReader(f))
|
||||
|
||||
|
||||
def write_csv(path, rows):
|
||||
with path.open('w', newline='') as f:
|
||||
if rows:
|
||||
writer = csv.DictWriter(f, fieldnames=list(rows[0]))
|
||||
writer.writeheader()
|
||||
writer.writerows(rows)
|
||||
|
||||
|
||||
def main():
|
||||
p = argparse.ArgumentParser()
|
||||
p.add_argument('--root', type=pathlib.Path, required=True)
|
||||
args = p.parse_args()
|
||||
root = args.root
|
||||
out = root / 'analysis'
|
||||
figures = root / 'figures'
|
||||
figures.mkdir(exist_ok=True)
|
||||
plt.rcParams.update({'font.size': 10, 'axes.spines.top': False, 'axes.spines.right': False,
|
||||
'pdf.fonttype': 42, 'svg.fonttype': 'none', 'figure.dpi': 120})
|
||||
def save(fig, name):
|
||||
fig.tight_layout()
|
||||
for ext in ('png', 'pdf', 'svg'):
|
||||
fig.savefig(figures / f'{name}.{ext}', dpi=200, bbox_inches='tight')
|
||||
plt.close(fig)
|
||||
|
||||
curves = read_csv(out / 'compression_curve.csv')
|
||||
summary = json.loads((out / 'summary.json').read_text())
|
||||
dimensional = {}
|
||||
for row in read_csv(out / 'inventory.csv'):
|
||||
ndim = int(row['ndim'])
|
||||
item = dimensional.setdefault(ndim, {'ndim': ndim, 'tensor_count': 0, 'weight_bytes': 0})
|
||||
item['tensor_count'] += 1
|
||||
item['weight_bytes'] += int(row['bytes'])
|
||||
dimensional_rows = [dimensional[key] for key in sorted(dimensional)]
|
||||
for item in dimensional_rows:
|
||||
item['model_byte_fraction'] = item['weight_bytes'] / summary['model_weight_bytes']
|
||||
with (out / 'dimensional_bytes.csv').open('w', newline='') as f:
|
||||
writer = csv.DictWriter(f, fieldnames=list(dimensional_rows[0]))
|
||||
writer.writeheader()
|
||||
writer.writerows(dimensional_rows)
|
||||
fig, axes = plt.subplots(1, 2, figsize=(11, 4))
|
||||
for scale in sorted({r['scale'] for r in curves}):
|
||||
rows = [r for r in curves if r['scale'] == scale]
|
||||
x = [100 * float(r['threshold']) for r in rows]
|
||||
axes[0].plot(x, [100 * float(r['net_saved_model_fraction']) for r in rows], 'o-', label=scale)
|
||||
axes[1].plot(x, [int(r['shared_groups']) for r in rows], 'o-', label=scale)
|
||||
axes[0].set(xlabel='Maximum relative weight error (%)', ylabel='Verified net savings / model bytes (%)')
|
||||
axes[1].set(xlabel='Maximum relative weight error (%)', ylabel='Non-singleton sharing groups')
|
||||
axes[0].legend(fontsize=8, loc='upper left')
|
||||
fig.suptitle('Conservative verified sharing; block scales are separate alternatives')
|
||||
save(fig, 'compression_frontier')
|
||||
|
||||
pair_sets = {mode: read_csv(out / f'{mode}_pairs.csv') for mode in ('tensor', 'block')}
|
||||
role_rows = []
|
||||
for mode, pairs in pair_sets.items():
|
||||
mode_objects = json.loads((out / f'{mode}_objects.json').read_text())
|
||||
buckets = {}
|
||||
for row in pairs:
|
||||
obj = mode_objects[int(row['a'])]
|
||||
key = (obj['role'], obj['scale'], tuple(obj['shape']), obj['dtype'])
|
||||
buckets.setdefault(key, []).append(row)
|
||||
for (role_name, scale_name, shape, dtype), rows in sorted(buckets.items(), key=lambda item: str(item[0])):
|
||||
errors = [float(r['error_member_to_rep']) for r in rows]
|
||||
cosines = [float(r['cosine']) for r in rows if r['cosine']]
|
||||
role_rows.append({'mode': mode, 'role': role_name, 'scale': scale_name,
|
||||
'shape': str(shape), 'ndim': len(shape), 'dtype': dtype,
|
||||
'verified_pairs': len(rows), 'minimum_direct_error': min(errors),
|
||||
'median_direct_error': float(np.median(errors)),
|
||||
'maximum_cosine': max(cosines) if cosines else '',
|
||||
'minimum_scaled_error': min(float(r['scaled_error']) for r in rows),
|
||||
'pairs_under_5pct_direct_error': sum(e <= .05 for e in errors),
|
||||
'pairs_under_20pct_direct_error': sum(e <= .20 for e in errors)})
|
||||
with (out / 'role_summary.csv').open('w', newline='') as f:
|
||||
if role_rows:
|
||||
writer = csv.DictWriter(f, fieldnames=list(role_rows[0]))
|
||||
writer.writeheader()
|
||||
writer.writerows(role_rows)
|
||||
fig, axes = plt.subplots(2, 2, figsize=(11, 7))
|
||||
for row, (mode, pairs) in enumerate(pair_sets.items()):
|
||||
for scale in sorted({r['scale_name'] for r in pairs}):
|
||||
group = [r for r in pairs if r['scale_name'] == scale]
|
||||
axes[row, 0].hist([float(r['cosine']) for r in group if r['cosine']], bins=np.linspace(-1, 1, 46), histtype='step', label=scale)
|
||||
errors = [float(r['error_member_to_rep']) for r in group]
|
||||
axes[row, 1].hist(errors, bins=np.linspace(0, max(1e-6, max(errors, default=0.) * 1.01), 46), histtype='step', label=scale)
|
||||
axes[row, 0].set(title=f'{mode}: exact cosine of screened candidates', xlabel='Cosine', ylabel='Candidate pairs')
|
||||
axes[row, 1].set(title=f'{mode}: exact direct replacement error', xlabel='Relative Frobenius error', ylabel='Candidate pairs')
|
||||
axes[row, 0].legend(fontsize=7, loc='upper right')
|
||||
save(fig, 'candidate_distributions')
|
||||
|
||||
pairs = pair_sets['tensor']
|
||||
roles = sorted({r['role'] for r in pairs})
|
||||
# Illustrate one repeated 2D projection; this is not a byte-weighted model summary.
|
||||
objects = json.loads((out / 'tensor_objects.json').read_text())
|
||||
candidates = [role for role in roles if any(o['role'] == role and len(o['shape']) == 2 for o in objects)]
|
||||
role = max(candidates or roles, key=lambda k: sum(r['role'] == k for r in pairs))
|
||||
chosen = [(i, o) for i, o in enumerate(objects) if o['role'] == role]
|
||||
chosen.sort(key=lambda item: item[1]['layer'])
|
||||
indices = {i: j for j, (i, _) in enumerate(chosen)}
|
||||
matrix = np.full((len(chosen), len(chosen)), np.nan)
|
||||
np.fill_diagonal(matrix, 1.)
|
||||
for r in pairs:
|
||||
a, b = int(r['a']), int(r['b'])
|
||||
if a in indices and b in indices and r['cosine']:
|
||||
matrix[indices[a], indices[b]] = matrix[indices[b], indices[a]] = float(r['cosine'])
|
||||
fig, ax = plt.subplots(figsize=(7, 6))
|
||||
cmap = plt.get_cmap('coolwarm').with_extremes(bad='white')
|
||||
im = ax.imshow(matrix, cmap=cmap, vmin=-1, vmax=1)
|
||||
for y, x in zip(*np.where(np.isnan(matrix))):
|
||||
ax.add_patch(Rectangle((x - .5, y - .5), 1, 1, facecolor='none',
|
||||
edgecolor='#999999', hatch='///', linewidth=0))
|
||||
labels = [o['layer'] for _, o in chosen]
|
||||
ticks = np.arange(0, len(labels), max(1, len(labels) // 10))
|
||||
ax.set_xticks(ticks, [labels[i] for i in ticks])
|
||||
ax.set_yticks(ticks, [labels[i] for i in ticks])
|
||||
ax.set(xlabel='Layer', ylabel='Layer', title=role.removeprefix('model.') + '\nHatching = unverified; diagonal is identity')
|
||||
ax.legend(handles=[Patch(facecolor='white', edgecolor='#999999', hatch='///', label='Unverified pair')],
|
||||
loc='upper center', bbox_to_anchor=(.5, -.12), fontsize=8, frameon=False)
|
||||
fig.colorbar(im, ax=ax, label='Exact cosine')
|
||||
save(fig, 'layer_heatmap')
|
||||
heatmap_rows = [{'a_layer': oa['layer'], 'b_layer': ob['layer'], 'role': role,
|
||||
'cosine': float(matrix[i, j]) if not np.isnan(matrix[i, j]) else '',
|
||||
'verified': bool(i != j and not np.isnan(matrix[i, j])), 'identity': i == j}
|
||||
for i, (_, oa) in enumerate(chosen) for j, (_, ob) in enumerate(chosen)]
|
||||
write_csv(out / 'heatmap_pairs.csv', heatmap_rows)
|
||||
|
||||
distance_rows = []
|
||||
fig, axes = plt.subplots(1, 2, figsize=(12, 4), sharey=True)
|
||||
for ax, mode in zip(axes, ('tensor', 'block')):
|
||||
mode_objects = json.loads((out / f'{mode}_objects.json').read_text())
|
||||
buckets = {}
|
||||
for row in pair_sets[mode]:
|
||||
if not row['cosine']:
|
||||
continue
|
||||
obj = mode_objects[int(row['a'])]
|
||||
stratum = f"{len(obj['shape'])}D" if mode == 'tensor' else obj['scale']
|
||||
buckets.setdefault(stratum, {}).setdefault(int(row['layer_distance']), []).append(float(row['cosine']))
|
||||
for stratum, by_distance in sorted(buckets.items()):
|
||||
distances = sorted(by_distance)
|
||||
medians = []
|
||||
for distance in distances:
|
||||
values = by_distance[distance]
|
||||
median = float(np.median(values))
|
||||
medians.append(median)
|
||||
distance_rows.append({'mode': mode, 'stratum': stratum, 'layer_distance': distance,
|
||||
'pairs': len(values), 'median_cosine': median,
|
||||
'q25': float(np.quantile(values, .25)), 'q75': float(np.quantile(values, .75))})
|
||||
ax.plot(distances, medians, 'o-', markersize=3, label=stratum)
|
||||
ax.set(xlabel='Absolute layer distance', ylabel='Median exact cosine',
|
||||
title=f'{mode}: dimension / scale strata', ylim=(-.1, 1.05))
|
||||
ax.legend(fontsize=8, loc='upper center', bbox_to_anchor=(.5, -.23), ncol=2, frameon=False)
|
||||
fig.suptitle('Screened candidates: module mixture and selection bias still apply')
|
||||
write_csv(out / 'layer_distance_stratified.csv', distance_rows)
|
||||
save(fig, 'layer_distance')
|
||||
|
||||
null_rows = []
|
||||
fig, axes = plt.subplots(3, 2, figsize=(12, 10))
|
||||
bins = np.linspace(-1, 1, 26)
|
||||
for col, mode in enumerate(('tensor', 'block')):
|
||||
mode_objects = json.loads((out / f'{mode}_objects.json').read_text())
|
||||
nulls = json.loads((out / f'{mode}_nulls.json').read_text())
|
||||
real_random = json.loads((out / f'{mode}_random_pairs.json').read_text())
|
||||
cohorts = [('shuffled', [r for r in nulls if r['method'] == 'shuffled']),
|
||||
('gaussian_norm_matched', [r for r in nulls if r['method'] == 'gaussian_norm_matched']),
|
||||
('random_actual_same_role', real_random)]
|
||||
for row, ndim in enumerate((1, 2, 3)):
|
||||
ax = axes[row, col]
|
||||
any_values = False
|
||||
for method, records in cohorts:
|
||||
values = np.asarray([r['cosine'] for r in records if r['cosine'] is not None
|
||||
and len(mode_objects[r['a']]['shape']) == ndim])
|
||||
if not len(values):
|
||||
continue
|
||||
any_values = True
|
||||
values = np.clip(values, -1, 1)
|
||||
ax.hist(values, bins=bins, density=True, histtype='step', label=f'{method} (n={len(values)})')
|
||||
counts, _ = np.histogram(values, bins=bins)
|
||||
for i, count in enumerate(counts):
|
||||
null_rows.append({'mode': mode, 'ndim': ndim, 'method': method, 'samples': len(values),
|
||||
'bin_left': float(bins[i]), 'bin_right': float(bins[i + 1]),
|
||||
'count': int(count), 'density': float(count / (len(values) * (bins[i + 1] - bins[i])))})
|
||||
ax.set(title=f'{mode}: {ndim}D sampled controls', xlabel='Sampled cosine', ylabel='Density')
|
||||
if any_values:
|
||||
ax.legend(fontsize=7, loc='upper right')
|
||||
else:
|
||||
ax.text(.5, .5, 'No eligible samples', transform=ax.transAxes, ha='center')
|
||||
fig.suptitle('Dimension-stratified controls: density normalized; role mixtures may differ')
|
||||
write_csv(out / 'null_histograms.csv', null_rows)
|
||||
save(fig, 'null_controls')
|
||||
|
||||
fig, axes = plt.subplots(1, 2, figsize=(11, 4))
|
||||
structure = json.loads((out / 'block_structure.json').read_text())
|
||||
for ax, ndim in zip(axes, (2, 3)):
|
||||
probes = [m for pair in structure for m in pair['modes'] if len(m['probe_shape']) == ndim and not m.get('trivial_mode_skipped')]
|
||||
for axis in sorted({m['axis'] for m in probes}):
|
||||
selected = [m for m in probes if m['axis'] == axis]
|
||||
ax.scatter([m['normalized_spectrum_distance'] for m in selected],
|
||||
[m['left_subspace_overlap'] for m in selected], s=18, alpha=.65, label=f'mode {axis}')
|
||||
ax.set(title=f'{ndim}D blocks: bounded mode-unfolding probes', xlabel='Normalized spectrum distance', ylabel='Leading left subspace overlap (k <= 4)')
|
||||
if probes:
|
||||
ax.legend(fontsize=8, loc='upper right')
|
||||
else:
|
||||
ax.text(.5, .5, 'No eligible probes', transform=ax.transAxes, ha='center')
|
||||
save(fig, 'block_structure')
|
||||
|
||||
parts = json.loads((out / 'block_partitions.json').read_text())
|
||||
lines = ['# Qwen3.5-2B parameter similarity pilot', '',
|
||||
f"- Revision: `{summary['revision']}`", f"- Stored parameters: {summary['parameter_count']:,}",
|
||||
f"- Weight bytes: {summary['model_weight_bytes']:,}",
|
||||
f"- Tensor dimensionalities: {summary['ndim_counts']}",
|
||||
f"- Analysis and grouping/export wall time: {summary['elapsed_seconds']:.1f} seconds",
|
||||
f"- Pair-comparison cutoff: {summary['analysis_budget_seconds']} seconds; final grouping/export is additional and remains under the supervisor hard limit.",
|
||||
f"- Peak analysis RSS: {summary['peak_rss_kib'] / 1024:.1f} MiB", '',
|
||||
'## Verified sharing estimates', '',
|
||||
'| Representation | Weight-error limit | Verified sharing groups | Conservative total groups* | Largest group | Net model savings |',
|
||||
'|---|---:|---:|---:|---:|---:|---:|']
|
||||
for r in curves:
|
||||
lines.append(f"| {r['scale']} | {float(r['threshold']):.0%} | {r['shared_groups']} | {r['conservative_partition_groups']} | {r['max_group_size']} | {float(r['net_saved_model_fraction']):.6%} |")
|
||||
lines += ['', '*Conservative total groups counts unexamined full blocks as singletons; boundary regions and ineligible tensors stay unchanged and are outside that block count.',
|
||||
'Savings count only verified replacement edges. Unsampled blocks remain unchanged. Different block scales must not be summed.',
|
||||
'Theoretical encoding: one stored representative per group, 16 bytes per object mapping and 64 bytes per shared-group descriptor; identity fallback if larger.',
|
||||
'Exact grouping uses direct replacement only. Scaled and spectral results are diagnostic.', '',
|
||||
'## Stratified diagnostics', '',
|
||||
'See `analysis/role_summary.csv` for same-role, same-shape statistics, separated by tensor dimensionality and block scale.',
|
||||
'See `analysis/dimensional_bytes.csv` for tensor counts versus weight-byte contributions by dimensionality.',
|
||||
'Heatmap missing pairs are hatched (`heatmap_pairs.csv`); identity entries are not empirical matches.',
|
||||
'Layer-distance medians are split by dimension/scale (`layer_distance_stratified.csv`); role and selection bias remain.',
|
||||
'Controls are split by dimension and density-normalized, with sample counts (`null_histograms.csv`); these are not significance tests.',
|
||||
'These are screened-candidate statistics, not estimates of all possible pairs. Small normalization vectors must not be conflated with large matrices.', '',
|
||||
'## Coverage', '']
|
||||
for mode in ('tensor', 'block'):
|
||||
coverage = json.loads((out / f'{mode}_coverage.json').read_text())
|
||||
lines.append(f'- {mode}: `{json.dumps(coverage)}`')
|
||||
lines += [f'- Complete-block partition totals (per alternative): `{json.dumps(parts)}`', '',
|
||||
'## Interpretation limits', '', *['- ' + s for s in summary['limitations']],
|
||||
'- Candidate histograms are selection-biased and do not represent the distribution of all tensor pairs.',
|
||||
'- 1D normalization weights may cluster strongly but carry few bytes; do not infer material compression from group counts.',
|
||||
'- No task accuracy, perplexity, inference speed or runtime-memory improvement has been measured.',
|
||||
'- Stored parameter counts may omit tied aliases; no duplicate-weight equality search is performed.', '',
|
||||
'## Figures', '', *[f'' for name in ['compression_frontier', 'candidate_distributions', 'layer_heatmap', 'layer_distance', 'null_controls', 'block_structure']]]
|
||||
(root / 'REPORT.md').write_text('\n'.join(lines) + '\n')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,58 @@
|
||||
"""Supervise the one-model job; no automatic download of subsequent models."""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import pathlib
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
|
||||
def main():
|
||||
p = argparse.ArgumentParser()
|
||||
p.add_argument('--root', type=pathlib.Path, required=True)
|
||||
args = p.parse_args()
|
||||
root = args.root.resolve()
|
||||
root.mkdir(exist_ok=True, parents=True)
|
||||
status_file = root / 'status.json'
|
||||
start = time.time()
|
||||
env = dict(os.environ, OPENBLAS_NUM_THREADS='1', OMP_NUM_THREADS='1')
|
||||
|
||||
def status(phase, **extra):
|
||||
status_file.write_text(json.dumps({'phase': phase, 'updated_unix': time.time(),
|
||||
'supervisor_started_unix': start, **extra}, indent=2))
|
||||
print(phase, flush=True)
|
||||
|
||||
def command(module, log, timeout, extra=()):
|
||||
with (root / log).open('w') as stream:
|
||||
subprocess.run([sys.executable, '-B', '-m', module, '--root', str(root), *extra],
|
||||
check=True, timeout=timeout, env=env, stdout=stream, stderr=subprocess.STDOUT)
|
||||
|
||||
try:
|
||||
status('waiting_for_download')
|
||||
download_wait_start = time.monotonic()
|
||||
while True:
|
||||
manifest_path = root / 'manifest.json'
|
||||
if manifest_path.exists():
|
||||
manifest = json.loads(manifest_path.read_text())
|
||||
if manifest.get('completed_unix') and all((root / 'weights' / f['name']).exists() for f in manifest['files']):
|
||||
break
|
||||
if time.monotonic() - download_wait_start > 7200:
|
||||
raise TimeoutError('Download wait exceeded 2 hours; no analysis started')
|
||||
time.sleep(10)
|
||||
status('analyzing', analysis_hard_limit_seconds=7200)
|
||||
command('pilot.analyze', 'analysis.log', 7200, ('--seconds', '6600'))
|
||||
status('plotting')
|
||||
command('pilot.plot', 'plot.log', 300)
|
||||
status('validating')
|
||||
command('pilot.validate', 'validate.log', 120)
|
||||
with (root / 'environment.txt').open('w') as f:
|
||||
subprocess.run([sys.executable, '-B', '-m', 'pip', 'freeze'], check=True, stdout=f, env=env)
|
||||
status('results_ready_weights_retained_until_archive_copy')
|
||||
except Exception as e:
|
||||
status('failed', error_type=type(e).__name__, error=str(e))
|
||||
raise
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,82 @@
|
||||
"""Read safetensors BF16 directly; never instantiate a model or use pickle."""
|
||||
import json
|
||||
import math
|
||||
import mmap
|
||||
import re
|
||||
import struct
|
||||
|
||||
import numpy as np
|
||||
|
||||
DTYPES = {'BF16': ('<u2', 2), 'F16': ('<f2', 2), 'F32': ('<f4', 4), 'F64': ('<f8', 8)}
|
||||
|
||||
|
||||
def decode(raw, dtype):
|
||||
if dtype == 'BF16':
|
||||
return (np.asarray(raw).astype(np.uint32) << 16).view(np.float32)
|
||||
return np.asarray(raw, dtype=np.float32)
|
||||
|
||||
|
||||
class Tensor:
|
||||
def __init__(self, file, base, name, meta):
|
||||
self.name, self.shape, self.dtype = name, tuple(meta['shape']), meta['dtype']
|
||||
self.numel = math.prod(self.shape)
|
||||
self.bytes = meta['data_offsets'][1] - meta['data_offsets'][0]
|
||||
code, itemsize = DTYPES[self.dtype]
|
||||
if self.bytes != self.numel * itemsize:
|
||||
raise ValueError(f'Invalid byte length: {name}')
|
||||
self.raw = np.memmap(file, mode='r', dtype=code,
|
||||
offset=base + meta['data_offsets'][0], shape=(self.numel,))
|
||||
self.role = re.sub(r'\.(?:layers|blocks)\.\d+', '.layers.*', name)
|
||||
self.role = re.sub(r'\.experts\.\d+', '.experts.*', self.role)
|
||||
m = re.search(r'\.(?:layers|blocks)\.(\d+)', name)
|
||||
self.layer = int(m[1]) if m else -1
|
||||
|
||||
def read(self, selector=slice(None)):
|
||||
return decode(self.raw[selector], self.dtype)
|
||||
|
||||
def chunked(self, size=262144):
|
||||
for start in range(0, self.numel, size):
|
||||
yield self.read(slice(start, min(start + size, self.numel)))
|
||||
|
||||
def evict(self):
|
||||
if hasattr(self.raw._mmap, 'madvise'):
|
||||
self.raw._mmap.madvise(mmap.MADV_DONTNEED)
|
||||
|
||||
|
||||
def load_tensors(directory):
|
||||
tensors = {}
|
||||
for file in sorted(directory.glob('*.safetensors')):
|
||||
with file.open('rb') as f:
|
||||
size = struct.unpack('<Q', f.read(8))[0]
|
||||
if size > 100_000_000:
|
||||
raise ValueError('Oversized safetensors header')
|
||||
header = json.loads(f.read(size))
|
||||
for name, meta in header.items():
|
||||
if name == '__metadata__':
|
||||
continue
|
||||
if name in tensors:
|
||||
raise ValueError(f'Duplicate tensor name: {name}')
|
||||
tensors[name] = Tensor(file, 8 + size, name, meta)
|
||||
if not tensors:
|
||||
raise ValueError('No safetensors found')
|
||||
return tensors
|
||||
|
||||
|
||||
def metrics(x, y):
|
||||
"""x is member; y is representative. Float64 accumulation."""
|
||||
x, y = x.astype(np.float64), y.astype(np.float64)
|
||||
xx, yy, xy = float(x @ x), float(y @ y), float(x @ y)
|
||||
delta = float((x - y) @ (x - y))
|
||||
return finish(xx, yy, xy, delta, float(x.sum()), float(y.sum()), x.size)
|
||||
|
||||
|
||||
def finish(xx, yy, xy, delta, sx, sy, n):
|
||||
tiny = 1e-30
|
||||
vx, vy = max(0., xx - sx * sx / n), max(0., yy - sy * sy / n)
|
||||
return {'cosine': xy / np.sqrt(xx * yy) if xx * yy > tiny else None,
|
||||
'pearson': (xy - sx * sy / n) / np.sqrt(vx * vy) if vx * vy > tiny else None,
|
||||
'error_member_to_rep': np.sqrt(delta / max(xx, tiny)),
|
||||
'error_rep_to_member': np.sqrt(delta / max(yy, tiny)),
|
||||
'symmetric_error': np.sqrt(2 * delta / max(xx + yy, tiny)),
|
||||
'scaled_error': np.sqrt(max(0., xx - xy * xy / max(yy, tiny)) / max(xx, tiny)),
|
||||
'scale': xy / max(yy, tiny)}
|
||||
@@ -0,0 +1,85 @@
|
||||
"""Validate archived analysis before allowing deletion of downloaded weights."""
|
||||
import argparse
|
||||
import csv
|
||||
import hashlib
|
||||
import json
|
||||
import pathlib
|
||||
import shutil
|
||||
import time
|
||||
|
||||
|
||||
def validate(root):
|
||||
analysis = root / 'analysis'
|
||||
summary = json.loads((analysis / 'summary.json').read_text())
|
||||
assert summary['completed']
|
||||
manifest = json.loads((root / 'manifest.json').read_text())
|
||||
assert summary['revision'] == manifest['revision']
|
||||
for mode in ('tensor', 'block'):
|
||||
objects = json.loads((analysis / f'{mode}_objects.json').read_text())
|
||||
with (analysis / f'{mode}_pairs.csv').open() as f:
|
||||
pairs = list(csv.DictReader(f))
|
||||
coverage = json.loads((analysis / f'{mode}_coverage.json').read_text())
|
||||
assert len(pairs) == coverage['pairs_verified']
|
||||
edges = {}
|
||||
for p in pairs:
|
||||
a, b = int(p['a']), int(p['b'])
|
||||
assert objects[a]['shape'] == objects[b]['shape']
|
||||
assert objects[a]['role'] == objects[b]['role']
|
||||
edges[(b, a)] = float(p['error_member_to_rep'])
|
||||
edges[(a, b)] = float(p['error_rep_to_member'])
|
||||
for file in analysis.glob(f'groups_{mode}_*.json'):
|
||||
threshold = float(file.stem.rsplit('_', 1)[1])
|
||||
groups = json.loads(file.read_text())
|
||||
members = [i for g in groups for i in g['members']]
|
||||
assert len(set(members)) == len(members), file
|
||||
for g in groups:
|
||||
assert g['representative'] in g['members']
|
||||
for member in g['members']:
|
||||
if member != g['representative']:
|
||||
assert edges[(g['representative'], member)] <= threshold + 1e-12
|
||||
assert g['max_error'] <= threshold + 1e-12
|
||||
assert (root / 'REPORT.md').is_file()
|
||||
images = list((root / 'figures').glob('*.png'))
|
||||
assert len(images) >= 5
|
||||
for file in images:
|
||||
assert file.read_bytes()[:8] == b'\x89PNG\r\n\x1a\n'
|
||||
checksums = {}
|
||||
for file in sorted(root.rglob('*')):
|
||||
if (file.is_file() and 'weights' not in file.relative_to(root).parts
|
||||
and file.name not in ('checksums.json', 'validation.json', 'cleanup.json', 'status.json')
|
||||
and not file.name.endswith('.log')):
|
||||
checksums[str(file.relative_to(root))] = hashlib.sha256(file.read_bytes()).hexdigest()
|
||||
(root / 'checksums.json').write_text(json.dumps(checksums, indent=2))
|
||||
result = {'validated_unix': time.time(), 'files_checksummed': len(checksums),
|
||||
'group_edges_checked': True, 'passed': True}
|
||||
(root / 'validation.json').write_text(json.dumps(result, indent=2))
|
||||
return result
|
||||
|
||||
|
||||
def main():
|
||||
p = argparse.ArgumentParser()
|
||||
p.add_argument('--root', required=True, type=pathlib.Path)
|
||||
p.add_argument('--cleanup-weights', action='store_true')
|
||||
args = p.parse_args()
|
||||
root = args.root.resolve()
|
||||
result = validate(root)
|
||||
if args.cleanup_weights:
|
||||
weights = root / 'weights'
|
||||
manifest = json.loads((root / 'manifest.json').read_text())
|
||||
assert not weights.is_symlink()
|
||||
assert (weights / '.pilot-owned').read_text() == manifest['repo']
|
||||
expected_names = {entry['name'] for entry in manifest['files']} | {'.pilot-owned'}
|
||||
assert {file.name for file in weights.iterdir()} == expected_names
|
||||
assert all(file.is_file() and not file.is_symlink() for file in weights.iterdir())
|
||||
deleted_bytes = sum(file.stat().st_size for file in weights.iterdir())
|
||||
shutil.rmtree(weights)
|
||||
(root / 'cleanup.json').write_text(json.dumps({'deleted_unix': time.time(),
|
||||
'deleted_bytes': deleted_bytes,
|
||||
'deleted_directory': 'weights',
|
||||
'reason': 'User-authorized delete-after-analysis; archived results validated'}, indent=2))
|
||||
result['weights_removed'] = not weights.exists()
|
||||
print(json.dumps(result))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,5 @@
|
||||
# Existing requirements.txt supplies NumPy and plotting dependencies.
|
||||
# Install binary wheels only. Torch is a reference-check dependency, not the streaming runtime.
|
||||
tokenizers==0.23.3
|
||||
# Install separately with --index-url https://download.pytorch.org/whl/cpu:
|
||||
# torch==2.15.0+cpu
|
||||
@@ -0,0 +1,2 @@
|
||||
numpy==2.5.4
|
||||
matplotlib==3.11.2
|
||||
@@ -0,0 +1,25 @@
|
||||
# 实测激活率:SiLU 门控模型
|
||||
|
||||
覆盖24层、4条中英文文本,每条32 token;所有投影输出及 MLP 门控中间量,按64/128输出通道分块。
|
||||
|
||||
| 输出 | 精确零比例均值 | 0.01 RMS以下均值 | 0.1 RMS以下均值 | 32 token内曾活跃通道比例均值 | 90%能量所需通道比例均值 |
|
||||
|---|---:|---:|---:|---:|---:|
|
||||
| mlp.gate_proj.output | 0.0000% | 0.7811% | 7.8005% | 100.0000% | 63.3063% |
|
||||
| mlp.silu_gate | 0.0000% | 0.5031% | 5.1789% | 100.0000% | 70.1228% |
|
||||
| mlp.gated_product | 0.0000% | 3.1636% | 21.0460% | 100.0000% | 54.8421% |
|
||||
| self_attn.q_proj.output | 0.0000% | 1.2760% | 12.4831% | 100.0000% | 44.0277% |
|
||||
| self_attn.k_proj.output | 0.0000% | 1.2245% | 12.0834% | 100.0000% | 53.4587% |
|
||||
| self_attn.v_proj.output | 0.0000% | 1.3741% | 13.4814% | 100.0000% | 59.9447% |
|
||||
|
||||
128通道块×文本观测共27600条,整个32 token窗口全为精确零的块观测数为0。
|
||||
|
||||
表中是整模块逐文本统计再取均值;每个分块的独立统计见 metrics.csv。阈值是各自RMS的比例;阈值下激活率=1−近零比例。能量集中不等于可删除或可替换。
|
||||
|
||||
仅测该模型、四条短文本;不推断所有语言/长上下文。SiLU不把负值整体截零,负数不等于静默。未执行删块/跳算的质量验证。权重和既有缓存全部保留。
|
||||
|
||||
CPU测量耗时268.6秒,峰值RSS151.6 MiB。
|
||||
|
||||
mlp.gated_product: 128通道块在0.1 RMS阈值上的激活率范围 10.47%–89.04%。
|
||||
self_attn.q_proj.output: 128通道块在0.1 RMS阈值上的激活率范围 75.49%–100.00%。
|
||||
self_attn.k_proj.output: 128通道块在0.1 RMS阈值上的激活率范围 81.20%–93.95%。
|
||||
self_attn.v_proj.output: 128通道块在0.1 RMS阈值上的激活率范围 76.66%–91.89%。
|
||||
@@ -0,0 +1,200 @@
|
||||
{
|
||||
"mlp.gate_proj.output": {
|
||||
"observations": 96,
|
||||
"zero_rate": {
|
||||
"min": 0.0,
|
||||
"mean": 0.0,
|
||||
"max": 0.0
|
||||
},
|
||||
"negative_rate": {
|
||||
"min": 0.6825663248697916,
|
||||
"mean": 0.8063652250501846,
|
||||
"max": 0.9117279052734375
|
||||
},
|
||||
"near_001_rms": {
|
||||
"min": 0.005289713541666667,
|
||||
"mean": 0.007811493343777125,
|
||||
"max": 0.010442097981770834
|
||||
},
|
||||
"near_01_rms": {
|
||||
"min": 0.053070068359375,
|
||||
"mean": 0.07800536685519749,
|
||||
"max": 0.10291544596354167
|
||||
},
|
||||
"ever_active_001": {
|
||||
"min": 1.0,
|
||||
"mean": 1.0,
|
||||
"max": 1.0
|
||||
},
|
||||
"energy90_channel_fraction": {
|
||||
"min": 0.5314127604166666,
|
||||
"mean": 0.6330634223090278,
|
||||
"max": 0.78955078125
|
||||
}
|
||||
},
|
||||
"mlp.silu_gate": {
|
||||
"observations": 96,
|
||||
"zero_rate": {
|
||||
"min": 0.0,
|
||||
"mean": 0.0,
|
||||
"max": 0.0
|
||||
},
|
||||
"negative_rate": {
|
||||
"min": 0.6825663248697916,
|
||||
"mean": 0.8063652250501846,
|
||||
"max": 0.9117279052734375
|
||||
},
|
||||
"near_001_rms": {
|
||||
"min": 0.0018768310546875,
|
||||
"mean": 0.0050310028923882385,
|
||||
"max": 0.0082244873046875
|
||||
},
|
||||
"near_01_rms": {
|
||||
"min": 0.020375569661458332,
|
||||
"mean": 0.05178917778862848,
|
||||
"max": 0.09897359212239583
|
||||
},
|
||||
"ever_active_001": {
|
||||
"min": 1.0,
|
||||
"mean": 1.0,
|
||||
"max": 1.0
|
||||
},
|
||||
"energy90_channel_fraction": {
|
||||
"min": 0.04736328125,
|
||||
"mean": 0.7012278238932291,
|
||||
"max": 0.8186848958333334
|
||||
}
|
||||
},
|
||||
"mlp.gated_product": {
|
||||
"observations": 96,
|
||||
"zero_rate": {
|
||||
"min": 0.0,
|
||||
"mean": 0.0,
|
||||
"max": 0.0
|
||||
},
|
||||
"negative_rate": {
|
||||
"min": 0.4942118326822917,
|
||||
"mean": 0.5002292527092828,
|
||||
"max": 0.5078125
|
||||
},
|
||||
"near_001_rms": {
|
||||
"min": 0.020960489908854168,
|
||||
"mean": 0.03163560231526693,
|
||||
"max": 0.07338460286458333
|
||||
},
|
||||
"near_01_rms": {
|
||||
"min": 0.16080220540364584,
|
||||
"mean": 0.21046002705891928,
|
||||
"max": 0.4075215657552083
|
||||
},
|
||||
"ever_active_001": {
|
||||
"min": 1.0,
|
||||
"mean": 1.0,
|
||||
"max": 1.0
|
||||
},
|
||||
"energy90_channel_fraction": {
|
||||
"min": 0.06380208333333333,
|
||||
"mean": 0.5484212239583334,
|
||||
"max": 0.7263997395833334
|
||||
}
|
||||
},
|
||||
"self_attn.q_proj.output": {
|
||||
"observations": 24,
|
||||
"zero_rate": {
|
||||
"min": 0.0,
|
||||
"mean": 0.0,
|
||||
"max": 0.0
|
||||
},
|
||||
"negative_rate": {
|
||||
"min": 0.6834945678710938,
|
||||
"mean": 0.7360480626424154,
|
||||
"max": 0.7523574829101562
|
||||
},
|
||||
"near_001_rms": {
|
||||
"min": 0.01090240478515625,
|
||||
"mean": 0.012759526570638021,
|
||||
"max": 0.01556396484375
|
||||
},
|
||||
"near_01_rms": {
|
||||
"min": 0.106048583984375,
|
||||
"mean": 0.1248311996459961,
|
||||
"max": 0.148590087890625
|
||||
},
|
||||
"ever_active_001": {
|
||||
"min": 1.0,
|
||||
"mean": 1.0,
|
||||
"max": 1.0
|
||||
},
|
||||
"energy90_channel_fraction": {
|
||||
"min": 0.413330078125,
|
||||
"mean": 0.440277099609375,
|
||||
"max": 0.4931640625
|
||||
}
|
||||
},
|
||||
"self_attn.k_proj.output": {
|
||||
"observations": 24,
|
||||
"zero_rate": {
|
||||
"min": 0.0,
|
||||
"mean": 0.0,
|
||||
"max": 0.0
|
||||
},
|
||||
"negative_rate": {
|
||||
"min": 0.47601318359375,
|
||||
"mean": 0.49783579508463544,
|
||||
"max": 0.52325439453125
|
||||
},
|
||||
"near_001_rms": {
|
||||
"min": 0.00909423828125,
|
||||
"mean": 0.01224517822265625,
|
||||
"max": 0.01629638671875
|
||||
},
|
||||
"near_01_rms": {
|
||||
"min": 0.095458984375,
|
||||
"mean": 0.1208343505859375,
|
||||
"max": 0.15106201171875
|
||||
},
|
||||
"ever_active_001": {
|
||||
"min": 1.0,
|
||||
"mean": 1.0,
|
||||
"max": 1.0
|
||||
},
|
||||
"energy90_channel_fraction": {
|
||||
"min": 0.41015625,
|
||||
"mean": 0.5345865885416666,
|
||||
"max": 0.65234375
|
||||
}
|
||||
},
|
||||
"self_attn.v_proj.output": {
|
||||
"observations": 24,
|
||||
"zero_rate": {
|
||||
"min": 0.0,
|
||||
"mean": 0.0,
|
||||
"max": 0.0
|
||||
},
|
||||
"negative_rate": {
|
||||
"min": 0.49267578125,
|
||||
"mean": 0.5059967041015625,
|
||||
"max": 0.51666259765625
|
||||
},
|
||||
"near_001_rms": {
|
||||
"min": 0.0084228515625,
|
||||
"mean": 0.01374053955078125,
|
||||
"max": 0.02447509765625
|
||||
},
|
||||
"near_01_rms": {
|
||||
"min": 0.087890625,
|
||||
"mean": 0.1348139444986979,
|
||||
"max": 0.22296142578125
|
||||
},
|
||||
"ever_active_001": {
|
||||
"min": 1.0,
|
||||
"mean": 1.0,
|
||||
"max": 1.0
|
||||
},
|
||||
"energy90_channel_fraction": {
|
||||
"min": 0.291015625,
|
||||
"mean": 0.5994466145833334,
|
||||
"max": 0.732421875
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"passed": true,
|
||||
"rows": 83616,
|
||||
"layers": 24,
|
||||
"samples": 4,
|
||||
"all_zero_128_blocks": 0
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
{
|
||||
"REPORT.md": "f7f05bdbf9c9d096fe685ebb49ee15ff9866a99071302f9d7710b99b1a2e7db4",
|
||||
"aggregates.json": "57dfef79b14e73dafd39840661abacd8b44447884e640bb40b86f90ec441ee4d",
|
||||
"audit.json": "2a67fde20f6765d0dbfa783ff31c548f8135af5d74b6fc0e27a3e71c9cbdffb1",
|
||||
"figures/block_activity.pdf": "354cbc89dfc6da6d6611c6effc9dfefc6762e3da1d52bfba89cc11be8c8a9247",
|
||||
"figures/block_activity.png": "ddcfa1f0de30f4f8bdece16b3451ca1b5bf419fac5a5bb9bb944301983efaeec",
|
||||
"figures/block_activity.svg": "da85920dd0fc32ba439b650c8b695757d42eb72768b4017a78109cb8a2f24216",
|
||||
"figures/mlp_activation.pdf": "6eef102d1ecc7c21206da862f79d4edaed77a695f7c46d53550c3dc6f9087ee5",
|
||||
"figures/mlp_activation.png": "0e1a01ea5b21274eaa757c1c9cd493524003220eadcf7aa378e0eb39a37a7c40",
|
||||
"figures/mlp_activation.svg": "d7b0a67d2d1d501e6fa70aa4a38010bead93a450423e5cd6b3b1284794353c8f",
|
||||
"final-tests.txt": "7b52cfeffca483471297e30aa9658bc2c5c6585b682f5ee5fbbcc2e1b2b8abe4",
|
||||
"first-attempt.txt": "7a1f686db3b7412b681d516bdd83bb8ed7ab2a758a44a2d84c4c7968017d9c91",
|
||||
"full-tests.txt": "55fe658a01d15ed380f2b5cd01058f2374997ffde945025262e92fd966f0dc3b",
|
||||
"metrics.csv": "8f6205e7f446f83567485275f57f63ce64233eee711a4be89c30b37182772e4d",
|
||||
"run.txt": "9fe01020a3ac92db51a17042eef6af73f854dd2e81220d7a04cd5a843c6ff9ba",
|
||||
"source-hashes.json": "34d1c164e787ec6ec5b77d4bc056da2e72365e1cd4b5089741f2b979a3c8beb3",
|
||||
"summary.json": "1e7c3bace11673d10cb927d689348bf00790a9b074ccafed6ce44d898410ef98"
|
||||
}
|
||||
Binary file not shown.
Binary file not shown.
|
After Width: | Height: | Size: 99 KiB |
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 78 KiB |
Binary file not shown.
Binary file not shown.
|
After Width: | Height: | Size: 163 KiB |
File diff suppressed because it is too large
Load Diff
|
After Width: | Height: | Size: 76 KiB |
@@ -0,0 +1,7 @@
|
||||
test_known_activity (tests.test_activation.ActivationTests.test_known_activity) ... ok
|
||||
test_zero_and_scale (tests.test_activation.ActivationTests.test_zero_and_scale) ... ok
|
||||
|
||||
----------------------------------------------------------------------
|
||||
Ran 2 tests in 0.002s
|
||||
|
||||
OK
|
||||
@@ -0,0 +1,24 @@
|
||||
completed fresh-zh-1 20904
|
||||
completed fresh-zh-2 41808
|
||||
completed fresh-en-1 62712
|
||||
Traceback (most recent call last):
|
||||
File "/opt/exp-model-parameter-similarity/activation_probe.py", line 44, in <module>
|
||||
if __name__=='__main__':main()
|
||||
~~~~^^
|
||||
File "/opt/exp-model-parameter-similarity/activation_probe.py", line 39, in main
|
||||
lm.forward(np.array([ids],dtype=np.int64)); print('completed',s['id'],len(lm.rows),flush=True)
|
||||
~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/opt/exp-model-parameter-similarity/functional/model.py", line 220, in forward
|
||||
h = h + self.mlp(n,donor)
|
||||
~~~~~~~~^^^^^^^^^
|
||||
File "/opt/exp-model-parameter-similarity/activation_probe.py", line 30, in mlp
|
||||
return self.linear(g*u,p+'down_proj')
|
||||
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
|
||||
File "/opt/exp-model-parameter-similarity/activation_probe.py", line 22, in linear
|
||||
y=super().linear(x,name,rows); self.record(y,name+'.output')
|
||||
File "/opt/exp-model-parameter-similarity/functional/model.py", line 130, in linear
|
||||
self.guard()
|
||||
~~~~~~~~~~^^
|
||||
File "/opt/exp-model-parameter-similarity/functional/model.py", line 111, in guard
|
||||
raise MemoryError(f'Sustained host swap pressure plus constrained memory: available={available} KiB, process_swap={own_swap_kib} KiB')
|
||||
MemoryError: Sustained host swap pressure plus constrained memory: available=989040 KiB, process_swap=132360 KiB
|
||||
@@ -0,0 +1,33 @@
|
||||
test_known_activity (test_activation.ActivationTests.test_known_activity) ... ok
|
||||
test_zero_and_scale (test_activation.ActivationTests.test_zero_and_scale) ... ok
|
||||
test_assignment_against_exhaustive_and_inverse (test_crossqkv.CrossQKVTests.test_assignment_against_exhaustive_and_inverse) ... ok
|
||||
test_cached_mode_rejects_future_state_reuse (test_crossqkv.CrossQKVTests.test_cached_mode_rejects_future_state_reuse) ... ok
|
||||
test_controls_cover_multiple_projection_layers (test_crossqkv.CrossQKVTests.test_controls_cover_multiple_projection_layers) ... ok
|
||||
test_head_matrix_metrics_and_gram_are_diagnostic (test_crossqkv.CrossQKVTests.test_head_matrix_metrics_and_gram_are_diagnostic) ... ok
|
||||
test_probability_identity_and_nontrivial_queries (test_crossqkv.CrossQKVTests.test_probability_identity_and_nontrivial_queries) ... ok
|
||||
test_profile_matches_canonical_and_head_sum (test_crossqkv.CrossQKVTests.test_profile_matches_canonical_and_head_sum) ... ok
|
||||
test_recipient_value_gate_output_and_gqa_group_stay_fixed (test_crossqkv.CrossQKVTests.test_recipient_value_gate_output_and_gqa_group_stay_fixed) ... ok
|
||||
test_topk_tail_rank_can_change_with_tiny_probability_difference (test_crossqkv.CrossQKVTests.test_topk_tail_rank_can_change_with_tiny_probability_difference) ... ok
|
||||
test_causal_convolution_and_norm_boundary (test_functional.FunctionalTests.test_causal_convolution_and_norm_boundary) ... ok
|
||||
test_causality_and_identical_function_metrics (test_functional.FunctionalTests.test_causality_and_identical_function_metrics) ... ok
|
||||
test_neuron_function_metrics_and_targeted_replacement (test_functional.FunctionalTests.test_neuron_function_metrics_and_targeted_replacement) ... ok
|
||||
test_nonrotary_gain_identity_and_rotary_negative_control (test_functional.FunctionalTests.test_nonrotary_gain_identity_and_rotary_negative_control) ... ok
|
||||
test_streamed_projection_embedding_and_output_head (test_functional.FunctionalTests.test_streamed_projection_embedding_and_output_head) ... ok
|
||||
test_uniform_rmsnorm_function_bound_is_tight (test_functional.FunctionalTests.test_uniform_rmsnorm_function_bound_is_tight) ... ok
|
||||
test_3d_reader_and_exact_block (test_pilot.PilotTests.test_3d_reader_and_exact_block) ... ok
|
||||
test_bfloat16 (test_pilot.PilotTests.test_bfloat16) ... ok
|
||||
test_directional_replacement_error_and_scaling (test_pilot.PilotTests.test_directional_replacement_error_and_scaling) ... ok
|
||||
test_end_to_end_synthetic_groups_and_savings (test_pilot.PilotTests.test_end_to_end_synthetic_groups_and_savings) ... Inventory: 4 tensors, 33792 bytes
|
||||
tensor: sampling 4 objects
|
||||
block: sampling 22 objects
|
||||
block: verified 100/109
|
||||
{"model": "synthetic", "revision": "test", "tensor_count": 4, "parameter_count": 8448, "model_weight_bytes": 33792, "ndim_counts": {"2": 2, "3": 2}, "elapsed_seconds": 0.3087193910032511, "analysis_budget_seconds": 60, "peak_rss_kib": 57700, "python": "3.14.4", "numpy": "2.5.4", "seed": 20261109, "limitations": ["No inference or functional equivalence measured", "No exhaustive pair search", "Block search uses per-role/shape/scale reservoirs of 256", "Different block scales overlap; savings must not be added", "Block savings count only verified disjoint sampled blocks, no extrapolation", "Spectral probes use aligned bounded subvolumes, not whole tensors", "Shared encoding is theoretical, not a runnable compressed checkpoint", "No cross-role grouping, neuron permutation alignment, or training"], "completed": true}
|
||||
ok
|
||||
test_no_transitive_chaining (test_pilot.PilotTests.test_no_transitive_chaining) ... ok
|
||||
test_ranged_download_preserves_prefix (test_pilot.PilotTests.test_ranged_download_preserves_prefix) ... Weight download 36/36 bytes (100.0%)
|
||||
ok
|
||||
|
||||
----------------------------------------------------------------------
|
||||
Ran 22 tests in 33.253s
|
||||
|
||||
OK
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,4 @@
|
||||
completed fresh-zh-1 20904
|
||||
completed fresh-zh-2 20904
|
||||
completed fresh-en-1 20904
|
||||
completed fresh-en-2 20904
|
||||
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"activation/report.py": "e67bfa4af69e43cdceb803b77fe6a6dcc151163e72b3ff2bb7e6b69e7c6fa1fc",
|
||||
"activation/audit.py": "7dc8af7c31de379a50bdcbd3c41243eb190e6a50e8ec00a30182e82a34e73627",
|
||||
"activation/run.py": "cba62a2d8b6344625f233309b166b091028154890d93e48278eb338cbcd19b18",
|
||||
"activation/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
|
||||
"tests/test_activation.py": "dee1756b63cd03c2051cdc92d678d2e27aff639fbfd3d3cb3169140760da4c68",
|
||||
"functional/model.py": "48a2a97d0445dc6e08ac7f27a935637a7553417d0982eb94122d6cb02bbb499c"
|
||||
}
|
||||
@@ -0,0 +1,180 @@
|
||||
{
|
||||
"completed": true,
|
||||
"elapsed_seconds": 268.55615069903433,
|
||||
"peak_rss_kib": 155288,
|
||||
"config_activation": "silu",
|
||||
"layer_count": 24,
|
||||
"samples": [
|
||||
{
|
||||
"id": "fresh-zh-1",
|
||||
"language": "zh",
|
||||
"text": "\u690d\u7269\u5b66\u5bb6\u5728\u4e0d\u540c\u6d77\u62d4\u91c7\u96c6\u4e86\u53f6\u7247\u6837\u672c\uff0c\u5e76\u4e3a\u6bcf\u4e00\u4efd\u6807\u672c\u8bb0\u5f55\u6e29\u5ea6\u3001\u5149\u7167\u548c\u571f\u58e4\u6e7f\u5ea6\u3002\u56de\u5230\u5b9e\u9a8c\u5ba4\u540e\uff0c\u5979\u6309\u5730\u70b9\u6574\u7406\u6807\u7b7e\uff0c\u5148\u6838\u5bf9\u7f3a\u5931\u8bb0\u5f55\uff0c\u518d\u6bd4\u8f83\u4e0d\u540c\u73af\u5883\u4e2d\u7684\u53f6\u7247\u5f62\u72b6\u3002",
|
||||
"token_ids": [
|
||||
99951,
|
||||
101071,
|
||||
115802,
|
||||
111685,
|
||||
102099,
|
||||
95759,
|
||||
111583,
|
||||
109928,
|
||||
3709,
|
||||
115026,
|
||||
134176,
|
||||
116599,
|
||||
98151,
|
||||
99472,
|
||||
5205,
|
||||
114180,
|
||||
95793,
|
||||
104074,
|
||||
102454,
|
||||
1710,
|
||||
103744,
|
||||
103911,
|
||||
95946,
|
||||
3709,
|
||||
96343,
|
||||
96656,
|
||||
100560,
|
||||
99946,
|
||||
101659,
|
||||
3709,
|
||||
96218,
|
||||
113089
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "fresh-zh-2",
|
||||
"language": "zh",
|
||||
"text": "\u5982\u679c\u4e00\u4e2a\u76d2\u5b50\u91cc\u6709\u5341\u4e8c\u679a\u84dd\u8272\u68cb\u5b50\u548c\u516b\u679a\u767d\u8272\u68cb\u5b50\uff0c\u968f\u673a\u53d6\u51fa\u4e00\u679a\u540e\u4e0d\u653e\u56de\uff0c\u7b2c\u4e8c\u6b21\u53d6\u5230\u84dd\u8272\u7684\u6982\u7387\u9700\u8981\u6839\u636e\u7b2c\u4e00\u6b21\u7684\u7ed3\u679c\u5206\u522b\u8ba1\u7b97\u3002\u8bf7\u5148\u5217\u51fa\u5404\u4e2a\u4e8b\u4ef6\uff0c\u518d\u68c0\u67e5\u5b83\u4eec\u7684\u6982\u7387\u4e4b\u548c\u3002",
|
||||
"token_ids": [
|
||||
127356,
|
||||
98996,
|
||||
98970,
|
||||
95762,
|
||||
98175,
|
||||
101184,
|
||||
104145,
|
||||
99357,
|
||||
111324,
|
||||
96702,
|
||||
101184,
|
||||
103828,
|
||||
129730,
|
||||
3709,
|
||||
102038,
|
||||
96284,
|
||||
98938,
|
||||
101184,
|
||||
95946,
|
||||
122271,
|
||||
96120,
|
||||
3709,
|
||||
109387,
|
||||
96284,
|
||||
95839,
|
||||
116981,
|
||||
104514,
|
||||
122501,
|
||||
103720,
|
||||
109146,
|
||||
99446,
|
||||
97792
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "fresh-en-1",
|
||||
"language": "en",
|
||||
"text": "A technician recorded the vibration of a small motor before and after replacing its bearings. She used the same mounting position and sampling interval for both measurements. The final notebook included the raw traces, instrument settings, and a brief explanation of unusual observations.",
|
||||
"token_ids": [
|
||||
32,
|
||||
51715,
|
||||
12076,
|
||||
279,
|
||||
46219,
|
||||
314,
|
||||
264,
|
||||
2526,
|
||||
8604,
|
||||
1518,
|
||||
321,
|
||||
1238,
|
||||
24101,
|
||||
1141,
|
||||
33309,
|
||||
13,
|
||||
2838,
|
||||
1429,
|
||||
279,
|
||||
1788,
|
||||
32525,
|
||||
2234,
|
||||
321,
|
||||
24102,
|
||||
9581,
|
||||
364,
|
||||
2107,
|
||||
20934,
|
||||
13,
|
||||
561,
|
||||
1534,
|
||||
35935
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "fresh-en-2",
|
||||
"language": "en",
|
||||
"text": "The function walks through a list of integers and stores each running total in a new list. An empty input should return an empty result. Before changing the implementation, compare a short example by hand and make sure negative numbers preserve the intended behavior.",
|
||||
"token_ids": [
|
||||
760,
|
||||
709,
|
||||
21780,
|
||||
1472,
|
||||
264,
|
||||
1103,
|
||||
314,
|
||||
24959,
|
||||
321,
|
||||
10220,
|
||||
1754,
|
||||
4162,
|
||||
2702,
|
||||
303,
|
||||
264,
|
||||
491,
|
||||
1103,
|
||||
13,
|
||||
1473,
|
||||
4147,
|
||||
1879,
|
||||
1220,
|
||||
460,
|
||||
449,
|
||||
4147,
|
||||
1067,
|
||||
13,
|
||||
12851,
|
||||
9722,
|
||||
279,
|
||||
7873,
|
||||
11
|
||||
]
|
||||
}
|
||||
],
|
||||
"rows": 83616,
|
||||
"block_widths": [
|
||||
64,
|
||||
128
|
||||
],
|
||||
"threshold_reference": "RMS within each sample/module/block, thresholds 0.01 and 0.1; exact zero measured separately",
|
||||
"training": false,
|
||||
"weights_deleted": false,
|
||||
"limits": [
|
||||
"4 short texts, 32 tokens each, descriptive only",
|
||||
"Block is contiguous output channels, not Transformer block",
|
||||
"Small output does not prove removable computation or parameter interchangeability",
|
||||
"All samples previously used; no generalization claim",
|
||||
"Relative threshold differs between blocks; width=0 is whole module reference"
|
||||
]
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user