Archive research charts and compare GSM8K LoRA head concentration

This commit is contained in:
2026-11-11 00:06:15 +08:00
parent d6d985487c
commit 52e6d13fe5
142 changed files with 18505 additions and 0 deletions
+42
View File
@@ -0,0 +1,42 @@
# Paired task-finetuning comparison (registered before reading finetuned effects)
Primary checkpoint: Avifenesh/gdn-beta-gate-qwen3.5-2b-grpo,
HF revision038d8be9bd0edd3883fa019651b02bdf65609468, pure lora/seed0,
step150 after shared40-step warmup. GSM8K GRPO, not SFT. Selected by smallest
seed index before observing concentration, not by benchmark score or our metric.
No gate01/gate02 arm or architecture changes. Exact base revision matches retained
Qwen/Qwen3.5-2B15852e8c16360a2fea060d615a32b45270f8a8fc.
Download existing18.9MB F32 LoRA tensors, verify official file SHA. All144 tensors
must form72 A/B pairs on the4 declared GDN projections of18 linear-attention
layers, rank16 alpha32. Apply FP32 effective W +2BA in streamed row chunks, also
when decomposing source output heads. Do not quantize merged weights, mutate or
save over base weights, execute foreign .pt files, train or use GPU. Validate
against separate base-output +2B(Ax) forward and check head sums/null masks.
Primary comparison exactly matches the reported8.931822/16 statistic: all six
ordinary attention receivers3/7/11/15/19/23, previous-layer16 donor output heads,
all32 tokens computed, query7..31, original4 plain-text inputs and same token IDs.
Reuse corresponding immutable base single-ablation arrays, recompute FT baselines
and all16 single donor removals, source MLP/residual/recipient normalization exact.
Use the same sqrt(JS) distance influence, normalized shares and N_eff=1/sum(p²).
Preserve JS divergence as a sensitivity row; no change of ranking or threshold.
All4800 per-token/query-head observations are paired. No new joint-mask search:
the requested statistic depends on single removals, not the previous joint tests.
Secondary task-conditioned comparison: first4 original GSM8K test questions with
at least32 base-tokenizer tokens (length selection only, before model effects),
question-only plain text, first32 tokens. Freeze original author repository commit,
JSONL SHA, item indices, question hashes and exact token IDs. No answers, generation,
chat-template change or accuracy claim. Compute both base and FT on these same
questions; never compare old general-text base to new GSM8K FT as if paired.
Report overall median/P10/P90, paired differences, per receiver layer and per text,
strong/weak-quartile shares, and dominant donor identity agreement. No iid-token
p-values, no claim over training seeds (one fixed trained seed), no proof of
GSM8K accuracy improvement or long-context behavior. Positive, negative and near
zero changes all count as results. Source cohort remains4+4 short texts.
All model work/tests/plots on authorized CPU server8.211.154.134, bounded7200s.
Preserve all base/FT weights, adapters and caches; no automatic cleanup. Save raw
arrays, source provenance, SHA manifests, fixed chart gallery and reviewable PR.
+61
View File
@@ -0,0 +1,61 @@
# Existing GSM8K fine-tune: paired head-concentration comparison
First restore the original charts through [the fixed figure gallery](../docs/FIGURE_GALLERY.md).
The original 8.931822/16, P10=5.784081 and P90=11.672658 refer to sqrt(JS)
influences of all16 prior-layer heads on the six full-attention receivers.
Selected release: [Avifenesh/gdn-beta-gate-qwen3.5-2b-grpo](https://huggingface.co/Avifenesh/gdn-beta-gate-qwen3.5-2b-grpo),
pinned038d8be9bd0edd3883fa019651b02bdf65609468, pure lora/seed0. The publisher
records GSM8K GRPO from a shared warmup, rank16 alpha32, exact same original
Qwen3.5-2B revision. This is extra RL fine-tuning of the post-trained release,
not pretraining-to-SFT, and not a beta-gate architecture modification. Our seed
choice is fixed before seeing effects; no new training is performed.
`lora.py` validates all144 tensors/72 pairs, maps only declared GDN projections,
and provides read-only FP32 effective row chunks W+2BA. Source head decomposition
uses the same effective weights. `ExplicitLoRA` adds2B(Ax) independently to the
base projection as a numerical reference. No BF16 re-rounding is introduced:
the author's training code also warns that rounding merged increments can erase
small LoRA changes. Author configuration commit/SHA and public HF metadata are
saved alongside the protocol; foreign training code and pickle checkpoints are
not executed.
`run.py` reuses the exact original single-ablation arrays for the original cohort,
then computes matched finetuned states; it computes both models on4 question-only
GSM8K test prefixes as a separate cohort. Every input has32 tokens. All16 donor
heads, six receiver layers andquery7..31 remain identical to the original metric.
The task corpus is the canonical author's test JSONL at a fixed Git commit, and
selection uses only the first4 items with at least32 tokenizer tokens. All item
indices, question hashes and token IDs are recorded; answers are not model inputs.
`report.py` generates paired CDFs, per-layer medians/quantiles, paired change
histograms, raw observations and tables. `audit.py` independently reconstructs
effective-head counts via(sum d)^2/sum(d^2), checks pairing, exact old baselines,
head sums, null masks, adapter/dataset/base SHA and retained external caches.
`tests/test_finetune_compare.py` checks streamed row arithmetic, an independent
low-rank forward and head decomposition, plus exact zero-scale restoration.
No iid-token p-values, training-seed generalization, GSM8K accuracy result,
long-context claim, pruning recommendation or functional-duplicate claim.
The paired delta median is distinguished from the difference of pooled medians.
All weights, adapters and caches remain on the authorized CPU server until user
explicit deletion. Git stores all raw influence arrays, not the model/adapter.
Remote workflow on8.211.154.134:
```
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -B -m unittest tests.test_finetune_compare -v
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -B -m finetune_compare.run --source runs/cpu-qwen35-2b --parent runs/head-dominance --assets runs/finetune-head-comparison --out runs/finetune-head-comparison/NEW_EXPERIMENT
venv/bin/python -B -m finetune_compare.report runs/finetune-head-comparison/NEW_EXPERIMENT
venv/bin/python -B -m finetune_compare.audit runs/finetune-head-comparison/NEW_EXPERIMENT runs/head-dominance runs/finetune-head-comparison runs/cpu-qwen35-2b
```
The supplemental QKV field is measured only on the same six full-attention
receivers, so its baseline must not be confused with the older76.0% pooled
QKV effective-head fraction across all23 adjacent transitions. The gallery
preserves that original table separately; the requested8.931822 metric uses
exactly the matching six receivers in both studies.
The audit also records all72 low-rank update Frobenius norms (via small Gram
matrices) and final pre-normalization residual changes per text. These separate
a stable concentration statistic from an accidentally inactive adapter. Residual
changes are not logit distributions, accuracy differences or capability scores.
View File
+74
View File
@@ -0,0 +1,74 @@
"""Independent paired effective-head, source, preservation and delivery checks."""
import json,hashlib,sys
from pathlib import Path
import numpy as np
def sha(path):
with Path(path).open('rb') as f:return hashlib.file_digest(f,'sha256').hexdigest()
def main(root,parent,assets,source):
root,parent,assets,source=map(Path,[root,parent,assets,source]);status=json.loads((root/'status.json').read_text());assert status['state']=='completed' and len(status['links'])==96
samples=json.loads((root/'samples.json').read_text());assert set(samples)=={'legacy','gsm8k'} and all(len(v)==4 for v in samples.values());assert all(len(s['token_ids'])==32 for ss in samples.values() for s in ss)
assert status['explicit_lora_forward_relative_error']<1e-4;rawstats={};expected=set();obs=dict(np.load(root/'observations.npz'));summary=json.loads((root/'summary.json').read_text())
def check(x,record):
x=np.asarray(x,dtype=np.float64);x=x[np.isfinite(x)];assert record['count']==len(x)
for name,value in zip(('p10','median','p90','mean'),[*np.quantile(x,[.1,.5,.9]),x.mean()]):np.testing.assert_allclose(record[name],value,atol=1e-8,rtol=1e-6)
for cohort,ss in samples.items():
for role in ('base','finetuned'):
for sample in ss:
for layer in (2,6,10,14,18,22):
name=f'{cohort}-{role}-{sample["id"]}-{layer:02d}.npz';expected.add(name);d=dict(np.load(root/'raw'/name))
for key in ('Q','K','V','QKV','JS'):
assert d['single_'+key].shape==(16,32,8) and np.isfinite(d['single_'+key]).all() and np.min(d['single_'+key])>=0
assert np.max(d['null_'+key])<2e-5
if cohort=='legacy' and role=='base':
with np.load(parent/'raw'/f'{sample["id"]}-link-{layer:02d}.npz') as original:
for key in d:np.testing.assert_array_equal(d[key],original[key])
for metric,impacts in [('distance',np.sqrt(d['single_JS'])),('divergence',d['single_JS']),('QKV',d['single_QKV'])]:
x=impacts[:,7:,:].transpose(1,2,0).astype(np.float64);total=x.sum(-1);den=np.sum(x*x,-1);assert np.all(den>0)
neff=total*total/den
rawstats.setdefault((cohort,role,metric),[]).append(neff.ravel())
assert expected=={f.name for f in (root/'raw').glob('*.npz')}
for cohort in samples:
for metric in ('distance','divergence','QKV'):
for role in ('base','finetuned'):
x=np.concatenate(rawstats[(cohort,role,metric)]);assert len(x)==4800
np.testing.assert_allclose(obs['__'.join((cohort,role,metric,'effective_heads'))],x,rtol=1e-8,atol=1e-8)
check(x,summary[cohort][metric][role]['effective_heads'])
before=np.concatenate(rawstats[(cohort,'base',metric)]);after=np.concatenate(rawstats[(cohort,'finetuned',metric)]);check(after-before,summary[cohort][metric]['paired']['effective_heads_delta'])
base=summary['legacy']['distance']['base']['effective_heads'];np.testing.assert_allclose([base[x] for x in ('median','p10','p90')],[8.93182211792415,5.7840811612328125,11.67265834734814],rtol=1e-10)
controls=json.loads((root/'head-sum-controls.json').read_text());assert len(controls)==288 and all(c['relative_l2']<2e-5 for c in controls)
refs=json.loads((root/'baseline-references.json').read_text());assert len(refs)==24 and all(sha(v['source'])==v['sha256'] for v in refs)
assert sha(assets/'adapter.safetensors')=='93a3f43a4356f84da0d13ecc622fee13e41ba15d237cdaad6094f243899c62d6'
dataset=json.loads((root/'gsm8k-source.json').read_text());assert sha(assets/'gsm8k-test.jsonl')==dataset['sha256']
oldweights=list((source/'weights').glob('*.safetensors'));assert len(oldweights)==1
base_sha=sha(oldweights[0]);assert base_sha=='aa33250c4fc64891ddfaba3a314fd9542ea371843c387178b425fbcc5ed680b1'
from pilot.tensors import load_tensors
import re
tensors=load_tensors(assets);pairs={}
for name,t in tensors.items():
stem,which=name.split('.lora_');pairs.setdefault(stem,{})[which[0]]=t.read().reshape(t.shape).copy();t.evict()
updates=[]
for name,pair in pairs.items():
a=pair['A'].astype(np.float64);b=pair['B'].astype(np.float64)
norm2=4*float(np.sum((b.T@b)*(a@a.T).T))
updates.append({'module':name,'delta_frobenius':float(np.sqrt(max(norm2,0))),'A_nonzero':int(np.count_nonzero(a)),'B_nonzero':int(np.count_nonzero(b))})
assert len(updates)==72 and any(x['delta_frobenius']>0 for x in updates)
(root/'weight-update-summary.json').write_text(json.dumps({'scale':2,'modules':updates,'nonzero_update_modules':sum(x['delta_frobenius']>0 for x in updates),'norm_identity':'||2BA||_F^2 = 4 trace((B^T B)(A A^T))'},indent=2)+'\n')
changes=[]
for cohort,ss in samples.items():
for sample in ss:
basefile=parent/'cache'/sample['id']/'final.npy' if cohort=='legacy' else root/'cache'/f"{cohort}-base-{sample['id']}"/'final.npy'
ftfile=root/'cache'/f"{cohort}-finetuned-{sample['id']}"/'final.npy'
before=np.load(basefile).astype(np.float64);after=np.load(ftfile).astype(np.float64)
changes.append({'cohort':cohort,'sample':sample['id'],'final_pre_norm_residual_relative_l2':float(np.linalg.norm(after-before)/np.linalg.norm(before))})
assert any(x['final_pre_norm_residual_relative_l2']>0 for x in changes)
(root/'hidden-state-changes.json').write_text(json.dumps({'definition':'final residual before final RMSNorm, not logits, accuracy or LM quality','changes':changes},indent=2)+'\n')
caches=[{'path':str(p.resolve()),'bytes':p.stat().st_size,'sha256':sha(p)} for p in sorted((root/'cache').rglob('*')) if p.is_file()]
assert len(caches)==300
(root/'external-caches.json').write_text(json.dumps({'host':'8.211.154.134','base_weights':str(oldweights[0].resolve()),'base_sha256':base_sha,'adapter_path':str((assets/'adapter.safetensors').resolve()),'adapter_sha256':sha(assets/'adapter.safetensors'),'caches':caches,'automatic_deletion':False},indent=2)+'\n')
files=list(Path('finetune_compare').glob('*.py'))+[Path('finetune_compare/PROTOCOL.md'),Path('tests/test_finetune_compare.py'),Path('dominance/core.py'),Path('dominance/run.py'),Path('functional/model.py')]
(root/'source-hashes.json').write_text(json.dumps({str(f):sha(f) for f in files},indent=2)+'\n')
proof={'passed':True,'links':96,'paired_observations_per_cohort_metric':4800,'legacy_baseline_exact':True,'effective_head_formula_independently_recomputed':True,'head_sum_checks':288,'explicit_lora_forward_relative_error':status['explicit_lora_forward_relative_error'],'adapter_sha_verified':True,'base_sha_verified':True,'dataset_sha_verified':True,'retained_cache_files':len(caches),'nonzero_update_modules':sum(x['delta_frobenius']>0 for x in updates),'nonzero_hidden_state_changes_verified':True}
(root/'audit.json').write_text(json.dumps(proof,indent=2)+'\n');print(json.dumps(proof,indent=2),flush=True)
included=[p for p in root.rglob('*') if p.is_file() and 'cache' not in p.relative_to(root).parts and p.name!='checksums.json'];(root/'checksums.json').write_text(json.dumps({str(p.relative_to(root)):sha(p) for p in sorted(included)},indent=2)+'\n')
if __name__=='__main__':main(*sys.argv[1:])
@@ -0,0 +1,12 @@
{
"source_repository": "https://github.com/avifenesh/gdn-beta-gate",
"source_revision": "f12f61f24ebe5997aac58667f413dcd1dcd96f54",
"lora_configuration_path": "src/gdn_gate/broad/arms.py",
"lora_configuration_sha256": "fc5b8de096eaa259ab166b8083bb774f9e14859d566ce82ea15986cc8d6eb5b5",
"verified_fields": [
"LoraConfig r=rank",
"lora_alpha=2*rank",
"use_rslora not enabled"
],
"effective_scale": 2
}
+38
View File
@@ -0,0 +1,38 @@
93a3f43a4356f84da0d13ecc622fee13e41ba15d237cdaad6094f243899c62d6 lora/seed0/adapter.safetensors
b8d4737a5b940bcdfed72258600227a89f883dc863e9724bf84e457b46a53703 lora/seed0/adapter_meta.json
63d54e1c2f6bf7d07ca0f6e83a3299bd4b6d00740919bc7cae0b422af05a828e lora/seed1/adapter.safetensors
f8be6c4218876b7b12a47945cafe18fcc27d59ff06447db6d3e5b21f0bb9fc29 lora/seed1/adapter_meta.json
702b764e2c39b3529914dac1547148ef2f462d9ee04e256201e79f00b20ccc62 lora/seed2/adapter.safetensors
9acee331b9383f49a81cfc057163e034f06ea1ba4b8167fea441f0fbfd490f94 lora/seed2/adapter_meta.json
7dbf35899fe1186453e599ad8c7a85c56b7fad3d630e371ca026a1e95da2c5a7 lora/seed3/adapter.safetensors
b5bfbefa8ce6b428feea761c39a148f003bf02396ca6e01b68abfba2ab3f71af lora/seed3/adapter_meta.json
a5cb1e7fa74dff467bd7f163e1db23d30ef8b1a1c935cfa55dcc8cd1b39082a7 lora/seed4/adapter.safetensors
23a6c3f198978e79014361a326af20ec364ea266cf9b5e9ced4c9da06feab30e lora/seed4/adapter_meta.json
d513d499dfd409da88b3db7da025ba9872fb48ec92d6b29bc2ce7a092a69745d lora/seed5/adapter.safetensors
9384d0b4ce4cec6b3dc15756fb4d46c5625a4dacd3805af728fc37b89627d0ed lora/seed5/adapter_meta.json
fda9b4c15d283ef8ee778c4416d69b12e1922527ea330bfc67bab5cde1b6741a gate01/seed0/adapter.safetensors
131e60bc768412f4d344e5b17cbc811f35d149aad0c328bc6503646513bbd8fc gate01/seed0/adapter_meta.json
6ada79d07414641bed58d74fd23090937abcf52337e75e8f06483d502fa64c57 gate01/seed1/adapter.safetensors
0350240ed838a93a26f406ef0d63f29ced21ea2284c9430d07aefccd30f8d924 gate01/seed1/adapter_meta.json
a9359e3f3be68468604f9bf93f20f4b65114084f3fc6a3c13363e7062c4cf547 gate01/seed2/adapter.safetensors
17fa2e876770ec636f8a0a0eb0b3f7b58c91a5f6f9910c09c38c593d856b035f gate01/seed2/adapter_meta.json
a2d0c27fee0263c9a8152e0849975edefedc9d10061bbf789a3ead223db8703c gate01/seed3/adapter.safetensors
4da32ea52fe6e7c2a0a8858e4d3255540fcc7adf18a7ab2ce9fd3164e4bb0a05 gate01/seed3/adapter_meta.json
810b8f8d6147af3620cf08d2cf575c0eb2b0134ae0253a93ceeae5be9dca8695 gate01/seed4/adapter.safetensors
f943fc6874af4e4b24763ee6e0cf6e737bc62720045190441d0b33dd561a439f gate01/seed4/adapter_meta.json
b34e8d999612415c7395f341633ff3d2365a020b9e27b7e371c5f80f0105a486 gate01/seed5/adapter.safetensors
98983ecbf303bf1b10e11c816f850db13af8a0ca5ff2e4efd121f7fd70262918 gate01/seed5/adapter_meta.json
1fc5a4a4392c35820c26e24d0075824b3426c4a77f3119713b4a1e577d31df5b gate02/seed0/adapter.safetensors
0a17ea0542fb52751d0e628985f8d07240a3ccc4b44a7e2c970e7029c00f6c3f gate02/seed0/adapter_meta.json
b742f4ffed0e293f27e4646c364a85ae359a3eae1690d3840e531dca7fba9cd8 gate02/seed1/adapter.safetensors
4b6c7bf77edf3049ca904863d59d4d3bfdbe1d0b1b3f563b7fd3bea7f179f97c gate02/seed1/adapter_meta.json
dc36c27a98379535007f2df798953624b6a67f0677bd41dec42349f8ce868afa gate02/seed2/adapter.safetensors
20009502b1bd6aa9c67b509b77eac0aa0d367af7f347caf4d05c7a51367b8f61 gate02/seed2/adapter_meta.json
1dcb6132afeec8101daf9a3fd73d42141fb871bff11fff28069f238dcff6afd0 gate02/seed3/adapter.safetensors
518289370e0eb9fc064c3226c2b3811f94cec180a9ae366a026f30cf1d3a3131 gate02/seed3/adapter_meta.json
5b97b0601585925c19bf849ecb11ad68597d1b0d55551b917ec0f5091f85c2ff gate02/seed4/adapter.safetensors
8419e8e22aa583f97e570950baf37e7c6370856b7cacefaf9c5ebeb0ea4f731c gate02/seed4/adapter_meta.json
3f944a6d210958de5840e5b76e263f2105b9f49d15b9ab505950c2b29a5343af gate02/seed5/adapter.safetensors
94d9c1e202f7962da61fa060d27941bbef7ff3d23ca5548f5d73fb6f5ac39aac gate02/seed5/adapter_meta.json
9108bce2acfcc3ccd38d0dfae67ac33253b95bdc94133bd25deb4daa45d41700 warmup/seed0/adapter.safetensors
d2d4346179104830410be9d2776d79983481503daeb5c8383e2a062de43714ef warmup/seed0/adapter_meta.json
+379
View File
@@ -0,0 +1,379 @@
{
"_id": "6accb5a881b0c228f413b5e5",
"id": "Avifenesh/gdn-beta-gate-qwen3.5-2b-grpo",
"private": false,
"pipeline_tag": "text-generation",
"library_name": "pytorch",
"tags": [
"pytorch",
"lora",
"gated-deltanet",
"qwen3.5",
"beta-gate",
"grpo",
"reasoning",
"math",
"safetensors",
"research",
"text-generation",
"en",
"dataset:openai/gsm8k",
"base_model:Qwen/Qwen3.5-2B",
"base_model:adapter:Qwen/Qwen3.5-2B",
"license:apache-2.0",
"region:us"
],
"downloads": 0,
"likes": 2,
"modelId": "Avifenesh/gdn-beta-gate-qwen3.5-2b-grpo",
"author": "Avifenesh",
"sha": "038d8be9bd0edd3883fa019651b02bdf65609468",
"lastModified": "2026-11-09T19:45:55.000Z",
"gated": false,
"disabled": false,
"widgetData": [
{
"text": "Hi, what can you help me with?"
},
{
"text": "What is 84 * 3 / 2?"
},
{
"text": "Tell me an interesting fact about the universe!"
},
{
"text": "Explain quantum computing in simple terms."
}
],
"model-index": null,
"config": {},
"cardData": {
"license": "apache-2.0",
"base_model": "Qwen/Qwen3.5-2B",
"base_model_relation": "adapter",
"library_name": "pytorch",
"pipeline_tag": "text-generation",
"language": [
"en"
],
"datasets": [
"openai/gsm8k"
],
"tags": [
"lora",
"gated-deltanet",
"qwen3.5",
"beta-gate",
"grpo",
"reasoning",
"math",
"safetensors",
"research"
]
},
"siblings": [
{
"rfilename": ".gitattributes",
"blobId": "a6344aac8c09253b3b630fb776ae94478aa0275b",
"size": 1519
},
{
"rfilename": "README.md",
"blobId": "916419ce9530318d51d71325023fb2a5024475f0",
"size": 6527
},
{
"rfilename": "SHA256SUMS",
"blobId": "cc8927482e300cb79fed740be333a8ac5639386c",
"size": 3700
},
{
"rfilename": "gate01/seed0/adapter.safetensors",
"blobId": "86a05f3f118a88940a9122fab59a8e7db73d4b4f",
"size": 21293968,
"lfs": {
"sha256": "fda9b4c15d283ef8ee778c4416d69b12e1922527ea330bfc67bab5cde1b6741a",
"size": 21293968,
"pointerSize": 133
}
},
{
"rfilename": "gate01/seed0/adapter_meta.json",
"blobId": "0652fd0ab9647ed184e2c65a670da74de1a24a4e",
"size": 661
},
{
"rfilename": "gate01/seed1/adapter.safetensors",
"blobId": "596ed265e665aea6f674b81eecfdaaada202be45",
"size": 21293968,
"lfs": {
"sha256": "6ada79d07414641bed58d74fd23090937abcf52337e75e8f06483d502fa64c57",
"size": 21293968,
"pointerSize": 133
}
},
{
"rfilename": "gate01/seed1/adapter_meta.json",
"blobId": "934875448487af7efe63e3eae775b24d2d388964",
"size": 661
},
{
"rfilename": "gate01/seed2/adapter.safetensors",
"blobId": "6413c056f06232ad2ca3d513fa44cd01a90d251b",
"size": 21293968,
"lfs": {
"sha256": "a9359e3f3be68468604f9bf93f20f4b65114084f3fc6a3c13363e7062c4cf547",
"size": 21293968,
"pointerSize": 133
}
},
{
"rfilename": "gate01/seed2/adapter_meta.json",
"blobId": "abc653d6a65aa8b3b1f2a18aaf12dc1260409573",
"size": 661
},
{
"rfilename": "gate01/seed3/adapter.safetensors",
"blobId": "f17c2715463cadb293f36a0fc2c15fb0542990a6",
"size": 21293968,
"lfs": {
"sha256": "a2d0c27fee0263c9a8152e0849975edefedc9d10061bbf789a3ead223db8703c",
"size": 21293968,
"pointerSize": 133
}
},
{
"rfilename": "gate01/seed3/adapter_meta.json",
"blobId": "f511293207bf322ce11a4c26dc83cc57b20edf52",
"size": 661
},
{
"rfilename": "gate01/seed4/adapter.safetensors",
"blobId": "32503f87fd062bf0631c8eac7d8dbba09ef0e6cb",
"size": 21293968,
"lfs": {
"sha256": "810b8f8d6147af3620cf08d2cf575c0eb2b0134ae0253a93ceeae5be9dca8695",
"size": 21293968,
"pointerSize": 133
}
},
{
"rfilename": "gate01/seed4/adapter_meta.json",
"blobId": "3fef81591cd7a41c636c39e72e22d5b3394dc6a5",
"size": 661
},
{
"rfilename": "gate01/seed5/adapter.safetensors",
"blobId": "63b9a6dd509a288e6095ed9531fe76a055e2e5b4",
"size": 21293968,
"lfs": {
"sha256": "b34e8d999612415c7395f341633ff3d2365a020b9e27b7e371c5f80f0105a486",
"size": 21293968,
"pointerSize": 133
}
},
{
"rfilename": "gate01/seed5/adapter_meta.json",
"blobId": "d0daa05e6348923237cdac83935118e4f020e9f7",
"size": 661
},
{
"rfilename": "gate02/seed0/adapter.safetensors",
"blobId": "d566d86611edc89edb7cac62dc73d72619df55bd",
"size": 21293968,
"lfs": {
"sha256": "1fc5a4a4392c35820c26e24d0075824b3426c4a77f3119713b4a1e577d31df5b",
"size": 21293968,
"pointerSize": 133
}
},
{
"rfilename": "gate02/seed0/adapter_meta.json",
"blobId": "36173814d1d838077da330e92fa5f0b1b222ba17",
"size": 660
},
{
"rfilename": "gate02/seed1/adapter.safetensors",
"blobId": "cba04a7ef1bcb48952a723b6aa80d1b6847ef8dc",
"size": 21293968,
"lfs": {
"sha256": "b742f4ffed0e293f27e4646c364a85ae359a3eae1690d3840e531dca7fba9cd8",
"size": 21293968,
"pointerSize": 133
}
},
{
"rfilename": "gate02/seed1/adapter_meta.json",
"blobId": "3c7751cefbe375871d0b99bfda0b3ba1e6ca896c",
"size": 660
},
{
"rfilename": "gate02/seed2/adapter.safetensors",
"blobId": "af547c0fcaa9e69024e14818eb82c46f155dfa59",
"size": 21293968,
"lfs": {
"sha256": "dc36c27a98379535007f2df798953624b6a67f0677bd41dec42349f8ce868afa",
"size": 21293968,
"pointerSize": 133
}
},
{
"rfilename": "gate02/seed2/adapter_meta.json",
"blobId": "baaefc1b64b0dce0c87bbc2a7c4342f87111aba6",
"size": 660
},
{
"rfilename": "gate02/seed3/adapter.safetensors",
"blobId": "21066e00dfc55fc619ca74202ff8ad8dd6017d4e",
"size": 21293968,
"lfs": {
"sha256": "1dcb6132afeec8101daf9a3fd73d42141fb871bff11fff28069f238dcff6afd0",
"size": 21293968,
"pointerSize": 133
}
},
{
"rfilename": "gate02/seed3/adapter_meta.json",
"blobId": "f675496569bbec647fcb31784c4a6d9d83872ee0",
"size": 660
},
{
"rfilename": "gate02/seed4/adapter.safetensors",
"blobId": "bec0713c08368303a25e2ccef690d37a8cf2d54b",
"size": 21293968,
"lfs": {
"sha256": "5b97b0601585925c19bf849ecb11ad68597d1b0d55551b917ec0f5091f85c2ff",
"size": 21293968,
"pointerSize": 133
}
},
{
"rfilename": "gate02/seed4/adapter_meta.json",
"blobId": "8535e4793f089b37fa8bf5724cf49ac24fd2750c",
"size": 660
},
{
"rfilename": "gate02/seed5/adapter.safetensors",
"blobId": "128821f8b3b0591990a3b18c70ca66a16d69aff3",
"size": 21293968,
"lfs": {
"sha256": "3f944a6d210958de5840e5b76e263f2105b9f49d15b9ab505950c2b29a5343af",
"size": 21293968,
"pointerSize": 133
}
},
{
"rfilename": "gate02/seed5/adapter_meta.json",
"blobId": "ae2076c513dc6898a8e30f2cf0fafc4486d8f7c3",
"size": 660
},
{
"rfilename": "lora/seed0/adapter.safetensors",
"blobId": "c3d80937be6f8262769b08a53f5e19548c516420",
"size": 18932336,
"lfs": {
"sha256": "93a3f43a4356f84da0d13ecc622fee13e41ba15d237cdaad6094f243899c62d6",
"size": 18932336,
"pointerSize": 133
}
},
{
"rfilename": "lora/seed0/adapter_meta.json",
"blobId": "c67a9c7f3db4db8f73cc71d27d4a8a664967b624",
"size": 573
},
{
"rfilename": "lora/seed1/adapter.safetensors",
"blobId": "5f0b88be4fa8d1a7fd586ecc640b7ac73fb8a68d",
"size": 18932336,
"lfs": {
"sha256": "63d54e1c2f6bf7d07ca0f6e83a3299bd4b6d00740919bc7cae0b422af05a828e",
"size": 18932336,
"pointerSize": 133
}
},
{
"rfilename": "lora/seed1/adapter_meta.json",
"blobId": "2f1ebca329c8579362d80f0e9e731b832a139349",
"size": 573
},
{
"rfilename": "lora/seed2/adapter.safetensors",
"blobId": "d66df32e3e1e67c4c6b3eccf6229f223fa61a5f8",
"size": 18932336,
"lfs": {
"sha256": "702b764e2c39b3529914dac1547148ef2f462d9ee04e256201e79f00b20ccc62",
"size": 18932336,
"pointerSize": 133
}
},
{
"rfilename": "lora/seed2/adapter_meta.json",
"blobId": "7e4737b99685c9f9542f7d15ad4efebc45fc26f3",
"size": 573
},
{
"rfilename": "lora/seed3/adapter.safetensors",
"blobId": "abb761e518f2bcd7285c7c2cb11579943b942d55",
"size": 18932336,
"lfs": {
"sha256": "7dbf35899fe1186453e599ad8c7a85c56b7fad3d630e371ca026a1e95da2c5a7",
"size": 18932336,
"pointerSize": 133
}
},
{
"rfilename": "lora/seed3/adapter_meta.json",
"blobId": "655fd5600b67568bc9da02e4480a14483e0a1067",
"size": 573
},
{
"rfilename": "lora/seed4/adapter.safetensors",
"blobId": "8bd506ae2220c7d0b025d69a4fbeb9ddaf37198d",
"size": 18932336,
"lfs": {
"sha256": "a5cb1e7fa74dff467bd7f163e1db23d30ef8b1a1c935cfa55dcc8cd1b39082a7",
"size": 18932336,
"pointerSize": 133
}
},
{
"rfilename": "lora/seed4/adapter_meta.json",
"blobId": "d8d7b0df8c3d7d1e2f5ef2f2f6e02e22478e2055",
"size": 573
},
{
"rfilename": "lora/seed5/adapter.safetensors",
"blobId": "487c771e11d8e7ca1fa5bd2670d060e8f77fbaee",
"size": 18932336,
"lfs": {
"sha256": "d513d499dfd409da88b3db7da025ba9872fb48ec92d6b29bc2ce7a092a69745d",
"size": 18932336,
"pointerSize": 133
}
},
{
"rfilename": "lora/seed5/adapter_meta.json",
"blobId": "26fc54919eeaffc92a453ddd31cec660c2ab7f0b",
"size": 573
},
{
"rfilename": "warmup/seed0/adapter.safetensors",
"blobId": "378bc7e4f70afb2e947fc6f53ae822dc4bc422d2",
"size": 18932344,
"lfs": {
"sha256": "9108bce2acfcc3ccd38d0dfae67ac33253b95bdc94133bd25deb4daa45d41700",
"size": 18932344,
"pointerSize": 133
}
},
{
"rfilename": "warmup/seed0/adapter_meta.json",
"blobId": "20989af8c7f2ff5959625878167aca5e0fc7203a",
"size": 576
}
],
"spaces": [],
"createdAt": "2026-10-12T10:25:44.000Z",
"usedStorage": 388053976
}
@@ -0,0 +1,22 @@
{
"record": "qwen3.5-2b-grpo",
"arm": "lora",
"receipt_arm": "ctrl",
"seed": 0,
"step": 150,
"base_model": "Qwen/Qwen3.5-2B",
"base_revision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"recipe": "GRPO, correctness-gated length penalty alpha 0.2, 150 steps from the shared warm-up (paper Section 4)",
"lora_rank": 16,
"lora_alpha": 32,
"lora_targets": [
"in_proj_qkv",
"in_proj_a",
"in_proj_b",
"out_proj"
],
"gate": null,
"n_tensors": 144,
"n_params": 4727808,
"source_sha256": "de2794c085f5096fea53374ba1cea31075071bb6de3f227bb3350b3035970447"
}
+47
View File
@@ -0,0 +1,47 @@
"""Read-only FP32 effective weights for an existing pure-LoRA checkpoint."""
import re
import numpy as np
from functional.model import PREFIX,StreamingLM
from pilot.tensors import load_tensors
BASE_REVISION='15852e8c16360a2fea060d615a32b45270f8a8fc'
TARGETS={'in_proj_qkv','in_proj_a','in_proj_b','out_proj'}
class EffectiveWeight:
def __init__(self,base,a,b,scale):
assert a.ndim==b.ndim==2 and (b.shape[0],a.shape[1])==base.shape and a.shape[0]==b.shape[1]
self.base,self.a,self.b,self.scale=base,a,b,scale
self.shape,self.numel,self.dtype=base.shape,base.numel,'F32'
def read(self,selector=slice(None)):
if not isinstance(selector,slice):raise ValueError('Only contiguous row chunks supported')
start,stop,step=selector.indices(self.numel);dim=self.shape[1]
if step!=1 or start%dim or stop%dim:raise ValueError('Row aligned chunks required')
w=self.base.read(slice(start,stop)).reshape(-1,dim)
return (w+self.scale*(self.b[start//dim:stop//dim]@self.a)).astype(np.float32).reshape(-1)
def evict(self):self.base.evict()
def install(lm,directory,meta,multiplier=1.):
assert meta['base_model']=='Qwen/Qwen3.5-2B' and meta['base_revision']==BASE_REVISION and meta['gate'] is None and meta['arm']=='lora'
tensors=load_tensors(directory);pairs={};pattern=re.compile(r'^base_model\.model\.model\.(layers\.(\d+)\.linear_attn\.(in_proj_qkv|in_proj_a|in_proj_b|out_proj))\.lora_([AB])\.default\.weight$')
for name,tensor in tensors.items():
match=pattern.fullmatch(name)
if not match:raise ValueError('Unexpected trainable tensor: '+name)
target=PREFIX+match[1]+'.weight';assert lm.config['layer_types'][int(match[2])]=='linear_attention'
assert target in lm.tensors
value=tensor.read().reshape(tensor.shape).copy();tensor.evict();assert np.isfinite(value).all()
pairs.setdefault(target,{})[match[4]]=value
expected={PREFIX+f'layers.{layer}.linear_attn.{role}.weight' for layer,kind in enumerate(lm.config['layer_types']) if kind=='linear_attention' for role in TARGETS}
assert set(pairs)==expected and len(tensors)==meta['n_tensors']
scale=meta['lora_alpha']/meta['lora_rank']*multiplier
for target,pair in pairs.items():
assert set(pair)=={'A','B'} and pair['A'].shape[0]==meta['lora_rank']
lm.tensors[target]=EffectiveWeight(lm.tensors[target],pair['A'],pair['B'],scale)
return pairs,scale
class ExplicitLoRA(StreamingLM):
"""Independent reference: base projection + scaled low-rank output, no W merge."""
def linear(self,x,name,rows=256):
y=super().linear(x,name,rows);pair=self.pairs.get(PREFIX+name+'.weight')
if pair is not None:y=y+self.scale*((x@pair['A'].T)@pair['B'].T)
return y.astype(np.float32)
+73
View File
@@ -0,0 +1,73 @@
"""Paired concentration changes from saved immutable intervention arrays."""
import csv,json,sys
from pathlib import Path
import numpy as np
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
from dominance.core import concentration
def stats(x):
x=np.asarray(x,dtype=np.float64);v=x[np.isfinite(x)];q=np.quantile(v,[.1,.5,.9])
return {'count':int(v.size),'p10':float(q[0]),'median':float(q[1]),'p90':float(q[2]),'mean':float(v.mean())}
def csvwrite(path,rows):
with path.open('w') as f:w=csv.DictWriter(f,fieldnames=list(rows[0]));w.writeheader();w.writerows(rows)
def main(root):
root=Path(root);status=json.loads((root/'status.json').read_text());assert status['state']=='completed';samples=json.loads((root/'samples.json').read_text());pool={};observations={};summary={};rows=[];layer_rows=[]
for cohort in ('legacy','gsm8k'):
for role in ('base','finetuned'):
for sample in samples[cohort]:
for source in (2,6,10,14,18,22):
data=np.load(root/'raw'/f'{cohort}-{role}-{sample["id"]}-{source:02d}.npz')
for metric,key,transform in [('distance','JS',np.sqrt),('divergence','JS',lambda x:x),('QKV','QKV',lambda x:x)]:
impacts=transform(data['single_'+key])[:,7:,:].transpose(1,2,0);c=concentration(impacts)
record={'cohort':cohort,'role':role,'sample':sample['id'],'recipient':source+1,'metric':metric}
for name,x in c.items():
observations.setdefault((cohort,role,metric,name),[]).append(x.reshape(-1))
if name!='valid':
for stat,value in stats(x).items():record[name+'_'+stat]=value
pool.setdefault((cohort,role,metric,'recipient'),[]).append(np.full(200,source+1))
pool.setdefault((cohort,role,metric,'sample'),[]).append(np.full(200,sample['id']))
observations.setdefault((cohort,role,metric,'winner'),[]).append(np.argmax(impacts,-1).reshape(-1))
rows.append(record)
arrays={k:np.concatenate(v) for k,v in observations.items()};np.savez_compressed(root/'observations.npz',**{'__'.join(k):v for k,v in arrays.items()})
for cohort in ('legacy','gsm8k'):
summary[cohort]={}
for metric in ('distance','divergence','QKV'):
item={role:{name:stats(arrays[(cohort,role,metric,name)]) for name in ('effective_heads','effective_fraction','concentration','top1','top25','bottom25','heads90','total')} for role in ('base','finetuned')}
before=arrays[(cohort,'base',metric,'effective_heads')];after=arrays[(cohort,'finetuned',metric,'effective_heads')];delta=after-before
item['paired']={'effective_heads_delta':stats(delta),'increased_fraction':float(np.mean(delta>0)),'decreased_fraction':float(np.mean(delta<0)),'winner_agreement':float(np.mean(arrays[(cohort,'base',metric,'winner')]==arrays[(cohort,'finetuned',metric,'winner')]))}
summary[cohort][metric]=item
recipients=np.concatenate(pool[(cohort,'base',metric,'recipient')]);names=np.concatenate(pool[(cohort,'base',metric,'sample')])
for layer in (3,7,11,15,19,23):
which=recipients==layer;a=stats(before[which]);b=stats(after[which]);d=stats(delta[which]);layer_rows.append({'cohort':cohort,'metric':metric,'recipient':layer,'base_median':a['median'],'base_p10':a['p10'],'base_p90':a['p90'],'finetuned_median':b['median'],'finetuned_p10':b['p10'],'finetuned_p90':b['p90'],'paired_delta_median':d['median']})
for sample in samples[cohort]:
which=names==sample['id'];item.setdefault('per_text',{})[sample['id']]={'base':stats(before[which]),'finetuned':stats(after[which]),'paired_delta':stats(delta[which])}
expected=(8.93182211792415,5.7840811612328125,11.67265834734814);b=summary['legacy']['distance']['base']['effective_heads'];np.testing.assert_allclose([b[k] for k in ['median','p10','p90']],expected,rtol=1e-10)
(root/'summary.json').write_text(json.dumps(summary,indent=2)+'\n');csvwrite(root/'per-text-layer.csv',rows);csvwrite(root/'per-layer.csv',layer_rows)
figs=root/'figures';figs.mkdir(exist_ok=True)
def save(fig,name):
for ext in ('png','pdf','svg'):fig.savefig(figs/(name+'.'+ext),dpi=180)
plt.close(fig)
fig,axs=plt.subplots(1,2,figsize=(13,4.8),layout='constrained')
for ax,cohort in zip(axs,['legacy','gsm8k']):
for role,label,color in [('base','Original release','C0'),('finetuned','GSM8K GRPO LoRA seed0','C1')]:
x=np.sort(arrays[(cohort,role,'distance','effective_heads')]);ax.plot(x,np.arange(1,len(x)+1)/len(x),label=label,color=color)
ax.set(title='Original four texts' if cohort=='legacy' else 'Four GSM8K held-out question prefixes',xlabel='Effective donor heads: sqrt(JS) influence',ylabel='Empirical CDF',xlim=(1,16),ylim=(0,1));ax.grid(alpha=.2);ax.legend()
fig.suptitle('Same inputs before/after existing task fine-tuning; 32 tokens; one training seed; no training here');save(fig,'paired_effective_heads_cdf')
fig,axs=plt.subplots(1,2,figsize=(13,4.8),layout='constrained')
for ax,cohort in zip(axs,['legacy','gsm8k']):
for role,prefix,color in [('base','base','C0'),('finetuned','finetuned','C1')]:
rr=[r for r in layer_rows if r['cohort']==cohort and r['metric']=='distance'];x=[r['recipient'] for r in rr];med=[r[prefix+'_median'] for r in rr];lo=[r[prefix+'_p10'] for r in rr];hi=[r[prefix+'_p90'] for r in rr];ax.plot(x,med,'o-',color=color,label=role);ax.fill_between(x,lo,hi,color=color,alpha=.12)
ax.set(title=cohort,xlabel='Full-attention recipient layer index',ylabel='Effective donor heads: median / P10–P90',ylim=(1,16));ax.legend();ax.grid(alpha=.2)
fig.suptitle('All six full-attention receivers, sixteen donor heads each; descriptive dependent observations');save(fig,'paired_effective_heads_by_layer')
fig,axs=plt.subplots(1,2,figsize=(13,4.8),layout='constrained')
for ax,cohort in zip(axs,['legacy','gsm8k']):
delta=arrays[(cohort,'finetuned','distance','effective_heads')]-arrays[(cohort,'base','distance','effective_heads')];ax.hist(delta,bins=60,density=True,alpha=.7);ax.axvline(0,color='black',linestyle='--');ax.axvline(np.median(delta),color='C3',label='Paired median');ax.set(title=cohort,xlabel='Finetuned minus original effective head count',ylabel='Density');ax.legend()
fig.suptitle('Paired changes; positive = more distributed, negative = more concentrated');save(fig,'paired_effective_heads_delta')
lines=['# GSM8K任务微调前后的有效头数','', '比较原Qwen3.5-2B发布权重与同一revision上的纯LoRA seed0(额外GSM8K GRPO,非SFT)。没有训练新模型。主指标为√JS距离消融影响的N_eff;正变化表示更分散,负变化表示更集中。','', '| 输入组 | 原权重中位数 [P10,P90] | 微调后中位数 [P10,P90] | 配对变化中位数 | 增加的观测比例 |','|---|---:|---:|---:|---:|']
for cohort,label in [('legacy','原四条文本'),('gsm8k','GSM8K测试题前缀')]:
item=summary[cohort]['distance'];a=item['base']['effective_heads'];b=item['finetuned']['effective_heads'];d=item['paired']['effective_heads_delta'];lines.append(f"| {label} | {a['median']:.6f} [{a['p10']:.6f},{a['p90']:.6f}] | {b['median']:.6f} [{b['p10']:.6f},{b['p90']:.6f}] | {d['median']:+.6f} | {item['paired']['increased_fraction']:.1%} |")
lines+=['', '每组4条输入×6个接收层×25个query位置×8个接收query头=4800个配对观测;它们不是4800个独立文本样本。前后使用同一原模型tokenizer、完全相同token IDs。原四条输入的基线8.931822/16已精确复现。','', 'GSM8K输入取原作者固定版本test split中前4道至少32token的题目,仅用题目前32token,不含答案,不套不同聊天模板。另算了同组GSM8K输入的原权重基线;没有把旧通用文本与新任务文本混作配对。','', '## 方法与限定','', '所有6个普通注意力接收层和16个上游输出头均保留;每次移除一个上游头在全序列的投影输出,重算中间MLP/残差/归一化和检索分布。微调只改变18个GDN层的指定投影,普通注意力权重不变;隐藏状态和上游贡献仍会变化。','', '采用read-only FP32有效权重W+2BA,未把增量舍入为BF16、未修改底座;另用base projection+2B(Ax)独立前向校验。头输出求和、空消融和源数组复用校验均记录。JS散度及QKV复合相对变化也保留,避免只展示有利度量。这里QKV补充指标仅覆盖相同六个普通注意力接收层,其原基线不能与旧研究全部23连接的76.0%混作同一统计量。','', '单个发布者训练seed、4条短文本/组,仅支持这些输入上的描述性变化;不推广为所有微调都会如此,不报告GSM8K准确率,不据此推断删头可用或最终LM能力。配对变化中位数与两个总体中位数之差不是同一个统计量。','', '[公开微调权重与训练说明](https://huggingface.co/Avifenesh/gdn-beta-gate-qwen3.5-2b-grpo);固定版本、参数SHA、dataset版本和题目索引见protocol.json/gsm8k-source.json/samples.json。所有权重、适配器和缓存继续保留。']
(root/'REPORT.md').write_text('\n'.join(lines)+'\n');print((root/'REPORT.md').read_text(),end='')
if __name__=='__main__':main(sys.argv[1])
+72
View File
@@ -0,0 +1,72 @@
"""Paired CPU-only existing-adapter comparison for the published sqrt(JS) metric."""
import argparse,json,hashlib,time,resource,gc
from pathlib import Path
import numpy as np
from tokenizers import Tokenizer
from dominance.core import InfluenceLM,concentration
from dominance.run import distances
from .lora import install,ExplicitLoRA,BASE_REVISION
SOURCES=(2,6,10,14,18,22)
def dump(p,d):p.write_text(json.dumps(d,indent=2,allow_nan=False)+'\n')
def single(lm,cache,layer,path):
base=lm.recipient(cache['next_raw'],layer+1);hs=cache['heads'].shape[2];t=cache['raw'].shape[1];assert hs==16
scores={k:[] for k in ('Q','K','V','QKV','JS')}
for start in range(0,hs,4):
masks=np.zeros((4,t,hs),dtype=np.float32)
for n in range(4):masks[n,:,start+n]=1
effects=distances(base,lm.intervene(cache,layer,masks))
for k,v in effects.items():scores[k].extend(v)
null=distances(base,lm.intervene(cache,layer,np.zeros((1,t,hs),dtype=np.float32)))
assert max(float(np.max(v)) for v in null.values())<2e-5
payload={'single_'+k:np.asarray(v,dtype=np.float32) for k,v in scores.items()};payload.update({'null_'+k:v for k,v in null.items()});np.savez_compressed(path,**payload)
def main():
p=argparse.ArgumentParser();p.add_argument('--source',type=Path,required=True);p.add_argument('--parent',type=Path,required=True);p.add_argument('--assets',type=Path,required=True);p.add_argument('--out',type=Path,required=True);a=p.parse_args();a.out.mkdir(parents=True,exist_ok=False);(a.out/'cache').mkdir();(a.out/'raw').mkdir()
started=time.monotonic();meta=json.loads((a.assets/'adapter_meta.json').read_text());assert meta['base_revision']==BASE_REVISION
adapter_sha=hashlib.sha256((a.assets/'adapter.safetensors').read_bytes()).hexdigest();assert adapter_sha=='93a3f43a4356f84da0d13ecc622fee13e41ba15d237cdaad6094f243899c62d6'
tok=Tokenizer.from_file(str(a.source/'tokenizer.json'));legacy=json.loads((a.parent/'samples.json').read_text());gsm=[]
for s in legacy:assert s['token_ids']==tok.encode(s['text']).ids[:32]
for index,line in enumerate((a.assets/'gsm8k-test.jsonl').read_text().splitlines()):
q=json.loads(line)['question'];ids=tok.encode(q).ids
if len(ids)<32:continue
gsm.append({'id':f'gsm8k-test-{index:04d}','row_index':index,'text':q,'question_sha256':hashlib.sha256(q.encode()).hexdigest(),'token_ids':ids[:32]})
if len(gsm)==4:break
assert len(gsm)==4
cohorts={'legacy':legacy,'gsm8k':gsm};dump(a.out/'samples.json',cohorts);dump(a.out/'adapter_meta.json',meta);dump(a.out/'gsm8k-source.json',json.loads((a.assets/'gsm8k-source.json').read_text()))
protocol={'primary':'same six full-attention receivers and original4x32 token cohort as published8.931822/16','adapter_repo':'Avifenesh/gdn-beta-gate-qwen3.5-2b-grpo','adapter_revision':'038d8be9bd0edd3883fa019651b02bdf65609468','adapter_file':'lora/seed0/adapter.safetensors','adapter_sha256':adapter_sha,'base_revision':BASE_REVISION,'scale':2,'rank':16,'arm':'pure LoRA','training':'publisher-reported GSM8K GRPO; we do no training','sample_choice':'four original texts; first4 canonical GSM8K test questions with >=32tokens, length selection only, first32tokens; question only','source_layers':SOURCES,'primary_queries':list(range(7,32)),'main_metric':'concentration(sqrt(single_JS)); same FP32 tensors and sequence-wide donor-head deletion as prior run','cohort_pairing':'each comparison uses same question/tokenIDs before and after; base legacy arrays reused with elementwise checks','runtime':'read-only FP32 effective W+2BA; no BF16 re-rounding or quantization','controls':'head sum, null mask, explicit low-rank versus effective-weight full forward relL2<1e-4','budget_seconds':7200,'keep_all_weights_and_caches':True,'limitations':['one existing training seed','4 texts per cohort, 32 tokens each','dependent query/head observations, not iid textual samples','no generation or accuracy measurement','GDN targets only; ordinary attention weights stay unchanged','no pruning or functional interchangeability claim']}
dump(a.out/'protocol.json',protocol);status={'state':'running','links':[]};dump(a.out/'status.json',status);cfg=json.loads((a.source/'model_config.json').read_text());controls=[];references=[]
try:
for cohort,samples in cohorts.items():
for role in ('base','finetuned'):
reuse=cohort=='legacy' and role=='base';lm=None
if not reuse:
lm=InfluenceLM(a.source/'weights',cfg,seconds=max(1,7200-(time.monotonic()-started)))
if role=='finetuned':pairs,scale=install(lm,a.assets,meta)
for si,s in enumerate(samples):
if time.monotonic()-started>7200:raise TimeoutError('Paired CPU budget')
if not reuse:
cache=a.out/'cache'/f'{cohort}-{role}-{s["id"]}';cache.mkdir();ids=np.array([s['token_ids']],dtype=np.int64);final=lm.baseline(ids,cache);np.save(cache/'final.npy',final)
controls.extend({'cohort':cohort,'role':role,'sample':s['id'],**v} for v in lm.head_sum_errors)
if cohort=='legacy' and role=='finetuned' and si==0:
ref=ExplicitLoRA(a.source/'weights',cfg,seconds=max(1,7200-(time.monotonic()-started)));ref.pairs,ref.scale=pairs,scale;expected,_=ref.forward(ids);actual=lm.normalization(final,'norm');error=float(np.linalg.norm(actual.astype(np.float64)-expected)/np.linalg.norm(expected.astype(np.float64)));assert error<1e-4
status['explicit_lora_forward_relative_error']=error;del ref;gc.collect()
for layer in SOURCES:
out=a.out/'raw'/f'{cohort}-{role}-{s["id"]}-{layer:02d}.npz'
if reuse:
old=a.parent/'raw'/f'{s["id"]}-link-{layer:02d}.npz'
with np.load(old) as d:payload={prefix+k:d[prefix+k] for prefix in ('single_','null_') for k in ('Q','K','V','QKV','JS')}
np.savez_compressed(out,**payload)
with np.load(out) as d:
for k,v in payload.items():np.testing.assert_array_equal(d[k],v)
references.append({'source':str(old.resolve()),'sha256':hashlib.sha256(old.read_bytes()).hexdigest(),'derived':out.name})
else:
with np.load(cache/f'layer-{layer}.npz') as d:loaded={k:d[k] for k in d.files}
single(lm,loaded,layer,out);del loaded
status['links'].append({'cohort':cohort,'role':role,'sample':s['id'],'source':layer,'path':str(out.relative_to(a.out))});status['elapsed_seconds']=time.monotonic()-started;dump(a.out/'status.json',status);print('completed',cohort,role,s['id'],layer,flush=True)
dump(a.out/'head-sum-controls.json',controls);dump(a.out/'baseline-references.json',references)
if lm is not None:del lm;gc.collect()
status.update(state='completed',elapsed_seconds=time.monotonic()-started,peak_rss_kib=resource.getrusage(resource.RUSAGE_SELF).ru_maxrss);dump(a.out/'status.json',status)
except BaseException as e:
status.update(state='failed',error=repr(e),elapsed_seconds=time.monotonic()-started);dump(a.out/'status.json',status);raise
if __name__=='__main__':main()