Quantify adjacent-layer head influence concentration on CPU

This commit is contained in:
2026-11-10 22:38:15 +08:00
parent cd260a2483
commit 098ea8650c
136 changed files with 24841 additions and 0 deletions
+76
View File
@@ -0,0 +1,76 @@
# Head influence concentration: measurement protocol
Target: quantify the statistical distribution of large and small upstream head
influences on the next layer's Q/K/V; identity of dominant heads may vary at each
sample, query token and recipient head. No training, GPU, deletion or model edits.
Scope: all 23 adjacent layer transitions of the retained Qwen3.5-2B text model,
including 16-head linear attention and 8-head full attention. Measure effective
Q/K/V, and retrieval probability changes where the recipient uses full attention.
The intervening source-layer residual/MLP and recipient input normalization must
be recomputed after interventions. Residual stream is preserved: removing an
attention head is not removing all information reaching the recipient.
For each observation, remove one upstream attention head's projected contribution
at a time. Record nonnegative influence magnitude d_i and its absolute scale.
When sum(d_i)>0, p_i=d_i/sum(d_i) is normalized intervention influence, NOT an
additive causal attribution or a native attention probability over upstream heads.
All-zero influence has undefined concentration and must be flagged, not assigned
uniform or concentrated values. Record small total influences explicitly.
Primary continuous measures:
- Effective head count: N_eff=1/sum(p_i^2), range [1,H].
- Effective head fraction: N_eff/H, supports different H.
- Normalized concentration: C=(sum(p_i^2)-1/H)/(1-1/H), range [0,1].
These are related summaries, not independent evidence. Report distributions over
texts/tokens/recipient heads/layers, not only concentration of globally averaged
influences. Primary query positions 7 onward avoid trivial short-prefix attention.
Secondary intuitive measures: top1/top25%/top50% influence share, bottom25%
share, and number of heads reaching 90% influence. Preserve raw influence tables.
No RMS activity threshold is used to define influence concentration.
Joint interventions are required: per observation's weak-head set and strong-head
set must be removed together and measured, because individual ablations are not
additive and may conceal cancellation or synergy. Rankings may be input-specific;
these experiments cannot imply a universally removable fixed head set. Distinguish
local next-layer QKV/probability effects from unmeasured final LM quality.
CPU measurements, scientific plots, tests and audits execute on the previously
authorized server 8.211.154.134. All weights and existing caches remain retained.
## Fixed execution details (before real-model measurements)
Four texts are reused from crossqkv/fresh_samples.json (two Chinese/two English),
32 tokens each; no independent generalization claim. All tokens are computed,
queries7..31 are primary statistical observations. The observer caches source
residual inputs and exact per-head projected output components; their sum is
checked against the native branch output. First text's full forward is checked
against the canonical streaming forward, and every transition has a null-mask
control. No cached tensor is deleted.
For each query/recipient query-head, individual influence is the L2 change divided
by that same baseline vector's L2 (floor1e-12). Q,K,V are analyzed separately.
A declared balanced score sqrt((relative_Q^2+relative_K^2+relative_V^2)/3) is also
analyzed; it does not treat their different physical scales as directly additive.
Full attention: Q/K are measured after learned norm/partial RoPE, V before output
mixing, and actual softmax retrieval JS is measured. Linear attention: Q/K/V are
after causal convolution+SiLU; Q/K include the delta-rule L2 normalization and Q
scale. It has no ordinary softmax retrieval distribution. Full-attention GQA K/V
are repeated to query-head slots for aligned joint tests; these slots are not
independent KV heads, and statistical observations are not independent samples.
Joint weak-half/strong-half/random-half/all-head removal uses a separate coherent
sequence for each recipient query-head. At each source token, choose the donor
half by that recipient query-head's QKV influence ranking at the same token.
Recompute full recipient sequence; convolution and QK retrieval can therefore
propagate changes between positions. Only the intended recipient head is scored
for each scenario. This is an input-adaptive diagnostic, not a computationally
free head-selection policy. Random uses fixed seed31000+sample_index*100+layer.
Compare joint effects to sums of corresponding single effects but do not assume
additivity. For JS use the same joint sets (selected by QKV, not by JS).
No success criterion assumes that dominance exists. Report effective-head and
absolute-effect distributions even if contributions are diffuse. No final-LM
NLL or deployable pruning quality claim. Scientific output includes all raw
single/joint effects and deletion masks; large baseline caches remain external.
+75
View File
@@ -0,0 +1,75 @@
# Statistical head influence dominance (CPU)
This study answers how concentrated the **distribution of measured upstream-head
influences** is, without assuming that dominant head identities are fixed across
texts, positions or recipient heads. Read [registered protocol](PROTOCOL.md).
- `core.py`: normalized effective Q/K/V extraction for both attention types,
source projected head decomposition, exact nonlinear downstream recomputation,
coherent masks, continuous concentration and JS.
- `run.py`: four fixed prior texts, all23 adjacent connections, single-head and
weak/strong/random-half/all-head interventions, per-transition checkpointing,
canonical-forward/null controls, preserved baseline caches.
- `report.py`: full saved-data distributions, per-transition CSV, continuous
effective head plots, influence rank curves, joint tests and Chinese report.
- `audit.py`: independently recomputes concentration/masks from raw effects,
validates scope/null/head sums, and hashes retained caches, source and results.
- `tests/test_dominance.py`: continuous limiting cases, undefined zero-total,
token-dependent masks, JS identity, both canonical mixed-layer paths, and joint
interventions on a small checkpoint with known arrays.
Influence for Q/K/V is relative L2 change after deleting one whole donor attention
head (at all sequence positions), preserving the prior residual and recomputing
source MLP + recipient normalization. Source **output/value heads** are the units:
16 for linear attention,8 for full attention. These are not arbitrary individual
Q/K/V weight blocks. For a recipient query-head at one token, normalize donor
influences to p. Primary continuous summary is N_eff=1/sum(p²), and N_eff/H.
C=(sum(p²)-1/H)/(1-1/H) is a related0..1 concentration score. These are not
independent confirmations. No RMS near-zero threshold or R fitting is used.
Q/K/V are separate primary scales; balanced QKV score is RMS of their three
relative changes. It is a declared summary, not a norm in a single physical space.
Zero total implies undefined concentration, not a uniform distribution. Single-head relative effects before head-share normalization and true joint
effects are retained as well as shares. The saved metric named max_absolute is
the largest unsigned relative-L2 effect, not an unnormalized hidden-space norm.
The total field sums individual relative effects; it is not the measured joint
effect of removing the whole attention branch.
Full-attention recipients additionally have real token-key probability JS changes;
linear attention does not have an ordinary softmax attention probability matrix.
Full-attention K/V repeat their GQA groups to query-head slots for aligned joint
experiments, not as extra independent KV observations. Every headline statistic
is descriptive: query/token/head observations are dependent, and four short prior
texts do not establish universal dominance or independent generalization.
Joint masks vary per source token and recipient head. Recompute a coherent full
sequence for each recipient head, scoring that intended head afterward. Rank by
single QKV influence, remove weakest half/strongest half/a fixed-seed random half,
or all attention heads. This investigates interactions and cross-token effects;
it is not a validated free selector, fixed removable head subset or LM pruning.
Only run model work and scientific plotting on the authorized CPU server:
```
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -B -m unittest tests.test_dominance -v
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -B -m dominance.run --source runs/cpu-qwen35-2b --out runs/NEW_DOMINANCE_DIR --seconds 7200
venv/bin/python -B -m dominance.report runs/NEW_DOMINANCE_DIR
venv/bin/python -B -m dominance.audit runs/NEW_DOMINANCE_DIR runs/cpu-qwen35-2b
```
No training, GPU, new model download or automatic deletion. Raw intervention
arrays and masks are delivered to Git; large baseline caches stay on the server
with path/size/SHA metadata. Existing checkpoints/caches are never removed.
The saved sum of selected individual effects is only a query-wise mask-weighted
reference: individual ablations remove a head at all positions, whereas joint
masks can vary across positions. Their difference cannot isolate pure nonlinear
interaction, since temporal intervention supports also differ. Use the measured
weak/strong/random joint effects directly; do not call this reference an additive
causal decomposition or a Shapley attribution.
Scale sensitivity: after reviewing the first complete summaries, add sqrt(JS)
(Jensen–Shannon distance) alongside primary JS divergence. Divergence is roughly
quadratic for small changes, so shares and effective head counts depend on this
scale. All original primary statistics and raw interventions remain unchanged;
primary-analysis-summary.json preserves the pre-sensitivity aggregate for exact
audit comparison. Retrieval-scale plots and source tables show both definitions.
View File
+90
View File
@@ -0,0 +1,90 @@
"""Recompute saved concentration, masks, scale and scope checks; preserve caches."""
import json,hashlib,csv,sys
from pathlib import Path
import numpy as np
from .core import concentration
def main(root,source):
root=Path(root);source=Path(source);status=json.loads((root/'status.json').read_text());assert status['state']=='completed'
samples=json.loads((root/'samples.json').read_text());assert len(samples)==4 and all(len(s['token_ids'])==32 for s in samples)
expected={(s['id'],i) for s in samples for i in range(23)};actual={(x['sample'],x['source']) for x in status['completed_links']};assert actual==expected and len(status['completed_links'])==92
cfg=json.loads((source/'model_config.json').read_text())['text_config'];obs=dict(np.load(root/'observations.npz'));lookup=sorted(s['id'] for s in samples);counts={};undefined={};full_js_links=0;joint_values={}
for sample,layer in sorted(actual):
f=root/'raw'/f'{sample}-link-{layer:02d}.npz';d=dict(np.load(f));hs=8 if cfg['layer_types'][layer]=='full_attention' else 16;hr=8 if cfg['layer_types'][layer+1]=='full_attention' else 16
if hr==8:
full_js_links+=1;d['single_sqrtJS']=np.sqrt(d['single_JS'])
for label in ('weak_half','strong_half','random_half','all_heads'):
d['joint_'+label+'_sqrtJS']=np.sqrt(d['joint_'+label+'_JS'])
mask=d.get('mask_'+label,np.ones((hr,32,hs),dtype=np.float32))
d['individual_sum_'+label+'_sqrtJS']=np.sum(d['single_sqrtJS'].transpose(2,1,0)*mask,-1).T
d['null_sqrtJS']=np.sqrt(d['null_JS'])
masks={label:d['mask_'+label] for label in ('weak_half','strong_half','random_half')}
for label,m in masks.items():
assert m.shape==(hr,32,hs);assert np.isin(m,[0,1]).all();np.testing.assert_array_equal(m.sum(-1),hs//2)
np.testing.assert_array_equal(masks['weak_half']+masks['strong_half'],1)
ranked=np.argsort(d['single_QKV'].transpose(2,1,0),axis=-1,kind='stable')
np.testing.assert_array_equal(np.take_along_axis(masks['weak_half'],ranked[...,:hs//2],-1),1)
for key in ('Q','K','V','QKV')+(('JS','sqrtJS') if hr==8 else ()):
single=d['single_'+key];assert single.shape==(hs,32,hr) and np.isfinite(single).all() and np.min(single)>=0
effect=single[:,7:,:].transpose(1,2,0);stats=concentration(effect);counts[key]=counts.get(key,0)+effect.shape[0]*hr;undefined[key]=undefined.get(key,0)+int((~stats['valid']).sum())
choose=(obs[key+'_source']==layer)&(obs[key+'_sample_index']==lookup.index(sample))
assert int(choose.sum())==25*hr
for metric,values in stats.items():
if metric=='valid':continue
np.testing.assert_allclose(obs[key+'_'+metric][choose],values.ravel(),rtol=1e-6,atol=1e-7,equal_nan=True)
for label in ('weak_half','strong_half','random_half','all_heads'):
mask=masks.get(label,np.ones((hr,32,hs),dtype=np.float32));expected_sum=np.sum(single.transpose(2,1,0)*mask,-1).T
np.testing.assert_allclose(d['individual_sum_'+label+'_'+key],expected_sum,rtol=1e-6,atol=1e-8)
assert d['joint_'+label+'_'+key].shape==(32,hr) and np.isfinite(d['joint_'+label+'_'+key]).all()
joint_values.setdefault((key,label),[]).extend(d['joint_'+label+'_'+key][7:].ravel())
assert float(np.max(d['null_'+key]))<=2e-5
assert full_js_links==24
summary=json.loads((root/'analysis-summary.json').read_text())
def check_stats(values,record):
v=np.asarray(values,dtype=np.float64);v=v[np.isfinite(v)];assert len(v)==record['count']
if len(v):
for name,x in zip(['p10','median','p90','mean'],[*np.quantile(v,[.1,.5,.9]),v.mean()]):np.testing.assert_allclose(record[name],x,rtol=2e-6,atol=1e-8)
for key in counts:
for metric,record in summary['concentration'][key].items():check_stats(obs[key+'_'+metric],record)
primary=json.loads((root/'primary-analysis-summary.json').read_text())
for section in ('concentration','joint','paired_joint','by_recipient_type'):
for key,value in primary[section].items():assert summary[section][key]==value,(section,key)
full_sources=sorted({x['source'] for x in status['completed_links'] if x['full_attention_recipient']})
assert summary['full_attention_source_indices']==full_sources
for key,groups in summary['by_recipient_type'].items():
src=obs[key+'_source']
for label,metrics in groups.items():
choose=np.isin(src,full_sources) if label=='full_attention' else ~np.isin(src,full_sources)
for metric,record in metrics.items():check_stats(obs[key+'_'+metric][choose],record)
for (key,label),values in joint_values.items():check_stats(values,summary['joint'][key][label])
for key in counts:
weak=np.asarray(joint_values[(key,'weak_half')]);strong=np.asarray(joint_values[(key,'strong_half')]);random=np.asarray(joint_values[(key,'random_half')]);valid=weak>0;record=summary['paired_joint'][key]
assert record['zero_weak_effect_count']==int((~valid).sum())
np.testing.assert_allclose(record['weak_less_than_strong_fraction'],np.mean(weak<strong))
np.testing.assert_allclose(record['weak_less_than_random_fraction'],np.mean(weak<random))
check_stats(strong[valid]/weak[valid],record['strong_over_weak']);check_stats(random[valid]/weak[valid],record['random_over_weak'])
controls=json.loads((root/'head-sum-controls.json').read_text());assert len(controls)==96 and max(x['relative_l2'] for x in controls)<2e-5
assert status['canonical_forward_relative_error']<2e-5
caches=[]
for f in sorted((root/'cache').rglob('*')):
if f.is_file():caches.append({'path':str(f.resolve()),'bytes':f.stat().st_size,'sha256':hashlib.sha256(f.read_bytes()).hexdigest()})
weights=list((source/'weights').rglob('*.safetensors'));assert weights and sum(f.stat().st_size for f in weights)==4548221488
manifest=json.loads((source/'manifest.json').read_text());weight_records=[]
for f in weights:
with f.open('rb') as stream:sha=hashlib.file_digest(stream,'sha256').hexdigest()
expected=next(x['upstream_sha256'] for x in manifest['files'] if x['name']==f.name);assert sha==expected
weight_records.append({'path':str(f.resolve()),'bytes':f.stat().st_size,'sha256':sha,'matches_upstream':True})
external={'host':'8.211.154.134','checkpoint_directory':str((source/'weights').resolve()),'checkpoint_bytes':sum(f.stat().st_size for f in weights),'checkpoint_files':weight_records,'revision':'15852e8c16360a2fea060d615a32b45270f8a8fc','baseline_caches':caches,'automatic_deletion':False,'prior_caches_untouched':True}
(root/'external-caches.json').write_text(json.dumps(external,indent=2)+'\n')
reference=json.loads((source/'reference-check.json').read_text())
assert reference['passed'] and all(x['passed'] for x in reference['tests'])
assert hashlib.sha256(Path('functional/model.py').read_bytes()).hexdigest()==reference['implementation_sha256']
(root/'prior-official-reference-check.json').write_bytes((source/'reference-check.json').read_bytes())
files=list(Path('dominance').glob('*.py'))+[Path('dominance/PROTOCOL.md'),Path('dominance/README.md'),Path('tests/test_dominance.py'),Path('functional/model.py'),Path('crossqkv/fresh_samples.json')]
(root/'source-hashes.json').write_text(json.dumps({str(f):hashlib.sha256(f.read_bytes()).hexdigest() for f in files if not f.name.startswith('._')},indent=2)+'\n')
audit={'passed':True,'transitions':23,'texts':4,'link_observations':92,'primary_observations_per_metric':counts,'undefined_zero_total_per_metric':undefined,'full_attention_js_links':full_js_links,'head_sum_controls':96,'canonical_forward_relative_error':status['canonical_forward_relative_error'],'all_raw_concentration_and_masks_recomputed':True,'pooled_concentration_and_joint_summary_recomputed':True,'original_primary_summaries_unchanged':True,'sqrt_JS_scale_sensitivity_recomputed':True,'retained_baseline_cache_files':len(caches),'checkpoint_sha_matches_upstream':True,'prior_official_reference_check_passed_same_implementation':True,'prior_official_reference_tests':len(reference['tests'])}
(root/'audit.json').write_text(json.dumps(audit,indent=2)+'\n');print(json.dumps(audit,indent=2),flush=True)
included=[f for f in root.rglob('*') if f.is_file() and 'cache' not in f.relative_to(root).parts and f.name!='checksums.json']
(root/'checksums.json').write_text(json.dumps({str(f.relative_to(root)):hashlib.sha256(f.read_bytes()).hexdigest() for f in sorted(included)},indent=2)+'\n')
if __name__=='__main__':main(sys.argv[1],sys.argv[2])
+85
View File
@@ -0,0 +1,85 @@
"""Frozen-model head interventions and continuous influence concentration."""
import numpy as np
from functional.model import StreamingLM,PREFIX,rope,causal_conv,attention
def concentration(d):
"""Last axis donor head; zero total is undefined, represented by NaNs."""
d=np.asarray(d,dtype=np.float64)
if not np.isfinite(d).all() or np.any(d<0):raise ValueError('Finite nonnegative influences required')
h=d.shape[-1];total=d.sum(-1);valid=total>0
p=np.divide(d,total[...,None],out=np.zeros_like(d),where=valid[...,None]);square=np.sum(p*p,-1)
neff=np.divide(1,square,out=np.full_like(total,np.nan),where=valid)
ordered=np.sort(p,axis=-1)[...,::-1]
return {'total':total,'max_absolute':d.max(-1),'valid':valid,'effective_heads':neff,'effective_fraction':neff/h,
'concentration':np.where(valid,(square-1/h)/(1-1/h),np.nan),
'top1':np.where(valid,ordered[...,0],np.nan),'top25':np.where(valid,ordered[...,:max(1,h//4)].sum(-1),np.nan),
'top50':np.where(valid,ordered[...,:max(1,h//2)].sum(-1),np.nan),
'bottom25':np.where(valid,ordered[...,-max(1,h//4):].sum(-1),np.nan),
'heads90':np.where(valid,1+np.sum(np.cumsum(ordered,axis=-1)<.9,axis=-1),np.nan)}
def masks_from_influences(d,weak=True):
"""d [donor,T,recipient]; output masks [recipient,T,donor]."""
ranked=np.argsort(d.transpose(2,1,0),axis=-1,kind='stable');n=d.shape[0]//2
selected=ranked[...,:n] if weak else ranked[...,-n:]
mask=np.zeros_like(ranked,dtype=np.float32);np.put_along_axis(mask,selected,1,axis=-1)
return mask
def js_rows(a,b):
a=np.asarray(a,dtype=np.float64);b=np.asarray(b,dtype=np.float64);m=(a+b)/2
def term(x):return np.where(x>0,x*np.log(np.maximum(x,1e-300)/np.maximum(m,1e-300)),0)
return np.maximum(0,.5*np.sum(term(a)+term(b),axis=-1))
class InfluenceLM(StreamingLM):
capturing=False
def linear(self,x,name,rows=256):
y=super().linear(x,name,rows)
if self.capturing and (name.endswith('self_attn.o_proj') or name.endswith('linear_attn.out_proj')):
layer=int(name.split('.')[1]);nh=self.config['num_attention_heads'] if 'self_attn' in name else self.config['linear_num_value_heads']
xx=x.reshape(*x.shape[:2],nh,-1);dim=xx.shape[-1];t=self.tensors[PREFIX+name+'.weight'];od,idim=t.shape
out=np.empty((*x.shape[:2],nh,od),dtype=np.float32)
for start in range(0,od,256):
self.guard();stop=min(od,start+256);w=t.read(slice(start*idim,stop*idim)).reshape(stop-start,idim)
for head in range(nh):out[:,:,head,start:stop]=xx[:,:,head]@w[:,head*dim:(head+1)*dim].T
del w;t.evict()
err=float(np.linalg.norm(out.sum(2).astype(np.float64)-y)/max(np.linalg.norm(y.astype(np.float64)),1e-30))
if err>2e-5:raise AssertionError(('head sum mismatch',layer,err))
self.head_outputs=out;self.head_sum_errors.append({'layer':layer,'relative_l2':err})
return y
def baseline(self,ids,root):
h=self.embedding(ids);self.head_sum_errors=[]
for layer,kind in enumerate(self.config['layer_types']):
raw=h.copy();n=self.normalization(h,f'layers.{layer}.input_layernorm');self.capturing=True
out,_=self.full_attention(n,layer) if kind=='full_attention' else (self.linear_attention(n,layer),None)
self.capturing=False;heads=self.head_outputs;self.head_outputs=None
mid=raw+out;h=mid+self.mlp(self.normalization(mid,f'layers.{layer}.post_attention_layernorm'),layer)
np.savez(root/f'layer-{layer}.npz',raw=raw,out=out,heads=heads,next_raw=h)
print('baseline layer',layer,flush=True)
return h
def recipient(self,raw,layer):
h=self.normalization(raw,f'layers.{layer}.input_layernorm');c=self.config;b,t=h.shape[:2]
if c['layer_types'][layer]=='full_attention':
p=f'layers.{layer}.self_attn.';nh=c['num_attention_heads'];nk=c['num_key_value_heads'];d=c['head_dim']
q=self.linear(h,p+'q_proj').reshape(b,t,nh,2*d)[...,:d];k=self.linear(h,p+'k_proj').reshape(b,t,nk,d);v=self.linear(h,p+'v_proj').reshape(b,t,nk,d)
q=self.normalization(q,p+'q_norm');k=self.normalization(k,p+'k_norm');r=c['rope_parameters'];q,k=(rope(z,r['rope_theta'],r['partial_rotary_factor']) for z in (q,k))
k=np.repeat(k,nh//nk,axis=2);v=np.repeat(v,nh//nk,axis=2)
_,prob=attention(q,k,v)
return {'Q':q,'K':k,'V':v,'P':prob}
p=f'layers.{layer}.linear_attn.';kh=c['linear_num_key_heads'];vh=c['linear_num_value_heads'];kd=c['linear_key_head_dim'];vd=c['linear_value_head_dim']
x=causal_conv(self.linear(h,p+'in_proj_qkv'),self.small(p+'conv1d.weight')[:,0,:]);q,k,v=np.split(x,[kh*kd,2*kh*kd],axis=-1)
q=q.reshape(b,t,kh,kd);k=k.reshape(b,t,kh,kd);v=v.reshape(b,t,vh,vd)
if vh!=kh:q,k=np.repeat(q,vh//kh,axis=2),np.repeat(k,vh//kh,axis=2)
q=q/np.sqrt(np.sum(q*q,axis=-1,keepdims=True)+1e-6)/np.float32(np.sqrt(kd));k=k/np.sqrt(np.sum(k*k,axis=-1,keepdims=True)+1e-6)
return {'Q':q,'K':k,'V':v}
def intervene(self,cache,layer,masks):
# Each scenario has a full coherent token-dependent deletion mask.
removed=np.einsum('bth,thd->btd',masks,cache['heads'][0],optimize=True)
mid=cache['raw']+cache['out']-removed
nxt=mid+self.mlp(self.normalization(mid,f'layers.{layer}.post_attention_layernorm'),layer)
return self.recipient(nxt,layer+1)
+140
View File
@@ -0,0 +1,140 @@
"""Saved-data scientific analysis; no model invocation or fitting."""
import csv,json
from pathlib import Path
import numpy as np
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
from .core import concentration
def stats(x):
x=np.asarray(x,dtype=np.float64);valid=x[np.isfinite(x)]
if not valid.size:return {'count':0,'p10':None,'median':None,'p90':None,'mean':None}
q=np.quantile(valid,[.1,.5,.9]);return {'count':int(valid.size),'p10':float(q[0]),'median':float(q[1]),'p90':float(q[2]),'mean':float(valid.mean())}
def csvwrite(p,rows):
with p.open('w') as f:
w=csv.DictWriter(f,fieldnames=list(rows[0]));w.writeheader();w.writerows(rows)
def main(root):
root=Path(root);figdir=root/'figures';figdir.mkdir(exist_ok=True);status=json.loads((root/'status.json').read_text());assert status['state']=='completed'
rows=[];joint_rows=[];pool={};curves={};joint_pool={};obs={};winner=[];joint_by_layer={}
for entry in status['completed_links']:
sample=entry['sample'];source=entry['source'];data=dict(np.load(root/'raw'/f'{sample}-link-{source:02d}.npz'))
if 'single_JS' in data:
data['single_sqrtJS']=np.sqrt(data['single_JS'])
for label in ('weak_half','strong_half','random_half','all_heads'):
data['joint_'+label+'_sqrtJS']=np.sqrt(data['joint_'+label+'_JS'])
masks=data.get('mask_'+label,np.ones_like(data['mask_weak_half']))
data['individual_sum_'+label+'_sqrtJS']=np.sum(data['single_sqrtJS'].transpose(2,1,0)*masks,-1).T
for key in ('Q','K','V','QKV','JS','sqrtJS'):
if 'single_'+key not in data:continue
d=data['single_'+key][:,7:,:].transpose(1,2,0);c=concentration(d);valid=c['valid'];h=d.shape[-1]
metric={k:v.reshape(-1) for k,v in c.items() if k!='valid'}
row={'sample':sample,'source':source,'recipient':source+1,'metric':key,'source_heads':h,'recipient_attention_type':'full_attention' if entry['full_attention_recipient'] else 'linear_attention','observations':d.shape[0]*d.shape[1],'undefined_zero_total':int((~valid).sum())}
for k,v in metric.items():
pool.setdefault(key,{}).setdefault(k,[]).extend(v)
for s,z in stats(v).items():row[k+'_'+s]=z
rows.append(row)
# Raw per-observation statistics for independently reproducing figures.
for k,v in metric.items():obs.setdefault(key+'_'+k,[]).append(v.astype(np.float32))
n=d.shape[0]*d.shape[1]
for k,v in [('source',np.full(n,source)),('sample_index',np.full(n,sorted({e['sample'] for e in status['completed_links']}).index(sample))),('token',np.repeat(np.arange(7,32),d.shape[1])),('recipient_head',np.tile(np.arange(d.shape[1]),25))]:obs.setdefault(key+'_'+k,[]).append(v.astype(np.int16))
p=np.divide(d,d.sum(-1,keepdims=True),out=np.zeros_like(d),where=d.sum(-1,keepdims=True)>0);cum=np.concatenate([np.zeros((*d.shape[:2],1)),np.cumsum(np.sort(p,-1)[...,::-1],-1)],-1)
grid=np.linspace(0,1,101);cc=np.array([np.interp(grid,np.linspace(0,1,h+1),v) for v in cum[valid]])
curves.setdefault(key,[]).extend(cc)
win=np.argmax(d,-1);counts=np.bincount(win[valid],minlength=h)
winner.append({'sample':sample,'source':source,'metric':key,'most_frequent_winner_fraction':float(counts.max()/max(counts.sum(),1)),'distinct_winning_heads':int((counts>0).sum()),'head_counts':json.dumps(counts.tolist())})
for label in ('weak_half','strong_half','random_half','all_heads'):
values=data['joint_'+label+'_'+key][7:];summed=data['individual_sum_'+label+'_'+key][7:]
jr={'sample':sample,'source':source,'recipient':source+1,'metric':key,'removed':label}
for k,x in [('joint_effect',values),('sum_individual',summed)]:
for s,z in stats(x).items():jr[k+'_'+s]=z
joint_rows.append(jr);joint_pool.setdefault(key,{}).setdefault(label,[]).extend(values.ravel());joint_by_layer.setdefault((key,label,source),[]).extend(values.ravel())
csvwrite(root/'concentration-by-link.csv',rows);csvwrite(root/'joint-by-link.csv',joint_rows);csvwrite(root/'winner-identities.csv',winner)
np.savez_compressed(root/'observations.npz',**{k:np.concatenate(v) for k,v in obs.items()})
pooled={k:{m:stats(v) for m,v in metrics.items()} for k,metrics in pool.items()};joint_stats={k:{m:stats(v) for m,v in metrics.items()} for k,metrics in joint_pool.items()}
paired={}
for key,labels in joint_pool.items():
weak=np.asarray(labels['weak_half']);strong=np.asarray(labels['strong_half']);random=np.asarray(labels['random_half']);valid=weak>0
paired[key]={'weak_less_than_strong_fraction':float(np.mean(weak<strong)),'weak_less_than_random_fraction':float(np.mean(weak<random)),'zero_weak_effect_count':int((~valid).sum()),'strong_over_weak':stats(strong[valid]/weak[valid]),'random_over_weak':stats(random[valid]/weak[valid])}
full_sources=sorted({x['source'] for x in status['completed_links'] if x['full_attention_recipient']})
by_type={}
for key in pool:
src=np.concatenate(obs[key+'_source']);by_type[key]={}
for label,choose in [('full_attention',np.isin(src,full_sources)),('linear_attention',~np.isin(src,full_sources))]:
if not choose.any():continue
by_type[key][label]={m:stats(np.concatenate(obs[key+'_'+m])[choose]) for m in pool[key]}
summary={'scale_sensitivity':'sqrt(JS) added after reviewing completed primary summaries; primary JS and all raw interventions unchanged; squared divergences cannot be equated to probability mass or linear-scale effects','by_recipient_type':by_type,'full_attention_source_indices':full_sources,'paired_joint':paired,'pooling':'query7..31, per text/token/recipient-query-head; descriptive dependent observations, not independent samples','concentration':pooled,'joint':joint_stats,'counts':{'links':len(status['completed_links']),'texts':len(json.loads((root/'samples.json').read_text())),'layer_transitions':23},'sample_index_key':sorted({e['sample'] for e in status['completed_links']}),'winner_identity_file':'winner-identities.csv'}
(root/'analysis-summary.json').write_text(json.dumps(summary,indent=2,allow_nan=False)+'\n')
def save(fig,name):
for ext in ('png','pdf','svg'):fig.savefig(figdir/(name+'.'+ext),dpi=180)
plt.close(fig)
fig,axs=plt.subplots(1,2,figsize=(14,4.8),layout='constrained')
colors={'Q':'C0','K':'C1','V':'C2','QKV':'C3'}
for key in colors:
for ax,metric in zip(axs,['effective_fraction','top1']):
med=[];lo=[];hi=[]
source= np.concatenate(obs[key+'_source']);vals=np.concatenate(obs[key+'_'+metric])
for layer in range(23):
z=stats(vals[source==layer]);lo.append(z['p10']);med.append(z['median']);hi.append(z['p90'])
ax.plot(range(1,24),med,label=key,color=colors[key]);ax.fill_between(range(1,24),lo,hi,color=colors[key],alpha=.10)
axs[0].set(title='Continuous effective-head fraction (median, p10–p90)',ylabel='N_eff / number of donor heads')
axs[1].set(title='Largest donor-head influence share',ylabel='Top-1 normalized ablation effect')
for ax in axs:
for j,source in enumerate(full_sources):ax.axvspan(source+.6,source+1.4,color='gray',alpha=.08,label='Full-attention receiver' if j==0 else None)
ax.set(xlabel='Recipient layer index',ylim=(0,1.02));ax.legend();ax.grid(alpha=.2)
fig.suptitle('All 23 adjacent transitions; four texts, query positions 7–31; no training')
save(fig,'continuous_dominance')
fig,axs=plt.subplots(1,2,figsize=(13,4.8),layout='constrained');curve_rows=[]
for key in colors:
a=np.asarray(curves[key]);lo,med,hi=np.quantile(a,[.1,.5,.9],axis=0);x=np.linspace(0,1,101)
axs[0].plot(x,med,label=key,color=colors[key]);axs[0].fill_between(x,lo,hi,color=colors[key],alpha=.1)
for j,xx in enumerate(x):curve_rows.append({'metric':key,'fraction_ranked_heads':xx,'p10':lo[j],'median':med[j],'p90':hi[j]})
v=np.sort(np.asarray(pool[key]['effective_fraction']));axs[1].plot(v,np.arange(1,len(v)+1)/len(v),label=key,color=colors[key])
axs[0].plot([0,1],[0,1],':',color='gray',label='Uniform');axs[0].set(title='Influence concentration curve',xlabel='Fraction of donor heads, largest first',ylabel='Cumulative normalized single-head effects')
axs[1].set(title='Continuous effective-head fraction: empirical CDF',xlabel='N_eff / donor heads',ylabel='Fraction of observed cases')
for ax in axs:ax.set(xlim=(0,1),ylim=(0,1.02));ax.legend();ax.grid(alpha=.2)
fig.suptitle('Observation-wise ranking; dominant identities can differ; shaded p10–p90')
csvwrite(root/'concentration-curve.csv',curve_rows);save(fig,'ranked_influence_distribution')
fig,axs=plt.subplots(1,2,figsize=(14,4.8),layout='constrained')
for label,color in zip(['weak_half','strong_half','random_half','all_heads'],['C0','C3','C1','gray']):
for ax,key in zip(axs,['QKV','JS']):
selected=[r for r in joint_rows if r['metric']==key and r['removed']==label]
xs=sorted(set(r['recipient'] for r in selected));ys=[np.median(joint_by_layer[(key,label,x-1)]) for x in xs]
ax.plot(xs,np.maximum(ys,1e-12),'o-',label=label,color=color,markersize=4)
axs[0].set(title='Joint removal: balanced QKV relative effect',ylabel='Median token/head relative L2 score')
axs[1].set(title='Full-attention recipients: retrieval change',ylabel='Median token/head JS divergence (nats)')
for ax in axs:ax.set(xlabel='Recipient layer index',yscale='log');ax.legend();ax.grid(alpha=.2)
fig.suptitle('Equal-count weak/strong/random removal; token-wise masks; log display floor 1e-12')
save(fig,'joint_removal_effects')
fig,axs=plt.subplots(1,2,figsize=(13,4.8),layout='constrained')
for key,label,color in [('JS','JS divergence (squared scale)','C4'),('sqrtJS','sqrt(JS) distance','C5')]:
a=np.asarray(curves[key]);lo,med,hi=np.quantile(a,[.1,.5,.9],axis=0);x=np.linspace(0,1,101)
axs[0].plot(x,med,label=label,color=color);axs[0].fill_between(x,lo,hi,color=color,alpha=.12)
for j,xx in enumerate(x):curve_rows.append({'metric':key,'fraction_ranked_heads':xx,'p10':lo[j],'median':med[j],'p90':hi[j]})
v=np.sort(np.asarray(pool[key]['effective_fraction']));axs[1].plot(v,np.arange(1,len(v)+1)/len(v),label=label,color=color)
axs[0].plot([0,1],[0,1],':',color='gray',label='Uniform');axs[0].set(title='Retrieval influence rank curve',xlabel='Fraction of donor heads, largest first',ylabel='Cumulative normalized intervention effects')
axs[1].set(title='Sensitivity to divergence versus distance scale',xlabel='Effective donor-head fraction',ylabel='Fraction of observations')
for ax in axs:ax.set(xlim=(0,1),ylim=(0,1.02));ax.legend();ax.grid(alpha=.2)
fig.suptitle('Full-attention recipients; sqrt(JS) is post-summary scale sensitivity, not a new model run')
csvwrite(root/'concentration-curve.csv',curve_rows);save(fig,'retrieval_scale_sensitivity')
lines=['# 上游注意力头对下一层 Q/K/V 的影响集中度','', '实测全部23个相邻层连接、4条既有中英文文本各32 token。主统计取query7–31,按文本/token/接收query头分别计算,再汇总。不是独立样本的显著性检验。','', '影响是移除上游一个头后,下一层Q/K/V向量的相对L2变化;重新计算中间MLP/残差/归一化。各头影响归一化仅构成消融影响分布,不是天然可相加的贡献概率。','', '| 对象 | 有效头比例中位数 [P10,P90] | 最大头份额中位数 | 前25%头份额中位数 | 最小25%头份额中位数 | 90%影响所需头数中位数 |','|---|---:|---:|---:|---:|---:|']
for k in ('Q','K','V','QKV','JS','sqrtJS'):
if k not in pooled:continue
z=pooled[k];a=z['effective_fraction'];lines.append(f"| {k} | {a['median']:.3f} [{a['p10']:.3f},{a['p90']:.3f}] | {z['top1']['median']:.1%} | {z['top25']['median']:.1%} | {z['bottom25']['median']:.1%} | {z['heads90']['median']:.1f} |")
lines+=['','sqrtJS为JS距离尺度敏感性分析:在初次完整汇总后增加,原始消融和原JS主结果不变。JS散度近似平方尺度,归一化后会放大强弱差距;JS影响份额不是注意力概率质量份额。','', '按接收层类型分层(避免总体分布混合不同机制):','', '| 接收层类型 | Q有效头比例中位数 | K | V | QKV |','|---|---:|---:|---:|---:|']
for label in ('linear_attention','full_attention'):
lines.append('| '+label+' | '+' | '.join(format(by_type[k][label]['effective_fraction']['median'],'.3f') for k in ('Q','K','V','QKV'))+' |')
lines+=['','QKV是三个相对变化平方的均值再开根号,避免混加不同Q/K/V尺度;JS仅普通注意力接收层有。普通注意力GQA的K/V复制到query头索引用于配对,不是额外独立KV头。','', '| 联合移除 | QKV相对影响中位数 [P10,P90] | 检索JS中位数 [P10,P90] |','|---|---:|---:|']
for label in ('weak_half','strong_half','random_half','all_heads'):
a=joint_stats['QKV'][label];b=joint_stats['JS'][label];lines.append(f"| {label} | {a['median']:.5f} [{a['p10']:.5f},{a['p90']:.5f}] | {b['median']:.6f} [{b['p10']:.6f},{b['p90']:.6f}] |")
lines+=['', '配对观测:QKV弱半数的实际联合影响小于强半数的比例为'+format(paired['QKV']['weak_less_than_strong_fraction'],'.1%')+';强/弱影响比中位数为'+format(paired['QKV']['strong_over_weak']['median'],'.3f')+'。这是逐观测比值的中位数,不是两个总体中位数之比。弱效应为零的观测另计,不做除法。', '', '联合实验的弱/强/随机半数头,在各token、各接收头上分别选择,形成完整序列再重算。排名依据单头QKV影响,同样用于JS;它是输入自适应诊断,不是免费可用的剪枝策略。线性注意力的卷积和普通注意力的QK可能传播跨token影响。所有上游头移除仍保留残差背景,不能把它称为删除全部上游信息。', '', '连续指标:N_eff=1/Σp²;有效头比例=N_eff/H;C=(Σp²−1/H)/(1−1/H)。全零总影响时集中度未定义,原始数组保留NaN并在统计时排除、另计数量。单头相对L2影响在做头间占比归一化之前的总和及最大值均保留,避免把微小效应中的集中误认作重要。字段max_absolute指最大无符号相对效应,并非隐藏坐标中的未归一化L2;total是单头效应之和,真正全头联合效果另测。', '', '联合实验中保存的单头影响之和仅为按query掩码加权的参考。单头消融移除整段序列的一个头,联合掩码则可跨位置变化;二者的差不能单独解释成纯非线性交互,因为时间支持也不同。应直接比较实际弱/强/随机联合效果。', '', '## 文件与限制','', '`raw/*.npz`保留每次单头影响、联合影响、相应单头影响之和、删除掩码及空操作对照;`observations.npz`保留逐观测连续指标及索引;CSV和JSON为全部图表源表。原模型权重和大基线缓存继续保留在服务器。','', '只测短文本上的局部下一层作用;不代表全LM质量、可直接删头、所有输入普遍集中,或相似头可互相替代。所有头贡献可能是广泛分布的;报告不以发现稀疏为成功条件。','',f"CPU测量耗时{status['elapsed_seconds']:.1f}秒,峰值RSS{status['peak_rss_kib']/1024:.1f} MiB;无训练、GPU或清理。"]
(root/'REPORT.md').write_text('\n'.join(lines)+'\n')
print((root/'REPORT.md').read_text())
if __name__=='__main__':
import sys
main(sys.argv[1])
+84
View File
@@ -0,0 +1,84 @@
"""All adjacent layers, input-dependent influence distributions; CPU only."""
import argparse,json,time,resource,hashlib,traceback
from pathlib import Path
import numpy as np
from tokenizers import Tokenizer
from functional.model import StreamingLM
from .core import InfluenceLM,concentration,masks_from_influences,js_rows
def dump(path,data):path.write_text(json.dumps(data,indent=2,allow_nan=False)+'\n')
def distances(base,changed):
out={}
for key in ('Q','K','V'):
a=base[key].astype(np.float64);b=changed[key].astype(np.float64)
out[key]=np.linalg.norm(b-a,axis=-1)/np.maximum(np.linalg.norm(a,axis=-1),1e-12)
out['QKV']=np.sqrt(sum(out[k]**2 for k in ('Q','K','V'))/3)
if 'P' in base:out['JS']=js_rows(base['P'],changed['P']).transpose(0,2,1)
return out
def link(lm,cache,layer,path,seed):
started=time.monotonic();base=lm.recipient(cache['next_raw'],layer+1);hs=cache['heads'].shape[2];t=cache['raw'].shape[1];hr=base['Q'].shape[2]
# Whole donor head across the full sequence is the single-ablation intervention.
singles={k:[] for k in ('Q','K','V','QKV')+ (('JS',) if 'P' in base else ())}
for start in range(0,hs,4):
index=list(range(start,min(start+4,hs)));masks=np.zeros((len(index),t,hs),dtype=np.float32)
for n,h in enumerate(index):masks[n,:,h]=1
scores=distances(base,lm.intervene(cache,layer,masks))
for k,v in scores.items():singles[k].extend(v)
singles={k:np.asarray(v,dtype=np.float32) for k,v in singles.items()}
# Identical cardinality weak/strong/random removals, ranking by balanced QKV influence.
# One coherent full-sequence mask per recipient head; only that head's output scored.
rank=singles['QKV'];rng=np.random.default_rng(seed)
weak=masks_from_influences(rank,True);strong=masks_from_influences(rank,False)
random=np.zeros_like(weak)
for r in range(hr):
for q in range(t):random[r,q,rng.choice(hs,hs//2,replace=False)]=1
joint={};sum_individual={}
for label,masks in [('weak_half',weak),('strong_half',strong),('random_half',random),('all_heads',np.ones_like(weak))]:
values={k:[] for k in singles}
for start in range(0,hr,4):
stop=min(start+4,hr);scores=distances(base,lm.intervene(cache,layer,masks[start:stop]))
for k,v in scores.items():
for n,r in enumerate(range(start,stop)):values[k].append(v[n,:,r])
for k,v in values.items():
joint[label+'_'+k]=np.asarray(v,dtype=np.float32).T
sum_individual[label+'_'+k]=np.sum(singles[k].transpose(2,1,0)*masks,axis=-1).T
null=distances(base,lm.intervene(cache,layer,np.zeros((1,t,hs),dtype=np.float32)))
if max(float(np.max(v)) for v in null.values())>2e-5:raise AssertionError('Null intervention changed recipient')
payload={'single_'+k:v for k,v in singles.items()};payload.update({'joint_'+k:v for k,v in joint.items()});payload.update({'individual_sum_'+k:v for k,v in sum_individual.items()})
payload.update({'mask_weak_half':weak,'mask_strong_half':strong,'mask_random_half':random})
payload.update({'null_'+k:v for k,v in null.items()})
np.savez_compressed(path,**payload)
return {'source':layer,'recipient':layer+1,'source_heads':hs,'recipient_query_heads':hr,'full_attention_recipient':'P' in base,'elapsed_seconds':time.monotonic()-started,'null_max':max(float(np.max(v)) for v in null.values())}
def main():
p=argparse.ArgumentParser();p.add_argument('--source',type=Path,required=True);p.add_argument('--out',type=Path,required=True);p.add_argument('--seconds',type=int,default=7200);a=p.parse_args()
a.out.mkdir(parents=True,exist_ok=False);(a.out/'cache').mkdir();(a.out/'raw').mkdir();started=time.monotonic()
protocol={'scope':'all 23 adjacent layer transitions; source attention head contribution to next effective Q/K/V','samples':'4 previously authored zh/en texts, first32 tokens, descriptive','primary_queries':list(range(7,32)), 'influence':'L2 change / baseline token recipient-head L2; QKV score sqrt(mean of three squared relative changes))','linear_QK':'after causal convolution/SiLU and delta-rule L2 normalization; Q additionally divided by sqrt(key dim)','full_QK':'QK norm and partial RoPE applied; K/V repeated to query heads for indexing (not independent KV observations)','single_ablation':'remove complete source head projected output at all tokens; recompute source postattention normalization, MLP, residual and next input normalization','joint_ablation':'weak/strong/random half and all attention heads; one coherent token-varying removal mask for each recipient query head; ranking uses QKV single effects; no fitting','continuous':'effective heads=1/sum(p^2); fraction=effective/H; concentration=(sum(p^2)-1/H)/(1-1/H)','zero_total':'undefined concentration (NaN arrays), count explicitly','cost_budget_seconds':a.seconds,'no_training':True,'keep_all_weights_and_caches':True,'limitations':['not final LM quality or pruning proof','individual effects not additive; normalized scores not true contribution probabilities','token-varying masks are input-specific diagnostic and not a deployable fixed head subset','linear attention does not have an ordinary softmax probability matrix','Q/K/V scales handled separately; combined QKV is a declared balanced relative-effect score']}
dump(a.out/'protocol.json',protocol);status={'state':'running','completed_links':[]};dump(a.out/'status.json',status)
try:
lm=InfluenceLM(a.source/'weights',json.loads((a.source/'model_config.json').read_text()),seconds=a.seconds)
samples=json.loads(Path('crossqkv/fresh_samples.json').read_text())['samples'];tok=Tokenizer.from_file(str(a.source/'tokenizer.json'));controls=[]
for si,s in enumerate(samples):
lm.guard();s['token_ids']=tok.encode(s['text']).ids[:32];ids=np.array([s['token_ids']],dtype=np.int64);cache=a.out/'cache'/s['id'];cache.mkdir()
final=lm.baseline(ids,cache);np.save(cache/'final.npy',final)
for x in lm.head_sum_errors:controls.append({'sample':s['id'],**x})
if si==0:
reference,_=StreamingLM.forward(lm,ids);normalized=lm.normalization(final,'norm');err=float(np.linalg.norm(normalized-reference)/max(np.linalg.norm(reference),1e-30));assert err<2e-5;status['canonical_forward_relative_error']=err
dump(a.out/'head-sum-controls.json',controls);dump(a.out/'samples.json',samples)
for layer in range(23):
target=a.out/'raw'/f'{s["id"]}-link-{layer:02d}.npz'
with np.load(cache/f'layer-{layer}.npz') as data:loaded={k:data[k] for k in data.files}
info=link(lm,loaded,layer,target,31000+si*100+layer)
del loaded
status['completed_links'].append({'sample':s['id'],**info});status['elapsed_seconds']=time.monotonic()-started;dump(a.out/'status.json',status)
print('completed',s['id'],layer,'seconds',round(info['elapsed_seconds'],2),flush=True)
status.update(state='completed',elapsed_seconds=time.monotonic()-started,peak_rss_kib=resource.getrusage(resource.RUSAGE_SELF).ru_maxrss);dump(a.out/'status.json',status)
except BaseException as e:
status.update(state='failed',error=repr(e),elapsed_seconds=time.monotonic()-started);dump(a.out/'status.json',status);raise
if __name__=='__main__':main()