4.8 KiB
Head influence concentration: measurement protocol
Target: quantify the statistical distribution of large and small upstream head influences on the next layer's Q/K/V; identity of dominant heads may vary at each sample, query token and recipient head. No training, GPU, deletion or model edits.
Scope: all 23 adjacent layer transitions of the retained Qwen3.5-2B text model, including 16-head linear attention and 8-head full attention. Measure effective Q/K/V, and retrieval probability changes where the recipient uses full attention. The intervening source-layer residual/MLP and recipient input normalization must be recomputed after interventions. Residual stream is preserved: removing an attention head is not removing all information reaching the recipient.
For each observation, remove one upstream attention head's projected contribution at a time. Record nonnegative influence magnitude d_i and its absolute scale. When sum(d_i)>0, p_i=d_i/sum(d_i) is normalized intervention influence, NOT an additive causal attribution or a native attention probability over upstream heads. All-zero influence has undefined concentration and must be flagged, not assigned uniform or concentrated values. Record small total influences explicitly.
Primary continuous measures:
- Effective head count: N_eff=1/sum(p_i^2), range [1,H].
- Effective head fraction: N_eff/H, supports different H.
- Normalized concentration: C=(sum(p_i^2)-1/H)/(1-1/H), range [0,1]. These are related summaries, not independent evidence. Report distributions over texts/tokens/recipient heads/layers, not only concentration of globally averaged influences. Primary query positions 7 onward avoid trivial short-prefix attention.
Secondary intuitive measures: top1/top25%/top50% influence share, bottom25% share, and number of heads reaching 90% influence. Preserve raw influence tables. No RMS activity threshold is used to define influence concentration.
Joint interventions are required: per observation's weak-head set and strong-head set must be removed together and measured, because individual ablations are not additive and may conceal cancellation or synergy. Rankings may be input-specific; these experiments cannot imply a universally removable fixed head set. Distinguish local next-layer QKV/probability effects from unmeasured final LM quality.
CPU measurements, scientific plots, tests and audits execute on the previously authorized server 8.211.154.134. All weights and existing caches remain retained.
Fixed execution details (before real-model measurements)
Four texts are reused from crossqkv/fresh_samples.json (two Chinese/two English), 32 tokens each; no independent generalization claim. All tokens are computed, queries7..31 are primary statistical observations. The observer caches source residual inputs and exact per-head projected output components; their sum is checked against the native branch output. First text's full forward is checked against the canonical streaming forward, and every transition has a null-mask control. No cached tensor is deleted.
For each query/recipient query-head, individual influence is the L2 change divided by that same baseline vector's L2 (floor1e-12). Q,K,V are analyzed separately. A declared balanced score sqrt((relative_Q^2+relative_K^2+relative_V^2)/3) is also analyzed; it does not treat their different physical scales as directly additive. Full attention: Q/K are measured after learned norm/partial RoPE, V before output mixing, and actual softmax retrieval JS is measured. Linear attention: Q/K/V are after causal convolution+SiLU; Q/K include the delta-rule L2 normalization and Q scale. It has no ordinary softmax retrieval distribution. Full-attention GQA K/V are repeated to query-head slots for aligned joint tests; these slots are not independent KV heads, and statistical observations are not independent samples.
Joint weak-half/strong-half/random-half/all-head removal uses a separate coherent sequence for each recipient query-head. At each source token, choose the donor half by that recipient query-head's QKV influence ranking at the same token. Recompute full recipient sequence; convolution and QK retrieval can therefore propagate changes between positions. Only the intended recipient head is scored for each scenario. This is an input-adaptive diagnostic, not a computationally free head-selection policy. Random uses fixed seed31000+sample_index*100+layer. Compare joint effects to sums of corresponding single effects but do not assume additivity. For JS use the same joint sets (selected by QKV, not by JS).
No success criterion assumes that dominance exists. Report effective-head and absolute-effect distributions even if contributions are diffuse. No final-LM NLL or deployable pruning quality claim. Scientific output includes all raw single/joint effects and deletion masks; large baseline caches remain external.