77 lines
4.8 KiB
Markdown
77 lines
4.8 KiB
Markdown
# Head influence concentration: measurement protocol
|
|
|
|
Target: quantify the statistical distribution of large and small upstream head
|
|
influences on the next layer's Q/K/V; identity of dominant heads may vary at each
|
|
sample, query token and recipient head. No training, GPU, deletion or model edits.
|
|
|
|
Scope: all 23 adjacent layer transitions of the retained Qwen3.5-2B text model,
|
|
including 16-head linear attention and 8-head full attention. Measure effective
|
|
Q/K/V, and retrieval probability changes where the recipient uses full attention.
|
|
The intervening source-layer residual/MLP and recipient input normalization must
|
|
be recomputed after interventions. Residual stream is preserved: removing an
|
|
attention head is not removing all information reaching the recipient.
|
|
|
|
For each observation, remove one upstream attention head's projected contribution
|
|
at a time. Record nonnegative influence magnitude d_i and its absolute scale.
|
|
When sum(d_i)>0, p_i=d_i/sum(d_i) is normalized intervention influence, NOT an
|
|
additive causal attribution or a native attention probability over upstream heads.
|
|
All-zero influence has undefined concentration and must be flagged, not assigned
|
|
uniform or concentrated values. Record small total influences explicitly.
|
|
|
|
Primary continuous measures:
|
|
- Effective head count: N_eff=1/sum(p_i^2), range [1,H].
|
|
- Effective head fraction: N_eff/H, supports different H.
|
|
- Normalized concentration: C=(sum(p_i^2)-1/H)/(1-1/H), range [0,1].
|
|
These are related summaries, not independent evidence. Report distributions over
|
|
texts/tokens/recipient heads/layers, not only concentration of globally averaged
|
|
influences. Primary query positions 7 onward avoid trivial short-prefix attention.
|
|
|
|
Secondary intuitive measures: top1/top25%/top50% influence share, bottom25%
|
|
share, and number of heads reaching 90% influence. Preserve raw influence tables.
|
|
No RMS activity threshold is used to define influence concentration.
|
|
|
|
Joint interventions are required: per observation's weak-head set and strong-head
|
|
set must be removed together and measured, because individual ablations are not
|
|
additive and may conceal cancellation or synergy. Rankings may be input-specific;
|
|
these experiments cannot imply a universally removable fixed head set. Distinguish
|
|
local next-layer QKV/probability effects from unmeasured final LM quality.
|
|
|
|
CPU measurements, scientific plots, tests and audits execute on the previously
|
|
authorized server 8.211.154.134. All weights and existing caches remain retained.
|
|
|
|
## Fixed execution details (before real-model measurements)
|
|
|
|
Four texts are reused from crossqkv/fresh_samples.json (two Chinese/two English),
|
|
32 tokens each; no independent generalization claim. All tokens are computed,
|
|
queries7..31 are primary statistical observations. The observer caches source
|
|
residual inputs and exact per-head projected output components; their sum is
|
|
checked against the native branch output. First text's full forward is checked
|
|
against the canonical streaming forward, and every transition has a null-mask
|
|
control. No cached tensor is deleted.
|
|
|
|
For each query/recipient query-head, individual influence is the L2 change divided
|
|
by that same baseline vector's L2 (floor1e-12). Q,K,V are analyzed separately.
|
|
A declared balanced score sqrt((relative_Q^2+relative_K^2+relative_V^2)/3) is also
|
|
analyzed; it does not treat their different physical scales as directly additive.
|
|
Full attention: Q/K are measured after learned norm/partial RoPE, V before output
|
|
mixing, and actual softmax retrieval JS is measured. Linear attention: Q/K/V are
|
|
after causal convolution+SiLU; Q/K include the delta-rule L2 normalization and Q
|
|
scale. It has no ordinary softmax retrieval distribution. Full-attention GQA K/V
|
|
are repeated to query-head slots for aligned joint tests; these slots are not
|
|
independent KV heads, and statistical observations are not independent samples.
|
|
|
|
Joint weak-half/strong-half/random-half/all-head removal uses a separate coherent
|
|
sequence for each recipient query-head. At each source token, choose the donor
|
|
half by that recipient query-head's QKV influence ranking at the same token.
|
|
Recompute full recipient sequence; convolution and QK retrieval can therefore
|
|
propagate changes between positions. Only the intended recipient head is scored
|
|
for each scenario. This is an input-adaptive diagnostic, not a computationally
|
|
free head-selection policy. Random uses fixed seed31000+sample_index*100+layer.
|
|
Compare joint effects to sums of corresponding single effects but do not assume
|
|
additivity. For JS use the same joint sets (selected by QKV, not by JS).
|
|
|
|
No success criterion assumes that dominance exists. Report effective-head and
|
|
absolute-effect distributions even if contributions are diffuse. No final-LM
|
|
NLL or deployable pruning quality claim. Scientific output includes all raw
|
|
single/joint effects and deletion masks; large baseline caches remain external.
|