Quantify adjacent-layer head influence concentration on CPU
This commit is contained in:
@@ -0,0 +1,76 @@
|
||||
# Head influence concentration: measurement protocol
|
||||
|
||||
Target: quantify the statistical distribution of large and small upstream head
|
||||
influences on the next layer's Q/K/V; identity of dominant heads may vary at each
|
||||
sample, query token and recipient head. No training, GPU, deletion or model edits.
|
||||
|
||||
Scope: all 23 adjacent layer transitions of the retained Qwen3.5-2B text model,
|
||||
including 16-head linear attention and 8-head full attention. Measure effective
|
||||
Q/K/V, and retrieval probability changes where the recipient uses full attention.
|
||||
The intervening source-layer residual/MLP and recipient input normalization must
|
||||
be recomputed after interventions. Residual stream is preserved: removing an
|
||||
attention head is not removing all information reaching the recipient.
|
||||
|
||||
For each observation, remove one upstream attention head's projected contribution
|
||||
at a time. Record nonnegative influence magnitude d_i and its absolute scale.
|
||||
When sum(d_i)>0, p_i=d_i/sum(d_i) is normalized intervention influence, NOT an
|
||||
additive causal attribution or a native attention probability over upstream heads.
|
||||
All-zero influence has undefined concentration and must be flagged, not assigned
|
||||
uniform or concentrated values. Record small total influences explicitly.
|
||||
|
||||
Primary continuous measures:
|
||||
- Effective head count: N_eff=1/sum(p_i^2), range [1,H].
|
||||
- Effective head fraction: N_eff/H, supports different H.
|
||||
- Normalized concentration: C=(sum(p_i^2)-1/H)/(1-1/H), range [0,1].
|
||||
These are related summaries, not independent evidence. Report distributions over
|
||||
texts/tokens/recipient heads/layers, not only concentration of globally averaged
|
||||
influences. Primary query positions 7 onward avoid trivial short-prefix attention.
|
||||
|
||||
Secondary intuitive measures: top1/top25%/top50% influence share, bottom25%
|
||||
share, and number of heads reaching 90% influence. Preserve raw influence tables.
|
||||
No RMS activity threshold is used to define influence concentration.
|
||||
|
||||
Joint interventions are required: per observation's weak-head set and strong-head
|
||||
set must be removed together and measured, because individual ablations are not
|
||||
additive and may conceal cancellation or synergy. Rankings may be input-specific;
|
||||
these experiments cannot imply a universally removable fixed head set. Distinguish
|
||||
local next-layer QKV/probability effects from unmeasured final LM quality.
|
||||
|
||||
CPU measurements, scientific plots, tests and audits execute on the previously
|
||||
authorized server 8.211.154.134. All weights and existing caches remain retained.
|
||||
|
||||
## Fixed execution details (before real-model measurements)
|
||||
|
||||
Four texts are reused from crossqkv/fresh_samples.json (two Chinese/two English),
|
||||
32 tokens each; no independent generalization claim. All tokens are computed,
|
||||
queries7..31 are primary statistical observations. The observer caches source
|
||||
residual inputs and exact per-head projected output components; their sum is
|
||||
checked against the native branch output. First text's full forward is checked
|
||||
against the canonical streaming forward, and every transition has a null-mask
|
||||
control. No cached tensor is deleted.
|
||||
|
||||
For each query/recipient query-head, individual influence is the L2 change divided
|
||||
by that same baseline vector's L2 (floor1e-12). Q,K,V are analyzed separately.
|
||||
A declared balanced score sqrt((relative_Q^2+relative_K^2+relative_V^2)/3) is also
|
||||
analyzed; it does not treat their different physical scales as directly additive.
|
||||
Full attention: Q/K are measured after learned norm/partial RoPE, V before output
|
||||
mixing, and actual softmax retrieval JS is measured. Linear attention: Q/K/V are
|
||||
after causal convolution+SiLU; Q/K include the delta-rule L2 normalization and Q
|
||||
scale. It has no ordinary softmax retrieval distribution. Full-attention GQA K/V
|
||||
are repeated to query-head slots for aligned joint tests; these slots are not
|
||||
independent KV heads, and statistical observations are not independent samples.
|
||||
|
||||
Joint weak-half/strong-half/random-half/all-head removal uses a separate coherent
|
||||
sequence for each recipient query-head. At each source token, choose the donor
|
||||
half by that recipient query-head's QKV influence ranking at the same token.
|
||||
Recompute full recipient sequence; convolution and QK retrieval can therefore
|
||||
propagate changes between positions. Only the intended recipient head is scored
|
||||
for each scenario. This is an input-adaptive diagnostic, not a computationally
|
||||
free head-selection policy. Random uses fixed seed31000+sample_index*100+layer.
|
||||
Compare joint effects to sums of corresponding single effects but do not assume
|
||||
additivity. For JS use the same joint sets (selected by QKV, not by JS).
|
||||
|
||||
No success criterion assumes that dominance exists. Report effective-head and
|
||||
absolute-effect distributions even if contributions are diffuse. No final-LM
|
||||
NLL or deployable pruning quality claim. Scientific output includes all raw
|
||||
single/joint effects and deletion masks; large baseline caches remain external.
|
||||
Reference in New Issue
Block a user