Quantify adjacent-layer head influence concentration on CPU
This commit is contained in:
@@ -0,0 +1,75 @@
|
||||
# Statistical head influence dominance (CPU)
|
||||
|
||||
This study answers how concentrated the **distribution of measured upstream-head
|
||||
influences** is, without assuming that dominant head identities are fixed across
|
||||
texts, positions or recipient heads. Read [registered protocol](PROTOCOL.md).
|
||||
|
||||
- `core.py`: normalized effective Q/K/V extraction for both attention types,
|
||||
source projected head decomposition, exact nonlinear downstream recomputation,
|
||||
coherent masks, continuous concentration and JS.
|
||||
- `run.py`: four fixed prior texts, all23 adjacent connections, single-head and
|
||||
weak/strong/random-half/all-head interventions, per-transition checkpointing,
|
||||
canonical-forward/null controls, preserved baseline caches.
|
||||
- `report.py`: full saved-data distributions, per-transition CSV, continuous
|
||||
effective head plots, influence rank curves, joint tests and Chinese report.
|
||||
- `audit.py`: independently recomputes concentration/masks from raw effects,
|
||||
validates scope/null/head sums, and hashes retained caches, source and results.
|
||||
- `tests/test_dominance.py`: continuous limiting cases, undefined zero-total,
|
||||
token-dependent masks, JS identity, both canonical mixed-layer paths, and joint
|
||||
interventions on a small checkpoint with known arrays.
|
||||
|
||||
Influence for Q/K/V is relative L2 change after deleting one whole donor attention
|
||||
head (at all sequence positions), preserving the prior residual and recomputing
|
||||
source MLP + recipient normalization. Source **output/value heads** are the units:
|
||||
16 for linear attention,8 for full attention. These are not arbitrary individual
|
||||
Q/K/V weight blocks. For a recipient query-head at one token, normalize donor
|
||||
influences to p. Primary continuous summary is N_eff=1/sum(p²), and N_eff/H.
|
||||
C=(sum(p²)-1/H)/(1-1/H) is a related0..1 concentration score. These are not
|
||||
independent confirmations. No RMS near-zero threshold or R fitting is used.
|
||||
|
||||
Q/K/V are separate primary scales; balanced QKV score is RMS of their three
|
||||
relative changes. It is a declared summary, not a norm in a single physical space.
|
||||
Zero total implies undefined concentration, not a uniform distribution. Single-head relative effects before head-share normalization and true joint
|
||||
effects are retained as well as shares. The saved metric named max_absolute is
|
||||
the largest unsigned relative-L2 effect, not an unnormalized hidden-space norm.
|
||||
The total field sums individual relative effects; it is not the measured joint
|
||||
effect of removing the whole attention branch.
|
||||
|
||||
Full-attention recipients additionally have real token-key probability JS changes;
|
||||
linear attention does not have an ordinary softmax attention probability matrix.
|
||||
Full-attention K/V repeat their GQA groups to query-head slots for aligned joint
|
||||
experiments, not as extra independent KV observations. Every headline statistic
|
||||
is descriptive: query/token/head observations are dependent, and four short prior
|
||||
texts do not establish universal dominance or independent generalization.
|
||||
|
||||
Joint masks vary per source token and recipient head. Recompute a coherent full
|
||||
sequence for each recipient head, scoring that intended head afterward. Rank by
|
||||
single QKV influence, remove weakest half/strongest half/a fixed-seed random half,
|
||||
or all attention heads. This investigates interactions and cross-token effects;
|
||||
it is not a validated free selector, fixed removable head subset or LM pruning.
|
||||
|
||||
Only run model work and scientific plotting on the authorized CPU server:
|
||||
```
|
||||
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -B -m unittest tests.test_dominance -v
|
||||
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -B -m dominance.run --source runs/cpu-qwen35-2b --out runs/NEW_DOMINANCE_DIR --seconds 7200
|
||||
venv/bin/python -B -m dominance.report runs/NEW_DOMINANCE_DIR
|
||||
venv/bin/python -B -m dominance.audit runs/NEW_DOMINANCE_DIR runs/cpu-qwen35-2b
|
||||
```
|
||||
|
||||
No training, GPU, new model download or automatic deletion. Raw intervention
|
||||
arrays and masks are delivered to Git; large baseline caches stay on the server
|
||||
with path/size/SHA metadata. Existing checkpoints/caches are never removed.
|
||||
|
||||
The saved sum of selected individual effects is only a query-wise mask-weighted
|
||||
reference: individual ablations remove a head at all positions, whereas joint
|
||||
masks can vary across positions. Their difference cannot isolate pure nonlinear
|
||||
interaction, since temporal intervention supports also differ. Use the measured
|
||||
weak/strong/random joint effects directly; do not call this reference an additive
|
||||
causal decomposition or a Shapley attribution.
|
||||
|
||||
Scale sensitivity: after reviewing the first complete summaries, add sqrt(JS)
|
||||
(Jensen–Shannon distance) alongside primary JS divergence. Divergence is roughly
|
||||
quadratic for small changes, so shares and effective head counts depend on this
|
||||
scale. All original primary statistics and raw interventions remain unchanged;
|
||||
primary-analysis-summary.json preserves the pre-sensitivity aggregate for exact
|
||||
audit comparison. Retrieval-scale plots and source tables show both definitions.
|
||||
Reference in New Issue
Block a user