76 lines
4.9 KiB
Markdown
76 lines
4.9 KiB
Markdown
# Statistical head influence dominance (CPU)
|
||
|
||
This study answers how concentrated the **distribution of measured upstream-head
|
||
influences** is, without assuming that dominant head identities are fixed across
|
||
texts, positions or recipient heads. Read [registered protocol](PROTOCOL.md).
|
||
|
||
- `core.py`: normalized effective Q/K/V extraction for both attention types,
|
||
source projected head decomposition, exact nonlinear downstream recomputation,
|
||
coherent masks, continuous concentration and JS.
|
||
- `run.py`: four fixed prior texts, all23 adjacent connections, single-head and
|
||
weak/strong/random-half/all-head interventions, per-transition checkpointing,
|
||
canonical-forward/null controls, preserved baseline caches.
|
||
- `report.py`: full saved-data distributions, per-transition CSV, continuous
|
||
effective head plots, influence rank curves, joint tests and Chinese report.
|
||
- `audit.py`: independently recomputes concentration/masks from raw effects,
|
||
validates scope/null/head sums, and hashes retained caches, source and results.
|
||
- `tests/test_dominance.py`: continuous limiting cases, undefined zero-total,
|
||
token-dependent masks, JS identity, both canonical mixed-layer paths, and joint
|
||
interventions on a small checkpoint with known arrays.
|
||
|
||
Influence for Q/K/V is relative L2 change after deleting one whole donor attention
|
||
head (at all sequence positions), preserving the prior residual and recomputing
|
||
source MLP + recipient normalization. Source **output/value heads** are the units:
|
||
16 for linear attention,8 for full attention. These are not arbitrary individual
|
||
Q/K/V weight blocks. For a recipient query-head at one token, normalize donor
|
||
influences to p. Primary continuous summary is N_eff=1/sum(p²), and N_eff/H.
|
||
C=(sum(p²)-1/H)/(1-1/H) is a related0..1 concentration score. These are not
|
||
independent confirmations. No RMS near-zero threshold or R fitting is used.
|
||
|
||
Q/K/V are separate primary scales; balanced QKV score is RMS of their three
|
||
relative changes. It is a declared summary, not a norm in a single physical space.
|
||
Zero total implies undefined concentration, not a uniform distribution. Single-head relative effects before head-share normalization and true joint
|
||
effects are retained as well as shares. The saved metric named max_absolute is
|
||
the largest unsigned relative-L2 effect, not an unnormalized hidden-space norm.
|
||
The total field sums individual relative effects; it is not the measured joint
|
||
effect of removing the whole attention branch.
|
||
|
||
Full-attention recipients additionally have real token-key probability JS changes;
|
||
linear attention does not have an ordinary softmax attention probability matrix.
|
||
Full-attention K/V repeat their GQA groups to query-head slots for aligned joint
|
||
experiments, not as extra independent KV observations. Every headline statistic
|
||
is descriptive: query/token/head observations are dependent, and four short prior
|
||
texts do not establish universal dominance or independent generalization.
|
||
|
||
Joint masks vary per source token and recipient head. Recompute a coherent full
|
||
sequence for each recipient head, scoring that intended head afterward. Rank by
|
||
single QKV influence, remove weakest half/strongest half/a fixed-seed random half,
|
||
or all attention heads. This investigates interactions and cross-token effects;
|
||
it is not a validated free selector, fixed removable head subset or LM pruning.
|
||
|
||
Only run model work and scientific plotting on the authorized CPU server:
|
||
```
|
||
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -B -m unittest tests.test_dominance -v
|
||
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -B -m dominance.run --source runs/cpu-qwen35-2b --out runs/NEW_DOMINANCE_DIR --seconds 7200
|
||
venv/bin/python -B -m dominance.report runs/NEW_DOMINANCE_DIR
|
||
venv/bin/python -B -m dominance.audit runs/NEW_DOMINANCE_DIR runs/cpu-qwen35-2b
|
||
```
|
||
|
||
No training, GPU, new model download or automatic deletion. Raw intervention
|
||
arrays and masks are delivered to Git; large baseline caches stay on the server
|
||
with path/size/SHA metadata. Existing checkpoints/caches are never removed.
|
||
|
||
The saved sum of selected individual effects is only a query-wise mask-weighted
|
||
reference: individual ablations remove a head at all positions, whereas joint
|
||
masks can vary across positions. Their difference cannot isolate pure nonlinear
|
||
interaction, since temporal intervention supports also differ. Use the measured
|
||
weak/strong/random joint effects directly; do not call this reference an additive
|
||
causal decomposition or a Shapley attribution.
|
||
|
||
Scale sensitivity: after reviewing the first complete summaries, add sqrt(JS)
|
||
(Jensen–Shannon distance) alongside primary JS divergence. Divergence is roughly
|
||
quadratic for small changes, so shares and effective head counts depend on this
|
||
scale. All original primary statistics and raw interventions remain unchanged;
|
||
primary-analysis-summary.json preserves the pre-sensitivity aggregate for exact
|
||
audit comparison. Retrieval-scale plots and source tables show both definitions.
|