69 lines
4.0 KiB
Markdown
69 lines
4.0 KiB
Markdown
# Cross-layer QKV CPU study
|
|
|
|
This phase targets learned QKV functions across different layers, independent of cost.
|
|
The six full-attention layers (indices 3/7/11/15/19/23) form 15 unordered pairs.
|
|
No within-layer Q/K/V substitution, training, continuous R fitting, GPU, or new model download.
|
|
|
|
Read [fixed primary protocol](../docs/CROSS_LAYER_QKV_PROTOCOL.md) and
|
|
[findings with evidence limits](../docs/CROSS_LAYER_QKV_FINDINGS.md).
|
|
The original main protocol had no passing candidate. A later fixed one-source audit
|
|
showed near-self-token retrieval and small readout errors, but failed strict fresh
|
|
mean-JS/row-JS-P95/row-TV-P95 criteria; it is not relabeled as a primary pass.
|
|
|
|
- `core.py`: exact discrete head assignment, actual attention parts, common-space head
|
|
decomposition, fixed recipient V/gate/O transplants and temporal-cache controls.
|
|
- `protocol.py`, `fresh_samples.json`: predeclared layer/unit/gate scope and untouched fresh texts.
|
|
- `run.py`, `fresh.py`: calibration, frozen selection, old validation and gated fresh/network steps.
|
|
- `conditional.py`: separately registered post-screen metric audit on one fixed H23 candidate;
|
|
no fallback selection or later threshold changes.
|
|
- `audit_case.py`: recompute selected fresh probabilities, self mass, shared top5 mass and
|
|
actual-H3 versus same-H23 difference; no full-network intervention after failure.
|
|
- `report.py`, `audit.py`: saved-data scientific plots, raw-count/matching/probability audits,
|
|
SHA256 and explicit small Git delivery with external large-cache metadata.
|
|
|
|
Use the authorized remote CPU environment with the retained source checkpoint/cache:
|
|
|
|
```sh
|
|
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -B -m unittest tests.test_crossqkv -v
|
|
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 timeout 7200 venv/bin/python -B -m crossqkv.run --source runs/cpu-qwen35-2b --root runs/NEW_QKV_DIRECTORY
|
|
```
|
|
|
|
The main run requires a fresh directory and never overwrites prior protocol evidence.
|
|
The recorded execution used `runs/cross-layer-qkv-v2` after fixing a controls-variable
|
|
shadowing error in the first attempt; old logs and its profile cache are retained.
|
|
The first attempt took about 10.3 seconds, then a 60-second reserve was deducted from
|
|
the next 7140-second run limit. The main study finished in about 74.8 seconds.
|
|
The fixed conditional audit used only remaining shared budget and took about 105.3 seconds.
|
|
Final tests/plots/audits fit comfortably within the two-hour computation budget.
|
|
|
|
All weights and caches stay on the server until explicit user deletion. Git contains
|
|
all figure tables and the selected small probability matrices; large caches have exact
|
|
paths/sizes/hashes in `external-caches.json`. Old validation texts were used in earlier
|
|
experiments; the four fresh texts were fixed before the new cross-layer measurements.
|
|
|
|
## LiSA-style single-layer pilot
|
|
|
|
The pilot reuses layer 19's actual pre-softmax scores at layer 23, with the
|
|
recipient's own values, output gate and output projection. It compares native
|
|
attention, direct reuse, a frozen one-to-one head assignment, linear mixing
|
|
of donor heads, a 32-coordinate recipient-only control, and joint donor/recipient
|
|
mixing. The linear variants use a closed-form ridge fit to the native scores,
|
|
centred within each causal key row. The recipient branch takes a fixed subset of
|
|
post-RoPE coordinates rather than learning a low-rank Q/K projection. The
|
|
calibration uses 12 synthetic prompts without answer labels. All checkpoint
|
|
weights are frozen.
|
|
|
|
Evaluation scores the four candidate option tokens on 24 disjoint retrieval
|
|
prompts, including 12 with more table entries than calibration. Choice NLL
|
|
normalizes over those four tokens. Run against the retained checkpoint on the
|
|
remote CPU environment:
|
|
|
|
```sh
|
|
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -B -m crossqkv.lisa_pilot \
|
|
--source runs/cpu-qwen35-2b --out runs/NEW_LISA_DIRECTORY
|
|
```
|
|
|
|
The diagnostic forward still computes native full Q/K and does not measure
|
|
latency or parameter savings. Refer to LiSA (arXiv:2408.01890, §5.1) for its
|
|
learned nonlinear head aligner and learned low-rank Q/K projections.
|