Files
exp-model-parameter-similarity/crossqkv

Cross-layer QKV CPU study

This phase targets learned QKV functions across different layers, independent of cost. The six full-attention layers (indices 3/7/11/15/19/23) form 15 unordered pairs. No within-layer Q/K/V substitution, training, continuous R fitting, GPU, or new model download.

Read fixed primary protocol and findings with evidence limits. The original main protocol had no passing candidate. A later fixed one-source audit showed near-self-token retrieval and small readout errors, but failed strict fresh mean-JS/row-JS-P95/row-TV-P95 criteria; it is not relabeled as a primary pass.

  • core.py: exact discrete head assignment, actual attention parts, common-space head decomposition, fixed recipient V/gate/O transplants and temporal-cache controls.
  • protocol.py, fresh_samples.json: predeclared layer/unit/gate scope and untouched fresh texts.
  • run.py, fresh.py: calibration, frozen selection, old validation and gated fresh/network steps.
  • conditional.py: separately registered post-screen metric audit on one fixed H23 candidate; no fallback selection or later threshold changes.
  • audit_case.py: recompute selected fresh probabilities, self mass, shared top5 mass and actual-H3 versus same-H23 difference; no full-network intervention after failure.
  • report.py, audit.py: saved-data scientific plots, raw-count/matching/probability audits, SHA256 and explicit small Git delivery with external large-cache metadata.

Use the authorized remote CPU environment with the retained source checkpoint/cache:

OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -B -m unittest tests.test_crossqkv -v
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 timeout 7200 venv/bin/python -B -m crossqkv.run --source runs/cpu-qwen35-2b --root runs/NEW_QKV_DIRECTORY

The main run requires a fresh directory and never overwrites prior protocol evidence. The recorded execution used runs/cross-layer-qkv-v2 after fixing a controls-variable shadowing error in the first attempt; old logs and its profile cache are retained. The first attempt took about 10.3 seconds, then a 60-second reserve was deducted from the next 7140-second run limit. The main study finished in about 74.8 seconds. The fixed conditional audit used only remaining shared budget and took about 105.3 seconds. Final tests/plots/audits fit comfortably within the two-hour computation budget.

All weights and caches stay on the server until explicit user deletion. Git contains all figure tables and the selected small probability matrices; large caches have exact paths/sizes/hashes in external-caches.json. Old validation texts were used in earlier experiments; the four fresh texts were fixed before the new cross-layer measurements.

LiSA-style single-layer pilot

The pilot reuses layer 19's actual pre-softmax scores at layer 23, with the recipient's own values, output gate and output projection. It compares native attention, direct reuse, a frozen one-to-one head assignment, linear mixing of donor heads, a 32-coordinate recipient-only control, and joint donor/recipient mixing. The linear variants use a closed-form ridge fit to the native scores, centred within each causal key row. The recipient branch takes a fixed subset of post-RoPE coordinates rather than learning a low-rank Q/K projection. The calibration uses 12 synthetic prompts without answer labels. All checkpoint weights are frozen.

Evaluation scores the four candidate option tokens on 24 disjoint retrieval prompts, including 12 with more table entries than calibration. Choice NLL normalizes over those four tokens. Run against the retained checkpoint on the remote CPU environment:

OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -B -m crossqkv.lisa_pilot \
  --source runs/cpu-qwen35-2b --out runs/NEW_LISA_DIRECTORY

The diagnostic forward still computes native full Q/K and does not measure latency or parameter savings. Refer to LiSA (arXiv:2408.01890, §5.1) for its learned nonlinear head aligner and learned low-rank Q/K projections.