4.0 KiB
Existing GSM8K fine-tune: paired head-concentration comparison
First restore the original charts through the fixed figure gallery. The original 8.931822/16, P10=5.784081 and P90=11.672658 refer to sqrt(JS) influences of all16 prior-layer heads on the six full-attention receivers.
Selected release: Avifenesh/gdn-beta-gate-qwen3.5-2b-grpo, pinned038d8be9bd0edd3883fa019651b02bdf65609468, pure lora/seed0. The publisher records GSM8K GRPO from a shared warmup, rank16 alpha32, exact same original Qwen3.5-2B revision. This is extra RL fine-tuning of the post-trained release, not pretraining-to-SFT, and not a beta-gate architecture modification. Our seed choice is fixed before seeing effects; no new training is performed.
lora.py validates all144 tensors/72 pairs, maps only declared GDN projections,
and provides read-only FP32 effective row chunks W+2BA. Source head decomposition
uses the same effective weights. ExplicitLoRA adds2B(Ax) independently to the
base projection as a numerical reference. No BF16 re-rounding is introduced:
the author's training code also warns that rounding merged increments can erase
small LoRA changes. Author configuration commit/SHA and public HF metadata are
saved alongside the protocol; foreign training code and pickle checkpoints are
not executed.
run.py reuses the exact original single-ablation arrays for the original cohort,
then computes matched finetuned states; it computes both models on4 question-only
GSM8K test prefixes as a separate cohort. Every input has32 tokens. All16 donor
heads, six receiver layers andquery7..31 remain identical to the original metric.
The task corpus is the canonical author's test JSONL at a fixed Git commit, and
selection uses only the first4 items with at least32 tokenizer tokens. All item
indices, question hashes and token IDs are recorded; answers are not model inputs.
report.py generates paired CDFs, per-layer medians/quantiles, paired change
histograms, raw observations and tables. audit.py independently reconstructs
effective-head counts via(sum d)^2/sum(d^2), checks pairing, exact old baselines,
head sums, null masks, adapter/dataset/base SHA and retained external caches.
tests/test_finetune_compare.py checks streamed row arithmetic, an independent
low-rank forward and head decomposition, plus exact zero-scale restoration.
No iid-token p-values, training-seed generalization, GSM8K accuracy result, long-context claim, pruning recommendation or functional-duplicate claim. The paired delta median is distinguished from the difference of pooled medians. All weights, adapters and caches remain on the authorized CPU server until user explicit deletion. Git stores all raw influence arrays, not the model/adapter.
Remote workflow on8.211.154.134:
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -B -m unittest tests.test_finetune_compare -v
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -B -m finetune_compare.run --source runs/cpu-qwen35-2b --parent runs/head-dominance --assets runs/finetune-head-comparison --out runs/finetune-head-comparison/NEW_EXPERIMENT
venv/bin/python -B -m finetune_compare.report runs/finetune-head-comparison/NEW_EXPERIMENT
venv/bin/python -B -m finetune_compare.audit runs/finetune-head-comparison/NEW_EXPERIMENT runs/head-dominance runs/finetune-head-comparison runs/cpu-qwen35-2b
The supplemental QKV field is measured only on the same six full-attention receivers, so its baseline must not be confused with the older76.0% pooled QKV effective-head fraction across all23 adjacent transitions. The gallery preserves that original table separately; the requested8.931822 metric uses exactly the matching six receivers in both studies.
The audit also records all72 low-rank update Frobenius norms (via small Gram matrices) and final pre-normalization residual changes per text. These separate a stable concentration statistic from an accidentally inactive adapter. Residual changes are not logit distributions, accuracy differences or capability scores.