Archive research charts and compare GSM8K LoRA head concentration
This commit is contained in:
@@ -0,0 +1,61 @@
|
||||
# Existing GSM8K fine-tune: paired head-concentration comparison
|
||||
|
||||
First restore the original charts through [the fixed figure gallery](../docs/FIGURE_GALLERY.md).
|
||||
The original 8.931822/16, P10=5.784081 and P90=11.672658 refer to sqrt(JS)
|
||||
influences of all16 prior-layer heads on the six full-attention receivers.
|
||||
|
||||
Selected release: [Avifenesh/gdn-beta-gate-qwen3.5-2b-grpo](https://huggingface.co/Avifenesh/gdn-beta-gate-qwen3.5-2b-grpo),
|
||||
pinned038d8be9bd0edd3883fa019651b02bdf65609468, pure lora/seed0. The publisher
|
||||
records GSM8K GRPO from a shared warmup, rank16 alpha32, exact same original
|
||||
Qwen3.5-2B revision. This is extra RL fine-tuning of the post-trained release,
|
||||
not pretraining-to-SFT, and not a beta-gate architecture modification. Our seed
|
||||
choice is fixed before seeing effects; no new training is performed.
|
||||
|
||||
`lora.py` validates all144 tensors/72 pairs, maps only declared GDN projections,
|
||||
and provides read-only FP32 effective row chunks W+2BA. Source head decomposition
|
||||
uses the same effective weights. `ExplicitLoRA` adds2B(Ax) independently to the
|
||||
base projection as a numerical reference. No BF16 re-rounding is introduced:
|
||||
the author's training code also warns that rounding merged increments can erase
|
||||
small LoRA changes. Author configuration commit/SHA and public HF metadata are
|
||||
saved alongside the protocol; foreign training code and pickle checkpoints are
|
||||
not executed.
|
||||
|
||||
`run.py` reuses the exact original single-ablation arrays for the original cohort,
|
||||
then computes matched finetuned states; it computes both models on4 question-only
|
||||
GSM8K test prefixes as a separate cohort. Every input has32 tokens. All16 donor
|
||||
heads, six receiver layers andquery7..31 remain identical to the original metric.
|
||||
The task corpus is the canonical author's test JSONL at a fixed Git commit, and
|
||||
selection uses only the first4 items with at least32 tokenizer tokens. All item
|
||||
indices, question hashes and token IDs are recorded; answers are not model inputs.
|
||||
|
||||
`report.py` generates paired CDFs, per-layer medians/quantiles, paired change
|
||||
histograms, raw observations and tables. `audit.py` independently reconstructs
|
||||
effective-head counts via(sum d)^2/sum(d^2), checks pairing, exact old baselines,
|
||||
head sums, null masks, adapter/dataset/base SHA and retained external caches.
|
||||
`tests/test_finetune_compare.py` checks streamed row arithmetic, an independent
|
||||
low-rank forward and head decomposition, plus exact zero-scale restoration.
|
||||
|
||||
No iid-token p-values, training-seed generalization, GSM8K accuracy result,
|
||||
long-context claim, pruning recommendation or functional-duplicate claim.
|
||||
The paired delta median is distinguished from the difference of pooled medians.
|
||||
All weights, adapters and caches remain on the authorized CPU server until user
|
||||
explicit deletion. Git stores all raw influence arrays, not the model/adapter.
|
||||
|
||||
Remote workflow on8.211.154.134:
|
||||
```
|
||||
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -B -m unittest tests.test_finetune_compare -v
|
||||
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -B -m finetune_compare.run --source runs/cpu-qwen35-2b --parent runs/head-dominance --assets runs/finetune-head-comparison --out runs/finetune-head-comparison/NEW_EXPERIMENT
|
||||
venv/bin/python -B -m finetune_compare.report runs/finetune-head-comparison/NEW_EXPERIMENT
|
||||
venv/bin/python -B -m finetune_compare.audit runs/finetune-head-comparison/NEW_EXPERIMENT runs/head-dominance runs/finetune-head-comparison runs/cpu-qwen35-2b
|
||||
```
|
||||
|
||||
The supplemental QKV field is measured only on the same six full-attention
|
||||
receivers, so its baseline must not be confused with the older76.0% pooled
|
||||
QKV effective-head fraction across all23 adjacent transitions. The gallery
|
||||
preserves that original table separately; the requested8.931822 metric uses
|
||||
exactly the matching six receivers in both studies.
|
||||
|
||||
The audit also records all72 low-rank update Frobenius norms (via small Gram
|
||||
matrices) and final pre-normalization residual changes per text. These separate
|
||||
a stable concentration statistic from an accidentally inactive adapter. Residual
|
||||
changes are not logit distributions, accuracy differences or capability scores.
|
||||
Reference in New Issue
Block a user