Archive research charts and compare GSM8K LoRA head concentration
This commit is contained in:
@@ -0,0 +1,42 @@
|
||||
# Paired task-finetuning comparison (registered before reading finetuned effects)
|
||||
|
||||
Primary checkpoint: Avifenesh/gdn-beta-gate-qwen3.5-2b-grpo,
|
||||
HF revision038d8be9bd0edd3883fa019651b02bdf65609468, pure lora/seed0,
|
||||
step150 after shared40-step warmup. GSM8K GRPO, not SFT. Selected by smallest
|
||||
seed index before observing concentration, not by benchmark score or our metric.
|
||||
No gate01/gate02 arm or architecture changes. Exact base revision matches retained
|
||||
Qwen/Qwen3.5-2B15852e8c16360a2fea060d615a32b45270f8a8fc.
|
||||
|
||||
Download existing18.9MB F32 LoRA tensors, verify official file SHA. All144 tensors
|
||||
must form72 A/B pairs on the4 declared GDN projections of18 linear-attention
|
||||
layers, rank16 alpha32. Apply FP32 effective W +2BA in streamed row chunks, also
|
||||
when decomposing source output heads. Do not quantize merged weights, mutate or
|
||||
save over base weights, execute foreign .pt files, train or use GPU. Validate
|
||||
against separate base-output +2B(Ax) forward and check head sums/null masks.
|
||||
|
||||
Primary comparison exactly matches the reported8.931822/16 statistic: all six
|
||||
ordinary attention receivers3/7/11/15/19/23, previous-layer16 donor output heads,
|
||||
all32 tokens computed, query7..31, original4 plain-text inputs and same token IDs.
|
||||
Reuse corresponding immutable base single-ablation arrays, recompute FT baselines
|
||||
and all16 single donor removals, source MLP/residual/recipient normalization exact.
|
||||
Use the same sqrt(JS) distance influence, normalized shares and N_eff=1/sum(p²).
|
||||
Preserve JS divergence as a sensitivity row; no change of ranking or threshold.
|
||||
All4800 per-token/query-head observations are paired. No new joint-mask search:
|
||||
the requested statistic depends on single removals, not the previous joint tests.
|
||||
|
||||
Secondary task-conditioned comparison: first4 original GSM8K test questions with
|
||||
at least32 base-tokenizer tokens (length selection only, before model effects),
|
||||
question-only plain text, first32 tokens. Freeze original author repository commit,
|
||||
JSONL SHA, item indices, question hashes and exact token IDs. No answers, generation,
|
||||
chat-template change or accuracy claim. Compute both base and FT on these same
|
||||
questions; never compare old general-text base to new GSM8K FT as if paired.
|
||||
|
||||
Report overall median/P10/P90, paired differences, per receiver layer and per text,
|
||||
strong/weak-quartile shares, and dominant donor identity agreement. No iid-token
|
||||
p-values, no claim over training seeds (one fixed trained seed), no proof of
|
||||
GSM8K accuracy improvement or long-context behavior. Positive, negative and near
|
||||
zero changes all count as results. Source cohort remains4+4 short texts.
|
||||
|
||||
All model work/tests/plots on authorized CPU server8.211.154.134, bounded7200s.
|
||||
Preserve all base/FT weights, adapters and caches; no automatic cleanup. Save raw
|
||||
arrays, source provenance, SHA manifests, fixed chart gallery and reviewable PR.
|
||||
Reference in New Issue
Block a user