43 lines
2.8 KiB
Markdown
43 lines
2.8 KiB
Markdown
# Paired task-finetuning comparison (registered before reading finetuned effects)
|
|
|
|
Primary checkpoint: Avifenesh/gdn-beta-gate-qwen3.5-2b-grpo,
|
|
HF revision038d8be9bd0edd3883fa019651b02bdf65609468, pure lora/seed0,
|
|
step150 after shared40-step warmup. GSM8K GRPO, not SFT. Selected by smallest
|
|
seed index before observing concentration, not by benchmark score or our metric.
|
|
No gate01/gate02 arm or architecture changes. Exact base revision matches retained
|
|
Qwen/Qwen3.5-2B15852e8c16360a2fea060d615a32b45270f8a8fc.
|
|
|
|
Download existing18.9MB F32 LoRA tensors, verify official file SHA. All144 tensors
|
|
must form72 A/B pairs on the4 declared GDN projections of18 linear-attention
|
|
layers, rank16 alpha32. Apply FP32 effective W +2BA in streamed row chunks, also
|
|
when decomposing source output heads. Do not quantize merged weights, mutate or
|
|
save over base weights, execute foreign .pt files, train or use GPU. Validate
|
|
against separate base-output +2B(Ax) forward and check head sums/null masks.
|
|
|
|
Primary comparison exactly matches the reported8.931822/16 statistic: all six
|
|
ordinary attention receivers3/7/11/15/19/23, previous-layer16 donor output heads,
|
|
all32 tokens computed, query7..31, original4 plain-text inputs and same token IDs.
|
|
Reuse corresponding immutable base single-ablation arrays, recompute FT baselines
|
|
and all16 single donor removals, source MLP/residual/recipient normalization exact.
|
|
Use the same sqrt(JS) distance influence, normalized shares and N_eff=1/sum(p²).
|
|
Preserve JS divergence as a sensitivity row; no change of ranking or threshold.
|
|
All4800 per-token/query-head observations are paired. No new joint-mask search:
|
|
the requested statistic depends on single removals, not the previous joint tests.
|
|
|
|
Secondary task-conditioned comparison: first4 original GSM8K test questions with
|
|
at least32 base-tokenizer tokens (length selection only, before model effects),
|
|
question-only plain text, first32 tokens. Freeze original author repository commit,
|
|
JSONL SHA, item indices, question hashes and exact token IDs. No answers, generation,
|
|
chat-template change or accuracy claim. Compute both base and FT on these same
|
|
questions; never compare old general-text base to new GSM8K FT as if paired.
|
|
|
|
Report overall median/P10/P90, paired differences, per receiver layer and per text,
|
|
strong/weak-quartile shares, and dominant donor identity agreement. No iid-token
|
|
p-values, no claim over training seeds (one fixed trained seed), no proof of
|
|
GSM8K accuracy improvement or long-context behavior. Positive, negative and near
|
|
zero changes all count as results. Source cohort remains4+4 short texts.
|
|
|
|
All model work/tests/plots on authorized CPU server8.211.154.134, bounded7200s.
|
|
Preserve all base/FT weights, adapters and caches; no automatic cleanup. Save raw
|
|
arrays, source provenance, SHA manifests, fixed chart gallery and reviewable PR.
|