37 lines
2.6 KiB
Markdown
37 lines
2.6 KiB
Markdown
# CPU learned-function repetition pilot
|
|
|
|
Primary question: do two distinct trained parts implement similar functions? Cost is not a success condition.
|
|
- Existing learned RMSNorm function positive: True
|
|
- Bidirectional network quality screen passed: True
|
|
- Learned scalar MLP neuron search positive: False
|
|
- Original 11 candidate run state: completed; selected: []
|
|
- All model weights are retained on the server until the user explicitly requests deletion.
|
|
|
|
## Existing learned module evidence
|
|
|
|
Role: input_layernorm; layers 12 and 13; each maps 2048-dimensional input to output.
|
|
Uniform real-arithmetic relative-error bound, first-to-second: 0.101851851852; reverse: 0.113402061856.
|
|
Maximum heldout directional output relative L2: 0.04528510757179746; minimum cosine: 0.9994870299724158.
|
|
Same epsilon and nonzero gains: the common RMS denominator cancels. For any nonzero input x,
|
|
`||f_a(x)-f_b(x)|| / ||f_a(x)|| <= max_j |a_j-b_j|/|a_j|`; swap a/b for the reverse direction.
|
|
Selection used only the tight all-input bound across 552 same-role pairs, before inspecting inputs.
|
|
Real probes use four calibration and four heldout experiment-authored texts, 32 tokens each, and both source layers.
|
|
These are learned, nonconstant gain vectors. The claim concerns RMSNorm modules; it does not establish attention/MLP-body duplication.
|
|
|
|
- Layer 12 uses 13: mean heldout NLL delta 3.25651368766e-05; sequence P95 delta 0.000984639618287; screen pass=True.
|
|
- Layer 13 uses 12: mean heldout NLL delta -0.000363972997404; sequence P95 delta 0.000709721155035; screen pass=True.
|
|
|
|
## Negative results and separate algebraic result
|
|
|
|
The original whole-attention, whole-MLP and within-layer Q/K norm candidates did not pass their gates.
|
|
The scalar neuron search excludes low-RMS/near-constant profiles and selects at most one pair without heldout feedback; consult its search summary for coverage.
|
|
Q/K non-RoPE gain consolidation is a separately verified real-arithmetic identity, not evidence that two existing learned modules duplicate functions.
|
|
Earlier local CPU timing records are preserved for provenance; performance is outside the current success criterion.
|
|
|
|
## Limits and provenance
|
|
|
|
Original BF16 weights were decoded into streaming FP32 CPU operators; no training, scale fitting, R fitting or GPU.
|
|
The numerical reference includes 22 independent official Torch comparisons. Early adapter errors and two resource-guard stops are preserved as historical logs.
|
|
The persistent computation ledger includes a conservative 180-second reserve for early checks; no budget reset after that reserve.
|
|
The text sample is tiny and short. Network NLL checks are not broad task-accuracy validation.
|