exp-model-parameter-similarity/functional/README.md

# Bounded CPU functional pilot

This is the execution implementation of the CPU-first plan, not the completed static
parameter pilot. Frozen Qwen3.5-2B weights are decoded to FP32 a projection block at a
time. CPU only, no autograd/optimization, no rotation fitting, no model-hub Python code.
The user explicitly requires retaining this checkpoint until they request deletion.

## Fixed scope

- Batch of eight distinct experiment-authored plain texts, first 32 tokens each:
  four calibration and four heldout, two Chinese/two English in each split.
- Full attention pairs: (3,7), (11,15), (19,23).
- MLP pairs: (5,6), (20,21).
- Smaller-granularity Q/K RMSNorm pairs inside layers 3,7,11,15,19,23.
  Added before observing real weights/functions for this pilot; these are small storage
  opportunities, not a claim of substantial compute savings.
- At most one calibration-selected candidate, two separate substitution directions.
  Selection prefers component size, then worst symmetric output error. No heldout-driven retry.
- Same-input relative L2 <=0.05 in both directions and token P95 <=0.1 in both directions;
  attention additionally requires mean JS <=0.01 and top-5 overlap >=0.8.
- Heldout mean NLL increase <=0.01 nat/token and per-sequence P95 increase <=0.05.
  These are pilot decision criteria, not mathematical capability guarantees.

## Source and checks

`reference.json` pins the official Transformers Qwen3.5 source commit, URL and hash.
`reference_check.py` extracts only named mathematical definitions from that exact file,
removes optional fused-kernel decorators, and compares the NumPy implementation against
Torch operators, full toy decoder layers and a two-layer text forward. Chunked delta and
recurrent delta implementations are both checked, including sequence padding boundaries.
The official source is Apache-2.0, copyright Qwen Team and HuggingFace; the reference file
is fetched for testing and is not bundled here. FP32 equivalence is checked, not equality
to every deployed BF16 kernel.

`tests/test_functional.py` independently checks streamed projection/embedding/NLL,
causality, causal convolution, normalization conventions and basic metric identities.
These tests use tiny synthetic arrays. Real model evidence is produced only by `run.py`.

## Remote execution

Use the previously authorized CPU host and existing NumPy/Matplotlib environment. Install
only binary wheels: tokenizers==0.23.3 from HTTPS PyPI; torch==2.15.0+cpu from the official
CPU wheel index for independent reference checks. Use an experiment-owned on-disk TMPDIR:
the shared host's /tmp tmpfs was too small during the first wheel download. Do not change
system swap or shared temporary files.

```sh
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -B -m unittest tests.test_functional -v
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -B -m functional.reference_check --source runs/cpu-qwen35-2b/reference.py --output runs/cpu-qwen35-2b/reference-check.json
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -B -m functional.supervise --root runs/cpu-qwen35-2b
```

The supervisor waits at most one hour for binary dependencies and four hours for the
already started pinned, verified checkpoint download. It does not install, retry downloads,
search extra candidates, mutate Git or delete weights. After reference checks, it runs one
8-token real text capacity check, then the bounded functional pilot. Reference, preflight
and functional compute together are capped at two hours with a reporting reserve;
download wait is recorded separately. Projection memory is checked, RSS soft stop is
512 MiB, available-system-memory floor 256 MiB; sustained host swap IO combined with low available memory or >128 MiB process swap also stops the run.
A persistent ledger counts computation across restarts; early checks/preflight receive a conservative 180-second reserve.

`preflight.py` is a capacity smoke check, not a semantic oracle. `run.py` records exact
sample/token IDs, fixed configuration, source hashes, activations, per-text/per-head metrics,
selection, per-token NLL, intervention results and truthful stop coverage. `report.py` reads
saved data to produce PNG/PDF/SVG figures without reopening weights.

No generation throughput is measured in this short-text pilot. Any parameter-tying storage
benefit must be accounted for separately, including metadata. Neither same-input similarity
nor favorable tiny heldout NLL demonstrates broad mutual interchangeability or GPU speedup.

## Current primary result and completed extensions

The latest user goal is learned-function repetition, independent of costs. Existing
input RMSNorm modules at layer indices 12/13 have a proven worst two-direction real-arithmetic
error bound of 11.3402%, worst observed heldout relative error 4.5285%, minimum cosine 0.999487,
and small bidirectional heldout NLL changes. See [findings](../docs/CPU_FUNCTIONAL_FINDINGS.md).

- `neuron_reuse.py`: fixed four-layer scalar MLP function search, nontrivial activity
  gates, calibration-only selection and independent heldout/substitution logic; no pair passed.
- `norm_repetition.py`: same-role learned RMSNorm bound across 552 pairs, one deterministic
  selection independent of any input samples, real probes and two full-network substitutions.
- `gain_factorization.py`: architecture-derived identity on non-RoPE coordinates; a separate
  reparameterization result, not a learned-function repetition example.
- `benchmark_factorization.py`: historical one-attempt CPU allocation optimization and
  bitwise equivalence check. Performance is no longer a success condition.
- `storage.py`: explicit component layout accounting; the original candidate run selected
  no pair. This does not affect the primary function-repetition conclusion.

Do not expand the final positive claim to full attention or MLP bodies. Retain the
fixed checkpoint until the user explicitly requests deletion. Full tests and audit are
run remotely; no local build or model execution is required to inspect the archived results.