exp-model-parameter-similarity/docs/SOP.md

Parameter similarity SOP — 2B pilot draft

Status: v0.1, executed for the bounded Qwen3.5-2B pilot. See findings. Changes proposed from this experience require a small equivalence/performance check before the next model; no second model is started automatically.

Scope and decision

Analyze official, unquantized safetensors without instantiating a model, running inference, training, or executing repository-supplied code. Start with Qwen/Qwen3.5-2B only. The question is the number of representative-centered sharing groups under direct weight replacement error limits, and the net byte savings of a theoretical shared representation. Weight error is not task-accuracy loss, and parameter sharing does not imply fewer FLOPs.

Prerequisites

1. Confirm authorized remote host, free disk, memory, CPU, existing work, and access.
2. Install prebuilt wheels only. Never compile locally or silently use another runner.
3. Freeze the official repository revision and download file manifest including SHA256.
4. Allocate checkpoint size plus 3 GB disk margin. Use an experiment-owned directory, not a shared Hugging Face cache. Never delete pre-existing weights.

Bounded procedure

1. Run synthetic correctness tests on the remote host: BF16, 3D indexing, directional replacement error, and non-transitive groups.
2. Download one checkpoint sequentially and verify upstream sizes and LFS SHA256.
3. Scan each tensor in 262,144-element chunks. Record shape, dtype, axis role, layer, bytes, moments, max magnitude, and finite-value checks. Memory-map weights; evict scanned pages. Do not search for exactly equal weight tensors.
4. Whole-tensor screening: same role, same shape, same dtype. Sample 2,048 aligned, evenly spaced elements. Select up to 10 neighbors by each of cosine and relative error. Fairly distribute the 2,000-pair exact-verification cap across role groups.
5. Block screening: non-overlapping 64x64 and 128x128 2D partitions. For 3D tensors, use axis-0 slices and bounded volumes adapted to actual dimensions. Keep up to 256 uniformly reservoir-sampled blocks per role/shape/scale for the 1.7 GiB pilot host (below the planned maximum of 512). Record edge bytes and sampling coverage. Verify at most 20,000 block pairs. Never extrapolate sampled discoveries to unseen blocks or add savings across alternative scales.
6. Exact metrics: full-element cosine, Pearson, directional replacement errors, symmetric error, optimal-scalar residual. Tail diagnostics use samples and say so. Controls include random actual comparable pairs, shuffled sampled elements, and norm-matched Gaussian vectors. Save sampled versus exact error discrepancies. Bounded mode-unfolding SVD probes are diagnostic only, not a replacement test.
7. Form deterministic star groups at 1%, 2%, 5%, 10%, 20% relative weight error. Every member requires an exact verified edge to its representative. Unknown pairs remain independent; never use graph connected components as sharing groups.
8. Compute original bytes minus representatives, 16-byte mapping per analyzed object, and 64-byte descriptor per shared group. Fall back to unchanged storage if larger. Count savings against the entire model, preserving unsampled blocks and edge regions.
9. Save raw data, groups, thresholds, coverage, timings, resource usage, figures (PNG/PDF/SVG), model metadata and report. Validate every group against exact edges.
10. Copy results to the deliverable workspace and verify checksums before deleting the experiment-owned checkpoint. Save a cleanup record. Do not delete the venv, result data or any unrelated cache.
11. Review full diff, commit and push a work branch with Codex Bot identity, then create a PR for manual merge. Never silently initialize/push main or auto-merge.

Budget and stop conditions

• Real 2B analysis computation: hard wall-time limit 7,200 seconds. Reserve final time for summaries; download and report-generation time are recorded separately.
• Use one BLAS thread to avoid interference with existing services.
• Save exact-pair tables every 100 pairs; stage deadlines prevent whole-tensor work from consuming the complete block budget.
• On timeout or failure, retain source weights until partial results are recoverable and report coverage/failure truthfully. An internal planned cutoff may produce a complete bounded report with partial candidate coverage; always state verified/selected counts. An external kill or missing required report is an incomplete run, not a completed experiment.
• Empty or negligible low-error sharing is a valid negative result. Do not add scales, increase error thresholds, align permutations, train or expand to other models automatically.
• No runtime capability/performance claim without separately authorized evaluation.

Commands (remote, from project root)

OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 timeout 300 venv/bin/python -B -m unittest discover -s tests -v
venv/bin/python -B -m pilot.download --root runs/qwen35-2b
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 timeout 7200 venv/bin/python -B -m pilot.analyze --root runs/qwen35-2b --seconds 6600
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -B -m pilot.plot --root runs/qwen35-2b
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 timeout 300 venv/bin/python -B -m unittest discover -s tests -v > runs/qwen35-2b/tests.log 2>&1
venv/bin/python -B -m pilot.audit --root runs/qwen35-2b --analysis-code-commit 6121016
venv/bin/python -B -m pilot.validate --root runs/qwen35-2b
# Copy archived results, verify checksums at the destination, then:
venv/bin/python -B -m pilot.validate --root runs/qwen35-2b --cleanup-weights

Follow-up decisions after the pilot

• The pilot verified 2,000 tensor pairs and 14,778/20,000 selected block pairs before cutoff. It took about 111 minutes; the final six-test suite including revised figure checks passed in about 77 seconds.
• Hatch missing heatmap cells and label identity values as constructed. Separate layer-distance curves by dimension/scale; retain warnings about within-stratum module/selection bias. Normalize control histograms, display sample counts, and verify their areas equal one. Keep these diagnostics distinct from task-equivalence or statistical-significance claims.
• Avoid concurrent plotting tests and heavy weight IO on the shared host. Record IO wait, swap, CPU and RSS; a low process RSS is not proof of low total resource cost.
• Distinguish tensor/group counts from byte contributions: 1D tensors were about half the inventory count but only 0.0197% of weight bytes in this checkpoint.
• Keep fair candidate selection separate from IO-local execution order. Benchmark batching by source file/tensor without changing the selected pairs or metrics.
• The sampled raw blocks total about 244.5 MiB. Test an explicitly bounded raw BF16/F32 cache against the uncached implementation before adopting it; include fingerprint and temporary conversion memory in the budget. This was not enabled in the current run.
• Report both pre-metadata and net savings. Tiny blocks can have groups yet lose all savings to mapping overhead. Any alternative sparse encoding needs its own explicit accounting.
• Separate low-byte normalization groups from matrix/expert compression opportunities.
• Decide whether approximate-neighbor recall and block coverage justify another bounded pass.
• If meaningful candidates exist, request a small calibration-forward experiment to measure output distortion, then held-out loss/tasks, then implementation-specific memory and latency.
• For MoE, analyze router behavior and joint gate/up/down expert consistency before treating independently similar matrix blocks as interchangeable experts.
• Use these observations to revise this SOP before downloading a second model.

Training-free functional follow-up (design only)

Direct coordinate-aligned parameter similarity is a baseline, not a test of Q/K/V or MLP functional interchangeability. The follow-up uses same-input module probes without solving rotations or training any parameters, then single-site intervention and held-out quality/cost validation. See the bounded plan and primary-source literature. No forward pass, new download, or new model is started by adopting this document. The original static pilot remains inference-free; the proposed follow-up requires a separately confirmed execution environment.

Current functional-run retention override

On 2026-11-09 the user explicitly changed retention: keep the CPU functional pilot checkpoint until the user explicitly requests deletion. Completion, timeout, failure and archive validation do not authorize cleanup. The earlier pilot cleanup remains a historical action; do not apply its automatic cleanup step to the current run.