Add bounded 2B tensor and block similarity pipeline

docs/SOP.md

@@ -0,0 +1,90 @@
+# Parameter similarity SOP — 2B pilot draft
+
+Status: implementation draft; findings and revisions are added after the first real run.
+
+## Scope and decision
+
+Analyze official, unquantized safetensors without instantiating a model, running inference,
+training, or executing repository-supplied code. Start with Qwen/Qwen3.5-2B only.
+The question is the number of representative-centered sharing groups under direct weight
+replacement error limits, and the net byte savings of a theoretical shared representation.
+Weight error is not task-accuracy loss, and parameter sharing does not imply fewer FLOPs.
+
+## Prerequisites
+
+1. Confirm authorized remote host, free disk, memory, CPU, existing work, and access.
+2. Install prebuilt wheels only. Never compile locally or silently use another runner.
+3. Freeze the official repository revision and download file manifest including SHA256.
+4. Allocate checkpoint size plus 3 GB disk margin. Use an experiment-owned directory,
+   not a shared Hugging Face cache. Never delete pre-existing weights.
+
+## Bounded procedure
+
+1. Run synthetic correctness tests on the remote host: BF16, 3D indexing, directional
+   replacement error, and non-transitive groups.
+2. Download one checkpoint sequentially and verify upstream sizes and LFS SHA256.
+3. Scan each tensor in 262,144-element chunks. Record shape, dtype, axis role,
+   layer, bytes, moments, max magnitude, and finite-value checks. Memory-map weights;
+   evict scanned pages. Do not search for exactly equal weight tensors.
+4. Whole-tensor screening: same role, same shape, same dtype. Sample 2,048 aligned,
+   evenly spaced elements. Select up to 10 neighbors by each of cosine and relative
+   error. Fairly distribute the 2,000-pair exact-verification cap across role groups.
+5. Block screening: non-overlapping 64x64 and 128x128 2D partitions. For 3D tensors,
+   use axis-0 slices and bounded volumes adapted to actual dimensions. Keep up to
+   256 uniformly reservoir-sampled blocks per role/shape/scale for the 1.7 GiB pilot host
+   (below the planned maximum of 512). Record edge bytes
+   and sampling coverage. Verify at most 20,000 block pairs. Never extrapolate sampled
+   discoveries to unseen blocks or add savings across alternative scales.
+6. Exact metrics: full-element cosine, Pearson, directional replacement errors,
+   symmetric error, optimal-scalar residual. Tail diagnostics use samples and say so.
+   Controls include random actual comparable pairs, shuffled sampled elements,
+   and norm-matched Gaussian vectors. Save sampled versus exact error discrepancies.
+   Bounded mode-unfolding SVD probes are diagnostic only, not a replacement test.
+7. Form deterministic star groups at 1%, 2%, 5%, 10%, 20% relative weight error.
+   Every member requires an exact verified edge to its representative. Unknown
+   pairs remain independent; never use graph connected components as sharing groups.
+8. Compute original bytes minus representatives, 16-byte mapping per analyzed object,
+   and 64-byte descriptor per shared group. Fall back to unchanged storage if larger.
+   Count savings against the entire model, preserving unsampled blocks and edge regions.
+9. Save raw data, groups, thresholds, coverage, timings, resource usage, figures
+   (PNG/PDF/SVG), model metadata and report. Validate every group against exact edges.
+10. Copy results to the deliverable workspace and verify checksums before deleting
+    the experiment-owned checkpoint. Save a cleanup record. Do not delete the venv,
+    result data or any unrelated cache.
+11. Review full diff, commit and push a work branch with Codex Bot identity, then
+    create a PR for manual merge. Never silently initialize/push main or auto-merge.
+
+## Budget and stop conditions
+
+- Real 2B analysis computation: hard wall-time limit 7,200 seconds. Reserve final time
+  for summaries; download and report-generation time are recorded separately.
+- Use one BLAS thread to avoid interference with existing services.
+- Save exact-pair tables every 100 pairs; stage deadlines prevent whole-tensor work
+  from consuming the complete block budget.
+- On timeout or failure, retain source weights until partial results are recoverable
+  and report coverage/failure truthfully. Never call an incomplete run complete.
+- Empty or negligible low-error sharing is a valid negative result. Do not add scales,
+  increase error thresholds, align permutations, train or expand to other models automatically.
+- No runtime capability/performance claim without separately authorized evaluation.
+
+## Commands (remote, from project root)
+
+```sh
+OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -B -m unittest discover -s tests -v
+venv/bin/python -B -m pilot.download --root runs/qwen35-2b
+OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 timeout 7200 venv/bin/python -B -m pilot.analyze --root runs/qwen35-2b --seconds 6600
+OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -B -m pilot.plot --root runs/qwen35-2b
+venv/bin/python -B -m pilot.validate --root runs/qwen35-2b
+# Copy archived results, verify checksums at the destination, then:
+venv/bin/python -B -m pilot.validate --root runs/qwen35-2b --cleanup-weights
+```
+
+## Follow-up decisions after the pilot
+
+- Separate low-byte normalization groups from matrix/expert compression opportunities.
+- Decide whether approximate-neighbor recall and block coverage justify another bounded pass.
+- If meaningful candidates exist, request a small calibration-forward experiment to measure
+  output distortion, then held-out loss/tasks, then implementation-specific memory and latency.
+- For MoE, analyze router behavior and joint gate/up/down expert consistency before treating
+  independently similar matrix blocks as interchangeable experts.
+- Use these observations to revise this SOP before downloading a second model.