141 lines
9.3 KiB
Markdown
141 lines
9.3 KiB
Markdown
# Parameter similarity SOP — 2B pilot draft
|
|
|
|
Status: v0.1, executed for the bounded Qwen3.5-2B pilot. See [findings](PILOT_FINDINGS.md).
|
|
Changes proposed from this experience require a small equivalence/performance check before
|
|
the next model; no second model is started automatically.
|
|
|
|
## Scope and decision
|
|
|
|
Analyze official, unquantized safetensors without instantiating a model, running inference,
|
|
training, or executing repository-supplied code. Start with Qwen/Qwen3.5-2B only.
|
|
The question is the number of representative-centered sharing groups under direct weight
|
|
replacement error limits, and the net byte savings of a theoretical shared representation.
|
|
Weight error is not task-accuracy loss, and parameter sharing does not imply fewer FLOPs.
|
|
|
|
## Prerequisites
|
|
|
|
1. Confirm authorized remote host, free disk, memory, CPU, existing work, and access.
|
|
2. Install prebuilt wheels only. Never compile locally or silently use another runner.
|
|
3. Freeze the official repository revision and download file manifest including SHA256.
|
|
4. Allocate checkpoint size plus 3 GB disk margin. Use an experiment-owned directory,
|
|
not a shared Hugging Face cache. Never delete pre-existing weights.
|
|
|
|
## Bounded procedure
|
|
|
|
1. Run synthetic correctness tests on the remote host: BF16, 3D indexing, directional
|
|
replacement error, and non-transitive groups.
|
|
2. Download one checkpoint sequentially and verify upstream sizes and LFS SHA256.
|
|
3. Scan each tensor in 262,144-element chunks. Record shape, dtype, axis role,
|
|
layer, bytes, moments, max magnitude, and finite-value checks. Memory-map weights;
|
|
evict scanned pages. Do not search for exactly equal weight tensors.
|
|
4. Whole-tensor screening: same role, same shape, same dtype. Sample 2,048 aligned,
|
|
evenly spaced elements. Select up to 10 neighbors by each of cosine and relative
|
|
error. Fairly distribute the 2,000-pair exact-verification cap across role groups.
|
|
5. Block screening: non-overlapping 64x64 and 128x128 2D partitions. For 3D tensors,
|
|
use axis-0 slices and bounded volumes adapted to actual dimensions. Keep up to
|
|
256 uniformly reservoir-sampled blocks per role/shape/scale for the 1.7 GiB pilot host
|
|
(below the planned maximum of 512). Record edge bytes
|
|
and sampling coverage. Verify at most 20,000 block pairs. Never extrapolate sampled
|
|
discoveries to unseen blocks or add savings across alternative scales.
|
|
6. Exact metrics: full-element cosine, Pearson, directional replacement errors,
|
|
symmetric error, optimal-scalar residual. Tail diagnostics use samples and say so.
|
|
Controls include random actual comparable pairs, shuffled sampled elements,
|
|
and norm-matched Gaussian vectors. Save sampled versus exact error discrepancies.
|
|
Bounded mode-unfolding SVD probes are diagnostic only, not a replacement test.
|
|
7. Form deterministic star groups at 1%, 2%, 5%, 10%, 20% relative weight error.
|
|
Every member requires an exact verified edge to its representative. Unknown
|
|
pairs remain independent; never use graph connected components as sharing groups.
|
|
8. Compute original bytes minus representatives, 16-byte mapping per analyzed object,
|
|
and 64-byte descriptor per shared group. Fall back to unchanged storage if larger.
|
|
Count savings against the entire model, preserving unsampled blocks and edge regions.
|
|
9. Save raw data, groups, thresholds, coverage, timings, resource usage, figures
|
|
(PNG/PDF/SVG), model metadata and report. Validate every group against exact edges.
|
|
10. Copy results to the deliverable workspace and verify checksums before deleting
|
|
the experiment-owned checkpoint. Save a cleanup record. Do not delete the venv,
|
|
result data or any unrelated cache.
|
|
11. Review full diff, commit and push a work branch with Codex Bot identity, then
|
|
create a PR for manual merge. Never silently initialize/push main or auto-merge.
|
|
|
|
## Budget and stop conditions
|
|
|
|
- Real 2B analysis computation: hard wall-time limit 7,200 seconds. Reserve final time
|
|
for summaries; download and report-generation time are recorded separately.
|
|
- Use one BLAS thread to avoid interference with existing services.
|
|
- Save exact-pair tables every 100 pairs; stage deadlines prevent whole-tensor work
|
|
from consuming the complete block budget.
|
|
- On timeout or failure, retain source weights until partial results are recoverable
|
|
and report coverage/failure truthfully. An internal planned cutoff may produce a complete
|
|
bounded report with partial candidate coverage; always state verified/selected counts.
|
|
An external kill or missing required report is an incomplete run, not a completed experiment.
|
|
- Empty or negligible low-error sharing is a valid negative result. Do not add scales,
|
|
increase error thresholds, align permutations, train or expand to other models automatically.
|
|
- No runtime capability/performance claim without separately authorized evaluation.
|
|
|
|
## Commands (remote, from project root)
|
|
|
|
```sh
|
|
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 timeout 300 venv/bin/python -B -m unittest discover -s tests -v
|
|
venv/bin/python -B -m pilot.download --root runs/qwen35-2b
|
|
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 timeout 7200 venv/bin/python -B -m pilot.analyze --root runs/qwen35-2b --seconds 6600
|
|
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -B -m pilot.plot --root runs/qwen35-2b
|
|
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 timeout 300 venv/bin/python -B -m unittest discover -s tests -v > runs/qwen35-2b/tests.log 2>&1
|
|
venv/bin/python -B -m pilot.audit --root runs/qwen35-2b --analysis-code-commit 6121016
|
|
venv/bin/python -B -m pilot.validate --root runs/qwen35-2b
|
|
# Copy archived results, verify checksums at the destination, then:
|
|
venv/bin/python -B -m pilot.validate --root runs/qwen35-2b --cleanup-weights
|
|
```
|
|
|
|
## Follow-up decisions after the pilot
|
|
|
|
- The pilot verified 2,000 tensor pairs and 14,778/20,000 selected block pairs before cutoff.
|
|
It took about 111 minutes; the final six-test suite including revised figure checks passed in about 77 seconds.
|
|
- Hatch missing heatmap cells and label identity values as constructed. Separate layer-distance
|
|
curves by dimension/scale; retain warnings about within-stratum module/selection bias.
|
|
Normalize control histograms, display sample counts, and verify their areas equal one.
|
|
Keep these diagnostics distinct from task-equivalence or statistical-significance claims.
|
|
- Avoid concurrent plotting tests and heavy weight IO on the shared host. Record IO wait,
|
|
swap, CPU and RSS; a low process RSS is not proof of low total resource cost.
|
|
- Distinguish tensor/group counts from byte contributions: 1D tensors were about half the
|
|
inventory count but only 0.0197% of weight bytes in this checkpoint.
|
|
- Keep fair candidate selection separate from IO-local execution order. Benchmark batching
|
|
by source file/tensor without changing the selected pairs or metrics.
|
|
- The sampled raw blocks total about 244.5 MiB. Test an explicitly bounded raw BF16/F32
|
|
cache against the uncached implementation before adopting it; include fingerprint and
|
|
temporary conversion memory in the budget. This was not enabled in the current run.
|
|
- Report both pre-metadata and net savings. Tiny blocks can have groups yet lose all savings
|
|
to mapping overhead. Any alternative sparse encoding needs its own explicit accounting.
|
|
- Separate low-byte normalization groups from matrix/expert compression opportunities.
|
|
- Decide whether approximate-neighbor recall and block coverage justify another bounded pass.
|
|
- If meaningful candidates exist, request a small calibration-forward experiment to measure
|
|
output distortion, then held-out loss/tasks, then implementation-specific memory and latency.
|
|
- For MoE, analyze router behavior and joint gate/up/down expert consistency before treating
|
|
independently similar matrix blocks as interchangeable experts.
|
|
- Use these observations to revise this SOP before downloading a second model.
|
|
|
|
## Training-free functional follow-up (design only)
|
|
|
|
Direct coordinate-aligned parameter similarity is a baseline, not a test of Q/K/V or MLP
|
|
functional interchangeability. The follow-up uses same-input module probes without solving
|
|
rotations or training any parameters, then single-site intervention and held-out quality/cost
|
|
validation. See the [bounded plan](TRAINING_FREE_FUNCTIONAL_PLAN.md) and
|
|
[primary-source literature](FUNCTIONAL_REUSE_LITERATURE.md). No forward pass, new download,
|
|
or new model is started by adopting this document. The original static pilot remains
|
|
inference-free; the proposed follow-up requires a separately confirmed execution environment.
|
|
|
|
## Current functional-run retention override
|
|
|
|
On 2026-11-09 the user explicitly changed retention: keep the CPU functional pilot
|
|
checkpoint until the user explicitly requests deletion. Completion, timeout, failure
|
|
and archive validation do not authorize cleanup. The earlier pilot cleanup remains
|
|
a historical action; do not apply its automatic cleanup step to the current run.
|
|
|
|
## CPU functional follow-up outcome
|
|
|
|
The executed CPU study found a learned RMSNorm function-repetition example at text
|
|
layer indices 12/13. Use same-input function error and a uniform analytic bound as
|
|
primary evidence, with separate heldout network substitution checks. Costs are not
|
|
a success condition under the latest user instruction. See the
|
|
[executed findings](CPU_FUNCTIONAL_FINDINGS.md) and
|
|
[function-repetition criterion](LEARNED_FUNCTION_REPETITION.md).
|
|
Keep whole-module, neuron, normalization and algebraic-reparameterization results distinct.
|