60 lines
4.0 KiB
Markdown
60 lines
4.0 KiB
Markdown
# Model parameter similarity experiments
|
|
|
|
**[研究图册与固定在线副本](docs/FIGURE_GALLERY.md)**：连续分布、联合消融及原始PDF/SVG。
|
|
|
|
Research on repeated learned functions in **Qwen3.5-2B**: a completed static weight baseline
|
|
and a training-free CPU functional pilot. The CPU study found similar existing RMSNorm
|
|
functions at layer indices 12/13, with a uniform analytic error bound and heldout substitution checks.
|
|
Cost is not a success condition. The completed static pilot was inference-free and its
|
|
weights were deleted after archival; the functional pilot executes streaming FP32 text forwards.
|
|
The current CPU functional checkpoint is retained until the user explicitly requests deletion.
|
|
No model weights are committed here.
|
|
|
|
The latest cross-layer QKV study covers all 15 full-attention layer pairs. Its primary
|
|
protocol had no passing candidate; an exploratory H23-conditioned self-token retrieval
|
|
pattern is reported separately with its fresh-test failures.
|
|
|
|
- [Cross-layer QKV findings and evidence limits](docs/CROSS_LAYER_QKV_FINDINGS.md)
|
|
- [Cross-layer QKV fixed protocol](docs/CROSS_LAYER_QKV_PROTOCOL.md)
|
|
- [CPU learned-function repetition findings](docs/CPU_FUNCTIONAL_FINDINGS.md)
|
|
- [Current learned-function repetition criterion](docs/LEARNED_FUNCTION_REPETITION.md)
|
|
- [CPU-first feasibility and validation](docs/CPU_FIRST_VALIDATION.md)
|
|
- [Training-free functional reuse plan (proposed; not executed)](docs/TRAINING_FREE_FUNCTIONAL_PLAN.md)
|
|
- [Primary-source literature and applicability](docs/FUNCTIONAL_REUSE_LITERATURE.md)
|
|
- [Procedure and limits](docs/SOP.md)
|
|
- [2B pilot findings and limitations](docs/PILOT_FINDINGS.md)
|
|
- [Raw result report, data and figures](results/qwen35-2b/REPORT.md)
|
|
- `pilot/download.py`: revision-pinned official checkpoint download and verification.
|
|
- `pilot/tensors.py`: memory-mapped safetensors/BF16 reader and metrics.
|
|
- `pilot/analyze.py`: bounded tensor/block search, exact verification, representative groups.
|
|
- `pilot/plot.py`: scientific PNG/PDF/SVG figures and report from saved data.
|
|
- `pilot/validate.py`: result/group validation, checksums and guarded weight cleanup.
|
|
- `pilot/audit.py`: finite-metric checks, test evidence and archive provenance.
|
|
- `pilot/run.py`: one-model supervisor with bounded download wait and analysis timeouts.
|
|
- `tests/test_pilot.py`: synthetic numerical and 3D correctness tests.
|
|
|
|
The static baseline reports **verified net byte savings versus direct relative weight error**.
|
|
The functional study separately reports same-input error, an all-input RMSNorm function bound
|
|
and tiny heldout NLL changes; these are not broad task-accuracy guarantees. Different static
|
|
block scales are alternative partitions and their savings must not be summed.
|
|
|
|
Dependencies are pinned in `requirements.txt`; install only prebuilt wheels on the remote host.
|
|
Full resolved versions and run provenance accompany each result. See the SOP for commands.
|
|
|
|
## CPU block activation measurements
|
|
|
|
See [measurement definitions](activation/README.md) and [observed activation report](results/block-activation/REPORT.md) for all 24 layers, contiguous output-channel blocks, exact/near-zero rates and output energy concentration. This is an observational SiLU-gated study, not a pruning or reuse quality claim.
|
|
|
|
## Adjacent-layer head influence concentration
|
|
|
|
The CPU study measures all 23 adjacent transitions, observation-wise effective
|
|
head counts and Q/K/V influence shares, with equal-size weak/strong/random joint
|
|
removal controls. See [definitions and reproduction](dominance/README.md),
|
|
[fixed protocol](dominance/PROTOCOL.md), and [measured report](results/head-dominance/REPORT.md).
|
|
These are descriptive intervention effects, not additive contribution probabilities
|
|
or validated final-LM pruning gains.
|
|
|
|
## Existing GSM8K task-finetuning comparison
|
|
|
|
[Paired study and reproduction](finetune_compare/README.md) · [Fixed protocol](finetune_compare/PROTOCOL.md) · [Results](results/finetune-comparison/REPORT.md). This compares an existing pure-LoRA GRPO checkpoint to exactly the same original release; no training is performed here.
|