46 lines
2.6 KiB
Markdown
46 lines
2.6 KiB
Markdown
# Observed block activation, CPU only
|
|
|
|
Measures the retained Qwen3.5-2B checkpoint without training or substitutions.
|
|
This model uses SiLU-gated MLPs and sigmoid attention gates, not ReLU.
|
|
Leaky ReLU also does not generally create exact zero on negative inputs.
|
|
|
|
The probe records every linear projection in all 24 layers, plus MLP SiLU
|
|
and gate-times-up intermediates and full-attention sigmoid gates. It does not
|
|
record every internal recurrent state, convolution output or attention probability.
|
|
A block means contiguous **output channels** (64 or 128), hence contiguous
|
|
rows of a projection matrix; it is not a Transformer layer or arbitrary 2D tile.
|
|
MLP gated-product channels correspond to columns consumed by down_proj.
|
|
It is incorrect to call an individual weight active/inactive based only on its value.
|
|
|
|
For every sample, module and block, metrics.csv reports:
|
|
- exact zero and negative element fractions;
|
|
- near-zero fractions at |x| <= 0.01 RMS and <= 0.1 RMS;
|
|
- fraction of channels ever above 0.01 RMS over the 32 tokens;
|
|
- minimum fraction of channels carrying 90% squared-output energy.
|
|
|
|
RMS is computed within each observed sample/module/block; width=0 records the
|
|
whole module. Thus thresholds are explicitly relative and are not shared absolute
|
|
cutoffs across layers or blocks. Activation rate at a threshold is 1-near-zero rate.
|
|
A small value or an energy-concentrated output does not prove that a block can
|
|
be skipped, removed or shared without affecting LM quality. Exact zero is measured
|
|
in this FP32 streaming implementation of BF16 source weights; other execution dtypes
|
|
can have different underflow rates.
|
|
|
|
Four prior authored Chinese/English texts, 32 tokens each, descriptive measurements
|
|
only. No training, fresh-generalization claim, GPU, weight download or cleanup.
|
|
The existing streaming implementation was checked against the official implementation
|
|
in the earlier CPU study; this experiment changes only observation, not model equations.
|
|
|
|
Remote commands (authorized server 8.211.154.134):
|
|
```
|
|
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -B -m activation.run --source runs/cpu-qwen35-2b --out runs/NEW_ACTIVATION_DIR
|
|
OPENBLAS_NUM_THREADS=1 venv/bin/python -B -m unittest tests.test_activation -v
|
|
venv/bin/python -B activation/report.py runs/NEW_ACTIVATION_DIR
|
|
```
|
|
|
|
run.py measures; report.py plots saved rows and derives aggregates;
|
|
__init__.py marks the package; tests/test_activation.py checks exact sparsity,
|
|
ever-active behavior, all-zero inputs and scale invariance. Raw activations are
|
|
not saved or needed for these descriptive statistics. Existing caches and weights
|
|
are retained untouched. Full sample token IDs and CPU time/RSS appear in summary.json.
|