2.6 KiB
Observed block activation, CPU only
Measures the retained Qwen3.5-2B checkpoint without training or substitutions. This model uses SiLU-gated MLPs and sigmoid attention gates, not ReLU. Leaky ReLU also does not generally create exact zero on negative inputs.
The probe records every linear projection in all 24 layers, plus MLP SiLU and gate-times-up intermediates and full-attention sigmoid gates. It does not record every internal recurrent state, convolution output or attention probability. A block means contiguous output channels (64 or 128), hence contiguous rows of a projection matrix; it is not a Transformer layer or arbitrary 2D tile. MLP gated-product channels correspond to columns consumed by down_proj. It is incorrect to call an individual weight active/inactive based only on its value.
For every sample, module and block, metrics.csv reports:
- exact zero and negative element fractions;
- near-zero fractions at |x| <= 0.01 RMS and <= 0.1 RMS;
- fraction of channels ever above 0.01 RMS over the 32 tokens;
- minimum fraction of channels carrying 90% squared-output energy.
RMS is computed within each observed sample/module/block; width=0 records the whole module. Thus thresholds are explicitly relative and are not shared absolute cutoffs across layers or blocks. Activation rate at a threshold is 1-near-zero rate. A small value or an energy-concentrated output does not prove that a block can be skipped, removed or shared without affecting LM quality. Exact zero is measured in this FP32 streaming implementation of BF16 source weights; other execution dtypes can have different underflow rates.
Four prior authored Chinese/English texts, 32 tokens each, descriptive measurements only. No training, fresh-generalization claim, GPU, weight download or cleanup. The existing streaming implementation was checked against the official implementation in the earlier CPU study; this experiment changes only observation, not model equations.
Remote commands (authorized server 8.211.154.134):
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -B -m activation.run --source runs/cpu-qwen35-2b --out runs/NEW_ACTIVATION_DIR
OPENBLAS_NUM_THREADS=1 venv/bin/python -B -m unittest tests.test_activation -v
venv/bin/python -B activation/report.py runs/NEW_ACTIVATION_DIR
run.py measures; report.py plots saved rows and derives aggregates; __init__.py marks the package; tests/test_activation.py checks exact sparsity, ever-active behavior, all-zero inputs and scale invariance. Raw activations are not saved or needed for these descriptive statistics. Existing caches and weights are retained untouched. Full sample token IDs and CPU time/RSS appear in summary.json.