84 lines
5.1 KiB
Markdown
84 lines
5.1 KiB
Markdown
# Actual-model parameter clouds: frozen grouping and null protocol
|
|
|
|
Assets: retained Qwen3.5-2B original15852e8c16360a2fea060d615a32b45270f8a8fc,
|
|
and pure GSM8K GRPO LoRA seed0 at HF038d8be9bd0edd3883fa019651b02bdf65609468.
|
|
Use the established read-only FP32 W+2BA wrapper. No training, GPU, new models,
|
|
weight mutation, inference requirement or deletion. Every retained asset stays.
|
|
|
|
## What constitutes a distribution
|
|
|
|
A model has no unique canonical number of parameter distributions. Enumerate a
|
|
fixed family, not "all possible distributions":
|
|
1. Each ndim>=2 tensor flattened as [axis0, product(other axes)]: row vectors
|
|
and column vectors, when point count>=16 and feature dimension>=32.
|
|
2. Decoder language/MTP full-attention Q and interleaved gate rows per query
|
|
head; K/V rows per KV head. GDN concatenated Q/K/V rows per corresponding head.
|
|
No claim that these groups are disjoint or iid; whole-matrix views overlap.
|
|
3. Primary sample-size stratum has >=64 available points;16..63 is small-sample
|
|
descriptive only. One-dimensional norm/bias vectors have no cloud of vectors;
|
|
GDN kernel row dimension4 is low dimensional and excluded with explicit reason.
|
|
Vision parameters are covered by matrix/flattened views, not vision-head splits.
|
|
|
|
Include all checkpoint components (language, vision and MTP). Per view choose
|
|
up to512 point indices without replacement, split deterministically into calibration
|
|
and held-out halves. Use full vector coordinates up to dimension8192; only larger
|
|
vectors use a fixed random coordinate subset, labelled projected rather than full.
|
|
Use identical indices, coordinates and null random draws for original/FT.
|
|
No overlap-group parameter savings or total distribution counts are additive.
|
|
|
|
## Centers and radius/scaling
|
|
|
|
Analyze origin and calibration-mean centers separately. Fit R_max using calibration
|
|
norms only; do not silently clip held-out points outside R_max. Also fit calibration
|
|
RMS radius. Store held-out maximum, outside-radius mass and radial spread.
|
|
|
|
Null A: uniform d-dimensional ball. Simulate the SAME center-fitting and calibration
|
|
maximum/RMS fitting before held-out evaluation. For origin, also retain exact
|
|
expected max-fitted null CDF a^d*n_cal/(n_cal+1), a<=1; log probabilities remain
|
|
available and zero Monte Carlo hits are not exact zeros.
|
|
Null B: independent coordinate-uniform box bounded by calibration min/max,
|
|
including asymmetric coordinates. Fit its center/radii using its own calibration
|
|
sample by exactly the same procedure. Record energy/range mismatch and held-out
|
|
coordinate violations; this null is conditional on fitted bounds, not a statement
|
|
that all real vectors lie inside that box.
|
|
Null C: independent coordinate-uniform box with calibration mean mu and standard
|
|
deviation sigma, bounds mu +/- sqrt(3)*sigma (sample variance ddof1). This matches
|
|
first and diagonal second moments in its generating law. Use its own calibration
|
|
and held-out center/max/RMS fits too; RMS-normalized comparisons target shape
|
|
rather than a change of global scale. Matching diagonal moments is not covariance
|
|
matching and does not preserve correlations.
|
|
|
|
Use512 validation null draws (not iid model observations), same calibration count
|
|
as each data cloud. Reuse isotropic-ball nulls only for identical dimension/count
|
|
and center-fitting rules. Monte Carlo resolution/error must be stated. Fit both
|
|
centers from the same raw draws where applicable; paired FT uses common randomness.
|
|
|
|
## Metrics and interpretation
|
|
|
|
Retain measured norm arrays, empirical CDF grids, max-normalized interior fractions
|
|
at0.5/0.75/0.9/0.95/0.99 and outside1, RMS-normalized CDF from0.5 to1.5,
|
|
median radial ratios and radial spread. Prefer probability-point differences over
|
|
astronomical ratios against tiny a^d. Inspect positive AND negative CDF gaps:
|
|
more inner tail plus more outer tail is not global inward stochastic dominance.
|
|
|
|
Record exact zeros, coordinate energy participation (sum squares)^2/sum fourths,
|
|
coordinate-variance participation, empirical covariance/second-moment participation
|
|
via calibration Gram matrices. Label sample rank ceiling n_cal-1 (mean centered)
|
|
or n_cal (origin); compare to equally sampled nulls, do not interpret the raw
|
|
finite-sample participation as true intrinsic dimension.
|
|
|
|
Report both full-radius and moment-controlled results for each center, whole-view
|
|
and head subgroups, primary versus small sample/projection strata, original and
|
|
changed72-LoRA-target groups. Unchanged parameter views can reuse identical results
|
|
with explicit references rather than re-read/recompute. Results describe parameters,
|
|
not activations, model quality, functional duplicates, or pruning gains.
|
|
|
|
Execution budget: one CPU, 512 MiB RSS ceiling, 7200 seconds per invocation;
|
|
checkpoint every group, resume without replacing completed groups. Any incomplete
|
|
coverage is reported before interpretation. No numerical analysis runs locally.
|
|
|
|
Illustrative radial CDFs use these fixed original/origin views, selected by module
|
|
role before the census completes, not by the resulting similarity/interior score:
|
|
language layer0 linear-attention Q head0, layer3 full-attention Q head0, and layer0
|
|
MLP up-projection rows. Their plots are illustrations, not independent replications.
|