Compare commits

9 Commits

211 changed files with 784862 additions and 0 deletions
+6
View File
@@ -0,0 +1,6 @@
# Scientific artifacts must retain the bytes recorded in checksums.json.
results/** -text
# Generated CSV uses RFC 4180 line endings; Matplotlib SVG paths contain spaces.
results/**/*.csv -whitespace
results/**/*.svg -whitespace
results/**/*.log -whitespace
+10
View File
@@ -0,0 +1,10 @@
__pycache__/
.venv/
weights/
*.partial
*.log
# Large reproducible functional activations remain on the experiment server.
results/cpu-qwen35-2b/baseline-activations.npz
results/cpu-qwen35-2b/neuron-reuse-activations.npz
results/cpu-qwen35-2b/norm-repetition-input-profiles.npz
+34
View File
@@ -0,0 +1,34 @@
# Model parameter similarity experiments
Research on repeated learned functions in **Qwen3.5-2B**: a completed static weight baseline
and a training-free CPU functional pilot. The CPU study found similar existing RMSNorm
functions at layer indices 12/13, with a uniform analytic error bound and heldout substitution checks.
Cost is not a success condition. The completed static pilot was inference-free and its
weights were deleted after archival; the functional pilot executes streaming FP32 text forwards.
The current CPU functional checkpoint is retained until the user explicitly requests deletion.
No model weights are committed here.
- [CPU learned-function repetition findings](docs/CPU_FUNCTIONAL_FINDINGS.md)
- [Current learned-function repetition criterion](docs/LEARNED_FUNCTION_REPETITION.md)
- [CPU-first feasibility and validation](docs/CPU_FIRST_VALIDATION.md)
- [Training-free functional reuse plan (proposed; not executed)](docs/TRAINING_FREE_FUNCTIONAL_PLAN.md)
- [Primary-source literature and applicability](docs/FUNCTIONAL_REUSE_LITERATURE.md)
- [Procedure and limits](docs/SOP.md)
- [2B pilot findings and limitations](docs/PILOT_FINDINGS.md)
- [Raw result report, data and figures](results/qwen35-2b/REPORT.md)
- `pilot/download.py`: revision-pinned official checkpoint download and verification.
- `pilot/tensors.py`: memory-mapped safetensors/BF16 reader and metrics.
- `pilot/analyze.py`: bounded tensor/block search, exact verification, representative groups.
- `pilot/plot.py`: scientific PNG/PDF/SVG figures and report from saved data.
- `pilot/validate.py`: result/group validation, checksums and guarded weight cleanup.
- `pilot/audit.py`: finite-metric checks, test evidence and archive provenance.
- `pilot/run.py`: one-model supervisor with bounded download wait and analysis timeouts.
- `tests/test_pilot.py`: synthetic numerical and 3D correctness tests.
The static baseline reports **verified net byte savings versus direct relative weight error**.
The functional study separately reports same-input error, an all-input RMSNorm function bound
and tiny heldout NLL changes; these are not broad task-accuracy guarantees. Different static
block scales are alternative partitions and their savings must not be summed.
Dependencies are pinned in `requirements.txt`; install only prebuilt wheels on the remote host.
Full resolved versions and run provenance accompany each result. See the SOP for commands.
+75
View File
@@ -0,0 +1,75 @@
# CPU 先导:先找值得继续验证的信号
更新:2026-11-09。目的不是在 CPU 上完成广泛能力及高吞吐评测,
而是低成本判断实际输入分布上是否存在值得 GPU 进一步验证的复用候选。
全程不训练,不求 R。
## 已执行的容量检查
服务器 8.211.154.134:2 核 x86_64,约 1.7 GiB RAM;检查时可用约 892 MiB,
swap 已用约 2.2 GiB,根盘可用约 73 GiB。共享主机快照不是实验独占资源。
NumPy 2.5.4,OPENBLAS_NUM_THREADS=1、OMP_NUM_THREADS=1,远程 timeout 60 秒。
随机种子 42、float32、X 为 32×2048;每种 W 随机生成后连续执行 4 次 X@W,
第 1 次作为冷调用,其余 3 次单独报告。随机输入仅用于容量测量。
| W 形状 | W 大小 | 首次乘法 | 后续三次乘法(秒) |
|---|---|---|---|
| 2048×2048 | 16 MiB | 0.027816 | 0.027035、0.027207、0.025878 |
| 2048×6144 | 48 MiB | 0.086039 | 0.082543、0.110806、0.075336 |
输出均为有限数,进程峰值 RSS 88,220 KiB(约 86.2 MiB)。
耗时只包含矩阵乘法,不包含权重生成、磁盘读取、BF16 转换、模型其余操作及框架开销。
它支持继续评估小批量 CPU 投影计算,不足以外推完整模型时延或宣称发现可复用性。
没有模型权重参与,没有编译、下载或模型前向。
## 执行路线与证据等级
1. **实现与资源可行性**:核对固定 revision 的位置编码、归一化、GQA 和门控,
使用受信任的算子实现及小数组正确性检查;模型仓库代码不直接执行。
先确认首个短样本能以受控内存运行。不用不正确的“简化 Qwen”产生看似有价值的结果。
2. **真实分布上的候选**:batch=1、32 tokens,校准和留出各 4 段真实文本。
按顺序加载、计算、释放每层,保留很小的入口激活;探测预先固定的 3 个 full-attention 层对、2 个 MLP 对,以及 6 个 full-attention 层内的 Q/K 归一化模块对,共 11 对。
每对在两个来源层的隐藏输入上分别测试,记录 JS、top-k、完整分支输出误差。
3. **最小干预证据**:最多对 1 个候选做单处替换;冻结参数,在留出文本上看输出变化,
条件允许时流式计算 token NLL。词表输出头分块归约 logsumexp,避免一次物化大 logits。
若没有正确且可承受的干预实现,就止于“候选”,不称为验证成功。
4. **GPU 细化**:候选通过才扩展长度、样本、线性注意力、多处共享及质量—成本曲线。
先导主证据来自真实文本产生的隐藏状态,而不是高维空间上的均匀随机向量。
直接按真实样本出现频率聚合,就能近似按目标使用分布加权,不先引入昂贵的高维密度估计。
保留逐文本及尾部误差,少量低频但关键用例不被平均值掩盖。
## CPU 内存与停止约束
- 不同时加载完整模型的 BF16/FP32 副本;模型原始权重约 4.55 GB,大于当前 RAM。
- 单层/子矩阵流式读取,及时解除映射或释放页面;先设本进程 RSS 512 MiB 软停止线,
系统可用内存低于 256 MiB 或持续 swap 活动就停止并保存进度,不改全局 swap 配置。
软停止线不替代实际峰值检测,分配前须计算临时内存。
- 先实现最小正确路径;线性状态递推、长上下文和生成速度不塞入首轮。
- 总计算 120 分钟,阶段预算见功能方案;无候选、超时或数值/实现核对失败均停止。
- 容量探测已完成,真实模型阶段尚未执行。后续需要恢复本实验自有的固定 revision 权重,
权重持续保留,只有用户明确要求删除时才清理;不增加模型规模或自动租用 GPU。
CPU 上“找到输出接近的候选”能支持继续投入 GPU;
“在指定小样本中单处复用保持质量且节省明确资源”才是初步正例。
两者都不能替代大规模能力验证。CPU 时延不外推为 GPU 加速率。
## 当前权重保留规则
用户于 2026-11-09 明确要求:本轮模型权重保留,只有用户明确要求删除时才删除。
试验完成、超时、失败、归档或结果校验均不得自动触发清理。该要求覆盖先前逐模型分析后删除的约定。
上一轮已删除的历史记录保留;本轮恢复下载的权重受新规则保护。
## 本轮执行补充
实际实验分支为 `codex/cpu-reuse-experiment`。在读取真实权重/功能指标前,新增固定的
6 个层内 Q/K RMSNorm 增益共享候选,以检查较小粒度的功能复用;不是看过留出结果后扩充搜索。
最多选择一个候选,必须分别验证两个替换方向;完整模块失败不会被小模块通过所掩盖。
归一化共享最多节省少量字节,不预设能显著减少计算。
采集真实隐藏输入必须正确执行前置线性注意力;它不进入首轮复用候选,但其递推计算纳入官方参考对照。
## 用户最新目标修订
不再以成本/加速作为成功条件;主问题是现有学习部分的功能重复。
新增一次固定四层的 MLP 神经元函数搜索、独立留出和双向替换,
见 [功能重复判据](LEARNED_FUNCTION_REPETITION.md)。解析增益合并单独归类,不冒充学习部分重复。
+86
View File
@@ -0,0 +1,86 @@
# CPU 先导结论:存在已训练归一化模块的相似功能重复
本轮主目标按用户最新要求为功能重复的存在性,不以成本/加速为成功条件。
全部计算在 2 核、约 1.7 GiB RAM 的 CPU 服务器完成,无训练、缩放拟合或旋转搜索。
层编号均为从 0 开始的索引。
## 一个具体正例
固定官方 Qwen3.5-2B checkpoint 的两个独立已训练模块:
- `model.language_model.layers.12.input_layernorm`
- `model.language_model.layers.13.input_layernorm`
二者均为 2048→2048 的 RMSNorm,eps=1e-6,使用训练后的有效增益 gamma=1+weight。
增益标准差分别约 0.1901、0.1685,最小值约 0.5547、0.5703;不是零函数或恒定增益。
| 证据 | 结果 |
|---|---|
| 全输入域数学界(相对于层12输出) | 输出相对 L2 误差≤10.1852% |
| 反方向全输入域数学界(相对于层13输出) | 输出相对 L2 误差≤11.3402% |
| 4 条留出文本、两个来源层输入的最坏双向误差 | 4.5285% |
| 相同留出比较中的最低余弦 | 0.999487 |
| 层12使用层13,留出平均 token NLL 增量 | +0.0000325651 nat/token |
| 层13使用层12,留出平均 token NLL 增量 | −0.0003639730 nat/token |
两方向逐文本 NLL 增量 P95 分别约 0.00098464、0.00070972,均通过原定小样本质量门槛。
所有数值来自保存的逐文本、逐 token 原始数据。留出为 4 条不同的实验原创短文本,
每条截取 32 tokens;校准另有 4 条。它们不是广泛任务能力评测。
## 为什么不只是“权重看起来接近”
对相同 eps 和维数,f_a(x)=diag(a)x/sqrt(mean(x²)+eps)。任意非零实数输入满足:
||f_a(x)-f_b(x)|| / ||f_a(x)|| ≤ max_j |a_j-b_j|/|a_j|。
共同归一化分母消去;坐标基向量可达到最大值,因而该界是精确的最坏情况。
反向分母换成 |b_j|。本轮先穷举 24 层同角色 input/post-attention norm 的 552 对,
仅按最小双向最坏界选定一对,未使用任何输入样本作选择。≤20% 界有 18 对,≤10% 无配对;
本轮只验证最优的一对,不根据留出结果换选。
这个确定性界直接给出两个已训练函数整体接近的证据。真实输入上的误差更小,
与只关心目标输入分布的目标一致。浮点实测与实数数学界分别报告。
重新采集输入时,未修改的完整前向逐 bit 复现了最初保存的基线输出。
输入探测分别使用两个来源层,对相同输入同时应用两套增益,避免比较不同输入的输出。
从已加增益的激活还原共同归一化输入时有额外 FP32 舍入,未将其称为 bitwise 等价。
结论限定为两个已有 RMSNorm 模块实现相似映射;不把共同包含 RMS 操作本身当作证据。
二者学习增益不同且非恒定,输出非零。完整 attention/MLP 主体的重复仍未得到正例。
## 已完成的阴性搜索
1. 原定 3 对完整 attention、2 对完整 MLP、6 对层内 Q/K norm 均未通过预设门槛。
attention 检索分布 JS 范围约 0.145~0.318,也未通过 0.01 的门槛;
未进行头排列、连续坐标变换或训练对齐,不能外推为不存在其他可复用表示。
2. 在层 5、6、20、21 各 6144 个 MLP scalar 子函数中,排除低活跃和近常量单元后,
共精确筛查 7,450,139 个唯一同层配对。要求双向相对误差≤20%、余弦≥0.98、
Pearson≥0.95,无通过者。没有针对留出集不断更换候选或放宽门槛。
3. 这些结果只覆盖声明的模块、输入和阈值,不否定其他粒度/角色的功能重复。
## 单独归类的解析恒等式
Q/K 未参与 RoPE 的 192 个维度,其增益只以乘积影响点积,可合并到任一侧。
96 组真实输入检查最大分数差 3.8147e-6、最大概率差 8.0466e-7;固定第3层两个方向
留出 NLL 变化接近 FP32 舍入。详见 [推导](QK_GAIN_FACTORIZATION.md)。
这是重参数化等价,不作为“两个已有学习部分功能重复”的正例。先前性能测试原始数据
保留用于溯源,成本不属于当前结论的成功条件。
## 正确性、资源与保留规则
- 固定模型 revision:15852e8c16360a2fea060d615a32b45270f8a8fc,原始权重 SHA256 校验通过。
- 流式读取原始 BF16,逐块转为 FP32,不同时载入全模型。
- 与固定 Transformers 官方定义比较的 22 项检查通过,最大差约 5.4e-7。
- 保留早期参考适配器错误、临时分区空间不足及两次过于保守的换页停止记录。
只归还自身临时堆页,未改变系统 swap 设置;后续有界前向完成。
- 计算预算跨重启累计,初始保守计入 180 秒先前检查预留;具体时长及峰值见归档日志。
- 当前权重按用户最新指令持续保留,只在用户明确要求删除时清理。
原始表、样本/token IDs、激活、NLL、证明界与图表见 `results/cpu-qwen35-2b/`。
后续 GPU 阶段可增加长度、语言和任务来验证正例的稳健性;本轮不自动启动新模型或扩展搜索。
## Git 交付与保留缓存
全部图表源 CSV/JSON、逐 token NLL、证明界和日志已交付。三份较大的中间激活缓存
保留在原服务器及本地完整归档,路径、大小、SHA256 见结果目录的 external-activation-cache.json。
其完整原实验校验表为 checksums-full-server.json;Git交付包校验表为 checksums-functional.json。
没有删除模型或缓存;这些缓存可由固定代码和样本重新生成,也可从服务器按哈希取回。
+23
View File
@@ -0,0 +1,23 @@
# 功能复用文献核对
核对日期:2026-11-09。以下为原始论文摘要及相关章节核对,不是复现实验。
本项目约束:不训练任何参数;已有静态试验没有执行模型前向。下面的方法不自动获得执行授权。
| 原始来源 | 研究的共享对象 / 证据 | 训练条件与适用限制 | 对本项目的用途 |
|---|---|---|---|
| [LISA, 2408.01890v2](https://arxiv.org/html/2408.01890v2), §3–5 | 跨层 token 注意力分布;考察头对齐、浅层敏感性;基准能力和吞吐验证 | 学习头对齐网络及低秩差异修正;不能归类为免训练替换。论文的 attention weights 在此主要指注意力分布,不是 W_Q/W_K 权重 | 借鉴逐头比较、首 token 对照和替换敏感性验证;不采用训练模块 |
| [CLA, 2405.12981](https://arxiv.org/abs/2405.12981) | 相邻层共享 K/V heads,降低 KV cache;语言模型能力验证 | 摘要明确实验为从头训练 1B/3B 模型;不是现成 Qwen 的直接合并证据 | 区分 KV 缓存收益和参数/计算收益;作为架构可行性依据 |
| [Systematic Study, NAACL 2025](https://aclanthology.org/2025.naacl-short.34/) | 系统比较跨层 KV 共享配置;语言建模、下游任务及生成吞吐 | 摘要报告激进配置涉及额外训练及 prefill 代价;不能由摘要推断零训练可迁移性 | 复用方向、共享拓扑及 prefill/decode 成本须分别记录 |
| [MiniLM, 2002.10957v2](https://arxiv.org/abs/2002.10957) | 注意力分布及 value-value 关系蒸馏 | 明确训练学生模型;本项目不进行蒸馏 | 提醒检索分布和 value 内容需要分开考察 |
| [CKA, ICML 2019](https://proceedings.mlr.press/v97/kornblith19a.html) | 表征之间的关系相似性,不是模块替换实验 | 论文讨论表征比较及可逆变换不变性的局限;高 CKA 不是可替换保证 | 可作为候选诊断,不能取代输出误差、干预和成本评测 |
| [ALBERT, 1909.11942v6](https://arxiv.org/html/1909.11942v6), §3.1/4.5 | 跨层参数共享及训练后能力验证 | 共享架构参与训练;不能直接外推为已训练 checkpoint 可无损合并 | 参数存储减少与前向运算减少分别记账 |
## 对当前试验的推论
以上文献给出研究动机,没有证明 Qwen3.5-2B 已存在免训练的有效复用组。
静态权重试验也没有否定该可能性。应使用同输入功能探测筛选,再对实际计算图实施单处干预。
Qwen 的 full attention 与线性注意力需分别定义探测量;不能把 softmax attention 的结论照搬到状态递推模块。
数学工具参考:[SciPy orthogonal_procrustes](https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.orthogonal_procrustes.html)。
纯正交最小二乘有直接的矩阵分解解法,无须枚举旋转矩阵;但求解一个对齐并不自动带来可省略的计算。
本项目下一阶段默认不求 R,直接测目标函数和干预效果。
+58
View File
@@ -0,0 +1,58 @@
# 当前目标:已有学习部分是否重复实现相似函数
用户于 2026-11-09 明确:不以成本或加速作为成功条件,只关心相似功能重复是否存在。
权重继续保留,只有用户明确要求删除才清理;不训练、不拟合旋转或缩放。
## 什么可以作为正例
两个不同的已训练部分,在同一真实输入上产生接近、且并非恒定/接近零的输出;
独立留出输入仍满足预先固定的门槛。实际双向替换只作补充验证。
不能仅因替换一小部分后整模型 NLL 几乎不变,就认定它们实现相同函数。
Q/K 增益的解析合并证明同一计算的重参数化自由度,不等于两个现成学习部分功能重复。
前一轮 11 对完整/归一化模块未通过门槛,也不能否定更小粒度的重复。
不将两个模块都包含 SiLU、矩阵乘法等共同运算本身作为重复功能证据。
## 新增一次有界搜索:MLP 学习神经元
在已保存的第 5、6、20、21 层真实 MLP 入口输入上,研究
f_(l,j)(h) = SiLU(w_gate_(l,j)^T h) × (w_up_(l,j)^T h)。
每层 6,144 个神经元具有不同学习参数。比较同一层内两个 scalar 子函数,
不是比较来自不同输入的两个观测值,也不是把整层或整 tensor 称作重复。
下游 down-projection 的列保持各自原样;替换发生在两个投影和 SiLU 之后的激活值处。
1. 4 条校准文本、4 条留出文本,每条 32 tokens,中文/英文各半,沿用先前冻结样本。
2. 校准阶段只保留 RMS 至少为该层神经元中位数、std/RMS≥0.25、能量有效 token 数≥16
的非平凡单元。能量有效样本数定义为 (sum f²)² / sum f⁴,不是独立统计样本量。
3. 对这些单元做分块穷举;实际配对数必须记录。不用随机 hidden 向量作主要证据。
4. 预先固定门槛:双向直接相对 L2≤20%、余弦≥0.98、Pearson≥0.95;同时输出 5/10/20%
三档校准配对数量。没有拟合线性缩放、偏置或 R。
5. 全部层中仅按校准的最小最坏方向误差选择 1 对,冻结选择;无通过者停止。
6. 留出集要求同样的汇总函数门槛及活跃度条件,逐文本双向误差≤35%。不根据留出失败换候选。
7. 只有函数验证通过,才做两个方向的独立网络替换;留出平均 NLL 增量≤0.01 nat/token,
逐文本增量 P95≤0.05。它不是主要的功能重复判据,防止因单元贡献很小而得到假正例。
本阶段使用先前两小时累计预算的剩余额度,复用现有隐藏输入;不增加模型、样本规模或成本测试。
神经元层面的经验正例不证明完整层可以互换、不证明全输入空间等价,也不能推断独立任务能力。
留出只有 4 条文本,结论必须限定到此短文本先导分布;后续扩展文本才能检验稳健性。
## 同角色学习 RMSNorm:确定性函数误差界
静态试验已经发现部分语言层 post-attention norm 的参数接近。新增一次独立检验:
比较 24 层各自的 input/post-attention RMSNorm(同角色内共 552 对),
仅按训练后增益的全输入最坏误差界选 1 对,完全不依赖校准或留出输入。
增益需非零且不是常量向量(std≥0.001),不把没有学习参数的共同归一化运算当作正例。
相同 eps、相同形状下,f_a(x)=diag(a)x/sqrt(mean(x²)+eps)。对任意非零 x,
||f_a(x)-f_b(x)|| / ||f_a(x)|| ≤ max_j |a_j-b_j|/|a_j|。
反方向分母换为 |b_j|。该上界由相应坐标基向量达到,是实数算术下的精确最坏界。
因此可先证明两个现成学习模块的函数整体接近,而不只比较原始参数 w;实际增益为 1+w。
选择门槛:最坏双向界≤20%;同时记录≤5%、≤10%、≤20% 配对数量。只选最优界的一对。
随后重新采集其实际输入,要求留出逐文本直接双向输出误差≤5%,再测两个方向网络替换。
函数接近与替换后网络 NLL 分开报告;后者沿用已有 0.01/0.05 门槛,不决定函数存在性本身。
选定后不根据留出结果换配对;不搜索视觉模块、不增加模型、不以成本为条件。
归一化层正例不能推广为 attention 或 MLP 主体存在同样重复。
+132
View File
@@ -0,0 +1,132 @@
# Qwen3.5-2B 有界试验结果
本轮已完成约定时间预算内的试验。整 tensor 精确比较 2,000 对;子块在内部截止时
完成 14,778 / 20,000 对(73.89%),其余 5,222 对未计算。不能把本轮写成全量搜索。
结论:在本次候选、坐标对齐及误差阈值范围内,没有获得显著的直接权重共享收益。
这不等于证明该模型不能通过其他方法压缩,也不代表 MoE 模型具有同样结果。
## 已确认的试验范围
- 模型:官方 `Qwen/Qwen3.5-2B`,revision `15852e8c16360a2fea060d615a32b45270f8a8fc`。
- 原始 safetensors 文件为 4,548,221,488 字节,已核对官方 LFS SHA256。
- 共 632 个存储 tensor;权重数据本身为 4,548,144,832 字节。
- 实际存储参数 2,274,069,824 个,包含官方 checkpoint 中的视觉部分。
- 维数分布:319 个一维、294 个二维、18 个三维、1 个五维。
- 三维权重是 `[6144, 1, 4]` 的卷积核,包含长度为 1 的轴;不能把它解释为一般三维体积。
- 五维视觉权重纳入完整清单和全模型字节分母,本轮不扩展五维子块搜索。
- 不执行前向计算、能力评测、训练或蒸馏。
## 整 tensor 的结果
已完成 2,000 对同角色、同形状候选的精确比较。它们是经过近邻筛选的候选,
不是所有可能 tensor 对的穷举。
| 最大直接替换相对误差 | 非单成员共享组 | 最大组大小 | 净节省字节 | 占模型权重 |
|---|---:|---:|---:|---:|
| 1% | 0 | 1 | 0 | 0% |
| 2% | 0 | 1 | 0 | 0% |
| 5% | 1 | 2 | 0 | 0% |
| 10% | 11 | 6 | 32,704 | 0.000719% |
| 20% | 16 | 15 | 119,424 | 0.002626% |
20% 阈值下找到的共享组均为一维权重。组数增加不代表显著的字节收益。
这里的百分比是权重误差,绝不是任务能力损失百分比。
净节省按本轮声明的理论布局计算:保留组代表,每个分析对象 16 字节映射,
每个共享组 64 字节描述;若大于原存储,则保持原表示。
这部分结果不支持在本次搜索范围内通过直接共享整 tensor 显著压缩该 checkpoint。
它不排除其他未比较组合、子块共享、重参数化或带残差的压缩方法。
## 子块结果
共筛选 13,328 个抽样子块。20% 直接替换误差阈值下:
这里的 tensor 指完整命名权重数组,block 指其局部子张量,不是 Transformer block。
三维原始权重为 6144×1×4,本轮切片实际是 1×1×4(4 个数),局部块是
4×1×4(16 个数);小块的高余弦不能推广为整份三维权重相似。
| 表示 | 非单成员共享组 | 最大组 | 映射开销前节省 | 当前布局净节省 |
|---|---:|---:|---:|---:|
| 64×64 二维块 | 3 | 3 | 40,960 字节 | 0 |
| 128×128 二维块 | 1 | 2 | 32,768 字节 | 0 |
| 三维 axis-0 切片 | 45 | 12 | 872 字节 | 0 |
| 三维局部块 | 2 | 2 | 64 字节 | 0 |
二维共享候选均来自 `model.visual.pos_embed.weight`,未在已复核候选中形成大规模
投影矩阵共享。不同块尺度是替代方案,不能相加。
净收益为零与声明的映射成本有关,不应解读为完全没有相似子块。更稀疏或更紧凑的
索引格式可能降低开销,但本轮找到的二维毛节省也仅为几十 KB,尚不足以支持大幅
减容。三维切片组数量较多,但整组三维权重合计只占模型字节的约 0.01945%。
模型字节占比:一维 0.01972%、二维 99.89167%、三维 0.01945%、五维 0.06917%。
因此,应按字节贡献评估价值,不能按 tensor 数量或共享组数量推断收益。
全部分组是满足代表半径约束的可行方案,不保证达到最少组数或最优压缩率。
## 运行经验
1. 实际服务器内存约 1.7 GiB,新增的 60 GB 是磁盘扩容,不是内存。
使用分块 BF16 解码和 mmap 页面释放完成了完整权重扫描与整 tensor 比较。
2. 只看进程 RSS 不足以解释运行成本。`vmstat` 快照观察到 75%~85% 的 I/O 等待,
同时存在交换活动;这些是共享主机的快照,不能全部归因于本进程。
3. 与权重读取并行的绘图合成测试触发过 300 秒超时;观察到测试处于磁盘等待。
后续验证应在权重读取结束后串行执行,避免把资源争用误判为数值算法失败。
4. Matplotlib 图例改为固定位置,避免自动定位引入额外布局开销。
5. 公平地选择各角色的候选,与高效地安排读取顺序,是两个不同问题。
下一轮应评估“先固定公平候选集合,再按文件/源 tensor 分批执行”的效果,
比较是否得到相同指标及更低 I/O 开销;本轮不在中途改变候选集合。
6. 从已保存元数据计算,13,328 个抽样子块的原始数据合计 256,387,072 字节
(约 244.5 MiB),对应指纹约 100.1 MiB。这是下一轮原始子块缓存的容量估计,
不是本轮已经实现的缓存或实测加速。应先用小样本验证数值一致性、峰值内存和
I/O 改善,再决定是否作为默认方案;不能仅凭容量估计保证运行效率。
参数分析及分组汇总耗时 6,661.13 秒(约 111.0 分钟),分析进程峰值 RSS 约
201.3 MiB;这不包含系统文件缓存和同机服务内存。内部比较截止后保存覆盖率与
分组,整个参数分析未触及 7,200 秒进程硬上限。块比较阶段将本进程 IO 优先级
调整为 best-effort 7,未改变服务器全局设置。
最终修图版 6 项串行合成测试通过(约 77.2 秒),六类图表的 PNG/PDF/SVG 已生成,
并对全部六类 PNG 做视觉检查;程序还核验了分组成员与代表之间的精确误差边。
## 三张诊断图的修订与解读
- 热力图只展示 `linear_attn.in_proj_a.weight`(16×2048),不是整个层。18 个相关层
的 153 种配对中核验了 74 对;已测非对角余弦约 0.011~0.298。
对角线 1 是身份比较,不是跨层发现。未测格子改用斜线,避免与接近零的底色混淆。
- 原混合层距曲线受维数组成影响:层距 1 含 183 对一维、19 对二维;层距 8
含 52 对一维、75 对二维。一维余弦中位数约从 0.941 变为 0.883,
不能将混合曲线的陡降全部归因于层距。修订版按维数/块尺度分开,模块和筛选偏差仍存在。
- 随机对照分为保留分布的元素打乱、匹配范数的零均值高斯向量、真实同角色随机配对。
元素打乱后仍高余弦的情况主要出现在一维归一化、偏置和 A_log,提示均值/分布影响。
修订版按维数分面、按密度归一化,并显示样本数,避免将不同样本量的峰高当概率比较。
它们是诊断对照,不是显著性检验,也没有完全匹配每个模块的采样构成。
三维对照分层仅有约 3~6 个样本,尤其不能据其峰形作总体推断。
这次只从已有 CSV/JSON 重绘并导出诊断表,模型权重已删除,没有增加参数比较或重新运行模型。
## 限制及后续决策
- 只研究一个 checkpoint;近邻筛选、同角色分组和子块抽样均可能漏掉其他共享机会。
- 直方图和层距曲线存在候选筛选与模块混合偏差,不能用作总体分布或独立样本推断。
- 光谱/子空间分析仅使用有界子体积;分布或子空间相似不等于可直接替换。
- 本轮没有测量能力损失、困惑度、运行内存或推理速度,也没有产出可运行的压缩模型。
- 大幅减少推理计算量需要进一步的计算图和路由分析,单纯共享存储不能保证加速。
下一轮先讨论候选选择与执行局部性、原始子块缓存、映射格式和阶段预算,
再修订 SOP;不自动下载第二个模型。只有获得额外授权后,才进行校准前向及
留出任务测试,以验证替换造成的能力损失。
数据与标准化报告见 `../results/qwen35-2b/`;具体阈值、分组、覆盖率、来源和校验记录
随原始图表数据归档。权重在交付副本校验后按授权删除,重现实验需要按固定 revision
重新下载。
## 功能复用假设仍待验证
本轮坐标对齐权重结果不能排除 Q/K 的联合功能接近或 MLP 分支可复用。
下一阶段采用同一隐藏输入下的函数比较,不训练参数、不默认求正交 R;
注意力分布、完整分支输出、真实网络单处替换和运行收益分别验证。
详见 [免训练有界方案](TRAINING_FREE_FUNCTIONAL_PLAN.md) 与
[文献核对](FUNCTIONAL_REUSE_LITERATURE.md)。这些是方案,尚未执行。
+46
View File
@@ -0,0 +1,46 @@
# Q/K 未旋转维度的确定性增益合并(已数值验证,归为重参数化)
这是从已核对架构直接推导的恒等式,不是看留出指标后扩大近邻搜索。
原定 11 对候选保留原协议;本项是独立的解析对照,不训练、不求 R,也不选择层。
预先固定完整网络干预层为第 3 层,分别验证两个移动方向;不根据数据换层。
## 恒等式与前提
以完成 RMS 单位化、尚未乘可学习增益的 q/k 为 u/v,Q/K 增益向量为 a/b。
固定 checkpoint 的 norm 参数实际使用 a=1+w_Q,b=1+w_K。
模型 head_dim=256,partial_rotary_factor=0.25:前 64 维旋转,后 192 维不旋转。
逐头共享的增益向量与 GQA 结构已由官方代码确认。
对于后 192 维,任意 query/key 位置 p/s:
sum_j (a_j u_pj)(b_j v_sj) = sum_j u_pj ((a_j b_j) v_sj)
= sum_j ((a_j b_j) u_pj) v_sj。
前 64 维保留原增益和位置旋转。因而完整 QK 点积在实数算术中相等,softmax、V、
输出门控和 O 不变时,完整 attention 分支也是同一个函数。这个结论不依赖某一输入分布。
浮点乘法次序改变,不能承诺逐 bit 相等;需报告实际误差并验证留出 NLL。
不能把这一规则用于旋转维度,或未经核对的逐头独立增益架构。
## 验证与成本边界
1. 在已有真实 q/k 输入上,检查 6 个 full-attention 层的原始/两个合并方向分数、概率及输出。
2. 固定第 3 层进行两个方向的独立网络替换,不叠加,使用原留出文本和质量阈值。
3. 记录预先缓存增益后、相同 CPU 环境下原版与合并版局部归一化耗时;不外推 GPU 性能。
4. 双精度数学推导和 FP32 实测分开。没有消除 Q/K 投影或 RMS 方差计算。
将 Q 的未旋转增益移入 K 时,每 token、每层少做 8×192=1,536 次逐元素增益乘法;
反方向少做 2×192=384 次。它们只是归一化里的小部分操作,不是整个模型的百分比加速。
NumPy 切片/拼接可能抵消这点收益,必须报告测得的局部时延。
参数量从两个 256 维增益变为两个 64 维旋转增益及一个 192 维联合增益,即 512→320 个数。
但原增益以 BF16 存储、联合乘积以 FP32 保留时,载荷仍是 1,024 字节,没有 checkpoint
净压缩;元数据还可能增加。FP32 运行时缓存增益的理论载荷 2,048→1,280 字节,
不等于已测得整个模型 RSS 降低。另行 BF16 量化乘积会增加误差,不属于本次恒等式验证。
这是同一注意力计算内部的重参数化与冗余消除,不是两个现成 attention 层能互换的证明。
若得到正例,应将它作为小粒度确定性优化机会;较大模块互换仍由原候选试验单独回答。
## 完成状态
96 组真实输入分数/概率检查及固定第3层双向网络检查均通过。
该结果独立归类;主目标的已训练函数重复正例见 [CPU 功能结论](CPU_FUNCTIONAL_FINDINGS.md)。
+140
View File
@@ -0,0 +1,140 @@
# Parameter similarity SOP — 2B pilot draft
Status: v0.1, executed for the bounded Qwen3.5-2B pilot. See [findings](PILOT_FINDINGS.md).
Changes proposed from this experience require a small equivalence/performance check before
the next model; no second model is started automatically.
## Scope and decision
Analyze official, unquantized safetensors without instantiating a model, running inference,
training, or executing repository-supplied code. Start with Qwen/Qwen3.5-2B only.
The question is the number of representative-centered sharing groups under direct weight
replacement error limits, and the net byte savings of a theoretical shared representation.
Weight error is not task-accuracy loss, and parameter sharing does not imply fewer FLOPs.
## Prerequisites
1. Confirm authorized remote host, free disk, memory, CPU, existing work, and access.
2. Install prebuilt wheels only. Never compile locally or silently use another runner.
3. Freeze the official repository revision and download file manifest including SHA256.
4. Allocate checkpoint size plus 3 GB disk margin. Use an experiment-owned directory,
not a shared Hugging Face cache. Never delete pre-existing weights.
## Bounded procedure
1. Run synthetic correctness tests on the remote host: BF16, 3D indexing, directional
replacement error, and non-transitive groups.
2. Download one checkpoint sequentially and verify upstream sizes and LFS SHA256.
3. Scan each tensor in 262,144-element chunks. Record shape, dtype, axis role,
layer, bytes, moments, max magnitude, and finite-value checks. Memory-map weights;
evict scanned pages. Do not search for exactly equal weight tensors.
4. Whole-tensor screening: same role, same shape, same dtype. Sample 2,048 aligned,
evenly spaced elements. Select up to 10 neighbors by each of cosine and relative
error. Fairly distribute the 2,000-pair exact-verification cap across role groups.
5. Block screening: non-overlapping 64x64 and 128x128 2D partitions. For 3D tensors,
use axis-0 slices and bounded volumes adapted to actual dimensions. Keep up to
256 uniformly reservoir-sampled blocks per role/shape/scale for the 1.7 GiB pilot host
(below the planned maximum of 512). Record edge bytes
and sampling coverage. Verify at most 20,000 block pairs. Never extrapolate sampled
discoveries to unseen blocks or add savings across alternative scales.
6. Exact metrics: full-element cosine, Pearson, directional replacement errors,
symmetric error, optimal-scalar residual. Tail diagnostics use samples and say so.
Controls include random actual comparable pairs, shuffled sampled elements,
and norm-matched Gaussian vectors. Save sampled versus exact error discrepancies.
Bounded mode-unfolding SVD probes are diagnostic only, not a replacement test.
7. Form deterministic star groups at 1%, 2%, 5%, 10%, 20% relative weight error.
Every member requires an exact verified edge to its representative. Unknown
pairs remain independent; never use graph connected components as sharing groups.
8. Compute original bytes minus representatives, 16-byte mapping per analyzed object,
and 64-byte descriptor per shared group. Fall back to unchanged storage if larger.
Count savings against the entire model, preserving unsampled blocks and edge regions.
9. Save raw data, groups, thresholds, coverage, timings, resource usage, figures
(PNG/PDF/SVG), model metadata and report. Validate every group against exact edges.
10. Copy results to the deliverable workspace and verify checksums before deleting
the experiment-owned checkpoint. Save a cleanup record. Do not delete the venv,
result data or any unrelated cache.
11. Review full diff, commit and push a work branch with Codex Bot identity, then
create a PR for manual merge. Never silently initialize/push main or auto-merge.
## Budget and stop conditions
- Real 2B analysis computation: hard wall-time limit 7,200 seconds. Reserve final time
for summaries; download and report-generation time are recorded separately.
- Use one BLAS thread to avoid interference with existing services.
- Save exact-pair tables every 100 pairs; stage deadlines prevent whole-tensor work
from consuming the complete block budget.
- On timeout or failure, retain source weights until partial results are recoverable
and report coverage/failure truthfully. An internal planned cutoff may produce a complete
bounded report with partial candidate coverage; always state verified/selected counts.
An external kill or missing required report is an incomplete run, not a completed experiment.
- Empty or negligible low-error sharing is a valid negative result. Do not add scales,
increase error thresholds, align permutations, train or expand to other models automatically.
- No runtime capability/performance claim without separately authorized evaluation.
## Commands (remote, from project root)
```sh
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 timeout 300 venv/bin/python -B -m unittest discover -s tests -v
venv/bin/python -B -m pilot.download --root runs/qwen35-2b
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 timeout 7200 venv/bin/python -B -m pilot.analyze --root runs/qwen35-2b --seconds 6600
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -B -m pilot.plot --root runs/qwen35-2b
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 timeout 300 venv/bin/python -B -m unittest discover -s tests -v > runs/qwen35-2b/tests.log 2>&1
venv/bin/python -B -m pilot.audit --root runs/qwen35-2b --analysis-code-commit 6121016
venv/bin/python -B -m pilot.validate --root runs/qwen35-2b
# Copy archived results, verify checksums at the destination, then:
venv/bin/python -B -m pilot.validate --root runs/qwen35-2b --cleanup-weights
```
## Follow-up decisions after the pilot
- The pilot verified 2,000 tensor pairs and 14,778/20,000 selected block pairs before cutoff.
It took about 111 minutes; the final six-test suite including revised figure checks passed in about 77 seconds.
- Hatch missing heatmap cells and label identity values as constructed. Separate layer-distance
curves by dimension/scale; retain warnings about within-stratum module/selection bias.
Normalize control histograms, display sample counts, and verify their areas equal one.
Keep these diagnostics distinct from task-equivalence or statistical-significance claims.
- Avoid concurrent plotting tests and heavy weight IO on the shared host. Record IO wait,
swap, CPU and RSS; a low process RSS is not proof of low total resource cost.
- Distinguish tensor/group counts from byte contributions: 1D tensors were about half the
inventory count but only 0.0197% of weight bytes in this checkpoint.
- Keep fair candidate selection separate from IO-local execution order. Benchmark batching
by source file/tensor without changing the selected pairs or metrics.
- The sampled raw blocks total about 244.5 MiB. Test an explicitly bounded raw BF16/F32
cache against the uncached implementation before adopting it; include fingerprint and
temporary conversion memory in the budget. This was not enabled in the current run.
- Report both pre-metadata and net savings. Tiny blocks can have groups yet lose all savings
to mapping overhead. Any alternative sparse encoding needs its own explicit accounting.
- Separate low-byte normalization groups from matrix/expert compression opportunities.
- Decide whether approximate-neighbor recall and block coverage justify another bounded pass.
- If meaningful candidates exist, request a small calibration-forward experiment to measure
output distortion, then held-out loss/tasks, then implementation-specific memory and latency.
- For MoE, analyze router behavior and joint gate/up/down expert consistency before treating
independently similar matrix blocks as interchangeable experts.
- Use these observations to revise this SOP before downloading a second model.
## Training-free functional follow-up (design only)
Direct coordinate-aligned parameter similarity is a baseline, not a test of Q/K/V or MLP
functional interchangeability. The follow-up uses same-input module probes without solving
rotations or training any parameters, then single-site intervention and held-out quality/cost
validation. See the [bounded plan](TRAINING_FREE_FUNCTIONAL_PLAN.md) and
[primary-source literature](FUNCTIONAL_REUSE_LITERATURE.md). No forward pass, new download,
or new model is started by adopting this document. The original static pilot remains
inference-free; the proposed follow-up requires a separately confirmed execution environment.
## Current functional-run retention override
On 2026-11-09 the user explicitly changed retention: keep the CPU functional pilot
checkpoint until the user explicitly requests deletion. Completion, timeout, failure
and archive validation do not authorize cleanup. The earlier pilot cleanup remains
a historical action; do not apply its automatic cleanup step to the current run.
## CPU functional follow-up outcome
The executed CPU study found a learned RMSNorm function-repetition example at text
layer indices 12/13. Use same-input function error and a uniform analytic bound as
primary evidence, with separate heldout network substitution checks. Costs are not
a success condition under the latest user instruction. See the
[executed findings](CPU_FUNCTIONAL_FINDINGS.md) and
[function-repetition criterion](LEARNED_FUNCTION_REPETITION.md).
Keep whole-module, neuron, normalization and algebraic-reparameterization results distinct.
+125
View File
@@ -0,0 +1,125 @@
# 免训练功能复用:下一阶段有界方案
状态:研究设计,真实模型功能探测尚未执行。原始参数试验已完成;另已完成单线程合成 CPU 容量探测,未新增模型前向、下载或训练。
目标:寻找并验证至少一个在留出数据上保持约定质量、同时降低明确运行成本的复用实例;
若没有通过者,报告有界搜索的阴性结果,不保证一定能找到。
## 1. 硬约束与研究对象
- 冻结所有模型参数。不进行优化器更新、微调、蒸馏、LoRA、学习投影或低秩修正。
- 默认不求正交 R。校准集只用于选择有限候选和阈值,留出集不参与选择。
- 同类型模块配对;Q/K/V 联合作为功能系统,MLP 独立验证。保留归一化、位置处理、门控及头结构。
- 已存配置为 24 个文本层,其中 6 个 full attention、18 个线性注意力;执行前按固定 revision 的实际实现复核。
full attention 的 15 个无序层对可穷举;线性注意力首轮只检查 6 个预先固定的相邻同类型层对。
MLP 首轮检查 6 个预先固定、覆盖浅中深层的相邻层对。具体名单在运行前写入 manifest。
- 之前热力图的 in_proj_a 是线性注意力控制投影示例,不是 Q/K/V 功能相似性的测量。
## 2. 为什么不用先找 R
给定相同隐藏输入 H,且位置、mask 和缓存/递推状态一致,比较两套模块的实际检索函数:
S_i(H) = Q_i(H) K_i(H)^T / sqrt(d),P_i(H) = softmax(S_i(H) + mask)。
这里 Q/K 指完成该架构实际归一化和位置处理后的量。直接比较 P 可避开内部坐标差异;
logit 每行加常数不改变 softmax,因此同时报告去行均值的分数误差。
这不是“存在共同正交 R”的数学证明。反例 Q_2=2Q_1、K_2=K_1/2 保持所有点积,
但非零 Q 的范数改变,不能由正交变换产生。功能等价允许比共同正交旋转更广的关系。
相反,在单套 Q/K 上共同右乘 R 保持点积,只说明坐标不唯一,并未说明两个已训练层可复用。
实际 RoPE、GQA、归一化、门控也必须包含在函数边界内。
有限探针只能提供相应输入分布上的经验证据,不是对所有 H 的形式证明。
在无位置处理/归一化的纯线性特例中,可研究 W_Q W_K^T 的代数相等性;
一般模块不据此简化,首轮不物化所有大算子。
## 3. 同输入与真实输入分开
对候选 i/j,分别从原模型 i 和 j 的入口收集 H_i、H_j;分别比较 f_i(H_i) 与 f_j(H_i),
以及 f_i(H_j) 与 f_j(H_j)。避免只选有利于某一层的输入。模块前归一化如位于边界外,
必须明确分别测“归一化后核心”和“包含归一化的完整分支”。
这检验模块函数接近程度。实际网络的 f_i(H_i) 与 f_j(H_j) 使用不同输入,是另一项量。
权重共享仍需对每个输入计算;缓存上游输出用于下游则改变了输入,必须实际干预后检验。
MLP 测残差相加前的分支输出,防止共同输入 H 掩盖分支差异。
## 4. 指标和对照
| 对象 | 首轮指标 | 不能据此宣称的结论 |
|---|---|---|
| full attention 逐头 P | 平均及 P95 行 JS(自然对数,范围 0–ln2)、top-5 重合率;去行均值 S 相对误差 | P 相似不保证 V/O 输出相似 |
| 完整 attention 分支 | 相对 L2、逐 token 误差 P50/P95、余弦;包含 V/O/门控 | 平均低误差不保证长尾任务能力 |
| 线性注意力 | 同输入、同合法初始状态的分支输出及最终状态误差;后续状态继续使用测试 | 不构造不存在的 softmax 图;状态数值相似不等于长期稳定 |
| MLP | 同输入分支输出相对 L2、逐 token P95、余弦 | 不以各矩阵单独相似代替 gate/up/down 联合验证 |
相对误差统一为 ||candidate-reference|| / max(||reference||, epsilon),并记录低范数样本数量。
首轮筛选门槛预设:P 平均 JS ≤0.01 且 top-5 重合 ≥0.8;完整分支相对 L2 ≤0.05、
逐 token 相对误差 P95 ≤0.1。它们是筛选约定,不是文献保证或能力损失界。
线性模块和 MLP 使用输出门槛;只通过 P 的候选不能自动通过完整模块门槛。
对照:同模块重复测量、随机同类型层对;首 token 保留/移除并重新归一化两个版本;
排除自注意与局部窗口主导后再测;按序列/位置/浅中深层分层。零分母与无效行单独标记。
逐头记录,不先平均头。默认只用原头序;可选的离散头排列仅在校准集选定后冻结,
须满足 GQA 分组和 V/O 一致性,不允许把最优头匹配分数当作可执行替换结果。
CKA 仅作辅助筛选;共享位置模板、attention sink 和共同残差都可能制造高相似。
## 5. 资源与阶段预算
采用 CPU 先导、GPU 后续细化的两阶段路径;GPU 不作为寻找初步证据的前置条件。
当前服务器为 2 核 CPU、约 1.7 GiB RAM;60 GB 是磁盘扩容。详见
[CPU 先导设计及实测](CPU_FIRST_VALIDATION.md)。
CPU 阶段仅处理文本,batch=1、长度 32,校准/留出各 4 条真实且去重的文本;
仅检查 3 个预先固定、覆盖浅中深的 full-attention 层对和 2 个 MLP 层对。
完整保留合法 key,逐头比较,不通过采样 key 改变 softmax。
模型依赖实现一致性检查和容量探测完成后,才实施真实前向。
权重按层/子矩阵流式读取并释放,禁止全模型转换成 FP32;不把依赖换页的完整框架加载视为可行方案。
数据来源、许可、tokenizer revision、截断、样本 hash 和 seed 在运行前写入 manifest。
| CPU 阶段 | 上限 | 完成条件 / 停止条件 |
|---|---|---|
| 实现一致性及短样本容量探测 | 20 分钟 | 模块计算语义核对,内存峰值可控;OOM、持续换页或实现核对失败即停止 |
| 真实隐藏输入采集 | 40 分钟 | 流式文本前向采集固定入口,不保存所有层权重/注意力图;超时保存覆盖率 |
| 同输入功能比较 | 30 分钟 | 固定 5 对候选;无通过者停止;最多留下 1 个干预候选 |
| 单处干预及归档 | 30 分钟 | 在 4 条留出文本上检查输出或流式 token NLL;不足以完成则只交付候选证据 |
CPU 首轮计算上限 120 分钟,下载单独计时。当前仅执行了约 88 MiB 峰值 RSS 的
合成矩阵乘法探测,没有重下权重或执行真实前向。容量探测不能保证完整模型实现的峰值内存。
线性注意力需另行完成正确性核对,首轮先不加入,避免状态递推实现拖延初步判断。
短文本小样本无法覆盖长上下文或低频关键能力;阴性结果也不能排除更广泛候选。
只有发现值得跟进的候选,GPU 阶段才扩展到原定 27 对候选、长度 256、
校准/留出各 16 条及更多独立任务,细化单处/多处干预与真实性能。
届时由用户提供并确认 GPU 执行环境;当前不租用或假定已有 GPU。
## 6. 干预必须对应可节省的对象
| 干预 | 可以争取的收益 | 必须额外确认 |
|---|---|---|
| 跨层绑定权重 | 权重存储/加载内存 | 各层输入仍不同,通常不减少投影 FLOPs |
| 重用上游 attention 分布,保留下游 V/O | 部分 Q/K 投影与 QK/softmax 计算 | 保留分布的内存及实现开销;decode 未必值得缓存完整图 |
| 跨层共享 K/V cache | KV 内存,可能减少 K/V 投影 | GQA、位置、缓存布局与层依赖合法性 |
| 重用完整 attention/MLP 输出 | 对应分支计算 | 下游输入不同,必须测试整个原模型中的替换效应 |
选择的每个候选首先给出一种具体可实现干预;只有同输入指标通过却无实际可省操作,
标记“功能相似但暂无复用实现”。不把整模块可替换和其中 Q/K 可替换混为一谈。
首轮探索性质量门槛:留出平均 token NLL 增量 ≤0.01 nat/token,逐序列增量 P95 ≤0.05,
输出有限;同时保存全部逐序列差值,不将该小样本阈值称为统计显著性或普适能力保证。
候选只在校准集筛选;留出失败不回调阈值、不反复选择。
性能分别测 prefill 和固定 32-token decode;固定 batch/长度/dtype/backend,预热 3 次、
交错运行基线与干预各 10 次,报告原始时延和中位数/P95、峰值内存、KV/权重字节及理论运算变化。
插桩诊断时延不作为优化实现时延。若速度差落在运行波动内,不宣称加速;
若只减内存,则结论限定为内存收益。计入缓存、索引与数据移动开销。
## 7. 分组与最终证据
单处通过后,才讨论多成员组:每个成员必须通过对同一代表的定向替换检查;
不能用 A≈B、B≈C 推出 A 可代表 C。多处同时替换须再测累积误差和性能,
单处收益不能相加为最终模型收益。首轮不自动执行多处替换。
输出每对候选/每序列/每头指标 CSV、覆盖率和失败原因、质量—成本散点图、逐层热图、
基线与干预原始计时、环境及代码版本。结论分为:功能接近、单处干预通过、测得实际收益三档。
“存在一个免训练复用点”的操作性证据必须同时具有后两项;当前静态试验尚未提供这类证据。
相关依据与训练限制见 [文献核对](FUNCTIONAL_REUSE_LITERATURE.md)。
+91
View File
@@ -0,0 +1,91 @@
# Bounded CPU functional pilot
This is the execution implementation of the CPU-first plan, not the completed static
parameter pilot. Frozen Qwen3.5-2B weights are decoded to FP32 a projection block at a
time. CPU only, no autograd/optimization, no rotation fitting, no model-hub Python code.
The user explicitly requires retaining this checkpoint until they request deletion.
## Fixed scope
- Batch of eight distinct experiment-authored plain texts, first 32 tokens each:
four calibration and four heldout, two Chinese/two English in each split.
- Full attention pairs: (3,7), (11,15), (19,23).
- MLP pairs: (5,6), (20,21).
- Smaller-granularity Q/K RMSNorm pairs inside layers 3,7,11,15,19,23.
Added before observing real weights/functions for this pilot; these are small storage
opportunities, not a claim of substantial compute savings.
- At most one calibration-selected candidate, two separate substitution directions.
Selection prefers component size, then worst symmetric output error. No heldout-driven retry.
- Same-input relative L2 <=0.05 in both directions and token P95 <=0.1 in both directions;
attention additionally requires mean JS <=0.01 and top-5 overlap >=0.8.
- Heldout mean NLL increase <=0.01 nat/token and per-sequence P95 increase <=0.05.
These are pilot decision criteria, not mathematical capability guarantees.
## Source and checks
`reference.json` pins the official Transformers Qwen3.5 source commit, URL and hash.
`reference_check.py` extracts only named mathematical definitions from that exact file,
removes optional fused-kernel decorators, and compares the NumPy implementation against
Torch operators, full toy decoder layers and a two-layer text forward. Chunked delta and
recurrent delta implementations are both checked, including sequence padding boundaries.
The official source is Apache-2.0, copyright Qwen Team and HuggingFace; the reference file
is fetched for testing and is not bundled here. FP32 equivalence is checked, not equality
to every deployed BF16 kernel.
`tests/test_functional.py` independently checks streamed projection/embedding/NLL,
causality, causal convolution, normalization conventions and basic metric identities.
These tests use tiny synthetic arrays. Real model evidence is produced only by `run.py`.
## Remote execution
Use the previously authorized CPU host and existing NumPy/Matplotlib environment. Install
only binary wheels: tokenizers==0.23.3 from HTTPS PyPI; torch==2.15.0+cpu from the official
CPU wheel index for independent reference checks. Use an experiment-owned on-disk TMPDIR:
the shared host's /tmp tmpfs was too small during the first wheel download. Do not change
system swap or shared temporary files.
```sh
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -B -m unittest tests.test_functional -v
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -B -m functional.reference_check --source runs/cpu-qwen35-2b/reference.py --output runs/cpu-qwen35-2b/reference-check.json
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 venv/bin/python -B -m functional.supervise --root runs/cpu-qwen35-2b
```
The supervisor waits at most one hour for binary dependencies and four hours for the
already started pinned, verified checkpoint download. It does not install, retry downloads,
search extra candidates, mutate Git or delete weights. After reference checks, it runs one
8-token real text capacity check, then the bounded functional pilot. Reference, preflight
and functional compute together are capped at two hours with a reporting reserve;
download wait is recorded separately. Projection memory is checked, RSS soft stop is
512 MiB, available-system-memory floor 256 MiB; sustained host swap IO combined with low available memory or >128 MiB process swap also stops the run.
A persistent ledger counts computation across restarts; early checks/preflight receive a conservative 180-second reserve.
`preflight.py` is a capacity smoke check, not a semantic oracle. `run.py` records exact
sample/token IDs, fixed configuration, source hashes, activations, per-text/per-head metrics,
selection, per-token NLL, intervention results and truthful stop coverage. `report.py` reads
saved data to produce PNG/PDF/SVG figures without reopening weights.
No generation throughput is measured in this short-text pilot. Any parameter-tying storage
benefit must be accounted for separately, including metadata. Neither same-input similarity
nor favorable tiny heldout NLL demonstrates broad mutual interchangeability or GPU speedup.
## Current primary result and completed extensions
The latest user goal is learned-function repetition, independent of costs. Existing
input RMSNorm modules at layer indices 12/13 have a proven worst two-direction real-arithmetic
error bound of 11.3402%, worst observed heldout relative error 4.5285%, minimum cosine 0.999487,
and small bidirectional heldout NLL changes. See [findings](../docs/CPU_FUNCTIONAL_FINDINGS.md).
- `neuron_reuse.py`: fixed four-layer scalar MLP function search, nontrivial activity
gates, calibration-only selection and independent heldout/substitution logic; no pair passed.
- `norm_repetition.py`: same-role learned RMSNorm bound across 552 pairs, one deterministic
selection independent of any input samples, real probes and two full-network substitutions.
- `gain_factorization.py`: architecture-derived identity on non-RoPE coordinates; a separate
reparameterization result, not a learned-function repetition example.
- `benchmark_factorization.py`: historical one-attempt CPU allocation optimization and
bitwise equivalence check. Performance is no longer a success condition.
- `storage.py`: explicit component layout accounting; the original candidate run selected
no pair. This does not affect the primary function-repetition conclusion.
Do not expand the final positive claim to full attention or MLP bodies. Retain the
fixed checkpoint until the user explicitly requests deletion. Full tests and audit are
run remotely; no local build or model execution is required to inspect the archived results.
+1
View File
@@ -0,0 +1 @@
"""CPU-only, training-free functional reuse pilot."""
+86
View File
@@ -0,0 +1,86 @@
"""Audit saved functional evidence and archive only outputs, never checkpoint weights."""
import argparse
import csv
import hashlib
import json
from pathlib import Path
import platform
import subprocess
import time
import numpy as np
from .model import StreamingLM
from .norm_repetition import uniform_bounds
from .run import write_json
EXCLUDE={'tokenizer.json','reference.py','checksums-functional.json','audit-validation.json','archive-files.json'}
def archive_files(root):
files=[p for p in root.iterdir() if p.is_file() and p.name not in EXCLUDE and not p.name.endswith(('.partial','.safetensors'))]
files += [p for p in (root/'figures').glob('*') if p.is_file()]
return sorted(files)
def main():
p=argparse.ArgumentParser();p.add_argument('--root',type=Path,required=True);a=p.parse_args();root=a.root
manifest=json.loads((root/'manifest.json').read_text())
assert manifest['revision']=='15852e8c16360a2fea060d615a32b45270f8a8fc'
for f in manifest['files']:
if f['name'].endswith('.safetensors'):
path=root/'weights'/f['name']
assert path.is_file() and path.stat().st_size==f['bytes']
assert f['sha256']==f['upstream_sha256']
retention=json.loads((root/'retention-policy.json').read_text());assert retention['automatic_cleanup'] is False
reference=json.loads((root/'reference-check.json').read_text())
assert reference['passed'] and len(reference['tests'])==22
assert reference['implementation_sha256']==hashlib.sha256(Path('functional/model.py').read_bytes()).hexdigest()
tests=(root/'final-tests.txt').read_text();assert 'Ran 12 tests' in tests and '\nOK\n' in tests
result=json.loads((root/'norm-repetition-summary.json').read_text());assert result['state']=='completed'
assert result['learned_function_repetition_positive'] and result['bidirectional_network_quality_pass']
cfg=json.loads((root/'model_config.json').read_text());lm=StreamingLM(root/'weights',cfg,seconds=60)
c=result['selected'];ga=1+lm.small(f'layers.{c["a"]}.{c["role"]}.weight');gb=1+lm.small(f'layers.{c["b"]}.{c["role"]}.weight')
bounds=uniform_bounds(ga,gb)
np.testing.assert_allclose(bounds,[c['a_to_b_uniform_bound'],c['b_to_a_uniform_bound']],atol=1e-14,rtol=1e-14)
with (root/'norm-repetition-uniform-bounds.csv').open() as f:assert len(list(csv.DictReader(f)))==552
baseline=np.load(root/'baseline-heldout-token-nll.npy');assert baseline.shape==(4,31)
for rec in result['interventions']:
nll=np.load(root/f'norm-repetition-{rec["target"]}-uses-{rec["donor"]}-token-nll.npy')
delta=nll.mean(1)-baseline.mean(1)
np.testing.assert_allclose(delta,rec['sequence_nll_deltas'],atol=1e-13,rtol=1e-13)
assert abs(delta.mean()-rec['mean_nll_delta'])<1e-13
with (root/'norm-repetition-function-metrics.csv').open() as f:rows=list(csv.DictReader(f))
held=[r for r in rows if r['split']=='heldout'];assert len(held)==8
worst=max(max(float(r['relative_l2']),float(r['reverse_relative_l2'])) for r in held)
assert abs(worst-result['worst_heldout_relative_l2'])<1e-14
neuron=json.loads((root/'neuron-reuse-search-summary.json').read_text());assert sum(x['eligible_unique_pairs_checked'] for x in neuron)==7450139
ledger=json.loads((root/'compute-budget-ledger.json').read_text());assert ledger['spent_seconds']+600<7200
environment={'python':platform.python_version(),'platform':platform.platform(),'numpy':np.__version__,
'packages':subprocess.run([str(Path(__import__('sys').executable)),'-m','pip','freeze'],capture_output=True,text=True,check=True).stdout.splitlines(),
'weights_retained':True,'no_automatic_cleanup':True,'ledger_seconds_including_initial_reserve':ledger['spent_seconds'],
'final_tests_report_audit_additional_budget_upper_reserve_seconds':600,
'historical_error_logs':'Early reference adapter errors and preflight pressure stops; latest summaries/tests are authoritative.'}
write_json(root/'functional-environment.json',environment)
source_paths=[*Path('functional').glob('*.py'),Path('functional/reference.json'),Path('pilot/tensors.py'),Path('tests/test_functional.py')]
write_json(root/'final-source-hashes.json',{str(p):hashlib.sha256(p.read_bytes()).hexdigest() for p in sorted(source_paths)})
files=archive_files(root)
# Every numeric CSV value must be finite. Empty diagnostic fields stay unknown.
for path in files:
if path.suffix=='.csv':
with path.open() as f:
for row in csv.DictReader(f):
for value in row.values():
if value in (None,''):continue
try:number=float(value)
except ValueError:continue
assert np.isfinite(number),(path,value)
checksums={str(p.relative_to(root)):hashlib.sha256(p.read_bytes()).hexdigest() for p in files}
write_json(root/'checksums-functional.json',checksums)
selected=[*checksums,'checksums-functional.json','audit-validation.json','archive-files.json']
write_json(root/'archive-files.json',selected)
write_json(root/'audit-validation.json',{'passed':True,'validated_unix':time.time(),'checksummed_files':len(checksums),
'proof_recomputed_from_retained_weights':True,'heldout_nll_recomputed_from_saved_arrays':True,
'weights_present_and_retained':True,'archive_excludes_weights_and_tokenizer':True})
print(json.dumps({'passed':True,'checksummed_files':len(checksums),'weights_retained':True}))
if __name__=='__main__':main()
+83
View File
@@ -0,0 +1,83 @@
"""One allocation-reduction attempt, with a similarly optimized baseline.
Does not change weights or run the network again. Checks bitwise equality to the
network-tested factorization on every captured real Q/K input before timing.
"""
import argparse
import json
from pathlib import Path
import resource
import time
import numpy as np
from .model import StreamingLM
from .gain_factorization import consolidate,normalized,LAYERS
from .run import save_csv,write_json
def baseline_inplace(q,k,a,b):
q=normalized(q);q*=a
k=normalized(k);k*=b
return q,k
def factored_inplace(q,k,a,b,r,combined,move='q_to_k'):
q,k=normalized(q),normalized(k)
if move=='q_to_k':
q[...,:r]*=a[:r];k*=combined
else:
q*=combined;k[...,:r]*=b[:r]
return q,k
def main():
p=argparse.ArgumentParser();p.add_argument('--root',required=True,type=Path);a=p.parse_args();root=a.root
ledger_path=root/'compute-budget-ledger.json';ledger=json.loads(ledger_path.read_text());remaining=int(7200-ledger['spent_seconds']-20)
if remaining<60:raise TimeoutError('Cumulative budget insufficient')
start=time.monotonic();result={'state':'running'}
try:
cfg=json.loads((root/'model_config.json').read_text());lm=StreamingLM(root/'weights',cfg,seconds=min(remaining,300))
r=64;data=np.load(root/'baseline-activations.npz');checks=[]
for layer in LAYERS:
lm.guard();pfx=f'layers.{layer}.self_attn.'
ag,bg=1+lm.small(pfx+'q_norm.weight'),1+lm.small(pfx+'k_norm.weight')
q,k=data[f'qraw_{layer}'],data[f'kraw_{layer}']
for move in ('q_to_k','k_to_q'):
combined=np.concatenate([(bg if move=='q_to_k' else ag)[:r],ag[r:]*bg[r:]])
expected=consolidate(q,k,ag,bg,r,move)
actual=factored_inplace(q,k,ag,bg,r,combined,move)
for x,y in zip(expected,actual):np.testing.assert_array_equal(x,y)
checks.append({'layer':layer,'direction':move,'query_elements':int(q.size),'key_elements':int(k.size),'bitwise_equal':True})
q,k=data['qraw_3'][:1],data['kraw_3'][:1];pfx='layers.3.self_attn.'
ag,bg=1+lm.small(pfx+'q_norm.weight'),1+lm.small(pfx+'k_norm.weight')
combined=np.concatenate([bg[:r],ag[r:]*bg[r:]])
funcs={'baseline_legacy':lambda:(normalized(q)*ag,normalized(k)*bg),
'baseline_inplace':lambda:baseline_inplace(q,k,ag,bg),
'factored_inplace':lambda:factored_inplace(q,k,ag,bg,r,combined)}
for _ in range(3):
for fn in funcs.values():fn()
timing=[];names=list(funcs)
for repeat in range(30):
order=names[repeat%3:]+names[:repeat%3]
for name in order:
lm.guard();t=time.perf_counter()
for _ in range(50):funcs[name]()
timing.append({'repeat':repeat,'implementation':name,'seconds_per_call':(time.perf_counter()-t)/50,'calls_per_block':50})
save_csv(root/'gain-factorization-optimized-timing.csv',timing)
values={name:np.array([row['seconds_per_call'] for row in timing if row['implementation']==name]) for name in names}
ratio=values['baseline_inplace']/values['factored_inplace']
medians={name:float(np.median(v)) for name,v in values.items()}
result.update(state='completed',bitwise_checks=checks,medians_seconds=medians,
paired_baseline_over_factored_median=float(np.median(ratio)),
paired_ratio_p10=float(np.quantile(ratio,.1)),paired_ratio_p90=float(np.quantile(ratio,.9)),
fraction_blocks_factored_faster=float(np.mean(ratio>1)),
descriptive_local_gain=bool(np.median(ratio)>1.05 and np.mean(ratio>1)>=.8 and np.quantile(ratio,.1)>1),
claim_limit='Hot-array CPU normalization microbenchmark against equally allocation-optimized baseline; no full-model speedup or independent statistical significance claim.')
except Exception as exc:
result.update(state='stopped',error_type=type(exc).__name__,error=str(exc));raise
finally:
elapsed=time.monotonic()-start;result.update(elapsed_seconds=elapsed,peak_rss_kib=resource.getrusage(resource.RUSAGE_SELF).ru_maxrss)
ledger=json.loads(ledger_path.read_text());ledger['spent_seconds']+=elapsed;ledger['calls'].append({'module':'functional.benchmark_factorization','seconds':elapsed});write_json(ledger_path,ledger)
write_json(root/'gain-factorization-optimized-summary.json',result)
if __name__=='__main__':main()
+135
View File
@@ -0,0 +1,135 @@
"""Architecture-derived Q/K gain consolidation on non-RoPE dimensions only.
Fixed layer3 interventions; no fitting, layer selection or learned alignment.
Run only after baseline/candidate pilot finishes, within the same cumulative budget.
"""
import argparse
import csv
import json
from pathlib import Path
import resource
import time
import numpy as np
from .model import StreamingLM,norm,rope
from .run import probability_metrics,write_json,save_csv
LAYERS=[3,7,11,15,19,23]
def normalized(x):
return x/np.sqrt(np.mean(x*x,axis=-1,keepdims=True)+np.float32(1e-6))
def consolidate(q,k,a,b,r,move,joint=None):
q,k=normalized(q),normalized(k)
c=a[r:]*b[r:] if joint is None else joint
if move=='q_to_k':
q=np.concatenate([q[...,:r]*a[:r],q[...,r:]],axis=-1)
k=np.concatenate([k[...,:r]*b[:r],k[...,r:]*c],axis=-1)
else:
q=np.concatenate([q[...,:r]*a[:r],q[...,r:]*c],axis=-1)
k=np.concatenate([k[...,:r]*b[:r],k[...,r:]],axis=-1)
return q,k
class FactoredLM(StreamingLM):
def configure(self,layer,move):
self.factor_layer=layer;self.move=move
p=f'layers.{layer}.self_attn.'
self.a=1+self.small(p+'q_norm.weight');self.b=1+self.small(p+'k_norm.weight')
self.r=int(self.config['head_dim']*self.config['rope_parameters']['partial_rotary_factor'])
self.joint=self.a[self.r:]*self.b[self.r:]
def normalization(self,x,name):
p=f'layers.{self.factor_layer}.self_attn.'
if name not in (p+'q_norm',p+'k_norm'):return super().normalization(x,name)
# This path does not multiply the unit-gain tail, unlike a full-vector mask.
unit=x/np.sqrt(np.mean(x*x,axis=-1,keepdims=True)+self.eps)
isq=name==p+'q_norm';head=self.a if isq else self.b
skip=(isq and self.move=='q_to_k') or (not isq and self.move=='k_to_q')
tail=unit[...,self.r:] if skip else unit[...,self.r:]*self.joint
return np.concatenate([unit[...,:self.r]*head[:self.r],tail],axis=-1)
def main():
p=argparse.ArgumentParser();p.add_argument('--root',required=True,type=Path);a=p.parse_args();root=a.root
# Protocol is fixed before inspecting functional measurements.
protocol={'fixed_intervention_layer':3,'directions':['q_to_k','k_to_q'],'diagnostic_layers':LAYERS,'no_selection':True,'no_training':True,'precision':'FP32 product; no BF16 requantization','quality_gate':{'mean_nll_delta':.01,'sequence_p95_nll_delta':.05}}
protocol_path=root/'gain-factorization-protocol.json'
if protocol_path.exists():
old=json.loads(protocol_path.read_text());assert all(old[k]==v for k,v in protocol.items()), 'Protocol differs from preregistration'
else:write_json(protocol_path,{**protocol,'registered_unix':time.time()})
ledger_path=root/'compute-budget-ledger.json';ledger=json.loads(ledger_path.read_text())
remaining=int(7200-ledger['spent_seconds']-30)
if remaining<60:raise TimeoutError('No remaining cumulative budget for deterministic identity verification')
start=time.monotonic()
report={'state':'running','records':[],'interventions':[]}
try:
cfg=json.loads((root/'model_config.json').read_text());c=cfg['text_config'];r=int(c['head_dim']*c['rope_parameters']['partial_rotary_factor'])
assert c['rms_norm_eps']==1e-6
assert c['head_dim']==256 and r==64 and c['num_attention_heads']==8 and c['num_key_value_heads']==2
lm=FactoredLM(root/'weights',cfg,seconds=remaining)
data=np.load(root/'baseline-activations.npz');samples=json.loads((root/'samples-tokenized.json').read_text())['samples']
rp=c['rope_parameters'];repeats=c['num_attention_heads']//c['num_key_value_heads']
records=[]
for layer in LAYERS:
lm.guard()
pfx=f'layers.{layer}.self_attn.'
aw=lm.small(pfx+'q_norm.weight');bw=lm.small(pfx+'k_norm.weight');ag,bg=1+aw,1+bw
q,k=data[f'qraw_{layer}'],data[f'kraw_{layer}']
qr,kr=rope(norm(q,aw),rp['rope_theta'],rp['partial_rotary_factor']),rope(norm(k,bw),rp['rope_theta'],rp['partial_rotary_factor'])
ref=np.einsum('bthd,bshd->bhts',qr,np.repeat(kr,repeats,axis=2))/np.float32(np.sqrt(c['head_dim']))
mask=np.triu(np.ones(ref.shape[-2:],dtype=bool),1)
def probs(s):
s=np.where(mask,-np.inf,s);e=np.exp(s-s.max(-1,keepdims=True));return e/e.sum(-1,keepdims=True)
pref=probs(ref)
for move in ('q_to_k','k_to_q'):
qq,kk=consolidate(q,k,ag,bg,r,move)
qq,kk=rope(qq,rp['rope_theta'],rp['partial_rotary_factor']),rope(kk,rp['rope_theta'],rp['partial_rotary_factor'])
changed=np.einsum('bthd,bshd->bhts',qq,np.repeat(kk,repeats,axis=2))/np.float32(np.sqrt(c['head_dim']))
pp=probs(changed)
for bi,s in enumerate(samples):
diff=changed[bi].astype(np.float64)-ref[bi]
records.append({'layer':layer,'direction':move,'sample_id':s['id'],'split':s['split'],
'score_max_abs_error':float(np.abs(diff).max()),'score_relative_l2':float(np.linalg.norm(diff)/max(np.linalg.norm(ref[bi].astype(np.float64)),1e-12)),
'probability_max_abs_error':float(np.abs(pp[bi]-pref[bi]).max()),**probability_metrics(pref[bi:bi+1],pp[bi:bi+1])})
save_csv(root/'gain-factorization-score-metrics.csv',records)
held=np.array([i for i,s in enumerate(samples) if s['split']=='heldout']);ids=np.array([s['token_ids'] for s in samples],dtype=np.int64)
base=np.load(root/'baseline-heldout-token-nll.npy').mean(1)
for move in ('q_to_k','k_to_q'):
lm.configure(3,move)
h,_=lm.forward(ids[held])
nll=lm.nll(h,ids[held]);np.save(root/f'gain-factorization-{move}-token-nll.npy',nll)
delta=nll.mean(1)-base
report['interventions'].append({'layer':3,'direction':move,'sequence_nll_deltas':delta.tolist(),'mean_nll_delta':float(delta.mean()),'sequence_p95_delta':float(np.quantile(delta,.95)),'passed':bool(delta.mean()<=.01 and np.quantile(delta,.95)<=.05)})
write_json(root/'gain-factorization-summary.json',report)
# Local normalization timing uses precomputed gains for both implementations.
q,k=data['qraw_3'][:1],data['kraw_3'][:1];pfx='layers.3.self_attn.'
ag,bg=1+lm.small(pfx+'q_norm.weight'),1+lm.small(pfx+'k_norm.weight')
def baseline():return normalized(q)*ag,normalized(k)*bg
joint=ag[r:]*bg[r:]
def factored():return consolidate(q,k,ag,bg,r,'q_to_k',joint=joint)
for _ in range(3):baseline();factored()
timing=[]
for repeat in range(10):
for label,fn in (('baseline',baseline),('factored',factored)) if repeat%2==0 else (('factored',factored),('baseline',baseline)):
lm.guard();t=time.perf_counter()
for _ in range(20):fn()
timing.append({'repeat':repeat,'implementation':label,'seconds_per_call':(time.perf_counter()-t)/20})
save_csv(root/'gain-factorization-kernel-timing.csv',timing)
report.update(state='completed',maximum_score_abs_error=max(x['score_max_abs_error'] for x in records),maximum_probability_abs_error=max(x['probability_max_abs_error'] for x in records),
bidirectional_heldout_pass=all(x['passed'] for x in report['interventions']),
elementwise_gain_multiplies_saved_per_token_layer={'q_to_k':1536,'k_to_q':384},
original_bf16_payload_bytes=1024,factored_lossless_mixed_payload_bytes=1024,
fp32_cached_gain_payload_before=2048,fp32_cached_gain_payload_after=1280,
claim_limit='Exact real-arithmetic identity only on non-RoPE coordinates; empirical FP32/NLL checks on tiny text pilot. Not evidence that two existing full layers interchange, not model-wide speedup.')
except Exception as exc:
report.update(state='stopped',error_type=type(exc).__name__,error=str(exc));raise
finally:
elapsed=time.monotonic()-start
report.update(elapsed_seconds=elapsed,peak_rss_kib=resource.getrusage(resource.RUSAGE_SELF).ru_maxrss)
ledger=json.loads(ledger_path.read_text());ledger['spent_seconds']+=elapsed;ledger['calls'].append({'module':'functional.gain_factorization','seconds':elapsed});write_json(ledger_path,ledger)
write_json(root/'gain-factorization-summary.json',report)
if __name__=='__main__':main()
+247
View File
@@ -0,0 +1,247 @@
"""Streaming FP32 text-only Qwen3.5 reference, original BF16 weights.
Mathematical semantics checked against Transformers Qwen3.5, commit
fd3cf0140de03f7619ac95b5cfed05b711ff9e22 (Apache-2.0, Qwen Team/HuggingFace).
No cached decode, padding, images, generation, fused kernels or autograd.
Full-sequence gated delta recurrence replaces the algebraically equivalent
chunk algorithm; both are independently compared in remote tests.
"""
import ctypes
import json
import time
from pathlib import Path
import numpy as np
from pilot.tensors import load_tensors, decode
PREFIX = 'model.language_model.'
def sigmoid(x):
return np.exp(-np.logaddexp(np.float32(0), -x))
def silu(x):
return x * sigmoid(x)
def norm(x, weight, eps=1e-6, centered=True):
return (x / np.sqrt(np.mean(x*x, axis=-1, keepdims=True) + eps)
* (1 + weight if centered else weight)).astype(np.float32)
def rope(x, theta=10000000., fraction=.25):
# Text-only: all three MRoPE position axes coincide.
dim = int(x.shape[-1] * fraction)
inv = np.float32(1) / np.float32(theta) ** (np.arange(0, dim, 2, dtype=np.float32) / dim)
freq = np.arange(x.shape[1], dtype=np.float32)[:, None] * inv[None, :]
angle = np.concatenate([freq, freq], axis=-1)[None, :, None, :]
a = x[..., :dim]
rotated = np.concatenate([-a[..., dim//2:], a[..., :dim//2]], axis=-1)
return np.concatenate([a*np.cos(angle) + rotated*np.sin(angle), x[..., dim:]], axis=-1)
def delta_rule(q, k, v, g, beta):
q = q / np.sqrt(np.sum(q*q, axis=-1, keepdims=True) + 1e-6)
k = k / np.sqrt(np.sum(k*k, axis=-1, keepdims=True) + 1e-6)
q = q / np.float32(np.sqrt(q.shape[-1]))
state = np.zeros((q.shape[0], q.shape[2], q.shape[3], v.shape[3]), dtype=np.float32)
out = np.empty_like(v)
for t in range(q.shape[1]):
state *= np.exp(g[:, t, :, None, None])
predicted = np.sum(state * k[:, t, :, :, None], axis=-2)
delta = (v[:, t] - predicted) * beta[:, t, :, None]
state += k[:, t, :, :, None] * delta[:, :, None, :]
out[:, t] = np.sum(state * q[:, t, :, :, None], axis=-2)
return out, state
def causal_conv(x, kernel):
out = np.zeros_like(x)
width = kernel.shape[-1]
for j in range(width):
lag = width - 1 - j
if lag < x.shape[1]:
out[:, lag:] += x[:, :x.shape[1]-lag] * kernel[:, j]
return silu(out)
def attention(q, k, v):
repeats = q.shape[2] // k.shape[2]
k, v = np.repeat(k, repeats, axis=2), np.repeat(v, repeats, axis=2)
scores = np.einsum('bthd,bshd->bhts', q, k) / np.float32(np.sqrt(q.shape[-1]))
mask = np.triu(np.ones(scores.shape[-2:], dtype=bool), 1)
scores = np.where(mask, -np.inf, scores)
exp = np.exp(scores - scores.max(axis=-1, keepdims=True))
p = exp / exp.sum(axis=-1, keepdims=True)
return np.einsum('bhts,bshd->bthd', p, v), p
class StreamingLM:
def __init__(self, weights, config, seconds=7200, enforce_memory=True):
self.tensors = load_tensors(Path(weights))
self.config = config.get('text_config', config)
self.deadline = time.monotonic() + seconds
self.enforce_memory = enforce_memory
self.eps = self.config['rms_norm_eps']
self.norm_replacement = None
self.capture_qk = False
self.qk_inputs = {}
self.swap_observation = None
self.swap_pressure_windows = 0
self.malloc_trim = getattr(ctypes.CDLL(None), 'malloc_trim', None)
def guard(self):
if time.monotonic() > self.deadline:
raise TimeoutError('CPU computation deadline reached')
if not self.enforce_memory:
return
status = dict(line.split(':', 1) for line in Path('/proc/self/status').read_text().splitlines() if ':' in line)
rss_kib = int(status['VmRSS'].split()[0])
available = int(next(line.split()[1] for line in Path('/proc/meminfo').read_text().splitlines() if line.startswith('MemAvailable:')))
now = time.monotonic()
if self.swap_observation is None or now-self.swap_observation[0] >= 5:
counters = dict(line.split() for line in Path('/proc/vmstat').read_text().splitlines())
count = int(counters['pswpin'])+int(counters['pswpout'])
if self.swap_observation is not None:
elapsed = now-self.swap_observation[0]
rate = (count-self.swap_observation[1])*4096/elapsed
self.swap_pressure_windows = self.swap_pressure_windows+1 if rate > 4*1024**2 else 0
own_swap_kib = int(status.get('VmSwap', '0 kB').split()[0])
if self.swap_pressure_windows >= 3 and (available < 384*1024 or own_swap_kib > 128*1024):
raise MemoryError(f'Sustained host swap pressure plus constrained memory: available={available} KiB, process_swap={own_swap_kib} KiB')
self.swap_observation = (now,count)
if rss_kib > 512*1024 or available < 256*1024:
raise MemoryError(f'Soft memory stop: RSS {rss_kib} KiB, available {available} KiB')
def small(self, name):
t = self.tensors[PREFIX + name]
if t.numel > 1048576:
raise ValueError('Use blockwise linear for large tensors')
x = t.read().reshape(t.shape)
t.evict()
return x
def linear(self, x, name, rows=256):
t = self.tensors[PREFIX + name + '.weight']
outdim, indim = t.shape
flat = x.reshape(-1, indim)
result = np.empty((flat.shape[0], outdim), dtype=np.float32)
for start in range(0, outdim, rows):
self.guard()
end = min(start + rows, outdim)
w = t.read(slice(start*indim, end*indim)).reshape(end-start, indim)
result[:, start:end] = flat @ w.T
del w
t.evict()
if self.malloc_trim is not None:
self.malloc_trim(0) # Release only this process's freed temporary heap pages.
return result.reshape(*x.shape[:-1], outdim)
def embedding(self, ids):
t = self.tensors[PREFIX + 'embed_tokens.weight']
raw = t.raw.reshape(t.shape)
result = decode(raw[ids], t.dtype)
t.evict()
return result
def normalization(self, x, name):
if getattr(self, 'norm_replacement', None) and name == self.norm_replacement[0]:
name = self.norm_replacement[1]
return norm(x, self.small(name + '.weight'), self.eps)
def mlp(self, h, layer):
p = f'layers.{layer}.mlp.'
gate = silu(self.linear(h, p+'gate_proj'))
up = self.linear(h, p+'up_proj')
return self.linear(gate*up, p+'down_proj')
def full_attention(self, h, layer, shared_p=None):
p = f'layers.{layer}.self_attn.'
c = self.config
heads, kvheads, dim = c['num_attention_heads'], c['num_key_value_heads'], c['head_dim']
# Q and output gate are interleaved per head, not globally split.
mixed = self.linear(h, p+'q_proj').reshape(*h.shape[:-1], heads, 2*dim)
q, gate = mixed[..., :dim], mixed[..., dim:]
k = self.linear(h, p+'k_proj').reshape(*h.shape[:-1], kvheads, dim)
v = self.linear(h, p+'v_proj').reshape(*h.shape[:-1], kvheads, dim)
if getattr(self, 'capture_qk', False):
self.qk_inputs[layer] = (q.copy(), k.copy())
q = self.normalization(q, p+'q_norm')
k = self.normalization(k, p+'k_norm')
rp = c['rope_parameters']
q, k = (rope(x, rp['rope_theta'], rp['partial_rotary_factor']) for x in (q,k))
out, probs = attention(q,k,v)
if shared_p is not None:
out = np.einsum('bhts,bshd->bthd', shared_p, np.repeat(v, heads//kvheads, axis=2))
out = (out*sigmoid(gate)).reshape(*h.shape[:-1], -1)
return self.linear(out, p+'o_proj'), probs
def linear_attention(self, h, layer):
p = f'layers.{layer}.linear_attn.'
c = self.config
kh, vh = c['linear_num_key_heads'], c['linear_num_value_heads']
kd, vd = c['linear_key_head_dim'], c['linear_value_head_dim']
qkv = self.linear(h, p+'in_proj_qkv')
qkv = causal_conv(qkv, self.small(p+'conv1d.weight')[:,0,:])
q, k, v = np.split(qkv, [kh*kd, 2*kh*kd], axis=-1)
q = q.reshape(*h.shape[:-1], kh, kd)
k = k.reshape(*h.shape[:-1], kh, kd)
v = v.reshape(*h.shape[:-1], vh, vd)
if vh != kh:
q, k = np.repeat(q, vh//kh, axis=2), np.repeat(k, vh//kh, axis=2)
z = self.linear(h, p+'in_proj_z').reshape(*h.shape[:-1], vh, vd)
beta = sigmoid(self.linear(h, p+'in_proj_b'))
a = self.linear(h, p+'in_proj_a')
g = -np.exp(self.small(p+'A_log')) * np.logaddexp(np.float32(0), a+self.small(p+'dt_bias'))
out, _ = delta_rule(q,k,v,g,beta)
out = norm(out, self.small(p+'norm.weight'), self.eps, centered=False)*silu(z)
return self.linear(out.reshape(*h.shape[:-1], -1), p+'out_proj')
def forward(self, ids, capture=None, replacement=None, progress=None):
h = self.embedding(ids)
saved = {}
capture = set(capture or [])
for layer, kind in enumerate(self.config['layer_types']):
self.guard()
raw = h
n = self.normalization(h, f'layers.{layer}.input_layernorm')
if ('attn', layer) in capture:
saved[f'attn_{layer}'] = n.copy()
donor = replacement[2] if replacement and replacement[:2] == ('attn',layer) else layer
if kind == 'full_attention':
out, _ = self.full_attention(n,donor)
else:
out = self.linear_attention(n,donor)
h = raw + out
n = self.normalization(h, f'layers.{layer}.post_attention_layernorm')
if ('mlp',layer) in capture:
saved[f'mlp_{layer}'] = n.copy()
donor = replacement[2] if replacement and replacement[:2] == ('mlp',layer) else layer
h = h + self.mlp(n,donor)
if not np.isfinite(h).all():
raise FloatingPointError(f'Nonfinite hidden states at layer {layer}')
if progress:
progress(layer)
return self.normalization(h,'norm'), saved
def nll(self, h, ids, rows=512):
# Tied output embedding; online logsumexp avoids full vocabulary logits.
x = h[:,:-1].reshape(-1,h.shape[-1])
targets = ids[:,1:].reshape(-1)
t = self.tensors[PREFIX+'embed_tokens.weight']
mx = np.full(x.shape[0],-np.inf,dtype=np.float64)
total = np.zeros(x.shape[0],dtype=np.float64)
correct = np.empty(x.shape[0],dtype=np.float64)
for start in range(0,t.shape[0],rows):
self.guard()
end=min(start+rows,t.shape[0])
w=t.read(slice(start*t.shape[1],end*t.shape[1])).reshape(end-start,t.shape[1])
scores=x@w.T
m=np.maximum(mx,scores.max(axis=-1))
total=total*np.exp(mx-m)+np.exp(scores-m[:,None]).sum(axis=-1)
mx=m
which=(targets>=start)&(targets<end)
correct[which]=scores[np.flatnonzero(which),targets[which]-start]
del w,scores
t.evict()
return (mx+np.log(total)-correct).reshape(ids.shape[0],-1)
+134
View File
@@ -0,0 +1,134 @@
"""Find repeated learned scalar MLP functions on real hidden inputs, no training.
Four fixed layers; calibration-only selection; one frozen pair; independent heldout
functional test and two actual network substitutions. No scale fitting or R.
"""
import argparse
import hashlib
import json
from pathlib import Path
import resource
import time
import numpy as np
from .model import StreamingLM,silu
from .run import write_json,save_csv
LAYERS=[5,6,20,21]
def pair_metrics(x,y):
x,y=x.astype(np.float64),y.astype(np.float64)
xx,yy=float(x@x),float(y@y);delta=float((x-y)@(x-y))
xc,yc=x-x.mean(),y-y.mean()
return {'relative_l2_a_to_b':float(np.sqrt(delta/max(xx,1e-30))),
'relative_l2_b_to_a':float(np.sqrt(delta/max(yy,1e-30))),
'cosine':float(x@y/max(np.sqrt(xx*yy),1e-30)),
'pearson':float(xc@yc/max(np.linalg.norm(xc)*np.linalg.norm(yc),1e-30)),
'rms_a':float(np.sqrt(xx/x.size)),'rms_b':float(np.sqrt(yy/y.size)),
'std_over_rms_a':float(np.std(x)/max(np.sqrt(xx/x.size),1e-30)),
'std_over_rms_b':float(np.std(y)/max(np.sqrt(yy/y.size),1e-30)),
'effective_energy_tokens_a':float(xx*xx/max(float((x**4).sum()),1e-30)),
'effective_energy_tokens_b':float(yy*yy/max(float((y**4).sum()),1e-30))}
class NeuronReuseLM(StreamingLM):
def configure(self,layer,target,donor):self.reused_layer,self.target,self.donor=layer,target,donor
def mlp(self,h,layer):
p=f'layers.{layer}.mlp.'
z=silu(self.linear(h,p+'gate_proj'))*self.linear(h,p+'up_proj')
if layer==self.reused_layer:z[...,self.target]=z[...,self.donor]
return self.linear(z,p+'down_proj')
def main():
p=argparse.ArgumentParser();p.add_argument('--root',type=Path,required=True);a=p.parse_args();root=a.root
protocol={'layers':LAYERS,'unit':'post_silu_gate_times_up_scalar_function; keep each output column',
'thresholds':[.05,.10,.20],'maximum_directional_relative_l2':.20,'cosine_min':.98,'pearson_min':.95,
'rms_floor':'at least within-layer median calibration neuron RMS','std_over_rms_min':.25,
'effective_energy_tokens_min':16,'max_selected_pairs':1,'selection':'smallest worst directional calibration error; then layer/id',
'heldout_per_text_error_max':.35,'heldout_quality_mean_nll_delta_max':.01,'heldout_quality_sequence_p95_delta_max':.05,
'training':False,'scale_fitting':False,'cost_required':False}
protocol_path=root/'neuron-reuse-protocol.json'
if protocol_path.exists():
old=json.loads(protocol_path.read_text());assert all(old[k]==v for k,v in protocol.items())
else:write_json(protocol_path,{**protocol,'registered_unix':time.time()})
ledger_path=root/'compute-budget-ledger.json';ledger=json.loads(ledger_path.read_text());remaining=int(7200-ledger['spent_seconds']-30)
if remaining<120:raise TimeoutError('Remaining cumulative budget insufficient')
started=time.monotonic();result={'state':'running','selected':None,'function_pass':False,'interventions':[]}
try:
cfg=json.loads((root/'model_config.json').read_text());lm=NeuronReuseLM(root/'weights',cfg,seconds=remaining)
data=np.load(root/'baseline-activations.npz');samples=json.loads((root/'samples-tokenized.json').read_text())['samples']
cal=np.array([i for i,s in enumerate(samples) if s['split']=='calibration']);held=np.array([i for i,s in enumerate(samples) if s['split']=='heldout'])
activations={};summaries=[];best=None
for layer in LAYERS:
h=data[f'mlp_{layer}'];pfx=f'layers.{layer}.mlp.'
z=silu(lm.linear(h,pfx+'gate_proj'))*lm.linear(h,pfx+'up_proj');activations[layer]=z
# Profiles use all calibration tokens; never use heldout in selection.
x=np.ascontiguousarray(z[cal].reshape(-1,z.shape[-1]).T,dtype=np.float64)
n=x.shape[1];normsq=(x*x).sum(1);means=x.mean(1);variance=np.maximum(normsq-n*means**2,0)
rms=np.sqrt(normsq/n);floor=float(np.median(rms));effective=normsq**2/np.maximum((x**4).sum(1),1e-30)
active=(rms>=floor)&(np.sqrt(variance/np.maximum(normsq,1e-30))>=.25)&(effective>=16)
counts={str(t):0 for t in protocol['thresholds']};tested=0;layerbest=None
for start in range(0,x.shape[0],256):
lm.guard();end=min(start+256,x.shape[0]);dot=x[start:end]@x.T
delta=np.maximum(normsq[start:end,None]+normsq[None,:]-2*dot,0)
error=np.sqrt(delta/np.maximum(np.minimum(normsq[start:end,None],normsq[None,:]),1e-30))
cosine=dot/np.maximum(np.sqrt(normsq[start:end,None]*normsq[None,:]),1e-30)
pearson=(dot-n*means[start:end,None]*means[None,:])/np.maximum(np.sqrt(variance[start:end,None]*variance[None,:]),1e-30)
unique=np.arange(start,end)[:,None]<np.arange(x.shape[0])[None,:]
allowed=unique&active[start:end,None]&active[None,:]&(cosine>=.98)&(pearson>=.95)
tested+=int((unique&active[start:end,None]&active[None,:]).sum())
for threshold in protocol['thresholds']:counts[str(threshold)]+=int((allowed&(error<=threshold)).sum())
mask=allowed&(error<=.20)
if mask.any():
score=np.where(mask,error,np.inf);row,col=np.unravel_index(np.argmin(score),score.shape)
candidate={'layer':layer,'a':int(start+row),'b':int(col),'calibration_worst_relative_l2':float(score[row,col])}
key=lambda c:(c['calibration_worst_relative_l2'],c['layer'],c['a'],c['b'])
if layerbest is None or key(candidate)<key(layerbest):layerbest=candidate
if best is None or key(candidate)<key(best):best=candidate
summaries.append({'layer':layer,'neurons':int(x.shape[0]),'eligible_neurons':int(active.sum()),'eligible_unique_pairs_checked':tested,'rms_floor':floor,'calibration_pair_counts':counts,'best':layerbest})
write_json(root/'neuron-reuse-search-summary.json',summaries)
print(json.dumps(summaries[-1]),flush=True)
np.savez(root/'neuron-reuse-activations.npz',**{f'layer{k}':v for k,v in activations.items()})
write_json(root/'neuron-reuse-selection.json',{'selected':best,'selection_uses_heldout':False})
result['selected']=best
if best is not None:
layer,i,j=best['layer'],best['a'],best['b'];z=activations[layer]
metrics=[]
for split,index in [('calibration',cal),('heldout',held)]:
metrics.append({'split':split,'scope':'pooled',**pair_metrics(z[index,:,i].reshape(-1),z[index,:,j].reshape(-1))})
for idx in index:metrics.append({'split':split,'scope':samples[idx]['id'],**pair_metrics(z[idx,:,i],z[idx,:,j])})
save_csv(root/'neuron-reuse-function-metrics.csv',metrics)
pooled=next(r for r in metrics if r['split']=='heldout' and r['scope']=='pooled')
floor=next(r['rms_floor'] for r in summaries if r['layer']==layer)
function_pass=(max(pooled['relative_l2_a_to_b'],pooled['relative_l2_b_to_a'])<=.20 and pooled['cosine']>=.98 and pooled['pearson']>=.95 and
min(pooled['std_over_rms_a'],pooled['std_over_rms_b'])>=.25 and min(pooled['effective_energy_tokens_a'],pooled['effective_energy_tokens_b'])>=16 and
min(pooled['rms_a'],pooled['rms_b'])>=floor and
all(max(r['relative_l2_a_to_b'],r['relative_l2_b_to_a'])<=.35 for r in metrics if r['split']=='heldout' and r['scope']!='pooled'))
result.update(function_pass=bool(function_pass),heldout_function_metrics=pooled)
# Show parameter similarity separately from actual functional similarity.
wd=[]
for role in ('gate_proj','up_proj'):
t=lm.tensors[f'model.language_model.layers.{layer}.mlp.{role}.weight'];d=t.shape[1]
wa,wb=t.read(slice(i*d,(i+1)*d)),t.read(slice(j*d,(j+1)*d))
wd.append({'role':role,'weight_cosine':float(wa.astype(np.float64)@wb/max(np.linalg.norm(wa.astype(np.float64))*np.linalg.norm(wb.astype(np.float64)),1e-30))});t.evict()
result['weight_diagnostics']=wd
if function_pass:
ids=np.array([s['token_ids'] for s in samples],dtype=np.int64);base=np.load(root/'baseline-heldout-token-nll.npy').mean(1)
for target,donor in ((i,j),(j,i)):
lm.configure(layer,target,donor);h,_=lm.forward(ids[held]);nll=lm.nll(h,ids[held])
np.save(root/f'neuron-reuse-{target}-uses-{donor}-token-nll.npy',nll)
delta=nll.mean(1)-base
result['interventions'].append({'target':target,'donor':donor,'sequence_nll_deltas':delta.tolist(),'mean_nll_delta':float(delta.mean()),'sequence_p95_delta':float(np.quantile(delta,.95)),'passed':bool(delta.mean()<=.01 and np.quantile(delta,.95)<=.05)})
write_json(root/'neuron-reuse-summary.json',result)
# No fallback after looking at heldout.
result.update(state='completed',empirical_repeated_function_positive=bool(result['function_pass'] and len(result['interventions'])==2 and all(r['passed'] for r in result['interventions'])),
claim_limit='Distinct learned scalar MLP subfunctions on the specified short-text distribution; not whole-layer duplication, global equivalence or broad task-capability proof. No performance requirement.')
except Exception as exc:result.update(state='stopped',error_type=type(exc).__name__,error=str(exc));raise
finally:
elapsed=time.monotonic()-started;result.update(elapsed_seconds=elapsed,peak_rss_kib=resource.getrusage(resource.RUSAGE_SELF).ru_maxrss)
ledger=json.loads(ledger_path.read_text());ledger['spent_seconds']+=elapsed;ledger['calls'].append({'module':'functional.neuron_reuse','seconds':elapsed});write_json(ledger_path,ledger)
write_json(root/'neuron-reuse-summary.json',result)
if __name__=='__main__':main()
+108
View File
@@ -0,0 +1,108 @@
"""Existing learned RMSNorm modules: uniform function bound + independent probes.
Deterministic exhaustive selection by a mathematical all-input error bound, not
calibration/heldout fitting. Same role, shape and eps; one pair only.
"""
import argparse
import json
from pathlib import Path
import resource
import time
import numpy as np
from .model import StreamingLM,norm
from .run import write_json,save_csv,error_rows
ROLES=['input_layernorm','post_attention_layernorm']
def uniform_bounds(a,b):
if np.any(a==0) or np.any(b==0):return None
diff=np.abs(a.astype(np.float64)-b.astype(np.float64))
return float(np.max(diff/np.abs(a))),float(np.max(diff/np.abs(b)))
def main():
p=argparse.ArgumentParser();p.add_argument('--root',type=Path,required=True);args=p.parse_args();root=args.root
protocol={'roles':ROLES,'layers':list(range(24)),'same_role_only':True,'same_eps':1e-6,
'uniform_worst_error_limit':.20,'empirical_function_worst_relative_error_limit':.05,
'selection':'smallest maximum two-direction uniform bound, then role/layer; no input data used',
'minimum_gain_standard_deviation':.001,'max_selected_pairs':1,'heldout_nll_mean_delta_limit':.01,
'heldout_nll_sequence_p95_delta_limit':.05,'training':False,'cost_required':False}
path=root/'norm-repetition-protocol.json'
if path.exists():
existing=json.loads(path.read_text());assert all(existing[k]==v for k,v in protocol.items())
else:write_json(path,{**protocol,'registered_unix':time.time()})
ledger_path=root/'compute-budget-ledger.json';ledger=json.loads(ledger_path.read_text());remaining=int(7200-ledger['spent_seconds']-30)
if remaining<120:raise TimeoutError('Insufficient cumulative computation budget')
started=time.monotonic();result={'state':'running','selected':None,'interventions':[]}
try:
cfg=json.loads((root/'model_config.json').read_text());lm=StreamingLM(root/'weights',cfg,seconds=remaining)
assert lm.eps==1e-6
candidates=[];gains={}
for role in ROLES:
for layer in range(24):gains[(role,layer)]=1+lm.small(f'layers.{layer}.{role}.weight')
for i in range(24):
for j in range(i+1,24):
a,b=gains[(role,i)],gains[(role,j)]
bounds=uniform_bounds(a,b)
if bounds is None:continue
eligible=min(float(np.std(a)),float(np.std(b)))>=.001
candidates.append({'role':role,'a':i,'b':j,'a_to_b_uniform_bound':bounds[0],'b_to_a_uniform_bound':bounds[1],
'worst_uniform_bound':max(bounds),'eligible_nonconstant_gains':eligible,
'gain_std_a':float(np.std(a)),'gain_std_b':float(np.std(b)),
'gain_min_a':float(np.min(a)),'gain_min_b':float(np.min(b))})
save_csv(root/'norm-repetition-uniform-bounds.csv',candidates)
feasible=[c for c in candidates if c['eligible_nonconstant_gains'] and c['worst_uniform_bound']<=.20]
counts={str(t):sum(c['eligible_nonconstant_gains'] and c['worst_uniform_bound']<=t for c in candidates) for t in (.05,.10,.20)}
selected=sorted(feasible,key=lambda c:(c['worst_uniform_bound'],c['role'],c['a'],c['b']))[:1]
write_json(root/'norm-repetition-selection.json',{'selected':selected,'uniform_bound_counts':counts,'selection_reads_no_input_data':True})
result['uniform_bound_counts']=counts
if selected:
c=selected[0];result['selected']=c
# Selection is frozen before reading tokenized inputs or their activations.
samples=json.loads((root/'samples-tokenized.json').read_text())['samples'];ids=np.array([s['token_ids'] for s in samples],dtype=np.int64)
held=np.array([i for i,s in enumerate(samples) if s['split']=='heldout'])
kind='attn' if c['role']=='input_layernorm' else 'mlp'
captures=[(kind,c['a']),(kind,c['b'])]
h,saved=lm.forward(ids,capture=captures)
# The untouched recomputation must reproduce the saved real baseline.
original=np.load(root/'baseline-activations.npz')['final']
np.testing.assert_array_equal(h,original)
np.savez(root/'norm-repetition-input-profiles.npz',**saved)
ga,gb=gains[(c['role'],c['a'])],gains[(c['role'],c['b'])]
rows=[]
for source in (c['a'],c['b']):
n=saved[f'{kind}_{source}'];g=gains[(c['role'],source)]
# n = u * gamma_source. Reconstruct common normalized input u;
# source gains are nonzero. Additional rounding is ~FP32 only.
unit=n/g
fa,fb=unit*ga,unit*gb
rec=error_rows(fa,fb,samples,{'source_layer':source,'role':c['role'],'a':c['a'],'b':c['b']})
rows.extend(rec)
save_csv(root/'norm-repetition-function-metrics.csv',rows)
heldrows=[r for r in rows if r['split']=='heldout']
function_pass=all(max(r['relative_l2'],r['reverse_relative_l2'])<=.05 for r in heldrows)
result.update(uniform_function_similarity_proven=True,heldout_function_pass=bool(function_pass),
worst_heldout_relative_l2=max(max(r['relative_l2'],r['reverse_relative_l2']) for r in heldrows),
worst_heldout_cosine=min(r['cosine'] for r in heldrows),untouched_full_forward_bitwise_matches_baseline=True)
if function_pass:
baseline=np.load(root/'baseline-heldout-token-nll.npy').mean(1)
for target,donor in ((c['a'],c['b']),(c['b'],c['a'])):
lm.norm_replacement=(f'layers.{target}.{c["role"]}',f'layers.{donor}.{c["role"]}')
changed,_=lm.forward(ids[held]);nll=lm.nll(changed,ids[held]);np.save(root/f'norm-repetition-{target}-uses-{donor}-token-nll.npy',nll)
delta=nll.mean(1)-baseline
result['interventions'].append({'target':target,'donor':donor,'sequence_nll_deltas':delta.tolist(),
'mean_nll_delta':float(delta.mean()),'sequence_p95_nll_delta':float(np.quantile(delta,.95)),
'passed_quality':bool(delta.mean()<=.01 and np.quantile(delta,.95)<=.05)})
write_json(root/'norm-repetition-summary.json',result)
result['learned_function_repetition_positive']=bool(function_pass)
result['bidirectional_network_quality_pass']=bool(len(result['interventions'])==2 and all(x['passed_quality'] for x in result['interventions']))
result.update(state='completed',claim_limit='Existing learned RMSNorm module functions with identical eps and shape; uniform real-arithmetic relative bound and tiny real-text probes. Does not establish attention/MLP-body duplication or broad task capability.')
except Exception as exc:result.update(state='stopped',error_type=type(exc).__name__,error=str(exc));raise
finally:
elapsed=time.monotonic()-started;result.update(elapsed_seconds=elapsed,peak_rss_kib=resource.getrusage(resource.RUSAGE_SELF).ru_maxrss)
ledger=json.loads(ledger_path.read_text());ledger['spent_seconds']+=elapsed;ledger['calls'].append({'module':'functional.norm_repetition','seconds':elapsed});write_json(ledger_path,ledger)
write_json(root/'norm-repetition-summary.json',result)
if __name__=='__main__':main()
+46
View File
@@ -0,0 +1,46 @@
"""Package code-independent scientific outputs, retain large activation caches externally.
The experiment directory remains immutable and complete. The delivery directory
contains all figure source tables/NLL arrays plus hashes/locations for three large
reproducible activation caches. Never deletes model weights or cache files.
"""
import argparse
import hashlib
import json
from pathlib import Path
import shutil
from .run import write_json
CACHES={'baseline-activations.npz','neuron-reuse-activations.npz','norm-repetition-input-profiles.npz'}
def main():
p=argparse.ArgumentParser();p.add_argument('--root',required=True,type=Path);p.add_argument('--output',required=True,type=Path);a=p.parse_args()
root,out=a.root.resolve(),a.output.resolve();assert root!=out
out.mkdir(parents=True,exist_ok=True)
full=json.loads((root/'checksums-functional.json').read_text())
files=json.loads((root/'archive-files.json').read_text())
for name in files:
src=root/name
if name in CACHES:continue
dest=out/name;dest.parent.mkdir(parents=True,exist_ok=True);shutil.copy2(src,dest)
external=[]
for name in sorted(CACHES):
src=root/name;digest=hashlib.sha256(src.read_bytes()).hexdigest();assert digest==full[name]
external.append({'name':name,'server_path':str(src),'bytes':src.stat().st_size,'sha256':digest,'retained':True,'reproducible_by':'functional.run / functional.neuron_reuse / functional.norm_repetition'})
write_json(out/'external-activation-cache.json',{'host':'8.211.154.134','files':external,'not_model_weights':True,
'reason':'Large intermediate caches retained on the authorized server; all figure source tables and token NLL arrays are delivered in Git.'})
shutil.copy2(root/'checksums-functional.json',out/'checksums-full-server.json')
(out/'DELIVERY.md').write_text('# Scientific delivery and retained caches\n\nAll figure CSV/JSON source data, per-token NLL, protocols, reports, metadata and historical logs are included.\nThree large reproducible intermediate activation caches remain on 8.211.154.134 and in the original full local archive;\n`external-activation-cache.json` records exact locations, sizes and SHA256. No weight or cache was deleted.\nThe original full experiment manifest is `checksums-full-server.json`; the delivery-only manifest is\n`checksums-functional.json`. No model checkpoint or tokenizer payload is in this Git delivery.\n')
validation=json.loads((out/'audit-validation.json').read_text());validation.update(delivery_excludes_external_activation_caches=True,full_experiment_checksums='checksums-full-server.json',figure_source_tables_and_token_nll_complete=True)
write_json(out/'audit-validation.json',validation)
checks={}
for file in sorted(out.rglob('*')):
if file.is_file() and file.name not in ('checksums-functional.json','audit-validation.json','archive-files.json'):
checks[str(file.relative_to(out))]=hashlib.sha256(file.read_bytes()).hexdigest()
write_json(out/'checksums-functional.json',checks)
write_json(out/'archive-files.json',[*checks,'checksums-functional.json','audit-validation.json','archive-files.json'])
print(json.dumps({'delivery_files':len(checks)+3,'checksummed_files':len(checks),'external_caches':len(external),'weights_and_caches_retained':True}))
if __name__=='__main__':main()
+39
View File
@@ -0,0 +1,39 @@
"""One 8-token real text, bounded memory/semantic capacity smoke check."""
import argparse
import json
from pathlib import Path
import resource
import time
import numpy as np
from .model import StreamingLM
from .run import tokenize,write_json
def main():
p=argparse.ArgumentParser();p.add_argument('--root',required=True,type=Path);a=p.parse_args()
samples,ids=tokenize(a.root)
start=time.time()
lm=StreamingLM(a.root/'weights',json.loads((a.root/'model_config.json').read_text()),seconds=1200)
snapshots=[]
result={'passed':False,'sample_id':samples[0]['id'],'tokens':8}
def progress(layer):
status=dict(line.split(':',1) for line in Path('/proc/self/status').read_text().splitlines() if ':' in line)
snapshots.append({'layer':layer,'seconds':time.time()-start,'rss_kib':int(status['VmRSS'].split()[0]),
'process_swap_kib':int(status.get('VmSwap','0 kB').split()[0]),
'available_kib':int(next(line.split()[1] for line in Path('/proc/meminfo').read_text().splitlines() if line.startswith('MemAvailable:')))})
print(json.dumps(snapshots[-1]),flush=True)
try:
h,_=lm.forward(ids[:1,:8],progress=progress)
nll=lm.nll(h,ids[:1,:8])
if not np.isfinite(nll).all():raise FloatingPointError('Nonfinite NLL')
result.update(passed=True,mean_nll=float(nll.mean()),token_nll=nll.tolist())
except Exception as exc:
result.update(error_type=type(exc).__name__,error=str(exc))
raise
finally:
result.update(elapsed_seconds=time.time()-start,peak_rss_kib=resource.getrusage(resource.RUSAGE_SELF).ru_maxrss,
resource_snapshots=snapshots,limitation='Real-weight capacity check, not an independent correctness oracle or quality benchmark')
write_json(a.root/'preflight.json',result)
if __name__=='__main__':main()
+5
View File
@@ -0,0 +1,5 @@
{
"commit": "fd3cf0140de03f7619ac95b5cfed05b711ff9e22",
"url": "https://raw.githubusercontent.com/huggingface/transformers/fd3cf0140de03f7619ac95b5cfed05b711ff9e22/src/transformers/models/qwen3_5/modeling_qwen3_5.py",
"sha256": "4463b8a108f678c166755101d95f188179f54143305953d78147de0389b80ebc"
}
+142
View File
@@ -0,0 +1,142 @@
"""Remote-only numerical comparison with pinned official Torch definitions.
Reads only a known official Transformers source file; extracts named mathematical
operators/classes via AST, removing optional kernel decorators. Never imports the
complete model or executes checkpoint repository code. No compilation/build.
"""
import argparse
import ast
import hashlib
import json
from pathlib import Path
from types import SimpleNamespace
import numpy as np
from .model import StreamingLM, delta_rule, rope, norm
EXPECTED_COMMIT = 'fd3cf0140de03f7619ac95b5cfed05b711ff9e22'
NAMES = {'apply_mask_to_padding_states', 'l2norm', 'torch_chunk_gated_delta_rule', 'torch_recurrent_gated_delta_rule',
'causal_conv1d_fn', 'Qwen3_5RMSNormGated', 'Qwen3_5GatedDeltaNet',
'rotate_half', 'apply_rotary_pos_emb', 'repeat_kv', 'eager_attention_forward',
'Qwen3_5Attention', 'Qwen3_5MLP', 'Qwen3_5RMSNorm', 'Qwen3_5DecoderLayer',
'Qwen3_5TextRotaryEmbedding'}
def official_namespace(source):
import torch
import torch.nn.functional as F
tree = ast.parse(source)
nodes = []
for node in tree.body:
if isinstance(node, (ast.FunctionDef, ast.ClassDef)) and node.name in NAMES:
for child in ast.walk(node):
if isinstance(child, (ast.FunctionDef, ast.ClassDef)):
child.decorator_list = [d for d in child.decorator_list if isinstance(d, ast.Name) and d.id in ('staticmethod','classmethod')]
nodes.append(node)
assert {x.name for x in nodes} == NAMES
tree = ast.Module(body=[ast.ImportFrom(module='__future__',names=[ast.alias(name='annotations')],level=0),*nodes],type_ignores=[])
ns = {'torch':torch, 'nn':torch.nn, 'F':F, 'ACT2FN':{'silu':F.silu},
'GradientCheckpointingLayer':torch.nn.Module,
'is_torchdynamo_exporting':lambda:False,
'ALL_ATTENTION_FUNCTIONS':SimpleNamespace(get_interface=lambda name, fallback:fallback)}
exec(compile(ast.fix_missing_locations(tree),'pinned_transformers_reference','exec'),ns)
return ns
class ArrayLM(StreamingLM):
def __init__(self, arrays, config):
self.arrays, self.config, self.eps = arrays,config,config['rms_norm_eps']
def guard(self):
pass
def small(self,name):
return self.arrays[name]
def linear(self,x,name,rows=256):
return x@self.arrays[name+'.weight'].T
def embedding(self,ids):
return self.arrays['embed_tokens.weight'][ids]
def main():
p=argparse.ArgumentParser();p.add_argument('--source',type=Path,required=True);p.add_argument('--output',type=Path,required=True)
a=p.parse_args()
import torch
torch.set_num_threads(1)
torch.manual_seed(42)
provenance=json.loads(Path(__file__).with_name('reference.json').read_text())
assert hashlib.sha256(a.source.read_bytes()).hexdigest()==provenance['sha256'], 'Reference source hash mismatch'
ns=official_namespace(a.source.read_text())
cfg={'hidden_size':16,'num_attention_heads':2,'num_key_value_heads':1,'head_dim':8,
'intermediate_size':24,'hidden_act':'silu','attention_bias':False,'attention_dropout':0.,
'linear_num_value_heads':2,'linear_num_key_heads':2,'linear_key_head_dim':4,
'linear_value_head_dim':4,'linear_conv_kernel_dim':4,'rms_norm_eps':1e-6,
'layer_types':['linear_attention','full_attention'],'max_position_embeddings':4096,
'rope_parameters':{'rope_type':'default','rope_theta':10000000.,'partial_rotary_factor':.5,'mrope_section':[1,0,0]},
'_attn_implementation':'eager'}
config=SimpleNamespace(**cfg)
layers=[ns['Qwen3_5DecoderLayer'](config,i).eval() for i in range(2)]
for layer in layers:
for name,t in layer.named_parameters():
with torch.no_grad():
t.uniform_(-.2,.2)
if name=='linear_attn.norm.weight':t.add_(1.)
arrays={f'layers.{i}.'+k:v.detach().numpy().copy() for i,l in enumerate(layers) for k,v in l.state_dict().items()}
lm=ArrayLM(arrays,cfg)
rng=np.random.default_rng(123)
x=rng.standard_normal((2,7,16),dtype=np.float32)
tx=torch.from_numpy(x)
positions=torch.arange(7)[None,:]
rotary=ns['Qwen3_5TextRotaryEmbedding'](config)
cos,sin=rotary(tx,positions)
mask=torch.triu(torch.full((7,7),-float('inf')),diagonal=1)[None,None]
evidence=[]
def check(name,actual,expected,atol=2e-5,rtol=2e-5):
actual=np.asarray(actual);expected=expected.detach().numpy() if hasattr(expected,'detach') else np.asarray(expected)
np.testing.assert_allclose(actual,expected,atol=atol,rtol=rtol)
evidence.append({'test':name,'shape':list(actual.shape),'max_abs_error':float(np.max(np.abs(actual-expected))),'atol':atol,'rtol':rtol,'passed':True})
with torch.no_grad():
for i,layer in enumerate(layers):
n=lm.normalization(x,f'layers.{i}.input_layernorm')
tn=layer.input_layernorm(torch.from_numpy(x))
check(f'layer{i}_rmsnorm',n,tn)
if i==0:
out=lm.linear_attention(n,i)
tout=layer.linear_attn(tn)
else:
out,pr=lm.full_attention(n,i)
tout,tpr=layer.self_attn(tn,(cos,sin),mask)
check('attention_probabilities',pr,tpr)
check(f'layer{i}_mixer',out,tout)
h=x+out
n=lm.normalization(h,f'layers.{i}.post_attention_layernorm')
check(f'layer{i}_mlp',lm.mlp(n,i),layer.mlp(layer.post_attention_layernorm(torch.from_numpy(h))))
actual=h+lm.mlp(n,i)
expected=layer(torch.from_numpy(x),(cos,sin),attention_mask=mask if i else None)
check(f'layer{i}_complete',actual,expected)
# Independent recurrence/chunk equality, including nonmultiple sequence length.
for length in (1,7,33):
q=rng.standard_normal((2,length,2,4),dtype=np.float32)
k=rng.standard_normal(q.shape,dtype=np.float32)
v=rng.standard_normal((2,length,2,5),dtype=np.float32)
g=-np.abs(rng.standard_normal((2,length,2),dtype=np.float32))
beta=rng.uniform(size=g.shape).astype(np.float32)
out,state=delta_rule(q,k,v,g,beta)
args=[torch.from_numpy(z) for z in (q,k,v,g,beta)]
for kind in ('torch_recurrent_gated_delta_rule','torch_chunk_gated_delta_rule'):
ref,refstate=ns[kind](*args,output_final_state=True,use_qk_l2norm_in_kernel=True)
check(f'{kind}_length{length}_output',out,ref)
check(f'{kind}_length{length}_state',state,refstate)
# Full two-layer residual forward, final norm and embedding lookup.
arrays['embed_tokens.weight']=rng.standard_normal((31,16),dtype=np.float32)
arrays['norm.weight']=rng.standard_normal((16,),dtype=np.float32)*.1
ids=rng.integers(0,31,size=(2,7))
h=torch.from_numpy(arrays['embed_tokens.weight'][ids])
for i,l in enumerate(layers):h=l(h,(cos,sin),attention_mask=mask if i else None)
expected=h*torch.rsqrt(h.square().mean(-1,keepdim=True)+1e-6)*(1+torch.from_numpy(arrays['norm.weight']))
actual,_=lm.forward(ids)
check('complete_two_layer_text_forward',actual,expected)
result={'passed':True,'reference_commit':EXPECTED_COMMIT,'source_sha256':hashlib.sha256(a.source.read_bytes()).hexdigest(),
'torch':torch.__version__,'numpy':np.__version__,'implementation_sha256':hashlib.sha256(Path(__file__).with_name('model.py').read_bytes()).hexdigest(),'tests':evidence,'limitations':'FP32 CPU semantics on tiny synthetic configurations; not full-checkpoint BF16 kernel equivalence'}
a.output.write_text(json.dumps(result,indent=2))
print(json.dumps({'passed':True,'checks':len(evidence),'maximum_absolute_discrepancy':max(x['max_abs_error'] for x in evidence)}))
if __name__=='__main__':main()
+104
View File
@@ -0,0 +1,104 @@
"""Render scientific pilot figures using saved data; never reopen model weights."""
import argparse
import csv
import json
from pathlib import Path
import numpy as np
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
def main():
p=argparse.ArgumentParser();p.add_argument('--root',type=Path,required=True);a=p.parse_args();root=a.root
out=root/'figures';out.mkdir(exist_ok=True)
plt.rcParams.update({'font.size':10,'savefig.bbox':'tight','pdf.fonttype':42,'svg.fonttype':'none'})
def save(fig,name):
for ext in ('png','pdf','svg'):fig.savefig(out/f'{name}.{ext}',dpi=180)
plt.close(fig)
status=json.loads((root/'functional-status.json').read_text())
if (root/'candidate-summary.json').exists():
c=json.loads((root/'candidate-summary.json').read_text())
fig,ax=plt.subplots(figsize=(10,4.5))
ax.bar(np.arange(len(c)),[max(v['maximum_symmetric_relative_error'],1e-8) for v in c],color=['#238b45' if v['passed'] else '#9e9e9e' for v in c])
ax.axhline(.05,color='#b2182b',linestyle='--',label='Relative L2 screen = 0.05; other gates also apply')
ax.set_yscale('log');ax.set_xticks(np.arange(len(c)),[v['id'] for v in c],rotation=45,ha='right')
ax.set_ylabel('Worst calibration directional relative L2')
ax.set_title('Fixed candidates; same-input function comparison\n4 calibration texts, 32 tokens; no training')
ax.legend(loc='upper left',bbox_to_anchor=(0,-.38),fontsize=8);fig.tight_layout();save(fig,'candidate_function_errors')
if (root/'interventions.json').exists():
interventions=json.loads((root/'interventions.json').read_text())
fig,ax=plt.subplots(figsize=(8,4.5))
for idx,r in enumerate(interventions):
values=r['sequence_nll_deltas'];x=np.arange(len(values))
ax.plot(x,values,'o-',label=r['direction'])
ax.axhline(0,color='gray',linewidth=.7);ax.axhline(.01,color='#b2182b',linestyle='--',label='Mean-delta budget (not per-sequence bound)')
ax.set_xticks(np.arange(4),['heldout zh1','heldout zh2','heldout en1','heldout en2'])
ax.set_ylabel('Token NLL change (nat/token)');ax.set_title('Single-site interventions; frozen candidate\n4 heldout texts, 32 tokens; limited pilot evidence')
ax.legend(loc='upper left',bbox_to_anchor=(0,-.18),fontsize=8);fig.tight_layout();save(fig,'heldout_intervention_nll')
norm_path=root/'norm-repetition-summary.json'
repetition=json.loads(norm_path.read_text()) if norm_path.exists() else {}
if (root/'norm-repetition-function-metrics.csv').exists():
with (root/'norm-repetition-function-metrics.csv').open() as f:rows=list(csv.DictReader(f))
sample_ids=list(dict.fromkeys(row['sample_id'] for row in rows))
fig,ax=plt.subplots(figsize=(9,4.8))
for source in sorted({row['source_layer'] for row in rows},key=int):
rr=[r for r in rows if r['source_layer']==source]
ax.plot(range(len(rr)),[max(float(r['relative_l2']),float(r['reverse_relative_l2'])) for r in rr],'o-',label=f'Input captured at layer {source}')
ax.axhline(.05,color='#b2182b',ls='--',label='Heldout empirical screen = 0.05')
chosen=repetition.get('selected')
if chosen:ax.axhline(chosen['worst_uniform_bound'],color='#6a51a3',ls=':',label='Proven uniform all-input bound')
ax.axvline(3.5,color='gray',linewidth=.8)
ax.set_xticks(range(len(sample_ids)),sample_ids,rotation=30,ha='right');ax.set_ylabel('Worst of two directional relative L2 errors')
ax.set_title('Learned input RMSNorm at layer indices 12/13: same-input comparison\nCalibration (left) / independent heldout texts (right), 32 tokens each')
ax.legend(loc='upper left',bbox_to_anchor=(0,-.25),fontsize=8);fig.tight_layout();save(fig,'learned_norm_function_errors')
if repetition.get('interventions'):
fig,ax=plt.subplots(figsize=(8,4.5))
for v in repetition['interventions']:
ax.plot(range(4),v['sequence_nll_deltas'],'o-',label=f'Layer {v["target"]} uses {v["donor"]}; mean={v["mean_nll_delta"]:.5g}')
ax.axhline(0,color='gray',linewidth=.8);ax.set_xticks(range(4),['zh1','zh2','en1','en2'])
ax.set_xlabel('Independent heldout text (32 tokens)');ax.set_ylabel('Mean token NLL change (nat/token)')
ax.set_title('Bidirectional single-site substitution of existing learned RMSNorm modules')
ax.legend(loc='upper left',bbox_to_anchor=(0,-.18),fontsize=8);fig.tight_layout();save(fig,'learned_norm_network_substitution')
if (root/'gain-factorization-score-metrics.csv').exists():
with (root/'gain-factorization-score-metrics.csv').open() as f:rr=list(csv.DictReader(f))
layers=sorted({int(v['layer']) for v in rr});fig,axes=plt.subplots(1,2,figsize=(10,4))
for ax,key,label in zip(axes,['score_max_abs_error','probability_max_abs_error'],['Maximum absolute score error','Maximum absolute probability error']):
for direction in ('q_to_k','k_to_q'):
ax.plot(layers,[max(float(v[key]) for v in rr if int(v['layer'])==layer and v['direction']==direction) for layer in layers],'o-',label=direction)
ax.set_xlabel('Full-attention layer index');ax.set_xticks(layers);ax.set_ylabel(label);ax.ticklabel_format(axis='y',style='sci',scilimits=(0,0));ax.legend(loc='upper left',fontsize=8)
fig.suptitle('Supplement: algebraic reparameterization, separate from learned-function repetition');fig.tight_layout();save(fig,'algebraic_gain_identity')
neuron_path=root/'neuron-reuse-summary.json'
neuron=json.loads(neuron_path.read_text()) if neuron_path.exists() else {}
text=['# CPU learned-function repetition pilot','',
'Primary question: do two distinct trained parts implement similar functions? Cost is not a success condition.',
f'- Existing learned RMSNorm function positive: {repetition.get("learned_function_repetition_positive",False)}',
f'- Bidirectional network quality screen passed: {repetition.get("bidirectional_network_quality_pass",False)}',
f'- Learned scalar MLP neuron search positive: {neuron.get("empirical_repeated_function_positive",False)}',
f'- Original 11 candidate run state: {status["state"]}; selected: {status.get("selected_candidate",[])}',
'- All model weights are retained on the server until the user explicitly requests deletion.','']
if repetition.get('selected'):
c=repetition['selected'];text += ['## Existing learned module evidence','',
f'Role: {c["role"]}; layers {c["a"]} and {c["b"]}; each maps 2048-dimensional input to output.',
f'Uniform real-arithmetic relative-error bound, first-to-second: {c["a_to_b_uniform_bound"]:.12g}; reverse: {c["b_to_a_uniform_bound"]:.12g}.',
f'Maximum heldout directional output relative L2: {repetition.get("worst_heldout_relative_l2")}; minimum cosine: {repetition.get("worst_heldout_cosine")}.',
'Same epsilon and nonzero gains: the common RMS denominator cancels. For any nonzero input x,',
'`||f_a(x)-f_b(x)|| / ||f_a(x)|| <= max_j |a_j-b_j|/|a_j|`; swap a/b for the reverse direction.',
'Selection used only the tight all-input bound across 552 same-role pairs, before inspecting inputs.',
'Real probes use four calibration and four heldout experiment-authored texts, 32 tokens each, and both source layers.',
'These are learned, nonconstant gain vectors. The claim concerns RMSNorm modules; it does not establish attention/MLP-body duplication.','']
for v in repetition.get('interventions',[]):text += [f'- Layer {v["target"]} uses {v["donor"]}: mean heldout NLL delta {v["mean_nll_delta"]:.12g}; sequence P95 delta {v["sequence_p95_nll_delta"]:.12g}; screen pass={v["passed_quality"]}.']
text += ['', '## Negative results and separate algebraic result','',
'The original whole-attention, whole-MLP and within-layer Q/K norm candidates did not pass their gates.',
'The scalar neuron search excludes low-RMS/near-constant profiles and selects at most one pair without heldout feedback; consult its search summary for coverage.',
'Q/K non-RoPE gain consolidation is a separately verified real-arithmetic identity, not evidence that two existing learned modules duplicate functions.',
'Earlier local CPU timing records are preserved for provenance; performance is outside the current success criterion.','',
'## Limits and provenance','',
'Original BF16 weights were decoded into streaming FP32 CPU operators; no training, scale fitting, R fitting or GPU.',
'The numerical reference includes 22 independent official Torch comparisons. Early adapter errors and two resource-guard stops are preserved as historical logs.',
'The persistent computation ledger includes a conservative 180-second reserve for early checks; no budget reset after that reserve.',
'The text sample is tiny and short. Network NLL checks are not broad task-accuracy validation.','']
(root/'FUNCTIONAL_REPORT.md').write_text('\n'.join(text))
if __name__=='__main__':main()
+227
View File
@@ -0,0 +1,227 @@
"""Bounded CPU pilot. One fixed model; calibration selects, heldout only validates."""
import argparse
import csv
import hashlib
import json
import os
from pathlib import Path
import resource
import time
import traceback
import urllib.request
import numpy as np
from .model import StreamingLM, norm
ATTN_PAIRS=[(3,7),(11,15),(19,23)]
MLP_PAIRS=[(5,6),(20,21)]
NORM_LAYERS=[3,7,11,15,19,23]
REVISION='15852e8c16360a2fea060d615a32b45270f8a8fc'
def write_json(path,obj):
tmp=path.with_suffix('.tmp')
tmp.write_text(json.dumps(obj,indent=2,allow_nan=False)+'\n')
tmp.replace(path)
def error_rows(a,b,samples,metadata):
records=[]
for x,y,sample in zip(a,b,samples):
xf,yf=x.astype(np.float64).reshape(-1),y.astype(np.float64).reshape(-1)
xn,yn=np.linalg.norm(xf),np.linalg.norm(yf)
delta=np.linalg.norm(xf-yf)
tokenerr=np.linalg.norm((x-y).reshape(x.shape[0],-1).astype(np.float64),axis=-1)/np.maximum(np.linalg.norm(x.reshape(x.shape[0],-1).astype(np.float64),axis=-1),1e-12)
records.append({**metadata,'sample_id':sample['id'],'split':sample['split'],
'relative_l2':float(delta/max(xn,1e-12)),
'reverse_relative_l2':float(delta/max(yn,1e-12)),
'token_relative_p95':float(np.quantile(tokenerr,.95)),
'reverse_token_relative_p95':float(np.quantile(np.linalg.norm((x-y).reshape(x.shape[0],-1).astype(np.float64),axis=-1)/np.maximum(np.linalg.norm(y.reshape(x.shape[0],-1).astype(np.float64),axis=-1),1e-12),.95)),
'cosine':float(xf@yf/max(xn*yn,1e-30)),
'low_norm_tokens':int((np.linalg.norm(x.reshape(x.shape[0],-1),axis=-1)<1e-8).sum())})
return records
def probability_metrics(a,b):
m=(a+b)*.5
def term(p):
return np.where(p>0,p*np.log(np.maximum(p,1e-30)/np.maximum(m,1e-30)),0)
js=(term(a)+term(b)).sum(axis=-1)*.5
overlap=[]
for row in range(a.shape[-2]):
k=min(5,row+1)
ai=np.argsort(a[:,:,row,:row+1],axis=-1)[...,-k:]
bi=np.argsort(b[:,:,row,:row+1],axis=-1)[...,-k:]
overlap.append(np.mean((ai[...,None]==bi[...,None,:]).any(axis=-1),axis=-1))
# Attention-sink control: remove key zero, renormalize valid remaining mass.
aa,bb=a[...,1:],b[...,1:]
mass_a,mass_b=aa.sum(-1),bb.sum(-1)
valid=(mass_a>1e-12)&(mass_b>1e-12)
aa=aa/np.maximum(mass_a[...,None],1e-30)
bb=bb/np.maximum(mass_b[...,None],1e-30)
mm=(aa+bb)*.5
conditional_js=.5*(np.where(aa>0,aa*np.log(np.maximum(aa,1e-30)/np.maximum(mm,1e-30)),0)+np.where(bb>0,bb*np.log(np.maximum(bb,1e-30)/np.maximum(mm,1e-30)),0)).sum(-1)
return {'first_key_mass_a':float(a[...,0].mean()),'first_key_mass_b':float(b[...,0].mean()),
'js_without_first_key':float(conditional_js[valid].mean()) if valid.any() else None,
'without_first_key_valid_rows':int(valid.sum()),'js_mean':float(js.mean()),'js_p95':float(np.quantile(js,.95)),
'js_without_first_query':float(js[...,1:].mean()),
'top5_overlap':float(np.stack(overlap,axis=-1).mean())}
def save_csv(path,rows):
if not rows:return
keys=list(dict.fromkeys(k for r in rows for k in r))
with path.open('w',newline='') as f:
w=csv.DictWriter(f,fieldnames=keys);w.writeheader();w.writerows(rows)
def tokenize(root):
from tokenizers import Tokenizer
path=root/'tokenizer.json'
if not path.exists():
url=f'https://huggingface.co/Qwen/Qwen3.5-2B/resolve/{REVISION}/tokenizer.json'
with urllib.request.urlopen(url,timeout=180) as r:path.write_bytes(r.read())
tokenizer=Tokenizer.from_file(str(path))
samples=json.loads(Path(__file__).with_name('samples.json').read_text())['samples']
ids=[]
for s in samples:
tokens=tokenizer.encode(s['text'],add_special_tokens=False).ids
if len(tokens)<32:raise ValueError(f'Text too short: {s["id"]}')
s['text_sha256']=hashlib.sha256(s['text'].encode()).hexdigest()
s['full_token_count']=len(tokens);s['token_ids']=tokens[:32]
ids.append(tokens[:32])
write_json(root/'samples-tokenized.json',{'revision':REVISION,'tokenizer_sha256':hashlib.sha256(path.read_bytes()).hexdigest(),'samples':samples})
return samples,np.array(ids,dtype=np.int64)
def main():
p=argparse.ArgumentParser();p.add_argument('--root',type=Path,required=True);p.add_argument('--seconds',type=int,default=7200)
args=p.parse_args();root=args.root;root.mkdir(exist_ok=True,parents=True)
write_json(root/'retention-policy.json',{'weights':'retain_until_explicit_user_delete_request','automatic_cleanup':False,'instruction_date':'2026-11-09'})
samples,ids=tokenize(root)
cal=np.array([i for i,s in enumerate(samples) if s['split']=='calibration'])
held=np.array([i for i,s in enumerate(samples) if s['split']=='heldout'])
# Freeze all candidates/thresholds before opening weights or measuring functions.
experiment={'revision':REVISION,'attention_pairs':ATTN_PAIRS,'mlp_pairs':MLP_PAIRS,'norm_layers':NORM_LAYERS,
'seconds':args.seconds,'output_relative_l2_limit':.05,'token_p95_limit':.1,
'attention_js_limit':.01,'attention_top5_min':.8,'heldout_nll_mean_delta_limit':.01,
'heldout_nll_sequence_p95_delta_limit':.05,'max_intervention_candidates':1,
'directions_per_selected_candidate':2,'training':False,'search_R':False,'automatic_weight_cleanup':False,
'dtype':'original weights decoded to FP32; not a BF16 backend benchmark',
'blas_threads':os.environ.get('OPENBLAS_NUM_THREADS')}
write_json(root/'experiment.json',experiment)
started=time.time();status={'state':'running','started_unix':started,'phase':'baseline'}
write_json(root/'functional-status.json',status)
rows=[];headrows=[];interventions=[]
try:
config=json.loads((root/'model_config.json').read_text())
lm=StreamingLM(root/'weights',config,seconds=args.seconds)
overall_deadline=lm.deadline
lm.deadline=min(overall_deadline,time.monotonic()+2400)
lm.capture_qk=True
capture=[('attn',i) for pair in ATTN_PAIRS for i in pair]+[('mlp',i) for pair in MLP_PAIRS for i in pair]
def progress(layer):
status.update(last_completed_layer=layer,elapsed_seconds=time.time()-started)
write_json(root/'functional-status.json',status)
print(json.dumps({'phase':status['phase'],'layer':layer,'elapsed':time.time()-started}),flush=True)
h,saved=lm.forward(ids,capture=capture,progress=progress)
np.savez(root/'baseline-activations.npz',**saved,**{f'qraw_{k}':v[0] for k,v in lm.qk_inputs.items()},**{f'kraw_{k}':v[1] for k,v in lm.qk_inputs.items()},final=h)
# Baseline heldout NLL uses no selection information.
nll=lm.nll(h[held],ids[held]);np.save(root/'baseline-heldout-token-nll.npy',nll)
baseline_nll=nll.mean(axis=1)
lm.capture_qk=False
lm.deadline=min(overall_deadline,time.monotonic()+1800)
status['phase']='calibration';write_json(root/'functional-status.json',status)
candidates=[]
for kind,pairs in [('attn',ATTN_PAIRS),('mlp',MLP_PAIRS)]:
for i,j in pairs:
pairid=f'{kind}_{i}_{j}';pairrows=[]
for source in (i,j):
x=saved[f'{kind}_{source}'][cal]
if kind=='attn':
a,pa=lm.full_attention(x,i);b,pb=lm.full_attention(x,j)
pm={}
for bi,si in enumerate(cal):
for hi in range(pa.shape[1]):
hpm=probability_metrics(pa[bi:bi+1,hi:hi+1],pb[bi:bi+1,hi:hi+1])
headrows.append({'candidate':pairid,'source':source,'sample_id':samples[si]['id'],'head':hi,**hpm})
save_csv(root/'attention-head-metrics.csv',headrows)
else:
a,b=lm.mlp(x,i),lm.mlp(x,j);pm={}
rec=error_rows(a,b,[samples[t] for t in cal],{'candidate':pairid,'kind':kind,'source':source,**pm})
if kind=='attn':
for bi,r in enumerate(rec):r.update(probability_metrics(pa[bi:bi+1],pb[bi:bi+1]))
rows.extend(rec);pairrows.extend(rec)
save_csv(root/'calibration-metrics.csv',rows)
passing=all(r['relative_l2']<=.05 and r['reverse_relative_l2']<=.05 and r['token_relative_p95']<=.1 and r['reverse_token_relative_p95']<=.1 for r in pairrows)
if kind=='attn':passing=passing and all(r['js_mean']<=.01 and r['top5_overlap']>=.8 for r in pairrows)
prefix=f'model.language_model.layers.{i}.'+('self_attn.' if kind=='attn' else 'mlp.')
size=sum(t.bytes for name,t in lm.tensors.items() if name.startswith(prefix))
candidates.append({'id':pairid,'kind':kind,'i':i,'j':j,'passed':passing,'maximum_symmetric_relative_error':max(max(r['relative_l2'],r['reverse_relative_l2']) for r in pairrows),'donor_component_bytes':size})
for layer in NORM_LAYERS:
pairid=f'qk_norm_{layer}';pfx=f'layers.{layer}.self_attn.'
qw,kw=lm.small(pfx+'q_norm.weight'),lm.small(pfx+'k_norm.weight')
pairrows=[]
for source,x in zip(('q','k'),lm.qk_inputs[layer]):
a,b=norm(x[cal],qw,lm.eps),norm(x[cal],kw,lm.eps)
rec=error_rows(a,b,[samples[t] for t in cal],{'candidate':pairid,'kind':'qk_norm','source':source})
pairrows.extend(rec);rows.extend(rec)
passing=all(r['relative_l2']<=.05 and r['reverse_relative_l2']<=.05 and r['token_relative_p95']<=.1 and r['reverse_token_relative_p95']<=.1 for r in pairrows)
candidates.append({'id':pairid,'kind':'qk_norm','i':layer,'j':layer,'passed':passing,'maximum_symmetric_relative_error':max(max(r['relative_l2'],r['reverse_relative_l2']) for r in pairrows),'donor_component_bytes':lm.tensors['model.language_model.'+pfx+'q_norm.weight'].bytes})
save_csv(root/'calibration-metrics.csv',rows)
write_json(root/'candidate-summary.json',candidates)
passed=[c for c in candidates if c['passed']]
# Prefer larger potential savings, then smaller worst calibration error; never consult heldout.
selected=sorted(passed,key=lambda c:(-c['donor_component_bytes'],c['maximum_symmetric_relative_error'],c['id']))[:1]
write_json(root/'selection.json',{'selected':selected,'rule':'calibration pass; descending component bytes, ascending maximum symmetric error; maximum one candidate'})
if selected:
c=selected[0]
heldrows=[]
if c['kind']=='qk_norm':
pfx=f'layers.{c["i"]}.self_attn.'
qw,kw=lm.small(pfx+'q_norm.weight'),lm.small(pfx+'k_norm.weight')
for source,x in zip(('q','k'),lm.qk_inputs[c['i']]):
heldrows.extend(error_rows(norm(x[held],qw,lm.eps),norm(x[held],kw,lm.eps),[samples[t] for t in held],{'candidate':c['id'],'kind':'qk_norm','source':source}))
else:
for source in (c['i'],c['j']):
x=saved[f'{c["kind"]}_{source}'][held]
if c['kind']=='attn':
a,_=lm.full_attention(x,c['i']);b,_=lm.full_attention(x,c['j'])
else:a,b=lm.mlp(x,c['i']),lm.mlp(x,c['j'])
heldrows.extend(error_rows(a,b,[samples[t] for t in held],{'candidate':c['id'],'kind':c['kind'],'source':source}))
save_csv(root/'heldout-function-metrics.csv',heldrows)
held_function_pass=all(r['relative_l2']<=.05 and r['reverse_relative_l2']<=.05 and r['token_relative_p95']<=.1 and r['reverse_token_relative_p95']<=.1 for r in heldrows)
status['heldout_function_pass']=held_function_pass
lm.deadline=min(overall_deadline,time.monotonic()+1800)
status['phase']='intervention';write_json(root/'functional-status.json',status)
for reverse in (False,True):
replacement=None;lm.norm_replacement=None
if c['kind']=='qk_norm':
pfx=f'layers.{c["i"]}.self_attn.'
target,donor=('q_norm','k_norm') if not reverse else ('k_norm','q_norm')
lm.norm_replacement=(pfx+target,pfx+donor)
label=target+'_uses_'+donor
else:
target,donor=(c['i'],c['j']) if not reverse else (c['j'],c['i'])
replacement=(c['kind'],target,donor);label=f'{target}_uses_{donor}'
t=time.monotonic();modified,_=lm.forward(ids[held],replacement=replacement,progress=progress)
changed=lm.nll(modified,ids[held]);np.save(root/f'intervention-{label}-token-nll.npy',changed)
delta=changed.mean(axis=1)-baseline_nll
rec={'candidate':c['id'],'direction':label,'elapsed_seconds':time.monotonic()-t,
'nll_mean_delta':float(delta.mean()),'nll_sequence_p95_delta':float(np.quantile(delta,.95)),
'sequence_nll_deltas':delta.tolist(),'passed_quality':bool(delta.mean()<=.01 and np.quantile(delta,.95)<=.05)}
interventions.append(rec);write_json(root/'interventions.json',interventions)
# No fallback candidate if heldout fails.
status.update(state='completed',phase='done',selected_candidate=selected,
positive_bidirectional_quality=bool(interventions and status.get('heldout_function_pass') and all(x['passed_quality'] for x in interventions)),
elapsed_seconds=time.time()-started,peak_rss_kib=resource.getrusage(resource.RUSAGE_SELF).ru_maxrss,
claim_limit='Small text-only heldout pilot; no global interchangeability proof, no measured GPU speedup; component storage savings require explicit layout accounting.')
except Exception as exc:
status.update(state='stopped',error_type=type(exc).__name__,error=str(exc),elapsed_seconds=time.time()-started,
peak_rss_kib=resource.getrusage(resource.RUSAGE_SELF).ru_maxrss)
(root/'functional-error.txt').write_text(traceback.format_exc())
raise
finally:
write_json(root/'functional-status.json',status)
if __name__=='__main__':main()
+13
View File
@@ -0,0 +1,13 @@
{
"origin": "Experiment-authored plain text, no external corpus; fixed before functional metrics. Chinese/English balanced in each split. No chat template. First 32 tokenizer tokens retained.",
"samples": [
{"id":"cal-zh-1","split":"calibration","language":"zh","text":"清晨,研究员打开实验记录本,检查昨天保存的数据。他先确认文件完整,再比较两组观测结果。为了避免偶然误差,他为每一步留下时间和条件,并将尚未验证的推测单独记录。"},
{"id":"cal-zh-2","split":"calibration","language":"zh","text":"一辆列车从山间驶向海边,经过村庄、桥梁和开阔的田野。旅客望着窗外变化的景色,计划抵达后先步行到旅馆,再去市场购买晚餐需要的食材。天气预报说傍晚可能下雨。"},
{"id":"cal-en-1","split":"calibration","language":"en","text":"A small library opens every morning beside the river. The librarian checks returned books, repairs damaged covers, and places new arrivals on a wooden table. Visitors often stop to read before continuing their journey across the old bridge."},
{"id":"cal-en-2","split":"calibration","language":"en","text":"To compare two measurements, first record their units and the conditions under which they were collected. Compute the difference, inspect unusual observations, and repeat the procedure with independent examples. A short report should separate observations from explanations and unresolved questions."},
{"id":"test-zh-1","split":"heldout","language":"zh","text":"午后,工程师检查温室中的传感器。温度记录出现一个异常值,她没有立刻删除,而是查看设备状态并补做一次测量。确认原因以后,她调整了采样计划,在报告中说明这次变化。"},
{"id":"test-zh-2","split":"heldout","language":"zh","text":"周末,家人沿着湖边的小路散步。他们带了水和简单的午餐,打算在树林旁休息。途中遇到一处正在维修的桥,于是参考路标选择另一条道路,终于在日落前回到车站。"},
{"id":"test-en-1","split":"heldout","language":"en","text":"The gardener planted several rows of beans behind the house. Each evening she checked the soil and wrote down how much water the plants received. After a week of warm weather, the first green shoots appeared near the fence beside the apple tree."},
{"id":"test-en-2","split":"heldout","language":"en","text":"When a program reads a table, it should check column names and handle missing entries explicitly. The developer tested a small example, compared the output with a hand calculation, and saved the input alongside the result so another person could reproduce the check."}
]
}
+62
View File
@@ -0,0 +1,62 @@
"""Measure an explicit two-component alias storage layout, not inference speed.
Payloads remain on the server under weights/ and must not be committed or cleaned
without the user's explicit deletion request. Manifest/byte/hash evidence may be archived.
"""
import argparse
import hashlib
import json
from pathlib import Path
import numpy as np
from pilot.tensors import load_tensors,decode
from .run import write_json
def payload(t,path):
digest=hashlib.sha256()
with path.open('wb') as f:
for start in range(0,t.raw.size,262144):
b=t.raw[start:start+262144].tobytes();f.write(b);digest.update(b)
t.evict()
return {'file':path.name,'dtype':t.dtype,'shape':list(t.shape),'bytes':path.stat().st_size,'sha256':digest.hexdigest()}
def main():
p=argparse.ArgumentParser();p.add_argument('--root',type=Path,required=True);a=p.parse_args();root=a.root
selected=json.loads((root/'selection.json').read_text())['selected']
if not selected:
write_json(root/'storage-evidence.json',{'measured':False,'reason':'No calibration candidate selected'});return
c=selected[0]
if c['kind']!='qk_norm':
write_json(root/'storage-evidence.json',{'measured':False,'reason':'Only two-gain alias layout implemented in this bounded pilot; larger-module savings remain theoretical'});return
tensors=load_tensors(root/'weights')
prefix=f'model.language_model.layers.{c["i"]}.self_attn.'
names=[prefix+'q_norm.weight',prefix+'k_norm.weight']
# Original pair plus two alternative layouts; each is counted independently.
path=root/'component-layout'/'weights';path.mkdir(parents=True,exist_ok=True)
original={'format':'two_component_raw_v1','tensors':{}}
for short,name in zip(('q','k'),names):original['tensors'][name]=payload(tensors[name],path/f'{short}.bin')
original_bytes=json.dumps(original,separators=(',',':'),sort_keys=True).encode()
(path/'original-layout.json').write_bytes(original_bytes)
evidence=[]
for target,donor in ((names[0],names[1]),(names[1],names[0])):
entry=original['tensors'][donor]
layout={'format':'two_component_raw_v1','tensors':{donor:entry},'aliases':{target:donor}}
encoded=json.dumps(layout,separators=(',',':'),sort_keys=True).encode()
file=path/('share-'+('k' if donor==names[1] else 'q')+'.json');file.write_bytes(encoded)
t=tensors[donor]
raw=np.fromfile(path/entry['file'],dtype=t.raw.dtype)
reconstructed=decode(raw,t.dtype).reshape(t.shape)
np.testing.assert_array_equal(reconstructed,t.read().reshape(t.shape));t.evict()
before=sum(e['bytes'] for e in original['tensors'].values())+len(original_bytes)
after=entry['bytes']+len(encoded)
evidence.append({'target':target,'donor':donor,'original_payload_bytes':sum(e['bytes'] for e in original['tensors'].values()),
'shared_payload_bytes':entry['bytes'],'original_descriptor_bytes':len(original_bytes),
'shared_descriptor_bytes':len(encoded),'original_total_bytes':before,'shared_total_bytes':after,
'net_saved_bytes':before-after,'shared_descriptor_sha256':hashlib.sha256(encoded).hexdigest(),
'decoded_donor_exactly_matches_intervention_gain':True})
write_json(root/'storage-evidence.json',{'measured':True,'candidate':c['id'],'layouts':evidence,
'claim_limit':'Measured standalone component layout including descriptors, not an integrated compressed checkpoint, latency gain or full-model RSS reduction. Two sharing directions are alternatives and must not be added.'})
if __name__=='__main__':main()
+95
View File
@@ -0,0 +1,95 @@
"""One bounded CPU run; waits for verified download and binary dependencies.
No installation, retry search, cleanup, training, GPU selection or Git mutation.
"""
import argparse
import hashlib
import importlib.util
import json
from pathlib import Path
import subprocess
import sys
import time
import traceback
from .run import write_json
def main():
p=argparse.ArgumentParser();p.add_argument('--root',type=Path,required=True);p.add_argument('--download-wait-seconds',type=int,default=14400)
a=p.parse_args();root=a.root.resolve();start=time.monotonic()
state={'state':'waiting','phase':'dependencies','weight_retention':'until_explicit_user_delete_request'}
def status(**kwargs):
state.update(kwargs,supervisor_elapsed_seconds=time.monotonic()-start)
write_json(root/'supervisor-status.json',state)
def execute(module,args,timeout,log):
ledger_path=root/'compute-budget-ledger.json'
ledger=json.loads(ledger_path.read_text()) if ledger_path.exists() else {'spent_seconds':0.,'calls':[]}
timeout=min(timeout,int(7200-ledger['spent_seconds']))
if timeout<1:raise TimeoutError('Cumulative two-hour compute budget exhausted')
t=time.monotonic()
try:
with (root/log).open('w') as out:
r=subprocess.run([sys.executable,'-B','-m',module,*args],stdout=out,stderr=subprocess.STDOUT,timeout=timeout)
finally:
elapsed=time.monotonic()-t
ledger['spent_seconds']+=elapsed
ledger['calls'].append({'module':module,'seconds':elapsed})
write_json(ledger_path,ledger)
if r.returncode:raise RuntimeError(f'{module} failed: exit {r.returncode}, see {log}')
status()
try:
while True:
ready=False
if all(importlib.util.find_spec(name) for name in ('torch','tokenizers')):
try:
r=subprocess.run([sys.executable,'-B','-c','import torch, tokenizers; torch.empty(1)'],capture_output=True,timeout=30)
ready=r.returncode==0
except subprocess.TimeoutExpired:pass
if ready:break
if time.monotonic()-start>3600:raise TimeoutError('Binary reference dependencies not ready within one hour')
status()
time.sleep(15)
compute_start=time.monotonic()
status(state='running',phase='reference_check')
provenance={str(p):hashlib.sha256(p.read_bytes()).hexdigest() for p in sorted(Path('functional').glob('*')) if p.is_file()}
write_json(root/'functional-code-hashes.json',provenance)
execute('functional.reference_check',['--source',str(root/'reference.py'),'--output',str(root/'reference-check.json')],300,'reference-check.log')
reference_seconds=time.monotonic()-compute_start
status(state='waiting',phase='verified_weights',reference_seconds=reference_seconds)
while True:
path=root/'manifest.json'
try:
manifest=json.loads(path.read_text())
weights=[f for f in manifest.get('files',[]) if f['name'].endswith('.safetensors')]
ready=(manifest.get('completed_unix') is not None and weights and
all((root/'weights'/f['name']).is_file() and (root/'weights'/f['name']).stat().st_size==f['bytes'] and f['sha256']==f['upstream_sha256'] for f in weights))
except (OSError,ValueError,KeyError):ready=False
if ready:break
if time.monotonic()-start>a.download_wait_seconds:raise TimeoutError('Verified weight download not ready before wait deadline')
status()
time.sleep(15)
status(state='running',phase='real_weight_preflight')
compute_start=time.monotonic()
execute('functional.preflight',['--root',str(root)],1200,'preflight.log')
preflight_seconds=time.monotonic()-compute_start
remaining=int(7200-json.loads((root/'compute-budget-ledger.json').read_text())['spent_seconds']-60)
if remaining<60:raise TimeoutError('No remaining bounded functional computation budget')
status(phase='functional_analysis',preflight_seconds=preflight_seconds,remaining_seconds=remaining)
execute('functional.run',['--root',str(root),'--seconds',str(remaining)],remaining+15,'functional-run.log')
status(phase='storage_accounting')
execute('functional.storage',['--root',str(root)],120,'storage-accounting.log')
status(phase='report')
execute('functional.report',['--root',str(root)],120,'functional-report.log')
status(state='completed',phase='done')
except Exception as exc:
status(state='stopped',error_type=type(exc).__name__,error=str(exc))
(root/'supervisor-error.txt').write_text(traceback.format_exc())
# Retain all weights/results. No automatic retries or candidate expansion.
if not (root/'functional-status.json').exists():
write_json(root/'functional-status.json',{'state':'stopped','phase':state['phase'],'error_type':type(exc).__name__,'error':str(exc)})
try:execute('functional.report',['--root',str(root)],120,'functional-report.log')
except Exception:pass
raise
if __name__=='__main__':main()
+1
View File
@@ -0,0 +1 @@
"""Bounded, inference-free weight similarity pilot."""
+396
View File
@@ -0,0 +1,396 @@
"""Bounded CPU-only screening, exact verification and conservative sharing groups."""
import argparse
import collections
import csv
import hashlib
import itertools
import json
import math
import pathlib
import platform
import resource
import time
import numpy as np
from .tensors import decode, finish, load_tensors, metrics
THRESHOLDS = [.01, .02, .05, .10, .20]
def write_json(path, obj):
path.write_text(json.dumps(obj, indent=2, allow_nan=False))
def write_csv(path, rows):
if not rows:
path.write_text('')
return
with path.open('w', newline='') as f:
writer = csv.DictWriter(f, fieldnames=list(rows[0]))
writer.writeheader()
writer.writerows(rows)
def stable_seed(key, seed):
return int.from_bytes(hashlib.sha256((str(seed) + str(key)).encode()).digest()[:8], 'little')
def read_object(obj, tensors):
t = tensors[obj['tensor']]
if obj['kind'] == 'tensor':
return t.read()
slices = tuple(slice(s, s + b) for s, b in zip(obj['start'], obj['shape']))
return decode(t.raw.reshape(t.shape)[slices], t.dtype).ravel()
def sample_object(obj, tensors, sample_size=2048):
t = tensors[obj['tensor']]
n = math.prod(obj['shape'])
indices = np.linspace(0, n - 1, min(n, sample_size), dtype=np.int64)
if obj['kind'] == 'tensor':
return t.read(indices)
coords = np.unravel_index(indices, obj['shape'])
global_coords = tuple(c + s for c, s in zip(coords, obj['start']))
flat = np.ravel_multi_index(global_coords, t.shape)
return t.read(flat)
def exact_pair(a, b, tensors, deadline):
if a['kind'] != 'tensor':
result = metrics(read_object(a, tensors), read_object(b, tensors))
tensors[a['tensor']].evict()
tensors[b['tensor']].evict()
return result
ta, tb = tensors[a['tensor']], tensors[b['tensor']]
xx = yy = xy = delta = sx = sy = 0.
for start in range(0, ta.numel, 262144):
if time.monotonic() > deadline:
raise TimeoutError('Analysis deadline reached')
sl = slice(start, min(start + 262144, ta.numel))
x, y = ta.read(sl).astype(np.float64), tb.read(sl).astype(np.float64)
xx += float(x @ x)
yy += float(y @ y)
xy += float(x @ y)
d = x - y
delta += float(d @ d)
sx += float(x.sum())
sy += float(y.sum())
ta.evict()
tb.evict()
ta.evict()
tb.evict()
return finish(xx, yy, xy, delta, sx, sy, ta.numel)
def sharing_groups(objects, pairs, threshold):
"""Star groups; every member must have an exact verified edge to its representative."""
adjacent = collections.defaultdict(dict)
for pair in pairs:
a, b = pair['a'], pair['b']
if pair['error_member_to_rep'] <= threshold:
adjacent[b][a] = pair['error_member_to_rep']
if pair['error_rep_to_member'] <= threshold:
adjacent[a][b] = pair['error_rep_to_member']
unassigned = set(range(len(objects)))
order = sorted(unassigned, key=lambda i: (-len(adjacent[i]), objects[i]['id']))
groups = []
for rep in order:
if rep not in unassigned:
continue
members = sorted(set(adjacent[rep]) & unassigned)
members.append(rep)
unassigned.difference_update(members)
groups.append({'representative': rep, 'members': members,
'max_error': max([adjacent[rep][i] for i in members if i != rep] or [0.])})
return groups
def nearest_candidates(objects, features, cap, neighbors=10):
buckets = collections.defaultdict(list)
for i, obj in enumerate(objects):
buckets[(obj['role'], tuple(obj['shape']), obj['scale'], obj['dtype'])].append(i)
candidates = []
eligible_pairs = 0
for key, ids in sorted(buckets.items(), key=lambda item: str(item[0])):
eligible_pairs += len(ids) * (len(ids) - 1) // 2
if len(ids) < 2:
continue
x = np.stack([features[i] for i in ids]).astype(np.float64)
norm = np.linalg.norm(x, axis=1)
u = x / np.maximum(norm[:, None], 1e-30)
# Cap each bucket's reservoir to bound the screening matrix.
cos = u @ u.T
sq = np.maximum(0., norm[:, None] ** 2 + norm[None, :] ** 2 - 2 * (x @ x.T))
err = np.sqrt(sq / np.maximum(norm[:, None] ** 2, 1e-30))
np.fill_diagonal(cos, -np.inf)
np.fill_diagonal(err, np.inf)
seen = set()
for row, i in enumerate(ids):
near = set(np.argsort(err[row])[:neighbors]) | set(np.argsort(-cos[row])[:neighbors])
for col in near:
j = ids[col]
edge = (min(i, j), max(i, j))
if i == j or edge in seen:
continue
seen.add(edge)
candidates.append({'a': edge[0], 'b': edge[1], 'screen_error': float(min(err[row, col], err[col, row])),
'screen_cosine': float(cos[row, col])})
# Low-error candidates first, then guarantee role/scale diversity round-robin.
queues = collections.defaultdict(list)
for c in sorted(candidates, key=lambda c: (c['screen_error'], c['a'], c['b'])):
o = objects[c['a']]
queues[(o['role'], tuple(o['shape']), o['scale'])].append(c)
queues = [collections.deque(q) for _, q in sorted(queues.items(), key=lambda x: str(x[0]))]
chosen = []
while queues and len(chosen) < cap:
for q in queues:
if q and len(chosen) < cap:
chosen.append(q.popleft())
queues = [q for q in queues if q]
return chosen, {'eligible_pairs_in_screened_objects': eligible_pairs,
'candidate_pairs_before_cap': len(candidates), 'selected_pairs': len(chosen)}
def object_record(t, kind='tensor', shape=None, start=None, scale='whole'):
shape = tuple(shape or t.shape)
return {'id': t.name if kind == 'tensor' else f'{t.name}|{scale}|{start}',
'tensor': t.name, 'role': t.role, 'layer': t.layer, 'kind': kind, 'dtype': t.dtype,
'shape': shape, 'start': start, 'scale': scale,
'bytes': math.prod(shape) * (t.bytes // t.numel)}
def build_blocks(tensors, seed, deadline, reservoir_limit=256):
reservoirs, counts, rngs = {}, collections.Counter(), {}
totals = collections.defaultdict(lambda: {'blocks': 0, 'bytes': 0, 'edge_bytes': 0})
for t in tensors.values():
if len(t.shape) == 2:
specs = [(f'2d-{s}', (s, s)) for s in (64, 128) if min(t.shape) >= s]
elif len(t.shape) == 3:
specs = [('3d-axis0-slice', (1, min(64, t.shape[1]), min(64, t.shape[2]))),
('3d-volume', (min(4, t.shape[0]), min(32, t.shape[1]), min(32, t.shape[2])))]
else:
continue
for scale, shape in specs:
nblocks = math.prod(d // b for d, b in zip(t.shape, shape))
totals[scale]['blocks'] += nblocks
block_bytes = math.prod(shape) * (t.bytes // t.numel)
totals[scale]['bytes'] += nblocks * block_bytes
totals[scale]['edge_bytes'] += t.bytes - nblocks * block_bytes
for grid in itertools.product(*(range(d // b) for d, b in zip(t.shape, shape))):
if time.monotonic() > deadline:
raise TimeoutError('Deadline during block enumeration')
start = tuple(i * b for i, b in zip(grid, shape))
obj = object_record(t, 'block', shape, start, scale)
key = (t.role, shape, scale)
if key not in reservoirs:
reservoirs[key], rngs[key] = [], np.random.default_rng(stable_seed(key, seed))
counts[key] += 1
reservoir = reservoirs[key]
if len(reservoir) < reservoir_limit:
reservoir.append(obj)
else:
j = int(rngs[key].integers(counts[key]))
if j < reservoir_limit:
reservoir[j] = obj
objects = [obj for key in sorted(reservoirs, key=str) for obj in reservoirs[key]]
coverage = [{'role': key[0], 'shape': str(key[1]), 'scale': key[2],
'total_blocks': counts[key], 'sampled_blocks': len(reservoirs[key])}
for key in sorted(reservoirs, key=str)]
return objects, coverage, dict(totals)
def structural_probe(a, b, tensors):
"""Bounded mode unfoldings of aligned subvolumes; exploratory, never a grouping criterion."""
def small(obj):
t = tensors[obj['tensor']]
shape = obj['shape']
starts = obj['start'] or [0] * len(shape)
slices = tuple(slice(s, s + min(d, 32)) for s, d in zip(starts, shape))
return decode(t.raw.reshape(t.shape)[slices], t.dtype)
x, y = small(a), small(b)
result = []
if x.ndim < 2:
return result
for axis in range(x.ndim):
mx = np.moveaxis(x, axis, 0).reshape(x.shape[axis], -1)
my = np.moveaxis(y, axis, 0).reshape(y.shape[axis], -1)
if min(mx.shape) < 2:
result.append({'axis': axis, 'probe_shape': list(x.shape), 'trivial_mode_skipped': True})
continue
ux, sx, _ = np.linalg.svd(mx, full_matrices=False)
uy, sy, _ = np.linalg.svd(my, full_matrices=False)
k = min(4, max(1, min(len(sx), len(sy)) // 2))
result.append({'axis': axis, 'probe_shape': list(x.shape),
'k': k,
'normalized_spectrum_distance': float(np.linalg.norm(sx / max(np.linalg.norm(sx), 1e-30) - sy / max(np.linalg.norm(sy), 1e-30))),
'left_subspace_overlap': float(np.linalg.norm(ux[:, :k].T @ uy[:, :k]) ** 2 / k)})
return result
def analyze_mode(mode, objects, tensors, root, deadline, cap, seed, baseline_cap=100):
print(f'{mode}: sampling {len(objects)} objects', flush=True)
features = []
for obj in objects:
features.append(sample_object(obj, tensors))
tensors[obj['tensor']].evict()
candidates, coverage = nearest_candidates(objects, features, cap)
write_json(root / f'{mode}_objects.json', objects)
pairs, nulls, structures = [], [], []
rng = np.random.default_rng(seed)
random_pairs = []
comparable = collections.defaultdict(list)
for j, obj in enumerate(objects):
comparable[(obj['role'], tuple(obj['shape']), obj['scale'], obj['dtype'])].append(j)
for key, ids in sorted(comparable.items(), key=lambda item: str(item[0])):
if len(ids) < 2:
continue
for _ in range(3):
a, b = rng.choice(ids, size=2, replace=False).tolist()
random_pairs.append({'a': a, 'b': b, 'role': key[0], 'sampled': True,
**metrics(features[a], features[b])})
write_json(root / f'{mode}_random_pairs.json', random_pairs)
timeout = False
structural_counts = collections.Counter()
for i, candidate in enumerate(candidates):
if time.monotonic() > deadline:
timeout = True
break
a, b = candidate['a'], candidate['b']
try:
measured = exact_pair(objects[a], objects[b], tensors, deadline)
except TimeoutError:
timeout = True
break
row = {**candidate, **measured,
'a_id': objects[a]['id'], 'b_id': objects[b]['id'],
'role': objects[a]['role'], 'scale_name': objects[a]['scale'],
'layer_distance': abs(objects[a]['layer'] - objects[b]['layer'])}
row['screen_vs_exact_min_error_absolute_difference'] = abs(candidate['screen_error'] - min(measured['error_member_to_rep'], measured['error_rep_to_member']))
# Tail diagnostics are sampled, explicitly labelled, not claimed exact.
fx, fy = features[a].astype(np.float64), features[b].astype(np.float64)
row['sampled_error_q99_over_member_rms'] = float(np.quantile(np.abs(fx - fy), .99) / max(np.sqrt(np.mean(fx * fx)), 1e-30))
pairs.append(row)
if i < baseline_cap:
for method, y in [('shuffled', rng.permutation(fy)), ('gaussian_norm_matched', rng.normal(size=fy.size))]:
if method == 'gaussian_norm_matched':
y *= np.linalg.norm(fy) / max(np.linalg.norm(y), 1e-30)
nulls.append({'a': a, 'b': b, 'method': method, 'sampled': True, **metrics(fx, y)})
ndim = len(objects[a]['shape'])
if ndim >= 2 and structural_counts[ndim] < 20:
structures.append({'a': a, 'b': b, 'modes': structural_probe(objects[a], objects[b], tensors)})
tensors[objects[a]['tensor']].evict()
tensors[objects[b]['tensor']].evict()
structural_counts[ndim] += 1
if (i + 1) % 100 == 0:
write_csv(root / f'{mode}_pairs.csv', pairs)
print(f'{mode}: verified {i + 1}/{len(candidates)}', flush=True)
write_csv(root / f'{mode}_pairs.csv', pairs)
write_json(root / f'{mode}_nulls.json', nulls)
write_json(root / f'{mode}_structure.json', structures)
coverage.update({'objects_screened': len(objects), 'pairs_verified': len(pairs), 'timed_out': timeout,
'screen_sample_elements': 2048, 'candidate_neighbors_per_metric': 10})
write_json(root / f'{mode}_coverage.json', coverage)
return pairs
def summarize(mode, objects, pairs, model_bytes, root, partition_totals=None):
rows = []
scales = sorted({o['scale'] for o in objects})
for scale in scales:
ids = [i for i, o in enumerate(objects) if o['scale'] == scale]
selected = set(ids)
for threshold in THRESHOLDS:
groups = sharing_groups(objects, [p for p in pairs if p['a'] in selected], threshold)
groups = [g for g in groups if g['representative'] in selected]
original = sum(objects[i]['bytes'] for i in ids)
rep_bytes = sum(objects[g['representative']]['bytes'] for g in groups)
# Concrete theoretical layout: 16 bytes per object mapping, plus 64 bytes per shared group.
# Identity case falls back to original checkpoint at zero additional cost.
mapping_bytes = 16 * len(ids) + 64 * sum(len(g['members']) > 1 for g in groups)
proposed = rep_bytes + mapping_bytes
saved = max(0, original - proposed)
partition_count = partition_totals[scale]['blocks'] if partition_totals else len(ids)
rows.append({'mode': mode, 'scale': scale, 'threshold': threshold,
'objects': len(ids), 'groups': len(groups),
'total_partition_objects': partition_count,
'unexamined_partition_objects': partition_count - len(ids),
'conservative_partition_groups': partition_count - len(ids) + len(groups),
'shared_groups': sum(len(g['members']) > 1 for g in groups),
'max_group_size': max([len(g['members']) for g in groups] or [0]),
'verified_shared_bytes': sum(sum(objects[i]['bytes'] for i in g['members']) for g in groups if len(g['members']) > 1),
'examined_object_bytes': original, 'representative_bytes': rep_bytes,
'mapping_bytes_if_encoded': mapping_bytes, 'net_saved_bytes': saved,
'net_saved_model_fraction': saved / model_bytes,
'max_replacement_error': max([g['max_error'] for g in groups] or [0.])})
write_json(root / f'groups_{mode}_{scale}_{threshold:.2f}.json', groups)
return rows
def main():
p = argparse.ArgumentParser()
p.add_argument('--root', type=pathlib.Path, required=True)
p.add_argument('--seconds', type=int, default=7200)
p.add_argument('--seed', type=int, default=20261109)
args = p.parse_args()
started = time.monotonic()
deadline = started + args.seconds
root = args.root
output = root / 'analysis'
output.mkdir(exist_ok=True)
tensors = load_tensors(root / 'weights')
inventory = []
for t in tensors.values():
if time.monotonic() > deadline:
raise TimeoutError('Deadline during inventory')
n = 0
total = squared = 0.
max_abs = 0.
for x in t.chunked():
if not np.isfinite(x).all():
raise ValueError(f'Nonfinite weights: {t.name}')
z = x.astype(np.float64)
n += z.size
total += float(z.sum())
squared += float(z @ z)
max_abs = max(max_abs, float(np.abs(z).max()))
t.evict()
inventory.append({'name': t.name, 'shape': str(t.shape), 'ndim': len(t.shape), 'dtype': t.dtype,
'numel': t.numel, 'bytes': t.bytes, 'role': t.role, 'layer': t.layer,
'mean': total / n, 'rms': math.sqrt(squared / n), 'max_abs': max_abs})
t.evict()
write_csv(output / 'inventory.csv', inventory)
print(f'Inventory: {len(tensors)} tensors, {sum(t.bytes for t in tensors.values())} bytes', flush=True)
whole = [object_record(t) for t in tensors.values()]
# Stage budgets prevent the whole-tensor stage from consuming block-analysis time.
whole_deadline = min(deadline, started + args.seconds * .50)
whole_pairs = analyze_mode('tensor', whole, tensors, output, whole_deadline, 2000, args.seed)
blocks, block_coverage, block_totals = build_blocks(tensors, args.seed, deadline)
write_csv(output / 'block_sampling.csv', block_coverage)
write_json(output / 'block_partitions.json', block_totals)
block_pairs = analyze_mode('block', blocks, tensors, output, deadline, 20000, args.seed)
model_bytes = sum(t.bytes for t in tensors.values())
rows = summarize('tensor', whole, whole_pairs, model_bytes, output)
rows += summarize('block', blocks, block_pairs, model_bytes, output, block_totals)
write_csv(output / 'compression_curve.csv', rows)
summary = {'model': json.loads((root / 'manifest.json').read_text())['repo'],
'revision': json.loads((root / 'manifest.json').read_text())['revision'],
'tensor_count': len(tensors), 'parameter_count': sum(t.numel for t in tensors.values()),
'model_weight_bytes': model_bytes, 'ndim_counts': dict(collections.Counter(len(t.shape) for t in tensors.values())),
'elapsed_seconds': time.monotonic() - started, 'analysis_budget_seconds': args.seconds,
'peak_rss_kib': resource.getrusage(resource.RUSAGE_SELF).ru_maxrss,
'python': platform.python_version(), 'numpy': np.__version__, 'seed': args.seed,
'limitations': ['No inference or functional equivalence measured', 'No exhaustive pair search',
'Block search uses per-role/shape/scale reservoirs of 256',
'Different block scales overlap; savings must not be added',
'Block savings count only verified disjoint sampled blocks, no extrapolation',
'Spectral probes use aligned bounded subvolumes, not whole tensors',
'Shared encoding is theoretical, not a runnable compressed checkpoint',
'No cross-role grouping, neuron permutation alignment, or training'],
'completed': True}
write_json(output / 'summary.json', summary)
print(json.dumps(summary), flush=True)
if __name__ == '__main__':
main()
+75
View File
@@ -0,0 +1,75 @@
"""Audit saved numerical data and freeze provenance before archive/cleanup."""
import argparse
import csv
import hashlib
import json
import math
import os
import pathlib
import platform
import re
import shutil
import time
def main():
p = argparse.ArgumentParser()
p.add_argument('--root', required=True, type=pathlib.Path)
p.add_argument('--analysis-code-commit', help='Commit actually used for the weight analysis; omit if unknown')
args = p.parse_args()
root = args.root.resolve()
if not __debug__:
raise RuntimeError('Validation requires Python assertions enabled')
checked = {}
for mode in ('tensor', 'block'):
with (root / 'analysis' / f'{mode}_pairs.csv').open() as f:
rows = list(csv.DictReader(f))
coverage = json.loads((root / 'analysis' / f'{mode}_coverage.json').read_text())
if len(rows) != coverage['pairs_verified']:
raise ValueError('Coverage count mismatch')
for row in rows:
for key in ('cosine', 'pearson'):
if row[key] and (not math.isfinite(float(row[key])) or abs(float(row[key])) > 1 + 1e-7):
raise ValueError(f'Invalid correlation: {mode}/{key}')
for key in ('error_member_to_rep', 'error_rep_to_member', 'symmetric_error', 'scaled_error',
'sampled_error_q99_over_member_rms', 'screen_vs_exact_min_error_absolute_difference'):
value = float(row[key])
if not math.isfinite(value) or value < 0:
raise ValueError(f'Invalid error: {mode}/{key}')
if not math.isfinite(float(row['scale'])):
raise ValueError('Invalid scaling diagnostic')
checked[mode] = {'rows': len(rows), 'finite_metrics': True, 'coverage_matches': True}
test_log = (root / 'tests.log').read_text()
match = re.search(r'Ran (\d+) tests in ([\d.]+)s', test_log)
if not match or '\nOK\n' not in test_log:
raise ValueError('Final tests did not pass')
qc = {'passed': True, 'checked_unix': time.time(), 'numerical_tables': checked,
'final_tests': int(match[1]), 'final_test_seconds': float(match[2]),
'earlier_parallel_test_timeout_seconds': 300 if (root / 'tests-during-io-timeout.txt').exists() else None}
(root / 'numerical-qc.json').write_text(json.dumps(qc, indent=2))
for src, dest in [('tests.log', 'test-results.txt'), ('analysis.log', 'analysis-run.txt'),
('plot.log', 'plot-run.txt'), ('download-ranged.log', 'download-progress.txt'),
('supervisor.log', 'supervisor-run.txt')]:
shutil.copy2(root / src, root / dest)
mem = {}
for line in pathlib.Path('/proc/meminfo').read_text().splitlines():
key, value = line.split(':', 1)
if key in ('MemTotal', 'MemAvailable', 'SwapTotal', 'SwapFree'):
mem[key + '_KiB'] = int(value.split()[0])
environment = {'recorded_unix': time.time(), 'python': platform.python_version(),
'platform': platform.platform(), 'machine': platform.machine(),
'logical_cpus': os.cpu_count(), 'memory_snapshot': mem,
'blas_threads': 1, 'assertions_enabled': __debug__,
'analysis_io_priority': 'best-effort 7, applied during block verification',
'analysis_code_commit': args.analysis_code_commit,
'note': 'Shared host; see resource-observation.txt. Memory snapshot is post-analysis.'}
(root / 'execution-environment.json').write_text(json.dumps(environment, indent=2))
project = pathlib.Path(__file__).resolve().parents[1]
files = sorted((project / 'pilot').glob('*.py')) + [project / 'requirements.txt', project / 'tests/test_pilot.py']
hashes = [f'{hashlib.sha256(file.read_bytes()).hexdigest()} {file.relative_to(project)}' for file in files]
(root / 'code_checksums.txt').write_text('\n'.join(hashes) + '\n')
print(json.dumps(qc))
if __name__ == '__main__':
main()
+140
View File
@@ -0,0 +1,140 @@
"""Download pinned official safetensors, no framework or shared cache."""
import argparse
import concurrent.futures
import hashlib
import json
import pathlib
import os
import shutil
import time
import threading
import urllib.parse
import urllib.request
def fetch_json(url):
with urllib.request.urlopen(url, timeout=90) as response:
return json.load(response)
def ranged_download(url, part, total, workers=8):
"""Resume a sequential prefix; journal completed byte ranges before final hashing."""
journal = part.with_suffix(part.suffix + '.ranges.json')
if journal.exists():
state = json.loads(journal.read_text())
if state['total'] != total:
raise RuntimeError('Range journal size mismatch')
else:
state = {'total': total, 'prefix': part.stat().st_size if part.exists() else 0, 'done': []}
journal.write_text(json.dumps(state))
lock = threading.Lock()
fd = os.open(part, os.O_CREAT | os.O_RDWR, 0o600)
chunk_size = 64 * 1024 * 1024
ranges = [(start, min(start + chunk_size, total) - 1)
for start in range(state['prefix'], total, chunk_size)]
def one(bounds):
start, end = bounds
if start in state['done']:
return
for attempt in range(3):
try:
req = urllib.request.Request(url, headers={'Range': f'bytes={start}-{end}'})
with urllib.request.urlopen(req, timeout=180) as response:
expected = f'bytes {start}-{end}/{total}'
if response.status != 206 or response.headers.get('Content-Range') != expected:
raise RuntimeError('Server did not honor exact requested range')
offset = start
while chunk := response.read(min(4 * 1024 * 1024, end - offset + 1)):
written = os.pwrite(fd, chunk, offset)
if written != len(chunk):
raise RuntimeError('Short positional write')
offset += written
if offset > end:
break
if offset != end + 1:
raise RuntimeError('Incomplete range')
with lock:
os.fsync(fd)
state['done'].append(start)
tmp = journal.with_suffix('.tmp')
tmp.write_text(json.dumps(state))
tmp.replace(journal)
done = state['prefix'] + sum(b - a + 1 for a, b in ranges if a in state['done'])
print(f'Weight download {done}/{total} bytes ({100 * done / total:.1f}%)', flush=True)
return
except Exception:
if attempt == 2:
raise
time.sleep(2 ** attempt)
try:
with concurrent.futures.ThreadPoolExecutor(max_workers=workers) as pool:
list(pool.map(one, ranges))
finally:
os.close(fd)
journal.unlink()
def main():
p = argparse.ArgumentParser()
p.add_argument('--root', type=pathlib.Path, required=True)
p.add_argument('--repo', default='Qwen/Qwen3.5-2B')
args = p.parse_args()
root = args.root.resolve()
root.mkdir(parents=True, exist_ok=True)
weights = root / 'weights'
weights.mkdir(exist_ok=True)
marker = weights / '.pilot-owned'
marker.write_text(args.repo)
old_manifest = root / 'manifest.json'
pinned = json.loads(old_manifest.read_text())['revision'] if old_manifest.exists() else None
endpoint = f'https://huggingface.co/api/models/{args.repo}'
if pinned:
endpoint += f'/revision/{pinned}'
info = fetch_json(endpoint + '?blobs=true')
revision = info['sha']
files = [f for f in info['siblings'] if f['rfilename'].endswith('.safetensors')
or f['rfilename'] in ('config.json', 'model.safetensors.index.json', 'README.md')]
total = sum(f.get('size', 0) for f in files)
if shutil.disk_usage(root).free < total + 3_000_000_000:
raise RuntimeError('Insufficient space for checkpoint plus 3 GB margin')
manifest = {'repo': args.repo, 'revision': revision, 'expected_bytes': total,
'started_unix': time.time(), 'files': [], 'license': info.get('cardData', {}).get('license')}
(root / 'source_metadata.json').write_text(json.dumps(info, indent=2))
for entry in files:
name = entry['rfilename']
if pathlib.PurePosixPath(name).name != name:
raise RuntimeError(f'Unexpected nested path: {name}')
dest = weights / name
part = dest.with_suffix(dest.suffix + '.partial')
digest = hashlib.sha256()
url = f'https://huggingface.co/{args.repo}/resolve/{revision}/{urllib.parse.quote(name)}?download=true'
print(f'Downloading {name}, expected {entry.get("size")} bytes', flush=True)
if entry.get('size', 0) > 128 * 1024 * 1024:
ranged_download(url, part, entry['size'])
with part.open('rb') as stream:
while chunk := stream.read(4 * 1024 * 1024):
digest.update(chunk)
else:
with urllib.request.urlopen(url, timeout=180) as response, part.open('wb') as out:
while chunk := response.read(4 * 1024 * 1024):
out.write(chunk)
digest.update(chunk)
sha = digest.hexdigest()
if entry.get('size') and part.stat().st_size != entry['size']:
raise RuntimeError(f'Size mismatch: {name}')
expected_sha = entry.get('lfs', {}).get('sha256')
if expected_sha and sha != expected_sha:
raise RuntimeError(f'SHA256 mismatch: {name}')
part.rename(dest)
manifest['files'].append({'name': name, 'bytes': dest.stat().st_size,
'sha256': sha, 'upstream_sha256': expected_sha})
(root / 'manifest.json').write_text(json.dumps(manifest, indent=2))
manifest['completed_unix'] = time.time()
(root / 'manifest.json').write_text(json.dumps(manifest, indent=2))
shutil.copy2(weights / 'config.json', root / 'model_config.json')
print(json.dumps({'revision': revision, 'bytes': sum(f['bytes'] for f in manifest['files'])}), flush=True)
if __name__ == '__main__':
main()
+261
View File
@@ -0,0 +1,261 @@
"""Scientific figures and a compact result report from saved data only."""
import argparse
import csv
import json
import pathlib
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
from matplotlib.patches import Patch, Rectangle
import numpy as np
def read_csv(path):
with path.open() as f:
return list(csv.DictReader(f))
def write_csv(path, rows):
with path.open('w', newline='') as f:
if rows:
writer = csv.DictWriter(f, fieldnames=list(rows[0]))
writer.writeheader()
writer.writerows(rows)
def main():
p = argparse.ArgumentParser()
p.add_argument('--root', type=pathlib.Path, required=True)
args = p.parse_args()
root = args.root
out = root / 'analysis'
figures = root / 'figures'
figures.mkdir(exist_ok=True)
plt.rcParams.update({'font.size': 10, 'axes.spines.top': False, 'axes.spines.right': False,
'pdf.fonttype': 42, 'svg.fonttype': 'none', 'figure.dpi': 120})
def save(fig, name):
fig.tight_layout()
for ext in ('png', 'pdf', 'svg'):
fig.savefig(figures / f'{name}.{ext}', dpi=200, bbox_inches='tight')
plt.close(fig)
curves = read_csv(out / 'compression_curve.csv')
summary = json.loads((out / 'summary.json').read_text())
dimensional = {}
for row in read_csv(out / 'inventory.csv'):
ndim = int(row['ndim'])
item = dimensional.setdefault(ndim, {'ndim': ndim, 'tensor_count': 0, 'weight_bytes': 0})
item['tensor_count'] += 1
item['weight_bytes'] += int(row['bytes'])
dimensional_rows = [dimensional[key] for key in sorted(dimensional)]
for item in dimensional_rows:
item['model_byte_fraction'] = item['weight_bytes'] / summary['model_weight_bytes']
with (out / 'dimensional_bytes.csv').open('w', newline='') as f:
writer = csv.DictWriter(f, fieldnames=list(dimensional_rows[0]))
writer.writeheader()
writer.writerows(dimensional_rows)
fig, axes = plt.subplots(1, 2, figsize=(11, 4))
for scale in sorted({r['scale'] for r in curves}):
rows = [r for r in curves if r['scale'] == scale]
x = [100 * float(r['threshold']) for r in rows]
axes[0].plot(x, [100 * float(r['net_saved_model_fraction']) for r in rows], 'o-', label=scale)
axes[1].plot(x, [int(r['shared_groups']) for r in rows], 'o-', label=scale)
axes[0].set(xlabel='Maximum relative weight error (%)', ylabel='Verified net savings / model bytes (%)')
axes[1].set(xlabel='Maximum relative weight error (%)', ylabel='Non-singleton sharing groups')
axes[0].legend(fontsize=8, loc='upper left')
fig.suptitle('Conservative verified sharing; block scales are separate alternatives')
save(fig, 'compression_frontier')
pair_sets = {mode: read_csv(out / f'{mode}_pairs.csv') for mode in ('tensor', 'block')}
role_rows = []
for mode, pairs in pair_sets.items():
mode_objects = json.loads((out / f'{mode}_objects.json').read_text())
buckets = {}
for row in pairs:
obj = mode_objects[int(row['a'])]
key = (obj['role'], obj['scale'], tuple(obj['shape']), obj['dtype'])
buckets.setdefault(key, []).append(row)
for (role_name, scale_name, shape, dtype), rows in sorted(buckets.items(), key=lambda item: str(item[0])):
errors = [float(r['error_member_to_rep']) for r in rows]
cosines = [float(r['cosine']) for r in rows if r['cosine']]
role_rows.append({'mode': mode, 'role': role_name, 'scale': scale_name,
'shape': str(shape), 'ndim': len(shape), 'dtype': dtype,
'verified_pairs': len(rows), 'minimum_direct_error': min(errors),
'median_direct_error': float(np.median(errors)),
'maximum_cosine': max(cosines) if cosines else '',
'minimum_scaled_error': min(float(r['scaled_error']) for r in rows),
'pairs_under_5pct_direct_error': sum(e <= .05 for e in errors),
'pairs_under_20pct_direct_error': sum(e <= .20 for e in errors)})
with (out / 'role_summary.csv').open('w', newline='') as f:
if role_rows:
writer = csv.DictWriter(f, fieldnames=list(role_rows[0]))
writer.writeheader()
writer.writerows(role_rows)
fig, axes = plt.subplots(2, 2, figsize=(11, 7))
for row, (mode, pairs) in enumerate(pair_sets.items()):
for scale in sorted({r['scale_name'] for r in pairs}):
group = [r for r in pairs if r['scale_name'] == scale]
axes[row, 0].hist([float(r['cosine']) for r in group if r['cosine']], bins=np.linspace(-1, 1, 46), histtype='step', label=scale)
errors = [float(r['error_member_to_rep']) for r in group]
axes[row, 1].hist(errors, bins=np.linspace(0, max(1e-6, max(errors, default=0.) * 1.01), 46), histtype='step', label=scale)
axes[row, 0].set(title=f'{mode}: exact cosine of screened candidates', xlabel='Cosine', ylabel='Candidate pairs')
axes[row, 1].set(title=f'{mode}: exact direct replacement error', xlabel='Relative Frobenius error', ylabel='Candidate pairs')
axes[row, 0].legend(fontsize=7, loc='upper right')
save(fig, 'candidate_distributions')
pairs = pair_sets['tensor']
roles = sorted({r['role'] for r in pairs})
# Illustrate one repeated 2D projection; this is not a byte-weighted model summary.
objects = json.loads((out / 'tensor_objects.json').read_text())
candidates = [role for role in roles if any(o['role'] == role and len(o['shape']) == 2 for o in objects)]
role = max(candidates or roles, key=lambda k: sum(r['role'] == k for r in pairs))
chosen = [(i, o) for i, o in enumerate(objects) if o['role'] == role]
chosen.sort(key=lambda item: item[1]['layer'])
indices = {i: j for j, (i, _) in enumerate(chosen)}
matrix = np.full((len(chosen), len(chosen)), np.nan)
np.fill_diagonal(matrix, 1.)
for r in pairs:
a, b = int(r['a']), int(r['b'])
if a in indices and b in indices and r['cosine']:
matrix[indices[a], indices[b]] = matrix[indices[b], indices[a]] = float(r['cosine'])
fig, ax = plt.subplots(figsize=(7, 6))
cmap = plt.get_cmap('coolwarm').with_extremes(bad='white')
im = ax.imshow(matrix, cmap=cmap, vmin=-1, vmax=1)
for y, x in zip(*np.where(np.isnan(matrix))):
ax.add_patch(Rectangle((x - .5, y - .5), 1, 1, facecolor='none',
edgecolor='#999999', hatch='///', linewidth=0))
labels = [o['layer'] for _, o in chosen]
ticks = np.arange(0, len(labels), max(1, len(labels) // 10))
ax.set_xticks(ticks, [labels[i] for i in ticks])
ax.set_yticks(ticks, [labels[i] for i in ticks])
ax.set(xlabel='Layer', ylabel='Layer', title=role.removeprefix('model.') + '\nHatching = unverified; diagonal is identity')
ax.legend(handles=[Patch(facecolor='white', edgecolor='#999999', hatch='///', label='Unverified pair')],
loc='upper center', bbox_to_anchor=(.5, -.12), fontsize=8, frameon=False)
fig.colorbar(im, ax=ax, label='Exact cosine')
save(fig, 'layer_heatmap')
heatmap_rows = [{'a_layer': oa['layer'], 'b_layer': ob['layer'], 'role': role,
'cosine': float(matrix[i, j]) if not np.isnan(matrix[i, j]) else '',
'verified': bool(i != j and not np.isnan(matrix[i, j])), 'identity': i == j}
for i, (_, oa) in enumerate(chosen) for j, (_, ob) in enumerate(chosen)]
write_csv(out / 'heatmap_pairs.csv', heatmap_rows)
distance_rows = []
fig, axes = plt.subplots(1, 2, figsize=(12, 4), sharey=True)
for ax, mode in zip(axes, ('tensor', 'block')):
mode_objects = json.loads((out / f'{mode}_objects.json').read_text())
buckets = {}
for row in pair_sets[mode]:
if not row['cosine']:
continue
obj = mode_objects[int(row['a'])]
stratum = f"{len(obj['shape'])}D" if mode == 'tensor' else obj['scale']
buckets.setdefault(stratum, {}).setdefault(int(row['layer_distance']), []).append(float(row['cosine']))
for stratum, by_distance in sorted(buckets.items()):
distances = sorted(by_distance)
medians = []
for distance in distances:
values = by_distance[distance]
median = float(np.median(values))
medians.append(median)
distance_rows.append({'mode': mode, 'stratum': stratum, 'layer_distance': distance,
'pairs': len(values), 'median_cosine': median,
'q25': float(np.quantile(values, .25)), 'q75': float(np.quantile(values, .75))})
ax.plot(distances, medians, 'o-', markersize=3, label=stratum)
ax.set(xlabel='Absolute layer distance', ylabel='Median exact cosine',
title=f'{mode}: dimension / scale strata', ylim=(-.1, 1.05))
ax.legend(fontsize=8, loc='upper center', bbox_to_anchor=(.5, -.23), ncol=2, frameon=False)
fig.suptitle('Screened candidates: module mixture and selection bias still apply')
write_csv(out / 'layer_distance_stratified.csv', distance_rows)
save(fig, 'layer_distance')
null_rows = []
fig, axes = plt.subplots(3, 2, figsize=(12, 10))
bins = np.linspace(-1, 1, 26)
for col, mode in enumerate(('tensor', 'block')):
mode_objects = json.loads((out / f'{mode}_objects.json').read_text())
nulls = json.loads((out / f'{mode}_nulls.json').read_text())
real_random = json.loads((out / f'{mode}_random_pairs.json').read_text())
cohorts = [('shuffled', [r for r in nulls if r['method'] == 'shuffled']),
('gaussian_norm_matched', [r for r in nulls if r['method'] == 'gaussian_norm_matched']),
('random_actual_same_role', real_random)]
for row, ndim in enumerate((1, 2, 3)):
ax = axes[row, col]
any_values = False
for method, records in cohorts:
values = np.asarray([r['cosine'] for r in records if r['cosine'] is not None
and len(mode_objects[r['a']]['shape']) == ndim])
if not len(values):
continue
any_values = True
values = np.clip(values, -1, 1)
ax.hist(values, bins=bins, density=True, histtype='step', label=f'{method} (n={len(values)})')
counts, _ = np.histogram(values, bins=bins)
for i, count in enumerate(counts):
null_rows.append({'mode': mode, 'ndim': ndim, 'method': method, 'samples': len(values),
'bin_left': float(bins[i]), 'bin_right': float(bins[i + 1]),
'count': int(count), 'density': float(count / (len(values) * (bins[i + 1] - bins[i])))})
ax.set(title=f'{mode}: {ndim}D sampled controls', xlabel='Sampled cosine', ylabel='Density')
if any_values:
ax.legend(fontsize=7, loc='upper right')
else:
ax.text(.5, .5, 'No eligible samples', transform=ax.transAxes, ha='center')
fig.suptitle('Dimension-stratified controls: density normalized; role mixtures may differ')
write_csv(out / 'null_histograms.csv', null_rows)
save(fig, 'null_controls')
fig, axes = plt.subplots(1, 2, figsize=(11, 4))
structure = json.loads((out / 'block_structure.json').read_text())
for ax, ndim in zip(axes, (2, 3)):
probes = [m for pair in structure for m in pair['modes'] if len(m['probe_shape']) == ndim and not m.get('trivial_mode_skipped')]
for axis in sorted({m['axis'] for m in probes}):
selected = [m for m in probes if m['axis'] == axis]
ax.scatter([m['normalized_spectrum_distance'] for m in selected],
[m['left_subspace_overlap'] for m in selected], s=18, alpha=.65, label=f'mode {axis}')
ax.set(title=f'{ndim}D blocks: bounded mode-unfolding probes', xlabel='Normalized spectrum distance', ylabel='Leading left subspace overlap (k <= 4)')
if probes:
ax.legend(fontsize=8, loc='upper right')
else:
ax.text(.5, .5, 'No eligible probes', transform=ax.transAxes, ha='center')
save(fig, 'block_structure')
parts = json.loads((out / 'block_partitions.json').read_text())
lines = ['# Qwen3.5-2B parameter similarity pilot', '',
f"- Revision: `{summary['revision']}`", f"- Stored parameters: {summary['parameter_count']:,}",
f"- Weight bytes: {summary['model_weight_bytes']:,}",
f"- Tensor dimensionalities: {summary['ndim_counts']}",
f"- Analysis and grouping/export wall time: {summary['elapsed_seconds']:.1f} seconds",
f"- Pair-comparison cutoff: {summary['analysis_budget_seconds']} seconds; final grouping/export is additional and remains under the supervisor hard limit.",
f"- Peak analysis RSS: {summary['peak_rss_kib'] / 1024:.1f} MiB", '',
'## Verified sharing estimates', '',
'| Representation | Weight-error limit | Verified sharing groups | Conservative total groups* | Largest group | Net model savings |',
'|---|---:|---:|---:|---:|---:|---:|']
for r in curves:
lines.append(f"| {r['scale']} | {float(r['threshold']):.0%} | {r['shared_groups']} | {r['conservative_partition_groups']} | {r['max_group_size']} | {float(r['net_saved_model_fraction']):.6%} |")
lines += ['', '*Conservative total groups counts unexamined full blocks as singletons; boundary regions and ineligible tensors stay unchanged and are outside that block count.',
'Savings count only verified replacement edges. Unsampled blocks remain unchanged. Different block scales must not be summed.',
'Theoretical encoding: one stored representative per group, 16 bytes per object mapping and 64 bytes per shared-group descriptor; identity fallback if larger.',
'Exact grouping uses direct replacement only. Scaled and spectral results are diagnostic.', '',
'## Stratified diagnostics', '',
'See `analysis/role_summary.csv` for same-role, same-shape statistics, separated by tensor dimensionality and block scale.',
'See `analysis/dimensional_bytes.csv` for tensor counts versus weight-byte contributions by dimensionality.',
'Heatmap missing pairs are hatched (`heatmap_pairs.csv`); identity entries are not empirical matches.',
'Layer-distance medians are split by dimension/scale (`layer_distance_stratified.csv`); role and selection bias remain.',
'Controls are split by dimension and density-normalized, with sample counts (`null_histograms.csv`); these are not significance tests.',
'These are screened-candidate statistics, not estimates of all possible pairs. Small normalization vectors must not be conflated with large matrices.', '',
'## Coverage', '']
for mode in ('tensor', 'block'):
coverage = json.loads((out / f'{mode}_coverage.json').read_text())
lines.append(f'- {mode}: `{json.dumps(coverage)}`')
lines += [f'- Complete-block partition totals (per alternative): `{json.dumps(parts)}`', '',
'## Interpretation limits', '', *['- ' + s for s in summary['limitations']],
'- Candidate histograms are selection-biased and do not represent the distribution of all tensor pairs.',
'- 1D normalization weights may cluster strongly but carry few bytes; do not infer material compression from group counts.',
'- No task accuracy, perplexity, inference speed or runtime-memory improvement has been measured.',
'- Stored parameter counts may omit tied aliases; no duplicate-weight equality search is performed.', '',
'## Figures', '', *[f'![{name}](figures/{name}.png)' for name in ['compression_frontier', 'candidate_distributions', 'layer_heatmap', 'layer_distance', 'null_controls', 'block_structure']]]
(root / 'REPORT.md').write_text('\n'.join(lines) + '\n')
if __name__ == '__main__':
main()
+58
View File
@@ -0,0 +1,58 @@
"""Supervise the one-model job; no automatic download of subsequent models."""
import argparse
import json
import os
import pathlib
import subprocess
import sys
import time
def main():
p = argparse.ArgumentParser()
p.add_argument('--root', type=pathlib.Path, required=True)
args = p.parse_args()
root = args.root.resolve()
root.mkdir(exist_ok=True, parents=True)
status_file = root / 'status.json'
start = time.time()
env = dict(os.environ, OPENBLAS_NUM_THREADS='1', OMP_NUM_THREADS='1')
def status(phase, **extra):
status_file.write_text(json.dumps({'phase': phase, 'updated_unix': time.time(),
'supervisor_started_unix': start, **extra}, indent=2))
print(phase, flush=True)
def command(module, log, timeout, extra=()):
with (root / log).open('w') as stream:
subprocess.run([sys.executable, '-B', '-m', module, '--root', str(root), *extra],
check=True, timeout=timeout, env=env, stdout=stream, stderr=subprocess.STDOUT)
try:
status('waiting_for_download')
download_wait_start = time.monotonic()
while True:
manifest_path = root / 'manifest.json'
if manifest_path.exists():
manifest = json.loads(manifest_path.read_text())
if manifest.get('completed_unix') and all((root / 'weights' / f['name']).exists() for f in manifest['files']):
break
if time.monotonic() - download_wait_start > 7200:
raise TimeoutError('Download wait exceeded 2 hours; no analysis started')
time.sleep(10)
status('analyzing', analysis_hard_limit_seconds=7200)
command('pilot.analyze', 'analysis.log', 7200, ('--seconds', '6600'))
status('plotting')
command('pilot.plot', 'plot.log', 300)
status('validating')
command('pilot.validate', 'validate.log', 120)
with (root / 'environment.txt').open('w') as f:
subprocess.run([sys.executable, '-B', '-m', 'pip', 'freeze'], check=True, stdout=f, env=env)
status('results_ready_weights_retained_until_archive_copy')
except Exception as e:
status('failed', error_type=type(e).__name__, error=str(e))
raise
if __name__ == '__main__':
main()
+82
View File
@@ -0,0 +1,82 @@
"""Read safetensors BF16 directly; never instantiate a model or use pickle."""
import json
import math
import mmap
import re
import struct
import numpy as np
DTYPES = {'BF16': ('<u2', 2), 'F16': ('<f2', 2), 'F32': ('<f4', 4), 'F64': ('<f8', 8)}
def decode(raw, dtype):
if dtype == 'BF16':
return (np.asarray(raw).astype(np.uint32) << 16).view(np.float32)
return np.asarray(raw, dtype=np.float32)
class Tensor:
def __init__(self, file, base, name, meta):
self.name, self.shape, self.dtype = name, tuple(meta['shape']), meta['dtype']
self.numel = math.prod(self.shape)
self.bytes = meta['data_offsets'][1] - meta['data_offsets'][0]
code, itemsize = DTYPES[self.dtype]
if self.bytes != self.numel * itemsize:
raise ValueError(f'Invalid byte length: {name}')
self.raw = np.memmap(file, mode='r', dtype=code,
offset=base + meta['data_offsets'][0], shape=(self.numel,))
self.role = re.sub(r'\.(?:layers|blocks)\.\d+', '.layers.*', name)
self.role = re.sub(r'\.experts\.\d+', '.experts.*', self.role)
m = re.search(r'\.(?:layers|blocks)\.(\d+)', name)
self.layer = int(m[1]) if m else -1
def read(self, selector=slice(None)):
return decode(self.raw[selector], self.dtype)
def chunked(self, size=262144):
for start in range(0, self.numel, size):
yield self.read(slice(start, min(start + size, self.numel)))
def evict(self):
if hasattr(self.raw._mmap, 'madvise'):
self.raw._mmap.madvise(mmap.MADV_DONTNEED)
def load_tensors(directory):
tensors = {}
for file in sorted(directory.glob('*.safetensors')):
with file.open('rb') as f:
size = struct.unpack('<Q', f.read(8))[0]
if size > 100_000_000:
raise ValueError('Oversized safetensors header')
header = json.loads(f.read(size))
for name, meta in header.items():
if name == '__metadata__':
continue
if name in tensors:
raise ValueError(f'Duplicate tensor name: {name}')
tensors[name] = Tensor(file, 8 + size, name, meta)
if not tensors:
raise ValueError('No safetensors found')
return tensors
def metrics(x, y):
"""x is member; y is representative. Float64 accumulation."""
x, y = x.astype(np.float64), y.astype(np.float64)
xx, yy, xy = float(x @ x), float(y @ y), float(x @ y)
delta = float((x - y) @ (x - y))
return finish(xx, yy, xy, delta, float(x.sum()), float(y.sum()), x.size)
def finish(xx, yy, xy, delta, sx, sy, n):
tiny = 1e-30
vx, vy = max(0., xx - sx * sx / n), max(0., yy - sy * sy / n)
return {'cosine': xy / np.sqrt(xx * yy) if xx * yy > tiny else None,
'pearson': (xy - sx * sy / n) / np.sqrt(vx * vy) if vx * vy > tiny else None,
'error_member_to_rep': np.sqrt(delta / max(xx, tiny)),
'error_rep_to_member': np.sqrt(delta / max(yy, tiny)),
'symmetric_error': np.sqrt(2 * delta / max(xx + yy, tiny)),
'scaled_error': np.sqrt(max(0., xx - xy * xy / max(yy, tiny)) / max(xx, tiny)),
'scale': xy / max(yy, tiny)}
+85
View File
@@ -0,0 +1,85 @@
"""Validate archived analysis before allowing deletion of downloaded weights."""
import argparse
import csv
import hashlib
import json
import pathlib
import shutil
import time
def validate(root):
analysis = root / 'analysis'
summary = json.loads((analysis / 'summary.json').read_text())
assert summary['completed']
manifest = json.loads((root / 'manifest.json').read_text())
assert summary['revision'] == manifest['revision']
for mode in ('tensor', 'block'):
objects = json.loads((analysis / f'{mode}_objects.json').read_text())
with (analysis / f'{mode}_pairs.csv').open() as f:
pairs = list(csv.DictReader(f))
coverage = json.loads((analysis / f'{mode}_coverage.json').read_text())
assert len(pairs) == coverage['pairs_verified']
edges = {}
for p in pairs:
a, b = int(p['a']), int(p['b'])
assert objects[a]['shape'] == objects[b]['shape']
assert objects[a]['role'] == objects[b]['role']
edges[(b, a)] = float(p['error_member_to_rep'])
edges[(a, b)] = float(p['error_rep_to_member'])
for file in analysis.glob(f'groups_{mode}_*.json'):
threshold = float(file.stem.rsplit('_', 1)[1])
groups = json.loads(file.read_text())
members = [i for g in groups for i in g['members']]
assert len(set(members)) == len(members), file
for g in groups:
assert g['representative'] in g['members']
for member in g['members']:
if member != g['representative']:
assert edges[(g['representative'], member)] <= threshold + 1e-12
assert g['max_error'] <= threshold + 1e-12
assert (root / 'REPORT.md').is_file()
images = list((root / 'figures').glob('*.png'))
assert len(images) >= 5
for file in images:
assert file.read_bytes()[:8] == b'\x89PNG\r\n\x1a\n'
checksums = {}
for file in sorted(root.rglob('*')):
if (file.is_file() and 'weights' not in file.relative_to(root).parts
and file.name not in ('checksums.json', 'validation.json', 'cleanup.json', 'status.json')
and not file.name.endswith('.log')):
checksums[str(file.relative_to(root))] = hashlib.sha256(file.read_bytes()).hexdigest()
(root / 'checksums.json').write_text(json.dumps(checksums, indent=2))
result = {'validated_unix': time.time(), 'files_checksummed': len(checksums),
'group_edges_checked': True, 'passed': True}
(root / 'validation.json').write_text(json.dumps(result, indent=2))
return result
def main():
p = argparse.ArgumentParser()
p.add_argument('--root', required=True, type=pathlib.Path)
p.add_argument('--cleanup-weights', action='store_true')
args = p.parse_args()
root = args.root.resolve()
result = validate(root)
if args.cleanup_weights:
weights = root / 'weights'
manifest = json.loads((root / 'manifest.json').read_text())
assert not weights.is_symlink()
assert (weights / '.pilot-owned').read_text() == manifest['repo']
expected_names = {entry['name'] for entry in manifest['files']} | {'.pilot-owned'}
assert {file.name for file in weights.iterdir()} == expected_names
assert all(file.is_file() and not file.is_symlink() for file in weights.iterdir())
deleted_bytes = sum(file.stat().st_size for file in weights.iterdir())
shutil.rmtree(weights)
(root / 'cleanup.json').write_text(json.dumps({'deleted_unix': time.time(),
'deleted_bytes': deleted_bytes,
'deleted_directory': 'weights',
'reason': 'User-authorized delete-after-analysis; archived results validated'}, indent=2))
result['weights_removed'] = not weights.exists()
print(json.dumps(result))
if __name__ == '__main__':
main()
+5
View File
@@ -0,0 +1,5 @@
# Existing requirements.txt supplies NumPy and plotting dependencies.
# Install binary wheels only. Torch is a reference-check dependency, not the streaming runtime.
tokenizers==0.23.3
# Install separately with --index-url https://download.pytorch.org/whl/cpu:
# torch==2.15.0+cpu
+2
View File
@@ -0,0 +1,2 @@
numpy==2.5.4
matplotlib==3.11.2
+7
View File
@@ -0,0 +1,7 @@
# Scientific delivery and retained caches
All figure CSV/JSON source data, per-token NLL, protocols, reports, metadata and historical logs are included.
Three large reproducible intermediate activation caches remain on 8.211.154.134 and in the original full local archive;
`external-activation-cache.json` records exact locations, sizes and SHA256. No weight or cache was deleted.
The original full experiment manifest is `checksums-full-server.json`; the delivery-only manifest is
`checksums-functional.json`. No model checkpoint or tokenizer payload is in this Git delivery.
@@ -0,0 +1,36 @@
# CPU learned-function repetition pilot
Primary question: do two distinct trained parts implement similar functions? Cost is not a success condition.
- Existing learned RMSNorm function positive: True
- Bidirectional network quality screen passed: True
- Learned scalar MLP neuron search positive: False
- Original 11 candidate run state: completed; selected: []
- All model weights are retained on the server until the user explicitly requests deletion.
## Existing learned module evidence
Role: input_layernorm; layers 12 and 13; each maps 2048-dimensional input to output.
Uniform real-arithmetic relative-error bound, first-to-second: 0.101851851852; reverse: 0.113402061856.
Maximum heldout directional output relative L2: 0.04528510757179746; minimum cosine: 0.9994870299724158.
Same epsilon and nonzero gains: the common RMS denominator cancels. For any nonzero input x,
`||f_a(x)-f_b(x)|| / ||f_a(x)|| <= max_j |a_j-b_j|/|a_j|`; swap a/b for the reverse direction.
Selection used only the tight all-input bound across 552 same-role pairs, before inspecting inputs.
Real probes use four calibration and four heldout experiment-authored texts, 32 tokens each, and both source layers.
These are learned, nonconstant gain vectors. The claim concerns RMSNorm modules; it does not establish attention/MLP-body duplication.
- Layer 12 uses 13: mean heldout NLL delta 3.25651368766e-05; sequence P95 delta 0.000984639618287; screen pass=True.
- Layer 13 uses 12: mean heldout NLL delta -0.000363972997404; sequence P95 delta 0.000709721155035; screen pass=True.
## Negative results and separate algebraic result
The original whole-attention, whole-MLP and within-layer Q/K norm candidates did not pass their gates.
The scalar neuron search excludes low-RMS/near-constant profiles and selects at most one pair without heldout feedback; consult its search summary for coverage.
Q/K non-RoPE gain consolidation is a separately verified real-arithmetic identity, not evidence that two existing learned modules duplicate functions.
Earlier local CPU timing records are preserved for provenance; performance is outside the current success criterion.
## Limits and provenance
Original BF16 weights were decoded into streaming FP32 CPU operators; no training, scale fitting, R fitting or GPU.
The numerical reference includes 22 independent official Torch comparisons. Early adapter errors and two resource-guard stops are preserved as historical logs.
The persistent computation ledger includes a conservative 180-second reserve for early checks; no budget reset after that reserve.
The text sample is tiny and short. Network NLL checks are not broad task-accuracy validation.
+88
View File
@@ -0,0 +1,88 @@
[
"DELIVERY.md",
"FUNCTIONAL_REPORT.md",
"attention-head-metrics.csv",
"baseline-heldout-token-nll.npy",
"calibration-metrics.csv",
"candidate-summary.json",
"checksums-full-server.json",
"compute-budget-ledger.json",
"download.log",
"experiment.json",
"external-activation-cache.json",
"figures/algebraic_gain_identity.pdf",
"figures/algebraic_gain_identity.png",
"figures/algebraic_gain_identity.svg",
"figures/candidate_function_errors.pdf",
"figures/candidate_function_errors.png",
"figures/candidate_function_errors.svg",
"figures/learned_norm_function_errors.pdf",
"figures/learned_norm_function_errors.png",
"figures/learned_norm_function_errors.svg",
"figures/learned_norm_network_substitution.pdf",
"figures/learned_norm_network_substitution.png",
"figures/learned_norm_network_substitution.svg",
"final-report.log",
"final-source-hashes.json",
"final-tests.txt",
"functional-code-hashes.json",
"functional-environment.json",
"functional-report.log",
"functional-run.log",
"functional-status.json",
"gain-factorization-k_to_q-token-nll.npy",
"gain-factorization-kernel-timing.csv",
"gain-factorization-optimized-summary.json",
"gain-factorization-optimized-timing.csv",
"gain-factorization-optimized.log",
"gain-factorization-protocol.json",
"gain-factorization-q_to_k-token-nll.npy",
"gain-factorization-score-metrics.csv",
"gain-factorization-summary.json",
"gain-factorization.log",
"install-torch-disk-tmp.log",
"install-torch.log",
"manifest.json",
"model_config.json",
"neuron-reuse-protocol.json",
"neuron-reuse-search-summary.json",
"neuron-reuse-selection.json",
"neuron-reuse-summary.json",
"neuron-reuse.log",
"norm-repetition-12-uses-13-token-nll.npy",
"norm-repetition-13-uses-12-token-nll.npy",
"norm-repetition-function-metrics.csv",
"norm-repetition-protocol.json",
"norm-repetition-selection.json",
"norm-repetition-summary.json",
"norm-repetition-uniform-bounds.csv",
"norm-repetition.log",
"preflight-first-attempt.log",
"preflight-second-attempt.json",
"preflight-second-attempt.log",
"preflight.json",
"preflight.log",
"reference-check-first-attempt.log",
"reference-check-second-attempt.log",
"reference-check.json",
"reference-check.log",
"retention-policy.json",
"samples-tokenized.json",
"selection.json",
"source_metadata.json",
"storage-accounting.log",
"storage-evidence.json",
"supervisor-error.txt",
"supervisor-first-attempt-error.txt",
"supervisor-first-attempt.json",
"supervisor-second-attempt.json",
"supervisor-status.json",
"supervisor-third-attempt.json",
"supervisor.log",
"tokenizer-download.log",
"tokenizers-wheel.log",
"unit-tests.txt",
"checksums-functional.json",
"audit-validation.json",
"archive-files.json"
]
@@ -0,0 +1,193 @@
candidate,source,sample_id,head,first_key_mass_a,first_key_mass_b,js_without_first_key,without_first_key_valid_rows,js_mean,js_p95,js_without_first_query,top5_overlap
attn_3_7,3,cal-zh-1,0,0.178242027759552,0.44051653146743774,0.2887135446071625,31,0.2728434205055237,0.42193013429641724,0.2816448211669922,0.5875
attn_3_7,3,cal-zh-1,1,0.14070242643356323,0.6567835807800293,0.154605433344841,31,0.273810476064682,0.45863786339759827,0.2826431095600128,0.5375
attn_3_7,3,cal-zh-1,2,0.11500269174575806,0.4787790775299072,0.3218191862106323,31,0.35086748003959656,0.46183231472969055,0.3621857762336731,0.53125
attn_3_7,3,cal-zh-1,3,0.17224811017513275,0.6523524522781372,0.22310931980609894,31,0.28077495098114014,0.43149834871292114,0.289832204580307,0.5
attn_3_7,3,cal-zh-1,4,0.11534158885478973,0.6260660886764526,0.19559672474861145,31,0.2952086329460144,0.470927894115448,0.30473145842552185,0.53125
attn_3_7,3,cal-zh-1,5,0.0896417498588562,0.34022626280784607,0.22164911031723022,31,0.2483729124069214,0.3841992914676666,0.256384938955307,0.54375
attn_3_7,3,cal-zh-1,6,0.1247040405869484,0.7327346801757812,0.26632776856422424,31,0.37061089277267456,0.5524277091026306,0.38256609439849854,0.525
attn_3_7,3,cal-zh-1,7,0.05083201080560684,0.44692301750183105,0.04695317894220352,31,0.20246444642543793,0.4956357181072235,0.20899559557437897,0.65625
attn_3_7,3,cal-zh-2,0,0.14007353782653809,0.6036521196365356,0.2504415214061737,31,0.30694448947906494,0.4879894554615021,0.31684595346450806,0.59375
attn_3_7,3,cal-zh-2,1,0.14963164925575256,0.7174584865570068,0.16050270199775696,31,0.2995006740093231,0.44737952947616577,0.3091619610786438,0.58125
attn_3_7,3,cal-zh-2,2,0.1916198432445526,0.6128076314926147,0.24314163625240326,31,0.28848910331726074,0.42006126046180725,0.29779520630836487,0.63125
attn_3_7,3,cal-zh-2,3,0.11550820618867874,0.6839271783828735,0.22851790487766266,31,0.3617430627346039,0.4926488697528839,0.373412162065506,0.43125
attn_3_7,3,cal-zh-2,4,0.1432671993970871,0.714910089969635,0.16698074340820312,31,0.29421237111091614,0.44453173875808716,0.30370306968688965,0.55
attn_3_7,3,cal-zh-2,5,0.15627557039260864,0.4676961600780487,0.22456194460391998,31,0.24842627346515656,0.35863032937049866,0.25644001364707947,0.53125
attn_3_7,3,cal-zh-2,6,0.10968590527772903,0.8703562021255493,0.2677042484283447,31,0.44428783655166626,0.6032188534736633,0.45861977338790894,0.4625
attn_3_7,3,cal-zh-2,7,0.07148394733667374,0.5658342838287354,0.050552818924188614,31,0.24003271758556366,0.5258023738861084,0.24777570366859436,0.6875
attn_3_7,3,cal-en-1,0,0.11021165549755096,0.7443452477455139,0.2832939028739929,31,0.4321710467338562,0.5805379152297974,0.4461120665073395,0.5
attn_3_7,3,cal-en-1,1,0.14157001674175262,0.6908999681472778,0.15555045008659363,31,0.3069734573364258,0.46220970153808594,0.3168758153915405,0.5562499999999999
attn_3_7,3,cal-en-1,2,0.22632445394992828,0.6502443552017212,0.3039384186267853,31,0.33335912227630615,0.4760979115962982,0.34411266446113586,0.575
attn_3_7,3,cal-en-1,3,0.1636783331632614,0.749056875705719,0.18588410317897797,31,0.31960076093673706,0.4827522933483124,0.3299104571342468,0.5062500000000001
attn_3_7,3,cal-en-1,4,0.18975511193275452,0.5745540857315063,0.20487342774868011,31,0.2522231340408325,0.39599278569221497,0.26035937666893005,0.60625
attn_3_7,3,cal-en-1,5,0.1436786651611328,0.45787984132766724,0.1883384883403778,31,0.2209610939025879,0.3852173388004303,0.2280888706445694,0.6187499999999999
attn_3_7,3,cal-en-1,6,0.08422182500362396,0.8810427784919739,0.24220214784145355,31,0.4833376407623291,0.6262145638465881,0.49892911314964294,0.45625
attn_3_7,3,cal-en-1,7,0.09002429246902466,0.49231916666030884,0.053437694907188416,31,0.19592277705669403,0.4695172905921936,0.20224282145500183,0.6624999999999999
attn_3_7,3,cal-en-2,0,0.09620732069015503,0.7015095949172974,0.2738996148109436,31,0.3950675129890442,0.5772150158882141,0.40781161189079285,0.5874999999999999
attn_3_7,3,cal-en-2,1,0.1168593019247055,0.7301980257034302,0.160212442278862,31,0.3232686221599579,0.4884706735610962,0.3336966037750244,0.55625
attn_3_7,3,cal-en-2,2,0.21824826300144196,0.655203104019165,0.3244048058986664,31,0.3314599394798279,0.489634245634079,0.3421522378921509,0.49375
attn_3_7,3,cal-en-2,3,0.15237370133399963,0.7099946737289429,0.253733366727829,31,0.34891438484191895,0.4875939190387726,0.3601696491241455,0.4
attn_3_7,3,cal-en-2,4,0.20830751955509186,0.6707655191421509,0.16886863112449646,31,0.23844638466835022,0.41475459933280945,0.24613820016384125,0.58125
attn_3_7,3,cal-en-2,5,0.20089492201805115,0.4360402226448059,0.20298565924167633,31,0.19701015949249268,0.3117577135562897,0.20336531102657318,0.5875
attn_3_7,3,cal-en-2,6,0.08693217486143112,0.8322523832321167,0.28768372535705566,31,0.45854610204696655,0.604308545589447,0.47333794832229614,0.4625
attn_3_7,3,cal-en-2,7,0.12079077959060669,0.5237122178077698,0.08786331117153168,31,0.1976630836725235,0.4211302399635315,0.2040393203496933,0.71875
attn_3_7,7,cal-zh-1,0,0.08316324651241302,0.1214274913072586,0.22053609788417816,31,0.2072795033454895,0.32793501019477844,0.21396592259407043,0.5625
attn_3_7,7,cal-zh-1,1,0.11647684127092361,0.187703475356102,0.09203154593706131,31,0.09238047897815704,0.16611215472221375,0.09536049515008926,0.66875
attn_3_7,7,cal-zh-1,2,0.06550130248069763,0.07431766390800476,0.28803399205207825,31,0.2721514105796814,0.46315261721611023,0.28093045949935913,0.47500000000000003
attn_3_7,7,cal-zh-1,3,0.10703553259372711,0.24631237983703613,0.1426287740468979,31,0.150299072265625,0.26700741052627563,0.1551474630832672,0.56875
attn_3_7,7,cal-zh-1,4,0.04052526503801346,0.28515881299972534,0.11374201625585556,31,0.1819339096546173,0.2913167476654053,0.1878027319908142,0.5875
attn_3_7,7,cal-zh-1,5,0.03951618820428848,0.12756675481796265,0.12323599308729172,31,0.13915526866912842,0.22263947129249573,0.14364413917064667,0.6062500000000001
attn_3_7,7,cal-zh-1,6,0.09479819983243942,0.28205406665802,0.14449526369571686,31,0.16065017879009247,0.30267569422721863,0.16583245992660522,0.6375000000000001
attn_3_7,7,cal-zh-1,7,0.031424712389707565,0.10597768425941467,0.029422910884022713,31,0.05383709818124771,0.10532202571630478,0.055573780089616776,0.60625
attn_3_7,7,cal-zh-2,0,0.0902467742562294,0.11627861112356186,0.21221575140953064,31,0.19495421648025513,0.317206472158432,0.201243057847023,0.6187499999999999
attn_3_7,7,cal-zh-2,1,0.11616441607475281,0.1248721182346344,0.0952492356300354,31,0.08693543076515198,0.16702036559581757,0.08973980695009232,0.5874999999999999
attn_3_7,7,cal-zh-2,2,0.11578349769115448,0.08946777880191803,0.2738611102104187,31,0.2561575174331665,0.45070335268974304,0.26442068815231323,0.5
attn_3_7,7,cal-zh-2,3,0.11586448550224304,0.16896863281726837,0.10842260718345642,31,0.10158580541610718,0.21354182064533234,0.1048627570271492,0.60625
attn_3_7,7,cal-zh-2,4,0.044329218566417694,0.22534678876399994,0.15060655772686005,31,0.19093912839889526,0.3356643319129944,0.19709844887256622,0.61875
attn_3_7,7,cal-zh-2,5,0.05062376335263252,0.12924723327159882,0.1047384962439537,31,0.11341066658496857,0.2176104635000229,0.1170690730214119,0.6312500000000001
attn_3_7,7,cal-zh-2,6,0.08418147265911102,0.28032630681991577,0.1446688026189804,31,0.16730237007141113,0.24755442142486572,0.1726992279291153,0.5125
attn_3_7,7,cal-zh-2,7,0.0315190926194191,0.11244469881057739,0.02086171880364418,31,0.04779163748025894,0.10416773706674576,0.04933330416679382,0.65
attn_3_7,7,cal-en-1,0,0.08185911923646927,0.10868857800960541,0.24351894855499268,31,0.22597578167915344,0.35052549839019775,0.23326534032821655,0.66875
attn_3_7,7,cal-en-1,1,0.12180840969085693,0.12996530532836914,0.1148008182644844,31,0.10937604308128357,0.1990041583776474,0.11290428787469864,0.6625
attn_3_7,7,cal-en-1,2,0.12156520038843155,0.07304315268993378,0.32663044333457947,31,0.31218329071998596,0.4966602325439453,0.3222537338733673,0.4375
attn_3_7,7,cal-en-1,3,0.09531049430370331,0.21274563670158386,0.13099102675914764,31,0.14007462561130524,0.2635233998298645,0.14459317922592163,0.5874999999999999
attn_3_7,7,cal-en-1,4,0.051348961889743805,0.27256035804748535,0.17027397453784943,31,0.22669382393360138,0.33905521035194397,0.23400652408599854,0.59375
attn_3_7,7,cal-en-1,5,0.05210518091917038,0.14343655109405518,0.11446496844291687,31,0.12723194062709808,0.23429512977600098,0.13133619725704193,0.64375
attn_3_7,7,cal-en-1,6,0.09278014302253723,0.3040335178375244,0.20964911580085754,31,0.21833369135856628,0.34637004137039185,0.22537672519683838,0.53125
attn_3_7,7,cal-en-1,7,0.031563661992549896,0.11319160461425781,0.02739104814827442,31,0.053969405591487885,0.1002146303653717,0.055710360407829285,0.5625000000000001
attn_3_7,7,cal-en-2,0,0.07943743467330933,0.1426258534193039,0.21740850806236267,31,0.21001917123794556,0.35335251688957214,0.2167939990758896,0.58125
attn_3_7,7,cal-en-2,1,0.13507506251335144,0.15443003177642822,0.10350806266069412,31,0.09826720505952835,0.18229186534881592,0.10143710672855377,0.575
attn_3_7,7,cal-en-2,2,0.07290980964899063,0.08657881617546082,0.26718389987945557,31,0.259122759103775,0.4706645905971527,0.26748156547546387,0.45625
attn_3_7,7,cal-en-2,3,0.06336543709039688,0.2253912389278412,0.11350395530462265,31,0.1445191204547882,0.30314192175865173,0.1491810381412506,0.5875
attn_3_7,7,cal-en-2,4,0.04387465864419937,0.2312193661928177,0.14195290207862854,31,0.1880926638841629,0.3466520309448242,0.19416013360023499,0.5125
attn_3_7,7,cal-en-2,5,0.05407806113362312,0.1308087259531021,0.15231285989284515,31,0.16046105325222015,0.3131849467754364,0.16563721001148224,0.5874999999999999
attn_3_7,7,cal-en-2,6,0.10013270378112793,0.30412447452545166,0.17993809282779694,31,0.19388216733932495,0.34072187542915344,0.2001364380121231,0.5437500000000001
attn_3_7,7,cal-en-2,7,0.03161787986755371,0.14472994208335876,0.04679951071739197,31,0.0830649882555008,0.13866016268730164,0.08574451506137848,0.65625
attn_11_15,11,cal-zh-1,0,0.2780076861381531,0.40727436542510986,0.08935041725635529,31,0.08769044280052185,0.1605687141418457,0.09051916748285294,0.6937500000000001
attn_11_15,11,cal-zh-1,1,0.599311351776123,0.1917644888162613,0.1712544858455658,31,0.22863143682479858,0.39283493161201477,0.23600666224956512,0.5750000000000001
attn_11_15,11,cal-zh-1,2,0.15545204281806946,0.5101379156112671,0.08800674974918365,31,0.15655069053173065,0.3642641007900238,0.16160069406032562,0.73125
attn_11_15,11,cal-zh-1,3,0.11032958328723907,0.4199199676513672,0.08535198122262955,31,0.14290741086006165,0.27922654151916504,0.14751732349395752,0.7
attn_11_15,11,cal-zh-1,4,0.18725907802581787,0.3965998888015747,0.12378748506307602,31,0.1280820369720459,0.24588972330093384,0.13221368193626404,0.6937500000000001
attn_11_15,11,cal-zh-1,5,0.3307006359100342,0.43053850531578064,0.12539583444595337,31,0.1220257431268692,0.27262142300605774,0.1259620636701584,0.6187499999999999
attn_11_15,11,cal-zh-1,6,0.17290356755256653,0.4159870147705078,0.10268668830394745,31,0.12616759538650513,0.27286267280578613,0.13023751974105835,0.625
attn_11_15,11,cal-zh-1,7,0.12546084821224213,0.4914926290512085,0.11725502461194992,31,0.17813995480537415,0.28658801317214966,0.18388642370700836,0.6625000000000001
attn_11_15,11,cal-zh-2,0,0.2187591940164566,0.42793723940849304,0.09548557549715042,31,0.12088388949632645,0.2345586121082306,0.12478335946798325,0.6875
attn_11_15,11,cal-zh-2,1,0.5546900033950806,0.1175224781036377,0.16409580409526825,31,0.25557559728622437,0.37492382526397705,0.26381993293762207,0.6125
attn_11_15,11,cal-zh-2,2,0.11856245994567871,0.44309669733047485,0.08432265371084213,31,0.1634930670261383,0.37660637497901917,0.16876700520515442,0.65
attn_11_15,11,cal-zh-2,3,0.09019850194454193,0.3149072527885437,0.09793759882450104,31,0.1334252804517746,0.24160735309123993,0.13772933185100555,0.71875
attn_11_15,11,cal-zh-2,4,0.09969134628772736,0.20464956760406494,0.10986322164535522,31,0.11441972851753235,0.2605915665626526,0.11811067909002304,0.7
attn_11_15,11,cal-zh-2,5,0.16705867648124695,0.412482887506485,0.20107370615005493,31,0.21316100656986237,0.44814038276672363,0.22003716230392456,0.6000000000000001
attn_11_15,11,cal-zh-2,6,0.126181960105896,0.38109123706817627,0.10754827409982681,31,0.14080098271369934,0.2554462254047394,0.14534294605255127,0.6687500000000001
attn_11_15,11,cal-zh-2,7,0.11085975170135498,0.5181387662887573,0.10885247588157654,31,0.2060571014881134,0.35480910539627075,0.2127041071653366,0.66875
attn_11_15,11,cal-en-1,0,0.2882137894630432,0.45660895109176636,0.07201986759901047,31,0.08352888375520706,0.1487027406692505,0.08622336387634277,0.73125
attn_11_15,11,cal-en-1,1,0.597124457359314,0.13483837246894836,0.22424918413162231,31,0.29736971855163574,0.529407799243927,0.3069622814655304,0.51875
attn_11_15,11,cal-en-1,2,0.17265087366104126,0.4521681070327759,0.08010748028755188,31,0.13888895511627197,0.29661378264427185,0.14336924254894257,0.7375
attn_11_15,11,cal-en-1,3,0.10834433138370514,0.319663405418396,0.08455760031938553,31,0.11557236313819885,0.24554768204689026,0.11930052191019058,0.84375
attn_11_15,11,cal-en-1,4,0.157099187374115,0.3456929326057434,0.10397545993328094,31,0.11958309262990952,0.20836150646209717,0.12344060838222504,0.73125
attn_11_15,11,cal-en-1,5,0.30003029108047485,0.3683816194534302,0.11273742467164993,31,0.11081759631633759,0.26781079173088074,0.11439235508441925,0.6875
attn_11_15,11,cal-en-1,6,0.16862526535987854,0.5013703107833862,0.12122618407011032,31,0.1663668006658554,0.29310786724090576,0.17173348367214203,0.65
attn_11_15,11,cal-en-1,7,0.14973583817481995,0.5537562966346741,0.1512605994939804,31,0.22106251120567322,0.3912945091724396,0.22819358110427856,0.64375
attn_11_15,11,cal-en-2,0,0.3351830840110779,0.4993361234664917,0.08562158793210983,31,0.08981436491012573,0.19977226853370667,0.09271162003278732,0.675
attn_11_15,11,cal-en-2,1,0.5792163610458374,0.1499037742614746,0.1789509803056717,31,0.2536986768245697,0.41500288248062134,0.2618825435638428,0.55
attn_11_15,11,cal-en-2,2,0.20357520878314972,0.4823607802391052,0.11572320759296417,31,0.16091257333755493,0.35024091601371765,0.1661033034324646,0.6000000000000001
attn_11_15,11,cal-en-2,3,0.11477383971214294,0.4060356318950653,0.08803554624319077,31,0.14013110101222992,0.2660136818885803,0.14465147256851196,0.75
attn_11_15,11,cal-en-2,4,0.1895177811384201,0.2564787268638611,0.1033802479505539,31,0.10677468031644821,0.19931447505950928,0.11021901667118073,0.7125
attn_11_15,11,cal-en-2,5,0.2725520133972168,0.3527078926563263,0.1301337331533432,31,0.12496069073677063,0.22729209065437317,0.12899169325828552,0.6312500000000001
attn_11_15,11,cal-en-2,6,0.17815515398979187,0.4918685555458069,0.08680415898561478,31,0.1395310014486313,0.22975514829158783,0.14403200149536133,0.68125
attn_11_15,11,cal-en-2,7,0.1471988707780838,0.4305167496204376,0.15007153153419495,31,0.1933930218219757,0.31820762157440186,0.1996314972639084,0.60625
attn_11_15,15,cal-zh-1,0,0.20194272696971893,0.35520243644714355,0.1205381229519844,31,0.13211002945899963,0.2777846157550812,0.1363716572523117,0.63125
attn_11_15,15,cal-zh-1,1,0.23890593647956848,0.12377866357564926,0.21396537125110626,31,0.21463359892368317,0.41781342029571533,0.22155724465847015,0.6062500000000001
attn_11_15,15,cal-zh-1,2,0.13280686736106873,0.27719253301620483,0.1377590000629425,31,0.14551271498203278,0.2994697093963623,0.15020668506622314,0.65
attn_11_15,15,cal-zh-1,3,0.04135940596461296,0.23285867273807526,0.21063239872455597,31,0.24926505982875824,0.45159438252449036,0.2573058605194092,0.6937500000000001
attn_11_15,15,cal-zh-1,4,0.07648462802171707,0.4026046395301819,0.1620333045721054,31,0.23594452440738678,0.4158128798007965,0.24355560541152954,0.6500000000000001
attn_11_15,15,cal-zh-1,5,0.14696139097213745,0.41105949878692627,0.1549629122018814,31,0.18883343040943146,0.3439426124095917,0.19492481648921967,0.6187499999999999
attn_11_15,15,cal-zh-1,6,0.0764666199684143,0.34140151739120483,0.11151057481765747,31,0.16509875655174255,0.31560462713241577,0.17042450606822968,0.6625
attn_11_15,15,cal-zh-1,7,0.10250869393348694,0.5306729078292847,0.16751603782176971,31,0.25549793243408203,0.3858420252799988,0.26373982429504395,0.6062500000000001
attn_11_15,15,cal-zh-2,0,0.22696512937545776,0.35936564207077026,0.14069660007953644,31,0.13378489017486572,0.28451812267303467,0.1381005346775055,0.65
attn_11_15,15,cal-zh-2,1,0.25547248125076294,0.10696703940629959,0.23399728536605835,31,0.24166268110275269,0.39466142654418945,0.24945826828479767,0.5875
attn_11_15,15,cal-zh-2,2,0.17822568118572235,0.2864110469818115,0.08877820521593094,31,0.10718775540590286,0.2511947453022003,0.11064542829990387,0.6937500000000001
attn_11_15,15,cal-zh-2,3,0.04604611545801163,0.23604938387870789,0.22704336047172546,31,0.2641073167324066,0.4154779016971588,0.2726269066333771,0.70625
attn_11_15,15,cal-zh-2,4,0.08757959306240082,0.2918083965778351,0.22551555931568146,31,0.24939756095409393,0.48551467061042786,0.2574426531791687,0.6000000000000001
attn_11_15,15,cal-zh-2,5,0.11609973013401031,0.3603805601596832,0.21242640912532806,31,0.23965227603912354,0.4984361231327057,0.2473829984664917,0.51875
attn_11_15,15,cal-zh-2,6,0.0702231377363205,0.29952526092529297,0.12570251524448395,31,0.17031171917915344,0.32249733805656433,0.17580564320087433,0.6125
attn_11_15,15,cal-zh-2,7,0.0856853500008583,0.4777143895626068,0.1792273223400116,31,0.27432966232299805,0.47331610321998596,0.28317901492118835,0.65625
attn_11_15,15,cal-en-1,0,0.2137582153081894,0.36318275332450867,0.121327243745327,31,0.12573851644992828,0.2549639940261841,0.12979458272457123,0.64375
attn_11_15,15,cal-en-1,1,0.31107524037361145,0.1114300861954689,0.24769999086856842,31,0.28352850675582886,0.4685184955596924,0.29267460107803345,0.53125
attn_11_15,15,cal-en-1,2,0.17990511655807495,0.29312068223953247,0.12414757162332535,31,0.12930452823638916,0.3116541802883148,0.1334756314754486,0.74375
attn_11_15,15,cal-en-1,3,0.048217516392469406,0.23240914940834045,0.20335303246974945,31,0.2404388189315796,0.3818458914756775,0.2481948882341385,0.70625
attn_11_15,15,cal-en-1,4,0.10281141847372055,0.38790467381477356,0.18024703860282898,31,0.22299107909202576,0.4459393620491028,0.23018434643745422,0.70625
attn_11_15,15,cal-en-1,5,0.15091386437416077,0.3693487346172333,0.1916365921497345,31,0.21596358716487885,0.39266589283943176,0.2229301631450653,0.53125
attn_11_15,15,cal-en-1,6,0.0790158361196518,0.32175618410110474,0.11456245183944702,31,0.16404178738594055,0.30913954973220825,0.16933345794677734,0.6875
attn_11_15,15,cal-en-1,7,0.10605430603027344,0.5771000385284424,0.18373095989227295,31,0.28234609961509705,0.4291541278362274,0.2914540469646454,0.625
attn_11_15,15,cal-en-2,0,0.251964807510376,0.31455427408218384,0.10559824854135513,31,0.11457027494907379,0.24910703301429749,0.11826609075069427,0.675
attn_11_15,15,cal-en-2,1,0.31838059425354004,0.11868949234485626,0.257511168718338,31,0.2719646394252777,0.44276586174964905,0.2807376980781555,0.5687500000000001
attn_11_15,15,cal-en-2,2,0.26772984862327576,0.33754318952560425,0.0924711525440216,31,0.09108234941959381,0.191604882478714,0.09402050077915192,0.7375
attn_11_15,15,cal-en-2,3,0.06395735591650009,0.2933264374732971,0.20108558237552643,31,0.2424471080303192,0.4563605785369873,0.25026798248291016,0.66875
attn_11_15,15,cal-en-2,4,0.08209535479545593,0.3278740644454956,0.1982748657464981,31,0.24514394998550415,0.3975651264190674,0.2530518174171448,0.56875
attn_11_15,15,cal-en-2,5,0.09166160225868225,0.44504809379577637,0.17847460508346558,31,0.2476060837507248,0.3910382390022278,0.2555933892726898,0.4875
attn_11_15,15,cal-en-2,6,0.07513932138681412,0.45416805148124695,0.12279381603002548,31,0.22479453682899475,0.34578394889831543,0.2320459932088852,0.59375
attn_11_15,15,cal-en-2,7,0.1007680892944336,0.5731154084205627,0.18980540335178375,31,0.3043324649333954,0.41584962606430054,0.3141496479511261,0.5375000000000001
attn_19_23,19,cal-zh-1,0,0.0955502837896347,0.03435931354761124,0.08661527186632156,31,0.10083474218845367,0.1792067289352417,0.10408748686313629,0.73125
attn_19_23,19,cal-zh-1,1,0.10103033483028412,0.04235740005970001,0.16347354650497437,31,0.16801373660564423,0.37647515535354614,0.17343351244926453,0.6125
attn_19_23,19,cal-zh-1,2,0.047133591026067734,0.03135885298252106,0.47210854291915894,31,0.46042412519454956,0.6502387523651123,0.47527655959129333,0.5375000000000001
attn_19_23,19,cal-zh-1,3,0.07069262862205505,0.06008598208427429,0.12148955464363098,31,0.11885681003332138,0.26471832394599915,0.12269090116024017,0.6625000000000001
attn_19_23,19,cal-zh-1,4,0.10007724165916443,0.044495031237602234,0.16073106229305267,31,0.16553831100463867,0.3016175925731659,0.17087827622890472,0.68125
attn_19_23,19,cal-zh-1,5,0.06431633979082108,0.040679484605789185,0.0816011056303978,31,0.08355634659528732,0.1771341860294342,0.08625171333551407,0.8
attn_19_23,19,cal-zh-1,6,0.08145896345376968,0.03135719522833824,0.4502938687801361,31,0.44716787338256836,0.6495819091796875,0.4615926444530487,0.5
attn_19_23,19,cal-zh-1,7,0.08670078217983246,0.03888973593711853,0.09126521646976471,31,0.09830940514802933,0.18466481566429138,0.10148067027330399,0.70625
attn_19_23,19,cal-zh-2,0,0.06336566060781479,0.03736908361315727,0.07764910906553268,31,0.08064413070678711,0.16660116612911224,0.08324555307626724,0.8187500000000001
attn_19_23,19,cal-zh-2,1,0.0888693630695343,0.06530671566724777,0.1942276507616043,31,0.18916305899620056,0.33451300859451294,0.19526511430740356,0.6125
attn_19_23,19,cal-zh-2,2,0.07010967284440994,0.031316906213760376,0.3875632584095001,31,0.3861002027988434,0.6248269081115723,0.3985550105571747,0.65
attn_19_23,19,cal-zh-2,3,0.08096092939376831,0.08616353571414948,0.08690906316041946,31,0.08501926809549332,0.19463855028152466,0.08776183426380157,0.7937500000000001
attn_19_23,19,cal-zh-2,4,0.0936945378780365,0.04400797560811043,0.11036926507949829,31,0.11584575474262238,0.2122059315443039,0.11958272755146027,0.8187500000000001
attn_19_23,19,cal-zh-2,5,0.056567154824733734,0.03972743824124336,0.06345413625240326,31,0.06394530087709427,0.16273385286331177,0.066008061170578,0.7937500000000001
attn_19_23,19,cal-zh-2,6,0.08575102686882019,0.031342219561338425,0.4025454819202423,31,0.4032740592956543,0.5980108976364136,0.4162829518318176,0.60625
attn_19_23,19,cal-zh-2,7,0.10350999981164932,0.0646558329463005,0.10814103484153748,31,0.10938012599945068,0.23185187578201294,0.11290851980447769,0.7375
attn_19_23,19,cal-en-1,0,0.22517549991607666,0.09088313579559326,0.11986515671014786,31,0.1366778463125229,0.2391858845949173,0.14108680188655853,0.74375
attn_19_23,19,cal-en-1,1,0.23973168432712555,0.11891130357980728,0.17766058444976807,31,0.16800963878631592,0.2676515281200409,0.17342931032180786,0.66875
attn_19_23,19,cal-en-1,2,0.21452879905700684,0.03307320922613144,0.3807314932346344,31,0.4102427363395691,0.5871645212173462,0.42347636818885803,0.66875
attn_19_23,19,cal-en-1,3,0.21709948778152466,0.14377164840698242,0.0770971029996872,31,0.08144749701023102,0.15173088014125824,0.08407483249902725,0.8125
attn_19_23,19,cal-en-1,4,0.23791193962097168,0.19146007299423218,0.12041598558425903,31,0.11442764848470688,0.22197961807250977,0.11811887472867966,0.66875
attn_19_23,19,cal-en-1,5,0.2485853135585785,0.10372771322727203,0.08068431913852692,31,0.11313603073358536,0.23292472958564758,0.11678558588027954,0.7562500000000001
attn_19_23,19,cal-en-1,6,0.31094634532928467,0.031405214220285416,0.39337676763534546,31,0.45084744691848755,0.6603637933731079,0.4653908908367157,0.55
attn_19_23,19,cal-en-1,7,0.2927525043487549,0.15012584626674652,0.12347250431776047,31,0.12942291796207428,0.2467067539691925,0.13359786570072174,0.68125
attn_19_23,19,cal-en-2,0,0.19307208061218262,0.055295493453741074,0.1307787448167801,31,0.15591834485530853,0.3169764280319214,0.1609479784965515,0.5875
attn_19_23,19,cal-en-2,1,0.1491451859474182,0.10034040361642838,0.14188656210899353,31,0.1398725062608719,0.2532970607280731,0.14438451826572418,0.59375
attn_19_23,19,cal-en-2,2,0.11011949181556702,0.033411312848329544,0.4344772696495056,31,0.4410479664802551,0.6305621266365051,0.4552752673625946,0.5375
attn_19_23,19,cal-en-2,3,0.11554233729839325,0.08585710823535919,0.1385604739189148,31,0.13558831810951233,0.3369371294975281,0.13996215164661407,0.5874999999999999
attn_19_23,19,cal-en-2,4,0.1721733808517456,0.10996092110872269,0.1335446536540985,31,0.13328826427459717,0.24826908111572266,0.13758789002895355,0.625
attn_19_23,19,cal-en-2,5,0.18817278742790222,0.06915841996669769,0.11790769547224045,31,0.13643309473991394,0.2266169786453247,0.14083418250083923,0.65625
attn_19_23,19,cal-en-2,6,0.2252667248249054,0.03137722611427307,0.45010486245155334,31,0.4878306984901428,0.6629999876022339,0.503567099571228,0.49375
attn_19_23,19,cal-en-2,7,0.19858212769031525,0.07953452318906784,0.1386381983757019,31,0.15347915887832642,0.2583900988101959,0.15843011438846588,0.5687500000000001
attn_19_23,23,cal-zh-1,0,0.18854272365570068,0.06906979531049728,0.08094117045402527,31,0.10190774500370026,0.19865523278713226,0.10519509017467499,0.61875
attn_19_23,23,cal-zh-1,1,0.12637057900428772,0.094294473528862,0.11472687125205994,31,0.11000899970531464,0.20533926784992218,0.11355767399072647,0.60625
attn_19_23,23,cal-zh-1,2,0.06566040217876434,0.0320066399872303,0.3936012387275696,31,0.3894338607788086,0.6095475554466248,0.40199628472328186,0.53125
attn_19_23,23,cal-zh-1,3,0.10236738622188568,0.07628713548183441,0.0909285917878151,31,0.09045256674289703,0.16597077250480652,0.09337039291858673,0.65625
attn_19_23,23,cal-zh-1,4,0.11772166937589645,0.09994609653949738,0.11158378422260284,31,0.1082884818315506,0.2333190143108368,0.11178164929151535,0.6000000000000001
attn_19_23,23,cal-zh-1,5,0.1445631980895996,0.05731264501810074,0.06436286121606827,31,0.07913714647293091,0.15170255303382874,0.08168993890285492,0.6625000000000001
attn_19_23,23,cal-zh-1,6,0.13692563772201538,0.03156186267733574,0.42265841364860535,31,0.43705910444259644,0.6045969128608704,0.45115774869918823,0.5249999999999999
attn_19_23,23,cal-zh-1,7,0.17913174629211426,0.0774894654750824,0.055066999047994614,31,0.07003086805343628,0.11456935852766037,0.07228992879390717,0.6875
attn_19_23,23,cal-zh-2,0,0.08711955696344376,0.08616629242897034,0.09776511043310165,31,0.09617017209529877,0.2418135702610016,0.09927243739366531,0.7125
attn_19_23,23,cal-zh-2,1,0.1001269742846489,0.1142689436674118,0.11429562419652939,31,0.11118914932012558,0.19028690457344055,0.1147758960723877,0.6625000000000001
attn_19_23,23,cal-zh-2,2,0.13165870308876038,0.03296923264861107,0.32159483432769775,31,0.34172311425209045,0.5507174134254456,0.35274645686149597,0.675
attn_19_23,23,cal-zh-2,3,0.10557965934276581,0.08546337485313416,0.10869138687849045,31,0.10796000063419342,0.267814040184021,0.11144257336854935,0.71875
attn_19_23,23,cal-zh-2,4,0.0753588005900383,0.0625741258263588,0.09596671164035797,31,0.09504459798336029,0.18062597513198853,0.09811054915189743,0.6937500000000001
attn_19_23,23,cal-zh-2,5,0.0798945426940918,0.0568334236741066,0.07381810247898102,31,0.0762210339307785,0.13966575264930725,0.07867977768182755,0.74375
attn_19_23,23,cal-zh-2,6,0.0759878009557724,0.031536251306533813,0.39428046345710754,31,0.3959207534790039,0.5929224491119385,0.408692330121994,0.6375
attn_19_23,23,cal-zh-2,7,0.11138643324375153,0.07136214524507523,0.07186198234558105,31,0.07652410864830017,0.15722090005874634,0.07899262756109238,0.7312500000000001
attn_19_23,23,cal-en-1,0,0.12444712221622467,0.0752042829990387,0.10921256989240646,31,0.10872898995876312,0.22416618466377258,0.11223636567592621,0.6875
attn_19_23,23,cal-en-1,1,0.10719993710517883,0.08359251916408539,0.12698784470558167,31,0.12113931775093079,0.22020384669303894,0.12504702806472778,0.7000000000000001
attn_19_23,23,cal-en-1,2,0.13445816934108734,0.03345245122909546,0.3665089011192322,31,0.3763067126274109,0.5767470002174377,0.3884456157684326,0.65625
attn_19_23,23,cal-en-1,3,0.10536500066518784,0.07050738483667374,0.07684175670146942,31,0.07960161566734314,0.14911630749702454,0.08216940611600876,0.78125
attn_19_23,23,cal-en-1,4,0.0911213681101799,0.17202866077423096,0.10298387706279755,31,0.10929879546165466,0.20484577119350433,0.11282456666231155,0.61875
attn_19_23,23,cal-en-1,5,0.18703660368919373,0.12409132719039917,0.09463418275117874,31,0.09024524688720703,0.15500988066196442,0.09315639734268188,0.70625
attn_19_23,23,cal-en-1,6,0.13999314606189728,0.03152035176753998,0.4262007474899292,31,0.43630746006965637,0.6232555508613586,0.450381875038147,0.5187499999999999
attn_19_23,23,cal-en-1,7,0.11322036385536194,0.10936357825994492,0.07323051989078522,31,0.07213389873504639,0.1506829559803009,0.07446079701185226,0.6125
attn_19_23,23,cal-en-2,0,0.11863207072019577,0.13078393042087555,0.09141960740089417,31,0.08733218908309937,0.1577986180782318,0.09014936536550522,0.59375
attn_19_23,23,cal-en-2,1,0.07288858294487,0.1372600495815277,0.1089932844042778,31,0.11363835632801056,0.1698836386203766,0.11730410903692245,0.50625
attn_19_23,23,cal-en-2,2,0.07288709282875061,0.04088134318590164,0.376287043094635,31,0.3685654401779175,0.5480253100395203,0.3804546594619751,0.5375
attn_19_23,23,cal-en-2,3,0.06728185713291168,0.07526300847530365,0.09231049567461014,31,0.08894366025924683,0.15198129415512085,0.09181280434131622,0.6187499999999999
attn_19_23,23,cal-en-2,4,0.09862713515758514,0.12817975878715515,0.09404788166284561,31,0.08950259536504745,0.16771778464317322,0.09238976985216141,0.5562499999999999
attn_19_23,23,cal-en-2,5,0.07161959260702133,0.0855005532503128,0.10532733052968979,31,0.10144606977701187,0.1908321976661682,0.10471852868795395,0.5625
attn_19_23,23,cal-en-2,6,0.08469876646995544,0.03168217092752457,0.4412229359149933,31,0.43726012110710144,0.6215424537658691,0.45136526226997375,0.5437500000000001
attn_19_23,23,cal-en-2,7,0.10539951920509338,0.08276306837797165,0.07327506691217422,31,0.0738329142332077,0.13249149918556213,0.07621461898088455,0.63125
1 candidate source sample_id head first_key_mass_a first_key_mass_b js_without_first_key without_first_key_valid_rows js_mean js_p95 js_without_first_query top5_overlap
2 attn_3_7 3 cal-zh-1 0 0.178242027759552 0.44051653146743774 0.2887135446071625 31 0.2728434205055237 0.42193013429641724 0.2816448211669922 0.5875
3 attn_3_7 3 cal-zh-1 1 0.14070242643356323 0.6567835807800293 0.154605433344841 31 0.273810476064682 0.45863786339759827 0.2826431095600128 0.5375
4 attn_3_7 3 cal-zh-1 2 0.11500269174575806 0.4787790775299072 0.3218191862106323 31 0.35086748003959656 0.46183231472969055 0.3621857762336731 0.53125
5 attn_3_7 3 cal-zh-1 3 0.17224811017513275 0.6523524522781372 0.22310931980609894 31 0.28077495098114014 0.43149834871292114 0.289832204580307 0.5
6 attn_3_7 3 cal-zh-1 4 0.11534158885478973 0.6260660886764526 0.19559672474861145 31 0.2952086329460144 0.470927894115448 0.30473145842552185 0.53125
7 attn_3_7 3 cal-zh-1 5 0.0896417498588562 0.34022626280784607 0.22164911031723022 31 0.2483729124069214 0.3841992914676666 0.256384938955307 0.54375
8 attn_3_7 3 cal-zh-1 6 0.1247040405869484 0.7327346801757812 0.26632776856422424 31 0.37061089277267456 0.5524277091026306 0.38256609439849854 0.525
9 attn_3_7 3 cal-zh-1 7 0.05083201080560684 0.44692301750183105 0.04695317894220352 31 0.20246444642543793 0.4956357181072235 0.20899559557437897 0.65625
10 attn_3_7 3 cal-zh-2 0 0.14007353782653809 0.6036521196365356 0.2504415214061737 31 0.30694448947906494 0.4879894554615021 0.31684595346450806 0.59375
11 attn_3_7 3 cal-zh-2 1 0.14963164925575256 0.7174584865570068 0.16050270199775696 31 0.2995006740093231 0.44737952947616577 0.3091619610786438 0.58125
12 attn_3_7 3 cal-zh-2 2 0.1916198432445526 0.6128076314926147 0.24314163625240326 31 0.28848910331726074 0.42006126046180725 0.29779520630836487 0.63125
13 attn_3_7 3 cal-zh-2 3 0.11550820618867874 0.6839271783828735 0.22851790487766266 31 0.3617430627346039 0.4926488697528839 0.373412162065506 0.43125
14 attn_3_7 3 cal-zh-2 4 0.1432671993970871 0.714910089969635 0.16698074340820312 31 0.29421237111091614 0.44453173875808716 0.30370306968688965 0.55
15 attn_3_7 3 cal-zh-2 5 0.15627557039260864 0.4676961600780487 0.22456194460391998 31 0.24842627346515656 0.35863032937049866 0.25644001364707947 0.53125
16 attn_3_7 3 cal-zh-2 6 0.10968590527772903 0.8703562021255493 0.2677042484283447 31 0.44428783655166626 0.6032188534736633 0.45861977338790894 0.4625
17 attn_3_7 3 cal-zh-2 7 0.07148394733667374 0.5658342838287354 0.050552818924188614 31 0.24003271758556366 0.5258023738861084 0.24777570366859436 0.6875
18 attn_3_7 3 cal-en-1 0 0.11021165549755096 0.7443452477455139 0.2832939028739929 31 0.4321710467338562 0.5805379152297974 0.4461120665073395 0.5
19 attn_3_7 3 cal-en-1 1 0.14157001674175262 0.6908999681472778 0.15555045008659363 31 0.3069734573364258 0.46220970153808594 0.3168758153915405 0.5562499999999999
20 attn_3_7 3 cal-en-1 2 0.22632445394992828 0.6502443552017212 0.3039384186267853 31 0.33335912227630615 0.4760979115962982 0.34411266446113586 0.575
21 attn_3_7 3 cal-en-1 3 0.1636783331632614 0.749056875705719 0.18588410317897797 31 0.31960076093673706 0.4827522933483124 0.3299104571342468 0.5062500000000001
22 attn_3_7 3 cal-en-1 4 0.18975511193275452 0.5745540857315063 0.20487342774868011 31 0.2522231340408325 0.39599278569221497 0.26035937666893005 0.60625
23 attn_3_7 3 cal-en-1 5 0.1436786651611328 0.45787984132766724 0.1883384883403778 31 0.2209610939025879 0.3852173388004303 0.2280888706445694 0.6187499999999999
24 attn_3_7 3 cal-en-1 6 0.08422182500362396 0.8810427784919739 0.24220214784145355 31 0.4833376407623291 0.6262145638465881 0.49892911314964294 0.45625
25 attn_3_7 3 cal-en-1 7 0.09002429246902466 0.49231916666030884 0.053437694907188416 31 0.19592277705669403 0.4695172905921936 0.20224282145500183 0.6624999999999999
26 attn_3_7 3 cal-en-2 0 0.09620732069015503 0.7015095949172974 0.2738996148109436 31 0.3950675129890442 0.5772150158882141 0.40781161189079285 0.5874999999999999
27 attn_3_7 3 cal-en-2 1 0.1168593019247055 0.7301980257034302 0.160212442278862 31 0.3232686221599579 0.4884706735610962 0.3336966037750244 0.55625
28 attn_3_7 3 cal-en-2 2 0.21824826300144196 0.655203104019165 0.3244048058986664 31 0.3314599394798279 0.489634245634079 0.3421522378921509 0.49375
29 attn_3_7 3 cal-en-2 3 0.15237370133399963 0.7099946737289429 0.253733366727829 31 0.34891438484191895 0.4875939190387726 0.3601696491241455 0.4
30 attn_3_7 3 cal-en-2 4 0.20830751955509186 0.6707655191421509 0.16886863112449646 31 0.23844638466835022 0.41475459933280945 0.24613820016384125 0.58125
31 attn_3_7 3 cal-en-2 5 0.20089492201805115 0.4360402226448059 0.20298565924167633 31 0.19701015949249268 0.3117577135562897 0.20336531102657318 0.5875
32 attn_3_7 3 cal-en-2 6 0.08693217486143112 0.8322523832321167 0.28768372535705566 31 0.45854610204696655 0.604308545589447 0.47333794832229614 0.4625
33 attn_3_7 3 cal-en-2 7 0.12079077959060669 0.5237122178077698 0.08786331117153168 31 0.1976630836725235 0.4211302399635315 0.2040393203496933 0.71875
34 attn_3_7 7 cal-zh-1 0 0.08316324651241302 0.1214274913072586 0.22053609788417816 31 0.2072795033454895 0.32793501019477844 0.21396592259407043 0.5625
35 attn_3_7 7 cal-zh-1 1 0.11647684127092361 0.187703475356102 0.09203154593706131 31 0.09238047897815704 0.16611215472221375 0.09536049515008926 0.66875
36 attn_3_7 7 cal-zh-1 2 0.06550130248069763 0.07431766390800476 0.28803399205207825 31 0.2721514105796814 0.46315261721611023 0.28093045949935913 0.47500000000000003
37 attn_3_7 7 cal-zh-1 3 0.10703553259372711 0.24631237983703613 0.1426287740468979 31 0.150299072265625 0.26700741052627563 0.1551474630832672 0.56875
38 attn_3_7 7 cal-zh-1 4 0.04052526503801346 0.28515881299972534 0.11374201625585556 31 0.1819339096546173 0.2913167476654053 0.1878027319908142 0.5875
39 attn_3_7 7 cal-zh-1 5 0.03951618820428848 0.12756675481796265 0.12323599308729172 31 0.13915526866912842 0.22263947129249573 0.14364413917064667 0.6062500000000001
40 attn_3_7 7 cal-zh-1 6 0.09479819983243942 0.28205406665802 0.14449526369571686 31 0.16065017879009247 0.30267569422721863 0.16583245992660522 0.6375000000000001
41 attn_3_7 7 cal-zh-1 7 0.031424712389707565 0.10597768425941467 0.029422910884022713 31 0.05383709818124771 0.10532202571630478 0.055573780089616776 0.60625
42 attn_3_7 7 cal-zh-2 0 0.0902467742562294 0.11627861112356186 0.21221575140953064 31 0.19495421648025513 0.317206472158432 0.201243057847023 0.6187499999999999
43 attn_3_7 7 cal-zh-2 1 0.11616441607475281 0.1248721182346344 0.0952492356300354 31 0.08693543076515198 0.16702036559581757 0.08973980695009232 0.5874999999999999
44 attn_3_7 7 cal-zh-2 2 0.11578349769115448 0.08946777880191803 0.2738611102104187 31 0.2561575174331665 0.45070335268974304 0.26442068815231323 0.5
45 attn_3_7 7 cal-zh-2 3 0.11586448550224304 0.16896863281726837 0.10842260718345642 31 0.10158580541610718 0.21354182064533234 0.1048627570271492 0.60625
46 attn_3_7 7 cal-zh-2 4 0.044329218566417694 0.22534678876399994 0.15060655772686005 31 0.19093912839889526 0.3356643319129944 0.19709844887256622 0.61875
47 attn_3_7 7 cal-zh-2 5 0.05062376335263252 0.12924723327159882 0.1047384962439537 31 0.11341066658496857 0.2176104635000229 0.1170690730214119 0.6312500000000001
48 attn_3_7 7 cal-zh-2 6 0.08418147265911102 0.28032630681991577 0.1446688026189804 31 0.16730237007141113 0.24755442142486572 0.1726992279291153 0.5125
49 attn_3_7 7 cal-zh-2 7 0.0315190926194191 0.11244469881057739 0.02086171880364418 31 0.04779163748025894 0.10416773706674576 0.04933330416679382 0.65
50 attn_3_7 7 cal-en-1 0 0.08185911923646927 0.10868857800960541 0.24351894855499268 31 0.22597578167915344 0.35052549839019775 0.23326534032821655 0.66875
51 attn_3_7 7 cal-en-1 1 0.12180840969085693 0.12996530532836914 0.1148008182644844 31 0.10937604308128357 0.1990041583776474 0.11290428787469864 0.6625
52 attn_3_7 7 cal-en-1 2 0.12156520038843155 0.07304315268993378 0.32663044333457947 31 0.31218329071998596 0.4966602325439453 0.3222537338733673 0.4375
53 attn_3_7 7 cal-en-1 3 0.09531049430370331 0.21274563670158386 0.13099102675914764 31 0.14007462561130524 0.2635233998298645 0.14459317922592163 0.5874999999999999
54 attn_3_7 7 cal-en-1 4 0.051348961889743805 0.27256035804748535 0.17027397453784943 31 0.22669382393360138 0.33905521035194397 0.23400652408599854 0.59375
55 attn_3_7 7 cal-en-1 5 0.05210518091917038 0.14343655109405518 0.11446496844291687 31 0.12723194062709808 0.23429512977600098 0.13133619725704193 0.64375
56 attn_3_7 7 cal-en-1 6 0.09278014302253723 0.3040335178375244 0.20964911580085754 31 0.21833369135856628 0.34637004137039185 0.22537672519683838 0.53125
57 attn_3_7 7 cal-en-1 7 0.031563661992549896 0.11319160461425781 0.02739104814827442 31 0.053969405591487885 0.1002146303653717 0.055710360407829285 0.5625000000000001
58 attn_3_7 7 cal-en-2 0 0.07943743467330933 0.1426258534193039 0.21740850806236267 31 0.21001917123794556 0.35335251688957214 0.2167939990758896 0.58125
59 attn_3_7 7 cal-en-2 1 0.13507506251335144 0.15443003177642822 0.10350806266069412 31 0.09826720505952835 0.18229186534881592 0.10143710672855377 0.575
60 attn_3_7 7 cal-en-2 2 0.07290980964899063 0.08657881617546082 0.26718389987945557 31 0.259122759103775 0.4706645905971527 0.26748156547546387 0.45625
61 attn_3_7 7 cal-en-2 3 0.06336543709039688 0.2253912389278412 0.11350395530462265 31 0.1445191204547882 0.30314192175865173 0.1491810381412506 0.5875
62 attn_3_7 7 cal-en-2 4 0.04387465864419937 0.2312193661928177 0.14195290207862854 31 0.1880926638841629 0.3466520309448242 0.19416013360023499 0.5125
63 attn_3_7 7 cal-en-2 5 0.05407806113362312 0.1308087259531021 0.15231285989284515 31 0.16046105325222015 0.3131849467754364 0.16563721001148224 0.5874999999999999
64 attn_3_7 7 cal-en-2 6 0.10013270378112793 0.30412447452545166 0.17993809282779694 31 0.19388216733932495 0.34072187542915344 0.2001364380121231 0.5437500000000001
65 attn_3_7 7 cal-en-2 7 0.03161787986755371 0.14472994208335876 0.04679951071739197 31 0.0830649882555008 0.13866016268730164 0.08574451506137848 0.65625
66 attn_11_15 11 cal-zh-1 0 0.2780076861381531 0.40727436542510986 0.08935041725635529 31 0.08769044280052185 0.1605687141418457 0.09051916748285294 0.6937500000000001
67 attn_11_15 11 cal-zh-1 1 0.599311351776123 0.1917644888162613 0.1712544858455658 31 0.22863143682479858 0.39283493161201477 0.23600666224956512 0.5750000000000001
68 attn_11_15 11 cal-zh-1 2 0.15545204281806946 0.5101379156112671 0.08800674974918365 31 0.15655069053173065 0.3642641007900238 0.16160069406032562 0.73125
69 attn_11_15 11 cal-zh-1 3 0.11032958328723907 0.4199199676513672 0.08535198122262955 31 0.14290741086006165 0.27922654151916504 0.14751732349395752 0.7
70 attn_11_15 11 cal-zh-1 4 0.18725907802581787 0.3965998888015747 0.12378748506307602 31 0.1280820369720459 0.24588972330093384 0.13221368193626404 0.6937500000000001
71 attn_11_15 11 cal-zh-1 5 0.3307006359100342 0.43053850531578064 0.12539583444595337 31 0.1220257431268692 0.27262142300605774 0.1259620636701584 0.6187499999999999
72 attn_11_15 11 cal-zh-1 6 0.17290356755256653 0.4159870147705078 0.10268668830394745 31 0.12616759538650513 0.27286267280578613 0.13023751974105835 0.625
73 attn_11_15 11 cal-zh-1 7 0.12546084821224213 0.4914926290512085 0.11725502461194992 31 0.17813995480537415 0.28658801317214966 0.18388642370700836 0.6625000000000001
74 attn_11_15 11 cal-zh-2 0 0.2187591940164566 0.42793723940849304 0.09548557549715042 31 0.12088388949632645 0.2345586121082306 0.12478335946798325 0.6875
75 attn_11_15 11 cal-zh-2 1 0.5546900033950806 0.1175224781036377 0.16409580409526825 31 0.25557559728622437 0.37492382526397705 0.26381993293762207 0.6125
76 attn_11_15 11 cal-zh-2 2 0.11856245994567871 0.44309669733047485 0.08432265371084213 31 0.1634930670261383 0.37660637497901917 0.16876700520515442 0.65
77 attn_11_15 11 cal-zh-2 3 0.09019850194454193 0.3149072527885437 0.09793759882450104 31 0.1334252804517746 0.24160735309123993 0.13772933185100555 0.71875
78 attn_11_15 11 cal-zh-2 4 0.09969134628772736 0.20464956760406494 0.10986322164535522 31 0.11441972851753235 0.2605915665626526 0.11811067909002304 0.7
79 attn_11_15 11 cal-zh-2 5 0.16705867648124695 0.412482887506485 0.20107370615005493 31 0.21316100656986237 0.44814038276672363 0.22003716230392456 0.6000000000000001
80 attn_11_15 11 cal-zh-2 6 0.126181960105896 0.38109123706817627 0.10754827409982681 31 0.14080098271369934 0.2554462254047394 0.14534294605255127 0.6687500000000001
81 attn_11_15 11 cal-zh-2 7 0.11085975170135498 0.5181387662887573 0.10885247588157654 31 0.2060571014881134 0.35480910539627075 0.2127041071653366 0.66875
82 attn_11_15 11 cal-en-1 0 0.2882137894630432 0.45660895109176636 0.07201986759901047 31 0.08352888375520706 0.1487027406692505 0.08622336387634277 0.73125
83 attn_11_15 11 cal-en-1 1 0.597124457359314 0.13483837246894836 0.22424918413162231 31 0.29736971855163574 0.529407799243927 0.3069622814655304 0.51875
84 attn_11_15 11 cal-en-1 2 0.17265087366104126 0.4521681070327759 0.08010748028755188 31 0.13888895511627197 0.29661378264427185 0.14336924254894257 0.7375
85 attn_11_15 11 cal-en-1 3 0.10834433138370514 0.319663405418396 0.08455760031938553 31 0.11557236313819885 0.24554768204689026 0.11930052191019058 0.84375
86 attn_11_15 11 cal-en-1 4 0.157099187374115 0.3456929326057434 0.10397545993328094 31 0.11958309262990952 0.20836150646209717 0.12344060838222504 0.73125
87 attn_11_15 11 cal-en-1 5 0.30003029108047485 0.3683816194534302 0.11273742467164993 31 0.11081759631633759 0.26781079173088074 0.11439235508441925 0.6875
88 attn_11_15 11 cal-en-1 6 0.16862526535987854 0.5013703107833862 0.12122618407011032 31 0.1663668006658554 0.29310786724090576 0.17173348367214203 0.65
89 attn_11_15 11 cal-en-1 7 0.14973583817481995 0.5537562966346741 0.1512605994939804 31 0.22106251120567322 0.3912945091724396 0.22819358110427856 0.64375
90 attn_11_15 11 cal-en-2 0 0.3351830840110779 0.4993361234664917 0.08562158793210983 31 0.08981436491012573 0.19977226853370667 0.09271162003278732 0.675
91 attn_11_15 11 cal-en-2 1 0.5792163610458374 0.1499037742614746 0.1789509803056717 31 0.2536986768245697 0.41500288248062134 0.2618825435638428 0.55
92 attn_11_15 11 cal-en-2 2 0.20357520878314972 0.4823607802391052 0.11572320759296417 31 0.16091257333755493 0.35024091601371765 0.1661033034324646 0.6000000000000001
93 attn_11_15 11 cal-en-2 3 0.11477383971214294 0.4060356318950653 0.08803554624319077 31 0.14013110101222992 0.2660136818885803 0.14465147256851196 0.75
94 attn_11_15 11 cal-en-2 4 0.1895177811384201 0.2564787268638611 0.1033802479505539 31 0.10677468031644821 0.19931447505950928 0.11021901667118073 0.7125
95 attn_11_15 11 cal-en-2 5 0.2725520133972168 0.3527078926563263 0.1301337331533432 31 0.12496069073677063 0.22729209065437317 0.12899169325828552 0.6312500000000001
96 attn_11_15 11 cal-en-2 6 0.17815515398979187 0.4918685555458069 0.08680415898561478 31 0.1395310014486313 0.22975514829158783 0.14403200149536133 0.68125
97 attn_11_15 11 cal-en-2 7 0.1471988707780838 0.4305167496204376 0.15007153153419495 31 0.1933930218219757 0.31820762157440186 0.1996314972639084 0.60625
98 attn_11_15 15 cal-zh-1 0 0.20194272696971893 0.35520243644714355 0.1205381229519844 31 0.13211002945899963 0.2777846157550812 0.1363716572523117 0.63125
99 attn_11_15 15 cal-zh-1 1 0.23890593647956848 0.12377866357564926 0.21396537125110626 31 0.21463359892368317 0.41781342029571533 0.22155724465847015 0.6062500000000001
100 attn_11_15 15 cal-zh-1 2 0.13280686736106873 0.27719253301620483 0.1377590000629425 31 0.14551271498203278 0.2994697093963623 0.15020668506622314 0.65
101 attn_11_15 15 cal-zh-1 3 0.04135940596461296 0.23285867273807526 0.21063239872455597 31 0.24926505982875824 0.45159438252449036 0.2573058605194092 0.6937500000000001
102 attn_11_15 15 cal-zh-1 4 0.07648462802171707 0.4026046395301819 0.1620333045721054 31 0.23594452440738678 0.4158128798007965 0.24355560541152954 0.6500000000000001
103 attn_11_15 15 cal-zh-1 5 0.14696139097213745 0.41105949878692627 0.1549629122018814 31 0.18883343040943146 0.3439426124095917 0.19492481648921967 0.6187499999999999
104 attn_11_15 15 cal-zh-1 6 0.0764666199684143 0.34140151739120483 0.11151057481765747 31 0.16509875655174255 0.31560462713241577 0.17042450606822968 0.6625
105 attn_11_15 15 cal-zh-1 7 0.10250869393348694 0.5306729078292847 0.16751603782176971 31 0.25549793243408203 0.3858420252799988 0.26373982429504395 0.6062500000000001
106 attn_11_15 15 cal-zh-2 0 0.22696512937545776 0.35936564207077026 0.14069660007953644 31 0.13378489017486572 0.28451812267303467 0.1381005346775055 0.65
107 attn_11_15 15 cal-zh-2 1 0.25547248125076294 0.10696703940629959 0.23399728536605835 31 0.24166268110275269 0.39466142654418945 0.24945826828479767 0.5875
108 attn_11_15 15 cal-zh-2 2 0.17822568118572235 0.2864110469818115 0.08877820521593094 31 0.10718775540590286 0.2511947453022003 0.11064542829990387 0.6937500000000001
109 attn_11_15 15 cal-zh-2 3 0.04604611545801163 0.23604938387870789 0.22704336047172546 31 0.2641073167324066 0.4154779016971588 0.2726269066333771 0.70625
110 attn_11_15 15 cal-zh-2 4 0.08757959306240082 0.2918083965778351 0.22551555931568146 31 0.24939756095409393 0.48551467061042786 0.2574426531791687 0.6000000000000001
111 attn_11_15 15 cal-zh-2 5 0.11609973013401031 0.3603805601596832 0.21242640912532806 31 0.23965227603912354 0.4984361231327057 0.2473829984664917 0.51875
112 attn_11_15 15 cal-zh-2 6 0.0702231377363205 0.29952526092529297 0.12570251524448395 31 0.17031171917915344 0.32249733805656433 0.17580564320087433 0.6125
113 attn_11_15 15 cal-zh-2 7 0.0856853500008583 0.4777143895626068 0.1792273223400116 31 0.27432966232299805 0.47331610321998596 0.28317901492118835 0.65625
114 attn_11_15 15 cal-en-1 0 0.2137582153081894 0.36318275332450867 0.121327243745327 31 0.12573851644992828 0.2549639940261841 0.12979458272457123 0.64375
115 attn_11_15 15 cal-en-1 1 0.31107524037361145 0.1114300861954689 0.24769999086856842 31 0.28352850675582886 0.4685184955596924 0.29267460107803345 0.53125
116 attn_11_15 15 cal-en-1 2 0.17990511655807495 0.29312068223953247 0.12414757162332535 31 0.12930452823638916 0.3116541802883148 0.1334756314754486 0.74375
117 attn_11_15 15 cal-en-1 3 0.048217516392469406 0.23240914940834045 0.20335303246974945 31 0.2404388189315796 0.3818458914756775 0.2481948882341385 0.70625
118 attn_11_15 15 cal-en-1 4 0.10281141847372055 0.38790467381477356 0.18024703860282898 31 0.22299107909202576 0.4459393620491028 0.23018434643745422 0.70625
119 attn_11_15 15 cal-en-1 5 0.15091386437416077 0.3693487346172333 0.1916365921497345 31 0.21596358716487885 0.39266589283943176 0.2229301631450653 0.53125
120 attn_11_15 15 cal-en-1 6 0.0790158361196518 0.32175618410110474 0.11456245183944702 31 0.16404178738594055 0.30913954973220825 0.16933345794677734 0.6875
121 attn_11_15 15 cal-en-1 7 0.10605430603027344 0.5771000385284424 0.18373095989227295 31 0.28234609961509705 0.4291541278362274 0.2914540469646454 0.625
122 attn_11_15 15 cal-en-2 0 0.251964807510376 0.31455427408218384 0.10559824854135513 31 0.11457027494907379 0.24910703301429749 0.11826609075069427 0.675
123 attn_11_15 15 cal-en-2 1 0.31838059425354004 0.11868949234485626 0.257511168718338 31 0.2719646394252777 0.44276586174964905 0.2807376980781555 0.5687500000000001
124 attn_11_15 15 cal-en-2 2 0.26772984862327576 0.33754318952560425 0.0924711525440216 31 0.09108234941959381 0.191604882478714 0.09402050077915192 0.7375
125 attn_11_15 15 cal-en-2 3 0.06395735591650009 0.2933264374732971 0.20108558237552643 31 0.2424471080303192 0.4563605785369873 0.25026798248291016 0.66875
126 attn_11_15 15 cal-en-2 4 0.08209535479545593 0.3278740644454956 0.1982748657464981 31 0.24514394998550415 0.3975651264190674 0.2530518174171448 0.56875
127 attn_11_15 15 cal-en-2 5 0.09166160225868225 0.44504809379577637 0.17847460508346558 31 0.2476060837507248 0.3910382390022278 0.2555933892726898 0.4875
128 attn_11_15 15 cal-en-2 6 0.07513932138681412 0.45416805148124695 0.12279381603002548 31 0.22479453682899475 0.34578394889831543 0.2320459932088852 0.59375
129 attn_11_15 15 cal-en-2 7 0.1007680892944336 0.5731154084205627 0.18980540335178375 31 0.3043324649333954 0.41584962606430054 0.3141496479511261 0.5375000000000001
130 attn_19_23 19 cal-zh-1 0 0.0955502837896347 0.03435931354761124 0.08661527186632156 31 0.10083474218845367 0.1792067289352417 0.10408748686313629 0.73125
131 attn_19_23 19 cal-zh-1 1 0.10103033483028412 0.04235740005970001 0.16347354650497437 31 0.16801373660564423 0.37647515535354614 0.17343351244926453 0.6125
132 attn_19_23 19 cal-zh-1 2 0.047133591026067734 0.03135885298252106 0.47210854291915894 31 0.46042412519454956 0.6502387523651123 0.47527655959129333 0.5375000000000001
133 attn_19_23 19 cal-zh-1 3 0.07069262862205505 0.06008598208427429 0.12148955464363098 31 0.11885681003332138 0.26471832394599915 0.12269090116024017 0.6625000000000001
134 attn_19_23 19 cal-zh-1 4 0.10007724165916443 0.044495031237602234 0.16073106229305267 31 0.16553831100463867 0.3016175925731659 0.17087827622890472 0.68125
135 attn_19_23 19 cal-zh-1 5 0.06431633979082108 0.040679484605789185 0.0816011056303978 31 0.08355634659528732 0.1771341860294342 0.08625171333551407 0.8
136 attn_19_23 19 cal-zh-1 6 0.08145896345376968 0.03135719522833824 0.4502938687801361 31 0.44716787338256836 0.6495819091796875 0.4615926444530487 0.5
137 attn_19_23 19 cal-zh-1 7 0.08670078217983246 0.03888973593711853 0.09126521646976471 31 0.09830940514802933 0.18466481566429138 0.10148067027330399 0.70625
138 attn_19_23 19 cal-zh-2 0 0.06336566060781479 0.03736908361315727 0.07764910906553268 31 0.08064413070678711 0.16660116612911224 0.08324555307626724 0.8187500000000001
139 attn_19_23 19 cal-zh-2 1 0.0888693630695343 0.06530671566724777 0.1942276507616043 31 0.18916305899620056 0.33451300859451294 0.19526511430740356 0.6125
140 attn_19_23 19 cal-zh-2 2 0.07010967284440994 0.031316906213760376 0.3875632584095001 31 0.3861002027988434 0.6248269081115723 0.3985550105571747 0.65
141 attn_19_23 19 cal-zh-2 3 0.08096092939376831 0.08616353571414948 0.08690906316041946 31 0.08501926809549332 0.19463855028152466 0.08776183426380157 0.7937500000000001
142 attn_19_23 19 cal-zh-2 4 0.0936945378780365 0.04400797560811043 0.11036926507949829 31 0.11584575474262238 0.2122059315443039 0.11958272755146027 0.8187500000000001
143 attn_19_23 19 cal-zh-2 5 0.056567154824733734 0.03972743824124336 0.06345413625240326 31 0.06394530087709427 0.16273385286331177 0.066008061170578 0.7937500000000001
144 attn_19_23 19 cal-zh-2 6 0.08575102686882019 0.031342219561338425 0.4025454819202423 31 0.4032740592956543 0.5980108976364136 0.4162829518318176 0.60625
145 attn_19_23 19 cal-zh-2 7 0.10350999981164932 0.0646558329463005 0.10814103484153748 31 0.10938012599945068 0.23185187578201294 0.11290851980447769 0.7375
146 attn_19_23 19 cal-en-1 0 0.22517549991607666 0.09088313579559326 0.11986515671014786 31 0.1366778463125229 0.2391858845949173 0.14108680188655853 0.74375
147 attn_19_23 19 cal-en-1 1 0.23973168432712555 0.11891130357980728 0.17766058444976807 31 0.16800963878631592 0.2676515281200409 0.17342931032180786 0.66875
148 attn_19_23 19 cal-en-1 2 0.21452879905700684 0.03307320922613144 0.3807314932346344 31 0.4102427363395691 0.5871645212173462 0.42347636818885803 0.66875
149 attn_19_23 19 cal-en-1 3 0.21709948778152466 0.14377164840698242 0.0770971029996872 31 0.08144749701023102 0.15173088014125824 0.08407483249902725 0.8125
150 attn_19_23 19 cal-en-1 4 0.23791193962097168 0.19146007299423218 0.12041598558425903 31 0.11442764848470688 0.22197961807250977 0.11811887472867966 0.66875
151 attn_19_23 19 cal-en-1 5 0.2485853135585785 0.10372771322727203 0.08068431913852692 31 0.11313603073358536 0.23292472958564758 0.11678558588027954 0.7562500000000001
152 attn_19_23 19 cal-en-1 6 0.31094634532928467 0.031405214220285416 0.39337676763534546 31 0.45084744691848755 0.6603637933731079 0.4653908908367157 0.55
153 attn_19_23 19 cal-en-1 7 0.2927525043487549 0.15012584626674652 0.12347250431776047 31 0.12942291796207428 0.2467067539691925 0.13359786570072174 0.68125
154 attn_19_23 19 cal-en-2 0 0.19307208061218262 0.055295493453741074 0.1307787448167801 31 0.15591834485530853 0.3169764280319214 0.1609479784965515 0.5875
155 attn_19_23 19 cal-en-2 1 0.1491451859474182 0.10034040361642838 0.14188656210899353 31 0.1398725062608719 0.2532970607280731 0.14438451826572418 0.59375
156 attn_19_23 19 cal-en-2 2 0.11011949181556702 0.033411312848329544 0.4344772696495056 31 0.4410479664802551 0.6305621266365051 0.4552752673625946 0.5375
157 attn_19_23 19 cal-en-2 3 0.11554233729839325 0.08585710823535919 0.1385604739189148 31 0.13558831810951233 0.3369371294975281 0.13996215164661407 0.5874999999999999
158 attn_19_23 19 cal-en-2 4 0.1721733808517456 0.10996092110872269 0.1335446536540985 31 0.13328826427459717 0.24826908111572266 0.13758789002895355 0.625
159 attn_19_23 19 cal-en-2 5 0.18817278742790222 0.06915841996669769 0.11790769547224045 31 0.13643309473991394 0.2266169786453247 0.14083418250083923 0.65625
160 attn_19_23 19 cal-en-2 6 0.2252667248249054 0.03137722611427307 0.45010486245155334 31 0.4878306984901428 0.6629999876022339 0.503567099571228 0.49375
161 attn_19_23 19 cal-en-2 7 0.19858212769031525 0.07953452318906784 0.1386381983757019 31 0.15347915887832642 0.2583900988101959 0.15843011438846588 0.5687500000000001
162 attn_19_23 23 cal-zh-1 0 0.18854272365570068 0.06906979531049728 0.08094117045402527 31 0.10190774500370026 0.19865523278713226 0.10519509017467499 0.61875
163 attn_19_23 23 cal-zh-1 1 0.12637057900428772 0.094294473528862 0.11472687125205994 31 0.11000899970531464 0.20533926784992218 0.11355767399072647 0.60625
164 attn_19_23 23 cal-zh-1 2 0.06566040217876434 0.0320066399872303 0.3936012387275696 31 0.3894338607788086 0.6095475554466248 0.40199628472328186 0.53125
165 attn_19_23 23 cal-zh-1 3 0.10236738622188568 0.07628713548183441 0.0909285917878151 31 0.09045256674289703 0.16597077250480652 0.09337039291858673 0.65625
166 attn_19_23 23 cal-zh-1 4 0.11772166937589645 0.09994609653949738 0.11158378422260284 31 0.1082884818315506 0.2333190143108368 0.11178164929151535 0.6000000000000001
167 attn_19_23 23 cal-zh-1 5 0.1445631980895996 0.05731264501810074 0.06436286121606827 31 0.07913714647293091 0.15170255303382874 0.08168993890285492 0.6625000000000001
168 attn_19_23 23 cal-zh-1 6 0.13692563772201538 0.03156186267733574 0.42265841364860535 31 0.43705910444259644 0.6045969128608704 0.45115774869918823 0.5249999999999999
169 attn_19_23 23 cal-zh-1 7 0.17913174629211426 0.0774894654750824 0.055066999047994614 31 0.07003086805343628 0.11456935852766037 0.07228992879390717 0.6875
170 attn_19_23 23 cal-zh-2 0 0.08711955696344376 0.08616629242897034 0.09776511043310165 31 0.09617017209529877 0.2418135702610016 0.09927243739366531 0.7125
171 attn_19_23 23 cal-zh-2 1 0.1001269742846489 0.1142689436674118 0.11429562419652939 31 0.11118914932012558 0.19028690457344055 0.1147758960723877 0.6625000000000001
172 attn_19_23 23 cal-zh-2 2 0.13165870308876038 0.03296923264861107 0.32159483432769775 31 0.34172311425209045 0.5507174134254456 0.35274645686149597 0.675
173 attn_19_23 23 cal-zh-2 3 0.10557965934276581 0.08546337485313416 0.10869138687849045 31 0.10796000063419342 0.267814040184021 0.11144257336854935 0.71875
174 attn_19_23 23 cal-zh-2 4 0.0753588005900383 0.0625741258263588 0.09596671164035797 31 0.09504459798336029 0.18062597513198853 0.09811054915189743 0.6937500000000001
175 attn_19_23 23 cal-zh-2 5 0.0798945426940918 0.0568334236741066 0.07381810247898102 31 0.0762210339307785 0.13966575264930725 0.07867977768182755 0.74375
176 attn_19_23 23 cal-zh-2 6 0.0759878009557724 0.031536251306533813 0.39428046345710754 31 0.3959207534790039 0.5929224491119385 0.408692330121994 0.6375
177 attn_19_23 23 cal-zh-2 7 0.11138643324375153 0.07136214524507523 0.07186198234558105 31 0.07652410864830017 0.15722090005874634 0.07899262756109238 0.7312500000000001
178 attn_19_23 23 cal-en-1 0 0.12444712221622467 0.0752042829990387 0.10921256989240646 31 0.10872898995876312 0.22416618466377258 0.11223636567592621 0.6875
179 attn_19_23 23 cal-en-1 1 0.10719993710517883 0.08359251916408539 0.12698784470558167 31 0.12113931775093079 0.22020384669303894 0.12504702806472778 0.7000000000000001
180 attn_19_23 23 cal-en-1 2 0.13445816934108734 0.03345245122909546 0.3665089011192322 31 0.3763067126274109 0.5767470002174377 0.3884456157684326 0.65625
181 attn_19_23 23 cal-en-1 3 0.10536500066518784 0.07050738483667374 0.07684175670146942 31 0.07960161566734314 0.14911630749702454 0.08216940611600876 0.78125
182 attn_19_23 23 cal-en-1 4 0.0911213681101799 0.17202866077423096 0.10298387706279755 31 0.10929879546165466 0.20484577119350433 0.11282456666231155 0.61875
183 attn_19_23 23 cal-en-1 5 0.18703660368919373 0.12409132719039917 0.09463418275117874 31 0.09024524688720703 0.15500988066196442 0.09315639734268188 0.70625
184 attn_19_23 23 cal-en-1 6 0.13999314606189728 0.03152035176753998 0.4262007474899292 31 0.43630746006965637 0.6232555508613586 0.450381875038147 0.5187499999999999
185 attn_19_23 23 cal-en-1 7 0.11322036385536194 0.10936357825994492 0.07323051989078522 31 0.07213389873504639 0.1506829559803009 0.07446079701185226 0.6125
186 attn_19_23 23 cal-en-2 0 0.11863207072019577 0.13078393042087555 0.09141960740089417 31 0.08733218908309937 0.1577986180782318 0.09014936536550522 0.59375
187 attn_19_23 23 cal-en-2 1 0.07288858294487 0.1372600495815277 0.1089932844042778 31 0.11363835632801056 0.1698836386203766 0.11730410903692245 0.50625
188 attn_19_23 23 cal-en-2 2 0.07288709282875061 0.04088134318590164 0.376287043094635 31 0.3685654401779175 0.5480253100395203 0.3804546594619751 0.5375
189 attn_19_23 23 cal-en-2 3 0.06728185713291168 0.07526300847530365 0.09231049567461014 31 0.08894366025924683 0.15198129415512085 0.09181280434131622 0.6187499999999999
190 attn_19_23 23 cal-en-2 4 0.09862713515758514 0.12817975878715515 0.09404788166284561 31 0.08950259536504745 0.16771778464317322 0.09238976985216141 0.5562499999999999
191 attn_19_23 23 cal-en-2 5 0.07161959260702133 0.0855005532503128 0.10532733052968979 31 0.10144606977701187 0.1908321976661682 0.10471852868795395 0.5625
192 attn_19_23 23 cal-en-2 6 0.08469876646995544 0.03168217092752457 0.4412229359149933 31 0.43726012110710144 0.6215424537658691 0.45136526226997375 0.5437500000000001
193 attn_19_23 23 cal-en-2 7 0.10539951920509338 0.08276306837797165 0.07327506691217422 31 0.0738329142332077 0.13249149918556213 0.07621461898088455 0.63125
@@ -0,0 +1,12 @@
{
"passed": true,
"validated_unix": 1794204667.0190518,
"checksummed_files": 83,
"proof_recomputed_from_retained_weights": true,
"heldout_nll_recomputed_from_saved_arrays": true,
"weights_present_and_retained": true,
"archive_excludes_weights_and_tokenizer": true,
"delivery_excludes_external_activation_caches": true,
"full_experiment_checksums": "checksums-full-server.json",
"figure_source_tables_and_token_nll_complete": true
}
Binary file not shown.
@@ -0,0 +1,89 @@
candidate,kind,source,sample_id,split,relative_l2,reverse_relative_l2,token_relative_p95,reverse_token_relative_p95,cosine,low_norm_tokens,first_key_mass_a,first_key_mass_b,js_without_first_key,without_first_key_valid_rows,js_mean,js_p95,js_without_first_query,top5_overlap
attn_3_7,attn,3,cal-zh-1,calibration,1.75886973383732,1.2170194989630583,2.3326619869739242,1.3829599318796777,-0.001709405865837512,0,0.12333933264017105,0.5467976927757263,0.21484676003456116,248,0.2868691682815552,0.4794144630432129,0.2961229979991913,0.5515625
attn_3_7,attn,3,cal-zh-2,calibration,2.0666622735799773,1.1397562572129543,2.6608388071181586,1.2319401946881372,0.004627105486643174,0,0.1346932351589203,0.6545802354812622,0.1990504413843155,248,0.3104545772075653,0.512997567653656,0.3204692304134369,0.55859375
attn_3_7,attn,3,cal-en-1,calibration,1.9684586970451095,1.1687255334183004,2.5460869484043305,1.3209705033413297,-0.011291865355731309,0,0.1436830461025238,0.6550427675247192,0.2021898329257965,248,0.31806862354278564,0.5598381161689758,0.3283289074897766,0.56015625
attn_3_7,attn,3,cal-en-2,calibration,2.1157833337453664,1.1413057345289206,2.896573536355606,1.279392467480941,-0.010752011897128681,0,0.1500767469406128,0.6574594974517822,0.2199564427137375,248,0.3112969994544983,0.5577057003974915,0.32133883237838745,0.5484375
attn_3_7,attn,7,cal-zh-1,calibration,1.3340187814155438,1.42344327376934,1.4289276609634798,1.638356412458448,0.05265559345262384,0,0.07230515778064728,0.17881479859352112,0.144265815615654,248,0.15721087157726288,0.3267841339111328,0.16228218376636505,0.5890625
attn_3_7,attn,7,cal-zh-2,calibration,1.3371723133481364,1.418275178551954,1.4336050300809013,1.6659403627988982,0.05349503065318622,0,0.081089086830616,0.1558690071105957,0.13882805407047272,248,0.1448846012353897,0.3248489499092102,0.14955829083919525,0.590625
attn_3_7,attn,7,cal-en-1,calibration,1.405999268639987,1.3246539988272832,1.532711571208276,1.4346161336828325,0.07054514898566913,0,0.08104264736175537,0.16970808804035187,0.1672150194644928,248,0.17672982811927795,0.38887327909469604,0.18243078887462616,0.5859375
attn_3_7,attn,7,cal-en-2,calibration,1.366473684595312,1.4009631878257847,1.4664096677417215,1.59944923669573,0.0431210172272288,0,0.07256138324737549,0.1774885654449463,0.1528259664773941,248,0.16717864573001862,0.3629882335662842,0.17257151007652283,0.5625
attn_11_15,attn,11,cal-zh-1,calibration,2.7606058538571316,0.9703348793344666,2.4183673572578086,1.0857279648648543,0.2588922604193846,0,0.24492809176445007,0.4079643785953522,0.11288609355688095,248,0.1462744027376175,0.3241320848464966,0.15099292993545532,0.6625
attn_11_15,attn,11,cal-zh-2,calibration,2.5044424857870546,0.9843061509576916,2.592796177795181,1.1478459097975344,0.23612970738935357,0,0.18575024604797363,0.3524782657623291,0.12114741653203964,248,0.16847708821296692,0.3656128942966461,0.17391183972358704,0.66328125
attn_11_15,attn,11,cal-en-1,calibration,2.649979511635331,0.9795891219874814,2.4632888494810317,1.1094207865752836,0.23948147120552113,0,0.24272800981998444,0.39156001806259155,0.11876673251390457,248,0.15664874017238617,0.337306410074234,0.16170193254947662,0.6929687500000001
attn_11_15,attn,11,cal-en-2,calibration,2.7191409308620766,0.9759398005295231,2.5703854656788505,1.1257459370791087,0.24568690002591262,0,0.2525215446949005,0.3836510181427002,0.11734011769294739,248,0.15115201473236084,0.3331157863140106,0.1560278832912445,0.65078125
attn_11_15,attn,15,cal-zh-1,calibration,3.7267750977169203,0.9943195388381949,1.8571948100572258,1.4659544762265972,0.15463241076860404,0,0.12717953324317932,0.33434635400772095,0.15986472368240356,248,0.19836202263832092,0.40538591146469116,0.20476080477237701,0.6398437499999999
attn_11_15,attn,15,cal-zh-2,calibration,3.484117411382163,0.9958762023786318,1.9555333671772042,1.5034310789069971,0.15731411020281283,0,0.13328716158866882,0.30227771401405334,0.17917340993881226,248,0.21005423367023468,0.41705968976020813,0.21683017909526825,0.628125
attn_11_15,attn,15,cal-en-1,calibration,3.5714882610996845,0.9941664364521509,1.8393702843739006,1.5570787013529828,0.16007660497298756,0,0.14896893501281738,0.3320315480232239,0.17083808779716492,248,0.2080441117286682,0.4197816848754883,0.21475520730018616,0.646875
attn_11_15,attn,15,cal-en-2,calibration,3.783713712256275,0.9885545400787986,2.0440754255967826,1.5107488942763196,0.17418988407724514,0,0.15646213293075562,0.35803988575935364,0.1682518720626831,248,0.2177426666021347,0.419617235660553,0.22476662695407867,0.6046875
attn_19_23,attn,19,cal-zh-1,calibration,3.1071992972679547,1.080818751446137,3.9981135079246157,1.1800456131328154,-0.06780946108706973,0,0.08087001740932465,0.04044787213206291,0.20344728231430054,248,0.20533767342567444,0.5970298051834106,0.21196146309375763,0.65390625
attn_19_23,attn,19,cal-zh-2,calibration,3.5223723842603682,1.066376496758468,4.7369591160509135,1.1555630164896329,-0.07515431162135316,0,0.08035354316234589,0.04998621344566345,0.17885737121105194,248,0.17917148768901825,0.5459575057029724,0.18495121598243713,0.72890625
attn_19_23,attn,19,cal-en-1,calibration,3.7338041826396506,1.0390406833474568,4.724166114393082,1.1075162239519407,-0.0038920109670231525,0,0.24834147095680237,0.10791976749897003,0.18416298925876617,248,0.20052647590637207,0.5714813470840454,0.20699506998062134,0.69375
attn_19_23,attn,19,cal-en-2,calibration,3.3431946049591024,1.0618542149778405,4.853161886124916,1.1462103851521934,-0.04195946512998055,0,0.169009268283844,0.07061692327260971,0.21073731780052185,248,0.22293230891227722,0.6151081323623657,0.23012366890907288,0.58125
attn_19_23,attn,23,cal-zh-1,calibration,2.981238336078164,1.0546738615923779,3.342676305565292,1.0914181503369393,0.018114203094761517,0,0.1326604187488556,0.06724601984024048,0.1667337417602539,248,0.17328983545303345,0.5646554827690125,0.17887982726097107,0.6109375
attn_19_23,attn,23,cal-zh-2,calibration,3.0395249971461062,1.0612041520781252,3.9775899363194043,1.111839148442333,-0.006099502317895858,0,0.09588906913995743,0.06764671951532364,0.15978427231311798,248,0.16259410977363586,0.5136446356773376,0.16783908009529114,0.6968749999999999
attn_19_23,attn,23,cal-en-1,calibration,3.044278410194487,1.074729515389283,4.006931562831888,1.1811940201100057,-0.04307182449714221,0,0.12535521388053894,0.08747006952762604,0.1720750480890274,248,0.1742202639579773,0.5411514639854431,0.17984026670455933,0.66015625
attn_19_23,attn,23,cal-en-2,calibration,2.9710652272259535,1.0799822705908066,3.68900214533588,1.1411985546671786,-0.0470831335221732,0,0.08650433272123337,0.08903923630714417,0.17286045849323273,248,0.17006517946720123,0.538245439529419,0.1755511462688446,0.56875
mlp_5_6,mlp,5,cal-zh-1,calibration,1.2343440431264474,1.2448870136879528,1.4262659492232905,1.3330956688296154,0.23172673358878942,0,,,,,,,,
mlp_5_6,mlp,5,cal-zh-2,calibration,1.2339654450576971,1.2535104800781527,1.32561843838562,1.329338987748564,0.22672917563938177,0,,,,,,,,
mlp_5_6,mlp,5,cal-en-1,calibration,1.2263457192811411,1.2317267194400943,1.3474685872410954,1.321453653023078,0.2447481896291194,0,,,,,,,,
mlp_5_6,mlp,5,cal-en-2,calibration,1.2077976165854318,1.2358845908873308,1.3068781752715062,1.3085733890221656,0.2539150140171328,0,,,,,,,,
mlp_5_6,mlp,6,cal-zh-1,calibration,1.4662701939974856,1.3822306441896342,1.7481556583915787,1.7624732809638852,-0.011619417250216499,0,,,,,,,,
mlp_5_6,mlp,6,cal-zh-2,calibration,1.4798445991262743,1.3637763123448376,1.6959117760237137,1.4894484186140093,-0.005750877367744503,0,,,,,,,,
mlp_5_6,mlp,6,cal-en-1,calibration,1.464591579412775,1.3766590597797903,1.689543163635455,1.5032948778211077,-0.006204174235411236,0,,,,,,,,
mlp_5_6,mlp,6,cal-en-2,calibration,1.4368842673613405,1.4008394919323996,1.6226235618647424,1.6896559214993705,-0.006099379702393146,0,,,,,,,,
mlp_20_21,mlp,20,cal-zh-1,calibration,1.3631629826891347,1.3480373135423591,1.4508876157140895,1.4575901158218467,0.0812649689016906,0,,,,,,,,
mlp_20_21,mlp,20,cal-zh-2,calibration,1.4305121300612613,1.314068906091469,1.5976490501921767,1.4075788062559882,0.06371076852261268,0,,,,,,,,
mlp_20_21,mlp,20,cal-en-1,calibration,1.366192568032983,1.35226347926907,1.5789533371800022,1.5827466919073778,0.07632635231947486,0,,,,,,,,
mlp_20_21,mlp,20,cal-en-2,calibration,1.4386161527552375,1.3312892565009653,1.5821573752114773,1.4214461315124072,0.045400137548793094,0,,,,,,,,
mlp_20_21,mlp,21,cal-zh-1,calibration,1.389882929510158,1.3382556516005442,1.4917751564635304,1.4560073575192687,0.07070714893962673,0,,,,,,,,
mlp_20_21,mlp,21,cal-zh-2,calibration,1.4299639779406776,1.3285677793009314,1.6132811807602088,1.41117608068315,0.052803821787486716,0,,,,,,,,
mlp_20_21,mlp,21,cal-en-1,calibration,1.3453233062044447,1.4107978765350015,1.5399413812441296,1.7867758701030254,0.052139705952592985,0,,,,,,,,
mlp_20_21,mlp,21,cal-en-2,calibration,1.4554092594533905,1.3180744101503734,1.5831740681011888,1.3865901398082663,0.045747088042376524,0,,,,,,,,
qk_norm_3,qk_norm,q,cal-zh-1,calibration,0.22945597860473388,0.23195008932522895,0.24221562711880595,0.24569969899481256,0.973447272274363,0,,,,,,,,
qk_norm_3,qk_norm,q,cal-zh-2,calibration,0.2290088724166579,0.23181448107574626,0.24542894690924066,0.24779202863840974,0.9735303499099761,0,,,,,,,,
qk_norm_3,qk_norm,q,cal-en-1,calibration,0.23455492648208914,0.2376969550882275,0.24989139477418182,0.2529845119820021,0.972212040535478,0,,,,,,,,
qk_norm_3,qk_norm,q,cal-en-2,calibration,0.22901516213757628,0.23221465328764443,0.24944281350466996,0.2558284584833273,0.9735059067182225,0,,,,,,,,
qk_norm_3,qk_norm,k,cal-zh-1,calibration,0.19278924917702345,0.19044985719492102,0.21696829135487192,0.21613622029669263,0.9817161843232514,0,,,,,,,,
qk_norm_3,qk_norm,k,cal-zh-2,calibration,0.1914737037557284,0.18949712918171538,0.22059518754606067,0.22263213250570094,0.98191197888193,0,,,,,,,,
qk_norm_3,qk_norm,k,cal-en-1,calibration,0.19741689848184946,0.1948868605635172,0.25540160556856767,0.2544707934700031,0.9808462075453858,0,,,,,,,,
qk_norm_3,qk_norm,k,cal-en-2,calibration,0.21059744905193553,0.20938264759903177,0.24188706740832974,0.24203851319432876,0.9779690077710401,0,,,,,,,,
qk_norm_7,qk_norm,q,cal-zh-1,calibration,0.20847388940801903,0.20369709176427747,0.2170900495487928,0.21347731005429094,0.9790359000979272,0,,,,,,,,
qk_norm_7,qk_norm,q,cal-zh-2,calibration,0.19935697980676284,0.19546641114417262,0.21545417284813181,0.20622001465458153,0.9807104227014257,0,,,,,,,,
qk_norm_7,qk_norm,q,cal-en-1,calibration,0.20377890669282792,0.1992044511742432,0.2180618268213074,0.21066534853291913,0.9799609125477599,0,,,,,,,,
qk_norm_7,qk_norm,q,cal-en-2,calibration,0.2049050681850306,0.20143020818672328,0.21865291387428726,0.20943815578722963,0.9795092387917065,0,,,,,,,,
qk_norm_7,qk_norm,k,cal-zh-1,calibration,0.11058006301891404,0.10891256044231754,0.12484384056818072,0.12331584938512573,0.9940936589516192,0,,,,,,,,
qk_norm_7,qk_norm,k,cal-zh-2,calibration,0.1025377174596076,0.10115362014926518,0.1190380009492926,0.11772165381976195,0.994906319610355,0,,,,,,,,
qk_norm_7,qk_norm,k,cal-en-1,calibration,0.11061820718242668,0.10874403924660925,0.1246694172087446,0.12218803487494587,0.9941314653674382,0,,,,,,,,
qk_norm_7,qk_norm,k,cal-en-2,calibration,0.1097129827694824,0.10824015260116268,0.12658812743768616,0.1257629217271933,0.9941536582549061,0,,,,,,,,
qk_norm_11,qk_norm,q,cal-zh-1,calibration,0.13162251097797606,0.1336688160421926,0.14723992137490324,0.1503968566479842,0.9913220883632566,0,,,,,,,,
qk_norm_11,qk_norm,q,cal-zh-2,calibration,0.1293519958781259,0.1314802843900349,0.15550840977823716,0.15903465263092675,0.9916295485881612,0,,,,,,,,
qk_norm_11,qk_norm,q,cal-en-1,calibration,0.12647077725692374,0.12852946057966494,0.14803436556570124,0.1502912423902487,0.9920027531738658,0,,,,,,,,
qk_norm_11,qk_norm,q,cal-en-2,calibration,0.1325318326855699,0.13489053569820655,0.15675391452711515,0.16029285586480232,0.9912169569277748,0,,,,,,,,
qk_norm_11,qk_norm,k,cal-zh-1,calibration,0.19235633022629203,0.19080289992858507,0.2290122037106061,0.22871738277610806,0.9816818019088915,0,,,,,,,,
qk_norm_11,qk_norm,k,cal-zh-2,calibration,0.20373761974053708,0.2026516990230578,0.22610646515999447,0.22641810348752162,0.9793703931467622,0,,,,,,,,
qk_norm_11,qk_norm,k,cal-en-1,calibration,0.20072570704403014,0.1997337419816918,0.23210824690537907,0.232000375512668,0.9799664235115935,0,,,,,,,,
qk_norm_11,qk_norm,k,cal-en-2,calibration,0.21292543194773159,0.2114735746425023,0.2540111544217496,0.253548340715113,0.9775093552656579,0,,,,,,,,
qk_norm_15,qk_norm,q,cal-zh-1,calibration,0.15982234891287764,0.1668593847865133,0.1806429713396301,0.18971996556779416,0.9875945267208645,0,,,,,,,,
qk_norm_15,qk_norm,q,cal-zh-2,calibration,0.15740332806536778,0.1630453842176067,0.17808143035469418,0.1865293914125642,0.987788243570594,0,,,,,,,,
qk_norm_15,qk_norm,q,cal-en-1,calibration,0.16051291705164175,0.16606195948802505,0.18552144269008958,0.19215103895989022,0.9872500535949923,0,,,,,,,,
qk_norm_15,qk_norm,q,cal-en-2,calibration,0.1646608571816797,0.17165005121760066,0.18445606954569943,0.19141765315408663,0.9867321292200623,0,,,,,,,,
qk_norm_15,qk_norm,k,cal-zh-1,calibration,0.12891191745886205,0.12454504053986937,0.1385991791084394,0.1331904813118359,0.9925662013931714,0,,,,,,,,
qk_norm_15,qk_norm,k,cal-zh-2,calibration,0.12161598809109211,0.11777013931789594,0.1368420974285716,0.1315377420603533,0.9933549654835544,0,,,,,,,,
qk_norm_15,qk_norm,k,cal-en-1,calibration,0.12197566460111624,0.11819082207382924,0.135322184241321,0.13196489880351644,0.9932886289657402,0,,,,,,,,
qk_norm_15,qk_norm,k,cal-en-2,calibration,0.12441025575886772,0.12064470117171276,0.1406080640306027,0.13504984145616641,0.9929676304409041,0,,,,,,,,
qk_norm_19,qk_norm,q,cal-zh-1,calibration,0.35607606695379634,0.35506507510461943,0.4419932151127893,0.43353427677209583,0.9367889544420241,0,,,,,,,,
qk_norm_19,qk_norm,q,cal-zh-2,calibration,0.3050017336245941,0.3063513991209177,0.35702892926241475,0.35499913749231526,0.9532908937392158,0,,,,,,,,
qk_norm_19,qk_norm,q,cal-en-1,calibration,0.3004172668571062,0.30248734097466856,0.3551434800248225,0.3574678230240773,0.9545873680313379,0,,,,,,,,
qk_norm_19,qk_norm,q,cal-en-2,calibration,0.332120216475189,0.33442277896716227,0.39234612989053624,0.3869494866775299,0.9444895843396817,0,,,,,,,,
qk_norm_19,qk_norm,k,cal-zh-1,calibration,0.1494891933687725,0.1453919459232531,0.19873146648565732,0.1911951771845603,0.9895189305032556,0,,,,,,,,
qk_norm_19,qk_norm,k,cal-zh-2,calibration,0.14909792210432973,0.14536281895078332,0.19307308117224836,0.18807685138750432,0.9894852004735976,0,,,,,,,,
qk_norm_19,qk_norm,k,cal-en-1,calibration,0.1529571262250341,0.14878974314439253,0.19553431973346902,0.18740449249037416,0.9890023265976378,0,,,,,,,,
qk_norm_19,qk_norm,k,cal-en-2,calibration,0.1608915089597201,0.15621975475373284,0.19992876686197072,0.1909273809291801,0.9878669550506567,0,,,,,,,,
qk_norm_23,qk_norm,q,cal-zh-1,calibration,0.5472715903763039,0.4942115602598659,0.5959124252670717,0.5246077344510891,0.8699706486172409,0,,,,,,,,
qk_norm_23,qk_norm,q,cal-zh-2,calibration,0.5066278025120974,0.4652273156892467,0.5698812253775749,0.505450409971441,0.8857874771365101,0,,,,,,,,
qk_norm_23,qk_norm,q,cal-en-1,calibration,0.5396383470456222,0.4881273164223247,0.6104980922233472,0.5338668240636818,0.8733304573488526,0,,,,,,,,
qk_norm_23,qk_norm,q,cal-en-2,calibration,0.5484944291943057,0.4948734608679084,0.6117866340678192,0.5354951226175159,0.8695786360146698,0,,,,,,,,
qk_norm_23,qk_norm,k,cal-zh-1,calibration,0.1313144920837702,0.13005524261632884,0.18219490064170027,0.1762085222381826,0.9915073560564274,0,,,,,,,,
qk_norm_23,qk_norm,k,cal-zh-2,calibration,0.12151622394280005,0.12124252528158447,0.1793544881596151,0.17483197862503747,0.9926360753692314,0,,,,,,,,
qk_norm_23,qk_norm,k,cal-en-1,calibration,0.12730373699904377,0.12630314407300633,0.17809052202752992,0.17316093196082644,0.9919917024971001,0,,,,,,,,
qk_norm_23,qk_norm,k,cal-en-2,calibration,0.15218433424490216,0.1492986535261055,0.1960834395064235,0.18960021577591676,0.9888227905277445,0,,,,,,,,
1 candidate kind source sample_id split relative_l2 reverse_relative_l2 token_relative_p95 reverse_token_relative_p95 cosine low_norm_tokens first_key_mass_a first_key_mass_b js_without_first_key without_first_key_valid_rows js_mean js_p95 js_without_first_query top5_overlap
2 attn_3_7 attn 3 cal-zh-1 calibration 1.75886973383732 1.2170194989630583 2.3326619869739242 1.3829599318796777 -0.001709405865837512 0 0.12333933264017105 0.5467976927757263 0.21484676003456116 248 0.2868691682815552 0.4794144630432129 0.2961229979991913 0.5515625
3 attn_3_7 attn 3 cal-zh-2 calibration 2.0666622735799773 1.1397562572129543 2.6608388071181586 1.2319401946881372 0.004627105486643174 0 0.1346932351589203 0.6545802354812622 0.1990504413843155 248 0.3104545772075653 0.512997567653656 0.3204692304134369 0.55859375
4 attn_3_7 attn 3 cal-en-1 calibration 1.9684586970451095 1.1687255334183004 2.5460869484043305 1.3209705033413297 -0.011291865355731309 0 0.1436830461025238 0.6550427675247192 0.2021898329257965 248 0.31806862354278564 0.5598381161689758 0.3283289074897766 0.56015625
5 attn_3_7 attn 3 cal-en-2 calibration 2.1157833337453664 1.1413057345289206 2.896573536355606 1.279392467480941 -0.010752011897128681 0 0.1500767469406128 0.6574594974517822 0.2199564427137375 248 0.3112969994544983 0.5577057003974915 0.32133883237838745 0.5484375
6 attn_3_7 attn 7 cal-zh-1 calibration 1.3340187814155438 1.42344327376934 1.4289276609634798 1.638356412458448 0.05265559345262384 0 0.07230515778064728 0.17881479859352112 0.144265815615654 248 0.15721087157726288 0.3267841339111328 0.16228218376636505 0.5890625
7 attn_3_7 attn 7 cal-zh-2 calibration 1.3371723133481364 1.418275178551954 1.4336050300809013 1.6659403627988982 0.05349503065318622 0 0.081089086830616 0.1558690071105957 0.13882805407047272 248 0.1448846012353897 0.3248489499092102 0.14955829083919525 0.590625
8 attn_3_7 attn 7 cal-en-1 calibration 1.405999268639987 1.3246539988272832 1.532711571208276 1.4346161336828325 0.07054514898566913 0 0.08104264736175537 0.16970808804035187 0.1672150194644928 248 0.17672982811927795 0.38887327909469604 0.18243078887462616 0.5859375
9 attn_3_7 attn 7 cal-en-2 calibration 1.366473684595312 1.4009631878257847 1.4664096677417215 1.59944923669573 0.0431210172272288 0 0.07256138324737549 0.1774885654449463 0.1528259664773941 248 0.16717864573001862 0.3629882335662842 0.17257151007652283 0.5625
10 attn_11_15 attn 11 cal-zh-1 calibration 2.7606058538571316 0.9703348793344666 2.4183673572578086 1.0857279648648543 0.2588922604193846 0 0.24492809176445007 0.4079643785953522 0.11288609355688095 248 0.1462744027376175 0.3241320848464966 0.15099292993545532 0.6625
11 attn_11_15 attn 11 cal-zh-2 calibration 2.5044424857870546 0.9843061509576916 2.592796177795181 1.1478459097975344 0.23612970738935357 0 0.18575024604797363 0.3524782657623291 0.12114741653203964 248 0.16847708821296692 0.3656128942966461 0.17391183972358704 0.66328125
12 attn_11_15 attn 11 cal-en-1 calibration 2.649979511635331 0.9795891219874814 2.4632888494810317 1.1094207865752836 0.23948147120552113 0 0.24272800981998444 0.39156001806259155 0.11876673251390457 248 0.15664874017238617 0.337306410074234 0.16170193254947662 0.6929687500000001
13 attn_11_15 attn 11 cal-en-2 calibration 2.7191409308620766 0.9759398005295231 2.5703854656788505 1.1257459370791087 0.24568690002591262 0 0.2525215446949005 0.3836510181427002 0.11734011769294739 248 0.15115201473236084 0.3331157863140106 0.1560278832912445 0.65078125
14 attn_11_15 attn 15 cal-zh-1 calibration 3.7267750977169203 0.9943195388381949 1.8571948100572258 1.4659544762265972 0.15463241076860404 0 0.12717953324317932 0.33434635400772095 0.15986472368240356 248 0.19836202263832092 0.40538591146469116 0.20476080477237701 0.6398437499999999
15 attn_11_15 attn 15 cal-zh-2 calibration 3.484117411382163 0.9958762023786318 1.9555333671772042 1.5034310789069971 0.15731411020281283 0 0.13328716158866882 0.30227771401405334 0.17917340993881226 248 0.21005423367023468 0.41705968976020813 0.21683017909526825 0.628125
16 attn_11_15 attn 15 cal-en-1 calibration 3.5714882610996845 0.9941664364521509 1.8393702843739006 1.5570787013529828 0.16007660497298756 0 0.14896893501281738 0.3320315480232239 0.17083808779716492 248 0.2080441117286682 0.4197816848754883 0.21475520730018616 0.646875
17 attn_11_15 attn 15 cal-en-2 calibration 3.783713712256275 0.9885545400787986 2.0440754255967826 1.5107488942763196 0.17418988407724514 0 0.15646213293075562 0.35803988575935364 0.1682518720626831 248 0.2177426666021347 0.419617235660553 0.22476662695407867 0.6046875
18 attn_19_23 attn 19 cal-zh-1 calibration 3.1071992972679547 1.080818751446137 3.9981135079246157 1.1800456131328154 -0.06780946108706973 0 0.08087001740932465 0.04044787213206291 0.20344728231430054 248 0.20533767342567444 0.5970298051834106 0.21196146309375763 0.65390625
19 attn_19_23 attn 19 cal-zh-2 calibration 3.5223723842603682 1.066376496758468 4.7369591160509135 1.1555630164896329 -0.07515431162135316 0 0.08035354316234589 0.04998621344566345 0.17885737121105194 248 0.17917148768901825 0.5459575057029724 0.18495121598243713 0.72890625
20 attn_19_23 attn 19 cal-en-1 calibration 3.7338041826396506 1.0390406833474568 4.724166114393082 1.1075162239519407 -0.0038920109670231525 0 0.24834147095680237 0.10791976749897003 0.18416298925876617 248 0.20052647590637207 0.5714813470840454 0.20699506998062134 0.69375
21 attn_19_23 attn 19 cal-en-2 calibration 3.3431946049591024 1.0618542149778405 4.853161886124916 1.1462103851521934 -0.04195946512998055 0 0.169009268283844 0.07061692327260971 0.21073731780052185 248 0.22293230891227722 0.6151081323623657 0.23012366890907288 0.58125
22 attn_19_23 attn 23 cal-zh-1 calibration 2.981238336078164 1.0546738615923779 3.342676305565292 1.0914181503369393 0.018114203094761517 0 0.1326604187488556 0.06724601984024048 0.1667337417602539 248 0.17328983545303345 0.5646554827690125 0.17887982726097107 0.6109375
23 attn_19_23 attn 23 cal-zh-2 calibration 3.0395249971461062 1.0612041520781252 3.9775899363194043 1.111839148442333 -0.006099502317895858 0 0.09588906913995743 0.06764671951532364 0.15978427231311798 248 0.16259410977363586 0.5136446356773376 0.16783908009529114 0.6968749999999999
24 attn_19_23 attn 23 cal-en-1 calibration 3.044278410194487 1.074729515389283 4.006931562831888 1.1811940201100057 -0.04307182449714221 0 0.12535521388053894 0.08747006952762604 0.1720750480890274 248 0.1742202639579773 0.5411514639854431 0.17984026670455933 0.66015625
25 attn_19_23 attn 23 cal-en-2 calibration 2.9710652272259535 1.0799822705908066 3.68900214533588 1.1411985546671786 -0.0470831335221732 0 0.08650433272123337 0.08903923630714417 0.17286045849323273 248 0.17006517946720123 0.538245439529419 0.1755511462688446 0.56875
26 mlp_5_6 mlp 5 cal-zh-1 calibration 1.2343440431264474 1.2448870136879528 1.4262659492232905 1.3330956688296154 0.23172673358878942 0
27 mlp_5_6 mlp 5 cal-zh-2 calibration 1.2339654450576971 1.2535104800781527 1.32561843838562 1.329338987748564 0.22672917563938177 0
28 mlp_5_6 mlp 5 cal-en-1 calibration 1.2263457192811411 1.2317267194400943 1.3474685872410954 1.321453653023078 0.2447481896291194 0
29 mlp_5_6 mlp 5 cal-en-2 calibration 1.2077976165854318 1.2358845908873308 1.3068781752715062 1.3085733890221656 0.2539150140171328 0
30 mlp_5_6 mlp 6 cal-zh-1 calibration 1.4662701939974856 1.3822306441896342 1.7481556583915787 1.7624732809638852 -0.011619417250216499 0
31 mlp_5_6 mlp 6 cal-zh-2 calibration 1.4798445991262743 1.3637763123448376 1.6959117760237137 1.4894484186140093 -0.005750877367744503 0
32 mlp_5_6 mlp 6 cal-en-1 calibration 1.464591579412775 1.3766590597797903 1.689543163635455 1.5032948778211077 -0.006204174235411236 0
33 mlp_5_6 mlp 6 cal-en-2 calibration 1.4368842673613405 1.4008394919323996 1.6226235618647424 1.6896559214993705 -0.006099379702393146 0
34 mlp_20_21 mlp 20 cal-zh-1 calibration 1.3631629826891347 1.3480373135423591 1.4508876157140895 1.4575901158218467 0.0812649689016906 0
35 mlp_20_21 mlp 20 cal-zh-2 calibration 1.4305121300612613 1.314068906091469 1.5976490501921767 1.4075788062559882 0.06371076852261268 0
36 mlp_20_21 mlp 20 cal-en-1 calibration 1.366192568032983 1.35226347926907 1.5789533371800022 1.5827466919073778 0.07632635231947486 0
37 mlp_20_21 mlp 20 cal-en-2 calibration 1.4386161527552375 1.3312892565009653 1.5821573752114773 1.4214461315124072 0.045400137548793094 0
38 mlp_20_21 mlp 21 cal-zh-1 calibration 1.389882929510158 1.3382556516005442 1.4917751564635304 1.4560073575192687 0.07070714893962673 0
39 mlp_20_21 mlp 21 cal-zh-2 calibration 1.4299639779406776 1.3285677793009314 1.6132811807602088 1.41117608068315 0.052803821787486716 0
40 mlp_20_21 mlp 21 cal-en-1 calibration 1.3453233062044447 1.4107978765350015 1.5399413812441296 1.7867758701030254 0.052139705952592985 0
41 mlp_20_21 mlp 21 cal-en-2 calibration 1.4554092594533905 1.3180744101503734 1.5831740681011888 1.3865901398082663 0.045747088042376524 0
42 qk_norm_3 qk_norm q cal-zh-1 calibration 0.22945597860473388 0.23195008932522895 0.24221562711880595 0.24569969899481256 0.973447272274363 0
43 qk_norm_3 qk_norm q cal-zh-2 calibration 0.2290088724166579 0.23181448107574626 0.24542894690924066 0.24779202863840974 0.9735303499099761 0
44 qk_norm_3 qk_norm q cal-en-1 calibration 0.23455492648208914 0.2376969550882275 0.24989139477418182 0.2529845119820021 0.972212040535478 0
45 qk_norm_3 qk_norm q cal-en-2 calibration 0.22901516213757628 0.23221465328764443 0.24944281350466996 0.2558284584833273 0.9735059067182225 0
46 qk_norm_3 qk_norm k cal-zh-1 calibration 0.19278924917702345 0.19044985719492102 0.21696829135487192 0.21613622029669263 0.9817161843232514 0
47 qk_norm_3 qk_norm k cal-zh-2 calibration 0.1914737037557284 0.18949712918171538 0.22059518754606067 0.22263213250570094 0.98191197888193 0
48 qk_norm_3 qk_norm k cal-en-1 calibration 0.19741689848184946 0.1948868605635172 0.25540160556856767 0.2544707934700031 0.9808462075453858 0
49 qk_norm_3 qk_norm k cal-en-2 calibration 0.21059744905193553 0.20938264759903177 0.24188706740832974 0.24203851319432876 0.9779690077710401 0
50 qk_norm_7 qk_norm q cal-zh-1 calibration 0.20847388940801903 0.20369709176427747 0.2170900495487928 0.21347731005429094 0.9790359000979272 0
51 qk_norm_7 qk_norm q cal-zh-2 calibration 0.19935697980676284 0.19546641114417262 0.21545417284813181 0.20622001465458153 0.9807104227014257 0
52 qk_norm_7 qk_norm q cal-en-1 calibration 0.20377890669282792 0.1992044511742432 0.2180618268213074 0.21066534853291913 0.9799609125477599 0
53 qk_norm_7 qk_norm q cal-en-2 calibration 0.2049050681850306 0.20143020818672328 0.21865291387428726 0.20943815578722963 0.9795092387917065 0
54 qk_norm_7 qk_norm k cal-zh-1 calibration 0.11058006301891404 0.10891256044231754 0.12484384056818072 0.12331584938512573 0.9940936589516192 0
55 qk_norm_7 qk_norm k cal-zh-2 calibration 0.1025377174596076 0.10115362014926518 0.1190380009492926 0.11772165381976195 0.994906319610355 0
56 qk_norm_7 qk_norm k cal-en-1 calibration 0.11061820718242668 0.10874403924660925 0.1246694172087446 0.12218803487494587 0.9941314653674382 0
57 qk_norm_7 qk_norm k cal-en-2 calibration 0.1097129827694824 0.10824015260116268 0.12658812743768616 0.1257629217271933 0.9941536582549061 0
58 qk_norm_11 qk_norm q cal-zh-1 calibration 0.13162251097797606 0.1336688160421926 0.14723992137490324 0.1503968566479842 0.9913220883632566 0
59 qk_norm_11 qk_norm q cal-zh-2 calibration 0.1293519958781259 0.1314802843900349 0.15550840977823716 0.15903465263092675 0.9916295485881612 0
60 qk_norm_11 qk_norm q cal-en-1 calibration 0.12647077725692374 0.12852946057966494 0.14803436556570124 0.1502912423902487 0.9920027531738658 0
61 qk_norm_11 qk_norm q cal-en-2 calibration 0.1325318326855699 0.13489053569820655 0.15675391452711515 0.16029285586480232 0.9912169569277748 0
62 qk_norm_11 qk_norm k cal-zh-1 calibration 0.19235633022629203 0.19080289992858507 0.2290122037106061 0.22871738277610806 0.9816818019088915 0
63 qk_norm_11 qk_norm k cal-zh-2 calibration 0.20373761974053708 0.2026516990230578 0.22610646515999447 0.22641810348752162 0.9793703931467622 0
64 qk_norm_11 qk_norm k cal-en-1 calibration 0.20072570704403014 0.1997337419816918 0.23210824690537907 0.232000375512668 0.9799664235115935 0
65 qk_norm_11 qk_norm k cal-en-2 calibration 0.21292543194773159 0.2114735746425023 0.2540111544217496 0.253548340715113 0.9775093552656579 0
66 qk_norm_15 qk_norm q cal-zh-1 calibration 0.15982234891287764 0.1668593847865133 0.1806429713396301 0.18971996556779416 0.9875945267208645 0
67 qk_norm_15 qk_norm q cal-zh-2 calibration 0.15740332806536778 0.1630453842176067 0.17808143035469418 0.1865293914125642 0.987788243570594 0
68 qk_norm_15 qk_norm q cal-en-1 calibration 0.16051291705164175 0.16606195948802505 0.18552144269008958 0.19215103895989022 0.9872500535949923 0
69 qk_norm_15 qk_norm q cal-en-2 calibration 0.1646608571816797 0.17165005121760066 0.18445606954569943 0.19141765315408663 0.9867321292200623 0
70 qk_norm_15 qk_norm k cal-zh-1 calibration 0.12891191745886205 0.12454504053986937 0.1385991791084394 0.1331904813118359 0.9925662013931714 0
71 qk_norm_15 qk_norm k cal-zh-2 calibration 0.12161598809109211 0.11777013931789594 0.1368420974285716 0.1315377420603533 0.9933549654835544 0
72 qk_norm_15 qk_norm k cal-en-1 calibration 0.12197566460111624 0.11819082207382924 0.135322184241321 0.13196489880351644 0.9932886289657402 0
73 qk_norm_15 qk_norm k cal-en-2 calibration 0.12441025575886772 0.12064470117171276 0.1406080640306027 0.13504984145616641 0.9929676304409041 0
74 qk_norm_19 qk_norm q cal-zh-1 calibration 0.35607606695379634 0.35506507510461943 0.4419932151127893 0.43353427677209583 0.9367889544420241 0
75 qk_norm_19 qk_norm q cal-zh-2 calibration 0.3050017336245941 0.3063513991209177 0.35702892926241475 0.35499913749231526 0.9532908937392158 0
76 qk_norm_19 qk_norm q cal-en-1 calibration 0.3004172668571062 0.30248734097466856 0.3551434800248225 0.3574678230240773 0.9545873680313379 0
77 qk_norm_19 qk_norm q cal-en-2 calibration 0.332120216475189 0.33442277896716227 0.39234612989053624 0.3869494866775299 0.9444895843396817 0
78 qk_norm_19 qk_norm k cal-zh-1 calibration 0.1494891933687725 0.1453919459232531 0.19873146648565732 0.1911951771845603 0.9895189305032556 0
79 qk_norm_19 qk_norm k cal-zh-2 calibration 0.14909792210432973 0.14536281895078332 0.19307308117224836 0.18807685138750432 0.9894852004735976 0
80 qk_norm_19 qk_norm k cal-en-1 calibration 0.1529571262250341 0.14878974314439253 0.19553431973346902 0.18740449249037416 0.9890023265976378 0
81 qk_norm_19 qk_norm k cal-en-2 calibration 0.1608915089597201 0.15621975475373284 0.19992876686197072 0.1909273809291801 0.9878669550506567 0
82 qk_norm_23 qk_norm q cal-zh-1 calibration 0.5472715903763039 0.4942115602598659 0.5959124252670717 0.5246077344510891 0.8699706486172409 0
83 qk_norm_23 qk_norm q cal-zh-2 calibration 0.5066278025120974 0.4652273156892467 0.5698812253775749 0.505450409971441 0.8857874771365101 0
84 qk_norm_23 qk_norm q cal-en-1 calibration 0.5396383470456222 0.4881273164223247 0.6104980922233472 0.5338668240636818 0.8733304573488526 0
85 qk_norm_23 qk_norm q cal-en-2 calibration 0.5484944291943057 0.4948734608679084 0.6117866340678192 0.5354951226175159 0.8695786360146698 0
86 qk_norm_23 qk_norm k cal-zh-1 calibration 0.1313144920837702 0.13005524261632884 0.18219490064170027 0.1762085222381826 0.9915073560564274 0
87 qk_norm_23 qk_norm k cal-zh-2 calibration 0.12151622394280005 0.12124252528158447 0.1793544881596151 0.17483197862503747 0.9926360753692314 0
88 qk_norm_23 qk_norm k cal-en-1 calibration 0.12730373699904377 0.12630314407300633 0.17809052202752992 0.17316093196082644 0.9919917024971001 0
89 qk_norm_23 qk_norm k cal-en-2 calibration 0.15218433424490216 0.1492986535261055 0.1960834395064235 0.18960021577591676 0.9888227905277445 0
@@ -0,0 +1,101 @@
[
{
"id": "attn_3_7",
"kind": "attn",
"i": 3,
"j": 7,
"passed": false,
"maximum_symmetric_relative_error": 2.1157833337453664,
"donor_component_bytes": 29361152
},
{
"id": "attn_11_15",
"kind": "attn",
"i": 11,
"j": 15,
"passed": false,
"maximum_symmetric_relative_error": 3.783713712256275,
"donor_component_bytes": 29361152
},
{
"id": "attn_19_23",
"kind": "attn",
"i": 19,
"j": 23,
"passed": false,
"maximum_symmetric_relative_error": 3.7338041826396506,
"donor_component_bytes": 29361152
},
{
"id": "mlp_5_6",
"kind": "mlp",
"i": 5,
"j": 6,
"passed": false,
"maximum_symmetric_relative_error": 1.4798445991262743,
"donor_component_bytes": 75497472
},
{
"id": "mlp_20_21",
"kind": "mlp",
"i": 20,
"j": 21,
"passed": false,
"maximum_symmetric_relative_error": 1.4554092594533905,
"donor_component_bytes": 75497472
},
{
"id": "qk_norm_3",
"kind": "qk_norm",
"i": 3,
"j": 3,
"passed": false,
"maximum_symmetric_relative_error": 0.2376969550882275,
"donor_component_bytes": 512
},
{
"id": "qk_norm_7",
"kind": "qk_norm",
"i": 7,
"j": 7,
"passed": false,
"maximum_symmetric_relative_error": 0.20847388940801903,
"donor_component_bytes": 512
},
{
"id": "qk_norm_11",
"kind": "qk_norm",
"i": 11,
"j": 11,
"passed": false,
"maximum_symmetric_relative_error": 0.21292543194773159,
"donor_component_bytes": 512
},
{
"id": "qk_norm_15",
"kind": "qk_norm",
"i": 15,
"j": 15,
"passed": false,
"maximum_symmetric_relative_error": 0.17165005121760066,
"donor_component_bytes": 512
},
{
"id": "qk_norm_19",
"kind": "qk_norm",
"i": 19,
"j": 19,
"passed": false,
"maximum_symmetric_relative_error": 0.35607606695379634,
"donor_component_bytes": 512
},
{
"id": "qk_norm_23",
"kind": "qk_norm",
"i": 23,
"j": 23,
"passed": false,
"maximum_symmetric_relative_error": 0.5484944291943057,
"donor_component_bytes": 512
}
]
@@ -0,0 +1,85 @@
{
"FUNCTIONAL_REPORT.md": "6bfde7e8843e4c87186f8229bf15ab1db919f08ddd8b40fd04aecdec20c7390d",
"attention-head-metrics.csv": "732ada825914f8db49b72083cb129cf1305434c84c88253cc32b499ca1a9a119",
"baseline-activations.npz": "a851b489a8917a63b07cb6615a47fd1399b36e62f62d6140be618c4966d204ef",
"baseline-heldout-token-nll.npy": "303853deb86cb09848b63e2f8b62673db5e730c3b7984c7e65fef3f61c33cda1",
"calibration-metrics.csv": "fff8d1e638c6dcfaed88df580de348083fb01621bae7cda7ca2f04ec29636137",
"candidate-summary.json": "dc1d738760fa87ef681e33a72caf67cdb68db7413b66f364f80a67b0233fdbdc",
"compute-budget-ledger.json": "33dac0f3e9aaf6a82a01c67c22f9a755dff9be8092710f5d3fa8236ce551b255",
"download.log": "9f5b76650a94f15729e949605b50ab172ff0fc3a1d791b35dd2a88d4ace02dd0",
"experiment.json": "911ba54b489547373b724bd0e719717d8f836807d8fad8aa221eadb5a525d05a",
"figures/algebraic_gain_identity.pdf": "b9c44ed9f2df66f7b9172e9a1650db31ec3915290fef3d59b77320a0510ee6de",
"figures/algebraic_gain_identity.png": "2e637bb3362ac1592959fbe8f84e67ba06b1ea00ec71dfeec80c45a7ce7520b1",
"figures/algebraic_gain_identity.svg": "553ac72babfc005f67a91564f8b737b5edd279d62fb013b8d5d87f6884c4f1ac",
"figures/candidate_function_errors.pdf": "370f9b888a6cff64d69bb5f52541e71c66695139cc0b1a45ca8115eae44aae33",
"figures/candidate_function_errors.png": "a469f33e4f0a8ba81d4cbcaa333ddcf822f6fa2e4f33ea2c0a6d7da3cbd814b9",
"figures/candidate_function_errors.svg": "a647cfcf58bd5d96770daed29cf14ffd245aef3a0c463706f3724bf84b800469",
"figures/learned_norm_function_errors.pdf": "cba6296e9297d8f14b1f3346b4ab4015845a54bc880d0f153c8d4b586fe16f90",
"figures/learned_norm_function_errors.png": "85b1bce927e1c152778b814629a52996bfdb4a0828995492f56e1fb12f68648f",
"figures/learned_norm_function_errors.svg": "ce867a8b8aedfd941dde51ae53ad2541f4e407235c2572092dca6446e71d8f8c",
"figures/learned_norm_network_substitution.pdf": "6d3440323d276b4a70209f71cd31e016d8eb78d257bdaa41ffc0468d2813a428",
"figures/learned_norm_network_substitution.png": "e0dca3ce8082abf3e904aaa27aa581bd1c7ed1ff7a19285dde613dd88b3738ad",
"figures/learned_norm_network_substitution.svg": "7753abfa3f85cda50fbc3a00491f54a597cbbb54fdb75f0c0b2fe7a7fcc131c5",
"final-report.log": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"final-source-hashes.json": "52398645bce397827c36048c417f721a1cd8f76dc38eeaee530b8a5031b09e91",
"final-tests.txt": "3eb63c08f7b8de12c4198e79da9909582d5d64cb821ef1b08d38de7e4e43eaf5",
"functional-code-hashes.json": "3756c43b0904c53d9a3ea1fe92496cc9f890cc7b033a3c9577ef53cecd4d8536",
"functional-environment.json": "b9a2b4b850b3f26cfa96efc1ac6a9470d678f17197f96a01d9c3bdf5111b148a",
"functional-report.log": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"functional-run.log": "95684864b79b72dbefdf5485fca00bca3e80719c9e925a2a6fe1a6b0a07825d3",
"functional-status.json": "fdda303b7e520f9e0065c65281df5b4bd6e9277edec52f9e5818047598a2a2ea",
"gain-factorization-k_to_q-token-nll.npy": "767f2a885532e347a654a86ed0a7f19bd4938c52296dcc74db26876acb3fe6a1",
"gain-factorization-kernel-timing.csv": "7ee790a456d4b9dac9d4756b71288f4df165c0c6111fe94657fa6804f91154d1",
"gain-factorization-optimized-summary.json": "0e303074384a3933577b2c3c29a042fe00e64cf308eb8f6190db8f7e3db55e7b",
"gain-factorization-optimized-timing.csv": "b5c85e1f9bcfb49bf317090470530a44546079c5a79c1df2365c00c36144dc10",
"gain-factorization-optimized.log": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"gain-factorization-protocol.json": "7690e4cf5dec415e414813e5beb3434ae62d51d95e68ad5de48e698980e26cab",
"gain-factorization-q_to_k-token-nll.npy": "6ea27edea56593c7269403ce5a0f7eb8673b9d80d1e00806d38bc6a2a681fff8",
"gain-factorization-score-metrics.csv": "7cfcc997980cd0566721e529f6bb1bca2f95858da50d5cf533eb31742ab0c230",
"gain-factorization-summary.json": "3bfde7dba2abd19e6dc945c05ad924181bc0927e0b88806337c9bbab9db7b359",
"gain-factorization.log": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"install-torch-disk-tmp.log": "85ee085d264707b8fd0031876c3ae9cf4e36151db9cbf53edef979f10d144c13",
"install-torch.log": "e297f77b38de146e85e6159025ea1e0ec93622fe26ef46698df180c7c96378b8",
"manifest.json": "28efbaf94a5034333339a35f08ead151459a7d20c6c2d2f07a451e7f3bf154de",
"model_config.json": "ed1c1723241f23f7f4e23430759cbd7dcfb4103cbdfe052bfe7626b57c2615b4",
"neuron-reuse-activations.npz": "0ce0b7ba4efef0164a68d91595e8eb47a6347f1f364eeda764835cfa09b09448",
"neuron-reuse-protocol.json": "cc4fcf06ebd4ee5420933534021006df6e62259b00fd922f6e8fabada89472f7",
"neuron-reuse-search-summary.json": "30a5ef914c4ff766e9bde09e3784a229bdbe7a10cea871bef3be327df8047c25",
"neuron-reuse-selection.json": "afafc7f8d35c562940f5afe47c5edaf3c5b7ca97744152bb5f9988b9910797a4",
"neuron-reuse-summary.json": "f14999d51d262c67f90e2e37918701b6b56107c7d5c47fad6da1cdfeb35d62a2",
"neuron-reuse.log": "be387fa429688a6a34131f0a5899a81586a2bde85cf209253c2da977e625168c",
"norm-repetition-12-uses-13-token-nll.npy": "fb05b796fd40122aa0a1b05ce92585c980f47a1cb1c771df4fabc720ade4bbd8",
"norm-repetition-13-uses-12-token-nll.npy": "5470a87f22f100ed8e43caf90a05831453b4277707d6d7085eecc4bd43ae8cb6",
"norm-repetition-function-metrics.csv": "d5f4eed05a9f5c6e0f971331dd9d0b834d6141b4aeb9e3c725a5bee1f4314593",
"norm-repetition-input-profiles.npz": "7e3415b8f28d8844658facf2c37415cc0f3a4de8f6d39d3e0c4b01e1b3a4d7e0",
"norm-repetition-protocol.json": "ad8d2319572b4a1daa5d3ae227bd344b5f1db7941ee567b43d1f25dbd963f6a0",
"norm-repetition-selection.json": "4dae5a62da0889c0ed4de7813a8f720bb94991e468b9d6cdee8c5327d549ac55",
"norm-repetition-summary.json": "70b5067d0820b5a5bc7062c71b9f29c5590d9bb12bf538beb961135d1aab8834",
"norm-repetition-uniform-bounds.csv": "48e71a9cac9410d110483cef8356e02b2dcdeae1120bcb243ae86903852fb32a",
"norm-repetition.log": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"preflight-first-attempt.log": "1a336568f0ed99ac306ab1d690ff00f325c9c73ec6be84df4ace4374d490c650",
"preflight-second-attempt.json": "a6b68d31710e45cbf09ee100b490c781547334f76ecfbb3e09332779e953b30c",
"preflight-second-attempt.log": "42a07ef0a3fd8cb1a6475bec6e7958f754a24012fb8269073d8625b1b891594f",
"preflight.json": "86113ecbf499f4a84ea4d7fd7893e8c12f2784b6e4678b5cbcbb0e022e295e4b",
"preflight.log": "66dd7d1e0b146fc448a1064efbde5d693126d3f3cbb2fc9abdeb4485ead7c951",
"reference-check-first-attempt.log": "ed77d6619c19551401938a54fae0ef1aa50846b943ea1ba3321750c9cb985428",
"reference-check-second-attempt.log": "5923802faaba183a099d0eb313d5dc2de4825b1d52ec6ce0699a8201f68d99af",
"reference-check.json": "e82b0d8634ccae0671269587da960e06cce33e0a0e2ed15db15d946f6399d9ed",
"reference-check.log": "e4a1d55d309bff39535ecab272bd2aa6d8fe183656724b7731e4171d03150f11",
"retention-policy.json": "e3cdb6707985d2fa1fcb7f088f8f46f9eafb89eea13e37f51c554f57790acd6e",
"samples-tokenized.json": "2e56e14a4eefedd702bfe09d3edb2f17f360f53dd420e29fcf0c0ec1affc0184",
"selection.json": "c8c86d12fe7db6fb8fffca8078ec0ff5438b97998088c89862db8f9b6152daa3",
"source_metadata.json": "d3be2a8560f4270b8af27891d1076cb7cf80bd84c4c19fe7790d5fb06f3a342e",
"storage-accounting.log": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"storage-evidence.json": "c6968f4143910cef9d1359d3cd3342d289b9c791c6e2a6ab598fe235b105e0d9",
"supervisor-error.txt": "4cba26d15e963620745ab3d4625f4de967859a9c43993f78ca1ce2e5c06aeb42",
"supervisor-first-attempt-error.txt": "e689f94e983b84cfc4e6b372175826d77dcd2d5c9b8d166598bc8607227afbba",
"supervisor-first-attempt.json": "93d7185df1810aec2aa396fd8247e188b8a37a821731ef681e66389695bd9a8e",
"supervisor-second-attempt.json": "7333a3634f7d169bc78d9eeb5e47d8fbd2970c8401b4d6160b92038ad2a89f55",
"supervisor-status.json": "ae078d540c9c683bac468313a85754d8ecf1a88be482a2d5895bbfca0c610404",
"supervisor-third-attempt.json": "e86ce8e2f55f75ab3c61ffd7b4a1517e627cebbdb85ee7f966615c5d683722ed",
"supervisor.log": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"tokenizer-download.log": "1d9c8b608f3c79c9d7af93c593543ddd14db3e077e2c108738bccd518df9a79a",
"tokenizers-wheel.log": "f0b4f7c1360ff1a00e04fe443e619682c7d7b024df55545d31f54951b3001480",
"unit-tests.txt": "fa8ffc7277ea722b9261da462b797710ed4ae3eb8975ad51937c419b3052e3af"
}
@@ -0,0 +1,85 @@
{
"DELIVERY.md": "8f4608add548dc4c6bc3785ccc262a4c695098c043563f43193edac904702cb1",
"FUNCTIONAL_REPORT.md": "6bfde7e8843e4c87186f8229bf15ab1db919f08ddd8b40fd04aecdec20c7390d",
"attention-head-metrics.csv": "732ada825914f8db49b72083cb129cf1305434c84c88253cc32b499ca1a9a119",
"baseline-heldout-token-nll.npy": "303853deb86cb09848b63e2f8b62673db5e730c3b7984c7e65fef3f61c33cda1",
"calibration-metrics.csv": "fff8d1e638c6dcfaed88df580de348083fb01621bae7cda7ca2f04ec29636137",
"candidate-summary.json": "dc1d738760fa87ef681e33a72caf67cdb68db7413b66f364f80a67b0233fdbdc",
"checksums-full-server.json": "152305bd23c849a75a0e0b5b686cadd93e5db017ea9f2733a52aa771353ed5f3",
"compute-budget-ledger.json": "33dac0f3e9aaf6a82a01c67c22f9a755dff9be8092710f5d3fa8236ce551b255",
"download.log": "9f5b76650a94f15729e949605b50ab172ff0fc3a1d791b35dd2a88d4ace02dd0",
"experiment.json": "911ba54b489547373b724bd0e719717d8f836807d8fad8aa221eadb5a525d05a",
"external-activation-cache.json": "ceb8ec04ac1ca6fe1a0466520b0c58180741339541b4ace4c5696c359ed25917",
"figures/algebraic_gain_identity.pdf": "b9c44ed9f2df66f7b9172e9a1650db31ec3915290fef3d59b77320a0510ee6de",
"figures/algebraic_gain_identity.png": "2e637bb3362ac1592959fbe8f84e67ba06b1ea00ec71dfeec80c45a7ce7520b1",
"figures/algebraic_gain_identity.svg": "553ac72babfc005f67a91564f8b737b5edd279d62fb013b8d5d87f6884c4f1ac",
"figures/candidate_function_errors.pdf": "370f9b888a6cff64d69bb5f52541e71c66695139cc0b1a45ca8115eae44aae33",
"figures/candidate_function_errors.png": "a469f33e4f0a8ba81d4cbcaa333ddcf822f6fa2e4f33ea2c0a6d7da3cbd814b9",
"figures/candidate_function_errors.svg": "a647cfcf58bd5d96770daed29cf14ffd245aef3a0c463706f3724bf84b800469",
"figures/learned_norm_function_errors.pdf": "cba6296e9297d8f14b1f3346b4ab4015845a54bc880d0f153c8d4b586fe16f90",
"figures/learned_norm_function_errors.png": "85b1bce927e1c152778b814629a52996bfdb4a0828995492f56e1fb12f68648f",
"figures/learned_norm_function_errors.svg": "ce867a8b8aedfd941dde51ae53ad2541f4e407235c2572092dca6446e71d8f8c",
"figures/learned_norm_network_substitution.pdf": "6d3440323d276b4a70209f71cd31e016d8eb78d257bdaa41ffc0468d2813a428",
"figures/learned_norm_network_substitution.png": "e0dca3ce8082abf3e904aaa27aa581bd1c7ed1ff7a19285dde613dd88b3738ad",
"figures/learned_norm_network_substitution.svg": "7753abfa3f85cda50fbc3a00491f54a597cbbb54fdb75f0c0b2fe7a7fcc131c5",
"final-report.log": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"final-source-hashes.json": "52398645bce397827c36048c417f721a1cd8f76dc38eeaee530b8a5031b09e91",
"final-tests.txt": "3eb63c08f7b8de12c4198e79da9909582d5d64cb821ef1b08d38de7e4e43eaf5",
"functional-code-hashes.json": "3756c43b0904c53d9a3ea1fe92496cc9f890cc7b033a3c9577ef53cecd4d8536",
"functional-environment.json": "b9a2b4b850b3f26cfa96efc1ac6a9470d678f17197f96a01d9c3bdf5111b148a",
"functional-report.log": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"functional-run.log": "95684864b79b72dbefdf5485fca00bca3e80719c9e925a2a6fe1a6b0a07825d3",
"functional-status.json": "fdda303b7e520f9e0065c65281df5b4bd6e9277edec52f9e5818047598a2a2ea",
"gain-factorization-k_to_q-token-nll.npy": "767f2a885532e347a654a86ed0a7f19bd4938c52296dcc74db26876acb3fe6a1",
"gain-factorization-kernel-timing.csv": "7ee790a456d4b9dac9d4756b71288f4df165c0c6111fe94657fa6804f91154d1",
"gain-factorization-optimized-summary.json": "0e303074384a3933577b2c3c29a042fe00e64cf308eb8f6190db8f7e3db55e7b",
"gain-factorization-optimized-timing.csv": "b5c85e1f9bcfb49bf317090470530a44546079c5a79c1df2365c00c36144dc10",
"gain-factorization-optimized.log": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"gain-factorization-protocol.json": "7690e4cf5dec415e414813e5beb3434ae62d51d95e68ad5de48e698980e26cab",
"gain-factorization-q_to_k-token-nll.npy": "6ea27edea56593c7269403ce5a0f7eb8673b9d80d1e00806d38bc6a2a681fff8",
"gain-factorization-score-metrics.csv": "7cfcc997980cd0566721e529f6bb1bca2f95858da50d5cf533eb31742ab0c230",
"gain-factorization-summary.json": "3bfde7dba2abd19e6dc945c05ad924181bc0927e0b88806337c9bbab9db7b359",
"gain-factorization.log": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"install-torch-disk-tmp.log": "85ee085d264707b8fd0031876c3ae9cf4e36151db9cbf53edef979f10d144c13",
"install-torch.log": "e297f77b38de146e85e6159025ea1e0ec93622fe26ef46698df180c7c96378b8",
"manifest.json": "28efbaf94a5034333339a35f08ead151459a7d20c6c2d2f07a451e7f3bf154de",
"model_config.json": "ed1c1723241f23f7f4e23430759cbd7dcfb4103cbdfe052bfe7626b57c2615b4",
"neuron-reuse-protocol.json": "cc4fcf06ebd4ee5420933534021006df6e62259b00fd922f6e8fabada89472f7",
"neuron-reuse-search-summary.json": "30a5ef914c4ff766e9bde09e3784a229bdbe7a10cea871bef3be327df8047c25",
"neuron-reuse-selection.json": "afafc7f8d35c562940f5afe47c5edaf3c5b7ca97744152bb5f9988b9910797a4",
"neuron-reuse-summary.json": "f14999d51d262c67f90e2e37918701b6b56107c7d5c47fad6da1cdfeb35d62a2",
"neuron-reuse.log": "be387fa429688a6a34131f0a5899a81586a2bde85cf209253c2da977e625168c",
"norm-repetition-12-uses-13-token-nll.npy": "fb05b796fd40122aa0a1b05ce92585c980f47a1cb1c771df4fabc720ade4bbd8",
"norm-repetition-13-uses-12-token-nll.npy": "5470a87f22f100ed8e43caf90a05831453b4277707d6d7085eecc4bd43ae8cb6",
"norm-repetition-function-metrics.csv": "d5f4eed05a9f5c6e0f971331dd9d0b834d6141b4aeb9e3c725a5bee1f4314593",
"norm-repetition-protocol.json": "ad8d2319572b4a1daa5d3ae227bd344b5f1db7941ee567b43d1f25dbd963f6a0",
"norm-repetition-selection.json": "4dae5a62da0889c0ed4de7813a8f720bb94991e468b9d6cdee8c5327d549ac55",
"norm-repetition-summary.json": "70b5067d0820b5a5bc7062c71b9f29c5590d9bb12bf538beb961135d1aab8834",
"norm-repetition-uniform-bounds.csv": "48e71a9cac9410d110483cef8356e02b2dcdeae1120bcb243ae86903852fb32a",
"norm-repetition.log": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"preflight-first-attempt.log": "1a336568f0ed99ac306ab1d690ff00f325c9c73ec6be84df4ace4374d490c650",
"preflight-second-attempt.json": "a6b68d31710e45cbf09ee100b490c781547334f76ecfbb3e09332779e953b30c",
"preflight-second-attempt.log": "42a07ef0a3fd8cb1a6475bec6e7958f754a24012fb8269073d8625b1b891594f",
"preflight.json": "86113ecbf499f4a84ea4d7fd7893e8c12f2784b6e4678b5cbcbb0e022e295e4b",
"preflight.log": "66dd7d1e0b146fc448a1064efbde5d693126d3f3cbb2fc9abdeb4485ead7c951",
"reference-check-first-attempt.log": "ed77d6619c19551401938a54fae0ef1aa50846b943ea1ba3321750c9cb985428",
"reference-check-second-attempt.log": "5923802faaba183a099d0eb313d5dc2de4825b1d52ec6ce0699a8201f68d99af",
"reference-check.json": "e82b0d8634ccae0671269587da960e06cce33e0a0e2ed15db15d946f6399d9ed",
"reference-check.log": "e4a1d55d309bff39535ecab272bd2aa6d8fe183656724b7731e4171d03150f11",
"retention-policy.json": "e3cdb6707985d2fa1fcb7f088f8f46f9eafb89eea13e37f51c554f57790acd6e",
"samples-tokenized.json": "2e56e14a4eefedd702bfe09d3edb2f17f360f53dd420e29fcf0c0ec1affc0184",
"selection.json": "c8c86d12fe7db6fb8fffca8078ec0ff5438b97998088c89862db8f9b6152daa3",
"source_metadata.json": "d3be2a8560f4270b8af27891d1076cb7cf80bd84c4c19fe7790d5fb06f3a342e",
"storage-accounting.log": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"storage-evidence.json": "c6968f4143910cef9d1359d3cd3342d289b9c791c6e2a6ab598fe235b105e0d9",
"supervisor-error.txt": "4cba26d15e963620745ab3d4625f4de967859a9c43993f78ca1ce2e5c06aeb42",
"supervisor-first-attempt-error.txt": "e689f94e983b84cfc4e6b372175826d77dcd2d5c9b8d166598bc8607227afbba",
"supervisor-first-attempt.json": "93d7185df1810aec2aa396fd8247e188b8a37a821731ef681e66389695bd9a8e",
"supervisor-second-attempt.json": "7333a3634f7d169bc78d9eeb5e47d8fbd2970c8401b4d6160b92038ad2a89f55",
"supervisor-status.json": "ae078d540c9c683bac468313a85754d8ecf1a88be482a2d5895bbfca0c610404",
"supervisor-third-attempt.json": "e86ce8e2f55f75ab3c61ffd7b4a1517e627cebbdb85ee7f966615c5d683722ed",
"supervisor.log": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"tokenizer-download.log": "1d9c8b608f3c79c9d7af93c593543ddd14db3e077e2c108738bccd518df9a79a",
"tokenizers-wheel.log": "f0b4f7c1360ff1a00e04fe443e619682c7d7b024df55545d31f54951b3001480",
"unit-tests.txt": "fa8ffc7277ea722b9261da462b797710ed4ae3eb8975ad51937c419b3052e3af"
}
@@ -0,0 +1,54 @@
{
"spent_seconds": 1167.355532555841,
"initial_reserve_reason": "Conservative reserve for early reference checks and stopped preflight, not a measured runtime",
"calls": [
{
"module": "functional.reference_check",
"seconds": 6.833657197188586
},
{
"module": "functional.preflight",
"seconds": 27.313251950312406
},
{
"module": "functional.report",
"seconds": 2.1849583038128912
},
{
"module": "functional.reference_check",
"seconds": 16.264290761202574
},
{
"module": "functional.preflight",
"seconds": 58.00242758495733
},
{
"module": "functional.run",
"seconds": 204.33689004788175
},
{
"module": "functional.storage",
"seconds": 1.2859071069397032
},
{
"module": "functional.report",
"seconds": 10.058376160915941
},
{
"module": "functional.gain_factorization",
"seconds": 232.08875080291182
},
{
"module": "functional.benchmark_factorization",
"seconds": 4.185962562914938
},
{
"module": "functional.neuron_reuse",
"seconds": 55.004304428584874
},
{
"module": "functional.norm_repetition",
"seconds": 369.7967556482181
}
]
}
+73
View File
@@ -0,0 +1,73 @@
Downloading README.md, expected 62814 bytes
Downloading config.json, expected 2908 bytes
Downloading model.safetensors-00001-of-00001.safetensors, expected 4548221488 bytes
Weight download 67108864/4548221488 bytes (1.5%)
Weight download 134217728/4548221488 bytes (3.0%)
Weight download 201326592/4548221488 bytes (4.4%)
Weight download 268435456/4548221488 bytes (5.9%)
Weight download 335544320/4548221488 bytes (7.4%)
Weight download 402653184/4548221488 bytes (8.9%)
Weight download 469762048/4548221488 bytes (10.3%)
Weight download 536870912/4548221488 bytes (11.8%)
Weight download 603979776/4548221488 bytes (13.3%)
Weight download 671088640/4548221488 bytes (14.8%)
Weight download 738197504/4548221488 bytes (16.2%)
Weight download 805306368/4548221488 bytes (17.7%)
Weight download 872415232/4548221488 bytes (19.2%)
Weight download 939524096/4548221488 bytes (20.7%)
Weight download 1006632960/4548221488 bytes (22.1%)
Weight download 1073741824/4548221488 bytes (23.6%)
Weight download 1140850688/4548221488 bytes (25.1%)
Weight download 1207959552/4548221488 bytes (26.6%)
Weight download 1275068416/4548221488 bytes (28.0%)
Weight download 1342177280/4548221488 bytes (29.5%)
Weight download 1409286144/4548221488 bytes (31.0%)
Weight download 1476395008/4548221488 bytes (32.5%)
Weight download 1543503872/4548221488 bytes (33.9%)
Weight download 1610612736/4548221488 bytes (35.4%)
Weight download 1677721600/4548221488 bytes (36.9%)
Weight download 1744830464/4548221488 bytes (38.4%)
Weight download 1811939328/4548221488 bytes (39.8%)
Weight download 1879048192/4548221488 bytes (41.3%)
Weight download 1946157056/4548221488 bytes (42.8%)
Weight download 2013265920/4548221488 bytes (44.3%)
Weight download 2080374784/4548221488 bytes (45.7%)
Weight download 2147483648/4548221488 bytes (47.2%)
Weight download 2214592512/4548221488 bytes (48.7%)
Weight download 2281701376/4548221488 bytes (50.2%)
Weight download 2348810240/4548221488 bytes (51.6%)
Weight download 2415919104/4548221488 bytes (53.1%)
Weight download 2483027968/4548221488 bytes (54.6%)
Weight download 2550136832/4548221488 bytes (56.1%)
Weight download 2617245696/4548221488 bytes (57.5%)
Weight download 2684354560/4548221488 bytes (59.0%)
Weight download 2751463424/4548221488 bytes (60.5%)
Weight download 2818572288/4548221488 bytes (62.0%)
Weight download 2885681152/4548221488 bytes (63.4%)
Weight download 2952790016/4548221488 bytes (64.9%)
Weight download 3019898880/4548221488 bytes (66.4%)
Weight download 3087007744/4548221488 bytes (67.9%)
Weight download 3154116608/4548221488 bytes (69.3%)
Weight download 3221225472/4548221488 bytes (70.8%)
Weight download 3288334336/4548221488 bytes (72.3%)
Weight download 3355443200/4548221488 bytes (73.8%)
Weight download 3422552064/4548221488 bytes (75.3%)
Weight download 3489660928/4548221488 bytes (76.7%)
Weight download 3556769792/4548221488 bytes (78.2%)
Weight download 3623878656/4548221488 bytes (79.7%)
Weight download 3690987520/4548221488 bytes (81.2%)
Weight download 3758096384/4548221488 bytes (82.6%)
Weight download 3825205248/4548221488 bytes (84.1%)
Weight download 3892314112/4548221488 bytes (85.6%)
Weight download 3959422976/4548221488 bytes (87.1%)
Weight download 4026531840/4548221488 bytes (88.5%)
Weight download 4093640704/4548221488 bytes (90.0%)
Weight download 4160749568/4548221488 bytes (91.5%)
Weight download 4227858432/4548221488 bytes (93.0%)
Weight download 4294967296/4548221488 bytes (94.4%)
Weight download 4346894896/4548221488 bytes (95.6%)
Weight download 4414003760/4548221488 bytes (97.0%)
Weight download 4481112624/4548221488 bytes (98.5%)
Weight download 4548221488/4548221488 bytes (100.0%)
Downloading model.safetensors.index.json, expected 64460 bytes
{"revision": "15852e8c16360a2fea060d615a32b45270f8a8fc", "bytes": 4548351670}
+49
View File
@@ -0,0 +1,49 @@
{
"revision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"attention_pairs": [
[
3,
7
],
[
11,
15
],
[
19,
23
]
],
"mlp_pairs": [
[
5,
6
],
[
20,
21
]
],
"norm_layers": [
3,
7,
11,
15,
19,
23
],
"seconds": 6849,
"output_relative_l2_limit": 0.05,
"token_p95_limit": 0.1,
"attention_js_limit": 0.01,
"attention_top5_min": 0.8,
"heldout_nll_mean_delta_limit": 0.01,
"heldout_nll_sequence_p95_delta_limit": 0.05,
"max_intervention_candidates": 1,
"directions_per_selected_candidate": 2,
"training": false,
"search_R": false,
"automatic_weight_cleanup": false,
"dtype": "original weights decoded to FP32; not a BF16 backend benchmark",
"blas_threads": "1"
}
@@ -0,0 +1,31 @@
{
"host": "8.211.154.134",
"files": [
{
"name": "baseline-activations.npz",
"server_path": "/opt/exp-model-parameter-similarity/runs/cpu-qwen35-2b/baseline-activations.npz",
"bytes": 38802964,
"sha256": "a851b489a8917a63b07cb6615a47fd1399b36e62f62d6140be618c4966d204ef",
"retained": true,
"reproducible_by": "functional.run / functional.neuron_reuse / functional.norm_repetition"
},
{
"name": "neuron-reuse-activations.npz",
"server_path": "/opt/exp-model-parameter-similarity/runs/cpu-qwen35-2b/neuron-reuse-activations.npz",
"bytes": 25166826,
"sha256": "0ce0b7ba4efef0164a68d91595e8eb47a6347f1f364eeda764835cfa09b09448",
"retained": true,
"reproducible_by": "functional.run / functional.neuron_reuse / functional.norm_repetition"
},
{
"name": "norm-repetition-input-profiles.npz",
"server_path": "/opt/exp-model-parameter-similarity/runs/cpu-qwen35-2b/norm-repetition-input-profiles.npz",
"bytes": 4194818,
"sha256": "7e3415b8f28d8844658facf2c37415cc0f3a4de8f6d39d3e0c4b01e1b3a4d7e0",
"retained": true,
"reproducible_by": "functional.run / functional.neuron_reuse / functional.norm_repetition"
}
],
"not_model_weights": true,
"reason": "Large intermediate caches retained on the authorized server; all figure source tables and token NLL arrays are delivered in Git."
}
Binary file not shown.

After

Width:  |  Height:  |  Size: 135 KiB

@@ -0,0 +1,581 @@
<?xml version="1.0" encoding="utf-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<svg xmlns:xlink="http://www.w3.org/1999/xlink" width="712.424062pt" height="285.518438pt" viewBox="0 0 712.424062 285.518438" xmlns="http://www.w3.org/2000/svg" version="1.1">
<metadata>
<rdf:RDF xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:cc="http://creativecommons.org/ns#" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
<cc:Work>
<dc:type rdf:resource="http://purl.org/dc/dcmitype/StillImage"/>
<dc:date>2026-11-09T14:11:02.311213</dc:date>
<dc:format>image/svg+xml</dc:format>
<dc:creator>
<cc:Agent>
<dc:title>Matplotlib v3.11.2, https://matplotlib.org/</dc:title>
</cc:Agent>
</dc:creator>
</cc:Work>
</rdf:RDF>
</metadata>
<defs>
<style type="text/css">*{stroke-linejoin: round; stroke-linecap: butt}</style>
</defs>
<g id="figure_1">
<g id="patch_1">
<path d="M 0 285.518438
L 712.424062 285.518438
L 712.424062 -0
L 0 -0
z
" style="fill: #ffffff"/>
</g>
<g id="axes_1">
<g id="patch_2">
<path d="M 50.466406 247.317656
L 353.864062 247.317656
L 353.864062 45.880469
L 50.466406 45.880469
z
" style="fill: #ffffff"/>
</g>
<g id="matplotlib.axis_1">
<g id="xtick_1">
<g id="line2d_1">
<defs>
<path id="m50faf94306" d="M 0 0
L 0 3.5
" style="stroke: #000000; stroke-width: 0.8"/>
</defs>
<g>
<use xlink:href="#m50faf94306" x="64.257209" y="247.317656" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_1">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: middle" x="64.257209" y="261.915313" transform="rotate(-0 64.257209 261.915313)">3</text>
</g>
</g>
<g id="xtick_2">
<g id="line2d_2">
<g>
<use xlink:href="#m50faf94306" x="119.420419" y="247.317656" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_2">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: middle" x="119.420419" y="261.915313" transform="rotate(-0 119.420419 261.915313)">7</text>
</g>
</g>
<g id="xtick_3">
<g id="line2d_3">
<g>
<use xlink:href="#m50faf94306" x="174.583629" y="247.317656" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_3">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: middle" x="174.583629" y="261.915313" transform="rotate(-0 174.583629 261.915313)">11</text>
</g>
</g>
<g id="xtick_4">
<g id="line2d_4">
<g>
<use xlink:href="#m50faf94306" x="229.746839" y="247.317656" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_4">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: middle" x="229.746839" y="261.915313" transform="rotate(-0 229.746839 261.915313)">15</text>
</g>
</g>
<g id="xtick_5">
<g id="line2d_5">
<g>
<use xlink:href="#m50faf94306" x="284.91005" y="247.317656" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_5">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: middle" x="284.91005" y="261.915313" transform="rotate(-0 284.91005 261.915313)">19</text>
</g>
</g>
<g id="xtick_6">
<g id="line2d_6">
<g>
<use xlink:href="#m50faf94306" x="340.07326" y="247.317656" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_6">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: middle" x="340.07326" y="261.915313" transform="rotate(-0 340.07326 261.915313)">23</text>
</g>
</g>
<g id="text_7">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: middle" x="202.165234" y="275.916094" transform="rotate(-0 202.165234 275.916094)">Full-attention layer index</text>
</g>
</g>
<g id="matplotlib.axis_2">
<g id="ytick_1">
<g id="line2d_7">
<defs>
<path id="m9a2e91a9aa" d="M 0 0
L -3.5 0
" style="stroke: #000000; stroke-width: 0.8"/>
</defs>
<g>
<use xlink:href="#m9a2e91a9aa" x="50.466406" y="229.265954" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_8">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: end" x="43.466406" y="233.064782" transform="rotate(-0 43.466406 233.064782)">2.00</text>
</g>
</g>
<g id="ytick_2">
<g id="line2d_8">
<g>
<use xlink:href="#m9a2e91a9aa" x="50.466406" y="205.263431" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_9">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: end" x="43.466406" y="209.06226" transform="rotate(-0 43.466406 209.06226)">2.25</text>
</g>
</g>
<g id="ytick_3">
<g id="line2d_9">
<g>
<use xlink:href="#m9a2e91a9aa" x="50.466406" y="181.260909" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_10">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: end" x="43.466406" y="185.059737" transform="rotate(-0 43.466406 185.059737)">2.50</text>
</g>
</g>
<g id="ytick_4">
<g id="line2d_10">
<g>
<use xlink:href="#m9a2e91a9aa" x="50.466406" y="157.258386" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_11">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: end" x="43.466406" y="161.057214" transform="rotate(-0 43.466406 161.057214)">2.75</text>
</g>
</g>
<g id="ytick_5">
<g id="line2d_11">
<g>
<use xlink:href="#m9a2e91a9aa" x="50.466406" y="133.255863" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_12">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: end" x="43.466406" y="137.054691" transform="rotate(-0 43.466406 137.054691)">3.00</text>
</g>
</g>
<g id="ytick_6">
<g id="line2d_12">
<g>
<use xlink:href="#m9a2e91a9aa" x="50.466406" y="109.25334" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_13">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: end" x="43.466406" y="113.052169" transform="rotate(-0 43.466406 113.052169)">3.25</text>
</g>
</g>
<g id="ytick_7">
<g id="line2d_13">
<g>
<use xlink:href="#m9a2e91a9aa" x="50.466406" y="85.250818" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_14">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: end" x="43.466406" y="89.049646" transform="rotate(-0 43.466406 89.049646)">3.50</text>
</g>
</g>
<g id="ytick_8">
<g id="line2d_14">
<g>
<use xlink:href="#m9a2e91a9aa" x="50.466406" y="61.248295" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_15">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: end" x="43.466406" y="65.047123" transform="rotate(-0 43.466406 65.047123)">3.75</text>
</g>
</g>
<g id="text_16">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: middle" x="14.798438" y="146.599063" transform="rotate(-90 14.798438 146.599063)">Maximum absolute score error</text>
</g>
<g id="text_17">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: start" x="50.466406" y="42.880469" transform="rotate(-0 50.466406 42.880469)">1e−6</text>
</g>
</g>
<g id="line2d_15">
<path d="M 64.257209 238.16142
L 119.420419 238.16142
L 174.583629 238.16142
L 229.746839 146.599063
L 284.91005 55.036705
L 340.07326 55.036705
" clip-path="url(#p0542a2a98b)" style="fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square"/>
<defs>
<path id="m9b4cc9d8d2" d="M 0 3
C 0.795609 3 1.55874 2.683901 2.12132 2.12132
C 2.683901 1.55874 3 0.795609 3 0
C 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132
C 1.55874 -2.683901 0.795609 -3 0 -3
C -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132
C -2.683901 -1.55874 -3 -0.795609 -3 0
C -3 0.795609 -2.683901 1.55874 -2.12132 2.12132
C -1.55874 2.683901 -0.795609 3 0 3
z
" style="stroke: #1f77b4"/>
</defs>
<g clip-path="url(#p0542a2a98b)">
<use xlink:href="#m9b4cc9d8d2" x="64.257209" y="238.16142" style="fill: #1f77b4; stroke: #1f77b4"/>
<use xlink:href="#m9b4cc9d8d2" x="119.420419" y="238.16142" style="fill: #1f77b4; stroke: #1f77b4"/>
<use xlink:href="#m9b4cc9d8d2" x="174.583629" y="238.16142" style="fill: #1f77b4; stroke: #1f77b4"/>
<use xlink:href="#m9b4cc9d8d2" x="229.746839" y="146.599063" style="fill: #1f77b4; stroke: #1f77b4"/>
<use xlink:href="#m9b4cc9d8d2" x="284.91005" y="55.036705" style="fill: #1f77b4; stroke: #1f77b4"/>
<use xlink:href="#m9b4cc9d8d2" x="340.07326" y="55.036705" style="fill: #1f77b4; stroke: #1f77b4"/>
</g>
</g>
<g id="line2d_16">
<path d="M 64.257209 238.16142
L 119.420419 238.16142
L 174.583629 146.599063
L 229.746839 238.16142
L 284.91005 238.16142
L 340.07326 55.036705
" clip-path="url(#p0542a2a98b)" style="fill: none; stroke: #ff7f0e; stroke-width: 1.5; stroke-linecap: square"/>
<defs>
<path id="m556f16804f" d="M 0 3
C 0.795609 3 1.55874 2.683901 2.12132 2.12132
C 2.683901 1.55874 3 0.795609 3 0
C 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132
C 1.55874 -2.683901 0.795609 -3 0 -3
C -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132
C -2.683901 -1.55874 -3 -0.795609 -3 0
C -3 0.795609 -2.683901 1.55874 -2.12132 2.12132
C -1.55874 2.683901 -0.795609 3 0 3
z
" style="stroke: #ff7f0e"/>
</defs>
<g clip-path="url(#p0542a2a98b)">
<use xlink:href="#m556f16804f" x="64.257209" y="238.16142" style="fill: #ff7f0e; stroke: #ff7f0e"/>
<use xlink:href="#m556f16804f" x="119.420419" y="238.16142" style="fill: #ff7f0e; stroke: #ff7f0e"/>
<use xlink:href="#m556f16804f" x="174.583629" y="146.599063" style="fill: #ff7f0e; stroke: #ff7f0e"/>
<use xlink:href="#m556f16804f" x="229.746839" y="238.16142" style="fill: #ff7f0e; stroke: #ff7f0e"/>
<use xlink:href="#m556f16804f" x="284.91005" y="238.16142" style="fill: #ff7f0e; stroke: #ff7f0e"/>
<use xlink:href="#m556f16804f" x="340.07326" y="55.036705" style="fill: #ff7f0e; stroke: #ff7f0e"/>
</g>
</g>
<g id="patch_3">
<path d="M 50.466406 247.317656
L 50.466406 45.880469
" style="fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square"/>
</g>
<g id="patch_4">
<path d="M 353.864062 247.317656
L 353.864062 45.880469
" style="fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square"/>
</g>
<g id="patch_5">
<path d="M 50.466406 247.317656
L 353.864062 247.317656
" style="fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square"/>
</g>
<g id="patch_6">
<path d="M 50.466406 45.880469
L 353.864062 45.880469
" style="fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square"/>
</g>
<g id="legend_1">
<g id="patch_7">
<path d="M 56.066406 76.281719
L 107.408906 76.281719
Q 109.008906 76.281719 109.008906 74.681719
L 109.008906 51.480469
Q 109.008906 49.880469 107.408906 49.880469
L 56.066406 49.880469
Q 54.466406 49.880469 54.466406 51.480469
L 54.466406 74.681719
Q 54.466406 76.281719 56.066406 76.281719
z
" style="fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter"/>
</g>
<g id="line2d_17">
<path d="M 57.666406 56.359219
L 65.666406 56.359219
L 73.666406 56.359219
" style="fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square"/>
<g>
<use xlink:href="#m9b4cc9d8d2" x="65.666406" y="56.359219" style="fill: #1f77b4; stroke: #1f77b4"/>
</g>
</g>
<g id="text_18">
<text style="font-size: 8px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: start" x="80.066406" y="59.159219" transform="rotate(-0 80.066406 59.159219)">q_to_k</text>
</g>
<g id="line2d_18">
<path d="M 57.666406 68.359844
L 65.666406 68.359844
L 73.666406 68.359844
" style="fill: none; stroke: #ff7f0e; stroke-width: 1.5; stroke-linecap: square"/>
<g>
<use xlink:href="#m556f16804f" x="65.666406" y="68.359844" style="fill: #ff7f0e; stroke: #ff7f0e"/>
</g>
</g>
<g id="text_19">
<text style="font-size: 8px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: start" x="80.066406" y="71.159844" transform="rotate(-0 80.066406 71.159844)">k_to_q</text>
</g>
</g>
</g>
<g id="axes_2">
<g id="patch_8">
<path d="M 401.826406 247.317656
L 705.224062 247.317656
L 705.224062 45.880469
L 401.826406 45.880469
z
" style="fill: #ffffff"/>
</g>
<g id="matplotlib.axis_3">
<g id="xtick_7">
<g id="line2d_19">
<g>
<use xlink:href="#m50faf94306" x="415.617209" y="247.317656" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_20">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: middle" x="415.617209" y="261.915313" transform="rotate(-0 415.617209 261.915313)">3</text>
</g>
</g>
<g id="xtick_8">
<g id="line2d_20">
<g>
<use xlink:href="#m50faf94306" x="470.780419" y="247.317656" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_21">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: middle" x="470.780419" y="261.915313" transform="rotate(-0 470.780419 261.915313)">7</text>
</g>
</g>
<g id="xtick_9">
<g id="line2d_21">
<g>
<use xlink:href="#m50faf94306" x="525.943629" y="247.317656" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_22">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: middle" x="525.943629" y="261.915313" transform="rotate(-0 525.943629 261.915313)">11</text>
</g>
</g>
<g id="xtick_10">
<g id="line2d_22">
<g>
<use xlink:href="#m50faf94306" x="581.106839" y="247.317656" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_23">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: middle" x="581.106839" y="261.915313" transform="rotate(-0 581.106839 261.915313)">15</text>
</g>
</g>
<g id="xtick_11">
<g id="line2d_23">
<g>
<use xlink:href="#m50faf94306" x="636.27005" y="247.317656" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_24">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: middle" x="636.27005" y="261.915313" transform="rotate(-0 636.27005 261.915313)">19</text>
</g>
</g>
<g id="xtick_12">
<g id="line2d_24">
<g>
<use xlink:href="#m50faf94306" x="691.43326" y="247.317656" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_25">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: middle" x="691.43326" y="261.915313" transform="rotate(-0 691.43326 261.915313)">23</text>
</g>
</g>
<g id="text_26">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: middle" x="553.525234" y="275.916094" transform="rotate(-0 553.525234 275.916094)">Full-attention layer index</text>
</g>
</g>
<g id="matplotlib.axis_4">
<g id="ytick_9">
<g id="line2d_25">
<g>
<use xlink:href="#m9a2e91a9aa" x="401.826406" y="242.241146" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_27">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: end" x="394.826406" y="246.039974" transform="rotate(-0 394.826406 246.039974)">5.0</text>
</g>
</g>
<g id="ytick_10">
<g id="line2d_26">
<g>
<use xlink:href="#m9a2e91a9aa" x="401.826406" y="211.517917" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_28">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: end" x="394.826406" y="215.316745" transform="rotate(-0 394.826406 215.316745)">5.5</text>
</g>
</g>
<g id="ytick_11">
<g id="line2d_27">
<g>
<use xlink:href="#m9a2e91a9aa" x="401.826406" y="180.794688" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_29">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: end" x="394.826406" y="184.593516" transform="rotate(-0 394.826406 184.593516)">6.0</text>
</g>
</g>
<g id="ytick_12">
<g id="line2d_28">
<g>
<use xlink:href="#m9a2e91a9aa" x="401.826406" y="150.071459" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_30">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: end" x="394.826406" y="153.870287" transform="rotate(-0 394.826406 153.870287)">6.5</text>
</g>
</g>
<g id="ytick_13">
<g id="line2d_29">
<g>
<use xlink:href="#m9a2e91a9aa" x="401.826406" y="119.34823" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_31">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: end" x="394.826406" y="123.147058" transform="rotate(-0 394.826406 123.147058)">7.0</text>
</g>
</g>
<g id="ytick_14">
<g id="line2d_30">
<g>
<use xlink:href="#m9a2e91a9aa" x="401.826406" y="88.625" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_32">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: end" x="394.826406" y="92.423829" transform="rotate(-0 394.826406 92.423829)">7.5</text>
</g>
</g>
<g id="ytick_15">
<g id="line2d_31">
<g>
<use xlink:href="#m9a2e91a9aa" x="401.826406" y="57.901771" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_33">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: end" x="394.826406" y="61.700599" transform="rotate(-0 394.826406 61.700599)">8.0</text>
</g>
</g>
<g id="text_34">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: middle" x="372.520938" y="146.599063" transform="rotate(-90 372.520938 146.599063)">Maximum absolute probability error</text>
</g>
<g id="text_35">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: start" x="401.826406" y="42.880469" transform="rotate(-0 401.826406 42.880469)">1e−7</text>
</g>
</g>
<g id="line2d_32">
<path d="M 415.617209 238.16142
L 470.780419 55.036705
L 525.943629 146.599063
L 581.106839 109.974119
L 636.27005 183.224006
L 691.43326 219.848949
" clip-path="url(#p601cd532af)" style="fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square"/>
<g clip-path="url(#p601cd532af)">
<use xlink:href="#m9b4cc9d8d2" x="415.617209" y="238.16142" style="fill: #1f77b4; stroke: #1f77b4"/>
<use xlink:href="#m9b4cc9d8d2" x="470.780419" y="55.036705" style="fill: #1f77b4; stroke: #1f77b4"/>
<use xlink:href="#m9b4cc9d8d2" x="525.943629" y="146.599063" style="fill: #1f77b4; stroke: #1f77b4"/>
<use xlink:href="#m9b4cc9d8d2" x="581.106839" y="109.974119" style="fill: #1f77b4; stroke: #1f77b4"/>
<use xlink:href="#m9b4cc9d8d2" x="636.27005" y="183.224006" style="fill: #1f77b4; stroke: #1f77b4"/>
<use xlink:href="#m9b4cc9d8d2" x="691.43326" y="219.848949" style="fill: #1f77b4; stroke: #1f77b4"/>
</g>
</g>
<g id="line2d_33">
<path d="M 415.617209 219.848949
L 470.780419 183.224006
L 525.943629 128.286591
L 581.106839 109.974119
L 636.27005 183.224006
L 691.43326 146.599063
" clip-path="url(#p601cd532af)" style="fill: none; stroke: #ff7f0e; stroke-width: 1.5; stroke-linecap: square"/>
<g clip-path="url(#p601cd532af)">
<use xlink:href="#m556f16804f" x="415.617209" y="219.848949" style="fill: #ff7f0e; stroke: #ff7f0e"/>
<use xlink:href="#m556f16804f" x="470.780419" y="183.224006" style="fill: #ff7f0e; stroke: #ff7f0e"/>
<use xlink:href="#m556f16804f" x="525.943629" y="128.286591" style="fill: #ff7f0e; stroke: #ff7f0e"/>
<use xlink:href="#m556f16804f" x="581.106839" y="109.974119" style="fill: #ff7f0e; stroke: #ff7f0e"/>
<use xlink:href="#m556f16804f" x="636.27005" y="183.224006" style="fill: #ff7f0e; stroke: #ff7f0e"/>
<use xlink:href="#m556f16804f" x="691.43326" y="146.599063" style="fill: #ff7f0e; stroke: #ff7f0e"/>
</g>
</g>
<g id="patch_9">
<path d="M 401.826406 247.317656
L 401.826406 45.880469
" style="fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square"/>
</g>
<g id="patch_10">
<path d="M 705.224062 247.317656
L 705.224062 45.880469
" style="fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square"/>
</g>
<g id="patch_11">
<path d="M 401.826406 247.317656
L 705.224062 247.317656
" style="fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square"/>
</g>
<g id="patch_12">
<path d="M 401.826406 45.880469
L 705.224062 45.880469
" style="fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square"/>
</g>
<g id="legend_2">
<g id="patch_13">
<path d="M 407.426406 76.281719
L 458.768906 76.281719
Q 460.368906 76.281719 460.368906 74.681719
L 460.368906 51.480469
Q 460.368906 49.880469 458.768906 49.880469
L 407.426406 49.880469
Q 405.826406 49.880469 405.826406 51.480469
L 405.826406 74.681719
Q 405.826406 76.281719 407.426406 76.281719
z
" style="fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter"/>
</g>
<g id="line2d_34">
<path d="M 409.026406 56.359219
L 417.026406 56.359219
L 425.026406 56.359219
" style="fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square"/>
<g>
<use xlink:href="#m9b4cc9d8d2" x="417.026406" y="56.359219" style="fill: #1f77b4; stroke: #1f77b4"/>
</g>
</g>
<g id="text_36">
<text style="font-size: 8px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: start" x="431.426406" y="59.159219" transform="rotate(-0 431.426406 59.159219)">q_to_k</text>
</g>
<g id="line2d_35">
<path d="M 409.026406 68.359844
L 417.026406 68.359844
L 425.026406 68.359844
" style="fill: none; stroke: #ff7f0e; stroke-width: 1.5; stroke-linecap: square"/>
<g>
<use xlink:href="#m556f16804f" x="417.026406" y="68.359844" style="fill: #ff7f0e; stroke: #ff7f0e"/>
</g>
</g>
<g id="text_37">
<text style="font-size: 8px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: start" x="431.426406" y="71.159844" transform="rotate(-0 431.426406 71.159844)">k_to_q</text>
</g>
</g>
</g>
<g id="text_38">
<text style="font-size: 12px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: middle" x="356.024063" y="16.318125" transform="rotate(-0 356.024063 16.318125)">Supplement: algebraic reparameterization, separate from learned-function repetition</text>
</g>
</g>
<defs>
<clipPath id="p0542a2a98b">
<rect x="50.466406" y="45.880469" width="303.397656" height="201.437187"/>
</clipPath>
<clipPath id="p601cd532af">
<rect x="401.826406" y="45.880469" width="303.397656" height="201.437187"/>
</clipPath>
</defs>
</svg>

After

Width:  |  Height:  |  Size: 29 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 79 KiB

@@ -0,0 +1,467 @@
<?xml version="1.0" encoding="utf-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<svg xmlns:xlink="http://www.w3.org/1999/xlink" width="712.218437pt" height="280.996571pt" viewBox="0 0 712.218437 280.996571" xmlns="http://www.w3.org/2000/svg" version="1.1">
<metadata>
<rdf:RDF xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:cc="http://creativecommons.org/ns#" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
<cc:Work>
<dc:type rdf:resource="http://purl.org/dc/dcmitype/StillImage"/>
<dc:date>2026-11-09T14:10:51.395327</dc:date>
<dc:format>image/svg+xml</dc:format>
<dc:creator>
<cc:Agent>
<dc:title>Matplotlib v3.11.2, https://matplotlib.org/</dc:title>
</cc:Agent>
</dc:creator>
</cc:Work>
</rdf:RDF>
</metadata>
<defs>
<style type="text/css">*{stroke-linejoin: round; stroke-linecap: butt}</style>
</defs>
<g id="figure_1">
<g id="patch_1">
<path d="M 0 280.996571
L 712.218437 280.996571
L 712.218437 0
L 0 0
z
" style="fill: #ffffff"/>
</g>
<g id="axes_1">
<g id="patch_2">
<path d="M 51.700781 195.511975
L 705.018437 195.511975
L 705.018437 37.922578
L 51.700781 37.922578
z
" style="fill: #ffffff"/>
</g>
<g id="patch_3">
<path d="M 81.397038 76335.575007
L 125.391493 76335.575007
L 125.391493 64.33392
L 81.397038 64.33392
z
" clip-path="url(#pf9e7526e4c)" style="fill: #9e9e9e"/>
</g>
<g id="patch_4">
<path d="M 136.390107 76335.575007
L 180.384562 76335.575007
L 180.384562 45.085733
L 136.390107 45.085733
z
" clip-path="url(#pf9e7526e4c)" style="fill: #9e9e9e"/>
</g>
<g id="patch_5">
<path d="M 191.383176 76335.575007
L 235.377631 76335.575007
L 235.377631 45.525425
L 191.383176 45.525425
z
" clip-path="url(#pf9e7526e4c)" style="fill: #9e9e9e"/>
</g>
<g id="patch_6">
<path d="M 246.376244 76335.575007
L 290.370699 76335.575007
L 290.370699 76.171564
L 246.376244 76.171564
z
" clip-path="url(#pf9e7526e4c)" style="fill: #9e9e9e"/>
</g>
<g id="patch_7">
<path d="M 301.369313 76335.575007
L 345.363768 76335.575007
L 345.363768 76.722901
L 301.369313 76.722901
z
" clip-path="url(#pf9e7526e4c)" style="fill: #9e9e9e"/>
</g>
<g id="patch_8">
<path d="M 356.362382 76335.575007
L 400.356837 76335.575007
L 400.356837 136.725903
L 356.362382 136.725903
z
" clip-path="url(#pf9e7526e4c)" style="fill: #9e9e9e"/>
</g>
<g id="patch_9">
<path d="M 411.355451 76335.575007
L 455.349906 76335.575007
L 455.349906 141.06981
L 411.355451 141.06981
z
" clip-path="url(#pf9e7526e4c)" style="fill: #9e9e9e"/>
</g>
<g id="patch_10">
<path d="M 466.348519 76335.575007
L 510.342974 76335.575007
L 510.342974 140.370183
L 466.348519 140.370183
z
" clip-path="url(#pf9e7526e4c)" style="fill: #9e9e9e"/>
</g>
<g id="patch_11">
<path d="M 521.341588 76335.575007
L 565.336043 76335.575007
L 565.336043 147.505598
L 521.341588 147.505598
z
" clip-path="url(#pf9e7526e4c)" style="fill: #9e9e9e"/>
</g>
<g id="patch_12">
<path d="M 576.334657 76335.575007
L 620.329112 76335.575007
L 620.329112 123.343194
L 576.334657 123.343194
z
" clip-path="url(#pf9e7526e4c)" style="fill: #9e9e9e"/>
</g>
<g id="patch_13">
<path d="M 631.327725 76335.575007
L 675.32218 76335.575007
L 675.32218 109.03712
L 631.327725 109.03712
z
" clip-path="url(#pf9e7526e4c)" style="fill: #9e9e9e"/>
</g>
<g id="matplotlib.axis_1">
<g id="xtick_1">
<g id="line2d_1">
<defs>
<path id="mc7fdb9fc78" d="M 0 0
L 0 3.5
" style="stroke: #000000; stroke-width: 0.8"/>
</defs>
<g>
<use xlink:href="#mc7fdb9fc78" x="103.394266" y="195.511975" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_1">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif" transform="translate(71.267864 238.312018) rotate(-45)">attn_3_7</text>
</g>
</g>
<g id="xtick_2">
<g id="line2d_2">
<g>
<use xlink:href="#mc7fdb9fc78" x="158.387335" y="195.511975" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_2">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif" transform="translate(117.262999 247.309952) rotate(-45)">attn_11_15</text>
</g>
</g>
<g id="xtick_3">
<g id="line2d_3">
<g>
<use xlink:href="#mc7fdb9fc78" x="213.380403" y="195.511975" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_3">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif" transform="translate(172.256067 247.309952) rotate(-45)">attn_19_23</text>
</g>
</g>
<g id="xtick_4">
<g id="line2d_4">
<g>
<use xlink:href="#mc7fdb9fc78" x="268.373472" y="195.511975" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_4">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif" transform="translate(237.264641 237.295) rotate(-45)">mlp_5_6</text>
</g>
</g>
<g id="xtick_5">
<g id="line2d_5">
<g>
<use xlink:href="#mc7fdb9fc78" x="323.366541" y="195.511975" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_5">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif" transform="translate(283.259775 246.292933) rotate(-45)">mlp_20_21</text>
</g>
</g>
<g id="xtick_6">
<g id="line2d_6">
<g>
<use xlink:href="#mc7fdb9fc78" x="378.359609" y="195.511975" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_6">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif" transform="translate(338.028559 246.517219) rotate(-45)">qk_norm_3</text>
</g>
</g>
<g id="xtick_7">
<g id="line2d_7">
<g>
<use xlink:href="#mc7fdb9fc78" x="433.352678" y="195.511975" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_7">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif" transform="translate(393.021627 246.517219) rotate(-45)">qk_norm_7</text>
</g>
</g>
<g id="xtick_8">
<g id="line2d_8">
<g>
<use xlink:href="#mc7fdb9fc78" x="488.345747" y="195.511975" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_8">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif" transform="translate(443.515729 251.016186) rotate(-45)">qk_norm_11</text>
</g>
</g>
<g id="xtick_9">
<g id="line2d_9">
<g>
<use xlink:href="#mc7fdb9fc78" x="543.338815" y="195.511975" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_9">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif" transform="translate(498.508798 251.016186) rotate(-45)">qk_norm_15</text>
</g>
</g>
<g id="xtick_10">
<g id="line2d_10">
<g>
<use xlink:href="#mc7fdb9fc78" x="598.331884" y="195.511975" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_10">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif" transform="translate(553.501867 251.016186) rotate(-45)">qk_norm_19</text>
</g>
</g>
<g id="xtick_11">
<g id="line2d_11">
<g>
<use xlink:href="#mc7fdb9fc78" x="653.324953" y="195.511975" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_11">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif" transform="translate(608.494935 251.016186) rotate(-45)">qk_norm_23</text>
</g>
</g>
</g>
<g id="matplotlib.axis_2">
<g id="ytick_1">
<g id="line2d_12">
<defs>
<path id="mdcab3d4c70" d="M 0 0
L -3.5 0
" style="stroke: #000000; stroke-width: 0.8"/>
</defs>
<g>
<use xlink:href="#mdcab3d4c70" x="51.700781" y="165.39636" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_12">
<!-- $\mathdefault{10^{-1}}$ -->
<g transform="translate(21.200781 170.04636)">
<text>
<tspan x="0" y="-0.762187" style="font-size: 10px; font-family: 'DejaVu Sans'">1</tspan>
<tspan x="6.362305" y="-0.762187" style="font-size: 10px; font-family: 'DejaVu Sans'">0</tspan>
<tspan x="12.820312" y="-4.892187" style="font-size: 7px; font-family: 'DejaVu Sans'">−</tspan>
<tspan x="18.685547" y="-4.892187" style="font-size: 7px; font-family: 'DejaVu Sans'">1</tspan>
</text>
</g>
</g>
</g>
<g id="ytick_2">
<g id="line2d_13">
<g>
<use xlink:href="#mdcab3d4c70" x="51.700781" y="89.149935" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_13">
<!-- $\mathdefault{10^{0}}$ -->
<g transform="translate(27.100781 93.849935)">
<text>
<tspan x="0" y="-0.674688" style="font-size: 10px; font-family: 'DejaVu Sans'">1</tspan>
<tspan x="6.362305" y="-0.674688" style="font-size: 10px; font-family: 'DejaVu Sans'">0</tspan>
<tspan x="12.820312" y="-4.804688" style="font-size: 7px; font-family: 'DejaVu Sans'">0</tspan>
</text>
</g>
</g>
</g>
<g id="ytick_3">
<g id="line2d_14">
<defs>
<path id="md42a30ea1c" d="M 0 0
L -2 0
" style="stroke: #000000; stroke-width: 0.6"/>
</defs>
<g>
<use xlink:href="#md42a30ea1c" x="51.700781" y="188.348821" style="stroke: #000000; stroke-width: 0.6"/>
</g>
</g>
</g>
<g id="ytick_4">
<g id="line2d_15">
<g>
<use xlink:href="#md42a30ea1c" x="51.700781" y="182.311534" style="stroke: #000000; stroke-width: 0.6"/>
</g>
</g>
</g>
<g id="ytick_5">
<g id="line2d_16">
<g>
<use xlink:href="#md42a30ea1c" x="51.700781" y="177.20708" style="stroke: #000000; stroke-width: 0.6"/>
</g>
</g>
</g>
<g id="ytick_6">
<g id="line2d_17">
<g>
<use xlink:href="#md42a30ea1c" x="51.700781" y="172.785402" style="stroke: #000000; stroke-width: 0.6"/>
</g>
</g>
</g>
<g id="ytick_7">
<g id="line2d_18">
<g>
<use xlink:href="#md42a30ea1c" x="51.700781" y="168.885205" style="stroke: #000000; stroke-width: 0.6"/>
</g>
</g>
</g>
<g id="ytick_8">
<g id="line2d_19">
<g>
<use xlink:href="#md42a30ea1c" x="51.700781" y="142.443899" style="stroke: #000000; stroke-width: 0.6"/>
</g>
</g>
</g>
<g id="ytick_9">
<g id="line2d_20">
<g>
<use xlink:href="#md42a30ea1c" x="51.700781" y="129.01757" style="stroke: #000000; stroke-width: 0.6"/>
</g>
</g>
</g>
<g id="ytick_10">
<g id="line2d_21">
<g>
<use xlink:href="#md42a30ea1c" x="51.700781" y="119.491438" style="stroke: #000000; stroke-width: 0.6"/>
</g>
</g>
</g>
<g id="ytick_11">
<g id="line2d_22">
<g>
<use xlink:href="#md42a30ea1c" x="51.700781" y="112.102396" style="stroke: #000000; stroke-width: 0.6"/>
</g>
</g>
</g>
<g id="ytick_12">
<g id="line2d_23">
<g>
<use xlink:href="#md42a30ea1c" x="51.700781" y="106.065109" style="stroke: #000000; stroke-width: 0.6"/>
</g>
</g>
</g>
<g id="ytick_13">
<g id="line2d_24">
<g>
<use xlink:href="#md42a30ea1c" x="51.700781" y="100.960655" style="stroke: #000000; stroke-width: 0.6"/>
</g>
</g>
</g>
<g id="ytick_14">
<g id="line2d_25">
<g>
<use xlink:href="#md42a30ea1c" x="51.700781" y="96.538977" style="stroke: #000000; stroke-width: 0.6"/>
</g>
</g>
</g>
<g id="ytick_15">
<g id="line2d_26">
<g>
<use xlink:href="#md42a30ea1c" x="51.700781" y="92.63878" style="stroke: #000000; stroke-width: 0.6"/>
</g>
</g>
</g>
<g id="ytick_16">
<g id="line2d_27">
<g>
<use xlink:href="#md42a30ea1c" x="51.700781" y="66.197474" style="stroke: #000000; stroke-width: 0.6"/>
</g>
</g>
</g>
<g id="ytick_17">
<g id="line2d_28">
<g>
<use xlink:href="#md42a30ea1c" x="51.700781" y="52.771145" style="stroke: #000000; stroke-width: 0.6"/>
</g>
</g>
</g>
<g id="ytick_18">
<g id="line2d_29">
<g>
<use xlink:href="#md42a30ea1c" x="51.700781" y="43.245012" style="stroke: #000000; stroke-width: 0.6"/>
</g>
</g>
</g>
<g id="text_14">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: middle" x="14.798438" y="116.717277" transform="rotate(-90 14.798438 116.717277)">Worst calibration directional relative L2</text>
</g>
</g>
<g id="line2d_30">
<path d="M 51.700781 188.348821
L 705.018437 188.348821
" clip-path="url(#pf9e7526e4c)" style="fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #b2182b; stroke-width: 1.5"/>
</g>
<g id="patch_14">
<path d="M 51.700781 195.511975
L 51.700781 37.922578
" style="fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square"/>
</g>
<g id="patch_15">
<path d="M 705.018437 195.511975
L 705.018437 37.922578
" style="fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square"/>
</g>
<g id="patch_16">
<path d="M 51.700781 195.511975
L 705.018437 195.511975
" style="fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square"/>
</g>
<g id="patch_17">
<path d="M 51.700781 37.922578
L 705.018437 37.922578
" style="fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square"/>
</g>
<g id="text_15">
<text style="font-size: 12px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif" transform="translate(226.773359 17.519297)">Fixed candidates; same-input function comparison</text>
<text style="font-size: 12px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif" transform="translate(254.193359 31.922578)">4 calibration texts, 32 tokens; no training</text>
</g>
<g id="legend_1">
<g id="patch_18">
<path d="M 57.300781 273.796571
L 280.582031 273.796571
Q 282.182031 273.796571 282.182031 272.196571
L 282.182031 260.995946
Q 282.182031 259.395946 280.582031 259.395946
L 57.300781 259.395946
Q 55.700781 259.395946 55.700781 260.995946
L 55.700781 272.196571
Q 55.700781 273.796571 57.300781 273.796571
z
" style="fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter"/>
</g>
<g id="line2d_31">
<path d="M 58.900781 265.874696
L 66.900781 265.874696
L 74.900781 265.874696
" style="fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #b2182b; stroke-width: 1.5"/>
</g>
<g id="text_16">
<text style="font-size: 8px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: start" x="81.300781" y="268.674696" transform="rotate(-0 81.300781 268.674696)">Relative L2 screen = 0.05; other gates also apply</text>
</g>
</g>
</g>
</g>
<defs>
<clipPath id="pf9e7526e4c">
<rect x="51.700781" y="37.922578" width="653.317656" height="157.589397"/>
</clipPath>
</defs>
</svg>

After

Width:  |  Height:  |  Size: 17 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 104 KiB

@@ -0,0 +1,388 @@
<?xml version="1.0" encoding="utf-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<svg xmlns:xlink="http://www.w3.org/1999/xlink" width="640.424062pt" height="312.347793pt" viewBox="0 0 640.424062 312.347793" xmlns="http://www.w3.org/2000/svg" version="1.1">
<metadata>
<rdf:RDF xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:cc="http://creativecommons.org/ns#" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
<cc:Work>
<dc:type rdf:resource="http://purl.org/dc/dcmitype/StillImage"/>
<dc:date>2026-11-09T14:10:55.472730</dc:date>
<dc:format>image/svg+xml</dc:format>
<dc:creator>
<cc:Agent>
<dc:title>Matplotlib v3.11.2, https://matplotlib.org/</dc:title>
</cc:Agent>
</dc:creator>
</cc:Work>
</rdf:RDF>
</metadata>
<defs>
<style type="text/css">*{stroke-linejoin: round; stroke-linecap: butt}</style>
</defs>
<g id="figure_1">
<g id="patch_1">
<path d="M 0 312.347793
L 640.424062 312.347793
L 640.424062 0
L 0 0
z
" style="fill: #ffffff"/>
</g>
<g id="axes_1">
<g id="patch_2">
<path d="M 50.466406 208.18075
L 633.224062 208.18075
L 633.224062 37.922578
L 50.466406 37.922578
z
" style="fill: #ffffff"/>
</g>
<g id="matplotlib.axis_1">
<g id="xtick_1">
<g id="line2d_1">
<defs>
<path id="mb7bbc15d82" d="M 0 0
L 0 3.5
" style="stroke: #000000; stroke-width: 0.8"/>
</defs>
<g>
<use xlink:href="#mb7bbc15d82" x="76.955391" y="208.18075" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_1">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif" transform="translate(41.486676 241.545565) rotate(-30)">cal-zh-1</text>
</g>
</g>
<g id="xtick_2">
<g id="line2d_2">
<g>
<use xlink:href="#mb7bbc15d82" x="152.638203" y="208.18075" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_2">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif" transform="translate(117.169489 241.545565) rotate(-30)">cal-zh-2</text>
</g>
</g>
<g id="xtick_3">
<g id="line2d_3">
<g>
<use xlink:href="#mb7bbc15d82" x="228.321016" y="208.18075" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_3">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif" transform="translate(192.068819 241.997909) rotate(-30)">cal-en-1</text>
</g>
</g>
<g id="xtick_4">
<g id="line2d_4">
<g>
<use xlink:href="#mb7bbc15d82" x="304.003828" y="208.18075" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_4">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif" transform="translate(267.751631 241.997909) rotate(-30)">cal-en-2</text>
</g>
</g>
<g id="xtick_5">
<g id="line2d_5">
<g>
<use xlink:href="#mb7bbc15d82" x="379.686641" y="208.18075" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_5">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif" transform="translate(340.062357 243.944784) rotate(-30)">test-zh-1</text>
</g>
</g>
<g id="xtick_6">
<g id="line2d_6">
<g>
<use xlink:href="#mb7bbc15d82" x="455.369453" y="208.18075" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_6">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif" transform="translate(415.74517 243.944784) rotate(-30)">test-zh-2</text>
</g>
</g>
<g id="xtick_7">
<g id="line2d_7">
<g>
<use xlink:href="#mb7bbc15d82" x="531.052266" y="208.18075" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_7">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif" transform="translate(490.6445 244.396451) rotate(-30)">test-en-1</text>
</g>
</g>
<g id="xtick_8">
<g id="line2d_8">
<g>
<use xlink:href="#mb7bbc15d82" x="606.735078" y="208.18075" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_8">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif" transform="translate(566.327312 244.396451) rotate(-30)">test-en-2</text>
</g>
</g>
</g>
<g id="matplotlib.axis_2">
<g id="ytick_1">
<g id="line2d_9">
<defs>
<path id="ma0713fe695" d="M 0 0
L -3.5 0
" style="stroke: #000000; stroke-width: 0.8"/>
</defs>
<g>
<use xlink:href="#ma0713fe695" x="50.466406" y="204.137497" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_9">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: end" x="43.466406" y="207.936325" transform="rotate(-0 43.466406 207.936325)">0.04</text>
</g>
</g>
<g id="ytick_2">
<g id="line2d_10">
<g>
<use xlink:href="#ma0713fe695" x="50.466406" y="182.54738" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_10">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: end" x="43.466406" y="186.346208" transform="rotate(-0 43.466406 186.346208)">0.05</text>
</g>
</g>
<g id="ytick_3">
<g id="line2d_11">
<g>
<use xlink:href="#ma0713fe695" x="50.466406" y="160.957263" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_11">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: end" x="43.466406" y="164.756091" transform="rotate(-0 43.466406 164.756091)">0.06</text>
</g>
</g>
<g id="ytick_4">
<g id="line2d_12">
<g>
<use xlink:href="#ma0713fe695" x="50.466406" y="139.367146" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_12">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: end" x="43.466406" y="143.165974" transform="rotate(-0 43.466406 143.165974)">0.07</text>
</g>
</g>
<g id="ytick_5">
<g id="line2d_13">
<g>
<use xlink:href="#ma0713fe695" x="50.466406" y="117.777029" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_13">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: end" x="43.466406" y="121.575857" transform="rotate(-0 43.466406 121.575857)">0.08</text>
</g>
</g>
<g id="ytick_6">
<g id="line2d_14">
<g>
<use xlink:href="#ma0713fe695" x="50.466406" y="96.186911" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_14">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: end" x="43.466406" y="99.98574" transform="rotate(-0 43.466406 99.98574)">0.09</text>
</g>
</g>
<g id="ytick_7">
<g id="line2d_15">
<g>
<use xlink:href="#ma0713fe695" x="50.466406" y="74.596794" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_15">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: end" x="43.466406" y="78.395623" transform="rotate(-0 43.466406 78.395623)">0.10</text>
</g>
</g>
<g id="ytick_8">
<g id="line2d_16">
<g>
<use xlink:href="#ma0713fe695" x="50.466406" y="53.006677" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_16">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: end" x="43.466406" y="56.805505" transform="rotate(-0 43.466406 56.805505)">0.11</text>
</g>
</g>
<g id="text_17">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: middle" x="14.798438" y="123.051664" transform="rotate(-90 14.798438 123.051664)">Worst of two directional relative L2 errors</text>
</g>
</g>
<g id="line2d_17">
<path d="M 76.955391 197.451805
L 152.638203 192.877368
L 228.321016 193.610655
L 304.003828 199.099794
L 379.686641 196.376861
L 455.369453 192.726888
L 531.052266 194.835109
L 606.735078 198.558148
" clip-path="url(#pc5e8d93515)" style="fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square"/>
<defs>
<path id="m8708773af9" d="M 0 3
C 0.795609 3 1.55874 2.683901 2.12132 2.12132
C 2.683901 1.55874 3 0.795609 3 0
C 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132
C 1.55874 -2.683901 0.795609 -3 0 -3
C -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132
C -2.683901 -1.55874 -3 -0.795609 -3 0
C -3 0.795609 -2.683901 1.55874 -2.12132 2.12132
C -1.55874 2.683901 -0.795609 3 0 3
z
" style="stroke: #1f77b4"/>
</defs>
<g clip-path="url(#pc5e8d93515)">
<use xlink:href="#m8708773af9" x="76.955391" y="197.451805" style="fill: #1f77b4; stroke: #1f77b4"/>
<use xlink:href="#m8708773af9" x="152.638203" y="192.877368" style="fill: #1f77b4; stroke: #1f77b4"/>
<use xlink:href="#m8708773af9" x="228.321016" y="193.610655" style="fill: #1f77b4; stroke: #1f77b4"/>
<use xlink:href="#m8708773af9" x="304.003828" y="199.099794" style="fill: #1f77b4; stroke: #1f77b4"/>
<use xlink:href="#m8708773af9" x="379.686641" y="196.376861" style="fill: #1f77b4; stroke: #1f77b4"/>
<use xlink:href="#m8708773af9" x="455.369453" y="192.726888" style="fill: #1f77b4; stroke: #1f77b4"/>
<use xlink:href="#m8708773af9" x="531.052266" y="194.835109" style="fill: #1f77b4; stroke: #1f77b4"/>
<use xlink:href="#m8708773af9" x="606.735078" y="198.558148" style="fill: #1f77b4; stroke: #1f77b4"/>
</g>
</g>
<g id="line2d_18">
<path d="M 76.955391 198.934893
L 152.638203 194.123521
L 228.321016 195.5305
L 304.003828 200.441742
L 379.686641 197.894474
L 455.369453 194.413382
L 531.052266 196.725517
L 606.735078 199.872446
" clip-path="url(#pc5e8d93515)" style="fill: none; stroke: #ff7f0e; stroke-width: 1.5; stroke-linecap: square"/>
<defs>
<path id="m5a845d07a0" d="M 0 3
C 0.795609 3 1.55874 2.683901 2.12132 2.12132
C 2.683901 1.55874 3 0.795609 3 0
C 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132
C 1.55874 -2.683901 0.795609 -3 0 -3
C -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132
C -2.683901 -1.55874 -3 -0.795609 -3 0
C -3 0.795609 -2.683901 1.55874 -2.12132 2.12132
C -1.55874 2.683901 -0.795609 3 0 3
z
" style="stroke: #ff7f0e"/>
</defs>
<g clip-path="url(#pc5e8d93515)">
<use xlink:href="#m5a845d07a0" x="76.955391" y="198.934893" style="fill: #ff7f0e; stroke: #ff7f0e"/>
<use xlink:href="#m5a845d07a0" x="152.638203" y="194.123521" style="fill: #ff7f0e; stroke: #ff7f0e"/>
<use xlink:href="#m5a845d07a0" x="228.321016" y="195.5305" style="fill: #ff7f0e; stroke: #ff7f0e"/>
<use xlink:href="#m5a845d07a0" x="304.003828" y="200.441742" style="fill: #ff7f0e; stroke: #ff7f0e"/>
<use xlink:href="#m5a845d07a0" x="379.686641" y="197.894474" style="fill: #ff7f0e; stroke: #ff7f0e"/>
<use xlink:href="#m5a845d07a0" x="455.369453" y="194.413382" style="fill: #ff7f0e; stroke: #ff7f0e"/>
<use xlink:href="#m5a845d07a0" x="531.052266" y="196.725517" style="fill: #ff7f0e; stroke: #ff7f0e"/>
<use xlink:href="#m5a845d07a0" x="606.735078" y="199.872446" style="fill: #ff7f0e; stroke: #ff7f0e"/>
</g>
</g>
<g id="line2d_19">
<path d="M 50.466406 182.54738
L 633.224062 182.54738
" clip-path="url(#pc5e8d93515)" style="fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #b2182b; stroke-width: 1.5"/>
</g>
<g id="line2d_20">
<path d="M 50.466406 45.661586
L 633.224062 45.661586
" clip-path="url(#pc5e8d93515)" style="fill: none; stroke-dasharray: 1.5,2.475; stroke-dashoffset: 0; stroke: #6a51a3; stroke-width: 1.5"/>
</g>
<g id="line2d_21">
<path d="M 341.845234 208.18075
L 341.845234 37.922578
" clip-path="url(#pc5e8d93515)" style="fill: none; stroke: #808080; stroke-width: 0.8; stroke-linecap: square"/>
</g>
<g id="patch_3">
<path d="M 50.466406 208.18075
L 50.466406 37.922578
" style="fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square"/>
</g>
<g id="patch_4">
<path d="M 633.224062 208.18075
L 633.224062 37.922578
" style="fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square"/>
</g>
<g id="patch_5">
<path d="M 50.466406 208.18075
L 633.224063 208.18075
" style="fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square"/>
</g>
<g id="patch_6">
<path d="M 50.466406 37.922578
L 633.224063 37.922578
" style="fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square"/>
</g>
<g id="text_18">
<text style="font-size: 12px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif" transform="translate(126.937422 17.519297)">Learned input RMSNorm at layer indices 12/13: same-input comparison</text>
<text style="font-size: 12px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif" transform="translate(137.227422 31.922578)">Calibration (left) / independent heldout texts (right), 32 tokens each</text>
</g>
<g id="legend_1">
<g id="patch_7">
<path d="M 56.066406 305.147793
L 211.338906 305.147793
Q 212.938906 305.147793 212.938906 303.547793
L 212.938906 256.345293
Q 212.938906 254.745293 211.338906 254.745293
L 56.066406 254.745293
Q 54.466406 254.745293 54.466406 256.345293
L 54.466406 303.547793
Q 54.466406 305.147793 56.066406 305.147793
z
" style="fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter"/>
</g>
<g id="line2d_22">
<path d="M 57.666406 261.224043
L 65.666406 261.224043
L 73.666406 261.224043
" style="fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square"/>
<g>
<use xlink:href="#m8708773af9" x="65.666406" y="261.224043" style="fill: #1f77b4; stroke: #1f77b4"/>
</g>
</g>
<g id="text_19">
<text style="font-size: 8px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: start" x="80.066406" y="264.024043" transform="rotate(-0 80.066406 264.024043)">Input captured at layer 12</text>
</g>
<g id="line2d_23">
<path d="M 57.666406 273.224668
L 65.666406 273.224668
L 73.666406 273.224668
" style="fill: none; stroke: #ff7f0e; stroke-width: 1.5; stroke-linecap: square"/>
<g>
<use xlink:href="#m5a845d07a0" x="65.666406" y="273.224668" style="fill: #ff7f0e; stroke: #ff7f0e"/>
</g>
</g>
<g id="text_20">
<text style="font-size: 8px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: start" x="80.066406" y="276.024668" transform="rotate(-0 80.066406 276.024668)">Input captured at layer 13</text>
</g>
<g id="line2d_24">
<path d="M 57.666406 285.225293
L 65.666406 285.225293
L 73.666406 285.225293
" style="fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #b2182b; stroke-width: 1.5"/>
</g>
<g id="text_21">
<text style="font-size: 8px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: start" x="80.066406" y="288.025293" transform="rotate(-0 80.066406 288.025293)">Heldout empirical screen = 0.05</text>
</g>
<g id="line2d_25">
<path d="M 57.666406 297.225918
L 65.666406 297.225918
L 73.666406 297.225918
" style="fill: none; stroke-dasharray: 1.5,2.475; stroke-dashoffset: 0; stroke: #6a51a3; stroke-width: 1.5"/>
</g>
<g id="text_22">
<text style="font-size: 8px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: start" x="80.066406" y="300.025918" transform="rotate(-0 80.066406 300.025918)">Proven uniform all-input bound</text>
</g>
</g>
</g>
</g>
<defs>
<clipPath id="pc5e8d93515">
<rect x="50.466406" y="37.922578" width="582.757656" height="170.258172"/>
</clipPath>
</defs>
</svg>

After

Width:  |  Height:  |  Size: 19 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 100 KiB

@@ -0,0 +1,286 @@
<?xml version="1.0" encoding="utf-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<svg xmlns:xlink="http://www.w3.org/1999/xlink" width="567.92875pt" height="308.816598pt" viewBox="0 0 567.92875 308.816598" xmlns="http://www.w3.org/2000/svg" version="1.1">
<metadata>
<rdf:RDF xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:cc="http://creativecommons.org/ns#" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
<cc:Work>
<dc:type rdf:resource="http://purl.org/dc/dcmitype/StillImage"/>
<dc:date>2026-11-09T14:10:58.596152</dc:date>
<dc:format>image/svg+xml</dc:format>
<dc:creator>
<cc:Agent>
<dc:title>Matplotlib v3.11.2, https://matplotlib.org/</dc:title>
</cc:Agent>
</dc:creator>
</cc:Work>
</rdf:RDF>
</metadata>
<defs>
<style type="text/css">*{stroke-linejoin: round; stroke-linecap: butt}</style>
</defs>
<g id="figure_1">
<g id="patch_1">
<path d="M 0 308.816598
L 567.92875 308.816598
L 567.92875 0
L 0 0
z
" style="fill: #ffffff"/>
</g>
<g id="axes_1">
<g id="patch_2">
<path d="M 71.571094 233.247975
L 560.72875 233.247975
L 560.72875 22.318125
L 71.571094 22.318125
z
" style="fill: #ffffff"/>
</g>
<g id="matplotlib.axis_1">
<g id="xtick_1">
<g id="line2d_1">
<defs>
<path id="m4de31d702a" d="M 0 0
L 0 3.5
" style="stroke: #000000; stroke-width: 0.8"/>
</defs>
<g>
<use xlink:href="#m4de31d702a" x="93.805533" y="233.247975" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_1">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: middle" x="93.805533" y="247.846412" transform="rotate(-0 93.805533 247.846412)">zh1</text>
</g>
</g>
<g id="xtick_2">
<g id="line2d_2">
<g>
<use xlink:href="#m4de31d702a" x="242.035125" y="233.247975" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_2">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: middle" x="242.035125" y="247.846412" transform="rotate(-0 242.035125 247.846412)">zh2</text>
</g>
</g>
<g id="xtick_3">
<g id="line2d_3">
<g>
<use xlink:href="#m4de31d702a" x="390.264718" y="233.247975" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_3">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: middle" x="390.264718" y="247.845631" transform="rotate(-0 390.264718 247.845631)">en1</text>
</g>
</g>
<g id="xtick_4">
<g id="line2d_4">
<g>
<use xlink:href="#m4de31d702a" x="538.494311" y="233.247975" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_4">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: middle" x="538.494311" y="247.845631" transform="rotate(-0 538.494311 247.845631)">en2</text>
</g>
</g>
<g id="text_5">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: middle" x="316.149922" y="261.847194" transform="rotate(-0 316.149922 261.847194)">Independent heldout text (32 tokens)</text>
</g>
</g>
<g id="matplotlib.axis_2">
<g id="ytick_1">
<g id="line2d_5">
<defs>
<path id="m32017c2f5a" d="M 0 0
L -3.5 0
" style="stroke: #000000; stroke-width: 0.8"/>
</defs>
<g>
<use xlink:href="#m32017c2f5a" x="71.571094" y="231.508072" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_6">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: end" x="64.571094" y="235.3069" transform="rotate(-0 64.571094 235.3069)">−0.0015</text>
</g>
</g>
<g id="ytick_2">
<g id="line2d_6">
<g>
<use xlink:href="#m32017c2f5a" x="71.571094" y="192.395623" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_7">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: end" x="64.571094" y="196.194451" transform="rotate(-0 64.571094 196.194451)">−0.0010</text>
</g>
</g>
<g id="ytick_3">
<g id="line2d_7">
<g>
<use xlink:href="#m32017c2f5a" x="71.571094" y="153.283173" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_8">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: end" x="64.571094" y="157.082001" transform="rotate(-0 64.571094 157.082001)">−0.0005</text>
</g>
</g>
<g id="ytick_4">
<g id="line2d_8">
<g>
<use xlink:href="#m32017c2f5a" x="71.571094" y="114.170724" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_9">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: end" x="64.571094" y="117.969552" transform="rotate(-0 64.571094 117.969552)">0.0000</text>
</g>
</g>
<g id="ytick_5">
<g id="line2d_9">
<g>
<use xlink:href="#m32017c2f5a" x="71.571094" y="75.058274" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_10">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: end" x="64.571094" y="78.857102" transform="rotate(-0 64.571094 78.857102)">0.0005</text>
</g>
</g>
<g id="ytick_6">
<g id="line2d_10">
<g>
<use xlink:href="#m32017c2f5a" x="71.571094" y="35.945825" style="stroke: #000000; stroke-width: 0.8"/>
</g>
</g>
<g id="text_11">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: end" x="64.571094" y="39.744653" transform="rotate(-0 64.571094 39.744653)">0.0010</text>
</g>
</g>
<g id="text_12">
<text style="font-size: 10px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: middle" x="14.798438" y="127.78305" transform="rotate(-90 14.798438 127.78305)">Mean token NLL change (nat/token)</text>
</g>
</g>
<g id="line2d_11">
<path d="M 93.805533 31.905845
L 242.035125 223.660255
L 390.264718 124.077707
L 538.494311 66.84947
" clip-path="url(#p78d521230c)" style="fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square"/>
<defs>
<path id="ma59ebf376b" d="M 0 3
C 0.795609 3 1.55874 2.683901 2.12132 2.12132
C 2.683901 1.55874 3 0.795609 3 0
C 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132
C 1.55874 -2.683901 0.795609 -3 0 -3
C -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132
C -2.683901 -1.55874 -3 -0.795609 -3 0
C -3 0.795609 -2.683901 1.55874 -2.12132 2.12132
C -1.55874 2.683901 -0.795609 3 0 3
z
" style="stroke: #1f77b4"/>
</defs>
<g clip-path="url(#p78d521230c)">
<use xlink:href="#ma59ebf376b" x="93.805533" y="31.905845" style="fill: #1f77b4; stroke: #1f77b4"/>
<use xlink:href="#ma59ebf376b" x="242.035125" y="223.660255" style="fill: #1f77b4; stroke: #1f77b4"/>
<use xlink:href="#ma59ebf376b" x="390.264718" y="124.077707" style="fill: #1f77b4; stroke: #1f77b4"/>
<use xlink:href="#ma59ebf376b" x="538.494311" y="66.84947" style="fill: #1f77b4; stroke: #1f77b4"/>
</g>
</g>
<g id="line2d_12">
<path d="M 93.805533 45.225019
L 242.035125 215.36524
L 390.264718 134.743944
L 538.494311 175.235695
" clip-path="url(#p78d521230c)" style="fill: none; stroke: #ff7f0e; stroke-width: 1.5; stroke-linecap: square"/>
<defs>
<path id="m2ce5b34917" d="M 0 3
C 0.795609 3 1.55874 2.683901 2.12132 2.12132
C 2.683901 1.55874 3 0.795609 3 0
C 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132
C 1.55874 -2.683901 0.795609 -3 0 -3
C -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132
C -2.683901 -1.55874 -3 -0.795609 -3 0
C -3 0.795609 -2.683901 1.55874 -2.12132 2.12132
C -1.55874 2.683901 -0.795609 3 0 3
z
" style="stroke: #ff7f0e"/>
</defs>
<g clip-path="url(#p78d521230c)">
<use xlink:href="#m2ce5b34917" x="93.805533" y="45.225019" style="fill: #ff7f0e; stroke: #ff7f0e"/>
<use xlink:href="#m2ce5b34917" x="242.035125" y="215.36524" style="fill: #ff7f0e; stroke: #ff7f0e"/>
<use xlink:href="#m2ce5b34917" x="390.264718" y="134.743944" style="fill: #ff7f0e; stroke: #ff7f0e"/>
<use xlink:href="#m2ce5b34917" x="538.494311" y="175.235695" style="fill: #ff7f0e; stroke: #ff7f0e"/>
</g>
</g>
<g id="line2d_13">
<path d="M 71.571094 114.170724
L 560.72875 114.170724
" clip-path="url(#p78d521230c)" style="fill: none; stroke: #808080; stroke-width: 0.8; stroke-linecap: square"/>
</g>
<g id="patch_3">
<path d="M 71.571094 233.247975
L 71.571094 22.318125
" style="fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square"/>
</g>
<g id="patch_4">
<path d="M 560.72875 233.247975
L 560.72875 22.318125
" style="fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square"/>
</g>
<g id="patch_5">
<path d="M 71.571094 233.247975
L 560.72875 233.247975
" style="fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square"/>
</g>
<g id="patch_6">
<path d="M 71.571094 22.318125
L 560.72875 22.318125
" style="fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square"/>
</g>
<g id="text_13">
<text style="font-size: 12px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: middle" x="316.149922" y="16.318125" transform="rotate(-0 316.149922 16.318125)">Bidirectional single-site substitution of existing learned RMSNorm modules</text>
</g>
<g id="legend_1">
<g id="patch_7">
<path d="M 77.171094 301.616598
L 257.259844 301.616598
Q 258.859844 301.616598 258.859844 300.016598
L 258.859844 276.815348
Q 258.859844 275.215348 257.259844 275.215348
L 77.171094 275.215348
Q 75.571094 275.215348 75.571094 276.815348
L 75.571094 300.016598
Q 75.571094 301.616598 77.171094 301.616598
z
" style="fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter"/>
</g>
<g id="line2d_14">
<path d="M 78.771094 281.694098
L 86.771094 281.694098
L 94.771094 281.694098
" style="fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square"/>
<g>
<use xlink:href="#ma59ebf376b" x="86.771094" y="281.694098" style="fill: #1f77b4; stroke: #1f77b4"/>
</g>
</g>
<g id="text_14">
<text style="font-size: 8px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: start" x="101.171094" y="284.494098" transform="rotate(-0 101.171094 284.494098)">Layer 12 uses 13; mean=3.2565e-05</text>
</g>
<g id="line2d_15">
<path d="M 78.771094 293.694723
L 86.771094 293.694723
L 94.771094 293.694723
" style="fill: none; stroke: #ff7f0e; stroke-width: 1.5; stroke-linecap: square"/>
<g>
<use xlink:href="#m2ce5b34917" x="86.771094" y="293.694723" style="fill: #ff7f0e; stroke: #ff7f0e"/>
</g>
</g>
<g id="text_15">
<text style="font-size: 8px; font-family: 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: start" x="101.171094" y="296.494723" transform="rotate(-0 101.171094 296.494723)">Layer 13 uses 12; mean=-0.00036397</text>
</g>
</g>
</g>
</g>
<defs>
<clipPath id="p78d521230c">
<rect x="71.571094" y="22.318125" width="489.157656" height="210.92985"/>
</clipPath>
</defs>
</svg>

After

Width:  |  Height:  |  Size: 13 KiB

@@ -0,0 +1,18 @@
{
"functional/__init__.py": "e572d3a9040dfb440b92e969df495d2a9e0edbb1bfe1318e5e66841cc6144067",
"functional/audit.py": "5e10c858fe574f4c9a51cdb9c61a94932d9e4bb6dac82082a94c2ef7d57a693b",
"functional/benchmark_factorization.py": "135759dbe1af48b86e327cf23a1ef656d000dfef83dea2dbf48d2514356ea729",
"functional/gain_factorization.py": "33f3620027c50bce913ad07ed12f42fad83d69a811652174c803467c295e77c9",
"functional/model.py": "48a2a97d0445dc6e08ac7f27a935637a7553417d0982eb94122d6cb02bbb499c",
"functional/neuron_reuse.py": "d408bc14242c7950b846b6d91870d88374cdb8ece7fc111a2ccdc25e725ca19c",
"functional/norm_repetition.py": "aac0c44904c2b059510a754f2f0b1247ffb4b2a1715ecee6281b5ed2b43612b3",
"functional/preflight.py": "87c13d21512e8cbb82bc8a8aeb614a3be06bddbb8489c52eb2bf316480b0f20c",
"functional/reference.json": "af6bf3040369580de3dff4a6733576a75c61bcbd7e94201d309cf28e9b75e3d6",
"functional/reference_check.py": "422cd70c4f1276590402b10261aff0fc744a656387eb260b54bac303353c277c",
"functional/report.py": "ca74122de373e80429fab0f8c2e2d0dba3f8f7c6ce046510e5bf28fa7b8d827d",
"functional/run.py": "89d3abe098c240ec29c3f6684a94e24b951c372e1393c8db5a48f79ef141aa48",
"functional/storage.py": "8387fac8c192fe050e6d275dff6fe55a405332ee18c09f3e3b5d4d537a3db59e",
"functional/supervise.py": "44c01fd9ba989ed68ef58313e490602a49f5da26aeb918ee0b6545aff766144e",
"pilot/tensors.py": "2e620849501b6bc98a59cbf9c39495326b0c85530e18e6166aa9edb68ac2423e",
"tests/test_functional.py": "82c8585f6a9f4e6323cd978621639163298a369bdf298dfa5103c020c3e71f38"
}
+23
View File
@@ -0,0 +1,23 @@
test_causal_convolution_and_norm_boundary (test_functional.FunctionalTests.test_causal_convolution_and_norm_boundary) ... ok
test_causality_and_identical_function_metrics (test_functional.FunctionalTests.test_causality_and_identical_function_metrics) ... ok
test_neuron_function_metrics_and_targeted_replacement (test_functional.FunctionalTests.test_neuron_function_metrics_and_targeted_replacement) ... ok
test_nonrotary_gain_identity_and_rotary_negative_control (test_functional.FunctionalTests.test_nonrotary_gain_identity_and_rotary_negative_control) ... ok
test_streamed_projection_embedding_and_output_head (test_functional.FunctionalTests.test_streamed_projection_embedding_and_output_head) ... ok
test_uniform_rmsnorm_function_bound_is_tight (test_functional.FunctionalTests.test_uniform_rmsnorm_function_bound_is_tight) ... ok
test_3d_reader_and_exact_block (test_pilot.PilotTests.test_3d_reader_and_exact_block) ... ok
test_bfloat16 (test_pilot.PilotTests.test_bfloat16) ... ok
test_directional_replacement_error_and_scaling (test_pilot.PilotTests.test_directional_replacement_error_and_scaling) ... ok
test_end_to_end_synthetic_groups_and_savings (test_pilot.PilotTests.test_end_to_end_synthetic_groups_and_savings) ... Inventory: 4 tensors, 33792 bytes
tensor: sampling 4 objects
block: sampling 22 objects
block: verified 100/109
{"model": "synthetic", "revision": "test", "tensor_count": 4, "parameter_count": 8448, "model_weight_bytes": 33792, "ndim_counts": {"2": 2, "3": 2}, "elapsed_seconds": 0.1379160056822002, "analysis_budget_seconds": 60, "peak_rss_kib": 53592, "python": "3.14.4", "numpy": "2.5.4", "seed": 20261109, "limitations": ["No inference or functional equivalence measured", "No exhaustive pair search", "Block search uses per-role/shape/scale reservoirs of 256", "Different block scales overlap; savings must not be added", "Block savings count only verified disjoint sampled blocks, no extrapolation", "Spectral probes use aligned bounded subvolumes, not whole tensors", "Shared encoding is theoretical, not a runnable compressed checkpoint", "No cross-role grouping, neuron permutation alignment, or training"], "completed": true}
ok
test_no_transitive_chaining (test_pilot.PilotTests.test_no_transitive_chaining) ... ok
test_ranged_download_preserves_prefix (test_pilot.PilotTests.test_ranged_download_preserves_prefix) ... Weight download 36/36 bytes (100.0%)
ok
----------------------------------------------------------------------
Ran 12 tests in 40.946s
OK
@@ -0,0 +1,12 @@
{
"functional/__init__.py": "e572d3a9040dfb440b92e969df495d2a9e0edbb1bfe1318e5e66841cc6144067",
"functional/model.py": "48a2a97d0445dc6e08ac7f27a935637a7553417d0982eb94122d6cb02bbb499c",
"functional/preflight.py": "87c13d21512e8cbb82bc8a8aeb614a3be06bddbb8489c52eb2bf316480b0f20c",
"functional/reference.json": "af6bf3040369580de3dff4a6733576a75c61bcbd7e94201d309cf28e9b75e3d6",
"functional/reference_check.py": "422cd70c4f1276590402b10261aff0fc744a656387eb260b54bac303353c277c",
"functional/report.py": "e52201fa910a062b8521b4b1b2ea7ac80d276dca06fa8a322b9d3cb926ce100d",
"functional/run.py": "89d3abe098c240ec29c3f6684a94e24b951c372e1393c8db5a48f79ef141aa48",
"functional/samples.json": "bccb6ed515a8703c6ab2fc29b5227e1d11823b2ac94c95495ecf24a0340ba103",
"functional/storage.py": "8387fac8c192fe050e6d275dff6fe55a405332ee18c09f3e3b5d4d537a3db59e",
"functional/supervise.py": "44c01fd9ba989ed68ef58313e490602a49f5da26aeb918ee0b6545aff766144e"
}
@@ -0,0 +1,34 @@
{
"python": "3.14.4",
"platform": "Linux-7.0.0-31-generic-x86_64-with-glibc2.43",
"numpy": "2.5.4",
"packages": [
"contourpy==1.4.0",
"cycler==0.12.1",
"filelock==4.2.4",
"fonttools==4.68.0",
"fsspec==2026.9.0",
"Jinja2==3.1.6",
"kiwisolver==1.5.1",
"MarkupSafe==3.0.3",
"matplotlib==3.11.2",
"mpmath==1.3.0",
"networkx==3.7",
"numpy==2.5.4",
"packaging==26.3",
"pillow==13.0.0",
"pyparsing==3.3.3",
"python-dateutil==4.0.0",
"setuptools==78.1.0",
"sympy==1.14.0",
"tokenizers==0.23.3",
"torch==2.15.0+cpu",
"typing_extensions==4.16.0",
"tzdata==2026.6"
],
"weights_retained": true,
"no_automatic_cleanup": true,
"ledger_seconds_including_initial_reserve": 1167.355532555841,
"final_tests_report_audit_additional_budget_upper_reserve_seconds": 600,
"historical_error_logs": "Early reference adapter errors and preflight pressure stops; latest summaries/tests are authoritative."
}
+24
View File
@@ -0,0 +1,24 @@
{"phase": "baseline", "layer": 0, "elapsed": 6.704378366470337}
{"phase": "baseline", "layer": 1, "elapsed": 12.428538799285889}
{"phase": "baseline", "layer": 2, "elapsed": 18.311316967010498}
{"phase": "baseline", "layer": 3, "elapsed": 22.585159063339233}
{"phase": "baseline", "layer": 4, "elapsed": 27.968208074569702}
{"phase": "baseline", "layer": 5, "elapsed": 34.463871240615845}
{"phase": "baseline", "layer": 6, "elapsed": 41.34764099121094}
{"phase": "baseline", "layer": 7, "elapsed": 44.77713179588318}
{"phase": "baseline", "layer": 8, "elapsed": 50.20741152763367}
{"phase": "baseline", "layer": 9, "elapsed": 55.92143273353577}
{"phase": "baseline", "layer": 10, "elapsed": 62.15613389015198}
{"phase": "baseline", "layer": 11, "elapsed": 65.01726865768433}
{"phase": "baseline", "layer": 12, "elapsed": 70.20210671424866}
{"phase": "baseline", "layer": 13, "elapsed": 76.30333161354065}
{"phase": "baseline", "layer": 14, "elapsed": 81.29059743881226}
{"phase": "baseline", "layer": 15, "elapsed": 85.60254788398743}
{"phase": "baseline", "layer": 16, "elapsed": 91.78613877296448}
{"phase": "baseline", "layer": 17, "elapsed": 98.67645645141602}
{"phase": "baseline", "layer": 18, "elapsed": 106.65264892578125}
{"phase": "baseline", "layer": 19, "elapsed": 109.9351601600647}
{"phase": "baseline", "layer": 20, "elapsed": 116.18031859397888}
{"phase": "baseline", "layer": 21, "elapsed": 122.5530002117157}
{"phase": "baseline", "layer": 22, "elapsed": 128.36239457130432}
{"phase": "baseline", "layer": 23, "elapsed": 131.58471012115479}
@@ -0,0 +1,11 @@
{
"state": "completed",
"started_unix": 1794202338.6853228,
"phase": "done",
"last_completed_layer": 23,
"elapsed_seconds": 196.65513348579407,
"selected_candidate": [],
"positive_bidirectional_quality": false,
"peak_rss_kib": 162372,
"claim_limit": "Small text-only heldout pilot; no global interchangeability proof, no measured GPU speedup; component storage savings require explicit layout accounting."
}
@@ -0,0 +1,21 @@
repeat,implementation,seconds_per_call
0,baseline,0.0036114158341661096
0,factored,0.003925090050324797
1,factored,0.0032154972897842525
1,baseline,0.0025859603192657232
2,baseline,0.0023861153982579707
2,factored,0.0025824901415035127
3,factored,0.001997539587318897
3,baseline,0.0016429506475105881
4,baseline,0.0017620180500671268
4,factored,0.0027307189535349607
5,factored,0.002951206616126001
5,baseline,0.0020800257567316293
6,baseline,0.0018509919987991451
6,factored,0.0024777360958978534
7,factored,0.0025671867886558177
7,baseline,0.002200991357676685
8,baseline,0.0020182608393952252
8,factored,0.002254301542416215
9,factored,0.0025048630544915796
9,baseline,0.002296067262068391
1 repeat implementation seconds_per_call
2 0 baseline 0.0036114158341661096
3 0 factored 0.003925090050324797
4 1 factored 0.0032154972897842525
5 1 baseline 0.0025859603192657232
6 2 baseline 0.0023861153982579707
7 2 factored 0.0025824901415035127
8 3 factored 0.001997539587318897
9 3 baseline 0.0016429506475105881
10 4 baseline 0.0017620180500671268
11 4 factored 0.0027307189535349607
12 5 factored 0.002951206616126001
13 5 baseline 0.0020800257567316293
14 6 baseline 0.0018509919987991451
15 6 factored 0.0024777360958978534
16 7 factored 0.0025671867886558177
17 7 baseline 0.002200991357676685
18 8 baseline 0.0020182608393952252
19 8 factored 0.002254301542416215
20 9 factored 0.0025048630544915796
21 9 baseline 0.002296067262068391
@@ -0,0 +1,102 @@
{
"state": "completed",
"bitwise_checks": [
{
"layer": 3,
"direction": "q_to_k",
"query_elements": 524288,
"key_elements": 131072,
"bitwise_equal": true
},
{
"layer": 3,
"direction": "k_to_q",
"query_elements": 524288,
"key_elements": 131072,
"bitwise_equal": true
},
{
"layer": 7,
"direction": "q_to_k",
"query_elements": 524288,
"key_elements": 131072,
"bitwise_equal": true
},
{
"layer": 7,
"direction": "k_to_q",
"query_elements": 524288,
"key_elements": 131072,
"bitwise_equal": true
},
{
"layer": 11,
"direction": "q_to_k",
"query_elements": 524288,
"key_elements": 131072,
"bitwise_equal": true
},
{
"layer": 11,
"direction": "k_to_q",
"query_elements": 524288,
"key_elements": 131072,
"bitwise_equal": true
},
{
"layer": 15,
"direction": "q_to_k",
"query_elements": 524288,
"key_elements": 131072,
"bitwise_equal": true
},
{
"layer": 15,
"direction": "k_to_q",
"query_elements": 524288,
"key_elements": 131072,
"bitwise_equal": true
},
{
"layer": 19,
"direction": "q_to_k",
"query_elements": 524288,
"key_elements": 131072,
"bitwise_equal": true
},
{
"layer": 19,
"direction": "k_to_q",
"query_elements": 524288,
"key_elements": 131072,
"bitwise_equal": true
},
{
"layer": 23,
"direction": "q_to_k",
"query_elements": 524288,
"key_elements": 131072,
"bitwise_equal": true
},
{
"layer": 23,
"direction": "k_to_q",
"query_elements": 524288,
"key_elements": 131072,
"bitwise_equal": true
}
],
"medians_seconds": {
"baseline_legacy": 0.0005341526120901108,
"baseline_inplace": 0.00034534828271716835,
"factored_inplace": 0.00048380128573626284
},
"paired_baseline_over_factored_median": 0.6901853298227001,
"paired_ratio_p10": 0.38049978760565983,
"paired_ratio_p90": 1.0876508522302268,
"fraction_blocks_factored_faster": 0.13333333333333333,
"descriptive_local_gain": false,
"claim_limit": "Hot-array CPU normalization microbenchmark against equally allocation-optimized baseline; no full-model speedup or independent statistical significance claim.",
"elapsed_seconds": 4.185962562914938,
"peak_rss_kib": 62168
}
@@ -0,0 +1,91 @@
repeat,implementation,seconds_per_call,calls_per_block
0,baseline_legacy,0.0005406431574374438,50
0,baseline_inplace,0.00030974450521171093,50
0,factored_inplace,0.0004578591044992208,50
1,baseline_inplace,0.0003033110871911049,50
1,factored_inplace,0.0005383838433772326,50
1,baseline_legacy,0.0005124531593173742,50
2,factored_inplace,0.0004439983796328306,50
2,baseline_legacy,0.0005330004263669252,50
2,baseline_inplace,0.00034275135956704614,50
3,baseline_legacy,0.00048629873432219027,50
3,baseline_inplace,0.00030624629929661753,50
3,factored_inplace,0.0004589695204049349,50
4,baseline_inplace,0.0003086038026958704,50
4,factored_inplace,0.0005195329245179891,50
4,baseline_legacy,0.000474175363779068,50
5,factored_inplace,0.00045118503272533416,50
5,baseline_legacy,0.00048273258842527865,50
5,baseline_inplace,0.0003844233974814415,50
6,baseline_legacy,0.0006650591176003218,50
6,baseline_inplace,0.0007703853026032448,50
6,factored_inplace,0.0004403704963624477,50
7,baseline_inplace,0.00038892081938683986,50
7,factored_inplace,0.0004549186024814844,50
7,baseline_legacy,0.0005137955769896508,50
8,factored_inplace,0.0004692125599831343,50
8,baseline_legacy,0.0005263624526560307,50
8,baseline_inplace,0.000349312499165535,50
9,baseline_legacy,0.0004857080150395632,50
9,baseline_inplace,0.0012390289548784494,50
9,factored_inplace,0.0018109613563865423,50
10,baseline_inplace,0.0013814480416476727,50
10,factored_inplace,0.0013840428367257118,50
10,baseline_legacy,0.001071744617074728,50
11,factored_inplace,0.0016566905751824379,50
11,baseline_legacy,0.001358420280739665,50
11,baseline_inplace,0.0003716108575463295,50
12,baseline_legacy,0.0008325341437011957,50
12,baseline_inplace,0.0003479452058672905,50
12,factored_inplace,0.0004391701612621546,50
13,baseline_inplace,0.0005428373627364636,50
13,factored_inplace,0.0015458692144602537,50
13,baseline_legacy,0.0013308324199169874,50
14,factored_inplace,0.0014238305203616618,50
14,baseline_legacy,0.0006564708799123764,50
14,baseline_inplace,0.0003204820398241281,50
15,baseline_legacy,0.0005279563181102276,50
15,baseline_inplace,0.00030857514590024946,50
15,factored_inplace,0.0005265675205737352,50
16,baseline_inplace,0.000458676153793931,50
16,factored_inplace,0.00045228853821754455,50
16,baseline_legacy,0.0004912188835442066,50
17,factored_inplace,0.0004910450987517834,50
17,baseline_legacy,0.0005007587838917971,50
17,baseline_inplace,0.0003247600421309471,50
18,baseline_legacy,0.0004860841576009989,50
18,baseline_inplace,0.00032584095373749733,50
18,factored_inplace,0.0004711645469069481,50
19,baseline_inplace,0.0003332006186246872,50
19,factored_inplace,0.0004859814327210188,50
19,baseline_legacy,0.0006259976234287023,50
20,factored_inplace,0.0005134599283337593,50
20,baseline_legacy,0.0006658115796744823,50
20,baseline_inplace,0.0003131803590804338,50
21,baseline_legacy,0.00048669124022126197,50
21,baseline_inplace,0.0003487722761929035,50
21,factored_inplace,0.00048093565739691256,50
22,baseline_inplace,0.000310970526188612,50
22,factored_inplace,0.0004317485820502043,50
22,baseline_legacy,0.0005281844362616539,50
23,factored_inplace,0.0004816211387515068,50
23,baseline_legacy,0.0005441690050065518,50
23,baseline_inplace,0.0012436784617602826,50
24,baseline_legacy,0.0013620165828615426,50
24,baseline_inplace,0.000990196019411087,50
24,factored_inplace,0.0010849225427955389,50
25,baseline_inplace,0.0019172614254057408,50
25,factored_inplace,0.0009820526745170355,50
25,baseline_legacy,0.0009186737984418869,50
26,factored_inplace,0.0013663648813962937,50
26,baseline_legacy,0.0009482460748404264,50
26,baseline_inplace,0.0005478111188858747,50
27,baseline_legacy,0.000548997139558196,50
27,baseline_inplace,0.0003215675055980682,50
27,factored_inplace,0.0004668480064719915,50
28,baseline_inplace,0.0003313558641821146,50
28,factored_inplace,0.00047064563259482386,50
28,baseline_legacy,0.0005353047978132963,50
29,factored_inplace,0.0006785523053258657,50
29,baseline_legacy,0.000497497720643878,50
29,baseline_inplace,0.000260401563718915,50
1 repeat implementation seconds_per_call calls_per_block
2 0 baseline_legacy 0.0005406431574374438 50
3 0 baseline_inplace 0.00030974450521171093 50
4 0 factored_inplace 0.0004578591044992208 50
5 1 baseline_inplace 0.0003033110871911049 50
6 1 factored_inplace 0.0005383838433772326 50
7 1 baseline_legacy 0.0005124531593173742 50
8 2 factored_inplace 0.0004439983796328306 50
9 2 baseline_legacy 0.0005330004263669252 50
10 2 baseline_inplace 0.00034275135956704614 50
11 3 baseline_legacy 0.00048629873432219027 50
12 3 baseline_inplace 0.00030624629929661753 50
13 3 factored_inplace 0.0004589695204049349 50
14 4 baseline_inplace 0.0003086038026958704 50
15 4 factored_inplace 0.0005195329245179891 50
16 4 baseline_legacy 0.000474175363779068 50
17 5 factored_inplace 0.00045118503272533416 50
18 5 baseline_legacy 0.00048273258842527865 50
19 5 baseline_inplace 0.0003844233974814415 50
20 6 baseline_legacy 0.0006650591176003218 50
21 6 baseline_inplace 0.0007703853026032448 50
22 6 factored_inplace 0.0004403704963624477 50
23 7 baseline_inplace 0.00038892081938683986 50
24 7 factored_inplace 0.0004549186024814844 50
25 7 baseline_legacy 0.0005137955769896508 50
26 8 factored_inplace 0.0004692125599831343 50
27 8 baseline_legacy 0.0005263624526560307 50
28 8 baseline_inplace 0.000349312499165535 50
29 9 baseline_legacy 0.0004857080150395632 50
30 9 baseline_inplace 0.0012390289548784494 50
31 9 factored_inplace 0.0018109613563865423 50
32 10 baseline_inplace 0.0013814480416476727 50
33 10 factored_inplace 0.0013840428367257118 50
34 10 baseline_legacy 0.001071744617074728 50
35 11 factored_inplace 0.0016566905751824379 50
36 11 baseline_legacy 0.001358420280739665 50
37 11 baseline_inplace 0.0003716108575463295 50
38 12 baseline_legacy 0.0008325341437011957 50
39 12 baseline_inplace 0.0003479452058672905 50
40 12 factored_inplace 0.0004391701612621546 50
41 13 baseline_inplace 0.0005428373627364636 50
42 13 factored_inplace 0.0015458692144602537 50
43 13 baseline_legacy 0.0013308324199169874 50
44 14 factored_inplace 0.0014238305203616618 50
45 14 baseline_legacy 0.0006564708799123764 50
46 14 baseline_inplace 0.0003204820398241281 50
47 15 baseline_legacy 0.0005279563181102276 50
48 15 baseline_inplace 0.00030857514590024946 50
49 15 factored_inplace 0.0005265675205737352 50
50 16 baseline_inplace 0.000458676153793931 50
51 16 factored_inplace 0.00045228853821754455 50
52 16 baseline_legacy 0.0004912188835442066 50
53 17 factored_inplace 0.0004910450987517834 50
54 17 baseline_legacy 0.0005007587838917971 50
55 17 baseline_inplace 0.0003247600421309471 50
56 18 baseline_legacy 0.0004860841576009989 50
57 18 baseline_inplace 0.00032584095373749733 50
58 18 factored_inplace 0.0004711645469069481 50
59 19 baseline_inplace 0.0003332006186246872 50
60 19 factored_inplace 0.0004859814327210188 50
61 19 baseline_legacy 0.0006259976234287023 50
62 20 factored_inplace 0.0005134599283337593 50
63 20 baseline_legacy 0.0006658115796744823 50
64 20 baseline_inplace 0.0003131803590804338 50
65 21 baseline_legacy 0.00048669124022126197 50
66 21 baseline_inplace 0.0003487722761929035 50
67 21 factored_inplace 0.00048093565739691256 50
68 22 baseline_inplace 0.000310970526188612 50
69 22 factored_inplace 0.0004317485820502043 50
70 22 baseline_legacy 0.0005281844362616539 50
71 23 factored_inplace 0.0004816211387515068 50
72 23 baseline_legacy 0.0005441690050065518 50
73 23 baseline_inplace 0.0012436784617602826 50
74 24 baseline_legacy 0.0013620165828615426 50
75 24 baseline_inplace 0.000990196019411087 50
76 24 factored_inplace 0.0010849225427955389 50
77 25 baseline_inplace 0.0019172614254057408 50
78 25 factored_inplace 0.0009820526745170355 50
79 25 baseline_legacy 0.0009186737984418869 50
80 26 factored_inplace 0.0013663648813962937 50
81 26 baseline_legacy 0.0009482460748404264 50
82 26 baseline_inplace 0.0005478111188858747 50
83 27 baseline_legacy 0.000548997139558196 50
84 27 baseline_inplace 0.0003215675055980682 50
85 27 factored_inplace 0.0004668480064719915 50
86 28 baseline_inplace 0.0003313558641821146 50
87 28 factored_inplace 0.00047064563259482386 50
88 28 baseline_legacy 0.0005353047978132963 50
89 29 factored_inplace 0.0006785523053258657 50
90 29 baseline_legacy 0.000497497720643878 50
91 29 baseline_inplace 0.000260401563718915 50
@@ -0,0 +1,23 @@
{
"fixed_intervention_layer": 3,
"directions": [
"q_to_k",
"k_to_q"
],
"diagnostic_layers": [
3,
7,
11,
15,
19,
23
],
"no_selection": true,
"no_training": true,
"precision": "FP32 product; no BF16 requantization",
"quality_gate": {
"mean_nll_delta": 0.01,
"sequence_p95_nll_delta": 0.05
},
"registered_unix": 1794202505.9768724
}
@@ -0,0 +1,97 @@
layer,direction,sample_id,split,score_max_abs_error,score_relative_l2,probability_max_abs_error,first_key_mass_a,first_key_mass_b,js_without_first_key,without_first_key_valid_rows,js_mean,js_p95,js_without_first_query,top5_overlap
3,q_to_k,cal-zh-1,calibration,1.9073486328125e-06,5.101167139945906e-08,3.5762786865234375e-07,0.12333933264017105,0.12333933264017105,6.144742492608657e-09,248,6.2515415066854985e-09,3.3901077500786414e-08,6.453204193235251e-09,1.0
3,q_to_k,cal-zh-2,calibration,1.9073486328125e-06,5.091678723372976e-08,5.066394805908203e-07,0.1346932351589203,0.1346932053565979,6.3971117292283e-09,248,5.0883031121884414e-09,3.329163433818394e-08,5.252442036862703e-09,1.0
3,q_to_k,cal-en-1,calibration,1.9073486328125e-06,5.108396212348145e-08,3.725290298461914e-07,0.1436830461025238,0.1436830461025238,6.142506059347852e-09,248,5.038649941724316e-09,2.9201892459695955e-08,5.2011870366186486e-09,1.0
3,q_to_k,cal-en-2,calibration,1.9073486328125e-06,5.1701194575127235e-08,3.8743019104003906e-07,0.1500767469406128,0.1500767469406128,6.898510651609513e-09,248,5.6701336959008586e-09,3.627561184771366e-08,5.853041162851014e-09,1.0
3,q_to_k,test-zh-1,heldout,1.9073486328125e-06,5.220060152896206e-08,4.172325134277344e-07,0.09449940919876099,0.09449941664934158,4.331650149680399e-09,248,5.2700910302405646e-09,3.2252330584015e-08,5.440094152930897e-09,1.0
3,q_to_k,test-zh-2,heldout,1.9073486328125e-06,5.0557365349437256e-08,3.5762786865234375e-07,0.12233515083789825,0.12233515828847885,4.755000837519674e-09,248,5.405037306616123e-09,2.982928037909005e-08,5.579393391741405e-09,1.0
3,q_to_k,test-en-1,heldout,1.9073486328125e-06,5.036456447011736e-08,4.76837158203125e-07,0.16284234821796417,0.16284234821796417,3.5884786253603806e-09,248,6.1204539214543274e-09,3.24787201577692e-08,6.317887990547888e-09,1.0
3,q_to_k,test-en-2,heldout,1.9073486328125e-06,5.110061520200993e-08,4.76837158203125e-07,0.1765710860490799,0.1765711009502411,4.401169650947168e-09,248,4.359010929988472e-09,3.009398241715644e-08,4.4996242287709265e-09,1.0
3,k_to_q,cal-zh-1,calibration,1.9073486328125e-06,5.1102714936484615e-08,3.2782554626464844e-07,0.12333933264017105,0.12333932518959045,5.54685852804937e-09,248,6.077549130623083e-09,3.364394274285587e-08,6.273598973649541e-09,1.0
3,k_to_q,cal-zh-2,calibration,1.9073486328125e-06,5.047857377998922e-08,5.066394805908203e-07,0.1346932351589203,0.1346932202577591,5.5039839352843956e-09,248,5.080743381569164e-09,3.255715697036976e-08,5.244638501267218e-09,1.0
3,k_to_q,cal-en-1,calibration,1.9073486328125e-06,5.134420550367216e-08,4.172325134277344e-07,0.1436830461025238,0.1436830461025238,5.426067595237782e-09,248,6.617717041024207e-09,3.062589826186013e-08,6.831191612377552e-09,1.0
3,k_to_q,cal-en-2,calibration,1.9073486328125e-06,5.226853937237364e-08,5.364418029785156e-07,0.1500767469406128,0.1500767469406128,7.92923859904704e-09,248,3.863607211940234e-09,2.7573278771342302e-08,3.988239516417025e-09,1.0
3,k_to_q,test-zh-1,heldout,1.9073486328125e-06,5.123835134005697e-08,4.76837158203125e-07,0.09449940919876099,0.09449940919876099,3.631874356813114e-09,248,5.632322164217385e-09,2.992773318055697e-08,5.814010162197292e-09,1.0
3,k_to_q,test-zh-2,heldout,1.9073486328125e-06,5.0224548540677525e-08,4.172325134277344e-07,0.12233515083789825,0.12233515083789825,5.022106286389771e-09,248,4.692121358118584e-09,3.066226383907633e-08,4.843480283511781e-09,1.0
3,k_to_q,test-en-1,heldout,1.9073486328125e-06,4.9758794253491996e-08,4.76837158203125e-07,0.16284234821796417,0.16284234821796417,3.619864186177324e-09,248,6.1619935820544924e-09,3.4145102745242184e-08,6.360767468294171e-09,1.0
3,k_to_q,test-en-2,heldout,1.9073486328125e-06,5.142298277550361e-08,2.980232238769531e-07,0.1765710860490799,0.1765710860490799,4.500570138787907e-09,248,5.100083466658134e-09,3.277366289466954e-08,5.264602087606818e-09,1.0
7,q_to_k,cal-zh-1,calibration,1.9073486328125e-06,5.222536694121921e-08,4.172325134277344e-07,0.17881479859352112,0.17881479859352112,5.7563616095990255e-09,248,5.532780456007913e-09,3.966695061308201e-08,5.711257244911394e-09,1.0
7,q_to_k,cal-zh-2,calibration,1.9073486328125e-06,5.170127993939104e-08,4.470348358154297e-07,0.1558690071105957,0.1558690071105957,4.651088403306858e-09,248,5.351785681284582e-09,4.158310318302938e-08,5.524424029346164e-09,1.0
7,q_to_k,cal-en-1,calibration,1.9073486328125e-06,5.378773497110193e-08,8.046627044677734e-07,0.16970808804035187,0.16970807313919067,7.066096152641421e-09,248,4.739160619493532e-09,3.499418710362079e-08,4.892036553627577e-09,1.0
7,q_to_k,cal-en-2,calibration,1.9073486328125e-06,5.317526550862134e-08,4.76837158203125e-07,0.1774885654449463,0.1774885654449463,2.2907555852214045e-09,248,4.155392474558539e-09,3.55765088499993e-08,4.289437249838102e-09,1.0
7,q_to_k,test-zh-1,heldout,1.9073486328125e-06,5.211860231803155e-08,5.364418029785156e-07,0.13551732897758484,0.13551732897758484,3.869626841179752e-09,248,4.760383198743057e-09,3.7595043522742344e-08,4.91394391843869e-09,1.0
7,q_to_k,test-zh-2,heldout,1.9073486328125e-06,5.3270193367964134e-08,3.8743019104003906e-07,0.16214892268180847,0.16214892268180847,5.834590144360163e-09,248,3.3417555389547715e-09,3.232060308278051e-08,3.4495541978429856e-09,1.0
7,q_to_k,test-en-1,heldout,1.9073486328125e-06,5.1701003812639285e-08,5.364418029785156e-07,0.16952991485595703,0.16952991485595703,4.620759330720148e-09,248,3.3314184744170916e-09,3.060024056367183e-08,3.4388836223087083e-09,1.0
7,q_to_k,test-en-2,heldout,1.9073486328125e-06,5.3793891417825e-08,5.364418029785156e-07,0.20624876022338867,0.20624877512454987,5.72924729880242e-09,248,5.260036406440349e-09,3.098259782063906e-08,5.429714899918281e-09,1.0
7,k_to_q,cal-zh-1,calibration,1.9073486328125e-06,5.2622550638818123e-08,5.066394805908203e-07,0.17881479859352112,0.17881478369235992,5.618802756401919e-09,248,3.0388158656080577e-09,3.318642782801362e-08,3.1368421193889162e-09,1.0
7,k_to_q,cal-zh-2,calibration,1.9073486328125e-06,5.2178797749660394e-08,4.76837158203125e-07,0.1558690071105957,0.1558690220117569,5.6034279438677e-09,248,6.106143590756119e-09,3.9555089870191296e-08,6.3031158070714355e-09,1.0
7,k_to_q,cal-en-1,calibration,1.9073486328125e-06,5.331118520036339e-08,5.364418029785156e-07,0.16970808804035187,0.16970807313919067,4.860980507004342e-09,248,3.6745022580220166e-09,3.220852917706907e-08,3.7930347751569116e-09,1.0
7,k_to_q,cal-en-2,calibration,1.9073486328125e-06,5.3509503240043504e-08,4.172325134277344e-07,0.1774885654449463,0.1774885505437851,2.684577671274724e-09,248,3.914189861120576e-09,3.4250152936010636e-08,4.040454193443566e-09,1.0
7,k_to_q,test-zh-1,heldout,1.9073486328125e-06,5.186216628978779e-08,5.364418029785156e-07,0.13551732897758484,0.13551732897758484,7.021717873811895e-09,248,3.1829396895943773e-09,3.381542512670421e-08,3.285615113313156e-09,1.0
7,k_to_q,test-zh-2,heldout,1.9073486328125e-06,5.435688576791881e-08,5.960464477539062e-07,0.16214892268180847,0.16214892268180847,4.47947234860635e-09,248,5.434995564712608e-09,3.8530473034370516e-08,5.610317987958524e-09,1.0
7,k_to_q,test-en-1,heldout,1.9073486328125e-06,5.182678341307826e-08,4.76837158203125e-07,0.16952991485595703,0.16952991485595703,5.902526023504606e-09,248,3.811826410071717e-09,3.153426675339688e-08,3.934788495030261e-09,1.0
7,k_to_q,test-en-2,heldout,1.9073486328125e-06,5.389036154327309e-08,5.364418029785156e-07,0.20624876022338867,0.20624877512454987,4.393694741366971e-09,248,6.306869693162298e-09,3.442805152076289e-08,6.510317174246438e-09,1.0
11,q_to_k,cal-zh-1,calibration,1.9073486328125e-06,5.3373433612649885e-08,4.76837158203125e-07,0.24492809176445007,0.24492809176445007,5.236138189701478e-09,248,4.897512173585028e-09,3.2079633172088506e-08,5.055496465899978e-09,1.0
11,q_to_k,cal-zh-2,calibration,1.9073486328125e-06,5.3680472349671996e-08,4.76837158203125e-07,0.18575024604797363,0.18575023114681244,4.706864675796396e-09,248,6.692440379651998e-09,3.841551432515189e-08,6.908325467236409e-09,1.0
11,q_to_k,cal-en-1,calibration,1.9073486328125e-06,5.413658613357613e-08,6.556510925292969e-07,0.24272800981998444,0.24272799491882324,5.221439280944651e-09,248,5.445071948884106e-09,3.742365706216333e-08,5.620719445431632e-09,1.0
11,q_to_k,cal-en-2,calibration,1.9073486328125e-06,5.431145274674744e-08,6.258487701416016e-07,0.2525215446949005,0.2525215148925781,4.965174049686993e-09,248,3.2684353001855015e-09,3.038027784896258e-08,3.373868739942054e-09,1.0
11,q_to_k,test-zh-1,heldout,1.9073486328125e-06,4.975615695445315e-08,4.76837158203125e-07,0.20683012902736664,0.20683012902736664,4.743543779994752e-09,248,5.727621932294369e-09,3.784743896062537e-08,5.912383915784858e-09,1.0
11,q_to_k,test-zh-2,heldout,1.9073486328125e-06,5.341885392276808e-08,5.066394805908203e-07,0.2263796031475067,0.2263796031475067,5.8260534174792156e-09,248,4.675535514309104e-09,3.7589622081668495e-08,4.826359312204431e-09,1.0
11,q_to_k,test-en-1,heldout,1.9073486328125e-06,5.3474540038830716e-08,4.76837158203125e-07,0.242807075381279,0.242807075381279,5.4214881473058085e-09,248,3.967941974991618e-09,2.985759550711009e-08,4.095940031589862e-09,1.0
11,q_to_k,test-en-2,heldout,1.9073486328125e-06,5.447992218148785e-08,5.960464477539062e-07,0.2930097281932831,0.2930097281932831,6.2322125238267745e-09,248,3.5252292196474855e-09,3.577358498318972e-08,3.6389462554353713e-09,1.0
11,k_to_q,cal-zh-1,calibration,1.9073486328125e-06,5.28470006223089e-08,4.470348358154297e-07,0.24492809176445007,0.24492809176445007,5.811663150723234e-09,248,5.133418579106319e-09,3.507095414079231e-08,5.29901278412126e-09,1.0
11,k_to_q,cal-zh-2,calibration,1.9073486328125e-06,5.340567432956712e-08,5.066394805908203e-07,0.18575024604797363,0.18575024604797363,2.0245674026142524e-09,248,5.099674460495862e-09,3.472874965382289e-08,5.26418020285746e-09,1.0
11,k_to_q,cal-en-1,calibration,1.9073486328125e-06,5.450061260733951e-08,5.066394805908203e-07,0.24272800981998444,0.24272799491882324,6.1068083923032646e-09,248,6.209086134134623e-09,3.4522884106991114e-08,6.409379249561198e-09,1.0
11,k_to_q,cal-en-2,calibration,1.9073486328125e-06,5.419026691337401e-08,5.960464477539062e-07,0.2525215446949005,0.2525215446949005,6.355903803267893e-09,248,4.629203242956237e-09,3.0078894042162574e-08,4.778532236571209e-09,1.0
11,k_to_q,test-zh-1,heldout,2.86102294921875e-06,5.101806004082587e-08,4.76837158203125e-07,0.20683012902736664,0.20683011412620544,4.4362411522058665e-09,248,5.5794813214049555e-09,3.09307068846465e-08,5.759464460908248e-09,1.0
11,k_to_q,test-zh-2,heldout,1.9073486328125e-06,5.354370300835508e-08,5.364418029785156e-07,0.2263796031475067,0.22637958824634552,5.91522208992501e-09,248,5.174931594353893e-09,3.6706762074345534e-08,5.341864728336532e-09,1.0
11,k_to_q,test-en-1,heldout,1.9073486328125e-06,5.454591675148741e-08,5.066394805908203e-07,0.242807075381279,0.24280709028244019,4.439097978092832e-09,248,5.318464335601902e-09,3.0657005822831707e-08,5.490027543686438e-09,1.0
11,k_to_q,test-en-2,heldout,2.86102294921875e-06,5.424732981759204e-08,6.854534149169922e-07,0.2930097281932831,0.2930097281932831,6.705069388601714e-09,248,4.751804283387173e-09,3.399427228600871e-08,4.90508833550507e-09,1.0
15,q_to_k,cal-zh-1,calibration,1.9073486328125e-06,5.3514833863035024e-08,4.470348358154297e-07,0.33434635400772095,0.33434635400772095,4.285159338479616e-09,248,4.447109791527737e-09,3.414147897728981e-08,4.590564817164022e-09,1.0
15,q_to_k,cal-zh-2,calibration,1.9073486328125e-06,5.218853708885142e-08,4.76837158203125e-07,0.30227771401405334,0.30227774381637573,5.229982669163746e-09,248,7.101956356336814e-09,4.079232240883357e-08,7.3310517656466345e-09,1.0
15,q_to_k,cal-en-1,calibration,1.9073486328125e-06,5.4103478102894914e-08,4.76837158203125e-07,0.3320315480232239,0.3320315480232239,5.162816396619974e-09,248,4.775797535216952e-09,3.1169314240742096e-08,4.929855634827618e-09,1.0
15,q_to_k,cal-en-2,calibration,1.9073486328125e-06,5.31591652511587e-08,3.5762786865234375e-07,0.35803988575935364,0.35803985595703125,5.665685698375e-09,248,3.281495741802587e-09,3.316486996141066e-08,3.3873504001746824e-09,1.0
15,q_to_k,test-zh-1,heldout,1.9073486328125e-06,5.2584229204196615e-08,5.960464477539062e-07,0.3134230375289917,0.3134230375289917,4.611414805566483e-09,248,3.2052387410885785e-09,3.137078863346687e-08,3.3086335893273144e-09,1.0
15,q_to_k,test-zh-2,heldout,2.86102294921875e-06,5.334021097494736e-08,7.152557373046875e-07,0.31523996591567993,0.31523996591567993,6.134295738036144e-09,248,3.099508649739846e-09,3.3302633539733506e-08,3.1994928928469335e-09,1.0
15,q_to_k,test-en-1,heldout,1.9073486328125e-06,5.277330399580438e-08,5.662441253662109e-07,0.35721129179000854,0.35721129179000854,3.875603393765914e-09,248,6.00105520831562e-09,3.1643388354041235e-08,6.194637691692151e-09,1.0
15,q_to_k,test-en-2,heldout,1.9073486328125e-06,5.4245038413836726e-08,5.364418029785156e-07,0.3785627782344818,0.3785627782344818,5.588637108644434e-09,248,5.273080638801275e-09,3.732484543661485e-08,5.443180128850145e-09,1.0
15,k_to_q,cal-zh-1,calibration,1.9073486328125e-06,5.370409043564381e-08,4.76837158203125e-07,0.33434635400772095,0.33434635400772095,5.413625991934623e-09,248,7.632287690739759e-09,3.8797516310751234e-08,7.878490748680633e-09,1.0
15,k_to_q,cal-zh-2,calibration,1.9073486328125e-06,5.2645265789812776e-08,5.066394805908203e-07,0.30227771401405334,0.30227774381637573,6.007307984390309e-09,248,4.376814466411361e-09,3.4955100147726625e-08,4.518001972542152e-09,1.0
15,k_to_q,cal-en-1,calibration,1.9073486328125e-06,5.3713584045477537e-08,5.364418029785156e-07,0.3320315480232239,0.3320315480232239,3.789843550094929e-09,248,4.10175182707917e-09,3.3631373241860274e-08,4.2340664307971565e-09,1.0
15,k_to_q,cal-en-2,calibration,1.9073486328125e-06,5.329913616712736e-08,4.470348358154297e-07,0.35803988575935364,0.35803985595703125,5.377542411366676e-09,248,3.882600019267102e-09,3.18341726313065e-08,4.007845166853485e-09,1.0
15,k_to_q,test-zh-1,heldout,1.9073486328125e-06,5.1740198395570186e-08,5.960464477539062e-07,0.3134230375289917,0.3134230077266693,5.392958968286621e-09,248,3.644287094317633e-09,2.9920162347707446e-08,3.761844613592302e-09,1.0
15,k_to_q,test-zh-2,heldout,1.9073486328125e-06,5.17587827155604e-08,7.152557373046875e-07,0.31523996591567993,0.31523996591567993,6.041951827739922e-09,248,4.992775970436014e-09,3.2279736217333266e-08,5.1538333600831265e-09,1.0
15,k_to_q,test-en-1,heldout,1.9073486328125e-06,5.262549006299803e-08,6.258487701416016e-07,0.35721129179000854,0.35721129179000854,4.4030894486013494e-09,248,4.485939619769397e-09,3.2921569470545364e-08,4.630647421066669e-09,1.0
15,k_to_q,test-en-2,heldout,1.9073486328125e-06,5.460989035027149e-08,5.662441253662109e-07,0.3785627782344818,0.3785627782344818,6.211682723744616e-09,248,3.997459696591932e-09,3.3428371182253613e-08,4.1264098804560945e-09,1.0
19,q_to_k,cal-zh-1,calibration,1.9073486328125e-06,4.990160271767093e-08,5.364418029785156e-07,0.08087001740932465,0.08087002485990524,4.9786765821124845e-09,248,4.393741370734006e-09,3.151729544015325e-08,4.535475106592912e-09,1.0
19,q_to_k,cal-zh-2,calibration,1.9073486328125e-06,4.9742994871294574e-08,4.76837158203125e-07,0.08035354316234589,0.08035353571176529,5.756638277176762e-09,248,4.049901303204706e-09,3.086056210577226e-08,4.180543466958397e-09,1.0
19,q_to_k,cal-en-1,calibration,3.814697265625e-06,5.087667850358471e-08,5.066394805908203e-07,0.24834147095680237,0.24834147095680237,4.231536010479431e-09,248,4.469023373587788e-09,2.7867795182601185e-08,4.613185389246155e-09,1.0
19,q_to_k,cal-en-2,calibration,2.86102294921875e-06,5.025801246712497e-08,4.470348358154297e-07,0.169009268283844,0.169009268283844,5.525796265004601e-09,248,4.058557934172313e-09,2.8106194704946574e-08,4.1894789859497905e-09,1.0
19,q_to_k,test-zh-1,heldout,1.9073486328125e-06,5.0221761295632774e-08,4.172325134277344e-07,0.06396828591823578,0.06396827846765518,3.353163524622005e-09,248,4.251817564693283e-09,2.829117207170384e-08,4.3889731848878455e-09,1.0
19,q_to_k,test-zh-2,heldout,1.9073486328125e-06,5.0605528455235685e-08,5.960464477539062e-07,0.07303687185049057,0.07303687930107117,5.027091631859548e-09,248,3.928162239930089e-09,3.122080727280263e-08,4.054877322801076e-09,1.0
19,q_to_k,test-en-1,heldout,1.9073486328125e-06,5.211114332289351e-08,5.364418029785156e-07,0.2881055176258087,0.2881055176258087,3.5609850623785633e-09,248,4.370760198213475e-09,3.161588324473996e-08,4.511752305091932e-09,1.0
19,q_to_k,test-en-2,heldout,3.814697265625e-06,5.1614248682850084e-08,4.172325134277344e-07,0.08680859953165054,0.08680859953165054,4.462207936484219e-09,248,4.6642192330637045e-09,3.021447270157296e-08,4.814677989628535e-09,1.0
19,k_to_q,cal-zh-1,calibration,1.9073486328125e-06,5.113471077024349e-08,3.8743019104003906e-07,0.08087001740932465,0.08087001740932465,4.6697485878155476e-09,248,5.351755927307522e-09,3.250153568501446e-08,5.524393387190685e-09,1.0
19,k_to_q,cal-zh-2,calibration,1.9073486328125e-06,5.029181744672548e-08,5.662441253662109e-07,0.08035354316234589,0.08035354316234589,4.907285688915408e-09,248,5.877964781575429e-09,3.508033685761802e-08,6.06757666332669e-09,1.0
19,k_to_q,cal-en-1,calibration,1.9073486328125e-06,5.1180455196819575e-08,5.960464477539062e-07,0.24834147095680237,0.24834147095680237,5.958978199771536e-09,248,3.994703678955602e-09,2.9186557171101413e-08,4.123565044977795e-09,1.0
19,k_to_q,cal-en-2,calibration,1.9073486328125e-06,5.087171697055175e-08,5.662441253662109e-07,0.169009268283844,0.1690092831850052,6.885077397100758e-09,248,4.357643579311343e-09,2.7700329141566726e-08,4.498212913262023e-09,1.0
19,k_to_q,test-zh-1,heldout,1.9073486328125e-06,4.9891522075043136e-08,4.172325134277344e-07,0.06396828591823578,0.06396828591823578,3.918689817083987e-09,248,6.348101155850827e-09,3.146694638189729e-08,6.552878684118468e-09,1.0
19,k_to_q,test-zh-2,heldout,1.9073486328125e-06,5.0542783963068e-08,5.066394805908203e-07,0.07303687185049057,0.07303687185049057,6.0575562343956335e-09,248,3.8445029382216944e-09,2.87921402275515e-08,3.968519290964423e-09,1.0
19,k_to_q,test-en-1,heldout,1.9073486328125e-06,5.202611404804706e-08,5.364418029785156e-07,0.2881055176258087,0.28810548782348633,4.291980548742913e-09,248,5.839524419570807e-09,2.908876339802191e-08,6.027895960158958e-09,1.0
19,k_to_q,test-en-2,heldout,1.9073486328125e-06,5.31186969928777e-08,4.172325134277344e-07,0.08680859953165054,0.08680859208106995,5.296885152716868e-09,248,6.1070810630781125e-09,2.844184265882177e-08,6.304083477459699e-09,1.0
23,q_to_k,cal-zh-1,calibration,1.9073486328125e-06,4.932807759644251e-08,4.470348358154297e-07,0.06724601984024048,0.06724601984024048,4.144036225284253e-09,248,4.558249777630863e-09,2.929247777672117e-08,4.705289935458268e-09,1.0
23,q_to_k,cal-zh-2,calibration,1.9073486328125e-06,4.9029117626786305e-08,4.76837158203125e-07,0.06764671951532364,0.06764672696590424,3.972651985151288e-09,248,4.91240381705893e-09,3.200177900453127e-08,5.070868613898938e-09,1.0
23,q_to_k,cal-en-1,calibration,3.814697265625e-06,4.95385074720159e-08,3.8743019104003906e-07,0.08747006952762604,0.08747006952762604,3.265131720553427e-09,248,3.707365081595526e-09,3.0631660763447144e-08,3.826957417629728e-09,1.0
23,q_to_k,cal-en-2,calibration,1.9073486328125e-06,4.922304127933099e-08,4.76837158203125e-07,0.08903923630714417,0.08903922885656357,5.270068825780072e-09,248,3.997369546482332e-09,2.7743276120872906e-08,4.126317065811236e-09,1.0
23,q_to_k,test-zh-1,heldout,1.9073486328125e-06,4.8364042774388367e-08,3.5762786865234375e-07,0.05931711941957474,0.05931711569428444,3.816367222242434e-09,248,3.653502167466627e-09,2.7387025980374347e-08,3.771357004467291e-09,1.0
23,q_to_k,test-zh-2,heldout,3.814697265625e-06,4.96448251794115e-08,5.364418029785156e-07,0.08312603086233139,0.08312603831291199,5.321914020584018e-09,248,3.6995391194949434e-09,2.9038851323548442e-08,3.818878990813346e-09,1.0
23,q_to_k,test-en-1,heldout,3.814697265625e-06,5.034978202226261e-08,4.023313522338867e-07,0.1146349161863327,0.1146349087357521,4.471692793828197e-09,248,3.4011016225576896e-09,2.928134534840865e-08,3.5108145279849623e-09,1.0
23,q_to_k,test-en-2,heldout,3.814697265625e-06,4.994535705946019e-08,4.470348358154297e-07,0.08886055648326874,0.08886055648326874,4.2449421755463845e-09,248,3.4883309574240684e-09,2.892927319919636e-08,3.600857834129556e-09,1.0
23,k_to_q,cal-zh-1,calibration,2.86102294921875e-06,4.8831644467980875e-08,4.76837158203125e-07,0.06724601984024048,0.06724601984024048,3.977453033598977e-09,248,4.496496952555162e-09,2.8416268449404924e-08,4.64154537027639e-09,1.0
23,k_to_q,cal-zh-2,calibration,1.9073486328125e-06,4.9029117626786305e-08,4.470348358154297e-07,0.06764671951532364,0.06764672696590424,4.682830567759311e-09,248,2.651090902361375e-09,2.4917778063127116e-08,2.736610049680621e-09,1.0
23,k_to_q,cal-en-1,calibration,3.814697265625e-06,5.024324385126141e-08,3.8743019104003906e-07,0.08747006952762604,0.08747006952762604,3.7081571147012937e-09,248,4.977271039763309e-09,3.31379155227296e-08,5.13782838496013e-09,1.0
23,k_to_q,cal-en-2,calibration,1.9073486328125e-06,4.8518648048109764e-08,2.980232238769531e-07,0.08903923630714417,0.08903922885656357,3.85932619195728e-09,248,4.874880499272649e-09,2.9354460195918364e-08,5.0321347089266055e-09,1.0
23,k_to_q,test-zh-1,heldout,3.814697265625e-06,4.978168384729527e-08,4.172325134277344e-07,0.05931711941957474,0.05931711941957474,4.285771737499999e-09,248,4.133625886026948e-09,2.540235399806079e-08,4.266968556265738e-09,1.0
23,k_to_q,test-zh-2,heldout,1.9073486328125e-06,4.943963198590656e-08,6.556510925292969e-07,0.08312603086233139,0.08312603086233139,3.7222060988995054e-09,248,4.004627740528122e-09,2.6530234009669584e-08,4.133809294870616e-09,1.0
23,k_to_q,test-en-1,heldout,3.814697265625e-06,4.9439166765060464e-08,4.023313522338867e-07,0.1146349161863327,0.1146349161863327,6.534782048817078e-09,248,4.612270565473864e-09,2.6872147174117345e-08,4.7610533293607205e-09,1.0
23,k_to_q,test-en-2,heldout,1.9073486328125e-06,4.8735765106561524e-08,4.172325134277344e-07,0.08886055648326874,0.08886055648326874,4.1334708988927105e-09,248,4.0138719015203606e-09,2.8323970724386527e-08,4.1433514397226645e-09,1.0
1 layer direction sample_id split score_max_abs_error score_relative_l2 probability_max_abs_error first_key_mass_a first_key_mass_b js_without_first_key without_first_key_valid_rows js_mean js_p95 js_without_first_query top5_overlap
2 3 q_to_k cal-zh-1 calibration 1.9073486328125e-06 5.101167139945906e-08 3.5762786865234375e-07 0.12333933264017105 0.12333933264017105 6.144742492608657e-09 248 6.2515415066854985e-09 3.3901077500786414e-08 6.453204193235251e-09 1.0
3 3 q_to_k cal-zh-2 calibration 1.9073486328125e-06 5.091678723372976e-08 5.066394805908203e-07 0.1346932351589203 0.1346932053565979 6.3971117292283e-09 248 5.0883031121884414e-09 3.329163433818394e-08 5.252442036862703e-09 1.0
4 3 q_to_k cal-en-1 calibration 1.9073486328125e-06 5.108396212348145e-08 3.725290298461914e-07 0.1436830461025238 0.1436830461025238 6.142506059347852e-09 248 5.038649941724316e-09 2.9201892459695955e-08 5.2011870366186486e-09 1.0
5 3 q_to_k cal-en-2 calibration 1.9073486328125e-06 5.1701194575127235e-08 3.8743019104003906e-07 0.1500767469406128 0.1500767469406128 6.898510651609513e-09 248 5.6701336959008586e-09 3.627561184771366e-08 5.853041162851014e-09 1.0
6 3 q_to_k test-zh-1 heldout 1.9073486328125e-06 5.220060152896206e-08 4.172325134277344e-07 0.09449940919876099 0.09449941664934158 4.331650149680399e-09 248 5.2700910302405646e-09 3.2252330584015e-08 5.440094152930897e-09 1.0
7 3 q_to_k test-zh-2 heldout 1.9073486328125e-06 5.0557365349437256e-08 3.5762786865234375e-07 0.12233515083789825 0.12233515828847885 4.755000837519674e-09 248 5.405037306616123e-09 2.982928037909005e-08 5.579393391741405e-09 1.0
8 3 q_to_k test-en-1 heldout 1.9073486328125e-06 5.036456447011736e-08 4.76837158203125e-07 0.16284234821796417 0.16284234821796417 3.5884786253603806e-09 248 6.1204539214543274e-09 3.24787201577692e-08 6.317887990547888e-09 1.0
9 3 q_to_k test-en-2 heldout 1.9073486328125e-06 5.110061520200993e-08 4.76837158203125e-07 0.1765710860490799 0.1765711009502411 4.401169650947168e-09 248 4.359010929988472e-09 3.009398241715644e-08 4.4996242287709265e-09 1.0
10 3 k_to_q cal-zh-1 calibration 1.9073486328125e-06 5.1102714936484615e-08 3.2782554626464844e-07 0.12333933264017105 0.12333932518959045 5.54685852804937e-09 248 6.077549130623083e-09 3.364394274285587e-08 6.273598973649541e-09 1.0
11 3 k_to_q cal-zh-2 calibration 1.9073486328125e-06 5.047857377998922e-08 5.066394805908203e-07 0.1346932351589203 0.1346932202577591 5.5039839352843956e-09 248 5.080743381569164e-09 3.255715697036976e-08 5.244638501267218e-09 1.0
12 3 k_to_q cal-en-1 calibration 1.9073486328125e-06 5.134420550367216e-08 4.172325134277344e-07 0.1436830461025238 0.1436830461025238 5.426067595237782e-09 248 6.617717041024207e-09 3.062589826186013e-08 6.831191612377552e-09 1.0
13 3 k_to_q cal-en-2 calibration 1.9073486328125e-06 5.226853937237364e-08 5.364418029785156e-07 0.1500767469406128 0.1500767469406128 7.92923859904704e-09 248 3.863607211940234e-09 2.7573278771342302e-08 3.988239516417025e-09 1.0
14 3 k_to_q test-zh-1 heldout 1.9073486328125e-06 5.123835134005697e-08 4.76837158203125e-07 0.09449940919876099 0.09449940919876099 3.631874356813114e-09 248 5.632322164217385e-09 2.992773318055697e-08 5.814010162197292e-09 1.0
15 3 k_to_q test-zh-2 heldout 1.9073486328125e-06 5.0224548540677525e-08 4.172325134277344e-07 0.12233515083789825 0.12233515083789825 5.022106286389771e-09 248 4.692121358118584e-09 3.066226383907633e-08 4.843480283511781e-09 1.0
16 3 k_to_q test-en-1 heldout 1.9073486328125e-06 4.9758794253491996e-08 4.76837158203125e-07 0.16284234821796417 0.16284234821796417 3.619864186177324e-09 248 6.1619935820544924e-09 3.4145102745242184e-08 6.360767468294171e-09 1.0
17 3 k_to_q test-en-2 heldout 1.9073486328125e-06 5.142298277550361e-08 2.980232238769531e-07 0.1765710860490799 0.1765710860490799 4.500570138787907e-09 248 5.100083466658134e-09 3.277366289466954e-08 5.264602087606818e-09 1.0
18 7 q_to_k cal-zh-1 calibration 1.9073486328125e-06 5.222536694121921e-08 4.172325134277344e-07 0.17881479859352112 0.17881479859352112 5.7563616095990255e-09 248 5.532780456007913e-09 3.966695061308201e-08 5.711257244911394e-09 1.0
19 7 q_to_k cal-zh-2 calibration 1.9073486328125e-06 5.170127993939104e-08 4.470348358154297e-07 0.1558690071105957 0.1558690071105957 4.651088403306858e-09 248 5.351785681284582e-09 4.158310318302938e-08 5.524424029346164e-09 1.0
20 7 q_to_k cal-en-1 calibration 1.9073486328125e-06 5.378773497110193e-08 8.046627044677734e-07 0.16970808804035187 0.16970807313919067 7.066096152641421e-09 248 4.739160619493532e-09 3.499418710362079e-08 4.892036553627577e-09 1.0
21 7 q_to_k cal-en-2 calibration 1.9073486328125e-06 5.317526550862134e-08 4.76837158203125e-07 0.1774885654449463 0.1774885654449463 2.2907555852214045e-09 248 4.155392474558539e-09 3.55765088499993e-08 4.289437249838102e-09 1.0
22 7 q_to_k test-zh-1 heldout 1.9073486328125e-06 5.211860231803155e-08 5.364418029785156e-07 0.13551732897758484 0.13551732897758484 3.869626841179752e-09 248 4.760383198743057e-09 3.7595043522742344e-08 4.91394391843869e-09 1.0
23 7 q_to_k test-zh-2 heldout 1.9073486328125e-06 5.3270193367964134e-08 3.8743019104003906e-07 0.16214892268180847 0.16214892268180847 5.834590144360163e-09 248 3.3417555389547715e-09 3.232060308278051e-08 3.4495541978429856e-09 1.0
24 7 q_to_k test-en-1 heldout 1.9073486328125e-06 5.1701003812639285e-08 5.364418029785156e-07 0.16952991485595703 0.16952991485595703 4.620759330720148e-09 248 3.3314184744170916e-09 3.060024056367183e-08 3.4388836223087083e-09 1.0
25 7 q_to_k test-en-2 heldout 1.9073486328125e-06 5.3793891417825e-08 5.364418029785156e-07 0.20624876022338867 0.20624877512454987 5.72924729880242e-09 248 5.260036406440349e-09 3.098259782063906e-08 5.429714899918281e-09 1.0
26 7 k_to_q cal-zh-1 calibration 1.9073486328125e-06 5.2622550638818123e-08 5.066394805908203e-07 0.17881479859352112 0.17881478369235992 5.618802756401919e-09 248 3.0388158656080577e-09 3.318642782801362e-08 3.1368421193889162e-09 1.0
27 7 k_to_q cal-zh-2 calibration 1.9073486328125e-06 5.2178797749660394e-08 4.76837158203125e-07 0.1558690071105957 0.1558690220117569 5.6034279438677e-09 248 6.106143590756119e-09 3.9555089870191296e-08 6.3031158070714355e-09 1.0
28 7 k_to_q cal-en-1 calibration 1.9073486328125e-06 5.331118520036339e-08 5.364418029785156e-07 0.16970808804035187 0.16970807313919067 4.860980507004342e-09 248 3.6745022580220166e-09 3.220852917706907e-08 3.7930347751569116e-09 1.0
29 7 k_to_q cal-en-2 calibration 1.9073486328125e-06 5.3509503240043504e-08 4.172325134277344e-07 0.1774885654449463 0.1774885505437851 2.684577671274724e-09 248 3.914189861120576e-09 3.4250152936010636e-08 4.040454193443566e-09 1.0
30 7 k_to_q test-zh-1 heldout 1.9073486328125e-06 5.186216628978779e-08 5.364418029785156e-07 0.13551732897758484 0.13551732897758484 7.021717873811895e-09 248 3.1829396895943773e-09 3.381542512670421e-08 3.285615113313156e-09 1.0
31 7 k_to_q test-zh-2 heldout 1.9073486328125e-06 5.435688576791881e-08 5.960464477539062e-07 0.16214892268180847 0.16214892268180847 4.47947234860635e-09 248 5.434995564712608e-09 3.8530473034370516e-08 5.610317987958524e-09 1.0
32 7 k_to_q test-en-1 heldout 1.9073486328125e-06 5.182678341307826e-08 4.76837158203125e-07 0.16952991485595703 0.16952991485595703 5.902526023504606e-09 248 3.811826410071717e-09 3.153426675339688e-08 3.934788495030261e-09 1.0
33 7 k_to_q test-en-2 heldout 1.9073486328125e-06 5.389036154327309e-08 5.364418029785156e-07 0.20624876022338867 0.20624877512454987 4.393694741366971e-09 248 6.306869693162298e-09 3.442805152076289e-08 6.510317174246438e-09 1.0
34 11 q_to_k cal-zh-1 calibration 1.9073486328125e-06 5.3373433612649885e-08 4.76837158203125e-07 0.24492809176445007 0.24492809176445007 5.236138189701478e-09 248 4.897512173585028e-09 3.2079633172088506e-08 5.055496465899978e-09 1.0
35 11 q_to_k cal-zh-2 calibration 1.9073486328125e-06 5.3680472349671996e-08 4.76837158203125e-07 0.18575024604797363 0.18575023114681244 4.706864675796396e-09 248 6.692440379651998e-09 3.841551432515189e-08 6.908325467236409e-09 1.0
36 11 q_to_k cal-en-1 calibration 1.9073486328125e-06 5.413658613357613e-08 6.556510925292969e-07 0.24272800981998444 0.24272799491882324 5.221439280944651e-09 248 5.445071948884106e-09 3.742365706216333e-08 5.620719445431632e-09 1.0
37 11 q_to_k cal-en-2 calibration 1.9073486328125e-06 5.431145274674744e-08 6.258487701416016e-07 0.2525215446949005 0.2525215148925781 4.965174049686993e-09 248 3.2684353001855015e-09 3.038027784896258e-08 3.373868739942054e-09 1.0
38 11 q_to_k test-zh-1 heldout 1.9073486328125e-06 4.975615695445315e-08 4.76837158203125e-07 0.20683012902736664 0.20683012902736664 4.743543779994752e-09 248 5.727621932294369e-09 3.784743896062537e-08 5.912383915784858e-09 1.0
39 11 q_to_k test-zh-2 heldout 1.9073486328125e-06 5.341885392276808e-08 5.066394805908203e-07 0.2263796031475067 0.2263796031475067 5.8260534174792156e-09 248 4.675535514309104e-09 3.7589622081668495e-08 4.826359312204431e-09 1.0
40 11 q_to_k test-en-1 heldout 1.9073486328125e-06 5.3474540038830716e-08 4.76837158203125e-07 0.242807075381279 0.242807075381279 5.4214881473058085e-09 248 3.967941974991618e-09 2.985759550711009e-08 4.095940031589862e-09 1.0
41 11 q_to_k test-en-2 heldout 1.9073486328125e-06 5.447992218148785e-08 5.960464477539062e-07 0.2930097281932831 0.2930097281932831 6.2322125238267745e-09 248 3.5252292196474855e-09 3.577358498318972e-08 3.6389462554353713e-09 1.0
42 11 k_to_q cal-zh-1 calibration 1.9073486328125e-06 5.28470006223089e-08 4.470348358154297e-07 0.24492809176445007 0.24492809176445007 5.811663150723234e-09 248 5.133418579106319e-09 3.507095414079231e-08 5.29901278412126e-09 1.0
43 11 k_to_q cal-zh-2 calibration 1.9073486328125e-06 5.340567432956712e-08 5.066394805908203e-07 0.18575024604797363 0.18575024604797363 2.0245674026142524e-09 248 5.099674460495862e-09 3.472874965382289e-08 5.26418020285746e-09 1.0
44 11 k_to_q cal-en-1 calibration 1.9073486328125e-06 5.450061260733951e-08 5.066394805908203e-07 0.24272800981998444 0.24272799491882324 6.1068083923032646e-09 248 6.209086134134623e-09 3.4522884106991114e-08 6.409379249561198e-09 1.0
45 11 k_to_q cal-en-2 calibration 1.9073486328125e-06 5.419026691337401e-08 5.960464477539062e-07 0.2525215446949005 0.2525215446949005 6.355903803267893e-09 248 4.629203242956237e-09 3.0078894042162574e-08 4.778532236571209e-09 1.0
46 11 k_to_q test-zh-1 heldout 2.86102294921875e-06 5.101806004082587e-08 4.76837158203125e-07 0.20683012902736664 0.20683011412620544 4.4362411522058665e-09 248 5.5794813214049555e-09 3.09307068846465e-08 5.759464460908248e-09 1.0
47 11 k_to_q test-zh-2 heldout 1.9073486328125e-06 5.354370300835508e-08 5.364418029785156e-07 0.2263796031475067 0.22637958824634552 5.91522208992501e-09 248 5.174931594353893e-09 3.6706762074345534e-08 5.341864728336532e-09 1.0
48 11 k_to_q test-en-1 heldout 1.9073486328125e-06 5.454591675148741e-08 5.066394805908203e-07 0.242807075381279 0.24280709028244019 4.439097978092832e-09 248 5.318464335601902e-09 3.0657005822831707e-08 5.490027543686438e-09 1.0
49 11 k_to_q test-en-2 heldout 2.86102294921875e-06 5.424732981759204e-08 6.854534149169922e-07 0.2930097281932831 0.2930097281932831 6.705069388601714e-09 248 4.751804283387173e-09 3.399427228600871e-08 4.90508833550507e-09 1.0
50 15 q_to_k cal-zh-1 calibration 1.9073486328125e-06 5.3514833863035024e-08 4.470348358154297e-07 0.33434635400772095 0.33434635400772095 4.285159338479616e-09 248 4.447109791527737e-09 3.414147897728981e-08 4.590564817164022e-09 1.0
51 15 q_to_k cal-zh-2 calibration 1.9073486328125e-06 5.218853708885142e-08 4.76837158203125e-07 0.30227771401405334 0.30227774381637573 5.229982669163746e-09 248 7.101956356336814e-09 4.079232240883357e-08 7.3310517656466345e-09 1.0
52 15 q_to_k cal-en-1 calibration 1.9073486328125e-06 5.4103478102894914e-08 4.76837158203125e-07 0.3320315480232239 0.3320315480232239 5.162816396619974e-09 248 4.775797535216952e-09 3.1169314240742096e-08 4.929855634827618e-09 1.0
53 15 q_to_k cal-en-2 calibration 1.9073486328125e-06 5.31591652511587e-08 3.5762786865234375e-07 0.35803988575935364 0.35803985595703125 5.665685698375e-09 248 3.281495741802587e-09 3.316486996141066e-08 3.3873504001746824e-09 1.0
54 15 q_to_k test-zh-1 heldout 1.9073486328125e-06 5.2584229204196615e-08 5.960464477539062e-07 0.3134230375289917 0.3134230375289917 4.611414805566483e-09 248 3.2052387410885785e-09 3.137078863346687e-08 3.3086335893273144e-09 1.0
55 15 q_to_k test-zh-2 heldout 2.86102294921875e-06 5.334021097494736e-08 7.152557373046875e-07 0.31523996591567993 0.31523996591567993 6.134295738036144e-09 248 3.099508649739846e-09 3.3302633539733506e-08 3.1994928928469335e-09 1.0
56 15 q_to_k test-en-1 heldout 1.9073486328125e-06 5.277330399580438e-08 5.662441253662109e-07 0.35721129179000854 0.35721129179000854 3.875603393765914e-09 248 6.00105520831562e-09 3.1643388354041235e-08 6.194637691692151e-09 1.0
57 15 q_to_k test-en-2 heldout 1.9073486328125e-06 5.4245038413836726e-08 5.364418029785156e-07 0.3785627782344818 0.3785627782344818 5.588637108644434e-09 248 5.273080638801275e-09 3.732484543661485e-08 5.443180128850145e-09 1.0
58 15 k_to_q cal-zh-1 calibration 1.9073486328125e-06 5.370409043564381e-08 4.76837158203125e-07 0.33434635400772095 0.33434635400772095 5.413625991934623e-09 248 7.632287690739759e-09 3.8797516310751234e-08 7.878490748680633e-09 1.0
59 15 k_to_q cal-zh-2 calibration 1.9073486328125e-06 5.2645265789812776e-08 5.066394805908203e-07 0.30227771401405334 0.30227774381637573 6.007307984390309e-09 248 4.376814466411361e-09 3.4955100147726625e-08 4.518001972542152e-09 1.0
60 15 k_to_q cal-en-1 calibration 1.9073486328125e-06 5.3713584045477537e-08 5.364418029785156e-07 0.3320315480232239 0.3320315480232239 3.789843550094929e-09 248 4.10175182707917e-09 3.3631373241860274e-08 4.2340664307971565e-09 1.0
61 15 k_to_q cal-en-2 calibration 1.9073486328125e-06 5.329913616712736e-08 4.470348358154297e-07 0.35803988575935364 0.35803985595703125 5.377542411366676e-09 248 3.882600019267102e-09 3.18341726313065e-08 4.007845166853485e-09 1.0
62 15 k_to_q test-zh-1 heldout 1.9073486328125e-06 5.1740198395570186e-08 5.960464477539062e-07 0.3134230375289917 0.3134230077266693 5.392958968286621e-09 248 3.644287094317633e-09 2.9920162347707446e-08 3.761844613592302e-09 1.0
63 15 k_to_q test-zh-2 heldout 1.9073486328125e-06 5.17587827155604e-08 7.152557373046875e-07 0.31523996591567993 0.31523996591567993 6.041951827739922e-09 248 4.992775970436014e-09 3.2279736217333266e-08 5.1538333600831265e-09 1.0
64 15 k_to_q test-en-1 heldout 1.9073486328125e-06 5.262549006299803e-08 6.258487701416016e-07 0.35721129179000854 0.35721129179000854 4.4030894486013494e-09 248 4.485939619769397e-09 3.2921569470545364e-08 4.630647421066669e-09 1.0
65 15 k_to_q test-en-2 heldout 1.9073486328125e-06 5.460989035027149e-08 5.662441253662109e-07 0.3785627782344818 0.3785627782344818 6.211682723744616e-09 248 3.997459696591932e-09 3.3428371182253613e-08 4.1264098804560945e-09 1.0
66 19 q_to_k cal-zh-1 calibration 1.9073486328125e-06 4.990160271767093e-08 5.364418029785156e-07 0.08087001740932465 0.08087002485990524 4.9786765821124845e-09 248 4.393741370734006e-09 3.151729544015325e-08 4.535475106592912e-09 1.0
67 19 q_to_k cal-zh-2 calibration 1.9073486328125e-06 4.9742994871294574e-08 4.76837158203125e-07 0.08035354316234589 0.08035353571176529 5.756638277176762e-09 248 4.049901303204706e-09 3.086056210577226e-08 4.180543466958397e-09 1.0
68 19 q_to_k cal-en-1 calibration 3.814697265625e-06 5.087667850358471e-08 5.066394805908203e-07 0.24834147095680237 0.24834147095680237 4.231536010479431e-09 248 4.469023373587788e-09 2.7867795182601185e-08 4.613185389246155e-09 1.0
69 19 q_to_k cal-en-2 calibration 2.86102294921875e-06 5.025801246712497e-08 4.470348358154297e-07 0.169009268283844 0.169009268283844 5.525796265004601e-09 248 4.058557934172313e-09 2.8106194704946574e-08 4.1894789859497905e-09 1.0
70 19 q_to_k test-zh-1 heldout 1.9073486328125e-06 5.0221761295632774e-08 4.172325134277344e-07 0.06396828591823578 0.06396827846765518 3.353163524622005e-09 248 4.251817564693283e-09 2.829117207170384e-08 4.3889731848878455e-09 1.0
71 19 q_to_k test-zh-2 heldout 1.9073486328125e-06 5.0605528455235685e-08 5.960464477539062e-07 0.07303687185049057 0.07303687930107117 5.027091631859548e-09 248 3.928162239930089e-09 3.122080727280263e-08 4.054877322801076e-09 1.0
72 19 q_to_k test-en-1 heldout 1.9073486328125e-06 5.211114332289351e-08 5.364418029785156e-07 0.2881055176258087 0.2881055176258087 3.5609850623785633e-09 248 4.370760198213475e-09 3.161588324473996e-08 4.511752305091932e-09 1.0
73 19 q_to_k test-en-2 heldout 3.814697265625e-06 5.1614248682850084e-08 4.172325134277344e-07 0.08680859953165054 0.08680859953165054 4.462207936484219e-09 248 4.6642192330637045e-09 3.021447270157296e-08 4.814677989628535e-09 1.0
74 19 k_to_q cal-zh-1 calibration 1.9073486328125e-06 5.113471077024349e-08 3.8743019104003906e-07 0.08087001740932465 0.08087001740932465 4.6697485878155476e-09 248 5.351755927307522e-09 3.250153568501446e-08 5.524393387190685e-09 1.0
75 19 k_to_q cal-zh-2 calibration 1.9073486328125e-06 5.029181744672548e-08 5.662441253662109e-07 0.08035354316234589 0.08035354316234589 4.907285688915408e-09 248 5.877964781575429e-09 3.508033685761802e-08 6.06757666332669e-09 1.0
76 19 k_to_q cal-en-1 calibration 1.9073486328125e-06 5.1180455196819575e-08 5.960464477539062e-07 0.24834147095680237 0.24834147095680237 5.958978199771536e-09 248 3.994703678955602e-09 2.9186557171101413e-08 4.123565044977795e-09 1.0
77 19 k_to_q cal-en-2 calibration 1.9073486328125e-06 5.087171697055175e-08 5.662441253662109e-07 0.169009268283844 0.1690092831850052 6.885077397100758e-09 248 4.357643579311343e-09 2.7700329141566726e-08 4.498212913262023e-09 1.0
78 19 k_to_q test-zh-1 heldout 1.9073486328125e-06 4.9891522075043136e-08 4.172325134277344e-07 0.06396828591823578 0.06396828591823578 3.918689817083987e-09 248 6.348101155850827e-09 3.146694638189729e-08 6.552878684118468e-09 1.0
79 19 k_to_q test-zh-2 heldout 1.9073486328125e-06 5.0542783963068e-08 5.066394805908203e-07 0.07303687185049057 0.07303687185049057 6.0575562343956335e-09 248 3.8445029382216944e-09 2.87921402275515e-08 3.968519290964423e-09 1.0
80 19 k_to_q test-en-1 heldout 1.9073486328125e-06 5.202611404804706e-08 5.364418029785156e-07 0.2881055176258087 0.28810548782348633 4.291980548742913e-09 248 5.839524419570807e-09 2.908876339802191e-08 6.027895960158958e-09 1.0
81 19 k_to_q test-en-2 heldout 1.9073486328125e-06 5.31186969928777e-08 4.172325134277344e-07 0.08680859953165054 0.08680859208106995 5.296885152716868e-09 248 6.1070810630781125e-09 2.844184265882177e-08 6.304083477459699e-09 1.0
82 23 q_to_k cal-zh-1 calibration 1.9073486328125e-06 4.932807759644251e-08 4.470348358154297e-07 0.06724601984024048 0.06724601984024048 4.144036225284253e-09 248 4.558249777630863e-09 2.929247777672117e-08 4.705289935458268e-09 1.0
83 23 q_to_k cal-zh-2 calibration 1.9073486328125e-06 4.9029117626786305e-08 4.76837158203125e-07 0.06764671951532364 0.06764672696590424 3.972651985151288e-09 248 4.91240381705893e-09 3.200177900453127e-08 5.070868613898938e-09 1.0
84 23 q_to_k cal-en-1 calibration 3.814697265625e-06 4.95385074720159e-08 3.8743019104003906e-07 0.08747006952762604 0.08747006952762604 3.265131720553427e-09 248 3.707365081595526e-09 3.0631660763447144e-08 3.826957417629728e-09 1.0
85 23 q_to_k cal-en-2 calibration 1.9073486328125e-06 4.922304127933099e-08 4.76837158203125e-07 0.08903923630714417 0.08903922885656357 5.270068825780072e-09 248 3.997369546482332e-09 2.7743276120872906e-08 4.126317065811236e-09 1.0
86 23 q_to_k test-zh-1 heldout 1.9073486328125e-06 4.8364042774388367e-08 3.5762786865234375e-07 0.05931711941957474 0.05931711569428444 3.816367222242434e-09 248 3.653502167466627e-09 2.7387025980374347e-08 3.771357004467291e-09 1.0
87 23 q_to_k test-zh-2 heldout 3.814697265625e-06 4.96448251794115e-08 5.364418029785156e-07 0.08312603086233139 0.08312603831291199 5.321914020584018e-09 248 3.6995391194949434e-09 2.9038851323548442e-08 3.818878990813346e-09 1.0
88 23 q_to_k test-en-1 heldout 3.814697265625e-06 5.034978202226261e-08 4.023313522338867e-07 0.1146349161863327 0.1146349087357521 4.471692793828197e-09 248 3.4011016225576896e-09 2.928134534840865e-08 3.5108145279849623e-09 1.0
89 23 q_to_k test-en-2 heldout 3.814697265625e-06 4.994535705946019e-08 4.470348358154297e-07 0.08886055648326874 0.08886055648326874 4.2449421755463845e-09 248 3.4883309574240684e-09 2.892927319919636e-08 3.600857834129556e-09 1.0
90 23 k_to_q cal-zh-1 calibration 2.86102294921875e-06 4.8831644467980875e-08 4.76837158203125e-07 0.06724601984024048 0.06724601984024048 3.977453033598977e-09 248 4.496496952555162e-09 2.8416268449404924e-08 4.64154537027639e-09 1.0
91 23 k_to_q cal-zh-2 calibration 1.9073486328125e-06 4.9029117626786305e-08 4.470348358154297e-07 0.06764671951532364 0.06764672696590424 4.682830567759311e-09 248 2.651090902361375e-09 2.4917778063127116e-08 2.736610049680621e-09 1.0
92 23 k_to_q cal-en-1 calibration 3.814697265625e-06 5.024324385126141e-08 3.8743019104003906e-07 0.08747006952762604 0.08747006952762604 3.7081571147012937e-09 248 4.977271039763309e-09 3.31379155227296e-08 5.13782838496013e-09 1.0
93 23 k_to_q cal-en-2 calibration 1.9073486328125e-06 4.8518648048109764e-08 2.980232238769531e-07 0.08903923630714417 0.08903922885656357 3.85932619195728e-09 248 4.874880499272649e-09 2.9354460195918364e-08 5.0321347089266055e-09 1.0
94 23 k_to_q test-zh-1 heldout 3.814697265625e-06 4.978168384729527e-08 4.172325134277344e-07 0.05931711941957474 0.05931711941957474 4.285771737499999e-09 248 4.133625886026948e-09 2.540235399806079e-08 4.266968556265738e-09 1.0
95 23 k_to_q test-zh-2 heldout 1.9073486328125e-06 4.943963198590656e-08 6.556510925292969e-07 0.08312603086233139 0.08312603086233139 3.7222060988995054e-09 248 4.004627740528122e-09 2.6530234009669584e-08 4.133809294870616e-09 1.0
96 23 k_to_q test-en-1 heldout 3.814697265625e-06 4.9439166765060464e-08 4.023313522338867e-07 0.1146349161863327 0.1146349161863327 6.534782048817078e-09 248 4.612270565473864e-09 2.6872147174117345e-08 4.7610533293607205e-09 1.0
97 23 k_to_q test-en-2 heldout 1.9073486328125e-06 4.8735765106561524e-08 4.172325134277344e-07 0.08886055648326874 0.08886055648326874 4.1334708988927105e-09 248 4.0138719015203606e-09 2.8323970724386527e-08 4.1433514397226645e-09 1.0
@@ -0,0 +1,46 @@
{
"state": "completed",
"records": [],
"interventions": [
{
"layer": 3,
"direction": "q_to_k",
"sequence_nll_deltas": [
3.114725206643243e-07,
-5.23134924712565e-07,
1.7465536439331686e-07,
1.733758607791458e-07
],
"mean_nll_delta": 3.40922052810555e-08,
"sequence_p95_delta": 2.9094994722367315e-07,
"passed": true
},
{
"layer": 3,
"direction": "k_to_q",
"sequence_nll_deltas": [
4.0256782884995346e-07,
-8.45824208006718e-07,
2.1304516684139685e-07,
-3.860568877911419e-07
],
"mean_nll_delta": -1.540670250266274e-07,
"sequence_p95_delta": 3.741394295486699e-07,
"passed": true
}
],
"maximum_score_abs_error": 3.814697265625e-06,
"maximum_probability_abs_error": 8.046627044677734e-07,
"bidirectional_heldout_pass": true,
"elementwise_gain_multiplies_saved_per_token_layer": {
"q_to_k": 1536,
"k_to_q": 384
},
"original_bf16_payload_bytes": 1024,
"factored_lossless_mixed_payload_bytes": 1024,
"fp32_cached_gain_payload_before": 2048,
"fp32_cached_gain_payload_after": 1280,
"claim_limit": "Exact real-arithmetic identity only on non-RoPE coordinates; empirical FP32/NLL checks on tiny text pilot. Not evidence that two existing full layers interchange, not model-wide speedup.",
"elapsed_seconds": 232.08875080291182,
"peak_rss_kib": 83704
}
@@ -0,0 +1,39 @@
Looking in indexes: https://download.pytorch.org/whl/cpu
Collecting torch
Downloading https://download-r2.pytorch.org/whl/cpu/torch-2.15.0%2Bcpu-cp314-cp314-manylinux_2_28_x86_64.whl.metadata (35 kB)
Collecting filelock (from torch)
Downloading filelock-4.2.4-py3-none-any.whl.metadata (2.0 kB)
Collecting typing-extensions>=4.13.0 (from torch)
Downloading typing_extensions-4.16.0-py3-none-any.whl.metadata (3.3 kB)
Collecting setuptools>=77.0.3 (from torch)
Downloading https://download.pytorch.org/whl/setuptools-78.1.0-py3-none-any.whl.metadata (6.6 kB)
Collecting sympy>=1.13.3 (from torch)
Downloading sympy-1.14.0-py3-none-any.whl.metadata (12 kB)
Collecting networkx>=2.5.1 (from torch)
Downloading networkx-3.7-py3-none-any.whl.metadata (6.7 kB)
Collecting jinja2 (from torch)
Downloading https://download.pytorch.org/whl/jinja2-3.1.6-py3-none-any.whl.metadata (2.9 kB)
Collecting fsspec>=0.8.5 (from torch)
Downloading fsspec-2026.9.0-py3-none-any.whl.metadata (10 kB)
Collecting mpmath<1.4,>=1.1.0 (from sympy>=1.13.3->torch)
Downloading mpmath-1.3.0-py3-none-any.whl.metadata (8.6 kB)
Collecting MarkupSafe>=2.0 (from jinja2->torch)
Downloading https://download.pytorch.org/whl/markupsafe-3.0.3-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (2.7 kB)
Downloading https://download-r2.pytorch.org/whl/cpu/torch-2.15.0%2Bcpu-cp314-cp314-manylinux_2_28_x86_64.whl (209.5 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 209.5/209.5 MB 235.8 kB/s eta 0:00:00
Downloading fsspec-2026.9.0-py3-none-any.whl (221 kB)
Downloading networkx-3.7-py3-none-any.whl (2.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 143.1 kB/s eta 0:00:00
Downloading https://download.pytorch.org/whl/setuptools-78.1.0-py3-none-any.whl (1.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 368.1 kB/s eta 0:00:00
Downloading sympy-1.14.0-py3-none-any.whl (6.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.3/6.3 MB 27.0 kB/s eta 0:00:00
Downloading mpmath-1.3.0-py3-none-any.whl (536 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 536.2/536.2 kB 19.7 kB/s eta 0:00:00
Downloading typing_extensions-4.16.0-py3-none-any.whl (45 kB)
Downloading filelock-4.2.4-py3-none-any.whl (162 kB)
Downloading https://download.pytorch.org/whl/jinja2-3.1.6-py3-none-any.whl (134 kB)
Downloading https://download.pytorch.org/whl/markupsafe-3.0.3-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (23 kB)
Installing collected packages: mpmath, typing-extensions, sympy, setuptools, networkx, MarkupSafe, fsspec, filelock, jinja2, torch
Successfully installed MarkupSafe-3.0.3 filelock-4.2.4 fsspec-2026.9.0 jinja2-3.1.6 mpmath-1.3.0 networkx-3.7 setuptools-78.1.0 sympy-1.14.0 torch-2.15.0+cpu typing-extensions-4.16.0
+25
View File
@@ -0,0 +1,25 @@
Looking in indexes: https://download.pytorch.org/whl/cpu
Collecting torch
Downloading https://download-r2.pytorch.org/whl/cpu/torch-2.15.0%2Bcpu-cp314-cp314-manylinux_2_28_x86_64.whl.metadata (35 kB)
Collecting filelock (from torch)
Downloading filelock-4.2.4-py3-none-any.whl.metadata (2.0 kB)
Collecting typing-extensions>=4.13.0 (from torch)
Downloading typing_extensions-4.16.0-py3-none-any.whl.metadata (3.3 kB)
Collecting setuptools>=77.0.3 (from torch)
Downloading https://download.pytorch.org/whl/setuptools-78.1.0-py3-none-any.whl.metadata (6.6 kB)
Collecting sympy>=1.13.3 (from torch)
Downloading sympy-1.14.0-py3-none-any.whl.metadata (12 kB)
Collecting networkx>=2.5.1 (from torch)
Downloading networkx-3.7-py3-none-any.whl.metadata (6.7 kB)
Collecting jinja2 (from torch)
Downloading https://download.pytorch.org/whl/jinja2-3.1.6-py3-none-any.whl.metadata (2.9 kB)
Collecting fsspec>=0.8.5 (from torch)
Downloading fsspec-2026.9.0-py3-none-any.whl.metadata (10 kB)
Collecting mpmath<1.4,>=1.1.0 (from sympy>=1.13.3->torch)
Downloading mpmath-1.3.0-py3-none-any.whl.metadata (8.6 kB)
Collecting MarkupSafe>=2.0 (from jinja2->torch)
Downloading https://download.pytorch.org/whl/markupsafe-3.0.3-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (2.7 kB)
Downloading https://download-r2.pytorch.org/whl/cpu/torch-2.15.0%2Bcpu-cp314-cp314-manylinux_2_28_x86_64.whl (209.5 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━ 145.8/209.5 MB 447.0 kB/s eta 0:02:23
ERROR: Could not install packages due to an OSError: [Errno 28] No space left on device
+34
View File
@@ -0,0 +1,34 @@
{
"repo": "Qwen/Qwen3.5-2B",
"revision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"expected_bytes": 4548351670,
"started_unix": 1794169446.8545816,
"files": [
{
"name": "README.md",
"bytes": 62814,
"sha256": "c0e83a849c776e6fa843d011f023132d942a0f0140d903205bf1c363adad2275",
"upstream_sha256": null
},
{
"name": "config.json",
"bytes": 2908,
"sha256": "ed1c1723241f23f7f4e23430759cbd7dcfb4103cbdfe052bfe7626b57c2615b4",
"upstream_sha256": null
},
{
"name": "model.safetensors-00001-of-00001.safetensors",
"bytes": 4548221488,
"sha256": "aa33250c4fc64891ddfaba3a314fd9542ea371843c387178b425fbcc5ed680b1",
"upstream_sha256": "aa33250c4fc64891ddfaba3a314fd9542ea371843c387178b425fbcc5ed680b1"
},
{
"name": "model.safetensors.index.json",
"bytes": 64460,
"sha256": "aca8afed9da75b0f050b408d270766fd77627f1af401e240f61c3b47d0db02f9",
"upstream_sha256": null
}
],
"license": "apache-2.0",
"completed_unix": 1794173525.0766253
}
+96
View File
@@ -0,0 +1,96 @@
{
"architectures": [
"Qwen3_5ForConditionalGeneration"
],
"image_token_id": 248056,
"model_type": "qwen3_5",
"text_config": {
"attention_bias": false,
"attention_dropout": 0.0,
"attn_output_gate": true,
"dtype": "bfloat16",
"eos_token_id": 248044,
"full_attention_interval": 4,
"head_dim": 256,
"hidden_act": "silu",
"hidden_size": 2048,
"initializer_range": 0.02,
"intermediate_size": 6144,
"layer_types": [
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention"
],
"linear_conv_kernel_dim": 4,
"linear_key_head_dim": 128,
"linear_num_key_heads": 16,
"linear_num_value_heads": 16,
"linear_value_head_dim": 128,
"max_position_embeddings": 262144,
"mlp_only_layers": [],
"model_type": "qwen3_5_text",
"mtp_num_hidden_layers": 1,
"mtp_use_dedicated_embeddings": false,
"num_attention_heads": 8,
"num_hidden_layers": 24,
"num_key_value_heads": 2,
"rms_norm_eps": 1e-06,
"tie_word_embeddings": true,
"use_cache": true,
"vocab_size": 248320,
"mamba_ssm_dtype": "float32",
"rope_parameters": {
"mrope_interleaved": true,
"mrope_section": [
11,
11,
10
],
"rope_type": "default",
"rope_theta": 10000000,
"partial_rotary_factor": 0.25
}
},
"tie_word_embeddings": true,
"transformers_version": "4.57.0.dev0",
"video_token_id": 248057,
"vision_config": {
"deepstack_visual_indexes": [],
"depth": 24,
"hidden_act": "gelu_pytorch_tanh",
"hidden_size": 1024,
"in_channels": 3,
"initializer_range": 0.02,
"intermediate_size": 4096,
"model_type": "qwen3_5",
"num_heads": 16,
"num_position_embeddings": 2304,
"out_hidden_size": 2048,
"patch_size": 16,
"spatial_merge_size": 2,
"temporal_patch_size": 2
},
"vision_end_token_id": 248054,
"vision_start_token_id": 248053
}
@@ -0,0 +1,29 @@
{
"layers": [
5,
6,
20,
21
],
"unit": "post_silu_gate_times_up_scalar_function; keep each output column",
"thresholds": [
0.05,
0.1,
0.2
],
"maximum_directional_relative_l2": 0.2,
"cosine_min": 0.98,
"pearson_min": 0.95,
"rms_floor": "at least within-layer median calibration neuron RMS",
"std_over_rms_min": 0.25,
"effective_energy_tokens_min": 16,
"max_selected_pairs": 1,
"selection": "smallest worst directional calibration error; then layer/id",
"heldout_per_text_error_max": 0.35,
"heldout_quality_mean_nll_delta_max": 0.01,
"heldout_quality_sequence_p95_delta_max": 0.05,
"training": false,
"scale_fitting": false,
"cost_required": false,
"registered_unix": 1794203413.872667
}
@@ -0,0 +1,54 @@
[
{
"layer": 5,
"neurons": 6144,
"eligible_neurons": 2111,
"eligible_unique_pairs_checked": 2227105,
"rms_floor": 0.05843841928142779,
"calibration_pair_counts": {
"0.05": 0,
"0.1": 0,
"0.2": 0
},
"best": null
},
{
"layer": 6,
"neurons": 6144,
"eligible_neurons": 2090,
"eligible_unique_pairs_checked": 2183005,
"rms_floor": 0.05875614556680463,
"calibration_pair_counts": {
"0.05": 0,
"0.1": 0,
"0.2": 0
},
"best": null
},
{
"layer": 20,
"neurons": 6144,
"eligible_neurons": 1699,
"eligible_unique_pairs_checked": 1442451,
"rms_floor": 0.1179593664401982,
"calibration_pair_counts": {
"0.05": 0,
"0.1": 0,
"0.2": 0
},
"best": null
},
{
"layer": 21,
"neurons": 6144,
"eligible_neurons": 1788,
"eligible_unique_pairs_checked": 1597578,
"rms_floor": 0.12231686838397056,
"calibration_pair_counts": {
"0.05": 0,
"0.1": 0,
"0.2": 0
},
"best": null
}
]
@@ -0,0 +1,4 @@
{
"selected": null,
"selection_uses_heldout": false
}
@@ -0,0 +1,10 @@
{
"state": "completed",
"selected": null,
"function_pass": false,
"interventions": [],
"empirical_repeated_function_positive": false,
"claim_limit": "Distinct learned scalar MLP subfunctions on the specified short-text distribution; not whole-layer duplication, global equivalence or broad task-capability proof. No performance requirement.",
"elapsed_seconds": 55.004304428584874,
"peak_rss_kib": 185036
}
+4
View File
@@ -0,0 +1,4 @@
{"layer": 5, "neurons": 6144, "eligible_neurons": 2111, "eligible_unique_pairs_checked": 2227105, "rms_floor": 0.05843841928142779, "calibration_pair_counts": {"0.05": 0, "0.1": 0, "0.2": 0}, "best": null}
{"layer": 6, "neurons": 6144, "eligible_neurons": 2090, "eligible_unique_pairs_checked": 2183005, "rms_floor": 0.05875614556680463, "calibration_pair_counts": {"0.05": 0, "0.1": 0, "0.2": 0}, "best": null}
{"layer": 20, "neurons": 6144, "eligible_neurons": 1699, "eligible_unique_pairs_checked": 1442451, "rms_floor": 0.1179593664401982, "calibration_pair_counts": {"0.05": 0, "0.1": 0, "0.2": 0}, "best": null}
{"layer": 21, "neurons": 6144, "eligible_neurons": 1788, "eligible_unique_pairs_checked": 1597578, "rms_floor": 0.12231686838397056, "calibration_pair_counts": {"0.05": 0, "0.1": 0, "0.2": 0}, "best": null}
@@ -0,0 +1,17 @@
source_layer,role,a,b,sample_id,split,relative_l2,reverse_relative_l2,token_relative_p95,reverse_token_relative_p95,cosine,low_norm_tokens
12,input_layernorm,12,13,cal-zh-1,calibration,0.04185799286154599,0.04309664461005982,0.044446126787094885,0.04595947974911848,0.9995232826957284,0
12,input_layernorm,12,13,cal-zh-2,calibration,0.04383456526622353,0.04521540878040512,0.046012856287805166,0.047545525578214155,0.9994900129478096,0
12,input_layernorm,12,13,cal-en-1,calibration,0.04347907732513803,0.04487576882582582,0.04643377233703883,0.048238404047826186,0.9995243164548383,0
12,input_layernorm,12,13,cal-en-2,calibration,0.041111186492761374,0.04233333726141896,0.04398539072853077,0.04542340159464054,0.9995589314599114,0
12,input_layernorm,12,13,test-zh-1,heldout,0.04230986484298778,0.04359453156137845,0.04521637327744173,0.046674126177753086,0.9995251413848774,0
12,input_layernorm,12,13,test-zh-2,heldout,0.043902033219287144,0.04528510757179746,0.047435978938916296,0.04912927000592789,0.9994870299724158,0
12,input_layernorm,12,13,test-en-1,heldout,0.042965663401383455,0.04430863234604914,0.04650584597139243,0.04815192582322126,0.9995218127753321,0
12,input_layernorm,12,13,test-en-2,heldout,0.04135010433212149,0.042584214188789195,0.046368214514353176,0.047927790990014825,0.9995520355780412,0
13,input_layernorm,12,13,cal-zh-1,calibration,0.041182645479044225,0.04240971565525536,0.04352880565374195,0.045001128308403464,0.9995577794005558,0
13,input_layernorm,12,13,cal-zh-2,calibration,0.04327063750976668,0.04463822187523808,0.045272158865488125,0.04679408388648026,0.999518385770098,0
13,input_layernorm,12,13,cal-en-1,calibration,0.04262779877016355,0.04398654464567335,0.045497400019515064,0.04723836815505396,0.9995547796247127,0
13,input_layernorm,12,13,cal-en-2,calibration,0.040518961963038086,0.04171178072648754,0.043328686448901595,0.04470198445787224,0.9995758632252388,0
13,input_layernorm,12,13,test-zh-1,heldout,0.04162196134899919,0.04289161132734105,0.04428805531751076,0.045755187013951454,0.9995588682914622,0
13,input_layernorm,12,13,test-zh-2,heldout,0.043144621015003826,0.04450396583648517,0.045958426245692637,0.04754094135863426,0.9995211225735321,0
13,input_layernorm,12,13,test-en-1,heldout,0.042127619483222106,0.04343304280377689,0.045197718466321105,0.04683576056261672,0.999550812480402,0
13,input_layernorm,12,13,test-en-2,heldout,0.04076045660321344,0.04197546447097981,0.045561271843477975,0.04710887060271581,0.9995759438544739,0
1 source_layer role a b sample_id split relative_l2 reverse_relative_l2 token_relative_p95 reverse_token_relative_p95 cosine low_norm_tokens
2 12 input_layernorm 12 13 cal-zh-1 calibration 0.04185799286154599 0.04309664461005982 0.044446126787094885 0.04595947974911848 0.9995232826957284 0
3 12 input_layernorm 12 13 cal-zh-2 calibration 0.04383456526622353 0.04521540878040512 0.046012856287805166 0.047545525578214155 0.9994900129478096 0
4 12 input_layernorm 12 13 cal-en-1 calibration 0.04347907732513803 0.04487576882582582 0.04643377233703883 0.048238404047826186 0.9995243164548383 0
5 12 input_layernorm 12 13 cal-en-2 calibration 0.041111186492761374 0.04233333726141896 0.04398539072853077 0.04542340159464054 0.9995589314599114 0
6 12 input_layernorm 12 13 test-zh-1 heldout 0.04230986484298778 0.04359453156137845 0.04521637327744173 0.046674126177753086 0.9995251413848774 0
7 12 input_layernorm 12 13 test-zh-2 heldout 0.043902033219287144 0.04528510757179746 0.047435978938916296 0.04912927000592789 0.9994870299724158 0
8 12 input_layernorm 12 13 test-en-1 heldout 0.042965663401383455 0.04430863234604914 0.04650584597139243 0.04815192582322126 0.9995218127753321 0
9 12 input_layernorm 12 13 test-en-2 heldout 0.04135010433212149 0.042584214188789195 0.046368214514353176 0.047927790990014825 0.9995520355780412 0
10 13 input_layernorm 12 13 cal-zh-1 calibration 0.041182645479044225 0.04240971565525536 0.04352880565374195 0.045001128308403464 0.9995577794005558 0
11 13 input_layernorm 12 13 cal-zh-2 calibration 0.04327063750976668 0.04463822187523808 0.045272158865488125 0.04679408388648026 0.999518385770098 0
12 13 input_layernorm 12 13 cal-en-1 calibration 0.04262779877016355 0.04398654464567335 0.045497400019515064 0.04723836815505396 0.9995547796247127 0
13 13 input_layernorm 12 13 cal-en-2 calibration 0.040518961963038086 0.04171178072648754 0.043328686448901595 0.04470198445787224 0.9995758632252388 0
14 13 input_layernorm 12 13 test-zh-1 heldout 0.04162196134899919 0.04289161132734105 0.04428805531751076 0.045755187013951454 0.9995588682914622 0
15 13 input_layernorm 12 13 test-zh-2 heldout 0.043144621015003826 0.04450396583648517 0.045958426245692637 0.04754094135863426 0.9995211225735321 0
16 13 input_layernorm 12 13 test-en-1 heldout 0.042127619483222106 0.04343304280377689 0.045197718466321105 0.04683576056261672 0.999550812480402 0
17 13 input_layernorm 12 13 test-en-2 heldout 0.04076045660321344 0.04197546447097981 0.045561271843477975 0.04710887060271581 0.9995759438544739 0
@@ -0,0 +1,44 @@
{
"roles": [
"input_layernorm",
"post_attention_layernorm"
],
"layers": [
0,
1,
2,
3,
4,
5,
6,
7,
8,
9,
10,
11,
12,
13,
14,
15,
16,
17,
18,
19,
20,
21,
22,
23
],
"same_role_only": true,
"same_eps": 1e-06,
"uniform_worst_error_limit": 0.2,
"empirical_function_worst_relative_error_limit": 0.05,
"selection": "smallest maximum two-direction uniform bound, then role/layer; no input data used",
"minimum_gain_standard_deviation": 0.001,
"max_selected_pairs": 1,
"heldout_nll_mean_delta_limit": 0.01,
"heldout_nll_sequence_p95_delta_limit": 0.05,
"training": false,
"cost_required": false,
"registered_unix": 1794203690.521922
}
@@ -0,0 +1,23 @@
{
"selected": [
{
"role": "input_layernorm",
"a": 12,
"b": 13,
"a_to_b_uniform_bound": 0.10185185185185185,
"b_to_a_uniform_bound": 0.1134020618556701,
"worst_uniform_bound": 0.1134020618556701,
"eligible_nonconstant_gains": true,
"gain_std_a": 0.1900545060634613,
"gain_std_b": 0.16846242547035217,
"gain_min_a": 0.5546875,
"gain_min_b": 0.5703125
}
],
"uniform_bound_counts": {
"0.05": 0,
"0.1": 0,
"0.2": 18
},
"selection_reads_no_input_data": true
}
@@ -0,0 +1,59 @@
{
"state": "completed",
"selected": {
"role": "input_layernorm",
"a": 12,
"b": 13,
"a_to_b_uniform_bound": 0.10185185185185185,
"b_to_a_uniform_bound": 0.1134020618556701,
"worst_uniform_bound": 0.1134020618556701,
"eligible_nonconstant_gains": true,
"gain_std_a": 0.1900545060634613,
"gain_std_b": 0.16846242547035217,
"gain_min_a": 0.5546875,
"gain_min_b": 0.5703125
},
"interventions": [
{
"target": 12,
"donor": 13,
"sequence_nll_deltas": [
0.0010516456964495546,
-0.0013996762182379996,
-0.00012664743940504763,
0.0006049385086996928
],
"mean_nll_delta": 3.256513687655005e-05,
"sequence_p95_nll_delta": 0.0009846396182870752,
"passed_quality": true
},
{
"target": 13,
"donor": 12,
"sequence_nll_deltas": [
0.0008813779918703801,
-0.0012936356267054805,
-0.0002630009203641315,
-0.0007806334344149235
],
"mean_nll_delta": -0.00036397299740353883,
"sequence_p95_nll_delta": 0.000709721155035203,
"passed_quality": true
}
],
"uniform_bound_counts": {
"0.05": 0,
"0.1": 0,
"0.2": 18
},
"uniform_function_similarity_proven": true,
"heldout_function_pass": true,
"worst_heldout_relative_l2": 0.04528510757179746,
"worst_heldout_cosine": 0.9994870299724158,
"untouched_full_forward_bitwise_matches_baseline": true,
"learned_function_repetition_positive": true,
"bidirectional_network_quality_pass": true,
"claim_limit": "Existing learned RMSNorm module functions with identical eps and shape; uniform real-arithmetic relative bound and tiny real-text probes. Does not establish attention/MLP-body duplication or broad task capability.",
"elapsed_seconds": 369.7967556482181,
"peak_rss_kib": 71596
}
@@ -0,0 +1,553 @@
role,a,b,a_to_b_uniform_bound,b_to_a_uniform_bound,worst_uniform_bound,eligible_nonconstant_gains,gain_std_a,gain_std_b,gain_min_a,gain_min_b
input_layernorm,0,1,0.6032863849765259,1.5207100591715976,1.5207100591715976,True,0.11805251985788345,0.0930413156747818,0.794921875,0.607421875
input_layernorm,0,2,0.5692488262910798,1.3215258855585832,1.3215258855585832,True,0.11805251985788345,0.10473614931106567,0.794921875,0.6484375
input_layernorm,0,3,0.5469483568075117,1.2072538860103628,1.2072538860103628,True,0.11805251985788345,0.07621725648641586,0.794921875,0.7265625
input_layernorm,0,4,0.5751173708920188,1.3535911602209945,1.3535911602209945,True,0.11805251985788345,0.10847259312868118,0.794921875,0.662109375
input_layernorm,0,5,0.6396713615023474,1.775244299674267,1.775244299674267,True,0.11805251985788345,0.13199757039546967,0.794921875,0.5234375
input_layernorm,0,6,0.6643192488262911,1.979020979020979,1.979020979020979,True,0.11805251985788345,0.1498883068561554,0.794921875,0.47265625
input_layernorm,0,7,0.6830985915492958,2.1555555555555554,2.1555555555555554,True,0.11805251985788345,0.095920130610466,0.794921875,0.46484375
input_layernorm,0,8,0.6913145539906104,2.2395437262357416,2.2395437262357416,True,0.11805251985788345,0.20054922997951508,0.794921875,0.41796875
input_layernorm,0,9,0.7136150234741784,2.4918032786885247,2.4918032786885247,True,0.11805251985788345,0.22649425268173218,0.794921875,0.42578125
input_layernorm,0,10,0.7230046948356808,2.610169491525424,2.610169491525424,True,0.11805251985788345,0.2204512506723404,0.794921875,0.3828125
input_layernorm,0,11,0.7347417840375586,2.769911504424779,2.769911504424779,True,0.11805251985788345,0.16603894531726837,0.794921875,0.34375
input_layernorm,0,12,0.6338028169014085,1.7307692307692308,1.7307692307692308,True,0.11805251985788345,0.1900545060634613,0.794921875,0.5546875
input_layernorm,0,13,0.6349765258215962,1.7395498392282958,1.7395498392282958,True,0.11805251985788345,0.16846242547035217,0.794921875,0.5703125
input_layernorm,0,14,0.5516431924882629,1.2303664921465969,1.2303664921465969,True,0.11805251985788345,0.11825777590274811,0.794921875,0.6875
input_layernorm,0,15,0.49018199233716475,0.9614842649131048,0.9614842649131048,True,0.11805251985788345,0.06029079854488373,0.794921875,0.779296875
input_layernorm,0,16,0.5117370892018779,1.0480769230769231,1.0480769230769231,True,0.11805251985788345,0.09195870161056519,0.794921875,0.7763671875
input_layernorm,0,17,0.6232876712328768,0.7185185185185186,0.7185185185185186,True,0.11805251985788345,0.06483674794435501,0.794921875,1.0012359619140625
input_layernorm,0,18,0.8402948402948403,0.6637450199203188,0.8402948402948403,True,0.11805251985788345,0.06785038113594055,0.794921875,1.1640625
input_layernorm,0,19,1.0393120393120394,1.1248445660283513,1.1248445660283513,True,0.11805251985788345,0.19409698247909546,0.794921875,0.884765625
input_layernorm,0,20,1.113022113022113,0.9869156655168312,1.113022113022113,True,0.11805251985788345,0.08625888079404831,0.794921875,1.0262451171875
input_layernorm,0,21,1.1867321867321867,0.9545986426398315,1.1867321867321867,True,0.11805251985788345,0.07095508277416229,0.794921875,1.043212890625
input_layernorm,0,22,0.9656019656019657,0.9514018691588785,0.9656019656019657,True,0.11805251985788345,0.07214703410863876,0.794921875,1.044921875
input_layernorm,0,23,0.941031941031941,0.9568884723523898,0.9568884723523898,True,0.11805251985788345,0.15369786322116852,0.794921875,0.98577880859375
input_layernorm,1,2,0.16765578635014836,0.17593150850114556,0.17593150850114556,True,0.0930413156747818,0.10473614931106567,0.607421875,0.6484375
input_layernorm,1,3,0.28125,0.34967320261437906,0.34967320261437906,True,0.0930413156747818,0.07621725648641586,0.607421875,0.7265625
input_layernorm,1,4,0.1678622668579627,0.15668366741786266,0.1678622668579627,True,0.0930413156747818,0.10847259312868118,0.607421875,0.662109375
input_layernorm,1,5,0.23018147086914995,0.2990074441687345,0.2990074441687345,True,0.0930413156747818,0.13199757039546967,0.607421875,0.5234375
input_layernorm,1,6,0.34367141659681477,0.5236270753512133,0.5236270753512133,True,0.0930413156747818,0.1498883068561554,0.607421875,0.47265625
input_layernorm,1,7,0.3996670702179177,0.6657423745903706,0.6657423745903706,True,0.0930413156747818,0.095920130610466,0.607421875,0.46484375
input_layernorm,1,8,0.4585726004922067,0.8469696969696969,0.8469696969696969,True,0.0930413156747818,0.20054922997951508,0.607421875,0.41796875
input_layernorm,1,9,0.5138306789606035,1.056896551724138,1.056896551724138,True,0.0930413156747818,0.22649425268173218,0.607421875,0.42578125
input_layernorm,1,10,0.4786253143336127,0.9180064308681672,0.9180064308681672,True,0.0930413156747818,0.2204512506723404,0.607421875,0.3828125
input_layernorm,1,11,0.49209302325581394,0.9688644688644689,0.9688644688644689,True,0.0930413156747818,0.16603894531726837,0.607421875,0.34375
input_layernorm,1,12,0.3838141025641026,0.6228868660598179,0.6228868660598179,True,0.0930413156747818,0.1900545060634613,0.607421875,0.5546875
input_layernorm,1,13,0.39622641509433965,0.65625,0.65625,True,0.0930413156747818,0.16846242547035217,0.607421875,0.5703125
input_layernorm,1,14,0.27009646302250806,0.28465608465608466,0.28465608465608466,True,0.0930413156747818,0.11825777590274811,0.607421875,0.6875
input_layernorm,1,15,0.9292604501607717,0.5223386651958081,0.9292604501607717,True,0.0930413156747818,0.06029079854488373,0.607421875,0.779296875
input_layernorm,1,16,0.36012861736334406,0.2647754137115839,0.36012861736334406,True,0.0930413156747818,0.09195870161056519,0.607421875,0.7763671875
input_layernorm,1,17,0.7958860759493671,0.4431718061674009,0.7958860759493671,True,0.0930413156747818,0.06483674794435501,0.607421875,1.0012359619140625
input_layernorm,1,18,1.1044303797468353,0.524812030075188,1.1044303797468353,True,0.0930413156747818,0.06785038113594055,0.607421875,1.1640625
input_layernorm,1,19,1.3858520900321543,0.5808625336927223,1.3858520900321543,True,0.0930413156747818,0.19409698247909546,0.607421875,0.884765625
input_layernorm,1,20,1.206896551724138,0.546875,1.206896551724138,True,0.0930413156747818,0.08625888079404831,0.607421875,1.0262451171875
input_layernorm,1,21,1.630225080385852,0.6198044009779952,1.630225080385852,True,0.0930413156747818,0.07095508277416229,0.607421875,1.043212890625
input_layernorm,1,22,1.4043887147335423,0.5840938722294654,1.4043887147335423,True,0.0930413156747818,0.07214703410863876,0.607421875,1.044921875
input_layernorm,1,23,1.9453376205787782,0.6604803493449781,1.9453376205787782,True,0.0930413156747818,0.15369786322116852,0.607421875,0.98577880859375
input_layernorm,2,3,0.2867132867132867,0.4019607843137255,0.4019607843137255,True,0.10473614931106567,0.07621725648641586,0.6484375,0.7265625
input_layernorm,2,4,0.10583941605839416,0.11836734693877551,0.11836734693877551,True,0.10473614931106567,0.10847259312868118,0.6484375,0.662109375
input_layernorm,2,5,0.23737099402498643,0.31125356125356124,0.31125356125356124,True,0.10473614931106567,0.13199757039546967,0.6484375,0.5234375
input_layernorm,2,6,0.31070496083550914,0.45075757575757575,0.45075757575757575,True,0.10473614931106567,0.1498883068561554,0.6484375,0.47265625
input_layernorm,2,7,0.42205710955710957,0.7302747668263171,0.7302747668263171,True,0.10473614931106567,0.095920130610466,0.6484375,0.46484375
input_layernorm,2,8,0.4018264840182648,0.6717557251908397,0.6717557251908397,True,0.10473614931106567,0.20054922997951508,0.6484375,0.41796875
input_layernorm,2,9,0.4722187382415653,0.8947243346007605,0.8947243346007605,True,0.10473614931106567,0.22649425268173218,0.6484375,0.42578125
input_layernorm,2,10,0.46013404206147446,0.8523116438356164,0.8523116438356164,True,0.10473614931106567,0.2204512506723404,0.6484375,0.3828125
input_layernorm,2,11,0.5159817351598174,1.0660377358490567,1.0660377358490567,True,0.10473614931106567,0.16603894531726837,0.6484375,0.34375
input_layernorm,2,12,0.30617848970251715,0.4412928759894459,0.4412928759894459,True,0.10473614931106567,0.1900545060634613,0.6484375,0.5546875
input_layernorm,2,13,0.2900036175087423,0.4084578804347826,0.4084578804347826,True,0.10473614931106567,0.16846242547035217,0.6484375,0.5703125
input_layernorm,2,14,0.19501226492232215,0.24225495175215844,0.24225495175215844,True,0.10473614931106567,0.11825777590274811,0.6484375,0.6875
input_layernorm,2,15,0.7957142857142857,0.5678391959798995,0.7957142857142857,True,0.10473614931106567,0.06029079854488373,0.6484375,0.779296875
input_layernorm,2,16,0.2740963855421687,0.21513002364066194,0.2740963855421687,True,0.10473614931106567,0.09195870161056519,0.6484375,0.7763671875
input_layernorm,2,17,0.7093373493975904,0.4149779735682819,0.7093373493975904,True,0.10473614931106567,0.06483674794435501,0.6484375,1.0012359619140625
input_layernorm,2,18,1.0030120481927711,0.5007518796992482,1.0030120481927711,True,0.10473614931106567,0.06785038113594055,0.6484375,1.1640625
input_layernorm,2,19,1.2680722891566265,0.5590969455511288,1.2680722891566265,True,0.10473614931106567,0.19409698247909546,0.6484375,0.884765625
input_layernorm,2,20,1.0873493975903614,0.5209235209235209,1.0873493975903614,True,0.10473614931106567,0.08625888079404831,0.6484375,1.0262451171875
input_layernorm,2,21,1.463855421686747,0.5941320293398533,1.463855421686747,True,0.10473614931106567,0.07095508277416229,0.6484375,1.043212890625
input_layernorm,2,22,1.249266862170088,0.5554106910039114,1.249266862170088,True,0.10473614931106567,0.07214703410863876,0.6484375,1.044921875
input_layernorm,2,23,1.7448680351906158,0.6356837606837606,1.7448680351906158,True,0.10473614931106567,0.15369786322116852,0.6484375,0.98577880859375
input_layernorm,3,4,0.40522875816993464,0.28837209302325584,0.40522875816993464,True,0.07621725648641586,0.10847259312868118,0.7265625,0.662109375
input_layernorm,3,5,0.3660130718954248,0.39925373134328357,0.39925373134328357,True,0.07621725648641586,0.13199757039546967,0.7265625,0.5234375
input_layernorm,3,6,0.35609756097560974,0.553030303030303,0.553030303030303,True,0.07621725648641586,0.1498883068561554,0.7265625,0.47265625
input_layernorm,3,7,0.39575530586766544,0.6549586776859504,0.6549586776859504,True,0.07621725648641586,0.095920130610466,0.7265625,0.46484375
input_layernorm,3,8,0.42473118279569894,0.7383177570093458,0.7383177570093458,True,0.07621725648641586,0.20054922997951508,0.7265625,0.41796875
input_layernorm,3,9,0.49180741114192084,0.9677579365079365,0.9677579365079365,True,0.07621725648641586,0.22649425268173218,0.7265625,0.42578125
input_layernorm,3,10,0.47804878048780486,0.9158878504672897,0.9158878504672897,True,0.07621725648641586,0.2204512506723404,0.7265625,0.3828125
input_layernorm,3,11,0.545338441890166,1.199438202247191,1.199438202247191,True,0.07621725648641586,0.16603894531726837,0.7265625,0.34375
input_layernorm,3,12,0.40460669274228594,0.5290237467018469,0.5290237467018469,True,0.07621725648641586,0.1900545060634613,0.7265625,0.5546875
input_layernorm,3,13,0.3559322033898305,0.5441576086956522,0.5441576086956522,True,0.07621725648641586,0.16846242547035217,0.7265625,0.5703125
input_layernorm,3,14,0.28804587560652845,0.22363013698630138,0.28804587560652845,True,0.07621725648641586,0.11825777590274811,0.7265625,0.6875
input_layernorm,3,15,0.5853333333333334,0.36921783010933557,0.5853333333333334,True,0.07621725648641586,0.06029079854488373,0.7265625,0.779296875
input_layernorm,3,16,0.43790849673202614,0.30454545454545456,0.43790849673202614,True,0.07621725648641586,0.09195870161056519,0.7265625,0.7763671875
input_layernorm,3,17,0.5490196078431373,0.35443037974683544,0.5490196078431373,True,0.07621725648641586,0.06483674794435501,0.7265625,1.0012359619140625
input_layernorm,3,18,0.7876344086021505,0.4406015037593985,0.7876344086021505,True,0.07621725648641586,0.06785038113594055,0.7265625,1.1640625
input_layernorm,3,19,1.0241935483870968,0.5059760956175299,1.0241935483870968,True,0.07621725648641586,0.19409698247909546,0.7265625,0.884765625
input_layernorm,3,20,0.9153567110036276,0.4779040404040404,0.9153567110036276,True,0.07621725648641586,0.08625888079404831,0.7265625,1.0262451171875
input_layernorm,3,21,1.1706666666666667,0.5393120393120393,1.1706666666666667,True,0.07621725648641586,0.07095508277416229,0.7265625,1.043212890625
input_layernorm,3,22,1.0453333333333332,0.5110821382007823,1.0453333333333332,True,0.07621725648641586,0.07214703410863876,0.7265625,1.044921875
input_layernorm,3,23,1.496,0.5993589743589743,1.496,True,0.07621725648641586,0.15369786322116852,0.7265625,0.98577880859375
input_layernorm,4,5,0.21176470588235294,0.26865671641791045,0.26865671641791045,True,0.10847259312868118,0.13199757039546967,0.662109375,0.5234375
input_layernorm,4,6,0.2985507246376812,0.4256198347107438,0.4256198347107438,True,0.10847259312868118,0.1498883068561554,0.662109375,0.47265625
input_layernorm,4,7,0.42340116279069767,0.7343080413410638,0.7343080413410638,True,0.10847259312868118,0.095920130610466,0.662109375,0.46484375
input_layernorm,4,8,0.37971014492753624,0.6121495327102804,0.6121495327102804,True,0.10847259312868118,0.20054922997951508,0.662109375,0.41796875
input_layernorm,4,9,0.4610917537746806,0.8556034482758621,0.8556034482758621,True,0.10847259312868118,0.22649425268173218,0.662109375,0.42578125
input_layernorm,4,10,0.4318840579710145,0.7602040816326531,0.7602040816326531,True,0.10847259312868118,0.2204512506723404,0.662109375,0.3828125
input_layernorm,4,11,0.5041140976412507,1.0165929203539823,1.0165929203539823,True,0.10847259312868118,0.16603894531726837,0.662109375,0.34375
input_layernorm,4,12,0.31369924141008476,0.45708712613784136,0.45708712613784136,True,0.10847259312868118,0.1900545060634613,0.662109375,0.5546875
input_layernorm,4,13,0.3016249555212905,0.4318953804347826,0.4318953804347826,True,0.10847259312868118,0.16846242547035217,0.662109375,0.5703125
input_layernorm,4,14,0.177762982689747,0.21619433198380567,0.21619433198380567,True,0.10847259312868118,0.11825777590274811,0.662109375,0.6875
input_layernorm,4,15,0.799410029498525,0.5714938328003655,0.799410029498525,True,0.10847259312868118,0.06029079854488373,0.662109375,0.779296875
input_layernorm,4,16,0.2727919383521043,0.21432563338301044,0.2727919383521043,True,0.10847259312868118,0.09195870161056519,0.662109375,0.7763671875
input_layernorm,4,17,0.644927536231884,0.3920704845814978,0.644927536231884,True,0.10847259312868118,0.06483674794435501,0.662109375,1.0012359619140625
input_layernorm,4,18,0.927536231884058,0.48120300751879697,0.927536231884058,True,0.10847259312868118,0.06785038113594055,0.662109375,1.1640625
input_layernorm,4,19,1.191740412979351,0.5437415881561238,1.191740412979351,True,0.10847259312868118,0.19409698247909546,0.662109375,0.884765625
input_layernorm,4,20,1.0705882352941176,0.5170454545454546,1.0705882352941176,True,0.10847259312868118,0.08625888079404831,0.662109375,1.0262451171875
input_layernorm,4,21,1.4129793510324484,0.5855745721271394,1.4129793510324484,True,0.10847259312868118,0.07095508277416229,0.662109375,1.043212890625
input_layernorm,4,22,1.2558823529411764,0.5567144719687093,1.2558823529411764,True,0.10847259312868118,0.07214703410863876,0.662109375,1.044921875
input_layernorm,4,23,1.7529411764705882,0.6367521367521367,1.7529411764705882,True,0.10847259312868118,0.15369786322116852,0.662109375,0.98577880859375
input_layernorm,5,6,0.21281039684381528,0.27034198113207547,0.27034198113207547,True,0.13199757039546967,0.1498883068561554,0.5234375,0.47265625
input_layernorm,5,7,0.40684808612440193,0.6859087471641039,0.6859087471641039,True,0.13199757039546967,0.095920130610466,0.5234375,0.46484375
input_layernorm,5,8,0.3513911921872121,0.5417613636363636,0.5417613636363636,True,0.13199757039546967,0.20054922997951508,0.5234375,0.41796875
input_layernorm,5,9,0.41179234202490017,0.7000798722044729,0.7000798722044729,True,0.13199757039546967,0.22649425268173218,0.5234375,0.42578125
input_layernorm,5,10,0.35843218153687467,0.5586816720257235,0.5586816720257235,True,0.13199757039546967,0.2204512506723404,0.5234375,0.3828125
input_layernorm,5,11,0.4428827751196172,0.7949543746645196,0.7949543746645196,True,0.13199757039546967,0.16603894531726837,0.5234375,0.34375
input_layernorm,5,12,0.2774254169603007,0.3839401820546164,0.3839401820546164,True,0.13199757039546967,0.1900545060634613,0.5234375,0.5546875
input_layernorm,5,13,0.2767029052269517,0.3825577445652174,0.3825577445652174,True,0.13199757039546967,0.16846242547035217,0.5234375,0.5703125
input_layernorm,5,14,0.4141791044776119,0.2928759894459103,0.4141791044776119,True,0.13199757039546967,0.11825777590274811,0.5234375,0.6875
input_layernorm,5,15,1.2182835820895523,0.5492010092514719,1.2182835820895523,True,0.13199757039546967,0.06029079854488373,0.5234375,0.779296875
input_layernorm,5,16,0.5316051136363636,0.34709019244145606,0.5316051136363636,True,0.13199757039546967,0.09195870161056519,0.5234375,0.7763671875
input_layernorm,5,17,1.0124113475177305,0.5030837004405286,1.0124113475177305,True,0.13199757039546967,0.06483674794435501,0.5234375,1.0012359619140625
input_layernorm,5,18,1.3581560283687943,0.5759398496240602,1.3581560283687943,True,0.13199757039546967,0.06785038113594055,0.5234375,1.1640625
input_layernorm,5,19,1.7798507462686568,0.640268456375839,1.7798507462686568,True,0.13199757039546967,0.19409698247909546,0.5234375,0.884765625
input_layernorm,5,20,1.626865671641791,0.6193181818181818,1.626865671641791,True,0.13199757039546967,0.08625888079404831,0.5234375,1.0262451171875
input_layernorm,5,21,2.037313432835821,0.6707616707616708,2.037313432835821,True,0.13199757039546967,0.07095508277416229,0.5234375,1.043212890625
input_layernorm,5,22,1.8619402985074627,0.6505867014341591,1.8619402985074627,True,0.13199757039546967,0.07214703410863876,0.5234375,1.044921875
input_layernorm,5,23,2.4925373134328357,0.7136752136752137,2.4925373134328357,True,0.13199757039546967,0.15369786322116852,0.5234375,0.98577880859375
input_layernorm,6,7,0.37231012658227847,0.5931434333249307,0.5931434333249307,True,0.1498883068561554,0.095920130610466,0.47265625,0.46484375
input_layernorm,6,8,0.1901840490797546,0.23484848484848486,0.23484848484848486,True,0.1498883068561554,0.20054922997951508,0.47265625,0.41796875
input_layernorm,6,9,0.26697892271662765,0.36421725239616615,0.36421725239616615,True,0.1498883068561554,0.22649425268173218,0.47265625,0.42578125
input_layernorm,6,10,0.2579185520361991,0.3475609756097561,0.3475609756097561,True,0.1498883068561554,0.2204512506723404,0.47265625,0.3828125
input_layernorm,6,11,0.41044303797468357,0.6961889425657541,0.6961889425657541,True,0.1498883068561554,0.16603894531726837,0.47265625,0.34375
input_layernorm,6,12,0.2694422623723488,0.21225247524752475,0.2694422623723488,True,0.1498883068561554,0.1900545060634613,0.47265625,0.5546875
input_layernorm,6,13,0.2518703241895262,0.22759714512291832,0.2518703241895262,True,0.1498883068561554,0.16846242547035217,0.47265625,0.5703125
input_layernorm,6,14,0.5578512396694215,0.35809018567639256,0.5578512396694215,True,0.1498883068561554,0.11825777590274811,0.47265625,0.6875
input_layernorm,6,15,1.3971774193548387,0.5828427249789739,1.3971774193548387,True,0.1498883068561554,0.06029079854488373,0.47265625,0.779296875
input_layernorm,6,16,0.7217261904761905,0.4191875540190147,0.7217261904761905,True,0.1498883068561554,0.09195870161056519,0.47265625,0.7763671875
input_layernorm,6,17,1.3450413223140496,0.573568281938326,1.3450413223140496,True,0.1498883068561554,0.06483674794435501,0.47265625,1.0012359619140625
input_layernorm,6,18,1.7479338842975207,0.6360902255639098,1.7479338842975207,True,0.1498883068561554,0.06785038113594055,0.47265625,1.1640625
input_layernorm,6,19,2.1115702479338845,0.6786188579017264,2.1115702479338845,True,0.1498883068561554,0.19409698247909546,0.47265625,0.884765625
input_layernorm,6,20,1.8636363636363635,0.6507936507936508,1.8636363636363635,True,0.1498883068561554,0.08625888079404831,0.47265625,1.0262451171875
input_layernorm,6,21,2.282258064516129,0.6953316953316954,2.282258064516129,True,0.1498883068561554,0.07095508277416229,0.47265625,1.043212890625
input_layernorm,6,22,2.092741935483871,0.6766623207301173,2.092741935483871,True,0.1498883068561554,0.07214703410863876,0.47265625,1.044921875
input_layernorm,6,23,2.774193548387097,0.7350427350427351,2.774193548387097,True,0.1498883068561554,0.15369786322116852,0.47265625,0.98577880859375
input_layernorm,7,8,0.564910511721704,0.6390151515151515,0.6390151515151515,True,0.095920130610466,0.20054922997951508,0.46484375,0.41796875
input_layernorm,7,9,0.4822283841693975,0.8075657894736842,0.8075657894736842,True,0.095920130610466,0.22649425268173218,0.46484375,0.42578125
input_layernorm,7,10,0.4459289135366776,0.6383139067524116,0.6383139067524116,True,0.095920130610466,0.2204512506723404,0.46484375,0.3828125
input_layernorm,7,11,0.3941018766756032,0.6504424778761062,0.6504424778761062,True,0.095920130610466,0.16603894531726837,0.46484375,0.34375
input_layernorm,7,12,0.5676959619952494,0.449868073878628,0.5676959619952494,True,0.095920130610466,0.1900545060634613,0.46484375,0.5546875
input_layernorm,7,13,0.5866983372921615,0.46976902173913043,0.5866983372921615,True,0.095920130610466,0.16846242547035217,0.46484375,0.5703125
input_layernorm,7,14,0.630188679245283,0.38657407407407407,0.630188679245283,True,0.095920130610466,0.11825777590274811,0.46484375,0.6875
input_layernorm,7,15,1.4814049586776858,0.5970024979184013,1.4814049586776858,True,0.095920130610466,0.06029079854488373,0.46484375,0.779296875
input_layernorm,7,16,0.9477434679334917,0.4865853658536585,0.9477434679334917,True,0.095920130610466,0.09195870161056519,0.46484375,0.7763671875
input_layernorm,7,17,1.2394957983193278,0.5534709193245778,1.2394957983193278,True,0.095920130610466,0.06483674794435501,0.46484375,1.0012359619140625
input_layernorm,7,18,1.5630081300813008,0.6098334655035687,1.5630081300813008,True,0.095920130610466,0.06785038113594055,0.46484375,1.1640625
input_layernorm,7,19,2.19327731092437,0.6868421052631579,2.19327731092437,True,0.095920130610466,0.19409698247909546,0.46484375,0.884765625
input_layernorm,7,20,1.9201680672268908,0.6575539568345323,1.9201680672268908,True,0.095920130610466,0.08625888079404831,0.46484375,1.0262451171875
input_layernorm,7,21,2.3445378151260505,0.7010050251256281,2.3445378151260505,True,0.095920130610466,0.07095508277416229,0.46484375,1.043212890625
input_layernorm,7,22,2.1178861788617884,0.6792698826597132,2.1178861788617884,True,0.095920130610466,0.07214703410863876,0.46484375,1.044921875
input_layernorm,7,23,2.8048780487804876,0.7371794871794872,2.8048780487804876,True,0.095920130610466,0.15369786322116852,0.46484375,0.98577880859375
input_layernorm,8,9,0.2168141592920354,0.1781818181818182,0.2168141592920354,True,0.20054922997951508,0.22649425268173218,0.41796875,0.42578125
input_layernorm,8,10,0.19393939393939394,0.1813186813186813,0.19393939393939394,True,0.20054922997951508,0.2204512506723404,0.41796875,0.3828125
input_layernorm,8,11,0.39980670103092786,0.666129898013956,0.666129898013956,True,0.20054922997951508,0.16603894531726837,0.41796875,0.34375
input_layernorm,8,12,0.3672566371681416,0.2686084142394822,0.3672566371681416,True,0.20054922997951508,0.1900545060634613,0.41796875,0.5546875
input_layernorm,8,13,0.3644859813084112,0.2671232876712329,0.3644859813084112,True,0.20054922997951508,0.16846242547035217,0.41796875,0.5703125
input_layernorm,8,14,0.7616822429906542,0.4323607427055703,0.7616822429906542,True,0.20054922997951508,0.11825777590274811,0.41796875,0.6875
input_layernorm,8,15,1.677927927927928,0.6265769554247267,1.677927927927928,True,0.20054922997951508,0.06029079854488373,0.41796875,0.779296875
input_layernorm,8,16,0.9478114478114478,0.48660328435609335,0.9478114478114478,True,0.20054922997951508,0.09195870161056519,0.41796875,0.7763671875
input_layernorm,8,17,1.6518691588785046,0.6229074889867842,1.6518691588785046,True,0.20054922997951508,0.06483674794435501,0.41796875,1.0012359619140625
input_layernorm,8,18,2.107476635514019,0.6781954887218045,2.107476635514019,True,0.20054922997951508,0.06785038113594055,0.41796875,1.1640625
input_layernorm,8,19,2.5186915887850465,0.7158034528552457,2.5186915887850465,True,0.20054922997951508,0.19409698247909546,0.41796875,0.884765625
input_layernorm,8,20,2.282442748091603,0.6953488372093023,2.282442748091603,True,0.20054922997951508,0.08625888079404831,0.41796875,1.0262451171875
input_layernorm,8,21,2.6666666666666665,0.7272727272727273,2.6666666666666665,True,0.20054922997951508,0.07095508277416229,0.41796875,1.043212890625
input_layernorm,8,22,2.467289719626168,0.7115902964959568,2.467289719626168,True,0.20054922997951508,0.07214703410863876,0.41796875,1.044921875
input_layernorm,8,23,3.2162162162162162,0.7628205128205128,3.2162162162162162,True,0.20054922997951508,0.15369786322116852,0.41796875,0.98577880859375
input_layernorm,9,10,0.31333333333333335,0.3008849557522124,0.31333333333333335,True,0.22649425268173218,0.2204512506723404,0.42578125,0.3828125
input_layernorm,9,11,0.5083333333333333,0.5780998389694042,0.5780998389694042,True,0.22649425268173218,0.16603894531726837,0.42578125,0.34375
input_layernorm,9,12,0.452991452991453,0.31176470588235294,0.452991452991453,True,0.22649425268173218,0.1900545060634613,0.42578125,0.5546875
input_layernorm,9,13,0.4921259842519685,0.32981530343007914,0.4921259842519685,True,0.22649425268173218,0.16846242547035217,0.42578125,0.5703125
input_layernorm,9,14,0.8134920634920635,0.4485776805251641,0.8134920634920635,True,0.22649425268173218,0.11825777590274811,0.42578125,0.6875
input_layernorm,9,15,1.7727272727272727,0.639344262295082,1.7727272727272727,True,0.22649425268173218,0.06029079854488373,0.42578125,0.779296875
input_layernorm,9,16,1.2080152671755726,0.547104580812446,1.2080152671755726,True,0.22649425268173218,0.09195870161056519,0.42578125,0.7763671875
input_layernorm,9,17,1.7058823529411764,0.6304347826086957,1.7058823529411764,True,0.22649425268173218,0.06483674794435501,0.42578125,1.0012359619140625
input_layernorm,9,18,2.1470588235294117,0.6822429906542056,2.1470588235294117,True,0.22649425268173218,0.06785038113594055,0.42578125,1.1640625
input_layernorm,9,19,2.4873949579831933,0.7132530120481928,2.4873949579831933,True,0.22649425268173218,0.19409698247909546,0.42578125,0.884765625
input_layernorm,9,20,2.6134453781512605,0.7232558139534884,2.6134453781512605,True,0.22649425268173218,0.08625888079404831,0.42578125,1.0262451171875
input_layernorm,9,21,2.7394957983193278,0.7325842696629213,2.7394957983193278,True,0.22649425268173218,0.07095508277416229,0.42578125,1.043212890625
input_layernorm,9,22,2.403669724770642,0.706199460916442,2.403669724770642,True,0.22649425268173218,0.07214703410863876,0.42578125,1.044921875
input_layernorm,9,23,2.990990990990991,0.7494356659142212,2.990990990990991,True,0.22649425268173218,0.15369786322116852,0.42578125,0.98577880859375
input_layernorm,10,11,0.350418410041841,0.5394524959742351,0.5394524959742351,True,0.2204512506723404,0.16603894531726837,0.3828125,0.34375
input_layernorm,10,12,0.4489795918367347,0.30985915492957744,0.4489795918367347,True,0.2204512506723404,0.1900545060634613,0.3828125,0.5546875
input_layernorm,10,13,0.4897959183673469,0.3287671232876712,0.4897959183673469,True,0.2204512506723404,0.16846242547035217,0.3828125,0.5703125
input_layernorm,10,14,0.923469387755102,0.48010610079575594,0.923469387755102,True,0.2204512506723404,0.11825777590274811,0.3828125,0.6875
input_layernorm,10,15,1.8762755102040816,0.6523281596452328,1.8762755102040816,True,0.2204512506723404,0.06029079854488373,0.3828125,0.779296875
input_layernorm,10,16,1.1346863468634687,0.5315471045808124,1.1346863468634687,True,0.2204512506723404,0.09195870161056519,0.3828125,0.7763671875
input_layernorm,10,17,1.8954081632653061,0.654625550660793,1.8954081632653061,True,0.2204512506723404,0.06483674794435501,0.3828125,1.0012359619140625
input_layernorm,10,18,2.392857142857143,0.7052631578947368,2.392857142857143,True,0.2204512506723404,0.06785038113594055,0.3828125,1.1640625
input_layernorm,10,19,2.8418367346938775,0.7397078353253652,2.8418367346938775,True,0.2204512506723404,0.19409698247909546,0.3828125,0.884765625
input_layernorm,10,20,2.5357142857142856,0.7171717171717171,2.5357142857142856,True,0.2204512506723404,0.08625888079404831,0.3828125,1.0262451171875
input_layernorm,10,21,2.9285714285714284,0.7454545454545455,2.9285714285714284,True,0.2204512506723404,0.07095508277416229,0.3828125,1.043212890625
input_layernorm,10,22,2.7857142857142856,0.7358490566037735,2.7857142857142856,True,0.2204512506723404,0.07214703410863876,0.3828125,1.044921875
input_layernorm,10,23,3.3333333333333335,0.7692307692307693,3.3333333333333335,True,0.2204512506723404,0.15369786322116852,0.3828125,0.98577880859375
input_layernorm,11,12,0.7078651685393258,0.4144736842105263,0.7078651685393258,True,0.16603894531726837,0.1900545060634613,0.34375,0.5546875
input_layernorm,11,13,0.7368421052631579,0.42424242424242425,0.7368421052631579,True,0.16603894531726837,0.16846242547035217,0.34375,0.5703125
input_layernorm,11,14,1.1420454545454546,0.53315649867374,1.1420454545454546,True,0.16603894531726837,0.11825777590274811,0.34375,0.6875
input_layernorm,11,15,2.3516483516483517,0.7016393442622951,2.3516483516483517,True,0.16603894531726837,0.06029079854488373,0.34375,0.779296875
input_layernorm,11,16,1.414745145631068,0.5858776228169368,1.414745145631068,True,0.16603894531726837,0.09195870161056519,0.34375,0.7763671875
input_layernorm,11,17,2.2244318181818183,0.6898678414096916,2.2244318181818183,True,0.16603894531726837,0.06483674794435501,0.34375,1.0012359619140625
input_layernorm,11,18,2.778409090909091,0.7353383458646616,2.778409090909091,True,0.16603894531726837,0.06785038113594055,0.34375,1.1640625
input_layernorm,11,19,3.278409090909091,0.7662682602921647,3.278409090909091,True,0.16603894531726837,0.19409698247909546,0.34375,0.884765625
input_layernorm,11,20,3.056603773584906,0.7534883720930232,3.056603773584906,True,0.16603894531726837,0.08625888079404831,0.34375,1.0262451171875
input_layernorm,11,21,3.4831460674157304,0.7769423558897243,3.4831460674157304,True,0.16603894531726837,0.07095508277416229,0.34375,1.043212890625
input_layernorm,11,22,3.215909090909091,0.7628032345013477,3.215909090909091,True,0.16603894531726837,0.07214703410863876,0.34375,1.044921875
input_layernorm,11,23,4.142857142857143,0.8055555555555556,4.142857142857143,True,0.16603894531726837,0.15369786322116852,0.34375,0.98577880859375
input_layernorm,12,13,0.10185185185185185,0.1134020618556701,0.1134020618556701,True,0.1900545060634613,0.16846242547035217,0.5546875,0.5703125
input_layernorm,12,14,0.3644343302990897,0.26709554443650224,0.3644343302990897,True,0.1900545060634613,0.11825777590274811,0.5546875,0.6875
input_layernorm,12,15,1.0359589041095891,0.5088309503784693,1.0359589041095891,True,0.1900545060634613,0.06029079854488373,0.5546875,0.779296875
input_layernorm,12,16,0.6295774647887324,0.3863439930855661,0.6295774647887324,True,0.1900545060634613,0.09195870161056519,0.5546875,0.7763671875
input_layernorm,12,17,0.9982394366197183,0.4995594713656388,0.9982394366197183,True,0.1900545060634613,0.06483674794435501,0.5546875,1.0012359619140625
input_layernorm,12,18,1.341549295774648,0.5729323308270676,1.341549295774648,True,0.1900545060634613,0.06785038113594055,0.5546875,1.1640625
input_layernorm,12,19,1.6514084507042253,0.6228419654714475,1.6514084507042253,True,0.1900545060634613,0.19409698247909546,0.5546875,0.884765625
input_layernorm,12,20,1.5294117647058822,0.6046511627906976,1.5294117647058822,True,0.1900545060634613,0.08625888079404831,0.5546875,1.0262451171875
input_layernorm,12,21,1.7876712328767124,0.6412776412776413,1.7876712328767124,True,0.1900545060634613,0.07095508277416229,0.5546875,1.043212890625
input_layernorm,12,22,1.6267123287671232,0.6192959582790091,1.6267123287671232,True,0.1900545060634613,0.07214703410863876,0.5546875,1.044921875
input_layernorm,12,23,2.2054794520547945,0.688034188034188,2.2054794520547945,True,0.1900545060634613,0.15369786322116852,0.5546875,0.98577880859375
input_layernorm,13,14,0.3580163043478261,0.263631815907954,0.3580163043478261,True,0.16846242547035217,0.11825777590274811,0.5703125,0.6875
input_layernorm,13,15,1.0890410958904109,0.521311475409836,1.0890410958904109,True,0.16846242547035217,0.06029079854488373,0.5703125,0.779296875
input_layernorm,13,16,0.6413043478260869,0.39072847682119205,0.6413043478260869,True,0.16846242547035217,0.09195870161056519,0.5703125,0.7763671875
input_layernorm,13,17,0.9434931506849316,0.48546255506607927,0.9434931506849316,True,0.16846242547035217,0.06483674794435501,0.5703125,1.0012359619140625
input_layernorm,13,18,1.2773972602739727,0.5609022556390978,1.2773972602739727,True,0.16846242547035217,0.06785038113594055,0.5703125,1.1640625
input_layernorm,13,19,1.5787671232876712,0.6122177954847278,1.5787671232876712,True,0.16846242547035217,0.19409698247909546,0.5703125,0.884765625
input_layernorm,13,20,1.457142857142857,0.5930232558139535,1.457142857142857,True,0.16846242547035217,0.08625888079404831,0.5703125,1.0262451171875
input_layernorm,13,21,1.759322033898305,0.6375921375921376,1.759322033898305,True,0.16846242547035217,0.07095508277416229,0.5703125,1.043212890625
input_layernorm,13,22,1.6,0.6153846153846154,1.6,True,0.16846242547035217,0.07214703410863876,0.5703125,1.044921875
input_layernorm,13,23,2.1728813559322036,0.6848290598290598,2.1728813559322036,True,0.16846242547035217,0.15369786322116852,0.5703125,0.98577880859375
input_layernorm,14,15,0.6661931818181818,0.667953667953668,0.667953667953668,True,0.11825777590274811,0.06029079854488373,0.6875,0.779296875
input_layernorm,14,16,0.4027630180658874,0.2871212121212121,0.4027630180658874,True,0.11825777590274811,0.09195870161056519,0.6875,0.7763671875
input_layernorm,14,17,0.640807651434644,0.3905440414507772,0.640807651434644,True,0.11825777590274811,0.06483674794435501,0.6875,1.0012359619140625
input_layernorm,14,18,0.8575982996811902,0.46167048054919907,0.8575982996811902,True,0.11825777590274811,0.06785038113594055,0.6875,1.1640625
input_layernorm,14,19,1.1590909090909092,0.5774647887323944,1.1590909090909092,True,0.11825777590274811,0.19409698247909546,0.6875,0.884765625
input_layernorm,14,20,1.107843137254902,0.5255813953488372,1.107843137254902,True,0.11825777590274811,0.08625888079404831,0.6875,1.0262451171875
input_layernorm,14,21,1.2613636363636365,0.5577889447236181,1.2613636363636365,True,0.11825777590274811,0.07095508277416229,0.6875,1.043212890625
input_layernorm,14,22,1.0710227272727273,0.5171467764060357,1.0710227272727273,True,0.11825777590274811,0.07214703410863876,0.6875,1.044921875
input_layernorm,14,23,1.5056818181818181,0.6009070294784581,1.5056818181818181,True,0.11825777590274811,0.15369786322116852,0.6875,0.98577880859375
input_layernorm,15,16,0.8797573083287369,0.4993757802746567,0.8797573083287369,True,0.06029079854488373,0.09195870161056519,0.779296875,0.7763671875
input_layernorm,15,17,0.732297852900868,0.42273206751054854,0.732297852900868,True,0.06029079854488373,0.06483674794435501,0.779296875,1.0012359619140625
input_layernorm,15,18,0.8297872340425532,0.45348837209302323,0.8297872340425532,True,0.06029079854488373,0.06785038113594055,0.779296875,1.1640625
input_layernorm,15,19,1.0300751879699248,0.5074074074074074,1.0300751879699248,True,0.06029079854488373,0.19409698247909546,0.779296875,0.884765625
input_layernorm,15,20,1.0451127819548873,0.5110294117647058,1.0451127819548873,True,0.06029079854488373,0.08625888079404831,0.779296875,1.0262451171875
input_layernorm,15,21,1.150375939849624,0.534965034965035,1.150375939849624,True,0.06029079854488373,0.07095508277416229,0.779296875,1.043212890625
input_layernorm,15,22,1.005012531328321,0.50125,1.005012531328321,True,0.06029079854488373,0.07214703410863876,0.779296875,1.044921875
input_layernorm,15,23,1.100250626566416,0.5238663484486874,1.100250626566416,True,0.06029079854488373,0.15369786322116852,0.779296875,0.98577880859375
input_layernorm,16,17,0.3909313725490196,0.28105726872246695,0.3909313725490196,True,0.09195870161056519,0.06483674794435501,0.7763671875,1.0012359619140625
input_layernorm,16,18,0.6299019607843137,0.38646616541353385,0.6299019607843137,True,0.09195870161056519,0.06785038113594055,0.7763671875,1.1640625
input_layernorm,16,19,0.8952618453865336,0.7777777777777778,0.8952618453865336,True,0.09195870161056519,0.19409698247909546,0.7763671875,0.884765625
input_layernorm,16,20,0.7578027465667915,0.43110795454545453,0.7578027465667915,True,0.09195870161056519,0.08625888079404831,0.7763671875,1.0262451171875
input_layernorm,16,21,1.0324594257178528,0.507985257985258,1.0324594257178528,True,0.09195870161056519,0.07095508277416229,0.7763671875,1.043212890625
input_layernorm,16,22,0.9151061173533084,0.47783572359843546,0.9151061173533084,True,0.09195870161056519,0.07214703410863876,0.7763671875,1.044921875
input_layernorm,16,23,1.3370786516853932,0.5721153846153846,1.3370786516853932,True,0.09195870161056519,0.15369786322116852,0.7763671875,0.98577880859375
input_layernorm,17,18,0.20625,0.17098445595854922,0.20625,True,0.06483674794435501,0.06785038113594055,1.0012359619140625,1.1640625
input_layernorm,17,19,0.4493804959080726,0.6191966614501826,0.6191966614501826,True,0.06483674794435501,0.19409698247909546,1.0012359619140625,0.884765625
input_layernorm,17,20,0.35731807031888796,0.2632530120481928,0.35731807031888796,True,0.06483674794435501,0.08625888079404831,1.0012359619140625,1.0262451171875
input_layernorm,17,21,0.5803900259614804,0.367244804400978,0.5803900259614804,True,0.06483674794435501,0.07095508277416229,1.0012359619140625,1.043212890625
input_layernorm,17,22,0.4475112054729889,0.30915906127770537,0.4475112054729889,True,0.06483674794435501,0.07214703410863876,1.0012359619140625,1.044921875
input_layernorm,17,23,0.7790511605224256,0.45824255628177196,0.7790511605224256,True,0.06483674794435501,0.15369786322116852,1.0012359619140625,0.98577880859375
input_layernorm,18,19,0.40243142144638405,0.6734480959833072,0.6734480959833072,True,0.06785038113594055,0.19409698247909546,1.1640625,0.884765625
input_layernorm,18,20,0.17917304747320062,0.19578313253012047,0.19578313253012047,True,0.06785038113594055,0.08625888079404831,1.1640625,1.0262451171875
input_layernorm,18,21,0.3624161073825503,0.2660098522167488,0.3624161073825503,True,0.06785038113594055,0.07095508277416229,1.1640625,1.043212890625
input_layernorm,18,22,0.22378199834847234,0.27217125382262997,0.27217125382262997,True,0.06785038113594055,0.07214703410863876,1.1640625,1.044921875
input_layernorm,18,23,0.4929343308395677,0.5599088838268793,0.5599088838268793,True,0.06785038113594055,0.15369786322116852,1.1640625,0.98577880859375
input_layernorm,19,20,0.702803274621682,0.4127331002331002,0.702803274621682,True,0.19409698247909546,0.08625888079404831,0.884765625,1.0262451171875
input_layernorm,19,21,0.8282009724473258,0.4530141843971631,0.8282009724473258,True,0.19409698247909546,0.07095508277416229,0.884765625,1.043212890625
input_layernorm,19,22,0.6790923824959482,0.40444015444015446,0.6790923824959482,True,0.19409698247909546,0.07214703410863876,0.884765625,1.044921875
input_layernorm,19,23,0.34451054624121147,0.5234624145785877,0.5234624145785877,True,0.19409698247909546,0.15369786322116852,0.884765625,0.98577880859375
input_layernorm,20,21,0.2359208523592085,0.19088669950738915,0.2359208523592085,True,0.08625888079404831,0.07095508277416229,1.0262451171875,1.043212890625
input_layernorm,20,22,0.18092909535452323,0.2208955223880597,0.2208955223880597,True,0.08625888079404831,0.07214703410863876,1.0262451171875,1.044921875
input_layernorm,20,23,0.3733133433283358,0.5809627347949508,0.5809627347949508,True,0.08625888079404831,0.15369786322116852,1.0262451171875,0.98577880859375
input_layernorm,21,22,0.14822546972860126,0.17401960784313725,0.17401960784313725,True,0.07095508277416229,0.07214703410863876,1.043212890625,1.044921875
input_layernorm,21,23,0.3882978723404255,0.6347826086956522,0.6347826086956522,True,0.07095508277416229,0.15369786322116852,1.043212890625,0.98577880859375
input_layernorm,22,23,0.333976833976834,0.5014492753623189,0.5014492753623189,True,0.07214703410863876,0.15369786322116852,1.044921875,0.98577880859375
post_attention_layernorm,0,1,129.5,0.9923371647509579,129.5,True,0.108433797955513,0.09410307556390762,0.00390625,0.45703125
post_attention_layernorm,0,2,154.0,0.9935483870967742,154.0,True,0.108433797955513,0.10385182499885559,0.00390625,0.533203125
post_attention_layernorm,0,3,165.5,0.993993993993994,165.5,True,0.108433797955513,0.11546041816473007,0.00390625,0.48828125
post_attention_layernorm,0,4,181.5,0.9945205479452055,181.5,True,0.108433797955513,0.11874310672283173,0.00390625,0.4765625
post_attention_layernorm,0,5,194.25,0.9948783610755442,194.25,True,0.108433797955513,0.14567376673221588,0.00390625,0.421875
post_attention_layernorm,0,6,221.0,0.9954954954954955,221.0,True,0.108433797955513,0.13559450209140778,0.00390625,0.43359375
post_attention_layernorm,0,7,188.5,0.9947229551451188,188.5,True,0.108433797955513,0.17147497832775116,0.00390625,0.375
post_attention_layernorm,0,8,187.0,0.9946808510638298,187.0,True,0.108433797955513,0.18255221843719482,0.00390625,0.33984375
post_attention_layernorm,0,9,185.0,0.9946236559139785,185.0,True,0.108433797955513,0.17869991064071655,0.00390625,0.3515625
post_attention_layernorm,0,10,215.75,0.9953863898500577,215.75,True,0.108433797955513,0.16611140966415405,0.00390625,0.36328125
post_attention_layernorm,0,11,195.75,0.9949174078780177,195.75,True,0.108433797955513,0.1528426557779312,0.00390625,0.41796875
post_attention_layernorm,0,12,188.0,0.9947089947089947,188.0,True,0.108433797955513,0.12497646361589432,0.00390625,0.5234375
post_attention_layernorm,0,13,199.25,0.9950062421972534,199.25,True,0.108433797955513,0.09632012248039246,0.00390625,0.619140625
post_attention_layernorm,0,14,228.875,0.9956498096791735,228.875,True,0.108433797955513,0.053455036133527756,0.00390625,0.87646484375
post_attention_layernorm,0,15,197.25,0.9949558638083228,197.25,True,0.108433797955513,0.06801187992095947,0.00390625,0.736328125
post_attention_layernorm,0,16,189.5,0.994750656167979,189.5,True,0.108433797955513,0.08680811524391174,0.00390625,0.72265625
post_attention_layernorm,0,17,181.5,0.9945205479452055,181.5,True,0.108433797955513,0.09242258220911026,0.00390625,0.712890625
post_attention_layernorm,0,18,224.875,0.9955727725511898,224.875,True,0.108433797955513,0.09676424413919449,0.00390625,0.8095703125
post_attention_layernorm,0,19,233.5,0.9957356076759062,233.5,True,0.108433797955513,0.07613449543714523,0.00390625,0.81640625
post_attention_layernorm,0,20,209.5,0.995249406175772,209.5,True,0.108433797955513,0.07808392494916916,0.00390625,0.822265625
post_attention_layernorm,0,21,220.0,0.995475113122172,220.0,True,0.108433797955513,0.06570179015398026,0.00390625,0.86328125
post_attention_layernorm,0,22,248.5625,0.995992987728525,248.5625,True,0.108433797955513,0.05012959614396095,0.00390625,0.9713134765625
post_attention_layernorm,0,23,258.484375,0.9961462034082014,258.484375,True,0.108433797955513,0.06521295011043549,0.00390625,1.01361083984375
post_attention_layernorm,1,2,0.42512908777969016,0.29830917874396135,0.42512908777969016,True,0.09410307556390762,0.10385182499885559,0.45703125,0.533203125
post_attention_layernorm,1,3,0.27586206896551724,0.26858457997698504,0.27586206896551724,True,0.09410307556390762,0.11546041816473007,0.45703125,0.48828125
post_attention_layernorm,1,4,0.39846743295019155,0.28493150684931506,0.39846743295019155,True,0.09410307556390762,0.11874310672283173,0.45703125,0.4765625
post_attention_layernorm,1,5,0.49616858237547895,0.33162612035851474,0.49616858237547895,True,0.09410307556390762,0.14567376673221588,0.45703125,0.421875
post_attention_layernorm,1,6,0.7011494252873564,0.41216216216216217,0.7011494252873564,True,0.09410307556390762,0.13559450209140778,0.45703125,0.43359375
post_attention_layernorm,1,7,0.4521072796934866,0.45161290322580644,0.4521072796934866,True,0.09410307556390762,0.17147497832775116,0.45703125,0.375
post_attention_layernorm,1,8,0.5296052631578947,0.594488188976378,0.594488188976378,True,0.09410307556390762,0.18255221843719482,0.45703125,0.33984375
post_attention_layernorm,1,9,0.42528735632183906,0.5133079847908745,0.5133079847908745,True,0.09410307556390762,0.17869991064071655,0.45703125,0.3515625
post_attention_layernorm,1,10,0.6609195402298851,0.5073260073260073,0.6609195402298851,True,0.09410307556390762,0.16611140966415405,0.45703125,0.36328125
post_attention_layernorm,1,11,0.5076628352490421,0.3785807291666667,0.5076628352490421,True,0.09410307556390762,0.1528426557779312,0.45703125,0.41796875
post_attention_layernorm,1,12,0.7397203947368421,0.4274732620320856,0.7397203947368421,True,0.09410307556390762,0.12497646361589432,0.45703125,0.5234375
post_attention_layernorm,1,13,0.8725328947368421,0.46596398770311814,0.8725328947368421,True,0.09410307556390762,0.09632012248039246,0.45703125,0.619140625
post_attention_layernorm,1,14,1.08739406779661,0.5209337731540218,1.08739406779661,True,0.09410307556390762,0.053455036133527756,0.45703125,0.87646484375
post_attention_layernorm,1,15,1.7415254237288136,0.6352395672333848,1.7415254237288136,True,0.09410307556390762,0.06801187992095947,0.45703125,0.736328125
post_attention_layernorm,1,16,0.9901315789473685,0.4975206611570248,0.9901315789473685,True,0.09410307556390762,0.08680811524391174,0.45703125,0.72265625
post_attention_layernorm,1,17,1.009733606557377,0.5024216161101198,1.009733606557377,True,0.09410307556390762,0.09242258220911026,0.45703125,0.712890625
post_attention_layernorm,1,18,1.194078947368421,0.5442278860569715,1.194078947368421,True,0.09410307556390762,0.09676424413919449,0.45703125,0.8095703125
post_attention_layernorm,1,19,1.5277777777777777,0.6043956043956044,1.5277777777777777,True,0.09410307556390762,0.07613449543714523,0.45703125,0.81640625
post_attention_layernorm,1,20,1.3644067796610169,0.5770609318996416,1.3644067796610169,True,0.09410307556390762,0.07808392494916916,0.45703125,0.822265625
post_attention_layernorm,1,21,1.7166666666666666,0.6319018404907976,1.7166666666666666,True,0.09410307556390762,0.06570179015398026,0.45703125,0.86328125
post_attention_layernorm,1,22,2.377049180327869,0.7038834951456311,2.377049180327869,True,0.09410307556390762,0.05012959614396095,0.45703125,0.9713134765625
post_attention_layernorm,1,23,2.6324786324786325,0.7247058823529412,2.6324786324786325,True,0.09410307556390762,0.06521295011043549,0.45703125,1.01361083984375
post_attention_layernorm,2,3,0.3058252427184466,0.4405594405594406,0.4405594405594406,True,0.10385182499885559,0.11546041816473007,0.533203125,0.48828125
post_attention_layernorm,2,4,0.3094660194174757,0.44815465729349735,0.44815465729349735,True,0.10385182499885559,0.11874310672283173,0.533203125,0.4765625
post_attention_layernorm,2,5,0.3109903381642512,0.45135845749342685,0.45135845749342685,True,0.10385182499885559,0.14567376673221588,0.533203125,0.421875
post_attention_layernorm,2,6,0.432258064516129,0.5354659248956884,0.5354659248956884,True,0.10385182499885559,0.13559450209140778,0.533203125,0.43359375
post_attention_layernorm,2,7,0.3689613526570048,0.584688995215311,0.584688995215311,True,0.10385182499885559,0.17147497832775116,0.533203125,0.375
post_attention_layernorm,2,8,0.42462600690448793,0.738,0.738,True,0.10385182499885559,0.18255221843719482,0.533203125,0.33984375
post_attention_layernorm,2,9,0.42002301495972383,0.7242063492063492,0.7242063492063492,True,0.10385182499885559,0.17869991064071655,0.533203125,0.3515625
post_attention_layernorm,2,10,0.414368739953011,0.7075591216216216,0.7075591216216216,True,0.10385182499885559,0.16611140966415405,0.533203125,0.36328125
post_attention_layernorm,2,11,0.37333937198067635,0.5957600578173934,0.5957600578173934,True,0.10385182499885559,0.1528426557779312,0.533203125,0.41796875
post_attention_layernorm,2,12,0.46098066298342544,0.5644780349551252,0.5644780349551252,True,0.10385182499885559,0.12497646361589432,0.533203125,0.5234375
post_attention_layernorm,2,13,0.5725138121546961,0.52416014726185,0.5725138121546961,True,0.10385182499885559,0.09632012248039246,0.533203125,0.619140625
post_attention_layernorm,2,14,0.8044871794871795,0.4500659920809503,0.8044871794871795,True,0.10385182499885559,0.053455036133527756,0.533203125,0.87646484375
post_attention_layernorm,2,15,1.36996336996337,0.5780525502318392,1.36996336996337,True,0.10385182499885559,0.06801187992095947,0.533203125,0.736328125
post_attention_layernorm,2,16,0.6712707182320442,0.47909888895709285,0.6712707182320442,True,0.10385182499885559,0.08680811524391174,0.533203125,0.72265625
post_attention_layernorm,2,17,0.7513489208633094,0.4290115532734275,0.7513489208633094,True,0.10385182499885559,0.09242258220911026,0.533203125,0.712890625
post_attention_layernorm,2,18,0.8425414364640884,0.4572713643178411,0.8425414364640884,True,0.10385182499885559,0.09676424413919449,0.533203125,0.8095703125
post_attention_layernorm,2,19,1.1666666666666667,0.5384615384615384,1.1666666666666667,True,0.10385182499885559,0.07613449543714523,0.533203125,0.81640625
post_attention_layernorm,2,20,1.043956043956044,0.510752688172043,1.043956043956044,True,0.10385182499885559,0.07808392494916916,0.533203125,0.822265625
post_attention_layernorm,2,21,1.381294964028777,0.5800604229607251,1.381294964028777,True,0.10385182499885559,0.06570179015398026,0.533203125,0.86328125
post_attention_layernorm,2,22,1.9640287769784173,0.662621359223301,1.9640287769784173,True,0.10385182499885559,0.05012959614396095,0.533203125,0.9713134765625
post_attention_layernorm,2,23,2.1135531135531136,0.6788235294117647,2.1135531135531136,True,0.10385182499885559,0.06521295011043549,0.533203125,1.01361083984375
post_attention_layernorm,3,4,0.1832089552238806,0.22430333485609869,0.22430333485609869,True,0.11546041816473007,0.11874310672283173,0.48828125,0.4765625
post_attention_layernorm,3,5,0.1939149560117302,0.2405638926784902,0.2405638926784902,True,0.11546041816473007,0.14567376673221588,0.48828125,0.421875
post_attention_layernorm,3,6,0.3333333333333333,0.25,0.3333333333333333,True,0.11546041816473007,0.13559450209140778,0.48828125,0.43359375
post_attention_layernorm,3,7,0.2893772893772894,0.4072164948453608,0.4072164948453608,True,0.11546041816473007,0.17147497832775116,0.48828125,0.375
post_attention_layernorm,3,8,0.34798534798534797,0.5337078651685393,0.5337078651685393,True,0.11546041816473007,0.18255221843719482,0.48828125,0.33984375
post_attention_layernorm,3,9,0.34065934065934067,0.5166666666666667,0.5166666666666667,True,0.11546041816473007,0.17869991064071655,0.48828125,0.3515625
post_attention_layernorm,3,10,0.3333333333333333,0.5,0.5,True,0.11546041816473007,0.16611140966415405,0.48828125,0.36328125
post_attention_layernorm,3,11,0.2514124293785311,0.28756009615384615,0.28756009615384615,True,0.11546041816473007,0.1528426557779312,0.48828125,0.41796875
post_attention_layernorm,3,12,0.43326558265582654,0.3787001287001287,0.43326558265582654,True,0.11546041816473007,0.12497646361589432,0.48828125,0.5234375
post_attention_layernorm,3,13,0.5426829268292683,0.35177865612648224,0.5426829268292683,True,0.11546041816473007,0.09632012248039246,0.48828125,0.619140625
post_attention_layernorm,3,14,0.902027027027027,0.47424511545293074,0.902027027027027,True,0.11546041816473007,0.053455036133527756,0.48828125,0.87646484375
post_attention_layernorm,3,15,1.56,0.609375,1.56,True,0.11546041816473007,0.06801187992095947,0.48828125,0.736328125
post_attention_layernorm,3,16,0.8092643051771117,0.44728915662650603,0.8092643051771117,True,0.11546041816473007,0.08680811524391174,0.48828125,0.72265625
post_attention_layernorm,3,17,0.8933397683397684,0.47183278103492227,0.8933397683397684,True,0.11546041816473007,0.09242258220911026,0.48828125,0.712890625
post_attention_layernorm,3,18,0.9215,0.47957325006505336,0.9215,True,0.11546041816473007,0.09676424413919449,0.48828125,0.8095703125
post_attention_layernorm,3,19,1.366,0.577345731191885,1.366,True,0.11546041816473007,0.07613449543714523,0.48828125,0.81640625
post_attention_layernorm,3,20,1.169,0.5389580451821115,1.169,True,0.11546041816473007,0.07808392494916916,0.48828125,0.822265625
post_attention_layernorm,3,21,1.5396825396825398,0.60625,1.5396825396825398,True,0.11546041816473007,0.06570179015398026,0.48828125,0.86328125
post_attention_layernorm,3,22,2.2698412698412698,0.6941747572815534,2.2698412698412698,True,0.11546041816473007,0.05012959614396095,0.48828125,0.9713134765625
post_attention_layernorm,3,23,2.4,0.7058823529411765,2.4,True,0.11546041816473007,0.06521295011043549,0.48828125,1.01361083984375
post_attention_layernorm,4,5,0.16475095785440613,0.19724770642201836,0.19724770642201836,True,0.11874310672283173,0.14567376673221588,0.4765625,0.421875
post_attention_layernorm,4,6,0.2548342541436464,0.2030820033021464,0.2548342541436464,True,0.11874310672283173,0.13559450209140778,0.4765625,0.43359375
post_attention_layernorm,4,7,0.2785923753665689,0.3861788617886179,0.3861788617886179,True,0.11874310672283173,0.17147497832775116,0.4765625,0.375
post_attention_layernorm,4,8,0.3253731343283582,0.4823008849557522,0.4823008849557522,True,0.11874310672283173,0.18255221843719482,0.4765625,0.33984375
post_attention_layernorm,4,9,0.3128491620111732,0.45528455284552843,0.45528455284552843,True,0.11874310672283173,0.17869991064071655,0.4765625,0.3515625
post_attention_layernorm,4,10,0.30434782608695654,0.4375,0.4375,True,0.11874310672283173,0.16611140966415405,0.4765625,0.36328125
post_attention_layernorm,4,11,0.2160188457008245,0.27554086538461536,0.27554086538461536,True,0.11874310672283173,0.1528426557779312,0.4765625,0.41796875
post_attention_layernorm,4,12,0.3737012987012987,0.3658301158301158,0.3737012987012987,True,0.11874310672283173,0.12497646361589432,0.4765625,0.5234375
post_attention_layernorm,4,13,0.4785714285714286,0.32367149758454106,0.4785714285714286,True,0.11874310672283173,0.09632012248039246,0.4765625,0.619140625
post_attention_layernorm,4,14,0.9615384615384616,0.49019607843137253,0.9615384615384616,True,0.11874310672283173,0.053455036133527756,0.4765625,0.87646484375
post_attention_layernorm,4,15,1.6598360655737705,0.6240369799691834,1.6598360655737705,True,0.11874310672283173,0.06801187992095947,0.4765625,0.736328125
post_attention_layernorm,4,16,0.7994579945799458,0.4442771084337349,0.7994579945799458,True,0.11874310672283173,0.08680811524391174,0.4765625,0.72265625
post_attention_layernorm,4,17,0.9569672131147541,0.48900523560209425,0.9569672131147541,True,0.11874310672283173,0.09242258220911026,0.4765625,0.712890625
post_attention_layernorm,4,18,1.0363729508196722,0.5089308176100629,1.0363729508196722,True,0.11874310672283173,0.09676424413919449,0.4765625,0.8095703125
post_attention_layernorm,4,19,1.4246031746031746,0.5875613747954174,1.4246031746031746,True,0.11874310672283173,0.07613449543714523,0.4765625,0.81640625
post_attention_layernorm,4,20,1.2930327868852458,0.5638963360142984,1.2930327868852458,True,0.11874310672283173,0.07808392494916916,0.4765625,0.822265625
post_attention_layernorm,4,21,1.5806451612903225,0.6125,1.5806451612903225,True,0.11874310672283173,0.06570179015398026,0.4765625,0.86328125
post_attention_layernorm,4,22,2.3225806451612905,0.6990291262135923,2.3225806451612905,True,0.11874310672283173,0.05012959614396095,0.4765625,0.9713134765625
post_attention_layernorm,4,23,2.373015873015873,0.7035294117647058,2.373015873015873,True,0.11874310672283173,0.06521295011043549,0.4765625,1.01361083984375
post_attention_layernorm,5,6,0.32515337423312884,0.24537037037037038,0.32515337423312884,True,0.14567376673221588,0.13559450209140778,0.421875,0.43359375
post_attention_layernorm,5,7,0.1608832807570978,0.19172932330827067,0.19172932330827067,True,0.14567376673221588,0.17147497832775116,0.421875,0.375
post_attention_layernorm,5,8,0.2413793103448276,0.3181818181818182,0.3181818181818182,True,0.14567376673221588,0.18255221843719482,0.421875,0.33984375
post_attention_layernorm,5,9,0.24968314321926488,0.3327702702702703,0.3327702702702703,True,0.14567376673221588,0.17869991064071655,0.421875,0.3515625
post_attention_layernorm,5,10,0.2541436464088398,0.30067567567567566,0.30067567567567566,True,0.14567376673221588,0.16611140966415405,0.421875,0.36328125
post_attention_layernorm,5,11,0.214,0.23786808894230768,0.23786808894230768,True,0.14567376673221588,0.1528426557779312,0.421875,0.41796875
post_attention_layernorm,5,12,0.4103333333333333,0.3254906692406692,0.4103333333333333,True,0.14567376673221588,0.12497646361589432,0.421875,0.5234375
post_attention_layernorm,5,13,0.555045871559633,0.35693215339233036,0.555045871559633,True,0.14567376673221588,0.09632012248039246,0.421875,0.619140625
post_attention_layernorm,5,14,1.275092936802974,0.5604575163398693,1.275092936802974,True,0.14567376673221588,0.053455036133527756,0.421875,0.87646484375
post_attention_layernorm,5,15,1.962962962962963,0.6625,1.962962962962963,True,0.14567376673221588,0.06801187992095947,0.421875,0.736328125
post_attention_layernorm,5,16,1.063197026022305,0.5153153153153153,1.063197026022305,True,0.14567376673221588,0.08680811524391174,0.421875,0.72265625
post_attention_layernorm,5,17,1.2333715596330275,0.5522464698331194,1.2333715596330275,True,0.14567376673221588,0.09242258220911026,0.421875,0.712890625
post_attention_layernorm,5,18,1.2573327850877194,0.5569993017395792,1.2573327850877194,True,0.14567376673221588,0.09676424413919449,0.421875,0.8095703125
post_attention_layernorm,5,19,1.738532110091743,0.6348408710217756,1.738532110091743,True,0.14567376673221588,0.07613449543714523,0.421875,0.81640625
post_attention_layernorm,5,20,1.5136363636363637,0.6021699819168174,1.5136363636363637,True,0.14567376673221588,0.07808392494916916,0.421875,0.822265625
post_attention_layernorm,5,21,2.036697247706422,0.6706948640483383,2.036697247706422,True,0.14567376673221588,0.06570179015398026,0.421875,0.86328125
post_attention_layernorm,5,22,2.779816513761468,0.7354368932038835,2.779816513761468,True,0.14567376673221588,0.05012959614396095,0.421875,0.9713134765625
post_attention_layernorm,5,23,2.935185185185185,0.7458823529411764,2.935185185185185,True,0.14567376673221588,0.06521295011043549,0.421875,1.01361083984375
post_attention_layernorm,6,7,0.3519163763066202,0.543010752688172,0.543010752688172,True,0.13559450209140778,0.17147497832775116,0.43359375,0.375
post_attention_layernorm,6,8,0.4122905027932961,0.7015209125475285,0.7015209125475285,True,0.13559450209140778,0.18255221843719482,0.43359375,0.33984375
post_attention_layernorm,6,9,0.35195530726256985,0.5431034482758621,0.5431034482758621,True,0.13559450209140778,0.17869991064071655,0.43359375,0.3515625
post_attention_layernorm,6,10,0.3743016759776536,0.5982142857142857,0.5982142857142857,True,0.13559450209140778,0.16611140966415405,0.43359375,0.36328125
post_attention_layernorm,6,11,0.2961672473867596,0.4207920792079208,0.4207920792079208,True,0.13559450209140778,0.1528426557779312,0.43359375,0.41796875
post_attention_layernorm,6,12,0.3181818181818182,0.29812834224598933,0.3181818181818182,True,0.13559450209140778,0.12497646361589432,0.43359375,0.5234375
post_attention_layernorm,6,13,0.5526315789473685,0.3559322033898305,0.5526315789473685,True,0.13559450209140778,0.09632012248039246,0.43359375,0.619140625
post_attention_layernorm,6,14,1.1714801444043321,0.5394846217788861,1.1714801444043321,True,0.13559450209140778,0.053455036133527756,0.43359375,0.87646484375
post_attention_layernorm,6,15,1.927927927927928,0.6584615384615384,1.927927927927928,True,0.13559450209140778,0.06801187992095947,0.43359375,0.736328125
post_attention_layernorm,6,16,0.9859913793103449,0.4964731416169289,0.9859913793103449,True,0.13559450209140778,0.08680811524391174,0.43359375,0.72265625
post_attention_layernorm,6,17,1.2088963963963963,0.5472852408870762,1.2088963963963963,True,0.13559450209140778,0.09242258220911026,0.43359375,0.712890625
post_attention_layernorm,6,18,1.2401463963963963,0.5536006032424281,1.2401463963963963,True,0.13559450209140778,0.09676424413919449,0.43359375,0.8095703125
post_attention_layernorm,6,19,1.6644144144144144,0.6246830092983939,1.6644144144144144,True,0.13559450209140778,0.07613449543714523,0.43359375,0.81640625
post_attention_layernorm,6,20,1.5202702702702702,0.6032171581769437,1.5202702702702702,True,0.13559450209140778,0.07808392494916916,0.43359375,0.822265625
post_attention_layernorm,6,21,1.8782608695652174,0.6525679758308157,1.8782608695652174,True,0.13559450209140778,0.06570179015398026,0.43359375,0.86328125
post_attention_layernorm,6,22,2.6140350877192984,0.7233009708737864,2.6140350877192984,True,0.13559450209140778,0.05012959614396095,0.43359375,0.9713134765625
post_attention_layernorm,6,23,2.828828828828829,0.7388235294117647,2.828828828828829,True,0.13559450209140778,0.06521295011043549,0.43359375,1.01361083984375
post_attention_layernorm,7,8,0.19383825417201542,0.16236559139784945,0.19383825417201542,True,0.17147497832775116,0.18255221843719482,0.375,0.33984375
post_attention_layernorm,7,9,0.12010443864229765,0.13649851632047477,0.13649851632047477,True,0.17147497832775116,0.17869991064071655,0.375,0.3515625
post_attention_layernorm,7,10,0.15468549422336328,0.1339633129516398,0.15468549422336328,True,0.17147497832775116,0.16611140966415405,0.375,0.36328125
post_attention_layernorm,7,11,0.23232323232323232,0.1885245901639344,0.23232323232323232,True,0.17147497832775116,0.1528426557779312,0.375,0.41796875
post_attention_layernorm,7,12,0.5047169811320755,0.335423197492163,0.5047169811320755,True,0.17147497832775116,0.12497646361589432,0.375,0.5234375
post_attention_layernorm,7,13,0.7424242424242424,0.4260869565217391,0.7424242424242424,True,0.17147497832775116,0.09632012248039246,0.375,0.619140625
post_attention_layernorm,7,14,1.5932203389830508,0.6143790849673203,1.5932203389830508,True,0.17147497832775116,0.053455036133527756,0.375,0.87646484375
post_attention_layernorm,7,15,2.3645833333333335,0.7027863777089783,2.3645833333333335,True,0.17147497832775116,0.06801187992095947,0.375,0.736328125
post_attention_layernorm,7,16,1.3799283154121864,0.5798192771084337,1.3799283154121864,True,0.17147497832775116,0.08680811524391174,0.375,0.72265625
post_attention_layernorm,7,17,1.5019132653061225,0.6003058883507519,1.5019132653061225,True,0.17147497832775116,0.09242258220911026,0.375,0.712890625
post_attention_layernorm,7,18,1.579954954954955,0.6123963334788302,1.579954954954955,True,0.17147497832775116,0.09676424413919449,0.375,0.8095703125
post_attention_layernorm,7,19,2.085858585858586,0.6759410801963993,2.085858585858586,True,0.17147497832775116,0.07613449543714523,0.375,0.81640625
post_attention_layernorm,7,20,1.884020618556701,0.6532618409294012,1.884020618556701,True,0.17147497832775116,0.07808392494916916,0.375,0.822265625
post_attention_layernorm,7,21,2.377551020408163,0.7039274924471299,2.377551020408163,True,0.17147497832775116,0.06570179015398026,0.375,0.86328125
post_attention_layernorm,7,22,3.2916666666666665,0.7669902912621359,3.2916666666666665,True,0.17147497832775116,0.05012959614396095,0.375,0.9713134765625
post_attention_layernorm,7,23,3.381443298969072,0.7717647058823529,3.381443298969072,True,0.17147497832775116,0.06521295011043549,0.375,1.01361083984375
post_attention_layernorm,8,9,0.1510791366906475,0.13125,0.1510791366906475,True,0.18255221843719482,0.17869991064071655,0.33984375,0.3515625
post_attention_layernorm,8,10,0.17464424320827943,0.1486784140969163,0.17464424320827943,True,0.18255221843719482,0.16611140966415405,0.33984375,0.36328125
post_attention_layernorm,8,11,0.28421052631578947,0.22131147540983606,0.28421052631578947,True,0.18255221843719482,0.1528426557779312,0.33984375,0.41796875
post_attention_layernorm,8,12,0.6377551020408163,0.3894080996884735,0.6377551020408163,True,0.18255221843719482,0.12497646361589432,0.33984375,0.5234375
post_attention_layernorm,8,13,0.8956043956043956,0.47246376811594204,0.8956043956043956,True,0.18255221843719482,0.09632012248039246,0.33984375,0.619140625
post_attention_layernorm,8,14,1.7567567567567568,0.6372549019607843,1.7567567567567568,True,0.18255221843719482,0.053455036133527756,0.33984375,0.87646484375
post_attention_layernorm,8,15,2.7011494252873565,0.7298136645962733,2.7011494252873565,True,0.18255221843719482,0.06801187992095947,0.33984375,0.736328125
post_attention_layernorm,8,16,1.6141732283464567,0.6174698795180723,1.6141732283464567,True,0.18255221843719482,0.08680811524391174,0.33984375,0.72265625
post_attention_layernorm,8,17,1.7862215909090908,0.6410910017843487,1.7862215909090908,True,0.18255221843719482,0.09242258220911026,0.33984375,0.712890625
post_attention_layernorm,8,18,1.8256392045454546,0.6460977755435465,1.8256392045454546,True,0.18255221843719482,0.09676424413919449,0.33984375,0.8095703125
post_attention_layernorm,8,19,2.357142857142857,0.7021276595744681,2.357142857142857,True,0.18255221843719482,0.07613449543714523,0.33984375,0.81640625
post_attention_layernorm,8,20,2.143258426966292,0.6818588025022342,2.143258426966292,True,0.18255221843719482,0.07808392494916916,0.33984375,0.822265625
post_attention_layernorm,8,21,2.67816091954023,0.728125,2.67816091954023,True,0.18255221843719482,0.06570179015398026,0.33984375,0.86328125
post_attention_layernorm,8,22,3.735632183908046,0.7888349514563107,3.735632183908046,True,0.18255221843719482,0.05012959614396095,0.33984375,0.9713134765625
post_attention_layernorm,8,23,3.7222222222222223,0.788235294117647,3.7222222222222223,True,0.18255221843719482,0.06521295011043549,0.33984375,1.01361083984375
post_attention_layernorm,9,10,0.16532258064516128,0.14186851211072665,0.16532258064516128,True,0.17869991064071655,0.16611140966415405,0.3515625,0.36328125
post_attention_layernorm,9,11,0.2708333333333333,0.21311475409836064,0.2708333333333333,True,0.17869991064071655,0.1528426557779312,0.3515625,0.41796875
post_attention_layernorm,9,12,0.5735294117647058,0.3644859813084112,0.5735294117647058,True,0.17869991064071655,0.12497646361589432,0.3515625,0.5234375
post_attention_layernorm,9,13,0.8166666666666667,0.44954128440366975,0.8166666666666667,True,0.17869991064071655,0.09632012248039246,0.3515625,0.619140625
post_attention_layernorm,9,14,1.6645299145299146,0.6246992782678428,1.6645299145299146,True,0.17869991064071655,0.053455036133527756,0.3515625,0.87646484375
post_attention_layernorm,9,15,2.6055555555555556,0.7226502311248074,2.6055555555555556,True,0.17869991064071655,0.06801187992095947,0.3515625,0.736328125
post_attention_layernorm,9,16,1.4316239316239316,0.5887521968365553,1.4316239316239316,True,0.17869991064071655,0.08680811524391174,0.3515625,0.72265625
post_attention_layernorm,9,17,1.6527777777777777,0.6230366492146597,1.6527777777777777,True,0.17869991064071655,0.09242258220911026,0.3515625,0.712890625
post_attention_layernorm,9,18,1.7604166666666667,0.6377358490566037,1.7604166666666667,True,0.17869991064071655,0.09676424413919449,0.3515625,0.8095703125
post_attention_layernorm,9,19,2.1822916666666665,0.6857610474631751,2.1822916666666665,True,0.17869991064071655,0.07613449543714523,0.3515625,0.81640625
post_attention_layernorm,9,20,2.1083333333333334,0.67828418230563,2.1083333333333334,True,0.17869991064071655,0.07808392494916916,0.3515625,0.822265625
post_attention_layernorm,9,21,2.4916666666666667,0.7136038186157518,2.4916666666666667,True,0.17869991064071655,0.06570179015398026,0.3515625,0.86328125
post_attention_layernorm,9,22,3.382978723404255,0.7718446601941747,3.382978723404255,True,0.17869991064071655,0.05012959614396095,0.3515625,0.9713134765625
post_attention_layernorm,9,23,3.473684210526316,0.7764705882352941,3.473684210526316,True,0.17869991064071655,0.06521295011043549,0.3515625,1.01361083984375
post_attention_layernorm,10,11,0.23232323232323232,0.1885245901639344,0.23232323232323232,True,0.16611140966415405,0.1528426557779312,0.36328125,0.41796875
post_attention_layernorm,10,12,0.510752688172043,0.33807829181494664,0.510752688172043,True,0.16611140966415405,0.12497646361589432,0.36328125,0.5234375
post_attention_layernorm,10,13,0.796875,0.4434782608695652,0.796875,True,0.16611140966415405,0.09632012248039246,0.36328125,0.619140625
post_attention_layernorm,10,14,1.7079646017699115,0.630718954248366,1.7079646017699115,True,0.16611140966415405,0.053455036133527756,0.36328125,0.87646484375
post_attention_layernorm,10,15,2.4946236559139785,0.7138461538461538,2.4946236559139785,True,0.16611140966415405,0.06801187992095947,0.36328125,0.736328125
post_attention_layernorm,10,16,1.4557522123893805,0.5927927927927928,1.4557522123893805,True,0.16611140966415405,0.08680811524391174,0.36328125,0.72265625
post_attention_layernorm,10,17,1.6364247311827957,0.6206984450675503,1.6364247311827957,True,0.16611140966415405,0.09242258220911026,0.36328125,0.712890625
post_attention_layernorm,10,18,1.67372311827957,0.6259896946085208,1.67372311827957,True,0.16611140966415405,0.09676424413919449,0.36328125,0.8095703125
post_attention_layernorm,10,19,2.146276595744681,0.6821639898562976,2.146276595744681,True,0.16611140966415405,0.07613449543714523,0.36328125,0.81640625
post_attention_layernorm,10,20,1.976063829787234,0.6639857015192135,1.976063829787234,True,0.16611140966415405,0.07808392494916916,0.36328125,0.822265625
post_attention_layernorm,10,21,2.4123711340206184,0.7069486404833837,2.4123711340206184,True,0.16611140966415405,0.06570179015398026,0.36328125,0.86328125
post_attention_layernorm,10,22,3.247422680412371,0.7645631067961165,3.247422680412371,True,0.16611140966415405,0.05012959614396095,0.36328125,0.9713134765625
post_attention_layernorm,10,23,3.521276595744681,0.7788235294117647,3.521276595744681,True,0.16611140966415405,0.06521295011043549,0.36328125,1.01361083984375
post_attention_layernorm,11,12,0.3375,0.2523364485981308,0.3375,True,0.1528426557779312,0.12497646361589432,0.41796875,0.5234375
post_attention_layernorm,11,13,0.5131578947368421,0.3391304347826087,0.5131578947368421,True,0.1528426557779312,0.09632012248039246,0.41796875,0.619140625
post_attention_layernorm,11,14,1.2612781954887218,0.5577722360764755,1.2612781954887218,True,0.1528426557779312,0.053455036133527756,0.41796875,0.87646484375
post_attention_layernorm,11,15,1.9906542056074767,0.665625,1.9906542056074767,True,0.1528426557779312,0.06801187992095947,0.41796875,0.736328125
post_attention_layernorm,11,16,1.1914191419141915,0.5436746987951807,1.1914191419141915,True,0.1528426557779312,0.08680811524391174,0.41796875,0.72265625
post_attention_layernorm,11,17,1.249426605504587,0.5554422635737956,1.249426605504587,True,0.1528426557779312,0.09242258220911026,0.41796875,0.712890625
post_attention_layernorm,11,18,1.28125,0.5616438356164384,1.28125,True,0.1528426557779312,0.09676424413919449,0.41796875,0.8095703125
post_attention_layernorm,11,19,1.764018691588785,0.6382079459002535,1.764018691588785,True,0.1528426557779312,0.07613449543714523,0.41796875,0.81640625
post_attention_layernorm,11,20,1.5344036697247707,0.6054298642533936,1.5344036697247707,True,0.1528426557779312,0.07808392494916916,0.41796875,0.822265625
post_attention_layernorm,11,21,1.9369158878504673,0.6595067621320605,1.9369158878504673,True,0.1528426557779312,0.06570179015398026,0.41796875,0.86328125
post_attention_layernorm,11,22,2.7454545454545456,0.7330097087378641,2.7454545454545456,True,0.1528426557779312,0.05012959614396095,0.41796875,0.9713134765625
post_attention_layernorm,11,23,2.97196261682243,0.7482352941176471,2.97196261682243,True,0.1528426557779312,0.06521295011043549,0.41796875,1.01361083984375
post_attention_layernorm,12,13,0.26492537313432835,0.20943952802359883,0.26492537313432835,True,0.12497646361589432,0.09632012248039246,0.5234375,0.619140625
post_attention_layernorm,12,14,0.9213836477987422,0.4795417348608838,0.9213836477987422,True,0.12497646361589432,0.053455036133527756,0.5234375,0.87646484375
post_attention_layernorm,12,15,1.3861940298507462,0.5809225957779516,1.3861940298507462,True,0.12497646361589432,0.06801187992095947,0.5234375,0.736328125
post_attention_layernorm,12,16,0.9358600583090378,0.48343373493975905,0.9358600583090378,True,0.12497646361589432,0.08680811524391174,0.5234375,0.72265625
post_attention_layernorm,12,17,0.9096209912536443,0.4763358778625954,0.9096209912536443,True,0.12497646361589432,0.09242258220911026,0.5234375,0.712890625
post_attention_layernorm,12,18,0.9358600583090378,0.48343373493975905,0.9358600583090378,True,0.12497646361589432,0.09676424413919449,0.5234375,0.8095703125
post_attention_layernorm,12,19,1.2276119402985075,0.5510887772194305,1.2276119402985075,True,0.12497646361589432,0.07613449543714523,0.5234375,0.81640625
post_attention_layernorm,12,20,1.056985294117647,0.5138516532618409,1.056985294117647,True,0.12497646361589432,0.07808392494916916,0.5234375,0.822265625
post_attention_layernorm,12,21,1.4701492537313432,0.595166163141994,1.4701492537313432,True,0.12497646361589432,0.06570179015398026,0.5234375,0.86328125
post_attention_layernorm,12,22,2.0294117647058822,0.6699029126213593,2.0294117647058822,True,0.12497646361589432,0.05012959614396095,0.5234375,0.9713134765625
post_attention_layernorm,12,23,2.1481481481481484,0.6823529411764706,2.1481481481481484,True,0.12497646361589432,0.06521295011043549,0.5234375,1.01361083984375
post_attention_layernorm,13,14,0.6276595744680851,0.38562091503267976,0.6276595744680851,True,0.09632012248039246,0.053455036133527756,0.619140625,0.87646484375
post_attention_layernorm,13,15,1.0504731861198737,0.5123076923076924,1.0504731861198737,True,0.09632012248039246,0.06801187992095947,0.619140625,0.736328125
post_attention_layernorm,13,16,0.5697399527186762,0.3629518072289157,0.5697399527186762,True,0.09632012248039246,0.08680811524391174,0.619140625,0.72265625
post_attention_layernorm,13,17,0.5619930475086906,0.35979228486646886,0.5619930475086906,True,0.09632012248039246,0.09242258220911026,0.619140625,0.712890625
post_attention_layernorm,13,18,0.657010428736964,0.3965034965034965,0.657010428736964,True,0.09632012248039246,0.09676424413919449,0.619140625,0.8095703125
post_attention_layernorm,13,19,0.8659305993690851,0.4640743871513102,0.8659305993690851,True,0.09632012248039246,0.07613449543714523,0.619140625,0.81640625
post_attention_layernorm,13,20,0.7811594202898551,0.4385679414157852,0.7811594202898551,True,0.09632012248039246,0.07808392494916916,0.619140625,0.822265625
post_attention_layernorm,13,21,0.9826498422712934,0.49562450278440734,0.9826498422712934,True,0.09632012248039246,0.06570179015398026,0.619140625,0.86328125
post_attention_layernorm,13,22,1.5432098765432098,0.6067961165048543,1.5432098765432098,True,0.09632012248039246,0.05012959614396095,0.619140625,0.9713134765625
post_attention_layernorm,13,23,1.6813880126182965,0.6270588235294118,1.6813880126182965,True,0.09632012248039246,0.06521295011043549,0.619140625,1.01361083984375
post_attention_layernorm,14,15,0.4135667396061269,0.29256965944272445,0.4135667396061269,True,0.053455036133527756,0.06801187992095947,0.87646484375,0.736328125
post_attention_layernorm,14,16,0.2551505546751189,0.2418918918918919,0.2551505546751189,True,0.053455036133527756,0.08680811524391174,0.87646484375,0.72265625
post_attention_layernorm,14,17,0.28498372849837283,0.2595890410958904,0.28498372849837283,True,0.053455036133527756,0.09242258220911026,0.87646484375,0.712890625
post_attention_layernorm,14,18,0.39717282261741904,0.28426892950391647,0.39717282261741904,True,0.053455036133527756,0.09676424413919449,0.87646484375,0.8095703125
post_attention_layernorm,14,19,0.3989968080255358,0.28520208604954367,0.3989968080255358,True,0.053455036133527756,0.07613449543714523,0.87646484375,0.81640625
post_attention_layernorm,14,20,0.4737802097583219,0.3214727722772277,0.4737802097583219,True,0.053455036133527756,0.07808392494916916,0.87646484375,0.822265625
post_attention_layernorm,14,21,0.47515257192676547,0.32210401891252954,0.47515257192676547,True,0.053455036133527756,0.06570179015398026,0.87646484375,0.86328125
post_attention_layernorm,14,22,0.7402323125659979,0.4253640776699029,0.7402323125659979,True,0.053455036133527756,0.05012959614396095,0.87646484375,0.9713134765625
post_attention_layernorm,14,23,0.8220793140407289,0.4511764705882353,0.8220793140407289,True,0.053455036133527756,0.06521295011043549,0.87646484375,1.01361083984375
post_attention_layernorm,15,16,0.3656832298136646,0.576499388004896,0.576499388004896,True,0.06801187992095947,0.08680811524391174,0.736328125,0.72265625
post_attention_layernorm,15,17,0.29876160990712075,0.4260485651214128,0.4260485651214128,True,0.06801187992095947,0.09242258220911026,0.736328125,0.712890625
post_attention_layernorm,15,18,0.30124223602484473,0.4311111111111111,0.4311111111111111,True,0.06801187992095947,0.09676424413919449,0.736328125,0.8095703125
post_attention_layernorm,15,19,0.2840549102428722,0.22121710526315788,0.2840549102428722,True,0.06801187992095947,0.07613449543714523,0.736328125,0.81640625
post_attention_layernorm,15,20,0.33996683250414594,0.2537128712871287,0.33996683250414594,True,0.06801187992095947,0.07808392494916916,0.736328125,0.822265625
post_attention_layernorm,15,21,0.4206008583690987,0.29607250755287007,0.4206008583690987,True,0.06801187992095947,0.06570179015398026,0.736328125,0.86328125
post_attention_layernorm,15,22,0.45278969957081544,0.31166912850812406,0.45278969957081544,True,0.06801187992095947,0.05012959614396095,0.736328125,0.9713134765625
post_attention_layernorm,15,23,0.815378923352038,0.4491508152173913,0.815378923352038,True,0.06801187992095947,0.06521295011043549,0.736328125,1.01361083984375
post_attention_layernorm,16,17,0.17602657004830918,0.1496790757381258,0.17602657004830918,True,0.08680811524391174,0.09242258220911026,0.72265625,0.712890625
post_attention_layernorm,16,18,0.2154215976331361,0.17724022516354784,0.2154215976331361,True,0.08680811524391174,0.09676424413919449,0.72265625,0.8095703125
post_attention_layernorm,16,19,0.4420289855072464,0.3065326633165829,0.4420289855072464,True,0.08680811524391174,0.07613449543714523,0.72265625,0.81640625
post_attention_layernorm,16,20,0.33108935128518974,0.24873563218390804,0.33108935128518974,True,0.08680811524391174,0.07808392494916916,0.72265625,0.822265625
post_attention_layernorm,16,21,0.5990338164251208,0.37462235649546827,0.5990338164251208,True,0.08680811524391174,0.06570179015398026,0.72265625,0.86328125
post_attention_layernorm,16,22,1.0171358629130967,0.504247572815534,1.0171358629130967,True,0.08680811524391174,0.05012959614396095,0.72265625,0.9713134765625
post_attention_layernorm,16,23,1.0506634499396863,0.5123529411764706,1.0506634499396863,True,0.08680811524391174,0.06521295011043549,0.72265625,1.01361083984375
post_attention_layernorm,17,18,0.23767123287671232,0.19203099059214168,0.23767123287671232,True,0.09242258220911026,0.09676424413919449,0.712890625,0.8095703125
post_attention_layernorm,17,19,0.29360306178239476,0.22696534234995774,0.29360306178239476,True,0.09242258220911026,0.07613449543714523,0.712890625,0.81640625
post_attention_layernorm,17,20,0.20969441517386722,0.17334494773519163,0.20969441517386722,True,0.09242258220911026,0.07808392494916916,0.712890625,0.822265625
post_attention_layernorm,17,21,0.40350877192982454,0.2875,0.40350877192982454,True,0.09242258220911026,0.06570179015398026,0.712890625,0.86328125
post_attention_layernorm,17,22,0.8070175438596491,0.44660194174757284,0.8070175438596491,True,0.09242258220911026,0.05012959614396095,0.712890625,0.9713134765625
post_attention_layernorm,17,23,0.8589393110989612,0.46205882352941174,0.8589393110989612,True,0.09242258220911026,0.06521295011043549,0.712890625,1.01361083984375
post_attention_layernorm,18,19,0.27958115183246074,0.21849427168576105,0.27958115183246074,True,0.09676424413919449,0.07613449543714523,0.8095703125,0.81640625
post_attention_layernorm,18,20,0.20833333333333334,0.1724137931034483,0.20833333333333334,True,0.09676424413919449,0.07808392494916916,0.8095703125,0.822265625
post_attention_layernorm,18,21,0.4222222222222222,0.296875,0.4222222222222222,True,0.09676424413919449,0.06570179015398026,0.8095703125,0.86328125
post_attention_layernorm,18,22,0.8311111111111111,0.4538834951456311,0.8311111111111111,True,0.09676424413919449,0.05012959614396095,0.8095703125,0.9713134765625
post_attention_layernorm,18,23,0.8,0.4444444444444444,0.8,True,0.09676424413919449,0.06521295011043549,0.8095703125,1.01361083984375
post_attention_layernorm,19,20,0.151033386327504,0.1441947565543071,0.151033386327504,True,0.07613449543714523,0.07808392494916916,0.81640625,0.822265625
post_attention_layernorm,19,21,0.1678832116788321,0.14375,0.1678832116788321,True,0.07613449543714523,0.06570179015398026,0.81640625,0.86328125
post_attention_layernorm,19,22,0.5036496350364964,0.33495145631067963,0.5036496350364964,True,0.07613449543714523,0.05012959614396095,0.81640625,0.9713134765625
post_attention_layernorm,19,23,0.48267526188557613,0.34782608695652173,0.48267526188557613,True,0.07613449543714523,0.06521295011043549,0.81640625,1.01361083984375
post_attention_layernorm,20,21,0.2091324200913242,0.1729607250755287,0.2091324200913242,True,0.07808392494916916,0.06570179015398026,0.822265625,0.86328125
post_attention_layernorm,20,22,0.5154022988505748,0.3401092233009709,0.5154022988505748,True,0.07808392494916916,0.05012959614396095,0.822265625,0.9713134765625
post_attention_layernorm,20,23,0.5675426463808206,0.4219858156028369,0.5675426463808206,True,0.07808392494916916,0.06521295011043549,0.822265625,1.01361083984375
post_attention_layernorm,21,22,0.2875,0.22330097087378642,0.2875,True,0.06570179015398026,0.05012959614396095,0.86328125,0.9713134765625
post_attention_layernorm,21,23,0.3960546282245827,0.4148936170212766,0.4148936170212766,True,0.06570179015398026,0.06521295011043549,0.86328125,1.01361083984375
post_attention_layernorm,22,23,0.3124108416547789,0.34263295553618134,0.34263295553618134,True,0.05012959614396095,0.06521295011043549,0.9713134765625,1.01361083984375
1 role a b a_to_b_uniform_bound b_to_a_uniform_bound worst_uniform_bound eligible_nonconstant_gains gain_std_a gain_std_b gain_min_a gain_min_b
2 input_layernorm 0 1 0.6032863849765259 1.5207100591715976 1.5207100591715976 True 0.11805251985788345 0.0930413156747818 0.794921875 0.607421875
3 input_layernorm 0 2 0.5692488262910798 1.3215258855585832 1.3215258855585832 True 0.11805251985788345 0.10473614931106567 0.794921875 0.6484375
4 input_layernorm 0 3 0.5469483568075117 1.2072538860103628 1.2072538860103628 True 0.11805251985788345 0.07621725648641586 0.794921875 0.7265625
5 input_layernorm 0 4 0.5751173708920188 1.3535911602209945 1.3535911602209945 True 0.11805251985788345 0.10847259312868118 0.794921875 0.662109375
6 input_layernorm 0 5 0.6396713615023474 1.775244299674267 1.775244299674267 True 0.11805251985788345 0.13199757039546967 0.794921875 0.5234375
7 input_layernorm 0 6 0.6643192488262911 1.979020979020979 1.979020979020979 True 0.11805251985788345 0.1498883068561554 0.794921875 0.47265625
8 input_layernorm 0 7 0.6830985915492958 2.1555555555555554 2.1555555555555554 True 0.11805251985788345 0.095920130610466 0.794921875 0.46484375
9 input_layernorm 0 8 0.6913145539906104 2.2395437262357416 2.2395437262357416 True 0.11805251985788345 0.20054922997951508 0.794921875 0.41796875
10 input_layernorm 0 9 0.7136150234741784 2.4918032786885247 2.4918032786885247 True 0.11805251985788345 0.22649425268173218 0.794921875 0.42578125
11 input_layernorm 0 10 0.7230046948356808 2.610169491525424 2.610169491525424 True 0.11805251985788345 0.2204512506723404 0.794921875 0.3828125
12 input_layernorm 0 11 0.7347417840375586 2.769911504424779 2.769911504424779 True 0.11805251985788345 0.16603894531726837 0.794921875 0.34375
13 input_layernorm 0 12 0.6338028169014085 1.7307692307692308 1.7307692307692308 True 0.11805251985788345 0.1900545060634613 0.794921875 0.5546875
14 input_layernorm 0 13 0.6349765258215962 1.7395498392282958 1.7395498392282958 True 0.11805251985788345 0.16846242547035217 0.794921875 0.5703125
15 input_layernorm 0 14 0.5516431924882629 1.2303664921465969 1.2303664921465969 True 0.11805251985788345 0.11825777590274811 0.794921875 0.6875
16 input_layernorm 0 15 0.49018199233716475 0.9614842649131048 0.9614842649131048 True 0.11805251985788345 0.06029079854488373 0.794921875 0.779296875
17 input_layernorm 0 16 0.5117370892018779 1.0480769230769231 1.0480769230769231 True 0.11805251985788345 0.09195870161056519 0.794921875 0.7763671875
18 input_layernorm 0 17 0.6232876712328768 0.7185185185185186 0.7185185185185186 True 0.11805251985788345 0.06483674794435501 0.794921875 1.0012359619140625
19 input_layernorm 0 18 0.8402948402948403 0.6637450199203188 0.8402948402948403 True 0.11805251985788345 0.06785038113594055 0.794921875 1.1640625
20 input_layernorm 0 19 1.0393120393120394 1.1248445660283513 1.1248445660283513 True 0.11805251985788345 0.19409698247909546 0.794921875 0.884765625
21 input_layernorm 0 20 1.113022113022113 0.9869156655168312 1.113022113022113 True 0.11805251985788345 0.08625888079404831 0.794921875 1.0262451171875
22 input_layernorm 0 21 1.1867321867321867 0.9545986426398315 1.1867321867321867 True 0.11805251985788345 0.07095508277416229 0.794921875 1.043212890625
23 input_layernorm 0 22 0.9656019656019657 0.9514018691588785 0.9656019656019657 True 0.11805251985788345 0.07214703410863876 0.794921875 1.044921875
24 input_layernorm 0 23 0.941031941031941 0.9568884723523898 0.9568884723523898 True 0.11805251985788345 0.15369786322116852 0.794921875 0.98577880859375
25 input_layernorm 1 2 0.16765578635014836 0.17593150850114556 0.17593150850114556 True 0.0930413156747818 0.10473614931106567 0.607421875 0.6484375
26 input_layernorm 1 3 0.28125 0.34967320261437906 0.34967320261437906 True 0.0930413156747818 0.07621725648641586 0.607421875 0.7265625
27 input_layernorm 1 4 0.1678622668579627 0.15668366741786266 0.1678622668579627 True 0.0930413156747818 0.10847259312868118 0.607421875 0.662109375
28 input_layernorm 1 5 0.23018147086914995 0.2990074441687345 0.2990074441687345 True 0.0930413156747818 0.13199757039546967 0.607421875 0.5234375
29 input_layernorm 1 6 0.34367141659681477 0.5236270753512133 0.5236270753512133 True 0.0930413156747818 0.1498883068561554 0.607421875 0.47265625
30 input_layernorm 1 7 0.3996670702179177 0.6657423745903706 0.6657423745903706 True 0.0930413156747818 0.095920130610466 0.607421875 0.46484375
31 input_layernorm 1 8 0.4585726004922067 0.8469696969696969 0.8469696969696969 True 0.0930413156747818 0.20054922997951508 0.607421875 0.41796875
32 input_layernorm 1 9 0.5138306789606035 1.056896551724138 1.056896551724138 True 0.0930413156747818 0.22649425268173218 0.607421875 0.42578125
33 input_layernorm 1 10 0.4786253143336127 0.9180064308681672 0.9180064308681672 True 0.0930413156747818 0.2204512506723404 0.607421875 0.3828125
34 input_layernorm 1 11 0.49209302325581394 0.9688644688644689 0.9688644688644689 True 0.0930413156747818 0.16603894531726837 0.607421875 0.34375
35 input_layernorm 1 12 0.3838141025641026 0.6228868660598179 0.6228868660598179 True 0.0930413156747818 0.1900545060634613 0.607421875 0.5546875
36 input_layernorm 1 13 0.39622641509433965 0.65625 0.65625 True 0.0930413156747818 0.16846242547035217 0.607421875 0.5703125
37 input_layernorm 1 14 0.27009646302250806 0.28465608465608466 0.28465608465608466 True 0.0930413156747818 0.11825777590274811 0.607421875 0.6875
38 input_layernorm 1 15 0.9292604501607717 0.5223386651958081 0.9292604501607717 True 0.0930413156747818 0.06029079854488373 0.607421875 0.779296875
39 input_layernorm 1 16 0.36012861736334406 0.2647754137115839 0.36012861736334406 True 0.0930413156747818 0.09195870161056519 0.607421875 0.7763671875
40 input_layernorm 1 17 0.7958860759493671 0.4431718061674009 0.7958860759493671 True 0.0930413156747818 0.06483674794435501 0.607421875 1.0012359619140625
41 input_layernorm 1 18 1.1044303797468353 0.524812030075188 1.1044303797468353 True 0.0930413156747818 0.06785038113594055 0.607421875 1.1640625
42 input_layernorm 1 19 1.3858520900321543 0.5808625336927223 1.3858520900321543 True 0.0930413156747818 0.19409698247909546 0.607421875 0.884765625
43 input_layernorm 1 20 1.206896551724138 0.546875 1.206896551724138 True 0.0930413156747818 0.08625888079404831 0.607421875 1.0262451171875
44 input_layernorm 1 21 1.630225080385852 0.6198044009779952 1.630225080385852 True 0.0930413156747818 0.07095508277416229 0.607421875 1.043212890625
45 input_layernorm 1 22 1.4043887147335423 0.5840938722294654 1.4043887147335423 True 0.0930413156747818 0.07214703410863876 0.607421875 1.044921875
46 input_layernorm 1 23 1.9453376205787782 0.6604803493449781 1.9453376205787782 True 0.0930413156747818 0.15369786322116852 0.607421875 0.98577880859375
47 input_layernorm 2 3 0.2867132867132867 0.4019607843137255 0.4019607843137255 True 0.10473614931106567 0.07621725648641586 0.6484375 0.7265625
48 input_layernorm 2 4 0.10583941605839416 0.11836734693877551 0.11836734693877551 True 0.10473614931106567 0.10847259312868118 0.6484375 0.662109375
49 input_layernorm 2 5 0.23737099402498643 0.31125356125356124 0.31125356125356124 True 0.10473614931106567 0.13199757039546967 0.6484375 0.5234375
50 input_layernorm 2 6 0.31070496083550914 0.45075757575757575 0.45075757575757575 True 0.10473614931106567 0.1498883068561554 0.6484375 0.47265625
51 input_layernorm 2 7 0.42205710955710957 0.7302747668263171 0.7302747668263171 True 0.10473614931106567 0.095920130610466 0.6484375 0.46484375
52 input_layernorm 2 8 0.4018264840182648 0.6717557251908397 0.6717557251908397 True 0.10473614931106567 0.20054922997951508 0.6484375 0.41796875
53 input_layernorm 2 9 0.4722187382415653 0.8947243346007605 0.8947243346007605 True 0.10473614931106567 0.22649425268173218 0.6484375 0.42578125
54 input_layernorm 2 10 0.46013404206147446 0.8523116438356164 0.8523116438356164 True 0.10473614931106567 0.2204512506723404 0.6484375 0.3828125
55 input_layernorm 2 11 0.5159817351598174 1.0660377358490567 1.0660377358490567 True 0.10473614931106567 0.16603894531726837 0.6484375 0.34375
56 input_layernorm 2 12 0.30617848970251715 0.4412928759894459 0.4412928759894459 True 0.10473614931106567 0.1900545060634613 0.6484375 0.5546875
57 input_layernorm 2 13 0.2900036175087423 0.4084578804347826 0.4084578804347826 True 0.10473614931106567 0.16846242547035217 0.6484375 0.5703125
58 input_layernorm 2 14 0.19501226492232215 0.24225495175215844 0.24225495175215844 True 0.10473614931106567 0.11825777590274811 0.6484375 0.6875
59 input_layernorm 2 15 0.7957142857142857 0.5678391959798995 0.7957142857142857 True 0.10473614931106567 0.06029079854488373 0.6484375 0.779296875
60 input_layernorm 2 16 0.2740963855421687 0.21513002364066194 0.2740963855421687 True 0.10473614931106567 0.09195870161056519 0.6484375 0.7763671875
61 input_layernorm 2 17 0.7093373493975904 0.4149779735682819 0.7093373493975904 True 0.10473614931106567 0.06483674794435501 0.6484375 1.0012359619140625
62 input_layernorm 2 18 1.0030120481927711 0.5007518796992482 1.0030120481927711 True 0.10473614931106567 0.06785038113594055 0.6484375 1.1640625
63 input_layernorm 2 19 1.2680722891566265 0.5590969455511288 1.2680722891566265 True 0.10473614931106567 0.19409698247909546 0.6484375 0.884765625
64 input_layernorm 2 20 1.0873493975903614 0.5209235209235209 1.0873493975903614 True 0.10473614931106567 0.08625888079404831 0.6484375 1.0262451171875
65 input_layernorm 2 21 1.463855421686747 0.5941320293398533 1.463855421686747 True 0.10473614931106567 0.07095508277416229 0.6484375 1.043212890625
66 input_layernorm 2 22 1.249266862170088 0.5554106910039114 1.249266862170088 True 0.10473614931106567 0.07214703410863876 0.6484375 1.044921875
67 input_layernorm 2 23 1.7448680351906158 0.6356837606837606 1.7448680351906158 True 0.10473614931106567 0.15369786322116852 0.6484375 0.98577880859375
68 input_layernorm 3 4 0.40522875816993464 0.28837209302325584 0.40522875816993464 True 0.07621725648641586 0.10847259312868118 0.7265625 0.662109375
69 input_layernorm 3 5 0.3660130718954248 0.39925373134328357 0.39925373134328357 True 0.07621725648641586 0.13199757039546967 0.7265625 0.5234375
70 input_layernorm 3 6 0.35609756097560974 0.553030303030303 0.553030303030303 True 0.07621725648641586 0.1498883068561554 0.7265625 0.47265625
71 input_layernorm 3 7 0.39575530586766544 0.6549586776859504 0.6549586776859504 True 0.07621725648641586 0.095920130610466 0.7265625 0.46484375
72 input_layernorm 3 8 0.42473118279569894 0.7383177570093458 0.7383177570093458 True 0.07621725648641586 0.20054922997951508 0.7265625 0.41796875
73 input_layernorm 3 9 0.49180741114192084 0.9677579365079365 0.9677579365079365 True 0.07621725648641586 0.22649425268173218 0.7265625 0.42578125
74 input_layernorm 3 10 0.47804878048780486 0.9158878504672897 0.9158878504672897 True 0.07621725648641586 0.2204512506723404 0.7265625 0.3828125
75 input_layernorm 3 11 0.545338441890166 1.199438202247191 1.199438202247191 True 0.07621725648641586 0.16603894531726837 0.7265625 0.34375
76 input_layernorm 3 12 0.40460669274228594 0.5290237467018469 0.5290237467018469 True 0.07621725648641586 0.1900545060634613 0.7265625 0.5546875
77 input_layernorm 3 13 0.3559322033898305 0.5441576086956522 0.5441576086956522 True 0.07621725648641586 0.16846242547035217 0.7265625 0.5703125
78 input_layernorm 3 14 0.28804587560652845 0.22363013698630138 0.28804587560652845 True 0.07621725648641586 0.11825777590274811 0.7265625 0.6875
79 input_layernorm 3 15 0.5853333333333334 0.36921783010933557 0.5853333333333334 True 0.07621725648641586 0.06029079854488373 0.7265625 0.779296875
80 input_layernorm 3 16 0.43790849673202614 0.30454545454545456 0.43790849673202614 True 0.07621725648641586 0.09195870161056519 0.7265625 0.7763671875
81 input_layernorm 3 17 0.5490196078431373 0.35443037974683544 0.5490196078431373 True 0.07621725648641586 0.06483674794435501 0.7265625 1.0012359619140625
82 input_layernorm 3 18 0.7876344086021505 0.4406015037593985 0.7876344086021505 True 0.07621725648641586 0.06785038113594055 0.7265625 1.1640625
83 input_layernorm 3 19 1.0241935483870968 0.5059760956175299 1.0241935483870968 True 0.07621725648641586 0.19409698247909546 0.7265625 0.884765625
84 input_layernorm 3 20 0.9153567110036276 0.4779040404040404 0.9153567110036276 True 0.07621725648641586 0.08625888079404831 0.7265625 1.0262451171875
85 input_layernorm 3 21 1.1706666666666667 0.5393120393120393 1.1706666666666667 True 0.07621725648641586 0.07095508277416229 0.7265625 1.043212890625
86 input_layernorm 3 22 1.0453333333333332 0.5110821382007823 1.0453333333333332 True 0.07621725648641586 0.07214703410863876 0.7265625 1.044921875
87 input_layernorm 3 23 1.496 0.5993589743589743 1.496 True 0.07621725648641586 0.15369786322116852 0.7265625 0.98577880859375
88 input_layernorm 4 5 0.21176470588235294 0.26865671641791045 0.26865671641791045 True 0.10847259312868118 0.13199757039546967 0.662109375 0.5234375
89 input_layernorm 4 6 0.2985507246376812 0.4256198347107438 0.4256198347107438 True 0.10847259312868118 0.1498883068561554 0.662109375 0.47265625
90 input_layernorm 4 7 0.42340116279069767 0.7343080413410638 0.7343080413410638 True 0.10847259312868118 0.095920130610466 0.662109375 0.46484375
91 input_layernorm 4 8 0.37971014492753624 0.6121495327102804 0.6121495327102804 True 0.10847259312868118 0.20054922997951508 0.662109375 0.41796875
92 input_layernorm 4 9 0.4610917537746806 0.8556034482758621 0.8556034482758621 True 0.10847259312868118 0.22649425268173218 0.662109375 0.42578125
93 input_layernorm 4 10 0.4318840579710145 0.7602040816326531 0.7602040816326531 True 0.10847259312868118 0.2204512506723404 0.662109375 0.3828125
94 input_layernorm 4 11 0.5041140976412507 1.0165929203539823 1.0165929203539823 True 0.10847259312868118 0.16603894531726837 0.662109375 0.34375
95 input_layernorm 4 12 0.31369924141008476 0.45708712613784136 0.45708712613784136 True 0.10847259312868118 0.1900545060634613 0.662109375 0.5546875
96 input_layernorm 4 13 0.3016249555212905 0.4318953804347826 0.4318953804347826 True 0.10847259312868118 0.16846242547035217 0.662109375 0.5703125
97 input_layernorm 4 14 0.177762982689747 0.21619433198380567 0.21619433198380567 True 0.10847259312868118 0.11825777590274811 0.662109375 0.6875
98 input_layernorm 4 15 0.799410029498525 0.5714938328003655 0.799410029498525 True 0.10847259312868118 0.06029079854488373 0.662109375 0.779296875
99 input_layernorm 4 16 0.2727919383521043 0.21432563338301044 0.2727919383521043 True 0.10847259312868118 0.09195870161056519 0.662109375 0.7763671875
100 input_layernorm 4 17 0.644927536231884 0.3920704845814978 0.644927536231884 True 0.10847259312868118 0.06483674794435501 0.662109375 1.0012359619140625
101 input_layernorm 4 18 0.927536231884058 0.48120300751879697 0.927536231884058 True 0.10847259312868118 0.06785038113594055 0.662109375 1.1640625
102 input_layernorm 4 19 1.191740412979351 0.5437415881561238 1.191740412979351 True 0.10847259312868118 0.19409698247909546 0.662109375 0.884765625
103 input_layernorm 4 20 1.0705882352941176 0.5170454545454546 1.0705882352941176 True 0.10847259312868118 0.08625888079404831 0.662109375 1.0262451171875
104 input_layernorm 4 21 1.4129793510324484 0.5855745721271394 1.4129793510324484 True 0.10847259312868118 0.07095508277416229 0.662109375 1.043212890625
105 input_layernorm 4 22 1.2558823529411764 0.5567144719687093 1.2558823529411764 True 0.10847259312868118 0.07214703410863876 0.662109375 1.044921875
106 input_layernorm 4 23 1.7529411764705882 0.6367521367521367 1.7529411764705882 True 0.10847259312868118 0.15369786322116852 0.662109375 0.98577880859375
107 input_layernorm 5 6 0.21281039684381528 0.27034198113207547 0.27034198113207547 True 0.13199757039546967 0.1498883068561554 0.5234375 0.47265625
108 input_layernorm 5 7 0.40684808612440193 0.6859087471641039 0.6859087471641039 True 0.13199757039546967 0.095920130610466 0.5234375 0.46484375
109 input_layernorm 5 8 0.3513911921872121 0.5417613636363636 0.5417613636363636 True 0.13199757039546967 0.20054922997951508 0.5234375 0.41796875
110 input_layernorm 5 9 0.41179234202490017 0.7000798722044729 0.7000798722044729 True 0.13199757039546967 0.22649425268173218 0.5234375 0.42578125
111 input_layernorm 5 10 0.35843218153687467 0.5586816720257235 0.5586816720257235 True 0.13199757039546967 0.2204512506723404 0.5234375 0.3828125
112 input_layernorm 5 11 0.4428827751196172 0.7949543746645196 0.7949543746645196 True 0.13199757039546967 0.16603894531726837 0.5234375 0.34375
113 input_layernorm 5 12 0.2774254169603007 0.3839401820546164 0.3839401820546164 True 0.13199757039546967 0.1900545060634613 0.5234375 0.5546875
114 input_layernorm 5 13 0.2767029052269517 0.3825577445652174 0.3825577445652174 True 0.13199757039546967 0.16846242547035217 0.5234375 0.5703125
115 input_layernorm 5 14 0.4141791044776119 0.2928759894459103 0.4141791044776119 True 0.13199757039546967 0.11825777590274811 0.5234375 0.6875
116 input_layernorm 5 15 1.2182835820895523 0.5492010092514719 1.2182835820895523 True 0.13199757039546967 0.06029079854488373 0.5234375 0.779296875
117 input_layernorm 5 16 0.5316051136363636 0.34709019244145606 0.5316051136363636 True 0.13199757039546967 0.09195870161056519 0.5234375 0.7763671875
118 input_layernorm 5 17 1.0124113475177305 0.5030837004405286 1.0124113475177305 True 0.13199757039546967 0.06483674794435501 0.5234375 1.0012359619140625
119 input_layernorm 5 18 1.3581560283687943 0.5759398496240602 1.3581560283687943 True 0.13199757039546967 0.06785038113594055 0.5234375 1.1640625
120 input_layernorm 5 19 1.7798507462686568 0.640268456375839 1.7798507462686568 True 0.13199757039546967 0.19409698247909546 0.5234375 0.884765625
121 input_layernorm 5 20 1.626865671641791 0.6193181818181818 1.626865671641791 True 0.13199757039546967 0.08625888079404831 0.5234375 1.0262451171875
122 input_layernorm 5 21 2.037313432835821 0.6707616707616708 2.037313432835821 True 0.13199757039546967 0.07095508277416229 0.5234375 1.043212890625
123 input_layernorm 5 22 1.8619402985074627 0.6505867014341591 1.8619402985074627 True 0.13199757039546967 0.07214703410863876 0.5234375 1.044921875
124 input_layernorm 5 23 2.4925373134328357 0.7136752136752137 2.4925373134328357 True 0.13199757039546967 0.15369786322116852 0.5234375 0.98577880859375
125 input_layernorm 6 7 0.37231012658227847 0.5931434333249307 0.5931434333249307 True 0.1498883068561554 0.095920130610466 0.47265625 0.46484375
126 input_layernorm 6 8 0.1901840490797546 0.23484848484848486 0.23484848484848486 True 0.1498883068561554 0.20054922997951508 0.47265625 0.41796875
127 input_layernorm 6 9 0.26697892271662765 0.36421725239616615 0.36421725239616615 True 0.1498883068561554 0.22649425268173218 0.47265625 0.42578125
128 input_layernorm 6 10 0.2579185520361991 0.3475609756097561 0.3475609756097561 True 0.1498883068561554 0.2204512506723404 0.47265625 0.3828125
129 input_layernorm 6 11 0.41044303797468357 0.6961889425657541 0.6961889425657541 True 0.1498883068561554 0.16603894531726837 0.47265625 0.34375
130 input_layernorm 6 12 0.2694422623723488 0.21225247524752475 0.2694422623723488 True 0.1498883068561554 0.1900545060634613 0.47265625 0.5546875
131 input_layernorm 6 13 0.2518703241895262 0.22759714512291832 0.2518703241895262 True 0.1498883068561554 0.16846242547035217 0.47265625 0.5703125
132 input_layernorm 6 14 0.5578512396694215 0.35809018567639256 0.5578512396694215 True 0.1498883068561554 0.11825777590274811 0.47265625 0.6875
133 input_layernorm 6 15 1.3971774193548387 0.5828427249789739 1.3971774193548387 True 0.1498883068561554 0.06029079854488373 0.47265625 0.779296875
134 input_layernorm 6 16 0.7217261904761905 0.4191875540190147 0.7217261904761905 True 0.1498883068561554 0.09195870161056519 0.47265625 0.7763671875
135 input_layernorm 6 17 1.3450413223140496 0.573568281938326 1.3450413223140496 True 0.1498883068561554 0.06483674794435501 0.47265625 1.0012359619140625
136 input_layernorm 6 18 1.7479338842975207 0.6360902255639098 1.7479338842975207 True 0.1498883068561554 0.06785038113594055 0.47265625 1.1640625
137 input_layernorm 6 19 2.1115702479338845 0.6786188579017264 2.1115702479338845 True 0.1498883068561554 0.19409698247909546 0.47265625 0.884765625
138 input_layernorm 6 20 1.8636363636363635 0.6507936507936508 1.8636363636363635 True 0.1498883068561554 0.08625888079404831 0.47265625 1.0262451171875
139 input_layernorm 6 21 2.282258064516129 0.6953316953316954 2.282258064516129 True 0.1498883068561554 0.07095508277416229 0.47265625 1.043212890625
140 input_layernorm 6 22 2.092741935483871 0.6766623207301173 2.092741935483871 True 0.1498883068561554 0.07214703410863876 0.47265625 1.044921875
141 input_layernorm 6 23 2.774193548387097 0.7350427350427351 2.774193548387097 True 0.1498883068561554 0.15369786322116852 0.47265625 0.98577880859375
142 input_layernorm 7 8 0.564910511721704 0.6390151515151515 0.6390151515151515 True 0.095920130610466 0.20054922997951508 0.46484375 0.41796875
143 input_layernorm 7 9 0.4822283841693975 0.8075657894736842 0.8075657894736842 True 0.095920130610466 0.22649425268173218 0.46484375 0.42578125
144 input_layernorm 7 10 0.4459289135366776 0.6383139067524116 0.6383139067524116 True 0.095920130610466 0.2204512506723404 0.46484375 0.3828125
145 input_layernorm 7 11 0.3941018766756032 0.6504424778761062 0.6504424778761062 True 0.095920130610466 0.16603894531726837 0.46484375 0.34375
146 input_layernorm 7 12 0.5676959619952494 0.449868073878628 0.5676959619952494 True 0.095920130610466 0.1900545060634613 0.46484375 0.5546875
147 input_layernorm 7 13 0.5866983372921615 0.46976902173913043 0.5866983372921615 True 0.095920130610466 0.16846242547035217 0.46484375 0.5703125
148 input_layernorm 7 14 0.630188679245283 0.38657407407407407 0.630188679245283 True 0.095920130610466 0.11825777590274811 0.46484375 0.6875
149 input_layernorm 7 15 1.4814049586776858 0.5970024979184013 1.4814049586776858 True 0.095920130610466 0.06029079854488373 0.46484375 0.779296875
150 input_layernorm 7 16 0.9477434679334917 0.4865853658536585 0.9477434679334917 True 0.095920130610466 0.09195870161056519 0.46484375 0.7763671875
151 input_layernorm 7 17 1.2394957983193278 0.5534709193245778 1.2394957983193278 True 0.095920130610466 0.06483674794435501 0.46484375 1.0012359619140625
152 input_layernorm 7 18 1.5630081300813008 0.6098334655035687 1.5630081300813008 True 0.095920130610466 0.06785038113594055 0.46484375 1.1640625
153 input_layernorm 7 19 2.19327731092437 0.6868421052631579 2.19327731092437 True 0.095920130610466 0.19409698247909546 0.46484375 0.884765625
154 input_layernorm 7 20 1.9201680672268908 0.6575539568345323 1.9201680672268908 True 0.095920130610466 0.08625888079404831 0.46484375 1.0262451171875
155 input_layernorm 7 21 2.3445378151260505 0.7010050251256281 2.3445378151260505 True 0.095920130610466 0.07095508277416229 0.46484375 1.043212890625
156 input_layernorm 7 22 2.1178861788617884 0.6792698826597132 2.1178861788617884 True 0.095920130610466 0.07214703410863876 0.46484375 1.044921875
157 input_layernorm 7 23 2.8048780487804876 0.7371794871794872 2.8048780487804876 True 0.095920130610466 0.15369786322116852 0.46484375 0.98577880859375
158 input_layernorm 8 9 0.2168141592920354 0.1781818181818182 0.2168141592920354 True 0.20054922997951508 0.22649425268173218 0.41796875 0.42578125
159 input_layernorm 8 10 0.19393939393939394 0.1813186813186813 0.19393939393939394 True 0.20054922997951508 0.2204512506723404 0.41796875 0.3828125
160 input_layernorm 8 11 0.39980670103092786 0.666129898013956 0.666129898013956 True 0.20054922997951508 0.16603894531726837 0.41796875 0.34375
161 input_layernorm 8 12 0.3672566371681416 0.2686084142394822 0.3672566371681416 True 0.20054922997951508 0.1900545060634613 0.41796875 0.5546875
162 input_layernorm 8 13 0.3644859813084112 0.2671232876712329 0.3644859813084112 True 0.20054922997951508 0.16846242547035217 0.41796875 0.5703125
163 input_layernorm 8 14 0.7616822429906542 0.4323607427055703 0.7616822429906542 True 0.20054922997951508 0.11825777590274811 0.41796875 0.6875
164 input_layernorm 8 15 1.677927927927928 0.6265769554247267 1.677927927927928 True 0.20054922997951508 0.06029079854488373 0.41796875 0.779296875
165 input_layernorm 8 16 0.9478114478114478 0.48660328435609335 0.9478114478114478 True 0.20054922997951508 0.09195870161056519 0.41796875 0.7763671875
166 input_layernorm 8 17 1.6518691588785046 0.6229074889867842 1.6518691588785046 True 0.20054922997951508 0.06483674794435501 0.41796875 1.0012359619140625
167 input_layernorm 8 18 2.107476635514019 0.6781954887218045 2.107476635514019 True 0.20054922997951508 0.06785038113594055 0.41796875 1.1640625
168 input_layernorm 8 19 2.5186915887850465 0.7158034528552457 2.5186915887850465 True 0.20054922997951508 0.19409698247909546 0.41796875 0.884765625
169 input_layernorm 8 20 2.282442748091603 0.6953488372093023 2.282442748091603 True 0.20054922997951508 0.08625888079404831 0.41796875 1.0262451171875
170 input_layernorm 8 21 2.6666666666666665 0.7272727272727273 2.6666666666666665 True 0.20054922997951508 0.07095508277416229 0.41796875 1.043212890625
171 input_layernorm 8 22 2.467289719626168 0.7115902964959568 2.467289719626168 True 0.20054922997951508 0.07214703410863876 0.41796875 1.044921875
172 input_layernorm 8 23 3.2162162162162162 0.7628205128205128 3.2162162162162162 True 0.20054922997951508 0.15369786322116852 0.41796875 0.98577880859375
173 input_layernorm 9 10 0.31333333333333335 0.3008849557522124 0.31333333333333335 True 0.22649425268173218 0.2204512506723404 0.42578125 0.3828125
174 input_layernorm 9 11 0.5083333333333333 0.5780998389694042 0.5780998389694042 True 0.22649425268173218 0.16603894531726837 0.42578125 0.34375
175 input_layernorm 9 12 0.452991452991453 0.31176470588235294 0.452991452991453 True 0.22649425268173218 0.1900545060634613 0.42578125 0.5546875
176 input_layernorm 9 13 0.4921259842519685 0.32981530343007914 0.4921259842519685 True 0.22649425268173218 0.16846242547035217 0.42578125 0.5703125
177 input_layernorm 9 14 0.8134920634920635 0.4485776805251641 0.8134920634920635 True 0.22649425268173218 0.11825777590274811 0.42578125 0.6875
178 input_layernorm 9 15 1.7727272727272727 0.639344262295082 1.7727272727272727 True 0.22649425268173218 0.06029079854488373 0.42578125 0.779296875
179 input_layernorm 9 16 1.2080152671755726 0.547104580812446 1.2080152671755726 True 0.22649425268173218 0.09195870161056519 0.42578125 0.7763671875
180 input_layernorm 9 17 1.7058823529411764 0.6304347826086957 1.7058823529411764 True 0.22649425268173218 0.06483674794435501 0.42578125 1.0012359619140625
181 input_layernorm 9 18 2.1470588235294117 0.6822429906542056 2.1470588235294117 True 0.22649425268173218 0.06785038113594055 0.42578125 1.1640625
182 input_layernorm 9 19 2.4873949579831933 0.7132530120481928 2.4873949579831933 True 0.22649425268173218 0.19409698247909546 0.42578125 0.884765625
183 input_layernorm 9 20 2.6134453781512605 0.7232558139534884 2.6134453781512605 True 0.22649425268173218 0.08625888079404831 0.42578125 1.0262451171875
184 input_layernorm 9 21 2.7394957983193278 0.7325842696629213 2.7394957983193278 True 0.22649425268173218 0.07095508277416229 0.42578125 1.043212890625
185 input_layernorm 9 22 2.403669724770642 0.706199460916442 2.403669724770642 True 0.22649425268173218 0.07214703410863876 0.42578125 1.044921875
186 input_layernorm 9 23 2.990990990990991 0.7494356659142212 2.990990990990991 True 0.22649425268173218 0.15369786322116852 0.42578125 0.98577880859375
187 input_layernorm 10 11 0.350418410041841 0.5394524959742351 0.5394524959742351 True 0.2204512506723404 0.16603894531726837 0.3828125 0.34375
188 input_layernorm 10 12 0.4489795918367347 0.30985915492957744 0.4489795918367347 True 0.2204512506723404 0.1900545060634613 0.3828125 0.5546875
189 input_layernorm 10 13 0.4897959183673469 0.3287671232876712 0.4897959183673469 True 0.2204512506723404 0.16846242547035217 0.3828125 0.5703125
190 input_layernorm 10 14 0.923469387755102 0.48010610079575594 0.923469387755102 True 0.2204512506723404 0.11825777590274811 0.3828125 0.6875
191 input_layernorm 10 15 1.8762755102040816 0.6523281596452328 1.8762755102040816 True 0.2204512506723404 0.06029079854488373 0.3828125 0.779296875
192 input_layernorm 10 16 1.1346863468634687 0.5315471045808124 1.1346863468634687 True 0.2204512506723404 0.09195870161056519 0.3828125 0.7763671875
193 input_layernorm 10 17 1.8954081632653061 0.654625550660793 1.8954081632653061 True 0.2204512506723404 0.06483674794435501 0.3828125 1.0012359619140625
194 input_layernorm 10 18 2.392857142857143 0.7052631578947368 2.392857142857143 True 0.2204512506723404 0.06785038113594055 0.3828125 1.1640625
195 input_layernorm 10 19 2.8418367346938775 0.7397078353253652 2.8418367346938775 True 0.2204512506723404 0.19409698247909546 0.3828125 0.884765625
196 input_layernorm 10 20 2.5357142857142856 0.7171717171717171 2.5357142857142856 True 0.2204512506723404 0.08625888079404831 0.3828125 1.0262451171875
197 input_layernorm 10 21 2.9285714285714284 0.7454545454545455 2.9285714285714284 True 0.2204512506723404 0.07095508277416229 0.3828125 1.043212890625
198 input_layernorm 10 22 2.7857142857142856 0.7358490566037735 2.7857142857142856 True 0.2204512506723404 0.07214703410863876 0.3828125 1.044921875
199 input_layernorm 10 23 3.3333333333333335 0.7692307692307693 3.3333333333333335 True 0.2204512506723404 0.15369786322116852 0.3828125 0.98577880859375
200 input_layernorm 11 12 0.7078651685393258 0.4144736842105263 0.7078651685393258 True 0.16603894531726837 0.1900545060634613 0.34375 0.5546875
201 input_layernorm 11 13 0.7368421052631579 0.42424242424242425 0.7368421052631579 True 0.16603894531726837 0.16846242547035217 0.34375 0.5703125
202 input_layernorm 11 14 1.1420454545454546 0.53315649867374 1.1420454545454546 True 0.16603894531726837 0.11825777590274811 0.34375 0.6875
203 input_layernorm 11 15 2.3516483516483517 0.7016393442622951 2.3516483516483517 True 0.16603894531726837 0.06029079854488373 0.34375 0.779296875
204 input_layernorm 11 16 1.414745145631068 0.5858776228169368 1.414745145631068 True 0.16603894531726837 0.09195870161056519 0.34375 0.7763671875
205 input_layernorm 11 17 2.2244318181818183 0.6898678414096916 2.2244318181818183 True 0.16603894531726837 0.06483674794435501 0.34375 1.0012359619140625
206 input_layernorm 11 18 2.778409090909091 0.7353383458646616 2.778409090909091 True 0.16603894531726837 0.06785038113594055 0.34375 1.1640625
207 input_layernorm 11 19 3.278409090909091 0.7662682602921647 3.278409090909091 True 0.16603894531726837 0.19409698247909546 0.34375 0.884765625
208 input_layernorm 11 20 3.056603773584906 0.7534883720930232 3.056603773584906 True 0.16603894531726837 0.08625888079404831 0.34375 1.0262451171875
209 input_layernorm 11 21 3.4831460674157304 0.7769423558897243 3.4831460674157304 True 0.16603894531726837 0.07095508277416229 0.34375 1.043212890625
210 input_layernorm 11 22 3.215909090909091 0.7628032345013477 3.215909090909091 True 0.16603894531726837 0.07214703410863876 0.34375 1.044921875
211 input_layernorm 11 23 4.142857142857143 0.8055555555555556 4.142857142857143 True 0.16603894531726837 0.15369786322116852 0.34375 0.98577880859375
212 input_layernorm 12 13 0.10185185185185185 0.1134020618556701 0.1134020618556701 True 0.1900545060634613 0.16846242547035217 0.5546875 0.5703125
213 input_layernorm 12 14 0.3644343302990897 0.26709554443650224 0.3644343302990897 True 0.1900545060634613 0.11825777590274811 0.5546875 0.6875
214 input_layernorm 12 15 1.0359589041095891 0.5088309503784693 1.0359589041095891 True 0.1900545060634613 0.06029079854488373 0.5546875 0.779296875
215 input_layernorm 12 16 0.6295774647887324 0.3863439930855661 0.6295774647887324 True 0.1900545060634613 0.09195870161056519 0.5546875 0.7763671875
216 input_layernorm 12 17 0.9982394366197183 0.4995594713656388 0.9982394366197183 True 0.1900545060634613 0.06483674794435501 0.5546875 1.0012359619140625
217 input_layernorm 12 18 1.341549295774648 0.5729323308270676 1.341549295774648 True 0.1900545060634613 0.06785038113594055 0.5546875 1.1640625
218 input_layernorm 12 19 1.6514084507042253 0.6228419654714475 1.6514084507042253 True 0.1900545060634613 0.19409698247909546 0.5546875 0.884765625
219 input_layernorm 12 20 1.5294117647058822 0.6046511627906976 1.5294117647058822 True 0.1900545060634613 0.08625888079404831 0.5546875 1.0262451171875
220 input_layernorm 12 21 1.7876712328767124 0.6412776412776413 1.7876712328767124 True 0.1900545060634613 0.07095508277416229 0.5546875 1.043212890625
221 input_layernorm 12 22 1.6267123287671232 0.6192959582790091 1.6267123287671232 True 0.1900545060634613 0.07214703410863876 0.5546875 1.044921875
222 input_layernorm 12 23 2.2054794520547945 0.688034188034188 2.2054794520547945 True 0.1900545060634613 0.15369786322116852 0.5546875 0.98577880859375
223 input_layernorm 13 14 0.3580163043478261 0.263631815907954 0.3580163043478261 True 0.16846242547035217 0.11825777590274811 0.5703125 0.6875
224 input_layernorm 13 15 1.0890410958904109 0.521311475409836 1.0890410958904109 True 0.16846242547035217 0.06029079854488373 0.5703125 0.779296875
225 input_layernorm 13 16 0.6413043478260869 0.39072847682119205 0.6413043478260869 True 0.16846242547035217 0.09195870161056519 0.5703125 0.7763671875
226 input_layernorm 13 17 0.9434931506849316 0.48546255506607927 0.9434931506849316 True 0.16846242547035217 0.06483674794435501 0.5703125 1.0012359619140625
227 input_layernorm 13 18 1.2773972602739727 0.5609022556390978 1.2773972602739727 True 0.16846242547035217 0.06785038113594055 0.5703125 1.1640625
228 input_layernorm 13 19 1.5787671232876712 0.6122177954847278 1.5787671232876712 True 0.16846242547035217 0.19409698247909546 0.5703125 0.884765625
229 input_layernorm 13 20 1.457142857142857 0.5930232558139535 1.457142857142857 True 0.16846242547035217 0.08625888079404831 0.5703125 1.0262451171875
230 input_layernorm 13 21 1.759322033898305 0.6375921375921376 1.759322033898305 True 0.16846242547035217 0.07095508277416229 0.5703125 1.043212890625
231 input_layernorm 13 22 1.6 0.6153846153846154 1.6 True 0.16846242547035217 0.07214703410863876 0.5703125 1.044921875
232 input_layernorm 13 23 2.1728813559322036 0.6848290598290598 2.1728813559322036 True 0.16846242547035217 0.15369786322116852 0.5703125 0.98577880859375
233 input_layernorm 14 15 0.6661931818181818 0.667953667953668 0.667953667953668 True 0.11825777590274811 0.06029079854488373 0.6875 0.779296875
234 input_layernorm 14 16 0.4027630180658874 0.2871212121212121 0.4027630180658874 True 0.11825777590274811 0.09195870161056519 0.6875 0.7763671875
235 input_layernorm 14 17 0.640807651434644 0.3905440414507772 0.640807651434644 True 0.11825777590274811 0.06483674794435501 0.6875 1.0012359619140625
236 input_layernorm 14 18 0.8575982996811902 0.46167048054919907 0.8575982996811902 True 0.11825777590274811 0.06785038113594055 0.6875 1.1640625
237 input_layernorm 14 19 1.1590909090909092 0.5774647887323944 1.1590909090909092 True 0.11825777590274811 0.19409698247909546 0.6875 0.884765625
238 input_layernorm 14 20 1.107843137254902 0.5255813953488372 1.107843137254902 True 0.11825777590274811 0.08625888079404831 0.6875 1.0262451171875
239 input_layernorm 14 21 1.2613636363636365 0.5577889447236181 1.2613636363636365 True 0.11825777590274811 0.07095508277416229 0.6875 1.043212890625
240 input_layernorm 14 22 1.0710227272727273 0.5171467764060357 1.0710227272727273 True 0.11825777590274811 0.07214703410863876 0.6875 1.044921875
241 input_layernorm 14 23 1.5056818181818181 0.6009070294784581 1.5056818181818181 True 0.11825777590274811 0.15369786322116852 0.6875 0.98577880859375
242 input_layernorm 15 16 0.8797573083287369 0.4993757802746567 0.8797573083287369 True 0.06029079854488373 0.09195870161056519 0.779296875 0.7763671875
243 input_layernorm 15 17 0.732297852900868 0.42273206751054854 0.732297852900868 True 0.06029079854488373 0.06483674794435501 0.779296875 1.0012359619140625
244 input_layernorm 15 18 0.8297872340425532 0.45348837209302323 0.8297872340425532 True 0.06029079854488373 0.06785038113594055 0.779296875 1.1640625
245 input_layernorm 15 19 1.0300751879699248 0.5074074074074074 1.0300751879699248 True 0.06029079854488373 0.19409698247909546 0.779296875 0.884765625
246 input_layernorm 15 20 1.0451127819548873 0.5110294117647058 1.0451127819548873 True 0.06029079854488373 0.08625888079404831 0.779296875 1.0262451171875
247 input_layernorm 15 21 1.150375939849624 0.534965034965035 1.150375939849624 True 0.06029079854488373 0.07095508277416229 0.779296875 1.043212890625
248 input_layernorm 15 22 1.005012531328321 0.50125 1.005012531328321 True 0.06029079854488373 0.07214703410863876 0.779296875 1.044921875
249 input_layernorm 15 23 1.100250626566416 0.5238663484486874 1.100250626566416 True 0.06029079854488373 0.15369786322116852 0.779296875 0.98577880859375
250 input_layernorm 16 17 0.3909313725490196 0.28105726872246695 0.3909313725490196 True 0.09195870161056519 0.06483674794435501 0.7763671875 1.0012359619140625
251 input_layernorm 16 18 0.6299019607843137 0.38646616541353385 0.6299019607843137 True 0.09195870161056519 0.06785038113594055 0.7763671875 1.1640625
252 input_layernorm 16 19 0.8952618453865336 0.7777777777777778 0.8952618453865336 True 0.09195870161056519 0.19409698247909546 0.7763671875 0.884765625
253 input_layernorm 16 20 0.7578027465667915 0.43110795454545453 0.7578027465667915 True 0.09195870161056519 0.08625888079404831 0.7763671875 1.0262451171875
254 input_layernorm 16 21 1.0324594257178528 0.507985257985258 1.0324594257178528 True 0.09195870161056519 0.07095508277416229 0.7763671875 1.043212890625
255 input_layernorm 16 22 0.9151061173533084 0.47783572359843546 0.9151061173533084 True 0.09195870161056519 0.07214703410863876 0.7763671875 1.044921875
256 input_layernorm 16 23 1.3370786516853932 0.5721153846153846 1.3370786516853932 True 0.09195870161056519 0.15369786322116852 0.7763671875 0.98577880859375
257 input_layernorm 17 18 0.20625 0.17098445595854922 0.20625 True 0.06483674794435501 0.06785038113594055 1.0012359619140625 1.1640625
258 input_layernorm 17 19 0.4493804959080726 0.6191966614501826 0.6191966614501826 True 0.06483674794435501 0.19409698247909546 1.0012359619140625 0.884765625
259 input_layernorm 17 20 0.35731807031888796 0.2632530120481928 0.35731807031888796 True 0.06483674794435501 0.08625888079404831 1.0012359619140625 1.0262451171875
260 input_layernorm 17 21 0.5803900259614804 0.367244804400978 0.5803900259614804 True 0.06483674794435501 0.07095508277416229 1.0012359619140625 1.043212890625
261 input_layernorm 17 22 0.4475112054729889 0.30915906127770537 0.4475112054729889 True 0.06483674794435501 0.07214703410863876 1.0012359619140625 1.044921875
262 input_layernorm 17 23 0.7790511605224256 0.45824255628177196 0.7790511605224256 True 0.06483674794435501 0.15369786322116852 1.0012359619140625 0.98577880859375
263 input_layernorm 18 19 0.40243142144638405 0.6734480959833072 0.6734480959833072 True 0.06785038113594055 0.19409698247909546 1.1640625 0.884765625
264 input_layernorm 18 20 0.17917304747320062 0.19578313253012047 0.19578313253012047 True 0.06785038113594055 0.08625888079404831 1.1640625 1.0262451171875
265 input_layernorm 18 21 0.3624161073825503 0.2660098522167488 0.3624161073825503 True 0.06785038113594055 0.07095508277416229 1.1640625 1.043212890625
266 input_layernorm 18 22 0.22378199834847234 0.27217125382262997 0.27217125382262997 True 0.06785038113594055 0.07214703410863876 1.1640625 1.044921875
267 input_layernorm 18 23 0.4929343308395677 0.5599088838268793 0.5599088838268793 True 0.06785038113594055 0.15369786322116852 1.1640625 0.98577880859375
268 input_layernorm 19 20 0.702803274621682 0.4127331002331002 0.702803274621682 True 0.19409698247909546 0.08625888079404831 0.884765625 1.0262451171875
269 input_layernorm 19 21 0.8282009724473258 0.4530141843971631 0.8282009724473258 True 0.19409698247909546 0.07095508277416229 0.884765625 1.043212890625
270 input_layernorm 19 22 0.6790923824959482 0.40444015444015446 0.6790923824959482 True 0.19409698247909546 0.07214703410863876 0.884765625 1.044921875
271 input_layernorm 19 23 0.34451054624121147 0.5234624145785877 0.5234624145785877 True 0.19409698247909546 0.15369786322116852 0.884765625 0.98577880859375
272 input_layernorm 20 21 0.2359208523592085 0.19088669950738915 0.2359208523592085 True 0.08625888079404831 0.07095508277416229 1.0262451171875 1.043212890625
273 input_layernorm 20 22 0.18092909535452323 0.2208955223880597 0.2208955223880597 True 0.08625888079404831 0.07214703410863876 1.0262451171875 1.044921875
274 input_layernorm 20 23 0.3733133433283358 0.5809627347949508 0.5809627347949508 True 0.08625888079404831 0.15369786322116852 1.0262451171875 0.98577880859375
275 input_layernorm 21 22 0.14822546972860126 0.17401960784313725 0.17401960784313725 True 0.07095508277416229 0.07214703410863876 1.043212890625 1.044921875
276 input_layernorm 21 23 0.3882978723404255 0.6347826086956522 0.6347826086956522 True 0.07095508277416229 0.15369786322116852 1.043212890625 0.98577880859375
277 input_layernorm 22 23 0.333976833976834 0.5014492753623189 0.5014492753623189 True 0.07214703410863876 0.15369786322116852 1.044921875 0.98577880859375
278 post_attention_layernorm 0 1 129.5 0.9923371647509579 129.5 True 0.108433797955513 0.09410307556390762 0.00390625 0.45703125
279 post_attention_layernorm 0 2 154.0 0.9935483870967742 154.0 True 0.108433797955513 0.10385182499885559 0.00390625 0.533203125
280 post_attention_layernorm 0 3 165.5 0.993993993993994 165.5 True 0.108433797955513 0.11546041816473007 0.00390625 0.48828125
281 post_attention_layernorm 0 4 181.5 0.9945205479452055 181.5 True 0.108433797955513 0.11874310672283173 0.00390625 0.4765625
282 post_attention_layernorm 0 5 194.25 0.9948783610755442 194.25 True 0.108433797955513 0.14567376673221588 0.00390625 0.421875
283 post_attention_layernorm 0 6 221.0 0.9954954954954955 221.0 True 0.108433797955513 0.13559450209140778 0.00390625 0.43359375
284 post_attention_layernorm 0 7 188.5 0.9947229551451188 188.5 True 0.108433797955513 0.17147497832775116 0.00390625 0.375
285 post_attention_layernorm 0 8 187.0 0.9946808510638298 187.0 True 0.108433797955513 0.18255221843719482 0.00390625 0.33984375
286 post_attention_layernorm 0 9 185.0 0.9946236559139785 185.0 True 0.108433797955513 0.17869991064071655 0.00390625 0.3515625
287 post_attention_layernorm 0 10 215.75 0.9953863898500577 215.75 True 0.108433797955513 0.16611140966415405 0.00390625 0.36328125
288 post_attention_layernorm 0 11 195.75 0.9949174078780177 195.75 True 0.108433797955513 0.1528426557779312 0.00390625 0.41796875
289 post_attention_layernorm 0 12 188.0 0.9947089947089947 188.0 True 0.108433797955513 0.12497646361589432 0.00390625 0.5234375
290 post_attention_layernorm 0 13 199.25 0.9950062421972534 199.25 True 0.108433797955513 0.09632012248039246 0.00390625 0.619140625
291 post_attention_layernorm 0 14 228.875 0.9956498096791735 228.875 True 0.108433797955513 0.053455036133527756 0.00390625 0.87646484375
292 post_attention_layernorm 0 15 197.25 0.9949558638083228 197.25 True 0.108433797955513 0.06801187992095947 0.00390625 0.736328125
293 post_attention_layernorm 0 16 189.5 0.994750656167979 189.5 True 0.108433797955513 0.08680811524391174 0.00390625 0.72265625
294 post_attention_layernorm 0 17 181.5 0.9945205479452055 181.5 True 0.108433797955513 0.09242258220911026 0.00390625 0.712890625
295 post_attention_layernorm 0 18 224.875 0.9955727725511898 224.875 True 0.108433797955513 0.09676424413919449 0.00390625 0.8095703125
296 post_attention_layernorm 0 19 233.5 0.9957356076759062 233.5 True 0.108433797955513 0.07613449543714523 0.00390625 0.81640625
297 post_attention_layernorm 0 20 209.5 0.995249406175772 209.5 True 0.108433797955513 0.07808392494916916 0.00390625 0.822265625
298 post_attention_layernorm 0 21 220.0 0.995475113122172 220.0 True 0.108433797955513 0.06570179015398026 0.00390625 0.86328125
299 post_attention_layernorm 0 22 248.5625 0.995992987728525 248.5625 True 0.108433797955513 0.05012959614396095 0.00390625 0.9713134765625
300 post_attention_layernorm 0 23 258.484375 0.9961462034082014 258.484375 True 0.108433797955513 0.06521295011043549 0.00390625 1.01361083984375
301 post_attention_layernorm 1 2 0.42512908777969016 0.29830917874396135 0.42512908777969016 True 0.09410307556390762 0.10385182499885559 0.45703125 0.533203125
302 post_attention_layernorm 1 3 0.27586206896551724 0.26858457997698504 0.27586206896551724 True 0.09410307556390762 0.11546041816473007 0.45703125 0.48828125
303 post_attention_layernorm 1 4 0.39846743295019155 0.28493150684931506 0.39846743295019155 True 0.09410307556390762 0.11874310672283173 0.45703125 0.4765625
304 post_attention_layernorm 1 5 0.49616858237547895 0.33162612035851474 0.49616858237547895 True 0.09410307556390762 0.14567376673221588 0.45703125 0.421875
305 post_attention_layernorm 1 6 0.7011494252873564 0.41216216216216217 0.7011494252873564 True 0.09410307556390762 0.13559450209140778 0.45703125 0.43359375
306 post_attention_layernorm 1 7 0.4521072796934866 0.45161290322580644 0.4521072796934866 True 0.09410307556390762 0.17147497832775116 0.45703125 0.375
307 post_attention_layernorm 1 8 0.5296052631578947 0.594488188976378 0.594488188976378 True 0.09410307556390762 0.18255221843719482 0.45703125 0.33984375
308 post_attention_layernorm 1 9 0.42528735632183906 0.5133079847908745 0.5133079847908745 True 0.09410307556390762 0.17869991064071655 0.45703125 0.3515625
309 post_attention_layernorm 1 10 0.6609195402298851 0.5073260073260073 0.6609195402298851 True 0.09410307556390762 0.16611140966415405 0.45703125 0.36328125
310 post_attention_layernorm 1 11 0.5076628352490421 0.3785807291666667 0.5076628352490421 True 0.09410307556390762 0.1528426557779312 0.45703125 0.41796875
311 post_attention_layernorm 1 12 0.7397203947368421 0.4274732620320856 0.7397203947368421 True 0.09410307556390762 0.12497646361589432 0.45703125 0.5234375
312 post_attention_layernorm 1 13 0.8725328947368421 0.46596398770311814 0.8725328947368421 True 0.09410307556390762 0.09632012248039246 0.45703125 0.619140625
313 post_attention_layernorm 1 14 1.08739406779661 0.5209337731540218 1.08739406779661 True 0.09410307556390762 0.053455036133527756 0.45703125 0.87646484375
314 post_attention_layernorm 1 15 1.7415254237288136 0.6352395672333848 1.7415254237288136 True 0.09410307556390762 0.06801187992095947 0.45703125 0.736328125
315 post_attention_layernorm 1 16 0.9901315789473685 0.4975206611570248 0.9901315789473685 True 0.09410307556390762 0.08680811524391174 0.45703125 0.72265625
316 post_attention_layernorm 1 17 1.009733606557377 0.5024216161101198 1.009733606557377 True 0.09410307556390762 0.09242258220911026 0.45703125 0.712890625
317 post_attention_layernorm 1 18 1.194078947368421 0.5442278860569715 1.194078947368421 True 0.09410307556390762 0.09676424413919449 0.45703125 0.8095703125
318 post_attention_layernorm 1 19 1.5277777777777777 0.6043956043956044 1.5277777777777777 True 0.09410307556390762 0.07613449543714523 0.45703125 0.81640625
319 post_attention_layernorm 1 20 1.3644067796610169 0.5770609318996416 1.3644067796610169 True 0.09410307556390762 0.07808392494916916 0.45703125 0.822265625
320 post_attention_layernorm 1 21 1.7166666666666666 0.6319018404907976 1.7166666666666666 True 0.09410307556390762 0.06570179015398026 0.45703125 0.86328125
321 post_attention_layernorm 1 22 2.377049180327869 0.7038834951456311 2.377049180327869 True 0.09410307556390762 0.05012959614396095 0.45703125 0.9713134765625
322 post_attention_layernorm 1 23 2.6324786324786325 0.7247058823529412 2.6324786324786325 True 0.09410307556390762 0.06521295011043549 0.45703125 1.01361083984375
323 post_attention_layernorm 2 3 0.3058252427184466 0.4405594405594406 0.4405594405594406 True 0.10385182499885559 0.11546041816473007 0.533203125 0.48828125
324 post_attention_layernorm 2 4 0.3094660194174757 0.44815465729349735 0.44815465729349735 True 0.10385182499885559 0.11874310672283173 0.533203125 0.4765625
325 post_attention_layernorm 2 5 0.3109903381642512 0.45135845749342685 0.45135845749342685 True 0.10385182499885559 0.14567376673221588 0.533203125 0.421875
326 post_attention_layernorm 2 6 0.432258064516129 0.5354659248956884 0.5354659248956884 True 0.10385182499885559 0.13559450209140778 0.533203125 0.43359375
327 post_attention_layernorm 2 7 0.3689613526570048 0.584688995215311 0.584688995215311 True 0.10385182499885559 0.17147497832775116 0.533203125 0.375
328 post_attention_layernorm 2 8 0.42462600690448793 0.738 0.738 True 0.10385182499885559 0.18255221843719482 0.533203125 0.33984375
329 post_attention_layernorm 2 9 0.42002301495972383 0.7242063492063492 0.7242063492063492 True 0.10385182499885559 0.17869991064071655 0.533203125 0.3515625
330 post_attention_layernorm 2 10 0.414368739953011 0.7075591216216216 0.7075591216216216 True 0.10385182499885559 0.16611140966415405 0.533203125 0.36328125
331 post_attention_layernorm 2 11 0.37333937198067635 0.5957600578173934 0.5957600578173934 True 0.10385182499885559 0.1528426557779312 0.533203125 0.41796875
332 post_attention_layernorm 2 12 0.46098066298342544 0.5644780349551252 0.5644780349551252 True 0.10385182499885559 0.12497646361589432 0.533203125 0.5234375
333 post_attention_layernorm 2 13 0.5725138121546961 0.52416014726185 0.5725138121546961 True 0.10385182499885559 0.09632012248039246 0.533203125 0.619140625
334 post_attention_layernorm 2 14 0.8044871794871795 0.4500659920809503 0.8044871794871795 True 0.10385182499885559 0.053455036133527756 0.533203125 0.87646484375
335 post_attention_layernorm 2 15 1.36996336996337 0.5780525502318392 1.36996336996337 True 0.10385182499885559 0.06801187992095947 0.533203125 0.736328125
336 post_attention_layernorm 2 16 0.6712707182320442 0.47909888895709285 0.6712707182320442 True 0.10385182499885559 0.08680811524391174 0.533203125 0.72265625
337 post_attention_layernorm 2 17 0.7513489208633094 0.4290115532734275 0.7513489208633094 True 0.10385182499885559 0.09242258220911026 0.533203125 0.712890625
338 post_attention_layernorm 2 18 0.8425414364640884 0.4572713643178411 0.8425414364640884 True 0.10385182499885559 0.09676424413919449 0.533203125 0.8095703125
339 post_attention_layernorm 2 19 1.1666666666666667 0.5384615384615384 1.1666666666666667 True 0.10385182499885559 0.07613449543714523 0.533203125 0.81640625
340 post_attention_layernorm 2 20 1.043956043956044 0.510752688172043 1.043956043956044 True 0.10385182499885559 0.07808392494916916 0.533203125 0.822265625
341 post_attention_layernorm 2 21 1.381294964028777 0.5800604229607251 1.381294964028777 True 0.10385182499885559 0.06570179015398026 0.533203125 0.86328125
342 post_attention_layernorm 2 22 1.9640287769784173 0.662621359223301 1.9640287769784173 True 0.10385182499885559 0.05012959614396095 0.533203125 0.9713134765625
343 post_attention_layernorm 2 23 2.1135531135531136 0.6788235294117647 2.1135531135531136 True 0.10385182499885559 0.06521295011043549 0.533203125 1.01361083984375
344 post_attention_layernorm 3 4 0.1832089552238806 0.22430333485609869 0.22430333485609869 True 0.11546041816473007 0.11874310672283173 0.48828125 0.4765625
345 post_attention_layernorm 3 5 0.1939149560117302 0.2405638926784902 0.2405638926784902 True 0.11546041816473007 0.14567376673221588 0.48828125 0.421875
346 post_attention_layernorm 3 6 0.3333333333333333 0.25 0.3333333333333333 True 0.11546041816473007 0.13559450209140778 0.48828125 0.43359375
347 post_attention_layernorm 3 7 0.2893772893772894 0.4072164948453608 0.4072164948453608 True 0.11546041816473007 0.17147497832775116 0.48828125 0.375
348 post_attention_layernorm 3 8 0.34798534798534797 0.5337078651685393 0.5337078651685393 True 0.11546041816473007 0.18255221843719482 0.48828125 0.33984375
349 post_attention_layernorm 3 9 0.34065934065934067 0.5166666666666667 0.5166666666666667 True 0.11546041816473007 0.17869991064071655 0.48828125 0.3515625
350 post_attention_layernorm 3 10 0.3333333333333333 0.5 0.5 True 0.11546041816473007 0.16611140966415405 0.48828125 0.36328125
351 post_attention_layernorm 3 11 0.2514124293785311 0.28756009615384615 0.28756009615384615 True 0.11546041816473007 0.1528426557779312 0.48828125 0.41796875
352 post_attention_layernorm 3 12 0.43326558265582654 0.3787001287001287 0.43326558265582654 True 0.11546041816473007 0.12497646361589432 0.48828125 0.5234375
353 post_attention_layernorm 3 13 0.5426829268292683 0.35177865612648224 0.5426829268292683 True 0.11546041816473007 0.09632012248039246 0.48828125 0.619140625
354 post_attention_layernorm 3 14 0.902027027027027 0.47424511545293074 0.902027027027027 True 0.11546041816473007 0.053455036133527756 0.48828125 0.87646484375
355 post_attention_layernorm 3 15 1.56 0.609375 1.56 True 0.11546041816473007 0.06801187992095947 0.48828125 0.736328125
356 post_attention_layernorm 3 16 0.8092643051771117 0.44728915662650603 0.8092643051771117 True 0.11546041816473007 0.08680811524391174 0.48828125 0.72265625
357 post_attention_layernorm 3 17 0.8933397683397684 0.47183278103492227 0.8933397683397684 True 0.11546041816473007 0.09242258220911026 0.48828125 0.712890625
358 post_attention_layernorm 3 18 0.9215 0.47957325006505336 0.9215 True 0.11546041816473007 0.09676424413919449 0.48828125 0.8095703125
359 post_attention_layernorm 3 19 1.366 0.577345731191885 1.366 True 0.11546041816473007 0.07613449543714523 0.48828125 0.81640625
360 post_attention_layernorm 3 20 1.169 0.5389580451821115 1.169 True 0.11546041816473007 0.07808392494916916 0.48828125 0.822265625
361 post_attention_layernorm 3 21 1.5396825396825398 0.60625 1.5396825396825398 True 0.11546041816473007 0.06570179015398026 0.48828125 0.86328125
362 post_attention_layernorm 3 22 2.2698412698412698 0.6941747572815534 2.2698412698412698 True 0.11546041816473007 0.05012959614396095 0.48828125 0.9713134765625
363 post_attention_layernorm 3 23 2.4 0.7058823529411765 2.4 True 0.11546041816473007 0.06521295011043549 0.48828125 1.01361083984375
364 post_attention_layernorm 4 5 0.16475095785440613 0.19724770642201836 0.19724770642201836 True 0.11874310672283173 0.14567376673221588 0.4765625 0.421875
365 post_attention_layernorm 4 6 0.2548342541436464 0.2030820033021464 0.2548342541436464 True 0.11874310672283173 0.13559450209140778 0.4765625 0.43359375
366 post_attention_layernorm 4 7 0.2785923753665689 0.3861788617886179 0.3861788617886179 True 0.11874310672283173 0.17147497832775116 0.4765625 0.375
367 post_attention_layernorm 4 8 0.3253731343283582 0.4823008849557522 0.4823008849557522 True 0.11874310672283173 0.18255221843719482 0.4765625 0.33984375
368 post_attention_layernorm 4 9 0.3128491620111732 0.45528455284552843 0.45528455284552843 True 0.11874310672283173 0.17869991064071655 0.4765625 0.3515625
369 post_attention_layernorm 4 10 0.30434782608695654 0.4375 0.4375 True 0.11874310672283173 0.16611140966415405 0.4765625 0.36328125
370 post_attention_layernorm 4 11 0.2160188457008245 0.27554086538461536 0.27554086538461536 True 0.11874310672283173 0.1528426557779312 0.4765625 0.41796875
371 post_attention_layernorm 4 12 0.3737012987012987 0.3658301158301158 0.3737012987012987 True 0.11874310672283173 0.12497646361589432 0.4765625 0.5234375
372 post_attention_layernorm 4 13 0.4785714285714286 0.32367149758454106 0.4785714285714286 True 0.11874310672283173 0.09632012248039246 0.4765625 0.619140625
373 post_attention_layernorm 4 14 0.9615384615384616 0.49019607843137253 0.9615384615384616 True 0.11874310672283173 0.053455036133527756 0.4765625 0.87646484375
374 post_attention_layernorm 4 15 1.6598360655737705 0.6240369799691834 1.6598360655737705 True 0.11874310672283173 0.06801187992095947 0.4765625 0.736328125
375 post_attention_layernorm 4 16 0.7994579945799458 0.4442771084337349 0.7994579945799458 True 0.11874310672283173 0.08680811524391174 0.4765625 0.72265625
376 post_attention_layernorm 4 17 0.9569672131147541 0.48900523560209425 0.9569672131147541 True 0.11874310672283173 0.09242258220911026 0.4765625 0.712890625
377 post_attention_layernorm 4 18 1.0363729508196722 0.5089308176100629 1.0363729508196722 True 0.11874310672283173 0.09676424413919449 0.4765625 0.8095703125
378 post_attention_layernorm 4 19 1.4246031746031746 0.5875613747954174 1.4246031746031746 True 0.11874310672283173 0.07613449543714523 0.4765625 0.81640625
379 post_attention_layernorm 4 20 1.2930327868852458 0.5638963360142984 1.2930327868852458 True 0.11874310672283173 0.07808392494916916 0.4765625 0.822265625
380 post_attention_layernorm 4 21 1.5806451612903225 0.6125 1.5806451612903225 True 0.11874310672283173 0.06570179015398026 0.4765625 0.86328125
381 post_attention_layernorm 4 22 2.3225806451612905 0.6990291262135923 2.3225806451612905 True 0.11874310672283173 0.05012959614396095 0.4765625 0.9713134765625
382 post_attention_layernorm 4 23 2.373015873015873 0.7035294117647058 2.373015873015873 True 0.11874310672283173 0.06521295011043549 0.4765625 1.01361083984375
383 post_attention_layernorm 5 6 0.32515337423312884 0.24537037037037038 0.32515337423312884 True 0.14567376673221588 0.13559450209140778 0.421875 0.43359375
384 post_attention_layernorm 5 7 0.1608832807570978 0.19172932330827067 0.19172932330827067 True 0.14567376673221588 0.17147497832775116 0.421875 0.375
385 post_attention_layernorm 5 8 0.2413793103448276 0.3181818181818182 0.3181818181818182 True 0.14567376673221588 0.18255221843719482 0.421875 0.33984375
386 post_attention_layernorm 5 9 0.24968314321926488 0.3327702702702703 0.3327702702702703 True 0.14567376673221588 0.17869991064071655 0.421875 0.3515625
387 post_attention_layernorm 5 10 0.2541436464088398 0.30067567567567566 0.30067567567567566 True 0.14567376673221588 0.16611140966415405 0.421875 0.36328125
388 post_attention_layernorm 5 11 0.214 0.23786808894230768 0.23786808894230768 True 0.14567376673221588 0.1528426557779312 0.421875 0.41796875
389 post_attention_layernorm 5 12 0.4103333333333333 0.3254906692406692 0.4103333333333333 True 0.14567376673221588 0.12497646361589432 0.421875 0.5234375
390 post_attention_layernorm 5 13 0.555045871559633 0.35693215339233036 0.555045871559633 True 0.14567376673221588 0.09632012248039246 0.421875 0.619140625
391 post_attention_layernorm 5 14 1.275092936802974 0.5604575163398693 1.275092936802974 True 0.14567376673221588 0.053455036133527756 0.421875 0.87646484375
392 post_attention_layernorm 5 15 1.962962962962963 0.6625 1.962962962962963 True 0.14567376673221588 0.06801187992095947 0.421875 0.736328125
393 post_attention_layernorm 5 16 1.063197026022305 0.5153153153153153 1.063197026022305 True 0.14567376673221588 0.08680811524391174 0.421875 0.72265625
394 post_attention_layernorm 5 17 1.2333715596330275 0.5522464698331194 1.2333715596330275 True 0.14567376673221588 0.09242258220911026 0.421875 0.712890625
395 post_attention_layernorm 5 18 1.2573327850877194 0.5569993017395792 1.2573327850877194 True 0.14567376673221588 0.09676424413919449 0.421875 0.8095703125
396 post_attention_layernorm 5 19 1.738532110091743 0.6348408710217756 1.738532110091743 True 0.14567376673221588 0.07613449543714523 0.421875 0.81640625
397 post_attention_layernorm 5 20 1.5136363636363637 0.6021699819168174 1.5136363636363637 True 0.14567376673221588 0.07808392494916916 0.421875 0.822265625
398 post_attention_layernorm 5 21 2.036697247706422 0.6706948640483383 2.036697247706422 True 0.14567376673221588 0.06570179015398026 0.421875 0.86328125
399 post_attention_layernorm 5 22 2.779816513761468 0.7354368932038835 2.779816513761468 True 0.14567376673221588 0.05012959614396095 0.421875 0.9713134765625
400 post_attention_layernorm 5 23 2.935185185185185 0.7458823529411764 2.935185185185185 True 0.14567376673221588 0.06521295011043549 0.421875 1.01361083984375
401 post_attention_layernorm 6 7 0.3519163763066202 0.543010752688172 0.543010752688172 True 0.13559450209140778 0.17147497832775116 0.43359375 0.375
402 post_attention_layernorm 6 8 0.4122905027932961 0.7015209125475285 0.7015209125475285 True 0.13559450209140778 0.18255221843719482 0.43359375 0.33984375
403 post_attention_layernorm 6 9 0.35195530726256985 0.5431034482758621 0.5431034482758621 True 0.13559450209140778 0.17869991064071655 0.43359375 0.3515625
404 post_attention_layernorm 6 10 0.3743016759776536 0.5982142857142857 0.5982142857142857 True 0.13559450209140778 0.16611140966415405 0.43359375 0.36328125
405 post_attention_layernorm 6 11 0.2961672473867596 0.4207920792079208 0.4207920792079208 True 0.13559450209140778 0.1528426557779312 0.43359375 0.41796875
406 post_attention_layernorm 6 12 0.3181818181818182 0.29812834224598933 0.3181818181818182 True 0.13559450209140778 0.12497646361589432 0.43359375 0.5234375
407 post_attention_layernorm 6 13 0.5526315789473685 0.3559322033898305 0.5526315789473685 True 0.13559450209140778 0.09632012248039246 0.43359375 0.619140625
408 post_attention_layernorm 6 14 1.1714801444043321 0.5394846217788861 1.1714801444043321 True 0.13559450209140778 0.053455036133527756 0.43359375 0.87646484375
409 post_attention_layernorm 6 15 1.927927927927928 0.6584615384615384 1.927927927927928 True 0.13559450209140778 0.06801187992095947 0.43359375 0.736328125
410 post_attention_layernorm 6 16 0.9859913793103449 0.4964731416169289 0.9859913793103449 True 0.13559450209140778 0.08680811524391174 0.43359375 0.72265625
411 post_attention_layernorm 6 17 1.2088963963963963 0.5472852408870762 1.2088963963963963 True 0.13559450209140778 0.09242258220911026 0.43359375 0.712890625
412 post_attention_layernorm 6 18 1.2401463963963963 0.5536006032424281 1.2401463963963963 True 0.13559450209140778 0.09676424413919449 0.43359375 0.8095703125
413 post_attention_layernorm 6 19 1.6644144144144144 0.6246830092983939 1.6644144144144144 True 0.13559450209140778 0.07613449543714523 0.43359375 0.81640625
414 post_attention_layernorm 6 20 1.5202702702702702 0.6032171581769437 1.5202702702702702 True 0.13559450209140778 0.07808392494916916 0.43359375 0.822265625
415 post_attention_layernorm 6 21 1.8782608695652174 0.6525679758308157 1.8782608695652174 True 0.13559450209140778 0.06570179015398026 0.43359375 0.86328125
416 post_attention_layernorm 6 22 2.6140350877192984 0.7233009708737864 2.6140350877192984 True 0.13559450209140778 0.05012959614396095 0.43359375 0.9713134765625
417 post_attention_layernorm 6 23 2.828828828828829 0.7388235294117647 2.828828828828829 True 0.13559450209140778 0.06521295011043549 0.43359375 1.01361083984375
418 post_attention_layernorm 7 8 0.19383825417201542 0.16236559139784945 0.19383825417201542 True 0.17147497832775116 0.18255221843719482 0.375 0.33984375
419 post_attention_layernorm 7 9 0.12010443864229765 0.13649851632047477 0.13649851632047477 True 0.17147497832775116 0.17869991064071655 0.375 0.3515625
420 post_attention_layernorm 7 10 0.15468549422336328 0.1339633129516398 0.15468549422336328 True 0.17147497832775116 0.16611140966415405 0.375 0.36328125
421 post_attention_layernorm 7 11 0.23232323232323232 0.1885245901639344 0.23232323232323232 True 0.17147497832775116 0.1528426557779312 0.375 0.41796875
422 post_attention_layernorm 7 12 0.5047169811320755 0.335423197492163 0.5047169811320755 True 0.17147497832775116 0.12497646361589432 0.375 0.5234375
423 post_attention_layernorm 7 13 0.7424242424242424 0.4260869565217391 0.7424242424242424 True 0.17147497832775116 0.09632012248039246 0.375 0.619140625
424 post_attention_layernorm 7 14 1.5932203389830508 0.6143790849673203 1.5932203389830508 True 0.17147497832775116 0.053455036133527756 0.375 0.87646484375
425 post_attention_layernorm 7 15 2.3645833333333335 0.7027863777089783 2.3645833333333335 True 0.17147497832775116 0.06801187992095947 0.375 0.736328125
426 post_attention_layernorm 7 16 1.3799283154121864 0.5798192771084337 1.3799283154121864 True 0.17147497832775116 0.08680811524391174 0.375 0.72265625
427 post_attention_layernorm 7 17 1.5019132653061225 0.6003058883507519 1.5019132653061225 True 0.17147497832775116 0.09242258220911026 0.375 0.712890625
428 post_attention_layernorm 7 18 1.579954954954955 0.6123963334788302 1.579954954954955 True 0.17147497832775116 0.09676424413919449 0.375 0.8095703125
429 post_attention_layernorm 7 19 2.085858585858586 0.6759410801963993 2.085858585858586 True 0.17147497832775116 0.07613449543714523 0.375 0.81640625
430 post_attention_layernorm 7 20 1.884020618556701 0.6532618409294012 1.884020618556701 True 0.17147497832775116 0.07808392494916916 0.375 0.822265625
431 post_attention_layernorm 7 21 2.377551020408163 0.7039274924471299 2.377551020408163 True 0.17147497832775116 0.06570179015398026 0.375 0.86328125
432 post_attention_layernorm 7 22 3.2916666666666665 0.7669902912621359 3.2916666666666665 True 0.17147497832775116 0.05012959614396095 0.375 0.9713134765625
433 post_attention_layernorm 7 23 3.381443298969072 0.7717647058823529 3.381443298969072 True 0.17147497832775116 0.06521295011043549 0.375 1.01361083984375
434 post_attention_layernorm 8 9 0.1510791366906475 0.13125 0.1510791366906475 True 0.18255221843719482 0.17869991064071655 0.33984375 0.3515625
435 post_attention_layernorm 8 10 0.17464424320827943 0.1486784140969163 0.17464424320827943 True 0.18255221843719482 0.16611140966415405 0.33984375 0.36328125
436 post_attention_layernorm 8 11 0.28421052631578947 0.22131147540983606 0.28421052631578947 True 0.18255221843719482 0.1528426557779312 0.33984375 0.41796875
437 post_attention_layernorm 8 12 0.6377551020408163 0.3894080996884735 0.6377551020408163 True 0.18255221843719482 0.12497646361589432 0.33984375 0.5234375
438 post_attention_layernorm 8 13 0.8956043956043956 0.47246376811594204 0.8956043956043956 True 0.18255221843719482 0.09632012248039246 0.33984375 0.619140625
439 post_attention_layernorm 8 14 1.7567567567567568 0.6372549019607843 1.7567567567567568 True 0.18255221843719482 0.053455036133527756 0.33984375 0.87646484375
440 post_attention_layernorm 8 15 2.7011494252873565 0.7298136645962733 2.7011494252873565 True 0.18255221843719482 0.06801187992095947 0.33984375 0.736328125
441 post_attention_layernorm 8 16 1.6141732283464567 0.6174698795180723 1.6141732283464567 True 0.18255221843719482 0.08680811524391174 0.33984375 0.72265625
442 post_attention_layernorm 8 17 1.7862215909090908 0.6410910017843487 1.7862215909090908 True 0.18255221843719482 0.09242258220911026 0.33984375 0.712890625
443 post_attention_layernorm 8 18 1.8256392045454546 0.6460977755435465 1.8256392045454546 True 0.18255221843719482 0.09676424413919449 0.33984375 0.8095703125
444 post_attention_layernorm 8 19 2.357142857142857 0.7021276595744681 2.357142857142857 True 0.18255221843719482 0.07613449543714523 0.33984375 0.81640625
445 post_attention_layernorm 8 20 2.143258426966292 0.6818588025022342 2.143258426966292 True 0.18255221843719482 0.07808392494916916 0.33984375 0.822265625
446 post_attention_layernorm 8 21 2.67816091954023 0.728125 2.67816091954023 True 0.18255221843719482 0.06570179015398026 0.33984375 0.86328125
447 post_attention_layernorm 8 22 3.735632183908046 0.7888349514563107 3.735632183908046 True 0.18255221843719482 0.05012959614396095 0.33984375 0.9713134765625
448 post_attention_layernorm 8 23 3.7222222222222223 0.788235294117647 3.7222222222222223 True 0.18255221843719482 0.06521295011043549 0.33984375 1.01361083984375
449 post_attention_layernorm 9 10 0.16532258064516128 0.14186851211072665 0.16532258064516128 True 0.17869991064071655 0.16611140966415405 0.3515625 0.36328125
450 post_attention_layernorm 9 11 0.2708333333333333 0.21311475409836064 0.2708333333333333 True 0.17869991064071655 0.1528426557779312 0.3515625 0.41796875
451 post_attention_layernorm 9 12 0.5735294117647058 0.3644859813084112 0.5735294117647058 True 0.17869991064071655 0.12497646361589432 0.3515625 0.5234375
452 post_attention_layernorm 9 13 0.8166666666666667 0.44954128440366975 0.8166666666666667 True 0.17869991064071655 0.09632012248039246 0.3515625 0.619140625
453 post_attention_layernorm 9 14 1.6645299145299146 0.6246992782678428 1.6645299145299146 True 0.17869991064071655 0.053455036133527756 0.3515625 0.87646484375
454 post_attention_layernorm 9 15 2.6055555555555556 0.7226502311248074 2.6055555555555556 True 0.17869991064071655 0.06801187992095947 0.3515625 0.736328125
455 post_attention_layernorm 9 16 1.4316239316239316 0.5887521968365553 1.4316239316239316 True 0.17869991064071655 0.08680811524391174 0.3515625 0.72265625
456 post_attention_layernorm 9 17 1.6527777777777777 0.6230366492146597 1.6527777777777777 True 0.17869991064071655 0.09242258220911026 0.3515625 0.712890625
457 post_attention_layernorm 9 18 1.7604166666666667 0.6377358490566037 1.7604166666666667 True 0.17869991064071655 0.09676424413919449 0.3515625 0.8095703125
458 post_attention_layernorm 9 19 2.1822916666666665 0.6857610474631751 2.1822916666666665 True 0.17869991064071655 0.07613449543714523 0.3515625 0.81640625
459 post_attention_layernorm 9 20 2.1083333333333334 0.67828418230563 2.1083333333333334 True 0.17869991064071655 0.07808392494916916 0.3515625 0.822265625
460 post_attention_layernorm 9 21 2.4916666666666667 0.7136038186157518 2.4916666666666667 True 0.17869991064071655 0.06570179015398026 0.3515625 0.86328125
461 post_attention_layernorm 9 22 3.382978723404255 0.7718446601941747 3.382978723404255 True 0.17869991064071655 0.05012959614396095 0.3515625 0.9713134765625
462 post_attention_layernorm 9 23 3.473684210526316 0.7764705882352941 3.473684210526316 True 0.17869991064071655 0.06521295011043549 0.3515625 1.01361083984375
463 post_attention_layernorm 10 11 0.23232323232323232 0.1885245901639344 0.23232323232323232 True 0.16611140966415405 0.1528426557779312 0.36328125 0.41796875
464 post_attention_layernorm 10 12 0.510752688172043 0.33807829181494664 0.510752688172043 True 0.16611140966415405 0.12497646361589432 0.36328125 0.5234375
465 post_attention_layernorm 10 13 0.796875 0.4434782608695652 0.796875 True 0.16611140966415405 0.09632012248039246 0.36328125 0.619140625
466 post_attention_layernorm 10 14 1.7079646017699115 0.630718954248366 1.7079646017699115 True 0.16611140966415405 0.053455036133527756 0.36328125 0.87646484375
467 post_attention_layernorm 10 15 2.4946236559139785 0.7138461538461538 2.4946236559139785 True 0.16611140966415405 0.06801187992095947 0.36328125 0.736328125
468 post_attention_layernorm 10 16 1.4557522123893805 0.5927927927927928 1.4557522123893805 True 0.16611140966415405 0.08680811524391174 0.36328125 0.72265625
469 post_attention_layernorm 10 17 1.6364247311827957 0.6206984450675503 1.6364247311827957 True 0.16611140966415405 0.09242258220911026 0.36328125 0.712890625
470 post_attention_layernorm 10 18 1.67372311827957 0.6259896946085208 1.67372311827957 True 0.16611140966415405 0.09676424413919449 0.36328125 0.8095703125
471 post_attention_layernorm 10 19 2.146276595744681 0.6821639898562976 2.146276595744681 True 0.16611140966415405 0.07613449543714523 0.36328125 0.81640625
472 post_attention_layernorm 10 20 1.976063829787234 0.6639857015192135 1.976063829787234 True 0.16611140966415405 0.07808392494916916 0.36328125 0.822265625
473 post_attention_layernorm 10 21 2.4123711340206184 0.7069486404833837 2.4123711340206184 True 0.16611140966415405 0.06570179015398026 0.36328125 0.86328125
474 post_attention_layernorm 10 22 3.247422680412371 0.7645631067961165 3.247422680412371 True 0.16611140966415405 0.05012959614396095 0.36328125 0.9713134765625
475 post_attention_layernorm 10 23 3.521276595744681 0.7788235294117647 3.521276595744681 True 0.16611140966415405 0.06521295011043549 0.36328125 1.01361083984375
476 post_attention_layernorm 11 12 0.3375 0.2523364485981308 0.3375 True 0.1528426557779312 0.12497646361589432 0.41796875 0.5234375
477 post_attention_layernorm 11 13 0.5131578947368421 0.3391304347826087 0.5131578947368421 True 0.1528426557779312 0.09632012248039246 0.41796875 0.619140625
478 post_attention_layernorm 11 14 1.2612781954887218 0.5577722360764755 1.2612781954887218 True 0.1528426557779312 0.053455036133527756 0.41796875 0.87646484375
479 post_attention_layernorm 11 15 1.9906542056074767 0.665625 1.9906542056074767 True 0.1528426557779312 0.06801187992095947 0.41796875 0.736328125
480 post_attention_layernorm 11 16 1.1914191419141915 0.5436746987951807 1.1914191419141915 True 0.1528426557779312 0.08680811524391174 0.41796875 0.72265625
481 post_attention_layernorm 11 17 1.249426605504587 0.5554422635737956 1.249426605504587 True 0.1528426557779312 0.09242258220911026 0.41796875 0.712890625
482 post_attention_layernorm 11 18 1.28125 0.5616438356164384 1.28125 True 0.1528426557779312 0.09676424413919449 0.41796875 0.8095703125
483 post_attention_layernorm 11 19 1.764018691588785 0.6382079459002535 1.764018691588785 True 0.1528426557779312 0.07613449543714523 0.41796875 0.81640625
484 post_attention_layernorm 11 20 1.5344036697247707 0.6054298642533936 1.5344036697247707 True 0.1528426557779312 0.07808392494916916 0.41796875 0.822265625
485 post_attention_layernorm 11 21 1.9369158878504673 0.6595067621320605 1.9369158878504673 True 0.1528426557779312 0.06570179015398026 0.41796875 0.86328125
486 post_attention_layernorm 11 22 2.7454545454545456 0.7330097087378641 2.7454545454545456 True 0.1528426557779312 0.05012959614396095 0.41796875 0.9713134765625
487 post_attention_layernorm 11 23 2.97196261682243 0.7482352941176471 2.97196261682243 True 0.1528426557779312 0.06521295011043549 0.41796875 1.01361083984375
488 post_attention_layernorm 12 13 0.26492537313432835 0.20943952802359883 0.26492537313432835 True 0.12497646361589432 0.09632012248039246 0.5234375 0.619140625
489 post_attention_layernorm 12 14 0.9213836477987422 0.4795417348608838 0.9213836477987422 True 0.12497646361589432 0.053455036133527756 0.5234375 0.87646484375
490 post_attention_layernorm 12 15 1.3861940298507462 0.5809225957779516 1.3861940298507462 True 0.12497646361589432 0.06801187992095947 0.5234375 0.736328125
491 post_attention_layernorm 12 16 0.9358600583090378 0.48343373493975905 0.9358600583090378 True 0.12497646361589432 0.08680811524391174 0.5234375 0.72265625
492 post_attention_layernorm 12 17 0.9096209912536443 0.4763358778625954 0.9096209912536443 True 0.12497646361589432 0.09242258220911026 0.5234375 0.712890625
493 post_attention_layernorm 12 18 0.9358600583090378 0.48343373493975905 0.9358600583090378 True 0.12497646361589432 0.09676424413919449 0.5234375 0.8095703125
494 post_attention_layernorm 12 19 1.2276119402985075 0.5510887772194305 1.2276119402985075 True 0.12497646361589432 0.07613449543714523 0.5234375 0.81640625
495 post_attention_layernorm 12 20 1.056985294117647 0.5138516532618409 1.056985294117647 True 0.12497646361589432 0.07808392494916916 0.5234375 0.822265625
496 post_attention_layernorm 12 21 1.4701492537313432 0.595166163141994 1.4701492537313432 True 0.12497646361589432 0.06570179015398026 0.5234375 0.86328125
497 post_attention_layernorm 12 22 2.0294117647058822 0.6699029126213593 2.0294117647058822 True 0.12497646361589432 0.05012959614396095 0.5234375 0.9713134765625
498 post_attention_layernorm 12 23 2.1481481481481484 0.6823529411764706 2.1481481481481484 True 0.12497646361589432 0.06521295011043549 0.5234375 1.01361083984375
499 post_attention_layernorm 13 14 0.6276595744680851 0.38562091503267976 0.6276595744680851 True 0.09632012248039246 0.053455036133527756 0.619140625 0.87646484375
500 post_attention_layernorm 13 15 1.0504731861198737 0.5123076923076924 1.0504731861198737 True 0.09632012248039246 0.06801187992095947 0.619140625 0.736328125
501 post_attention_layernorm 13 16 0.5697399527186762 0.3629518072289157 0.5697399527186762 True 0.09632012248039246 0.08680811524391174 0.619140625 0.72265625
502 post_attention_layernorm 13 17 0.5619930475086906 0.35979228486646886 0.5619930475086906 True 0.09632012248039246 0.09242258220911026 0.619140625 0.712890625
503 post_attention_layernorm 13 18 0.657010428736964 0.3965034965034965 0.657010428736964 True 0.09632012248039246 0.09676424413919449 0.619140625 0.8095703125
504 post_attention_layernorm 13 19 0.8659305993690851 0.4640743871513102 0.8659305993690851 True 0.09632012248039246 0.07613449543714523 0.619140625 0.81640625
505 post_attention_layernorm 13 20 0.7811594202898551 0.4385679414157852 0.7811594202898551 True 0.09632012248039246 0.07808392494916916 0.619140625 0.822265625
506 post_attention_layernorm 13 21 0.9826498422712934 0.49562450278440734 0.9826498422712934 True 0.09632012248039246 0.06570179015398026 0.619140625 0.86328125
507 post_attention_layernorm 13 22 1.5432098765432098 0.6067961165048543 1.5432098765432098 True 0.09632012248039246 0.05012959614396095 0.619140625 0.9713134765625
508 post_attention_layernorm 13 23 1.6813880126182965 0.6270588235294118 1.6813880126182965 True 0.09632012248039246 0.06521295011043549 0.619140625 1.01361083984375
509 post_attention_layernorm 14 15 0.4135667396061269 0.29256965944272445 0.4135667396061269 True 0.053455036133527756 0.06801187992095947 0.87646484375 0.736328125
510 post_attention_layernorm 14 16 0.2551505546751189 0.2418918918918919 0.2551505546751189 True 0.053455036133527756 0.08680811524391174 0.87646484375 0.72265625
511 post_attention_layernorm 14 17 0.28498372849837283 0.2595890410958904 0.28498372849837283 True 0.053455036133527756 0.09242258220911026 0.87646484375 0.712890625
512 post_attention_layernorm 14 18 0.39717282261741904 0.28426892950391647 0.39717282261741904 True 0.053455036133527756 0.09676424413919449 0.87646484375 0.8095703125
513 post_attention_layernorm 14 19 0.3989968080255358 0.28520208604954367 0.3989968080255358 True 0.053455036133527756 0.07613449543714523 0.87646484375 0.81640625
514 post_attention_layernorm 14 20 0.4737802097583219 0.3214727722772277 0.4737802097583219 True 0.053455036133527756 0.07808392494916916 0.87646484375 0.822265625
515 post_attention_layernorm 14 21 0.47515257192676547 0.32210401891252954 0.47515257192676547 True 0.053455036133527756 0.06570179015398026 0.87646484375 0.86328125
516 post_attention_layernorm 14 22 0.7402323125659979 0.4253640776699029 0.7402323125659979 True 0.053455036133527756 0.05012959614396095 0.87646484375 0.9713134765625
517 post_attention_layernorm 14 23 0.8220793140407289 0.4511764705882353 0.8220793140407289 True 0.053455036133527756 0.06521295011043549 0.87646484375 1.01361083984375
518 post_attention_layernorm 15 16 0.3656832298136646 0.576499388004896 0.576499388004896 True 0.06801187992095947 0.08680811524391174 0.736328125 0.72265625
519 post_attention_layernorm 15 17 0.29876160990712075 0.4260485651214128 0.4260485651214128 True 0.06801187992095947 0.09242258220911026 0.736328125 0.712890625
520 post_attention_layernorm 15 18 0.30124223602484473 0.4311111111111111 0.4311111111111111 True 0.06801187992095947 0.09676424413919449 0.736328125 0.8095703125
521 post_attention_layernorm 15 19 0.2840549102428722 0.22121710526315788 0.2840549102428722 True 0.06801187992095947 0.07613449543714523 0.736328125 0.81640625
522 post_attention_layernorm 15 20 0.33996683250414594 0.2537128712871287 0.33996683250414594 True 0.06801187992095947 0.07808392494916916 0.736328125 0.822265625
523 post_attention_layernorm 15 21 0.4206008583690987 0.29607250755287007 0.4206008583690987 True 0.06801187992095947 0.06570179015398026 0.736328125 0.86328125
524 post_attention_layernorm 15 22 0.45278969957081544 0.31166912850812406 0.45278969957081544 True 0.06801187992095947 0.05012959614396095 0.736328125 0.9713134765625
525 post_attention_layernorm 15 23 0.815378923352038 0.4491508152173913 0.815378923352038 True 0.06801187992095947 0.06521295011043549 0.736328125 1.01361083984375
526 post_attention_layernorm 16 17 0.17602657004830918 0.1496790757381258 0.17602657004830918 True 0.08680811524391174 0.09242258220911026 0.72265625 0.712890625
527 post_attention_layernorm 16 18 0.2154215976331361 0.17724022516354784 0.2154215976331361 True 0.08680811524391174 0.09676424413919449 0.72265625 0.8095703125
528 post_attention_layernorm 16 19 0.4420289855072464 0.3065326633165829 0.4420289855072464 True 0.08680811524391174 0.07613449543714523 0.72265625 0.81640625
529 post_attention_layernorm 16 20 0.33108935128518974 0.24873563218390804 0.33108935128518974 True 0.08680811524391174 0.07808392494916916 0.72265625 0.822265625
530 post_attention_layernorm 16 21 0.5990338164251208 0.37462235649546827 0.5990338164251208 True 0.08680811524391174 0.06570179015398026 0.72265625 0.86328125
531 post_attention_layernorm 16 22 1.0171358629130967 0.504247572815534 1.0171358629130967 True 0.08680811524391174 0.05012959614396095 0.72265625 0.9713134765625
532 post_attention_layernorm 16 23 1.0506634499396863 0.5123529411764706 1.0506634499396863 True 0.08680811524391174 0.06521295011043549 0.72265625 1.01361083984375
533 post_attention_layernorm 17 18 0.23767123287671232 0.19203099059214168 0.23767123287671232 True 0.09242258220911026 0.09676424413919449 0.712890625 0.8095703125
534 post_attention_layernorm 17 19 0.29360306178239476 0.22696534234995774 0.29360306178239476 True 0.09242258220911026 0.07613449543714523 0.712890625 0.81640625
535 post_attention_layernorm 17 20 0.20969441517386722 0.17334494773519163 0.20969441517386722 True 0.09242258220911026 0.07808392494916916 0.712890625 0.822265625
536 post_attention_layernorm 17 21 0.40350877192982454 0.2875 0.40350877192982454 True 0.09242258220911026 0.06570179015398026 0.712890625 0.86328125
537 post_attention_layernorm 17 22 0.8070175438596491 0.44660194174757284 0.8070175438596491 True 0.09242258220911026 0.05012959614396095 0.712890625 0.9713134765625
538 post_attention_layernorm 17 23 0.8589393110989612 0.46205882352941174 0.8589393110989612 True 0.09242258220911026 0.06521295011043549 0.712890625 1.01361083984375
539 post_attention_layernorm 18 19 0.27958115183246074 0.21849427168576105 0.27958115183246074 True 0.09676424413919449 0.07613449543714523 0.8095703125 0.81640625
540 post_attention_layernorm 18 20 0.20833333333333334 0.1724137931034483 0.20833333333333334 True 0.09676424413919449 0.07808392494916916 0.8095703125 0.822265625
541 post_attention_layernorm 18 21 0.4222222222222222 0.296875 0.4222222222222222 True 0.09676424413919449 0.06570179015398026 0.8095703125 0.86328125
542 post_attention_layernorm 18 22 0.8311111111111111 0.4538834951456311 0.8311111111111111 True 0.09676424413919449 0.05012959614396095 0.8095703125 0.9713134765625
543 post_attention_layernorm 18 23 0.8 0.4444444444444444 0.8 True 0.09676424413919449 0.06521295011043549 0.8095703125 1.01361083984375
544 post_attention_layernorm 19 20 0.151033386327504 0.1441947565543071 0.151033386327504 True 0.07613449543714523 0.07808392494916916 0.81640625 0.822265625
545 post_attention_layernorm 19 21 0.1678832116788321 0.14375 0.1678832116788321 True 0.07613449543714523 0.06570179015398026 0.81640625 0.86328125
546 post_attention_layernorm 19 22 0.5036496350364964 0.33495145631067963 0.5036496350364964 True 0.07613449543714523 0.05012959614396095 0.81640625 0.9713134765625
547 post_attention_layernorm 19 23 0.48267526188557613 0.34782608695652173 0.48267526188557613 True 0.07613449543714523 0.06521295011043549 0.81640625 1.01361083984375
548 post_attention_layernorm 20 21 0.2091324200913242 0.1729607250755287 0.2091324200913242 True 0.07808392494916916 0.06570179015398026 0.822265625 0.86328125
549 post_attention_layernorm 20 22 0.5154022988505748 0.3401092233009709 0.5154022988505748 True 0.07808392494916916 0.05012959614396095 0.822265625 0.9713134765625
550 post_attention_layernorm 20 23 0.5675426463808206 0.4219858156028369 0.5675426463808206 True 0.07808392494916916 0.06521295011043549 0.822265625 1.01361083984375
551 post_attention_layernorm 21 22 0.2875 0.22330097087378642 0.2875 True 0.06570179015398026 0.05012959614396095 0.86328125 0.9713134765625
552 post_attention_layernorm 21 23 0.3960546282245827 0.4148936170212766 0.4148936170212766 True 0.06570179015398026 0.06521295011043549 0.86328125 1.01361083984375
553 post_attention_layernorm 22 23 0.3124108416547789 0.34263295553618134 0.34263295553618134 True 0.05012959614396095 0.06521295011043549 0.9713134765625 1.01361083984375
@@ -0,0 +1,29 @@
{"preflight_layer": 0, "seconds": 2.0485799312591553}
{"preflight_layer": 1, "seconds": 4.110389947891235}
{"preflight_layer": 2, "seconds": 5.677527189254761}
{"preflight_layer": 3, "seconds": 6.754502058029175}
{"preflight_layer": 4, "seconds": 8.340956926345825}
{"preflight_layer": 5, "seconds": 10.697716236114502}
{"preflight_layer": 6, "seconds": 12.576658964157104}
{"preflight_layer": 7, "seconds": 14.14732575416565}
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/opt/exp-model-parameter-similarity/functional/preflight.py", line 27, in <module>
if __name__=='__main__':main()
~~~~^^
File "/opt/exp-model-parameter-similarity/functional/preflight.py", line 18, in main
h,_=lm.forward(ids[:1,:8],progress=progress)
~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/exp-model-parameter-similarity/functional/model.py", line 215, in forward
h = h + self.mlp(n,donor)
~~~~~~~~^^^^^^^^^
File "/opt/exp-model-parameter-similarity/functional/model.py", line 149, in mlp
gate = silu(self.linear(h, p+'gate_proj'))
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
File "/opt/exp-model-parameter-similarity/functional/model.py", line 127, in linear
self.guard()
~~~~~~~~~~^^
File "/opt/exp-model-parameter-similarity/functional/model.py", line 108, in guard
raise MemoryError('Sustained host swap IO above 4 MiB/s in three observation windows')
MemoryError: Sustained host swap IO above 4 MiB/s in three observation windows

Some files were not shown because too many files have changed in this diff Show More