Expose GPT-4o tokenizer IDs for graph input and output
Verify syntax DAG / verify (push) Successful in 37s
Verify syntax DAG / verify (pull_request) Successful in 23s

This commit is contained in:
2026-11-04 16:48:22 +08:00
parent 688613878f
commit 53e0163941
4 changed files with 62 additions and 2 deletions
+13
View File
@@ -34,3 +34,16 @@ The first rule covers simple affirmative past-tense statements with a trailing
finite causal clause. Extraction is heuristic and depends on parser accuracy.
Commits are in-memory graph edits. The prototype uses explicit authorization
via the CLI flag and has no learned rewrite policy.
## Model token IDs
```sh
python -m syntax_dag --ids --commit 'The road flooded because it rained.'
```
JSON includes `encoding`, `input_ids`, `output_ids`, and the output `text`.
These are integer vocabulary IDs from tiktoken's `o200k_base` BPE encoding.
spaCy tokens still serve as syntax nodes. Their positions are independent of
BPE positions. A commit preserves the original input IDs, and the decoder
tokenizes the full rendered text so its output IDs reflect the new boundaries.
The tokenizer downloads its vocabulary on first use if it is absent from cache.