Expose GPT-4o tokenizer IDs for graph input and output
This commit is contained in:
@@ -34,3 +34,16 @@ The first rule covers simple affirmative past-tense statements with a trailing
|
||||
finite causal clause. Extraction is heuristic and depends on parser accuracy.
|
||||
Commits are in-memory graph edits. The prototype uses explicit authorization
|
||||
via the CLI flag and has no learned rewrite policy.
|
||||
|
||||
## Model token IDs
|
||||
|
||||
```sh
|
||||
python -m syntax_dag --ids --commit 'The road flooded because it rained.'
|
||||
```
|
||||
|
||||
JSON includes `encoding`, `input_ids`, `output_ids`, and the output `text`.
|
||||
These are integer vocabulary IDs from tiktoken's `o200k_base` BPE encoding.
|
||||
spaCy tokens still serve as syntax nodes. Their positions are independent of
|
||||
BPE positions. A commit preserves the original input IDs, and the decoder
|
||||
tokenizes the full rendered text so its output IDs reflect the new boundaries.
|
||||
The tokenizer downloads its vocabulary on first use if it is absent from cache.
|
||||
|
||||
Reference in New Issue
Block a user