Zayden-Jung/syntax-dag-transformer
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Expose model token IDs for syntax DAG input and output #2

Open
codex-bot wants to merge 0 commits from codex/syntax-dag into main
pull from: codex/syntax-dag
merge into: Zayden-Jung:main
Conversation 0 Commits - Files Changed -
codex-bot commented 2026-11-04 08:49:10 +00:00
Collaborator
Copy Link
Copy Source

Adds tiktoken o200k_base IDs for the original input and the decoded output, with a JSON CLI mode. The syntax graph still uses spaCy positions. Tests cover Unicode round trips and retokenization after a causal commit.

Validation passed on ubuntu24.04-x64-lean4.35: https://gitea.entroia.com/Zayden-Jung/syntax-dag-transformer/actions/runs/748

Adds tiktoken o200k_base IDs for the original input and the decoded output, with a JSON CLI mode. The syntax graph still uses spaCy positions. Tests cover Unicode round trips and retokenization after a causal commit. Validation passed on ubuntu24.04-x64-lean4.35: https://gitea.entroia.com/Zayden-Jung/syntax-dag-transformer/actions/runs/748
codex-bot added 1 commit 2026-11-04 08:49:11 +00:00
Expose GPT-4o tokenizer IDs for graph input and output
Verify syntax DAG / verify (push) Successful in 37s
Details
Verify syntax DAG / verify (pull_request) Successful in 23s
Details
53e0163941
Some checks are pending
Verify syntax DAG / verify (push) Successful in 37s
Details
Verify syntax DAG / verify (pull_request) Successful in 23s
Details
This branch is already included in the target branch. There is nothing to merge.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin codex/syntax-dag:codex/syntax-dag
git checkout codex/syntax-dag
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
codex-bot Zayden-Jung
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Zayden-Jung/syntax-dag-transformer#2
Delete Branch "codex/syntax-dag"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?