50 lines
2.1 KiB
Markdown
50 lines
2.1 KiB
Markdown
# Syntax DAG Transformer
|
|
|
|
An English-language prototype that combines spaCy's tokenizer and learned
|
|
dependency parser with explicit graph rules. The graph interface can support
|
|
other encoders and relation miners as the prototype evolves.
|
|
|
|
```text
|
|
The road flooded because it rained.
|
|
→ Cause: it rained. Effect: The road flooded.
|
|
```
|
|
|
|
Run with Python 3.12 after installing `requirements.txt`.
|
|
|
|
```sh
|
|
python -m syntax_dag --commit 'The road flooded because it rained.'
|
|
python -m unittest -v
|
|
```
|
|
|
|
Omit `--commit` to recover the original token surfaces and whitespace. The
|
|
command prints one final newline.
|
|
|
|
1. `Encoder` creates token nodes with original whitespace and head-to-dependent
|
|
edges. The causal rule adds clause nodes with edges to their member tokens.
|
|
A token can have incoming edges from both a clause and a syntactic head.
|
|
2. The rule mines a candidate `cause → effect` edge with provenance from a
|
|
trailing `because` clause. The candidate stays separate until committed.
|
|
3. `SyntaxDAG.commit` creates a new immutable graph revision and records the
|
|
proposal. A cycle rejects the entire edit. Repeating a commit on the same
|
|
revision returns that revision.
|
|
4. `Decoder` uses committed causal edges to select the clause order and labels.
|
|
Text outside the selected clauses retains its original surface form.
|
|
|
|
The first rule covers simple affirmative past-tense statements with a trailing
|
|
finite causal clause. Extraction is heuristic and depends on parser accuracy.
|
|
Commits are in-memory graph edits. The prototype uses explicit authorization
|
|
via the CLI flag and has no learned rewrite policy.
|
|
|
|
## Model token IDs
|
|
|
|
```sh
|
|
python -m syntax_dag --ids --commit 'The road flooded because it rained.'
|
|
```
|
|
|
|
JSON includes `encoding`, `input_ids`, `output_ids`, and the output `text`.
|
|
These are integer vocabulary IDs from tiktoken's `o200k_base` BPE encoding.
|
|
spaCy tokens still serve as syntax nodes. Their positions are independent of
|
|
BPE positions. A commit preserves the original input IDs, and the decoder
|
|
tokenizes the full rendered text so its output IDs reflect the new boundaries.
|
|
The tokenizer downloads its vocabulary on first use if it is absent from cache.
|