# Syntax DAG Transformer

An English-language prototype that combines spaCy's tokenizer and learned
dependency parser with explicit graph rules. The graph interface can support
other encoders and relation miners as the prototype evolves.

```text
The road flooded because it rained.
  → Cause: it rained. Effect: The road flooded.
```

Run with Python 3.12 after installing `requirements.txt`.

```sh
python -m syntax_dag --commit 'The road flooded because it rained.'
python -m unittest -v
```

Omit `--commit` to recover the original token surfaces and whitespace. The
command prints one final newline.

1. `Encoder` creates token nodes with original whitespace and head-to-dependent
   edges. The causal rule adds clause nodes with edges to their member tokens.
   A token can have incoming edges from both a clause and a syntactic head.
2. The rule mines a candidate `cause → effect` edge with provenance from a
   trailing `because` clause. The candidate stays separate until committed.
3. `SyntaxDAG.commit` creates a new immutable graph revision and records the
   proposal. A cycle rejects the entire edit. Repeating a commit on the same
   revision returns that revision.
4. `Decoder` uses committed causal edges to select the clause order and labels.
   Text outside the selected clauses retains its original surface form.

The first rule covers simple affirmative past-tense statements with a trailing
finite causal clause. Extraction is heuristic and depends on parser accuracy.
Commits are in-memory graph edits. The prototype uses explicit authorization
via the CLI flag and has no learned rewrite policy.

## Model token IDs

```sh
python -m syntax_dag --ids --commit 'The road flooded because it rained.'
```

JSON includes `encoding`, `input_ids`, `output_ids`, and the output `text`.
These are integer vocabulary IDs from tiktoken's `o200k_base` BPE encoding.
spaCy tokens still serve as syntax nodes. Their positions are independent of
BPE positions. A commit preserves the original input IDs, and the decoder
tokenizes the full rendered text so its output IDs reflect the new boundaries.
The tokenizer downloads its vocabulary on first use if it is absent from cache.