This repository contains the first-stage MVP for learning fact-level evidence slots from synthetic images and natural-language rules. The Composer is intentionally deferred until the extractor passes its fact-recovery checks.

Current scope

Generate deterministic scenes containing colored geometric objects and spatial relations.
Render scenes to PNG while preserving machine-readable fact triples.
Generate textual implication rules as normalized triples.
Build IID and held-out-combination splits without leaking held-out rule pairs into training.
Train visual and textual slot extractors with Hungarian set matching.
Measure triple exact match, component accuracy, precision, recall, and F1.
Run CPU smoke tests with a tiny frozen backbone or GPU experiments with frozen CLIP.

Quick start

uv sync --extra dev
uv run python -m data.build_splits --config configs/mvp.yaml --small
uv run python train_extractor.py --config configs/mvp.yaml --backbone tiny --epochs 2
uv run python -m eval.evaluate_slots --config configs/mvp.yaml \
  --checkpoint artifacts/tiny/best.pt --backbone tiny
uv run pytest

For the first real GPU experiment:

uv sync --extra dev --extra hf
uv run python -m data.build_splits --config configs/mvp.yaml
uv run python train_extractor.py --config configs/mvp.yaml --backbone clip

The CLIP path uses openai/clip-vit-base-patch32 by default. It is a deliberately small backbone for validating the extractor. A Qwen-VL adapter and the Composer belong to the next milestone after the evidence slots are shown to recover atomic facts reliably.

Success gate before Composer work

Text-rule triple F1 >= 0.95.
Visual-fact triple F1 >= 0.90 on the IID test split.
No severe slot collapse: predicted valid slots track the true fact count.
Held-out rule-combination performance is reported separately from IID performance.