composable-evidence-states-research/README.md

Composable Evidence States Research

This repository contains the first-stage MVP for learning fact-level evidence slots from synthetic images and natural-language rules. The Composer is intentionally deferred until the extractor passes its fact-recovery checks.

Current scope

• Generate deterministic scenes containing colored geometric objects and spatial relations.
• Render scenes to PNG while preserving machine-readable fact triples.
• Generate textual implication rules as normalized triples.
• Build IID and held-out-combination splits without leaking held-out rule pairs into training.
• Train visual and textual slot extractors with Hungarian set matching.
• Measure triple exact match, component accuracy, precision, recall, and F1.
• Run CPU smoke tests with a tiny frozen backbone or GPU experiments with frozen CLIP.

Quick start

uv sync --extra dev
uv run python -m data.build_splits --config configs/mvp.yaml --small
uv run python train_extractor.py --config configs/mvp.yaml --backbone tiny --epochs 2
uv run python -m eval.evaluate_slots --config configs/mvp.yaml \
  --checkpoint artifacts/tiny/best.pt --backbone tiny
uv run pytest

For the first real GPU experiment:

uv sync --extra dev --extra hf
uv run python -m data.build_splits --config configs/mvp.yaml
uv run python train_extractor.py --config configs/mvp.yaml --backbone clip

The CLIP path uses openai/clip-vit-base-patch32 by default. It is a deliberately small backbone for validating the extractor. A Qwen-VL adapter and the Composer belong to the next milestone after the evidence slots are shown to recover atomic facts reliably.

Success gate before Composer work

• Text-rule triple F1 >= 0.95.
• Visual-fact triple F1 >= 0.90 on the IID test split.
• No severe slot collapse: predicted valid slots track the true fact count.
• Held-out rule-combination performance is reported separately from IID performance.