47 lines
1.9 KiB
Markdown
47 lines
1.9 KiB
Markdown
# Composable Evidence States Research
|
|
|
|
This repository contains the first-stage MVP for learning fact-level evidence slots from
|
|
synthetic images and natural-language rules. The Composer is intentionally deferred until
|
|
the extractor passes its fact-recovery checks.
|
|
|
|
## Current scope
|
|
|
|
- Generate deterministic scenes containing colored geometric objects and spatial relations.
|
|
- Render scenes to PNG while preserving machine-readable fact triples.
|
|
- Generate textual implication rules as normalized triples.
|
|
- Build IID and held-out-combination splits without leaking held-out rule pairs into training.
|
|
- Train visual and textual slot extractors with Hungarian set matching.
|
|
- Measure triple exact match, component accuracy, precision, recall, and F1.
|
|
- Run CPU smoke tests with a tiny frozen backbone or GPU experiments with frozen CLIP.
|
|
|
|
## Quick start
|
|
|
|
```bash
|
|
uv sync --extra dev
|
|
uv run python -m data.build_splits --config configs/mvp.yaml --small
|
|
uv run python train_extractor.py --config configs/mvp.yaml --backbone tiny --epochs 2
|
|
uv run python -m eval.evaluate_slots --config configs/mvp.yaml \
|
|
--checkpoint artifacts/tiny/best.pt --backbone tiny
|
|
uv run pytest
|
|
```
|
|
|
|
For the first real GPU experiment:
|
|
|
|
```bash
|
|
uv sync --extra dev --extra hf
|
|
uv run python -m data.build_splits --config configs/mvp.yaml
|
|
uv run python train_extractor.py --config configs/mvp.yaml --backbone clip
|
|
```
|
|
|
|
The CLIP path uses `openai/clip-vit-base-patch32` by default. It is a deliberately small
|
|
backbone for validating the extractor. A Qwen-VL adapter and the Composer belong to the next
|
|
milestone after the evidence slots are shown to recover atomic facts reliably.
|
|
|
|
## Success gate before Composer work
|
|
|
|
- Text-rule triple F1 >= 0.95.
|
|
- Visual-fact triple F1 >= 0.90 on the IID test split.
|
|
- No severe slot collapse: predicted valid slots track the true fact count.
|
|
- Held-out rule-combination performance is reported separately from IID performance.
|
|
|