Files
ai-taxonomy-literature-anal…/scripts/fetch_data.sh
T
zzyang 13924f8985 Add LLM research survey site (Next.js pages router + SQLite)
- Next.js/React/TypeScript app: overview, paper explorer with FTS5 search,
  paper detail, global taxonomy tree with per-conference tag breakdown
- Python ingestion (ICLR/ICML 2026 via paperlists, ACL Anthology 2026) into
  data/papers.db, plus keyword-scored taxonomy (scripts/taxonomy.py)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bhs9LU8eMixk6g1kb1d3j7
2026-10-21 11:30:14 +00:00

17 lines
838 B
Bash
Executable File

#!/usr/bin/env bash
# Download raw accepted-paper data into data/raw (git-ignored).
# - papercopilot/paperlists -> ICLR 2026 (git-lfs file via media host) and ICML 2026
# - acl-org/acl-anthology -> 2026 volumes (ACL, Findings, EACL, LREC, CoNLL, SIGDIAL, TACL, CL)
set -euo pipefail
cd "$(dirname "$0")/.."
RAW=data/raw
mkdir -p "$RAW"
BASE=https://raw.githubusercontent.com/papercopilot/paperlists/main
curl -fsSL -o "$RAW/icml2026.json" "$BASE/icml/icml2026.json"
curl -fsSL -o "$RAW/iclr2026.json" https://media.githubusercontent.com/media/papercopilot/paperlists/main/iclr/iclr2026.json
if [ ! -d "$RAW/acl-anthology" ]; then
git clone --depth 1 --filter=blob:none --sparse https://github.com/acl-org/acl-anthology.git "$RAW/acl-anthology"
git -C "$RAW/acl-anthology" sparse-checkout set data/xml
fi
echo "done: $(ls $RAW)"