trumpsignal-backend/CLAUDE.md

# Trump Alpha — Backend

> AI-powered crypto signal aggregator. Surfaces four uncorrelated signal
> streams (Trump Truth Social, Macro Vibes, KOL talks-vs-trades, funding
> reversal) and runs an optional execution layer on Hyperliquid perps.
> Real money — handle every change to the trading layer like surgery.

This file is the **first thing an AI agent should read** when entering this
repo. It encodes the invariants that aren't visible from any single file.

---

## 🛑 Read this BEFORE touching anything trading-related

This backend manages real Hyperliquid leveraged positions for real users.
Bugs cost users money. **Five non-negotiable rules:**

1. **Two systems, one execution layer.** System 1 (Trump scalp) auto-opens.
   System 2 (Macro Vibes / reversal) is **MANAGE-ONLY since v2.0** — the
   user opens manually on Hyperliquid, then `/adopt` hands it to the bot.
   `process_post()` early-returns for sys2 sources. Re-enabling sys2
   auto-open would silently reintroduce all the leverage/budget/concurrency
   race conditions we excised. Don't do it without an ADR.

2. **`released_at` is the "user took back control" marker.** A trade with
   `released_at IS NOT NULL` is **OUT OF BOUNDS for the bot**:
   - `recovery.rehydrate_open_trades` skips it
   - `reconciler` skips it
   - `close_and_finalize`'s atomic claim requires `released_at IS NULL`
     unless `force=True` (only manual_close passes that)
   - `partial_derisk` and `pyramid_add` early-return idempotent-success
   - `/positions/open`, `/positions/today`, telegram_digest all filter it
   If you add ANY new code path that touches BotTrade rows, ask yourself
   "does this respect released_at?" Almost always yes.

3. **Effective exit params are FROZEN on the BotTrade row at open time.**
   See the `eff_*` columns on `BotTrade`. Recovery rebuilds the watchdog
   from these, NOT from the live Subscription. Without this, restarting
   the backend silently rewrites every open System-2 reversal's stop loss
   to the user's Trump scalp setting (1.5%). NEVER read live
   `Subscription.stop_loss_pct` etc. in the close path.

4. **HL leverage is what HL says, not what the user requested.** Hyperliquid
   silently clips the requested leverage to the asset's max (memes capped
   at 3×). `hyperliquid.open_position()` returns `effective_leverage` —
   bot_engine and adoption must use THAT value to compute
   `sys2_protective_stop_pct(lev)` and the derisk ladder. Using the
   requested value puts the stop OUTSIDE the real liquidation line.

5. **Per-wallet asyncio locks wrap "check + open" critical sections.**
   See `_wallet_lock` in `bot_engine` and `_wallet_adopt_lock` in
   `adoption`. Without them, two concurrent signals can both pass the
   daily-budget / concurrency / already-open check before either commits.

---

## What this product does (90 seconds)

```
Four signal sources → one bot → optional Hyperliquid execution

┌──────────────────────────┐
│ 1. Trump Truth Social    │── auto-classify (DeepSeek) → "buy"/"short"/"noise"
│    (every post, <3s)     │   if actionable: Trump scalp auto-open (System 1)
└──────────────────────────┘   Tight 1.5% SL, 12h cooldown, ≥30min min-hold

┌──────────────────────────┐
│ 2. Macro Vibes           │── 8 daily macro indicators (AHR999, F&G, etc.)
│    (BTC bottom + funding)│   + 2-of-3 bottom-reversal trigger
└──────────────────────────┘   Telegram alert with /adopt CTA — NO auto-open.
                                User opens on HL → /adopt → bot manages with
                                5-rung stop ladder, de-risk, pyramid, peak-trail.

┌──────────────────────────┐
│ 3. KOL talks-vs-trades   │── Substack/podcast ingest + ETH on-chain diff
│    (19 KOLs, daily)      │   Divergence (publicly bullish, secretly selling)
└──────────────────────────┘   is the platform's highest-conviction signal.
                                Telegram alert only — never auto-trades.

┌──────────────────────────┐
│ 4. Funding extreme       │── Hourly BTC perp funding scan
└──────────────────────────┘   Alert only (manage-only via /adopt like Macro)

┌──────────────────────────┐
│   Telegram daily digest  │── Once-a-day 3-section brief (Macro/KOL/Trump)
│   (per-user hour, opt-out)│  to every subscriber. Cron at minute=0 each hour.
└──────────────────────────┘
```

Free tier = read everything + Telegram alerts. Pro tier (Hyperliquid wallet
linked) = Trump auto-trade + /adopt manage-only flow for sys2.

---

## Stack

- **Python 3.9+** / FastAPI / async SQLAlchemy 2.x / APScheduler
- **DB**: SQLite dev, **Postgres prod**. All schema lives in
  `alembic/versions/NNN_*.py`, ordered. Currently at head **024**.
- **AI**: DeepSeek via OpenAI-compatible API (`AI_BASE_URL`, `AI_MODEL`).
  - Live scoring uses `AI_LIVE_MODEL` (~2s, latency-critical)
  - Batch / reanalysis uses `AI_MODEL` (quality, ~10s)
- **Trading**: Hyperliquid SDK; API-wallet keys are envelope-encrypted with
  `ENCRYPTION_KEY` (KEK), per-user DEK derivation via `crypto.py`.
- **Prices**: Binance WS (`binance.py`) feeds `price_store` + powers the
  `tp_sl_monitor` per-tick evaluator.
- **Telegram**: long-poll mode (single instance), HTML messages, inline
  keyboards. `telegram.py` send/edit/answer + `telegram_bot.py` commands.

---

## Module map (where things live)

```
app/
├── api/                    HTTP routes
│   ├── signals.py          POST /api/signals/ingest  ← scanners write here
│   ├── positions.py        /positions/open|today|close|grow|adopt|release|hl
│   ├── user.py             /subscribe|settings|manual-window|auto-trade
│   ├── telegram.py         /telegram/{preferences,bind,unbind,test}
│   ├── macro.py            /macro/{snapshot,history}
│   └── kol.py              /kol/{posts,digest,wallets,divergence}
├── services/
│   ├── bot_engine.py       ★ TRADING CORE — process_post, _execute_for_subscriber,
│   │                         close_and_finalize, partial_derisk, pyramid_add
│   ├── adoption.py         ★ /adopt + /release flow (sys2 manage-only)
│   ├── tp_sl_monitor.py    Per-price-tick close evaluator. on_price_tick is
│   │                         called from binance.py once per second
│   ├── hyperliquid.py      HL trader (open/close/reduce/leverage)
│   ├── recovery.py         Startup rehydration of open BotTrades into watchdog
│   ├── reconciler.py       Every 60s: compare DB ↔ HL state, mark drift
│   ├── circuit_breaker.py  Per-system (sys1/sys2) CB, daily DD + N-loss streak
│   ├── signal_categories.py CRITICAL CONFIG — sys1/sys2 sources, ladders,
│   │                         leverage clamping, protective stop formulas
│   ├── regime_filter.py    Sys1 only — recent-move / vol-contraction gates
│   ├── analysis.py         AI signal scoring (DeepSeek)
│   ├── entry_filter.py     Cheap text-based pre-filter (skip RT/URL-only)
│   ├── telegram.py         send_message / edit_message / answer_callback
│   ├── telegram_bot.py     Long-poll loop + /start /digest /adopt /release ...
│   ├── telegram_digest.py  Daily 3-section brief (rule-based; no LLM)
│   ├── price_store.py      In-memory latest price per asset
│   ├── price_backfill.py   Backfill historical 5min bars from Binance
│   ├── crypto.py           HL API-key envelope encryption
│   ├── scanner_state.py    In-memory toggle + observability for scanners
│   ├── macro/
│   │   ├── fetchers.py     8 macro indicator HTTP fetchers (each @_none_on_fail)
│   │   ├── scoring.py      Weighted composite -100..+100
│   │   └── poll.py         Daily UPSERT into macro_snapshots
│   ├── scanners/
│   │   ├── btc_bottom_reversal.py   2-of-3 AHR999 + 200WMA + Pi Bottom
│   │   ├── funding_reversal.py       Hourly funding extreme
│   │   └── sma_reclaim.py            (archive — not scheduled)
│   ├── kol_substack.py     RSS ingest for 19 KOL feeds
│   ├── kol_onchain.py      HL public API + Etherscan diff
│   ├── kol_divergence.py   Cross-ref talks vs trades within ±7d
│   ├── kol_analysis.py     AI ticker/direction/conviction extract
│   ├── bottom_indicators.py AHR999 / Pi Cycle / 200WMA math
│   └── funding_signal.py   Real-time funding extreme detector
├── scrapers/
│   └── truth_social.py     CNN + trumpstruth.org pollers (5s interval)
├── ws/
│   └── manager.py          WebSocket fan-out for live UI updates
├── models.py               ★ All SQLAlchemy models in one file
├── database.py             Async engine + session factory
├── config.py               Pydantic Settings — reads .env
└── main.py                 FastAPI lifespan, scheduler setup, route mount

alembic/versions/           Migrations (numbered NNN). Latest = 024
scripts/                    One-shot ops
├── preflight.py            Pre-launch readiness gate (env / DB / TG / AI)
├── launch_smoke.py         End-to-end smoke (14 checks against running API)
├── rescore_v5.py           Re-score every Post with current AI prompt
├── backfill_signals.py     Fill in signal for posts missing it
└── verify_sys2_lifecycle.py Manual System-2 lifecycle walk-through

tests/                      pytest, 64 tests, fast (<1s total)
```

---

## The two trading systems (memorise this)

```
                  System 1                  System 2
                  ────────                  ────────
Source            "truth"                   "btc_bottom_reversal"
                                            (+"funding_reversal" alert-only)
Trigger           Trump posts a thing       Daily scanner: 2-of-3 confluence
Latency need      Seconds (price moves fast) Days/weeks (signal lives a long time)
Open path         Auto (bot_engine.process_post fires _execute_for_subscriber)
                                            MANUAL: user opens on HL UI, then
                                            /adopt hands it to the bot
Stop loss         User-configured + tight   sys2_protective_stop_pct(actual_lev)
                  1.5% floor (TRUMP_*)      = 85% × (100/lev), capped at 35%
Exit model        TP / trailing / SL        5-rung stop ladder + downside de-risk
                                            + pyramid + peak-trail. NO TP.
Min hold          30 min (suppresses TP)    n/a
Max hold          6h                        18 months (ladder is the real exit)
Sizing            base × regime multiplier  Whatever user opened on HL
Concurrency cap   n/a                       3 positions / wallet (correlated beta)
Confidence min    88 (platform) / user      85 (platform)
Circuit breaker   sys1_*                    sys2_*  (independent)
Daily budget      Full daily_budget_usd     n/a — user controls notional on HL
Telegram alert    Trump alert format        Macro/funding alert + /adopt CTA
```

**If you're tempted to put sys2 logic in `_execute_for_subscriber`**: stop.
`process_post()` early-returns for sys2. The function only runs for sys1 now.
The dead sys2 branches inside `_execute_for_subscriber` are kept for diff
minimalism — don't extend them.

---

## The /adopt flow (System-2 lifecycle in detail)

```
1. Scanner fires
   └─ POST /api/signals/ingest  (source=btc_bottom_reversal, signal=buy)
   └─ Post row created
   └─ process_post() early-returns for sys2 (no auto-open)
   └─ notify_signal() → Telegram fan-out with /adopt CTA appended

2. User opens BTC long on Hyperliquid manually
   └─ size / leverage of their choice

3. User in bot: /adopt
   └─ adoption.list_hl_positions(wallet) reads HL state
   └─ Telegram inline keyboard: tap [🟢 BTC long $1500 @72k · 2x]
   └─ Mode picker: [📈 Standard] or [🚀 Aggressive]
   └─ adoption.adopt_position(wallet, asset, mode):
      a. Per-wallet asyncio lock acquired
      b. Pre-flight: no_subscription / no_hl_key / paper_mode /
         circuit_breaker (sys2 CB still gates adopt!) /
         already_adopted / concurrency_cap (3)
      c. Re-read HL state inside lock (fresh entry/size/lev)
      d. Resolve sys2_protective_stop_pct(HL_actual_leverage)
      e. INSERT BotTrade with eff_* frozen + sys2_mode + hl_order_id="adopted:<ts>"
         + trigger_post_id=NULL
      f. register_trade() with full ladder/de-risk/addon/peak_trail
   └─ Telegram confirmation w/ ladder summary

4. tp_sl_monitor drives the position
   └─ Stop ratchet, downside de-risk partial reduces, pyramid add-ons,
      peak-trail close, max_hold backstop. All staged through the
      lock-protected partial_derisk / pyramid_add / close_and_finalize.

5a. User wants out: /release
    └─ release_management(wallet, trade_id):
       a. Sets BotTrade.released_at = now
       b. unregister(trade_id) from watchdog
    └─ HL position UNTOUCHED — bot stops driving, user has manual control
    └─ Across restarts: recovery skips released rows (released_at filter)
    └─ Reconciler skips them. Released trades don't appear in
       /positions/open or the digest "your status" line.

5b. Bot drives the close (ladder / max-hold)
    └─ close_and_finalize() atomic claim sets closed_at, computes pnl
    └─ Trade is now CLOSED (closed_at set, exit_price + pnl_usd written)

5c. User force-closes via UI: POST /api/positions/{id}/close
    └─ manual_close calls close_and_finalize(force=True) — bypasses
       released_at guard. Works on adopted-and-released trades too.

6. Recovery on restart:
   └─ recovery.rehydrate_open_trades reads BotTrade WHERE closed_at IS NULL
      AND released_at IS NULL
   └─ For each: rebuild ladder from sys2_mode + (category OR adopted fallback).
      The fallback for trigger_post_id IS NULL is the critical fix —
      without it adopted trades lose their entire sys2 ladder on restart.
```

---

## Critical invariants checklist (when reviewing any trading change)

- [ ] Does this code path respect `released_at IS NULL`?
- [ ] Does it use `eff_*` (frozen) not live `Subscription.*` for exit math?
- [ ] If it opens a new position, does it use HL's actual leverage (not requested)?
- [ ] If it touches an open position concurrently, is it wrapped in the
      per-trade `_lock_for(trade_id)` lock?
- [ ] If it opens, is it inside `_wallet_lock(wallet)` so the budget /
      concurrency check is atomic with the write?
- [ ] If it closes, does it use the conditional `UPDATE ... WHERE closed_at
      IS NULL` atomic claim?
- [ ] Does it handle the `already_closed` path from HL gracefully (preserve
      banked partial PnL)?
- [ ] Does it correctly check sys1 vs sys2 CB independently?

---

## Common workflows

### Add a new signal source

1. Decide: System 1 (auto-trade) or System 2 (alert + /adopt) or alert-only?
2. Write a scanner under `app/services/scanners/NEW.py` that posts to
   `POST /api/signals/ingest` with `{source: "your_new_source", ...}`.
3. Schedule it in `app/main.py` (`_scheduler.add_job`).
4. Add the source to `signal_categories.SYSTEM_1_SOURCES` or
   `SYSTEM_2_SOURCES` if it should trade. Leave it out if alert-only.
5. Add a Telegram preference column to `TelegramBinding` (migration)
   + a mapping entry in `telegram._pref_column_for_source`.
6. Add a label to `telegram._source_label` and `_signal_emoji`.
7. Add a `/yoursource on|off` command in `telegram_bot.py`.
8. If sys2: extend `signal_categories._CATEGORY_EXITS` if it needs a custom
   exit profile (otherwise default works).

### Add a new bot command

1. `_cmd_x` async function in `telegram_bot.py`.
2. Route it in `_handle_message`.
3. If it needs inline buttons: build `reply_markup` payload, handle
   callbacks in `_handle_callback` (route by `callback_data` prefix).
4. Update `HELP_TEXT` and remind the user to add it to BotFather
   `/setcommands` after deploy.

### Add a column to an existing table

1. New migration `alembic/versions/NNN_description.py`.
   - Use `op.batch_alter_table` (sqlite-compatible).
   - Default values via `server_default=` so backfill is implicit.
2. Mirror the field on the SQLAlchemy model in `app/models.py`.
3. Apply locally: `DATABASE_URL=<sqlite> alembic upgrade head`.

### Deploy

```bash
# On the server:
DATABASE_URL=$PROD_URL alembic upgrade head
systemctl restart trumpalpha-backend   # or whatever the unit is
python scripts/preflight.py             # MUST pass before flipping traffic
python scripts/launch_smoke.py --base https://api.trumpalpha.io
```

---

## Testing

```bash
source venv/bin/activate
python -m pytest tests/ -q           # full suite, ~0.5s
python scripts/preflight.py          # env + DB + TG + AI auth checks
python scripts/launch_smoke.py       # 14 end-to-end checks vs running API
```

64 tests. Adoption + telegram_digest are snapshot-style (no real HL/AI).
End-to-end trading is verified manually via the bot.

---

## Telegram bot mechanics (since it's a custom integration)

- **Long-poll mode** via `getUpdates`. Only ONE process can long-poll a
  given bot token at a time — if you horizontally scale, switch to
  webhook (not done yet).
- Bot must be re-bound via `@BotFather` `/setcommands` whenever new
  commands are added (the slash-menu users see is separate from what
  the bot internally handles).
- `send_message` returns `False` on failure; per-user binding rows track
  `total_alerts_sent` / `total_alerts_failed` counters.
- **Inline keyboards** = the `reply_markup` payload to `sendMessage`.
  Callback data is capped at 64 bytes; keep it short (`adopt:mode:BTC:standard`).
  `_handle_callback` MUST end with `answer_callback` or the button spins
  forever on the user's client.
- Free tier = walletless `/start` (chat_id only). Pro tier = wallet bound
  via `/start CODE` where CODE comes from Settings UI.

---

## Why "Macro Vibes" became manage-only (the ADR)

V1.0: System 2 auto-opened sys2 trades on user wallets. Carried real
execution surface: leverage clipping, daily budget split, concurrency caps,
sys2 paper branches, key handling per user. Audit surfaced ~6 bugs.

V2.0 (current): sys2 manage-only. The strategy is day-K — entry delay of
24h doesn't matter. The valuable part is multi-month exit management
(5-rung ladder, de-risk, pyramid, peak-trail), which still runs against
positions the user adopts.

**Net effect**: massive reduction in execution risk surface; same alpha
(strategy logic unchanged); legal/responsibility shifts from "bot opened
this for you" to "you opened it, bot manages your discipline".

---

## Things that LOOK like bugs but aren't

- **`_execute_for_subscriber` has lots of `if sub["_is_system_2"]` branches.**
  Dead code under v2.0 (process_post early-returns for sys2). Kept for diff
  minimalism — don't extend or re-enable.
- **`funding_reversal` source is in `SYSTEM_2_SOURCES`? No.** It's
  intentionally NOT in either supported set — it ingests as a Post for
  audit + sends Telegram alert via the CTA path, but doesn't trigger any
  auto-trading. Adopt still works (it's asset-based, not signal-based).
- **`Subscription.sys2_budget_pct` defaults to 0.7.** Legacy field from the
  auto-open era. With v2.0 manage-only, it's effectively unused — sys1
  (Trump) reads full `daily_budget_usd`. Don't read it for new code.
- **Adopted trades have `hl_order_id` starting with `"adopted:"`.** Distinct
  from auto-opened (HL order id integer) and paper (`"paper"` literal).
  Useful for telemetry filtering.

---

## Open known issues (not blocking launch but worth fixing later)

- **Trump daily budget split**: `_execute_for_subscriber` still computes
  `daily_cap = total × (1 - sys2_pct)` for sys1 — i.e. Trump gets only 30%
  of the configured daily budget by default. With sys2 manage-only there's
  no reason to split. Fix: when not sys2, use full `total_cap`. (Low-prio:
  users who notice can just set `sys2_budget_pct = 0` in DB.)
- **HL high-leverage adoption**: if a user has a 25× position on HL, our
  `sys2_protective_stop_pct` clamps to 10× (the strategy's design max),
  which gives a stop FURTHER from entry than the real liquidation point.
  Position could get HL-liquidated before our stop fires. Fix: reject
  adoptions where actual HL leverage > `SYS2_MAX_LEVERAGE`.
- **`adopt:choose:BTC` callback may show stale prices** if user takes >60s
  to tap (HL fees, partial fills can change entry/size). adopt_position
  re-reads HL at mode-tap time so the FROZEN BotTrade is always fresh,
  but the picker label could be outdated.
- **Telegram bot offset on restart**: `getUpdates` may replay last 24h of
  messages on backend restart. Stale `/adopt` could re-fire. User can
  /release to recover.

---

## Repos in this project

- **This repo** (`/Users/k/Public/Claude/backend`) — Python/FastAPI backend
- **Sibling frontend** (`/Users/k/Public/Claude/trumpsignal`) — Next.js 16
  dashboard at trumpsignal.com. See its own CLAUDE.md.

Both deployed independently. Backend serves the JSON API + Telegram bot.
Frontend is a thin SPA over the API + WebSocket.