Pre-launch hardening: KOL module, Telegram, scanners, WS resilience

Big-picture changes since b941223:

KOL pipeline (new) — Substack/podcast/blog RSS → AI ticker extraction →
on-chain wallet diff → talks-vs-trades divergence detection. Daily polls,
19 feeds, divergence emits Post + Telegram fan-out.

Telegram push (new) — walletless free tier + wallet-linked Pro upgrade,
in-bot preference commands (/trump /btc /funding /kol /conf /quiet),
signed-envelope API for dashboard. Disconnect-wallet keeps free
subscription.

BTC funding-rate reversal scanner (new) — hourly cron, 30d cumulative
funding threshold + mean-revert + 7d price confirm, emits via
/api/signals/ingest. BTC bottom-reversal scanner promoted to System 2.

WS broadcast rewrite — per-client send timeout + parallel fan-out
(asyncio.gather). Fixes "Binance WS no close frame" reconnect storms +
APScheduler 11-min job misses, both caused by one slow client stalling
the kline loop.

Error visibility — three silent-error sites (trumpstruth/truth_social
fetchers, funding_reversal scanner) now include exception type name so
httpx ConnectError-style empty-message errors stop logging blank lines.
Telegram bot loop now classifies ReadTimeout vs network vs unknown +
logger.exception for the unknown bucket.

Security hygiene — trumpsignal.db untracked from git (held subscriber
wallets + encrypted HL keys + 22 bot trades); .gitignore now blocks
*.db/.next/backups. CORS only allows FRONTEND_URL in production.

New ops scripts —
  - scripts/preflight.py: env/DB/Telegram/AI auth verification gate
  - scripts/backup_db.sh: cron-friendly daily DB backup (SQLite + Postgres)
  - scripts/seed_kol_wallets.py: idempotent KOL on-chain wallet seeder

15 new Alembic migrations (007-021) covering convex strategy fields,
phase-1 safety, two-system frozen exits, invalidation prices, dynamic
SYS2 leverage, staged de-risk + pyramiding, peak gain tracking, risk
mode, auto-trade + grow flags, KOL module, KOL on-chain, KOL divergence,
Telegram bindings + walletless.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
k
2026-05-25 00:52:56 +08:00
parent b941223c88
commit 5fb1d52026
81 changed files with 13251 additions and 158 deletions
+12
View File
@@ -0,0 +1,12 @@
# Archived scanners (not in the live path)
These were generic System-2 scanners superseded by the focused
`btc_bottom_reversal` state machine. Nothing imports them; they are NOT
scheduled and do NOT register with scanner_state. Kept for reference/history.
- vcp_breakout.py — volatility-contraction breakout (bidirectional)
- weekly_rsi_reversal.py — weekly RSI extreme + recovery (bidirectional)
To revive: re-add an APScheduler job in app/main.py and a
scanner_state.register() call, and ensure the source tag is in
signal_categories.SYSTEM_2_SOURCES.
@@ -0,0 +1,251 @@
"""
VCP-style breakout scanner — example of an external signal module.
What this scans for (Minervini-light, adapted for crypto perps):
1. Volatility contraction: 7-day ATR ≤ 50% of 30-day ATR
(the price has been getting tighter — supply is drying up)
2. Breakout confirmation: current 4h close > max(prior 30-day highs)
(a real break of the range, not just touching the top)
3. Volume confirmation: last 24h volume ≥ 1.5× 30-day average
(institutional participation, not retail noise)
4. Cooldown: no signal for the same asset in the past 48h
(avoid stacking)
When all 4 conditions hit, the scanner POSTs to /api/signals/ingest. From
there the trade goes through the same convex-strategy pipeline as Trump
signals — regime filter, sizing, trailing stop, all of it.
Why this module exists:
Trump signals have 1-2 true catalysts per month. Adding a TA-based source
that finds setups continuously means the bot has SOMETHING to act on most
weeks, instead of waiting for geopolitical fireworks. The user's hypothesis
is that consolidation breakouts have asymmetric upside; the existing
trailing-stop logic is purpose-built to capture that.
Tunables (don't hardcode in caller — change them HERE if you want to test):
ATR_RECENT_DAYS, ATR_BASELINE_DAYS, ATR_RATIO_MAX,
BREAKOUT_LOOKBACK_DAYS, VOLUME_MULT_MIN, COOLDOWN_HOURS.
Asset list:
Default = SOL only. Extend ASSETS_TO_SCAN with more once you've watched
this run for a week and confirmed it doesn't fire-hose noise.
"""
from __future__ import annotations
import logging
import statistics
from datetime import datetime, timedelta, timezone
from typing import Optional
import httpx
from app.config import settings
from app.services.market_data import for_asset, drop_in_progress_bar
from app.services import scanner_state
logger = logging.getLogger(__name__)
SCANNER_NAME = "vcp_breakout"
scanner_state.register(SCANNER_NAME)
# ─── Tunables ───────────────────────────────────────────────────────────────
ATR_RECENT_DAYS = 7
ATR_BASELINE_DAYS = 30
ATR_RATIO_MAX = 0.5 # recent ATR must be ≤ 50% of baseline
BREAKOUT_LOOKBACK_DAYS = 30 # break of this many days' high
VOLUME_MULT_MIN = 1.5 # current 24h vol vs 30-day avg
COOLDOWN_HOURS = 48 # don't re-signal same asset within this window
# Assets to scan. Provider selection (Binance vs HL) is automatic via
# market_data.for_asset() — HL-native perps like TRUMP/HYPE route to HL
# automatically. Add to HL_NATIVE_ASSETS in market_data.py if needed.
ASSETS_TO_SCAN = [
"SOL",
# "ETH", "LINK", "AVAX", # Binance has them
# "TRUMP", "HYPE", # HL-native (routed automatically)
]
# Cooldown via scanner_state.in_cooldown — DB-backed, restart-safe.
# Data fetching is delegated to market_data.for_asset() — see that module
# for Binance vs Hyperliquid routing.
# ─── Signal logic ───────────────────────────────────────────────────────────
def _atr_proxy(candles: list[dict]) -> Optional[float]:
"""Mean of (high-low)/close — simple ATR proxy, dimensionless."""
ranges = [(c["high"] - c["low"]) / c["close"] for c in candles if c["close"] > 0]
return statistics.fmean(ranges) if ranges else None
def evaluate_breakout(candles: list[dict]) -> tuple[bool, dict]:
"""Returns (is_signal, debug_info). All 4 gates must pass.
Pure function — no side effects, fully testable.
"""
if len(candles) < ATR_BASELINE_DAYS * 6:
return False, {"reason": "insufficient_data", "bars": len(candles)}
# 1. Volatility contraction
recent_bars = candles[-ATR_RECENT_DAYS * 6:]
baseline_bars = candles[-ATR_BASELINE_DAYS * 6:]
atr_recent = _atr_proxy(recent_bars)
atr_baseline = _atr_proxy(baseline_bars)
if not atr_recent or not atr_baseline:
return False, {"reason": "atr_calc_failed"}
atr_ratio = atr_recent / atr_baseline
if atr_ratio > ATR_RATIO_MAX:
return False, {"reason": "no_contraction", "atr_ratio": round(atr_ratio, 3)}
# 2. Range break — UP through prior high (long) or DOWN through prior
# low (short). Lookback excludes the current bar.
lookback = candles[-BREAKOUT_LOOKBACK_DAYS * 6 : -1]
if not lookback:
return False, {"reason": "no_lookback"}
prior_high = max(c["high"] for c in lookback)
prior_low = min(c["low"] for c in lookback)
current_close = candles[-1]["close"]
if current_close > prior_high:
direction, ref, gap = "buy", prior_high, (current_close - prior_high) / prior_high * 100
elif current_close < prior_low:
direction, ref, gap = "short", prior_low, (prior_low - current_close) / prior_low * 100
else:
return False, {"reason": "no_range_break", "close": current_close,
"prior_high": prior_high, "prior_low": prior_low}
# 3. Volume confirmation (same for both directions — a real break needs
# participation regardless of which way it goes)
recent_vol_24h = sum(c["volume"] for c in candles[-6:])
baseline_avg_24h = sum(c["volume"] for c in candles[-180:]) / 30
if baseline_avg_24h <= 0:
return False, {"reason": "no_volume_data"}
vol_ratio = recent_vol_24h / baseline_avg_24h
if vol_ratio < VOLUME_MULT_MIN:
return False, {"reason": "weak_volume", "vol_ratio": round(vol_ratio, 2)}
return True, {
"direction": direction,
"atr_ratio": round(atr_ratio, 3),
"vol_ratio": round(vol_ratio, 2),
"break_ref": round(ref, 4),
"break_at": round(current_close, 4),
"headroom_pct": round(gap, 2),
}
def _confidence_from(debug: dict) -> int:
"""Map the signal's quality into a 0-100 confidence used by sizing.
The tighter the contraction and the bigger the volume spike, the higher
the confidence. Capped at 95 — leave headroom so AI signals at 100 stay
distinguishable.
"""
score = 70 # baseline for "all 4 gates passed"
# Tighter contraction → higher confidence
if debug.get("atr_ratio", 1) < 0.35: score += 10
elif debug.get("atr_ratio", 1) < 0.45: score += 5
# Bigger volume spike → higher confidence
if debug.get("vol_ratio", 0) >= 3.0: score += 10
elif debug.get("vol_ratio", 0) >= 2.0: score += 5
return min(score, 95)
# ─── Ingest call ────────────────────────────────────────────────────────────
async def _emit_signal(asset: str, debug: dict) -> None:
"""POST the signal to /api/signals/ingest. Treats the local server as the
target — same machine, no network hop in dev.
"""
if not settings.ingest_api_key:
logger.warning("VCP: signal would fire on %s but INGEST_API_KEY is empty", asset)
return
confidence = _confidence_from(debug)
direction = debug["direction"]
way = "breakout ↑" if direction == "buy" else "breakdown ↓"
rel = "above" if direction == "buy" else "below"
payload = {
"source": "breakout",
"external_id": f"vcp-{asset}-{direction}-{datetime.now(timezone.utc).strftime('%Y%m%d-%H%M')}",
"text": (
f"VCP {way} on {asset}: ATR ratio {debug['atr_ratio']} "
f"(7d/30d), {debug['vol_ratio']}× normal volume, "
f"close ${debug['break_at']} {rel} {BREAKOUT_LOOKBACK_DAYS}-day "
f"level ${debug['break_ref']} ({debug['headroom_pct']}%)"
),
"signal": direction,
"target_asset": asset,
"confidence": confidence,
"category": "vcp_breakout",
"expected_move_pct": 5.0, # historical VCP median expansion
}
async with httpx.AsyncClient(timeout=10) as client:
resp = await client.post(
"http://localhost:8000/api/signals/ingest",
json=payload,
headers={"X-Ingest-Key": settings.ingest_api_key},
)
if resp.status_code >= 400:
logger.error("VCP ingest failed (%d): %s", resp.status_code, resp.text[:200])
else:
logger.info("VCP signal emitted for %s, confidence=%d: %s",
asset, confidence, resp.json())
# ─── Scheduler entry point ──────────────────────────────────────────────────
async def scan_once() -> None:
"""Single scan pass over ASSETS_TO_SCAN. Called every 30 minutes.
Kill-switch + DB-backed cooldown. Drops in-progress 4h bar before
evaluating — without that, the breakout test ran against a partial bar
and could flicker FIRE / no-FIRE within a 4h window.
"""
if not scanner_state.is_enabled(SCANNER_NAME):
logger.debug("VCP scanner disabled — skipping run")
return
fired_any = False
error_msg: Optional[str] = None
for asset in ASSETS_TO_SCAN:
# Cooldown — DB-backed, restart-safe.
cooldown_days = COOLDOWN_HOURS / 24
if await scanner_state.in_cooldown("breakout", asset, cooldown_days):
continue
provider = for_asset(asset)
try:
candles = await provider.fetch_4h(asset, days=ATR_BASELINE_DAYS + 1)
except Exception as exc:
error_msg = f"{asset}: {exc}"
logger.error("VCP fetch failed for %s via %s: %s", asset, provider.name, exc)
continue
candles = drop_in_progress_bar(candles, "4h")
if not candles:
logger.warning("VCP: %s returned 0 candles from %s — symbol may not exist",
asset, provider.name)
continue
is_signal, debug = evaluate_breakout(candles)
logger.info("VCP scan %s [%s]: %s%s", asset, provider.name,
"FIRE" if is_signal else "no", debug)
if is_signal:
await _emit_signal(asset, debug)
fired_any = True
if error_msg:
scanner_state.record_run(SCANNER_NAME, "error", error_msg)
elif fired_any:
scanner_state.record_run(SCANNER_NAME, "fired")
else:
scanner_state.record_run(SCANNER_NAME, "ok")
@@ -0,0 +1,264 @@
"""
Weekly RSI extreme + recovery scanner.
What it catches:
Severe weekly oversold conditions that flush capitulating holders, followed
by the first sign of strength. Historical examples this would have hit:
BTC 2022-11 ($15.5k bottom)
ETH 2022-06 ($900 bottom)
SOL 2022-12 ($8 bottom)
AAVE 2022-06 ($45 bottom)
Trigger logic (intentionally strict):
PRE-CONDITION: ≥ 4 consecutive completed weekly bars with RSI(14) < 30
(this is the capitulation phase — sustained extreme weakness)
TRIGGER: current completed week's RSI(14) >= 35
(i.e. the FIRST recovery week — RSI has lifted off the floor)
COOLDOWN: 60 days — these events happen ~once per year per asset.
No re-firing on the same setup.
Companion exit parameters (passed to the bot via signal payload — wider stops
to match the longer holding period and higher single-trade conviction):
SL = 8%
TRAILING_ACTIVATE = 15%
TRAILING_STOP = 6%
MAX_HOLD = 60 days
This is a high-conviction, low-frequency signal. Expect 0-3 fires per asset
per year. Don't tune it for higher frequency — that defeats the point.
"""
from __future__ import annotations
import logging
from datetime import datetime, timedelta, timezone
from typing import Optional
import httpx
from app.config import settings
from app.services.market_data import REVERSAL_BASKET, for_asset, drop_in_progress_bar
from app.services import scanner_state
SCANNER_NAME = "weekly_rsi_reversal"
scanner_state.register(SCANNER_NAME)
logger = logging.getLogger(__name__)
# ─── Tunables ───────────────────────────────────────────────────────────────
RSI_PERIOD = 14
RSI_OVERSOLD = 30 # "deeply oversold" floor (long setup)
RSI_RECOVERY_TRIGGER = 35 # bounce-off line for the long
RSI_OVERBOUGHT = 70 # "euphoric" ceiling (short setup)
RSI_ROLLOVER_TRIGGER = 65 # roll-off line for the short
WEEKS_OF_OVERSOLD = 4 # consecutive extreme weeks required (both sides)
WEEKS_TO_FETCH = 40 # enough history for stable Wilder's RSI
COOLDOWN_DAYS = 60
# Exit profile passed to /signals/ingest. Caller can override on the bot side.
PAYLOAD_CONFIDENCE = 88 # high conviction — top of the regime-filter score
PAYLOAD_EXPECTED_MOVE = 30.0 # historical median move after capitulation
# Cooldown is now DB-backed via scanner_state.in_cooldown() — survives
# restart. No more in-memory _last_signal_at dict.
# ─── RSI math (Wilder's smoothed) ───────────────────────────────────────────
def calculate_rsi(closes: list[float], period: int = 14) -> list[Optional[float]]:
"""Wilder's RSI. Returns a list aligned with `closes` where the first
`period` entries are None (not enough history yet).
Implementation note: the first valid RSI uses a SIMPLE mean of the first
`period` gains/losses, subsequent values use exponential smoothing with
alpha = 1/period. This matches TradingView, Binance UI, and most charting
tools — the "Cutler's RSI" (pure SMA) variant would give different results.
"""
n = len(closes)
rsi: list[Optional[float]] = [None] * n
if n < period + 1:
return rsi
gains = [max(closes[i] - closes[i - 1], 0) for i in range(1, n)]
losses = [max(closes[i - 1] - closes[i], 0) for i in range(1, n)]
# First valid RSI = simple mean over first `period` deltas
avg_gain = sum(gains[:period]) / period
avg_loss = sum(losses[:period]) / period
if avg_loss == 0:
rsi[period] = 100.0
else:
rs = avg_gain / avg_loss
rsi[period] = 100 - 100 / (1 + rs)
# Subsequent values use Wilder's smoothing
for i in range(period + 1, n):
avg_gain = (avg_gain * (period - 1) + gains[i - 1]) / period
avg_loss = (avg_loss * (period - 1) + losses[i - 1]) / period
if avg_loss == 0:
rsi[i] = 100.0
else:
rs = avg_gain / avg_loss
rsi[i] = 100 - 100 / (1 + rs)
return rsi
# ─── Signal logic ───────────────────────────────────────────────────────────
def evaluate_rsi_reversal(weekly_candles: list[dict]) -> tuple[bool, dict]:
"""Pure function — no side effects, fully testable.
Returns (is_signal, debug_info).
"""
if len(weekly_candles) < RSI_PERIOD + WEEKS_OF_OVERSOLD + 2:
return False, {"reason": "insufficient_data", "bars": len(weekly_candles)}
closes = [c["close"] for c in weekly_candles]
rsi = calculate_rsi(closes, RSI_PERIOD)
# The MOST RECENT weekly bar is `closes[-1]`. We treat it as the "current"
# recovery candidate. Look BACKWARDS for WEEKS_OF_OVERSOLD prior bars all
# with RSI < RSI_OVERSOLD.
current_rsi = rsi[-1]
if current_rsi is None:
return False, {"reason": "rsi_not_computable"}
window = rsi[-(WEEKS_OF_OVERSOLD + 1):-1] # the N weeks before current
if any(r is None for r in window):
return False, {"reason": "history_gaps_in_window"}
this_close = weekly_candles[-1]["close"]
prev_close = weekly_candles[-2]["close"]
# ── LONG: capitulation bottom ──────────────────────────────────────────
# Sustained RSI<30, now lifting ≥35, price turning up.
if (current_rsi >= RSI_RECOVERY_TRIGGER
and all(r < RSI_OVERSOLD for r in window)
and this_close > prev_close):
low_w = min(c["low"] for c in weekly_candles[-(WEEKS_OF_OVERSOLD + 1):])
return True, {
"direction": "buy",
"current_rsi": round(current_rsi, 1),
"window_rsi": [round(r, 1) for r in window],
"move_pct": round((this_close - low_w) / low_w * 100, 2) if low_w > 0 else 0.0,
"trigger_close": round(this_close, 4),
}
# ── SHORT: euphoric top ────────────────────────────────────────────────
# Sustained RSI>70, now rolling ≤65, price turning down. Symmetric mirror.
if (current_rsi <= RSI_ROLLOVER_TRIGGER
and all(r > RSI_OVERBOUGHT for r in window)
and this_close < prev_close):
high_w = max(c["high"] for c in weekly_candles[-(WEEKS_OF_OVERSOLD + 1):])
return True, {
"direction": "short",
"current_rsi": round(current_rsi, 1),
"window_rsi": [round(r, 1) for r in window],
"move_pct": round((high_w - this_close) / high_w * 100, 2) if high_w > 0 else 0.0,
"trigger_close": round(this_close, 4),
}
# Neither side qualified — report the closest miss for the log.
return False, {
"reason": "no_setup",
"current_rsi": round(current_rsi, 1),
"window_rsi": [round(r, 1) for r in window],
}
# ─── Ingest emission ────────────────────────────────────────────────────────
async def _emit_signal(asset: str, debug: dict) -> None:
if not settings.ingest_api_key:
logger.warning("RSI-reversal would fire on %s but INGEST_API_KEY empty", asset)
return
direction = debug["direction"]
side_word = "capitulation bottom" if direction == "buy" else "euphoric top"
payload = {
"source": "rsi_reversal",
"external_id": f"rsi-{asset}-{direction}-{datetime.now(timezone.utc).strftime('%Y%W')}",
"text": (
f"Weekly RSI {side_word} on {asset}: RSI {debug['current_rsi']} after "
f"{WEEKS_OF_OVERSOLD}wk extreme (window: {debug['window_rsi']}). "
f"{debug['move_pct']}% move off the extreme @ ${debug['trigger_close']}."
),
"signal": direction,
"target_asset": asset,
"confidence": PAYLOAD_CONFIDENCE,
"category": "rsi_extreme_reversal",
"expected_move_pct": PAYLOAD_EXPECTED_MOVE,
}
async with httpx.AsyncClient(timeout=10) as client:
resp = await client.post(
"http://localhost:8000/api/signals/ingest",
json=payload,
headers={"X-Ingest-Key": settings.ingest_api_key},
)
if resp.status_code >= 400:
logger.error("RSI-reversal ingest failed (%d): %s", resp.status_code, resp.text[:200])
else:
logger.info("RSI-reversal signal emitted for %s: %s", asset, resp.json())
# ─── Scheduler entry point ──────────────────────────────────────────────────
async def scan_once() -> None:
"""One scan pass over REVERSAL_BASKET. Called by APScheduler.
Kill-switch aware: short-circuits if the operator disabled this scanner.
Cooldown is DB-backed (queries posts table), so restarts don't reset
state.
"""
if not scanner_state.is_enabled(SCANNER_NAME):
logger.debug("RSI-reversal scanner disabled — skipping run")
return
fired_any = False
error_msg: Optional[str] = None
for asset in REVERSAL_BASKET:
# Cooldown — DB-backed.
if await scanner_state.in_cooldown("rsi_reversal", asset, COOLDOWN_DAYS):
continue
provider = for_asset(asset)
try:
candles = await provider.fetch_1w(asset, weeks=WEEKS_TO_FETCH)
except Exception as exc:
error_msg = f"{asset}: {exc}"
logger.error("RSI-reversal fetch failed for %s via %s: %s",
asset, provider.name, exc)
continue
candles = drop_in_progress_bar(candles, "1w")
if not candles:
logger.warning("RSI-reversal: %s returned 0 weekly bars from %s",
asset, provider.name)
continue
is_signal, debug = evaluate_rsi_reversal(candles)
if is_signal:
logger.info("RSI-reversal scan %s [%s]: FIRE — %s", asset, provider.name, debug)
await _emit_signal(asset, debug)
fired_any = True
else:
logger.debug("RSI-reversal scan %s [%s]: no — %s", asset, provider.name, debug)
if error_msg:
scanner_state.record_run(SCANNER_NAME, "error", error_msg)
elif fired_any:
scanner_state.record_run(SCANNER_NAME, "fired")
else:
scanner_state.record_run(SCANNER_NAME, "ok")