Pre-launch hardening: KOL module, Telegram, scanners, WS resilience
Big-picture changes since b941223:
KOL pipeline (new) — Substack/podcast/blog RSS → AI ticker extraction →
on-chain wallet diff → talks-vs-trades divergence detection. Daily polls,
19 feeds, divergence emits Post + Telegram fan-out.
Telegram push (new) — walletless free tier + wallet-linked Pro upgrade,
in-bot preference commands (/trump /btc /funding /kol /conf /quiet),
signed-envelope API for dashboard. Disconnect-wallet keeps free
subscription.
BTC funding-rate reversal scanner (new) — hourly cron, 30d cumulative
funding threshold + mean-revert + 7d price confirm, emits via
/api/signals/ingest. BTC bottom-reversal scanner promoted to System 2.
WS broadcast rewrite — per-client send timeout + parallel fan-out
(asyncio.gather). Fixes "Binance WS no close frame" reconnect storms +
APScheduler 11-min job misses, both caused by one slow client stalling
the kline loop.
Error visibility — three silent-error sites (trumpstruth/truth_social
fetchers, funding_reversal scanner) now include exception type name so
httpx ConnectError-style empty-message errors stop logging blank lines.
Telegram bot loop now classifies ReadTimeout vs network vs unknown +
logger.exception for the unknown bucket.
Security hygiene — trumpsignal.db untracked from git (held subscriber
wallets + encrypted HL keys + 22 bot trades); .gitignore now blocks
*.db/.next/backups. CORS only allows FRONTEND_URL in production.
New ops scripts —
- scripts/preflight.py: env/DB/Telegram/AI auth verification gate
- scripts/backup_db.sh: cron-friendly daily DB backup (SQLite + Postgres)
- scripts/seed_kol_wallets.py: idempotent KOL on-chain wallet seeder
15 new Alembic migrations (007-021) covering convex strategy fields,
phase-1 safety, two-system frozen exits, invalidation prices, dynamic
SYS2 leverage, staged de-risk + pyramiding, peak gain tracking, risk
mode, auto-trade + grow flags, KOL module, KOL on-chain, KOL divergence,
Telegram bindings + walletless.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
k
@@ -0,0 +1,379 @@
|
||||
"""
|
||||
Funding rate extreme + reversal scanner.
|
||||
|
||||
What it catches:
|
||||
Crowded perp positioning that's about to unwind. When one side has been
|
||||
paying ridiculous funding for weeks, the eventual unwind ("the squeeze")
|
||||
is the cleanest single-day move you'll find. Examples:
|
||||
|
||||
BTC 2024-08 funding deeply +ve for 30d → 14% flush down within a week
|
||||
SOL 2023-09 funding deeply -ve for 30d → 60% rally over the next month
|
||||
ETH 2024-Q4 funding +3.5% on 30d → 12% pullback
|
||||
|
||||
Trigger logic:
|
||||
|
||||
PRE-CONDITION: Sum of last 30 days of funding rate (per 8h cycle)
|
||||
exceeds ±FUNDING_EXTREME_PCT. The sign tells us which
|
||||
side is overcrowded:
|
||||
|
||||
sum > +3% → longs have been paying — bearish setup
|
||||
(signal = short)
|
||||
sum < -3% → shorts paying — bullish setup
|
||||
(signal = buy)
|
||||
|
||||
TRIGGER: Funding direction has STARTED to mean-revert
|
||||
(last 3 funding cycles closer to zero than the 30d avg)
|
||||
AND price has begun moving in the contrarian direction
|
||||
(last 7 days price move at least 3% in that direction).
|
||||
|
||||
COOLDOWN: 14 days. Funding extremes can persist for weeks but
|
||||
each unwind is a distinct event.
|
||||
|
||||
Companion exits — tighter than RSI/SMA because funding moves play out faster
|
||||
(days, not weeks):
|
||||
SL = 4%
|
||||
TRAILING_ACTIVATE = 10%
|
||||
TRAILING_STOP = 5%
|
||||
MAX_HOLD = 30 days
|
||||
|
||||
This is the highest-frequency of the three reversal signals — expect 3-5
|
||||
fires per asset per year — and the most "alpha-like" (most market participants
|
||||
ignore funding entirely).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import statistics
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Optional
|
||||
|
||||
import httpx
|
||||
|
||||
from app.config import settings
|
||||
from app.services import scanner_state
|
||||
from app.services.market_data import REVERSAL_BASKET, for_asset, drop_in_progress_bar
|
||||
|
||||
# Promoted from booster → standalone scanner. Still importable by
|
||||
# btc_bottom_reversal for confluence boost; ALSO emits its own signals via
|
||||
# scan_once() registered with the APScheduler in app/main.py.
|
||||
|
||||
SCANNER_NAME = "funding_reversal"
|
||||
scanner_state.register(SCANNER_NAME)
|
||||
|
||||
ASSET = "BTC" # BTC-only for now; extend to ETH/SOL after results validate
|
||||
|
||||
# How much funding+price history to load each tick.
|
||||
# - Binance: 8h cadence → 30d ≈ 90 cycles
|
||||
# - HL: 1h cadence → capped at 500 rows ≈ 20.8d (handled inside evaluate)
|
||||
FUNDING_DAYS_LOOKBACK = 30
|
||||
PRICE_DAYS_LOOKBACK = 35 # need 7d confirm + buffer for in-progress bar
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# ─── Tunables ───────────────────────────────────────────────────────────────
|
||||
FUNDING_HISTORY_DAYS = 30
|
||||
FUNDING_EXTREME_THRESHOLD = 0.03 # 3% cumulative — extreme one-sided pressure
|
||||
# CRITICAL: "recent" is in HOURS, not CYCLES. Binance funding is 8h-cadence
|
||||
# (3 cycles/day) but HL is 1h-cadence (24 cycles/day) — a fixed "3 cycles
|
||||
# lookback" means 24h on Binance but only 3h on HL. We pick cycles dynamically
|
||||
# based on the response's actual cadence so 24h of funding history is always
|
||||
# the comparison window.
|
||||
FUNDING_REVERSAL_LOOKBACK_HOURS = 24
|
||||
FUNDING_REVERSAL_RATIO = 0.5 # recent avg must be ≤ 50% of 30d-avg in magnitude
|
||||
PRICE_CONFIRM_DAYS = 7
|
||||
PRICE_CONFIRM_PCT = 3.0 # price moved 3%+ in contrarian direction
|
||||
COOLDOWN_DAYS = 14
|
||||
|
||||
PAYLOAD_CONFIDENCE = 82 # slightly lower than RSI/SMA — funding can chop
|
||||
PAYLOAD_EXPECTED_MOVE = 10.0
|
||||
EMIT_STANDALONE_SIGNALS = True # promoted from booster → standalone signal source
|
||||
|
||||
|
||||
# Cooldown via scanner_state.in_cooldown — DB-backed, restart-safe.
|
||||
|
||||
|
||||
# ─── Signal logic ───────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _detect_cadence_hours(funding: list[dict]) -> float:
|
||||
"""Infer the funding cadence (hours between cycles) from the response.
|
||||
|
||||
Binance returns ~8h intervals; HL returns ~1h. We can't assume — the
|
||||
response itself is the source of truth. Average over the most recent 10
|
||||
intervals to smooth out occasional gaps.
|
||||
"""
|
||||
if len(funding) < 2:
|
||||
return 8.0 # safe Binance default
|
||||
sample = funding[-min(11, len(funding)):]
|
||||
diffs = [sample[i]["time_ms"] - sample[i - 1]["time_ms"] for i in range(1, len(sample))]
|
||||
if not diffs:
|
||||
return 8.0
|
||||
return statistics.fmean(diffs) / 3_600_000.0 # ms → hours
|
||||
|
||||
|
||||
MIN_COVERAGE_DAYS = 14 # below this, the signal is statistically too noisy
|
||||
|
||||
|
||||
def evaluate_funding_reversal(
|
||||
funding_history: list[dict],
|
||||
daily_candles: list[dict],
|
||||
) -> tuple[bool, dict]:
|
||||
"""Pure function. Returns (is_signal, debug).
|
||||
Debug contains `direction` key on real signals — 'buy' or 'short'.
|
||||
|
||||
Cross-venue safety: HL caps fundingHistory at 500 rows (~20.8 days for
|
||||
1h cadence), Binance can give a full 30 days. We compute the actual
|
||||
coverage span from the data and scale the cumulative-funding threshold
|
||||
proportionally — so 3% over 30 days and 2.08% over 20.8 days are
|
||||
treated as equivalent in daily-average terms.
|
||||
"""
|
||||
if not funding_history or len(funding_history) < 2:
|
||||
return False, {"reason": "insufficient_funding_history",
|
||||
"cycles": len(funding_history)}
|
||||
|
||||
cadence_h = _detect_cadence_hours(funding_history)
|
||||
|
||||
# Actual time span the response covers — bounded by HL's 500-row cap
|
||||
# for HL, or by what we asked for on Binance.
|
||||
span_ms = funding_history[-1]["time_ms"] - funding_history[0]["time_ms"]
|
||||
actual_days = span_ms / 86_400_000
|
||||
|
||||
if actual_days < MIN_COVERAGE_DAYS:
|
||||
return False, {"reason": "insufficient_coverage_days",
|
||||
"actual_days": round(actual_days, 1),
|
||||
"min_required": MIN_COVERAGE_DAYS}
|
||||
|
||||
rates = [f["rate"] for f in funding_history]
|
||||
cumulative_funding = sum(rates)
|
||||
avg_full_window = statistics.fmean(rates)
|
||||
|
||||
# Scale the extreme threshold by ACTUAL coverage so cross-venue results
|
||||
# are comparable. Binance: 30d → threshold ×1. HL: 20.8d → threshold ×0.69.
|
||||
scaled_threshold = FUNDING_EXTREME_THRESHOLD * (actual_days / FUNDING_HISTORY_DAYS)
|
||||
|
||||
# "Recent" lookback in TIME units, not cycles. 24h of cycles regardless
|
||||
# of venue. Min 3 to avoid pathological cases (very short history).
|
||||
recent_cycles = max(3, int(FUNDING_REVERSAL_LOOKBACK_HOURS / cadence_h))
|
||||
recent_cycles = min(recent_cycles, len(rates)) # don't over-slice
|
||||
avg_recent_cycle = statistics.fmean(rates[-recent_cycles:])
|
||||
|
||||
# 2. Identify direction (which side is overcrowded)
|
||||
if cumulative_funding > scaled_threshold:
|
||||
# Longs have been paying → expect SHORT squeeze
|
||||
direction = "short"
|
||||
elif cumulative_funding < -scaled_threshold:
|
||||
# Shorts have been paying → expect LONG rally
|
||||
direction = "buy"
|
||||
else:
|
||||
return False, {
|
||||
"reason": "no_extreme",
|
||||
"cumulative_funding_pct": round(cumulative_funding * 100, 3),
|
||||
"scaled_threshold_pct": round(scaled_threshold * 100, 3),
|
||||
"coverage_days": round(actual_days, 1),
|
||||
}
|
||||
|
||||
# 3. Funding must be MEAN-REVERTING (recent cycles softer than 30d avg)
|
||||
# Use absolute magnitude — what matters is "the pressure is easing",
|
||||
# not the direction of zero-crossing.
|
||||
if abs(avg_full_window) == 0:
|
||||
return False, {"reason": "degenerate_avg"}
|
||||
revert_ratio = abs(avg_recent_cycle) / abs(avg_full_window)
|
||||
if revert_ratio > FUNDING_REVERSAL_RATIO:
|
||||
return False, {
|
||||
"reason": "funding_still_extreme",
|
||||
"revert_ratio": round(revert_ratio, 2),
|
||||
"needed_below": FUNDING_REVERSAL_RATIO,
|
||||
"direction": direction,
|
||||
}
|
||||
|
||||
# 4. Price confirms the contrarian move (last 7 days)
|
||||
if not daily_candles or len(daily_candles) < PRICE_CONFIRM_DAYS + 1:
|
||||
return False, {"reason": "insufficient_price_history",
|
||||
"have": len(daily_candles), "need": PRICE_CONFIRM_DAYS + 1}
|
||||
|
||||
first_close = daily_candles[-(PRICE_CONFIRM_DAYS + 1)]["close"]
|
||||
last_close = daily_candles[-1]["close"]
|
||||
if first_close == 0:
|
||||
return False, {"reason": "bad_first_close"}
|
||||
pct_change = (last_close - first_close) / first_close * 100
|
||||
|
||||
if direction == "buy" and pct_change < PRICE_CONFIRM_PCT:
|
||||
return False, {"reason": "price_not_yet_recovering",
|
||||
"pct_7d": round(pct_change, 2), "direction": direction}
|
||||
if direction == "short" and pct_change > -PRICE_CONFIRM_PCT:
|
||||
return False, {"reason": "price_not_yet_falling",
|
||||
"pct_7d": round(pct_change, 2), "direction": direction}
|
||||
|
||||
boost = {
|
||||
"direction": direction,
|
||||
"cum_funding_30d_pct": round(cumulative_funding * 100, 3),
|
||||
"recent_avg_cycle_pct": round(avg_recent_cycle * 100, 4),
|
||||
"revert_ratio": round(revert_ratio, 2),
|
||||
"price_7d_pct": round(pct_change, 2),
|
||||
"trigger_close": round(last_close, 4),
|
||||
}
|
||||
if not EMIT_STANDALONE_SIGNALS:
|
||||
return False, {"reason": "boost_only", "boost": boost}
|
||||
return True, boost
|
||||
|
||||
|
||||
# ─── Standalone scanner plumbing ────────────────────────────────────────────
|
||||
|
||||
|
||||
async def _fetch_inputs() -> tuple[list[dict], list[dict]]:
|
||||
"""Pull funding history + daily candles for ASSET.
|
||||
|
||||
We bypass provider.fetch_1d in favor of fapi.binance.com/fapi/v1/klines
|
||||
because (a) funding lives on the futures venue anyway — same liquidity
|
||||
pool, same trade flow — and (b) the spot api.binance.com host is
|
||||
occasionally geo-blocked, while fapi is more reliably reachable.
|
||||
"""
|
||||
provider = for_asset(ASSET)
|
||||
funding = await provider.fetch_funding(ASSET, days=FUNDING_DAYS_LOOKBACK)
|
||||
|
||||
end_ms = int(datetime.now(timezone.utc).timestamp() * 1000)
|
||||
start_ms = end_ms - PRICE_DAYS_LOOKBACK * 24 * 3600 * 1000
|
||||
async with httpx.AsyncClient(timeout=20) as client:
|
||||
resp = await client.get(
|
||||
"https://fapi.binance.com/fapi/v1/klines",
|
||||
params={"symbol": f"{ASSET}USDT", "interval": "1d",
|
||||
"startTime": start_ms, "endTime": end_ms, "limit": 1000},
|
||||
)
|
||||
resp.raise_for_status()
|
||||
raw_daily = [
|
||||
{"time_ms": r[0], "open": float(r[1]), "high": float(r[2]),
|
||||
"low": float(r[3]), "close": float(r[4]), "volume": float(r[5])}
|
||||
for r in resp.json()
|
||||
]
|
||||
daily = drop_in_progress_bar(raw_daily, "1d")
|
||||
return funding, daily
|
||||
|
||||
|
||||
def _summary(debug: dict) -> str:
|
||||
direction = debug.get("direction", "?").upper()
|
||||
cum = debug.get("cum_funding_30d_pct")
|
||||
recent = debug.get("recent_avg_cycle_pct")
|
||||
revert = debug.get("revert_ratio")
|
||||
p7d = debug.get("price_7d_pct")
|
||||
return (
|
||||
f"BTC funding extreme reversal — {direction}. "
|
||||
f"30d cumulative funding {cum}% (crowded {'longs' if direction == 'SHORT' else 'shorts'}); "
|
||||
f"last-24h cycle avg {recent}% (revert ratio {revert}); "
|
||||
f"price 7d {p7d:+.2f}% confirms the unwind."
|
||||
)
|
||||
|
||||
|
||||
async def _emit_signal(debug: dict) -> bool:
|
||||
"""POST to the ingest endpoint. Idempotent via external_id (per-day)."""
|
||||
if not settings.ingest_api_key:
|
||||
logger.error("Funding reversal would fire but INGEST_API_KEY empty — not recorded")
|
||||
return False
|
||||
direction = debug["direction"] # 'buy' or 'short'
|
||||
expected_move = PAYLOAD_EXPECTED_MOVE if direction == "buy" else -PAYLOAD_EXPECTED_MOVE
|
||||
payload = {
|
||||
"source": "funding_reversal",
|
||||
"external_id": f"funding-{ASSET}-{direction}-{datetime.now(timezone.utc).strftime('%Y%m%d')}",
|
||||
"text": _summary(debug),
|
||||
"signal": direction,
|
||||
"target_asset": ASSET,
|
||||
"confidence": PAYLOAD_CONFIDENCE,
|
||||
"category": f"funding_reversal_{direction}",
|
||||
"expected_move_pct": expected_move,
|
||||
"invalidation_price": debug.get("trigger_close"),
|
||||
}
|
||||
async with httpx.AsyncClient(timeout=10) as client:
|
||||
resp = await client.post(
|
||||
"http://localhost:8000/api/signals/ingest",
|
||||
json=payload,
|
||||
headers={"X-Ingest-Key": settings.ingest_api_key},
|
||||
)
|
||||
if resp.status_code >= 400:
|
||||
logger.error("Funding reversal ingest failed (%d): %s",
|
||||
resp.status_code, resp.text[:200])
|
||||
return False
|
||||
logger.info("Funding reversal signal emitted: %s", resp.json())
|
||||
return True
|
||||
|
||||
|
||||
async def scan_once() -> None:
|
||||
"""Hourly tick. Idempotent (cooldown-gated, external_id-dedupe at ingest)."""
|
||||
if not scanner_state.is_enabled(SCANNER_NAME):
|
||||
logger.debug("Funding reversal scanner disabled — skipping")
|
||||
return
|
||||
if await scanner_state.in_cooldown("funding_reversal", ASSET, COOLDOWN_DAYS):
|
||||
logger.debug("Funding reversal cooldown active (%dd)", COOLDOWN_DAYS)
|
||||
scanner_state.record_run(SCANNER_NAME, "ok", "cooldown")
|
||||
return
|
||||
|
||||
try:
|
||||
funding, daily = await _fetch_inputs()
|
||||
fired, debug = evaluate_funding_reversal(funding, daily)
|
||||
except Exception as exc:
|
||||
# Always include exception type — httpx errors often have empty .args
|
||||
# which formatted as just "Funding reversal scan failed:" before.
|
||||
logger.exception("Funding reversal scan failed: %s (%s)",
|
||||
type(exc).__name__, exc)
|
||||
scanner_state.record_run(SCANNER_NAME, "error",
|
||||
f"{type(exc).__name__}: {exc}"[:200])
|
||||
return
|
||||
|
||||
if fired:
|
||||
logger.info("Funding reversal FIRE — %s", debug)
|
||||
emitted = await _emit_signal(debug)
|
||||
scanner_state.record_run(SCANNER_NAME, "fired" if emitted else "error",
|
||||
None if emitted else "ingest_failed")
|
||||
else:
|
||||
logger.info("Funding reversal no — %s", debug.get("reason"))
|
||||
scanner_state.record_run(SCANNER_NAME, "ok", debug.get("reason"))
|
||||
|
||||
|
||||
# ─── Read API helper — current snapshot for the BTC page tab ────────────────
|
||||
|
||||
|
||||
async def get_current_snapshot() -> dict:
|
||||
"""Live read for the frontend BTC page funding tab. Returns the latest
|
||||
funding rate, the 24h running average, cumulative 30d sum, and the verdict
|
||||
of evaluate_funding_reversal() against current data. Cheap to call — only
|
||||
network cost is two market_data fetches the scanner would do anyway."""
|
||||
try:
|
||||
funding, daily = await _fetch_inputs()
|
||||
except Exception as exc:
|
||||
logger.exception("funding snapshot fetch failed")
|
||||
return {"ok": False, "error": f"{type(exc).__name__}: {exc}"}
|
||||
|
||||
if not funding:
|
||||
return {"ok": False, "error": "no_funding_data"}
|
||||
|
||||
cadence_h = _detect_cadence_hours(funding)
|
||||
rates = [f["rate"] for f in funding]
|
||||
cum_30d_pct = sum(rates) * 100
|
||||
span_days = (funding[-1]["time_ms"] - funding[0]["time_ms"]) / 86_400_000
|
||||
latest = rates[-1] * 100
|
||||
# Last-24h equivalent average per-cycle rate (in %)
|
||||
recent_n = max(3, int(24 / cadence_h)) if cadence_h > 0 else 24
|
||||
recent_n = min(recent_n, len(rates))
|
||||
last_24h_avg = (sum(rates[-recent_n:]) / recent_n) * 100
|
||||
|
||||
fired, debug = evaluate_funding_reversal(funding, daily)
|
||||
|
||||
# 7-day funding history for the sparkline (truncate to keep payload small)
|
||||
history = [
|
||||
{"t": int(f["time_ms"]), "rate_pct": round(f["rate"] * 100, 5)}
|
||||
for f in funding[-int(min(len(funding), 24 * 7 / max(cadence_h, 0.5))) :]
|
||||
]
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"asset": ASSET,
|
||||
"cadence_hours": round(cadence_h, 2),
|
||||
"coverage_days": round(span_days, 1),
|
||||
"latest_rate_pct": round(latest, 5),
|
||||
"last_24h_avg_pct": round(last_24h_avg, 5),
|
||||
"cum_30d_pct": round(cum_30d_pct, 3),
|
||||
"extreme_threshold_pct": round(FUNDING_EXTREME_THRESHOLD * 100, 3),
|
||||
"signal_fired": fired,
|
||||
"debug": debug,
|
||||
"history": history,
|
||||
}
|
||||
Reference in New Issue
Block a user