Pre-launch hardening: KOL module, Telegram, scanners, WS resilience
Big-picture changes since b941223:
KOL pipeline (new) — Substack/podcast/blog RSS → AI ticker extraction →
on-chain wallet diff → talks-vs-trades divergence detection. Daily polls,
19 feeds, divergence emits Post + Telegram fan-out.
Telegram push (new) — walletless free tier + wallet-linked Pro upgrade,
in-bot preference commands (/trump /btc /funding /kol /conf /quiet),
signed-envelope API for dashboard. Disconnect-wallet keeps free
subscription.
BTC funding-rate reversal scanner (new) — hourly cron, 30d cumulative
funding threshold + mean-revert + 7d price confirm, emits via
/api/signals/ingest. BTC bottom-reversal scanner promoted to System 2.
WS broadcast rewrite — per-client send timeout + parallel fan-out
(asyncio.gather). Fixes "Binance WS no close frame" reconnect storms +
APScheduler 11-min job misses, both caused by one slow client stalling
the kline loop.
Error visibility — three silent-error sites (trumpstruth/truth_social
fetchers, funding_reversal scanner) now include exception type name so
httpx ConnectError-style empty-message errors stop logging blank lines.
Telegram bot loop now classifies ReadTimeout vs network vs unknown +
logger.exception for the unknown bucket.
Security hygiene — trumpsignal.db untracked from git (held subscriber
wallets + encrypted HL keys + 22 bot trades); .gitignore now blocks
*.db/.next/backups. CORS only allows FRONTEND_URL in production.
New ops scripts —
- scripts/preflight.py: env/DB/Telegram/AI auth verification gate
- scripts/backup_db.sh: cron-friendly daily DB backup (SQLite + Postgres)
- scripts/seed_kol_wallets.py: idempotent KOL on-chain wallet seeder
15 new Alembic migrations (007-021) covering convex strategy fields,
phase-1 safety, two-system frozen exits, invalidation prices, dynamic
SYS2 leverage, staged de-risk + pyramiding, peak gain tracking, risk
mode, auto-trade + grow flags, KOL module, KOL on-chain, KOL divergence,
Telegram bindings + walletless.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
k
@@ -0,0 +1,177 @@
|
||||
"""On-chain metrics for the BTC bottom-reversal scanner.
|
||||
|
||||
Two providers, same interface (fetch_mvrv_z_score / fetch_sth_sopr → list[float]):
|
||||
|
||||
BitcoinDataClient — DEFAULT. bitcoin-data.com public API. FREE, no key.
|
||||
Returns PRE-COMPUTED MVRV-Z and STH-SOPR (we don't
|
||||
need realized-cap or local math — realized cap is an
|
||||
on-chain quantity that CANNOT be derived from price
|
||||
data, so a vendor is unavoidable; this is the free one).
|
||||
Free tier: 10 requests/HOUR. The scanner runs once a
|
||||
day and needs 2 calls — well within budget — but we
|
||||
add a 6-hour in-process cache as a safety net against
|
||||
restarts / manual re-runs hammering the limit.
|
||||
|
||||
GlassnodeClient — Optional paid fallback. Used automatically if
|
||||
GLASSNODE_API_KEY is set (higher resolution / SLA).
|
||||
|
||||
get_onchain_client() picks the right one. btc_bottom_reversal.py only
|
||||
depends on the interface, never the concrete class.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
from typing import Optional
|
||||
|
||||
import httpx
|
||||
|
||||
from app.config import settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# ─── Provider 1: bitcoin-data.com (free, default) ───────────────────────────
|
||||
|
||||
BITCOIN_DATA_BASE = "https://bitcoin-data.com/v1"
|
||||
|
||||
# Endpoint → (path, json value key). Both return a full daily history list
|
||||
# of {"d","unixTs",<key>} when called with no query params.
|
||||
_BD_ENDPOINTS = {
|
||||
"mvrv_z": ("/mvrv-zscore", "mvrvZscore"),
|
||||
"sth_sopr": ("/sth-sopr", "sthSopr"),
|
||||
}
|
||||
|
||||
# Fresh-fetch window: don't re-hit the API if the on-disk copy is younger
|
||||
# than this. The scanner runs daily; 18h keeps it to ~1 fetch/day/metric
|
||||
# (2 req/day total, vs the 10 req/HOUR free cap).
|
||||
_BD_FRESH_S = 18 * 3600
|
||||
# How stale on-disk data may be and still be USABLE when the API is down /
|
||||
# rate-limited. A daily MVRV-Z / STH-SOPR barely moves over 2 days, so
|
||||
# serving 48h-old data beats skipping a scan or, worse, acting on nothing.
|
||||
_BD_STALE_OK_S = 48 * 3600
|
||||
|
||||
# Disk cache survives restarts — critical given the tight free rate limit.
|
||||
_BD_CACHE_DIR = os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(__file__))), ".cache")
|
||||
_BD_CACHE_FILE = os.path.join(_BD_CACHE_DIR, "onchain_bitcoin_data.json")
|
||||
|
||||
|
||||
def _load_disk_cache() -> dict:
|
||||
try:
|
||||
with open(_BD_CACHE_FILE) as f:
|
||||
return json.load(f)
|
||||
except (FileNotFoundError, json.JSONDecodeError, OSError):
|
||||
return {}
|
||||
|
||||
|
||||
def _save_disk_cache(cache: dict) -> None:
|
||||
try:
|
||||
os.makedirs(_BD_CACHE_DIR, exist_ok=True)
|
||||
tmp = _BD_CACHE_FILE + ".tmp"
|
||||
with open(tmp, "w") as f:
|
||||
json.dump(cache, f)
|
||||
os.replace(tmp, _BD_CACHE_FILE) # atomic
|
||||
except OSError as exc:
|
||||
logger.warning("onchain disk-cache write failed: %s", exc)
|
||||
|
||||
|
||||
class BitcoinDataClient:
|
||||
"""Free, key-less. Returns pre-computed metric series (already the
|
||||
Z-score / SOPR value — no local computation needed). Disk-cached so the
|
||||
10 req/hour free cap and process restarts can't starve the scanner."""
|
||||
|
||||
async def _series(self, metric: str, days: int) -> list[float]:
|
||||
now = time.time()
|
||||
cache = _load_disk_cache()
|
||||
entry = cache.get(metric) # {"ts": epoch, "values": [...]}
|
||||
age = (now - entry["ts"]) if entry else None
|
||||
|
||||
# Fresh enough → no network call at all.
|
||||
if entry and age is not None and age < _BD_FRESH_S:
|
||||
vals = entry["values"]
|
||||
return vals[-days:] if days else vals
|
||||
|
||||
path, key = _BD_ENDPOINTS[metric]
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=20) as client:
|
||||
resp = await client.get(f"{BITCOIN_DATA_BASE}{path}")
|
||||
if resp.status_code == 429:
|
||||
raise RuntimeError("rate_limited")
|
||||
resp.raise_for_status()
|
||||
rows = resp.json() or []
|
||||
values = [float(r[key]) for r in rows if r.get(key) is not None]
|
||||
if not values:
|
||||
raise RuntimeError("empty_response")
|
||||
cache[metric] = {"ts": now, "values": values}
|
||||
_save_disk_cache(cache)
|
||||
return values[-days:] if days else values
|
||||
except Exception as exc:
|
||||
# Network/limit failure: fall back to disk if it's still usable.
|
||||
if entry and age is not None and age < _BD_STALE_OK_S:
|
||||
logger.warning("bitcoin-data.com %s fetch failed (%s) — "
|
||||
"serving disk cache aged %.1fh",
|
||||
metric, exc, age / 3600)
|
||||
vals = entry["values"]
|
||||
return vals[-days:] if days else vals
|
||||
raise RuntimeError(
|
||||
f"bitcoin-data.com {metric} unavailable ({exc}) and no usable cache"
|
||||
) from exc
|
||||
|
||||
async def fetch_mvrv_z_score(self, asset: str = "BTC", days: int = 730) -> list[float]:
|
||||
if asset.upper() != "BTC":
|
||||
return [] # bitcoin-data.com is BTC-only
|
||||
return await self._series("mvrv_z", days)
|
||||
|
||||
async def fetch_sth_sopr(self, asset: str = "BTC", days: int = 30) -> list[float]:
|
||||
if asset.upper() != "BTC":
|
||||
return []
|
||||
return await self._series("sth_sopr", days)
|
||||
|
||||
|
||||
# ─── Provider 2: Glassnode (paid, optional) ─────────────────────────────────
|
||||
|
||||
GLASSNODE_BASE_URL = "https://api.glassnode.com/v1/metrics"
|
||||
|
||||
|
||||
class GlassnodeClient:
|
||||
def __init__(self, api_key: Optional[str] = None):
|
||||
self.api_key = api_key if api_key is not None else settings.glassnode_api_key
|
||||
|
||||
async def _get_series(self, path: str, asset: str, days: int) -> list[dict]:
|
||||
if not self.api_key:
|
||||
raise RuntimeError("GLASSNODE_API_KEY is not configured")
|
||||
end = int(datetime.now(timezone.utc).timestamp())
|
||||
start = end - days * 86_400
|
||||
async with httpx.AsyncClient(timeout=20) as client:
|
||||
resp = await client.get(
|
||||
f"{GLASSNODE_BASE_URL}{path}",
|
||||
params={"a": asset, "api_key": self.api_key,
|
||||
"s": start, "u": end, "i": "24h"},
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json() or []
|
||||
|
||||
async def fetch_mvrv_z_score(self, asset: str = "BTC", days: int = 730) -> list[float]:
|
||||
rows = await self._get_series("/market/mvrv_z_score", asset, days)
|
||||
return [float(r["v"]) for r in rows if r.get("v") is not None]
|
||||
|
||||
async def fetch_sth_sopr(self, asset: str = "BTC", days: int = 30) -> list[float]:
|
||||
rows = await self._get_series("/indicators/sopr_less_155", asset, days)
|
||||
return [float(r["v"]) for r in rows if r.get("v") is not None]
|
||||
|
||||
|
||||
# ─── Factory ────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def get_onchain_client():
|
||||
"""Glassnode if a key is configured (paid, higher SLA), else the free
|
||||
bitcoin-data.com client. Both expose the same async interface."""
|
||||
if settings.glassnode_api_key:
|
||||
logger.info("On-chain provider: Glassnode (paid key configured)")
|
||||
return GlassnodeClient()
|
||||
logger.info("On-chain provider: bitcoin-data.com (free, no key)")
|
||||
return BitcoinDataClient()
|
||||
Reference in New Issue
Block a user