Z/trumpsignal-backend
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Pre-launch hardening: KOL module, Telegram, scanners, WS resilience

Browse Source 
Big-picture changes since b941223:

KOL pipeline (new) — Substack/podcast/blog RSS → AI ticker extraction →
on-chain wallet diff → talks-vs-trades divergence detection. Daily polls,
19 feeds, divergence emits Post + Telegram fan-out.

Telegram push (new) — walletless free tier + wallet-linked Pro upgrade,
in-bot preference commands (/trump /btc /funding /kol /conf /quiet),
signed-envelope API for dashboard. Disconnect-wallet keeps free
subscription.

BTC funding-rate reversal scanner (new) — hourly cron, 30d cumulative
funding threshold + mean-revert + 7d price confirm, emits via
/api/signals/ingest. BTC bottom-reversal scanner promoted to System 2.

WS broadcast rewrite — per-client send timeout + parallel fan-out
(asyncio.gather). Fixes "Binance WS no close frame" reconnect storms +
APScheduler 11-min job misses, both caused by one slow client stalling
the kline loop.

Error visibility — three silent-error sites (trumpstruth/truth_social
fetchers, funding_reversal scanner) now include exception type name so
httpx ConnectError-style empty-message errors stop logging blank lines.
Telegram bot loop now classifies ReadTimeout vs network vs unknown +
logger.exception for the unknown bucket.

Security hygiene — trumpsignal.db untracked from git (held subscriber
wallets + encrypted HL keys + 22 bot trades); .gitignore now blocks
*.db/.next/backups. CORS only allows FRONTEND_URL in production.

New ops scripts —
  - scripts/preflight.py: env/DB/Telegram/AI auth verification gate
  - scripts/backup_db.sh: cron-friendly daily DB backup (SQLite + Postgres)
  - scripts/seed_kol_wallets.py: idempotent KOL on-chain wallet seeder

15 new Alembic migrations (007-021) covering convex strategy fields,
phase-1 safety, two-system frozen exits, invalidation prices, dynamic
SYS2 leverage, staged de-risk + pyramiding, peak gain tracking, risk
mode, auto-trade + grow flags, KOL module, KOL on-chain, KOL divergence,
Telegram bindings + walletless.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
k
2026-05-25 00:52:56 +08:00
parent b941223c88
commit 5fb1d52026
81 changed files with 13251 additions and 158 deletions
Download Patch File Download Diff File Expand all files Collapse all files
app/api/dev.py
+158
View File
@@ -57,9 +57,167 @@ async def insert_fake_post(
return {"id": post.id, "text": post.text[:80]}
@router.post("/dev/fake-signal")
async def inject_fake_signal(
symbol: str = "ETHUSDT",
close: float = 1850.42,
tbr: float = 0.72,
vol_mult: float = 3.1,
bb_pct: float = 8.5,
btc_trend: str = "↑ uptrend",
):
"""Inject a synthetic funding_signal for end-to-end testing."""
from datetime import datetime, timezone
from app.services import funding_signal as fs
now = datetime.now(timezone.utc)
alert = {
"type": "funding_signal",
"symbol": symbol,
"time": now.isoformat(),
"close": close,
"tbr": tbr,
"vol_mult": vol_mult,
"bb_pct": bb_pct,
"bb_upper": round(close * 1.002, 4),
"btc_trend": btc_trend,
"enabled": fs.is_enabled(),
}
fs._recent_signals.append(alert)
if fs.is_enabled():
await manager.broadcast(alert)
return {"status": "broadcast", "alert": alert}
else:
return {"status": "recorded_only (monitor OFF)", "alert": alert}
@router.post("/dev/reanalyze")
async def trigger_reanalyze(
background_tasks: BackgroundTasks,
limit: int = 500,
dry_run: bool = False,
delay_secs: float = 0.5,
legacy_signals: bool = False,
model: str = "",
):
"""
Batch re-run AI analysis on unscored posts.
model: override the AI model (default: ai_live_model=flash for speed).
Runs in the background — poll GET /dev/reanalyze/status for progress.
"""
from app.services.reanalyze import reanalyze_unscored, get_state
from app.config import settings as _s
state = get_state()
if state["running"]:
return {"status": "already_running", "state": state}
effective_model = model or _s.ai_live_model # default to flash
background_tasks.add_task(
reanalyze_unscored, AsyncSessionLocal,
limit=limit, dry_run=dry_run, delay_secs=delay_secs,
legacy_signals=legacy_signals, model=effective_model,
)
return {
"status": "started",
"limit": limit,
"dry_run": dry_run,
"delay_secs": delay_secs,
"legacy_signals": legacy_signals,
"model": effective_model,
"message": f"Re-analyzing up to {limit} posts in background. Poll /api/dev/reanalyze/status.",
}
@router.get("/dev/reanalyze/status")
async def reanalyze_status():
"""Current progress of the background re-analyzer."""
from app.services.reanalyze import get_state
return get_state()
@router.post("/dev/backfill-prices")
async def trigger_price_backfill(background_tasks: BackgroundTasks, asset: str = "BTC"):
"""触发历史帖子价格回溯（后台运行）"""
from app.services.price_backfill import backfill_price_impact
background_tasks.add_task(backfill_price_impact, AsyncSessionLocal, asset)
return {"status": "started", "asset": asset, "message": "后台回溯中，约需 1-2 分钟"}
# ─── Convex-strategy backtest ──────────────────────────────────────────────
@router.post("/dev/backtest/post")
async def backtest_one_post(
post_id: int,
stop_loss_pct: float = 1.5,
trailing_stop_pct: float = 2.5,
trailing_activate_at_pct: float = 5.0,
take_profit_pct: float = 0.0, # 0 = treat as None (no fixed TP)
max_hold_hours: int = 168,
):
"""Replay one post through the new convex-strategy exit rules.
Pulls 1m candles from Binance for the post's hold window and simulates
trailing-stop + SL + max-hold. Useful for sanity-checking parameters.
Pass `take_profit_pct=0` to disable the fixed TP and run pure trailing.
"""
from app.services.backtest import backtest_post, BacktestParams
params = BacktestParams(
stop_loss_pct=stop_loss_pct,
trailing_stop_pct=trailing_stop_pct or None,
trailing_activate_at_pct=trailing_activate_at_pct or None,
take_profit_pct=take_profit_pct or None,
max_hold_hours=max_hold_hours,
)
r = await backtest_post(post_id, params)
if r is None:
return {"status": "skipped", "post_id": post_id}
return {"status": "ok", "result": r.to_dict()}
@router.post("/dev/paper-mode")
async def toggle_paper_mode(
wallet: str,
enabled: bool,
db: AsyncSession = Depends(get_db),
):
"""Flip paper_mode for a subscription. Dev-only — no signature required.
Paper mode: trades are simulated end-to-end (entry/exit from Binance,
DB row with hl_order_id='paper') but no Hyperliquid call is made.
"""
from sqlalchemy import select
from app.models import Subscription
wallet = wallet.lower().strip()
result = await db.execute(select(Subscription).where(Subscription.wallet_address == wallet))
sub = result.scalar_one_or_none()
if sub is None:
return {"status": "not_found", "wallet": wallet}
sub.paper_mode = bool(enabled)
await db.commit()
return {"status": "ok", "wallet": wallet, "paper_mode": sub.paper_mode}
@router.post("/dev/backtest/batch")
async def backtest_batch_route(
limit: int = 50,
min_confidence: int = 80,
stop_loss_pct: float = 1.5,
trailing_stop_pct: float = 2.5,
trailing_activate_at_pct: float = 5.0,
take_profit_pct: float = 0.0,
max_hold_hours: int = 168,
):
"""Batch backtest: every directional post with conf ≥ min_confidence.
WARNING: synchronous — for 50 posts at ~10s each on Binance fetches this
can take several minutes. Run with limit=5 first to sanity-check.
"""
from app.services.backtest import backtest_batch, BacktestParams
params = BacktestParams(
stop_loss_pct=stop_loss_pct,
trailing_stop_pct=trailing_stop_pct or None,
trailing_activate_at_pct=trailing_activate_at_pct or None,
take_profit_pct=take_profit_pct or None,
max_hold_hours=max_hold_hours,
)
return await backtest_batch(limit=limit, min_confidence=min_confidence, params=params)
app/api/funding_reversal.py
+22
View File
@@ -0,0 +1,22 @@
"""
API endpoints for the BTC funding-rate reversal signal.
GET /api/funding/snapshot
Live snapshot of current funding state + signal verdict + 7d history.
Powers the BTC page "Funding" tab.
Historical fired signals are returned via the standard /api/posts endpoint
with source=funding_reversal — same plumbing as btc_bottom_reversal.
"""
from fastapi import APIRouter
from app.services.scanners.funding_reversal import get_current_snapshot
router = APIRouter(prefix="/funding", tags=["funding"])
@router.get("/snapshot")
async def snapshot() -> dict:
"""Cheap (≤2 outbound requests). Frontend polls every ~5min."""
return await get_current_snapshot()
app/api/funding_signal.py
+38
View File
@@ -0,0 +1,38 @@
"""
API endpoints for the breakout signal monitor.
GET /api/signal/status — current state (enabled, recent signals)
POST /api/signal/toggle — flip the on/off switch
GET /api/signal/history — last N signals (fired regardless of enabled state)
"""
from fastapi import APIRouter
from app.services.funding_signal import (
set_enabled,
is_enabled,
get_recent_signals,
get_status,
)
router = APIRouter(prefix="/signal", tags=["signal"])
@router.get("/status")
async def status():
return get_status()
@router.post("/toggle")
async def toggle(enabled: bool):
"""
Body: ?enabled=true or ?enabled=false
Example: POST /api/signal/toggle?enabled=true
"""
set_enabled(enabled)
return {"enabled": is_enabled()}
@router.get("/history")
async def history(limit: int = 20):
return get_recent_signals(limit)
app/api/kol.py
+395
View File
@@ -0,0 +1,395 @@
"""KOL module — public read API.
Endpoints:
GET /api/kol/posts list KOL posts (newest first), with summary +
tickers but WITHOUT raw_text to keep payload small
GET /api/kol/posts/{id} full detail incl. raw_text for the modal/page view
"""
import json
import logging
from collections import defaultdict
from datetime import datetime, timedelta, timezone
from typing import Any, List, Optional
from fastapi import APIRouter, Depends, Header, HTTPException, Query
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.config import settings
from app.database import get_db
from app.models import KolDivergence, KolHoldingChange, KolHoldingSnapshot, KolPost, KolWallet, iso_utc
def _verify_admin_key(x_ingest_key: Optional[str]) -> None:
"""Shared-secret auth for write endpoints (wallet add, scan trigger).
Same key as the signal ingest endpoint. Fail-closed if INGEST_API_KEY unset."""
expected = settings.ingest_api_key
if not expected:
raise HTTPException(503, "write endpoint disabled (INGEST_API_KEY not configured)")
if not x_ingest_key:
raise HTTPException(401, "missing X-Ingest-Key header")
if x_ingest_key != expected:
raise HTTPException(401, "invalid X-Ingest-Key")
router = APIRouter()
logger = logging.getLogger(__name__)
def _parse_tickers(raw: Optional[str]) -> List[dict]:
if not raw:
return []
try:
data = json.loads(raw)
return data if isinstance(data, list) else []
except json.JSONDecodeError:
return []
def _summary_dto(post: KolPost) -> dict:
"""List-view shape: no raw_text, no content_hash."""
return {
"id": post.id,
"kol_handle": post.kol_handle,
"source": post.source,
"url": post.url,
"title": post.title,
"published_at": iso_utc(post.published_at),
"summary": post.summary,
"tickers": _parse_tickers(post.tickers_json),
"analyzed_at": iso_utc(post.analyzed_at),
"analysis_model": post.analysis_model,
}
def _detail_dto(post: KolPost) -> dict:
base = _summary_dto(post)
base["raw_text"] = post.raw_text
return base
@router.get("/kol/posts")
async def list_kol_posts(
handle: Optional[str] = Query(default=None, description="filter by kol_handle"),
source: Optional[str] = Query(default=None, description="substack | twitter"),
limit: int = Query(default=50, ge=1, le=200),
page: int = Query(default=1, ge=1),
db: AsyncSession = Depends(get_db),
) -> dict[str, Any]:
stmt = select(KolPost)
if handle:
stmt = stmt.where(KolPost.kol_handle == handle)
if source:
stmt = stmt.where(KolPost.source == source)
stmt = stmt.order_by(KolPost.published_at.desc()).offset((page - 1) * limit).limit(limit)
rows = (await db.execute(stmt)).scalars().all()
return {"items": [_summary_dto(p) for p in rows], "page": page, "limit": limit}
@router.get("/kol/posts/{post_id}")
async def get_kol_post(post_id: int, db: AsyncSession = Depends(get_db)) -> dict:
row = (await db.execute(select(KolPost).where(KolPost.id == post_id))).scalar_one_or_none()
if row is None:
raise HTTPException(status_code=404, detail="KOL post not found")
return _detail_dto(row)
# ── Action ordering for picking the "dominant" call per ticker ─────────
# When the same ticker appears across multiple posts with different actions,
# the strongest signal wins. buy/sell (explicit position) outrank
# bullish/bearish (directional view) outrank mention (passing reference).
_ACTION_STRENGTH = {"buy": 3, "sell": 3, "bullish": 2, "bearish": 2, "mention": 1}
_ACTION_SIDE = {"buy": "long", "bullish": "long",
"sell": "short", "bearish": "short", "mention": "neutral"}
@router.get("/kol/digest")
async def kol_digest(
days: int = Query(default=7, ge=1, le=90,
description="rolling window in days"),
db: AsyncSession = Depends(get_db),
) -> dict[str, Any]:
"""Aggregate ticker calls across recent KOL posts.
For each ticker mentioned in posts within the window, returns the
dominant action, the highest conviction, and the list of (KOL, post,
action, conviction) calls. Sorted by post_count desc, then conviction.
"""
since = datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(days=days)
rows = (await db.execute(
select(KolPost)
.where(KolPost.published_at >= since)
.where(KolPost.tickers_json.is_not(None))
.order_by(KolPost.published_at.desc())
)).scalars().all()
# ticker -> list of call dicts
bucket: dict[str, list[dict]] = defaultdict(list)
for post in rows:
try:
tickers = json.loads(post.tickers_json or "[]")
except json.JSONDecodeError:
continue
for t in tickers:
sym = (t.get("ticker") or "").upper()
action = (t.get("action") or "mention").lower()
if not sym or action == "mention":
# Don't surface 'mention' on the digest — too noisy. Users
# can still see it on the post detail.
continue
bucket[sym].append({
"post_id": post.id,
"kol_handle": post.kol_handle,
"post_title": post.title,
"published_at": iso_utc(post.published_at),
"action": action,
"conviction": float(t.get("conviction") or 0.0),
"quote": (t.get("quote") or "")[:240],
})
tickers_out: list[dict] = []
for sym, calls in bucket.items():
# Dominant action: pick the action with the highest sum of conviction,
# broken by strength tier then count. Ensures one strong "buy" beats
# three weak "bullish" mentions.
action_weights: dict[str, float] = defaultdict(float)
for c in calls:
action_weights[c["action"]] += c["conviction"]
dominant = max(
action_weights.items(),
key=lambda kv: (_ACTION_STRENGTH.get(kv[0], 0), kv[1]),
)[0]
max_conv = max(c["conviction"] for c in calls)
unique_kols = sorted({c["kol_handle"] for c in calls})
tickers_out.append({
"ticker": sym,
"dominant_action": dominant,
"side": _ACTION_SIDE.get(dominant, "neutral"),
"max_conviction": round(max_conv, 2),
"post_count": len(calls),
"kol_count": len(unique_kols),
"kols": unique_kols,
"calls": sorted(calls, key=lambda c: c["conviction"], reverse=True),
})
tickers_out.sort(
key=lambda t: (t["kol_count"], t["post_count"], t["max_conviction"]),
reverse=True,
)
return {
"window_days": days,
"since": iso_utc(since),
"post_count": len(rows),
"ticker_count": len(tickers_out),
"tickers": tickers_out,
}
# ── A-tier: on-chain wallets + holdings ──────────────────────────────────────
def _wallet_dto(w: KolWallet) -> dict:
return {
"id": w.id,
"handle": w.handle,
"chain": w.chain,
"address": w.address,
"label": w.label,
"source_url": w.source_url,
"active": w.active,
"added_at": iso_utc(w.added_at),
}
def _snapshot_dto(s: KolHoldingSnapshot) -> dict:
return {
"id": s.id,
"wallet_id": s.wallet_id,
"snapshot_date": s.snapshot_date,
"holdings": json.loads(s.holdings_json or "[]"),
"total_usd": s.total_usd,
"source": s.source,
"created_at": iso_utc(s.created_at),
}
def _change_dto(c: KolHoldingChange, handle: str) -> dict:
return {
"id": c.id,
"wallet_id": c.wallet_id,
"handle": handle,
"detected_at": iso_utc(c.detected_at),
"ticker": c.ticker,
"change_type": c.change_type,
"usd_before": c.usd_before,
"usd_after": c.usd_after,
"pct_change": c.pct_change,
}
@router.get("/kol/wallets")
async def list_kol_wallets(
handle: Optional[str] = Query(default=None),
db: AsyncSession = Depends(get_db),
) -> dict[str, Any]:
stmt = select(KolWallet).where(KolWallet.active == True)
if handle:
stmt = stmt.where(KolWallet.handle == handle)
stmt = stmt.order_by(KolWallet.handle)
rows = (await db.execute(stmt)).scalars().all()
return {"wallets": [_wallet_dto(w) for w in rows]}
@router.post("/kol/wallets")
async def add_kol_wallet(
body: dict,
x_ingest_key: Optional[str] = Header(None, alias="X-Ingest-Key"),
db: AsyncSession = Depends(get_db),
) -> dict:
"""Add a new KOL wallet address. Body: {handle, chain, address, label?, source_url?}.
Auth: requires X-Ingest-Key header matching INGEST_API_KEY."""
_verify_admin_key(x_ingest_key)
required = {"handle", "chain", "address"}
if not required.issubset(body):
raise HTTPException(status_code=422, detail=f"Required fields: {required}")
# Check duplicate
existing = (await db.execute(
select(KolWallet).where(
KolWallet.chain == body["chain"],
KolWallet.address == body["address"].lower(),
)
)).scalar_one_or_none()
if existing:
raise HTTPException(status_code=409, detail="Wallet already tracked")
wallet = KolWallet(
handle=body["handle"],
chain=body["chain"],
address=body["address"].lower(),
label=body.get("label"),
source_url=body.get("source_url"),
)
db.add(wallet)
await db.commit()
await db.refresh(wallet)
return _wallet_dto(wallet)
@router.get("/kol/wallets/{wallet_id}/snapshots")
async def get_wallet_snapshots(
wallet_id: int,
limit: int = Query(default=30, ge=1, le=90),
db: AsyncSession = Depends(get_db),
) -> dict[str, Any]:
wallet = (await db.execute(
select(KolWallet).where(KolWallet.id == wallet_id)
)).scalar_one_or_none()
if not wallet:
raise HTTPException(status_code=404, detail="Wallet not found")
snaps = (await db.execute(
select(KolHoldingSnapshot)
.where(KolHoldingSnapshot.wallet_id == wallet_id)
.order_by(KolHoldingSnapshot.snapshot_date.desc())
.limit(limit)
)).scalars().all()
return {
"wallet": _wallet_dto(wallet),
"snapshots": [_snapshot_dto(s) for s in snaps],
}
@router.get("/kol/changes")
async def list_holding_changes(
handle: Optional[str] = Query(default=None),
days: int = Query(default=7, ge=1, le=90),
db: AsyncSession = Depends(get_db),
) -> dict[str, Any]:
"""Recent on-chain position changes across all tracked KOL wallets."""
since = datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(days=days)
stmt = (
select(KolHoldingChange, KolWallet.handle)
.join(KolWallet, KolHoldingChange.wallet_id == KolWallet.id)
.where(KolHoldingChange.detected_at >= since)
)
if handle:
stmt = stmt.where(KolWallet.handle == handle)
stmt = stmt.order_by(KolHoldingChange.detected_at.desc()).limit(200)
rows = (await db.execute(stmt)).all()
changes = [_change_dto(c, h) for c, h in rows]
return {
"window_days": days,
"since": iso_utc(since),
"count": len(changes),
"changes": changes,
}
# ── Talks-vs-trades divergence ────────────────────────────────────────────────
def _divergence_dto(d: KolDivergence) -> dict:
return {
"id": d.id,
"handle": d.handle,
"ticker": d.ticker,
"signal_type": d.signal_type, # divergence | alignment
"direction": d.direction, # long | short
"post_id": d.post_id,
"post_action": d.post_action,
"post_conviction":d.post_conviction,
"post_at": iso_utc(d.post_at),
"onchain_action": d.onchain_action,
"usd_before": d.usd_before,
"usd_after": d.usd_after,
"onchain_at": iso_utc(d.onchain_at),
"days_apart": d.days_apart,
"created_at": iso_utc(d.created_at),
}
@router.get("/kol/divergence")
async def list_divergence(
handle: Optional[str] = Query(default=None),
ticker: Optional[str] = Query(default=None),
signal_type: Optional[str] = Query(default=None, description="divergence | alignment"),
days: int = Query(default=30, ge=1, le=180),
db: AsyncSession = Depends(get_db),
) -> dict[str, Any]:
"""List talks-vs-trades cross-signal pairs.
signal_type=divergence → KOL said X but chain did the opposite (high alpha).
signal_type=alignment → KOL's words matched their on-chain action (reinforced signal).
"""
since = datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(days=days)
stmt = select(KolDivergence).where(KolDivergence.created_at >= since)
if handle:
stmt = stmt.where(KolDivergence.handle == handle)
if ticker:
stmt = stmt.where(KolDivergence.ticker == ticker.upper())
if signal_type:
stmt = stmt.where(KolDivergence.signal_type == signal_type)
stmt = stmt.order_by(KolDivergence.created_at.desc()).limit(200)
rows = (await db.execute(stmt)).scalars().all()
return {
"window_days": days,
"since": iso_utc(since),
"count": len(rows),
"items": [_divergence_dto(r) for r in rows],
}
@router.post("/kol/divergence/scan")
async def trigger_divergence_scan(
lookback_days: int = Query(default=30, ge=1, le=90),
x_ingest_key: Optional[str] = Header(None, alias="X-Ingest-Key"),
) -> dict[str, Any]:
"""Manually trigger the talks-vs-trades scan. Returns newly written pairs.
Auth: requires X-Ingest-Key header matching INGEST_API_KEY."""
_verify_admin_key(x_ingest_key)
from app.services.kol_divergence import run_divergence_scan
results = await run_divergence_scan(lookback_days=lookback_days)
return {"new_pairs": len(results), "items": results}
app/api/performance.py
+19 -2
View File
@@ -1,26 +1,43 @@
import logging
from datetime import datetime, timedelta, timezone
from fastapi import APIRouter, Depends
from fastapi import APIRouter, Depends, Query
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.database import get_db
from app.models import BotTrade
from app.schemas import BotPerformance
from app.services.signed_request import verify_signed_request
router = APIRouter()
logger = logging.getLogger(__name__)
PERIOD_DAYS = 30
ACTION_VIEW_PERFORMANCE = "view_performance"
@router.get("/performance", response_model=BotPerformance)
async def get_performance(db: AsyncSession = Depends(get_db)):
async def get_performance(
wallet: str = Query(..., description="Wallet address (lower-cased internally)"),
ts: int = Query(..., description="Signed timestamp (ms)"),
sig: str = Query(..., description="EIP-191 signature"),
db: AsyncSession = Depends(get_db),
):
wallet = wallet.lower().strip()
verify_signed_request(
action=ACTION_VIEW_PERFORMANCE,
wallet=wallet,
timestamp_ms=ts,
signature=sig,
body=None,
allow_replay=True,
)
since = datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(days=PERIOD_DAYS)
result = await db.execute(
select(BotTrade)
.where(BotTrade.wallet_address == wallet)
.where(BotTrade.closed_at.is_not(None))
.where(BotTrade.opened_at >= since)
.order_by(BotTrade.opened_at.asc())
app/api/positions.py
+392
View File
@@ -0,0 +1,392 @@
"""
Open positions + today's P&L — surface what every trader checks first.
The existing endpoints answer "what closed?" (trades) and "what's a signal?"
(posts). Neither answers "what do I currently hold?" — which is the very
first question on every trading dashboard.
This module exposes two reads for a connected wallet:
GET /api/positions/open?wallet=...
List every BotTrade with closed_at IS NULL for the wallet, enriched
with current price (from price_store) and unrealized PnL.
Works for both paper and real trades.
GET /api/positions/today?wallet=...
Realized P&L for trades closed since UTC midnight. Plus open count.
Cheap aggregate — drives the "today" tile on dashboards.
Both are no-auth (same trust model as /user/{wallet}/public). The wallet
address is the access key; anyone who knows it can read its state.
"""
from __future__ import annotations
import logging
from datetime import datetime, timezone
from typing import Optional
from fastapi import APIRouter, Depends, HTTPException, Query, Request
from pydantic import BaseModel
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.database import get_db
from app.models import BotTrade, Subscription, iso_utc
from app.services.crypto import decrypt_api_key
from app.services.price_store import price_store
from app.services.signed_request import verify_signed_request
router = APIRouter()
logger = logging.getLogger(__name__)
# ─── Schemas ────────────────────────────────────────────────────────────────
class OpenPosition(BaseModel):
trade_id: int
asset: str
side: str # "long" | "short"
entry_price: float
current_price: Optional[float] = None # None if price_store lacks the asset
size_usd: Optional[float] = None # OPEN notional (after de-risk)
leverage: Optional[int] = None
opened_at: str
hold_minutes: int
unrealized_pct: Optional[float] = None # signed in trade direction
unrealized_usd: Optional[float] = None # on the OPEN portion only
is_paper: bool = False
trigger_post_id: Optional[int] = None
# System-2 staged lifecycle (so the UI doesn't misreport a de-risked /
# pyramided trade). realized_usd = PnL already banked by partial de-risk.
realized_usd: Optional[float] = None
derisk_steps: int = 0
addon_steps: int = 0
grow_mode: bool = False
class OpenPositionsResponse(BaseModel):
wallet: str
count: int
positions: list[OpenPosition]
class TodayStatsResponse(BaseModel):
wallet: str
realized_pnl_usd: float
trades_closed: int
wins: int
losses: int
open_count: int
# PnL already banked by staged de-risk on positions that are STILL open
# (cumulative for those trades — not date-scoped, since per-step times
# aren't tracked). Surfaced separately so it's visible without corrupting
# the strict closed-trade realised figure.
open_realized_usd: float = 0.0
# ─── Helpers ────────────────────────────────────────────────────────────────
def _enrich(trade: BotTrade) -> OpenPosition:
"""Add current price + unrealized PnL to a BotTrade row.
Uses price_store (in-memory Binance ticks). For assets we don't stream
(TRUMP, HYPE, niche perps) current_price will be None and the UI shows
"live price unavailable".
"""
now_aware = datetime.now(timezone.utc)
opened_aware = trade.opened_at.replace(tzinfo=timezone.utc)
hold_min = int((now_aware - opened_aware).total_seconds() // 60)
# Only the STILL-OPEN fraction is on the book. After staged de-risk,
# remaining_fraction < 1.0 and the rest was already realised — marking
# the full size_usd to market would overstate both size and PnL.
rem_frac = trade.remaining_fraction if trade.remaining_fraction is not None else 1.0
open_notional = round((trade.size_usd or 0.0) * rem_frac, 2)
realized = round(trade.realized_partial_pnl_usd or 0.0, 2)
current = price_store.latest_price(trade.asset)
unrealized_pct: Optional[float] = None
unrealized_usd: Optional[float] = None
if current is not None and trade.entry_price:
raw_pct = (current - trade.entry_price) / trade.entry_price
signed_pct = (raw_pct if trade.side == "long" else -raw_pct) * 100
unrealized_pct = round(signed_pct, 3)
if open_notional:
unrealized_usd = round(open_notional * signed_pct / 100, 2)
return OpenPosition(
trade_id=trade.id,
asset=trade.asset,
side=trade.side,
entry_price=trade.entry_price,
current_price=current,
size_usd=open_notional,
leverage=trade.leverage,
opened_at=iso_utc(trade.opened_at) or "",
hold_minutes=hold_min,
unrealized_pct=unrealized_pct,
unrealized_usd=unrealized_usd,
is_paper=(trade.hl_order_id == "paper"),
trigger_post_id=trade.trigger_post_id,
realized_usd=(realized if realized else None),
derisk_steps=(trade.derisk_steps_done or 0),
addon_steps=(trade.addon_steps_done or 0),
grow_mode=bool(getattr(trade, "grow_mode", False)),
)
# ─── Endpoints ──────────────────────────────────────────────────────────────
@router.get("/positions/open", response_model=OpenPositionsResponse)
async def get_open_positions(
wallet: str = Query(..., description="Wallet address (lower-cased internally)"),
ts: int = Query(..., description="Signed timestamp (ms)"),
sig: str = Query(..., description="EIP-191 signature"),
db: AsyncSession = Depends(get_db),
):
"""Live open positions for the wallet, with mark-to-market PnL."""
wallet = wallet.lower().strip()
verify_signed_request(
action=ACTION_VIEW_POSITIONS,
wallet=wallet,
timestamp_ms=ts,
signature=sig,
body=None,
allow_replay=True,
)
rows = await db.execute(
select(BotTrade).where(
BotTrade.wallet_address == wallet,
BotTrade.closed_at.is_(None),
).order_by(BotTrade.opened_at.desc())
)
trades = rows.scalars().all()
positions = [_enrich(t) for t in trades]
return OpenPositionsResponse(wallet=wallet, count=len(positions), positions=positions)
@router.get("/positions/today", response_model=TodayStatsResponse)
async def get_today_stats(
wallet: str = Query(...),
ts: int = Query(..., description="Signed timestamp (ms)"),
sig: str = Query(..., description="EIP-191 signature"),
db: AsyncSession = Depends(get_db),
):
"""Today's realized P&L (since UTC midnight) + open count.
Used for the "today" KPI tile. Cheap — one indexed range scan + a
one-shot count for opens.
"""
wallet = wallet.lower().strip()
verify_signed_request(
action=ACTION_VIEW_POSITIONS,
wallet=wallet,
timestamp_ms=ts,
signature=sig,
body=None,
allow_replay=True,
)
midnight = datetime.now(timezone.utc).replace(
hour=0, minute=0, second=0, microsecond=0, tzinfo=None
)
closed_rows = await db.execute(
select(BotTrade).where(
BotTrade.wallet_address == wallet,
BotTrade.closed_at >= midnight,
BotTrade.pnl_usd.is_not(None),
)
)
closed = closed_rows.scalars().all()
open_rows = await db.execute(
select(BotTrade).where(
BotTrade.wallet_address == wallet,
BotTrade.closed_at.is_(None),
)
)
open_trades = open_rows.scalars().all()
open_count = len(open_trades)
open_realized = sum(t.realized_partial_pnl_usd or 0.0 for t in open_trades)
realized = sum(t.pnl_usd or 0 for t in closed)
wins = sum(1 for t in closed if (t.pnl_usd or 0) > 0)
losses = sum(1 for t in closed if (t.pnl_usd or 0) < 0)
return TodayStatsResponse(
wallet=wallet,
realized_pnl_usd=round(realized, 2),
trades_closed=len(closed),
wins=wins,
losses=losses,
open_count=open_count,
open_realized_usd=round(open_realized, 2),
)
# ─── Manual close (P0.1 safety valve) ───────────────────────────────────────
ACTION_CLOSE_TRADE = "close_trade"
ACTION_SET_GROW = "set_trade_grow"
ACTION_VIEW_POSITIONS = "view_positions"
class CloseTradeResponse(BaseModel):
status: str
trade_id: int
exit_price: Optional[float] = None
pnl_usd: Optional[float] = None
reason: str
note: Optional[str] = None
@router.post("/positions/{trade_id}/close", response_model=CloseTradeResponse)
async def manual_close(
trade_id: int,
request: Request,
db: AsyncSession = Depends(get_db),
):
"""User-initiated emergency close. Always available — bypasses CB, schedule,
even circuit breaker lockout. Wallet ownership is verified by signature.
Path: trade_id → look up wallet → verify signature against THAT wallet →
call close_and_finalize. Reason stamped as "manual" so it's distinguishable
in the trade log from TP / SL / trailing / max_hold exits.
The signed body is {"trade_id": <id>} so a leaked sig for trade X can't be
replayed to close trade Y.
"""
raw = await request.json()
wallet = (raw.get("wallet") or "").lower().strip()
timestamp = raw.get("timestamp")
signature = raw.get("signature")
if not wallet or not isinstance(timestamp, int) or not isinstance(signature, str):
raise HTTPException(422, "wallet, timestamp, signature required")
# Load the trade. Must be open AND owned by the signing wallet.
trade = (await db.execute(select(BotTrade).where(BotTrade.id == trade_id))).scalar_one_or_none()
if trade is None:
raise HTTPException(404, f"trade {trade_id} not found")
if trade.wallet_address.lower() != wallet:
raise HTTPException(403, "trade belongs to a different wallet")
if trade.closed_at is not None:
raise HTTPException(409, f"trade {trade_id} is already closed")
verify_signed_request(
action=ACTION_CLOSE_TRADE,
wallet=wallet,
timestamp_ms=timestamp,
signature=signature,
body={"trade_id": trade_id},
)
# Find the API key from the subscription.
sub = (await db.execute(
select(Subscription).where(Subscription.wallet_address == wallet)
)).scalar_one_or_none()
if sub is None:
raise HTTPException(404, "subscription not found")
# Paper trades close synthetically; live trades need the decrypted HL key.
api_key = ""
if trade.hl_order_id != "paper":
if not sub.hl_api_key:
raise HTTPException(400, "wallet has no HL API key — cannot close live position")
try:
api_key = decrypt_api_key(sub.hl_api_key)
except Exception as exc:
raise HTTPException(500, f"key decryption failed: {exc}")
# close_and_finalize handles BOTH paper and live branches internally.
from app.services.bot_engine import close_and_finalize
leverage = trade.leverage if trade.leverage is not None else sub.leverage
await close_and_finalize(
trade_id=trade.id,
api_key=api_key,
leverage=leverage,
asset=trade.asset,
wallet=wallet,
reason="manual",
)
closed = (await db.execute(select(BotTrade).where(BotTrade.id == trade_id))).scalar_one()
return CloseTradeResponse(
status="ok",
trade_id=trade_id,
exit_price=closed.exit_price,
pnl_usd=closed.pnl_usd,
reason="manual",
)
class GrowResponse(BaseModel):
trade_id: int
grow_mode: bool
@router.post("/positions/{trade_id}/grow", response_model=GrowResponse)
async def set_trade_grow(
trade_id: int,
request: Request,
db: AsyncSession = Depends(get_db),
):
"""Flip the per-trade Grow switch (pyramiding). Signed + ownership-checked.
Body: { wallet, timestamp, signature, trade_id, enabled }
grow=on → this winner is scaled INTO on confirmed trend.
grow=off → hold + protective de-risk / ratchet only (de-risk + stop-loss
always run regardless — never user-toggleable).
Takes effect immediately on the live monitor (no restart needed).
"""
raw = await request.json()
wallet = (raw.get("wallet") or "").lower().strip()
timestamp = raw.get("timestamp")
signature = raw.get("signature")
enabled = raw.get("enabled")
body_tid = raw.get("trade_id")
if not wallet or not isinstance(timestamp, int) or not isinstance(signature, str):
raise HTTPException(422, "wallet, timestamp, signature required")
if not isinstance(enabled, bool):
raise HTTPException(422, "enabled (bool) required")
if body_tid != trade_id:
raise HTTPException(400, "trade_id mismatch (path vs signed body)")
trade = (await db.execute(select(BotTrade).where(BotTrade.id == trade_id))).scalar_one_or_none()
if trade is None:
raise HTTPException(404, f"trade {trade_id} not found")
if trade.wallet_address.lower() != wallet:
raise HTTPException(403, "trade belongs to a different wallet")
if trade.closed_at is not None:
raise HTTPException(409, f"trade {trade_id} is already closed")
verify_signed_request(
action=ACTION_SET_GROW,
wallet=wallet,
timestamp_ms=timestamp,
signature=signature,
body={"trade_id": trade_id, "enabled": enabled},
)
trade.grow_mode = enabled
await db.commit()
# Apply to the live monitor immediately so it takes effect this tick.
try:
from app.services.tp_sl_monitor import _watched
wt = _watched.get(trade_id)
if wt is not None:
wt.grow_mode = enabled
except Exception as exc:
logger.warning("grow toggle: live monitor update skipped trade %d: %s",
trade_id, exc)
logger.info("Grow %s for trade %d by %s",
"ON" if enabled else "OFF", trade_id, wallet)
return GrowResponse(trade_id=trade_id, grow_mode=enabled)
app/api/posts.py
+1
View File
@@ -61,6 +61,7 @@ def _post_to_schema(post: Post) -> TrumpPost:
target_asset=post.target_asset,
category=post.category,
expected_move_pct=post.expected_move_pct,
invalidation_price=post.invalidation_price,
)
app/api/scanners.py
+182
View File
@@ -0,0 +1,182 @@
"""
Scanner control endpoints — kill switch + per-scanner toggle + status.
GET /scanners is public (read-only status, polled by the UI).
ALL mutating endpoints (toggle / all-disable / all-enable) require a SIGNED
request from a SUBSCRIBED wallet. Rationale: the scanner switch is a global
control — flipping it OFF stops the signal engine for every user. Before, it
was unauthenticated and any anonymous visitor could kill it. Now the caller
must prove wallet ownership (EIP-191 signature) AND already be a subscriber.
# See what's running (still public)
curl http://localhost:8000/api/scanners
Mutations are no longer curl-able without a wallet signature — that's the
point. Use the dashboard (which signs via the connected wallet).
"""
from __future__ import annotations
import logging
from typing import Optional
from fastapi import APIRouter, Depends, HTTPException, Request
from pydantic import BaseModel
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.database import get_db
from app.models import Subscription
from app.services import scanner_state
from app.services.signed_request import verify_signed_request
router = APIRouter()
logger = logging.getLogger(__name__)
ACTION_TOGGLE_SCANNER = "toggle_scanner"
ACTION_SCANNER_KILL = "scanner_kill_all"
ACTION_SCANNER_REVIVE = "scanner_revive_all"
class ScannerSummary(BaseModel):
name: str
enabled: bool
last_run_at: Optional[str] = None
last_status: str
last_message: Optional[str] = None
last_fired_at: Optional[str] = None
total_runs: int
total_fires: int
consecutive_errors: int
class ScannersResponse(BaseModel):
count: int
enabled: int
scanners: list[ScannerSummary]
class ToggleResponse(BaseModel):
status: str
name: Optional[str] = None
enabled: Optional[bool] = None
count: Optional[int] = None
async def _require_subscribed_signer(
request: Request,
db: AsyncSession,
*,
action: str,
body: Optional[dict],
) -> str:
"""Verify the signed envelope and that the signer is a subscriber.
Expects JSON body: { wallet, timestamp, signature, ...body fields }.
`body` is the canonical signed payload (must match what the client
hashed) — pass None for no-body actions (kill/revive).
Returns the verified lower-cased wallet.
"""
raw = await request.json()
wallet = (raw.get("wallet") or "").lower().strip()
timestamp = raw.get("timestamp")
signature = raw.get("signature")
if not wallet:
raise HTTPException(422, "wallet required")
if not isinstance(timestamp, int):
raise HTTPException(422, "timestamp (ms) required")
if not isinstance(signature, str) or not signature:
raise HTTPException(422, "signature required")
verify_signed_request(
action=action,
wallet=wallet,
timestamp_ms=timestamp,
signature=signature,
body=body,
)
sub = (await db.execute(
select(Subscription).where(Subscription.wallet_address == wallet)
)).scalar_one_or_none()
if sub is None:
raise HTTPException(403, "Wallet is not a subscriber — scanner control denied")
return wallet
@router.get("/scanners", response_model=ScannersResponse)
async def list_scanners() -> ScannersResponse:
"""Operational snapshot of every registered scanner. Polled by the UI
every 15-30s. Idempotent + cheap (in-memory dict read). Public."""
all_states = scanner_state.get_all()
return ScannersResponse(
count=len(all_states),
enabled=sum(1 for s in all_states if s.enabled),
scanners=[ScannerSummary(**s.to_dict()) for s in all_states],
)
@router.post("/scanners/{name}/toggle", response_model=ToggleResponse)
async def toggle_scanner(
name: str,
request: Request,
db: AsyncSession = Depends(get_db),
) -> ToggleResponse:
"""Flip ONE scanner. Signed + subscriber-gated.
Body: { wallet, timestamp, signature, name, enabled }
The signed body is { "enabled": <bool>, "name": <str> }.
"""
raw = await request.json()
enabled = raw.get("enabled")
body_name = raw.get("name")
if not isinstance(enabled, bool):
raise HTTPException(422, "enabled (bool) required")
if body_name != name:
raise HTTPException(400, "scanner name mismatch (path vs signed body)")
# NOTE: request.json() is cached by Starlette, so re-reading inside the
# helper returns the same payload.
wallet = await _require_subscribed_signer(
request, db, action=ACTION_TOGGLE_SCANNER,
body={"enabled": enabled, "name": name},
)
s = scanner_state.set_enabled(name, enabled)
if s is None:
raise HTTPException(404, f"unknown scanner {name!r}")
logger.info("Scanner %s set enabled=%s by %s", name, s.enabled, wallet)
return ToggleResponse(status="ok", name=name, enabled=s.enabled)
@router.post("/scanners/all/disable", response_model=ToggleResponse)
async def kill_switch(
request: Request,
db: AsyncSession = Depends(get_db),
) -> ToggleResponse:
"""Emergency stop — disables ALL scanners. Signed + subscriber-gated.
Body: { wallet, timestamp, signature }. Does NOT close open positions —
manage those via /positions/{id}/close or the Trades page."""
wallet = await _require_subscribed_signer(
request, db, action=ACTION_SCANNER_KILL, body=None,
)
n = scanner_state.disable_all()
logger.warning("ALL scanners disabled (%d) by %s", n, wallet)
return ToggleResponse(status="killed", count=n)
@router.post("/scanners/all/enable", response_model=ToggleResponse)
async def revive_all(
request: Request,
db: AsyncSession = Depends(get_db),
) -> ToggleResponse:
"""Re-enable every scanner that was disabled. Signed + subscriber-gated.
Body: { wallet, timestamp, signature }."""
wallet = await _require_subscribed_signer(
request, db, action=ACTION_SCANNER_REVIVE, body=None,
)
n = scanner_state.enable_all()
logger.info("ALL scanners re-enabled (%d) by %s", n, wallet)
return ToggleResponse(status="revived", count=n)
app/api/signals.py
+243
View File
@@ -0,0 +1,243 @@
"""
Generic signal ingestion endpoint.
The BTC bottom scanner POSTs a signal here and it flows through the shared
execution pipeline: sizing → risk caps → Hyperliquid execution → trailing
stop monitor. Unknown sources are accepted into the posts table for audit,
but the execution layer fails closed and will not trade them.
Why one endpoint instead of many?
The trusted scanners and the Trump scraper share the same trade execution
path. The downstream code sees a `Post` row with `signal=buy|short`, then
checks whether that post source is allowed to trade.
Auth: shared secret in X-Ingest-Key header. Fail-closed if INGEST_API_KEY
is empty in env (so a fresh deploy can't accidentally accept random POSTs).
Source field convention:
- "truth" → Trump posts (existing scraper). Reserved — rejected here.
- "btc_bottom_reversal" → Bitcoin Bottom scanner
- "<custom>" → Audit/storage only unless whitelisted in
signal_categories.py.
"""
from __future__ import annotations
import hashlib
import logging
from datetime import datetime, timezone
from typing import Optional
from fastapi import APIRouter, Depends, Header, HTTPException
from pydantic import BaseModel, Field, field_validator
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.config import settings
from app.database import get_db
from app.models import Post
router = APIRouter()
logger = logging.getLogger(__name__)
# ─── Schema ─────────────────────────────────────────────────────────────────
class SignalIngestRequest(BaseModel):
"""Payload for POST /api/signals/ingest.
Mirrors the Post schema closely so a signal here becomes a Post row
1:1, with no field translation. Downstream code is unaware of the source.
"""
source: str = Field(..., min_length=2, max_length=32,
description="Signal source tag, e.g. 'btc_bottom_reversal'. 'truth' is reserved.")
external_id: str = Field(..., min_length=4, max_length=64,
description="Caller-supplied unique key. Used for idempotent retries.")
text: str = Field(..., min_length=1, max_length=4000,
description="Human-readable description of why the signal fired. Shown in UI.")
signal: str = Field(..., description="'buy' or 'short' — non-actionable signals should not be ingested.")
target_asset: str = Field(..., min_length=2, max_length=16,
description="Hyperliquid perp ticker, e.g. 'SOL', 'BTC'.")
confidence: int = Field(..., ge=0, le=100,
description="0–100. Drives position sizing (≥90 boosts multiplier).")
category: str = Field(..., min_length=2, max_length=24,
description="Subtype within source, e.g. 'btc_bottom_reversal_long'.")
expected_move_pct: Optional[float] = Field(None, ge=0, le=100,
description="Optional: caller's expected % move. UI display only.")
invalidation_price: Optional[float] = Field(None, ge=0,
description="Optional thesis invalidation level for System 2.")
published_at: Optional[datetime] = Field(None,
description="Defaults to server now (UTC). Set explicitly only for backfill.")
@field_validator("signal")
@classmethod
def _signal_must_be_directional(cls, v: str) -> str:
if v not in ("buy", "short"):
raise ValueError(f"signal must be 'buy' or 'short' (got {v!r})")
return v
@field_validator("source")
@classmethod
def _no_reserved_sources(cls, v: str) -> str:
if v.lower() == "truth":
raise ValueError("'truth' is reserved for the Trump scraper. Use a different source tag.")
return v.lower()
class SignalIngestResponse(BaseModel):
status: str # "accepted" | "duplicate" | "skipped"
post_id: Optional[int] = None
dedup_against: Optional[int] = None
note: Optional[str] = None
# ─── Auth helper ────────────────────────────────────────────────────────────
def _verify_ingest_key(x_ingest_key: Optional[str]) -> None:
"""Fail-closed auth check.
- INGEST_API_KEY unset → return 503 (endpoint disabled by config)
- Header missing or wrong → 401
"""
expected = settings.ingest_api_key
if not expected:
raise HTTPException(503, "signal ingest endpoint is disabled (INGEST_API_KEY not configured)")
if not x_ingest_key:
raise HTTPException(401, "missing X-Ingest-Key header")
if x_ingest_key != expected:
raise HTTPException(401, "invalid X-Ingest-Key")
# ─── Endpoint ───────────────────────────────────────────────────────────────
@router.post("/signals/ingest", response_model=SignalIngestResponse)
async def ingest_signal(
body: SignalIngestRequest,
x_ingest_key: Optional[str] = Header(None, alias="X-Ingest-Key"),
db: AsyncSession = Depends(get_db),
) -> SignalIngestResponse:
"""Accept a signal from an external trading module.
Pipeline behaviour:
1. Auth (shared secret in header)
2. Dedup by external_id (idempotent — same id returns the existing post_id)
3. Insert as a Post row (source = caller-supplied)
4. If the source is whitelisted for trading, hand off to
bot_engine.process_post. Otherwise store for audit and return
status="skipped".
Note: the entry filter ([A] in the pipeline) is SKIPPED for non-truth
sources. That filter is tuned for Trump's writing patterns; technical
signals are their own catalyst and don't need text-based gating.
"""
_verify_ingest_key(x_ingest_key)
# Hash external_id the same way truth_social.py does — keeps the dedup
# key short and uniform across sources.
ext_id_hashed = hashlib.md5(f"{body.source}:{body.external_id}".encode()).hexdigest()
# Idempotency check
existing = await db.execute(select(Post).where(Post.external_id == ext_id_hashed))
prior = existing.scalar_one_or_none()
if prior:
return SignalIngestResponse(
status="duplicate",
post_id=prior.id,
dedup_against=prior.id,
note=f"external_id {body.external_id!r} already ingested",
)
# Publication time defaults to server now (naive UTC, matching other rows)
pub_naive = (body.published_at or datetime.now(timezone.utc))
if pub_naive.tzinfo is not None:
pub_naive = pub_naive.astimezone(timezone.utc).replace(tzinfo=None)
post = Post(
external_id=ext_id_hashed,
text=body.text,
source=body.source,
published_at=pub_naive,
sentiment="bullish" if body.signal == "buy" else "bearish",
ai_confidence=body.confidence,
relevant=True,
signal=body.signal,
target_asset=body.target_asset.upper(),
category=body.category,
expected_move_pct=body.expected_move_pct,
invalidation_price=body.invalidation_price,
# `analysis_version` doubles as a provenance tag — easy to query in DB
analysis_version=f"ingest:{body.source}",
prefilter_reason="external_signal", # bypasses entry-filter audit
)
db.add(post)
await db.commit()
await db.refresh(post)
logger.info(
"Ingested signal: source=%s id=%s → post_id=%d, %s/%s conf=%d category=%s",
body.source, body.external_id, post.id,
body.signal, body.target_asset, body.confidence, body.category,
)
from app.services.signal_categories import is_supported_trading_source
if not is_supported_trading_source(post.source):
logger.info("Signal %d stored but skipped: unsupported trading source=%s",
post.id, post.source)
return SignalIngestResponse(
status="skipped",
post_id=post.id,
note=f"source {post.source!r} is not enabled for live trading",
)
# Hand off to the same trade pipeline used by Trump posts.
# Runs sizing → risk caps → HL trade → trailing-stop monitor.
try:
from app.services.bot_engine import process_post
await process_post(post, db)
except Exception as exc:
# Don't fail the HTTP response — the post is already in DB, and
# process_post failures shouldn't make the external module retry
# (which would duplicate signals). Log and return success.
logger.error("process_post failed for ingested signal %d: %s", post.id, exc)
# Broadcast to UI so the new signal shows up live, same as Trump posts.
try:
from app.scrapers.truth_social import _post_to_ws_payload
from app.ws.manager import manager
await manager.broadcast(_post_to_ws_payload(post))
except Exception as exc:
logger.warning("WS broadcast failed for signal %d: %s", post.id, exc)
# Fan out to Telegram subscribers (fire-and-forget; never blocks ingest).
try:
from app.services.telegram import notify_signal
notify_signal(post)
except Exception as exc:
logger.warning("Telegram notify failed for signal %d: %s", post.id, exc)
return SignalIngestResponse(status="accepted", post_id=post.id)
@router.get("/signals/sources")
async def list_sources(db: AsyncSession = Depends(get_db)) -> dict:
"""List distinct signal sources we've seen, with counts.
Helpful for spot-checking that ingestion is working and for the UI to
show a source filter without hard-coding the list.
"""
from sqlalchemy import func
rows = await db.execute(
select(Post.source, func.count(Post.id), func.max(Post.published_at))
.group_by(Post.source)
.order_by(func.count(Post.id).desc())
)
return {
"sources": [
{"source": r[0], "count": r[1], "latest": r[2].isoformat() if r[2] else None}
for r in rows.all()
]
}
app/api/subscribe.py
+29 -6
View File
@@ -1,7 +1,7 @@
import logging
from datetime import datetime, timezone
from fastapi import APIRouter, Depends
from fastapi import APIRouter, Depends, Request
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
@@ -17,15 +17,29 @@ ACTION_SUBSCRIBE = "subscribe"
@router.post("/subscribe", response_model=SubscribeResponse)
async def subscribe(body: SubscribeRequest, db: AsyncSession = Depends(get_db)):
async def subscribe(request: Request, db: AsyncSession = Depends(get_db)):
"""Activate a subscription. Optional `paper_mode` flag in the body lets
new users try the system safely (no Hyperliquid call, simulated fills).
Signed message body is the RAW dict so the canonical hash matches what
the frontend signed — same pattern as set_user_settings.
"""
raw = await request.json()
body = SubscribeRequest(**raw)
wallet = body.wallet.lower().strip()
# The signed body excludes the envelope fields. For backwards compatibility
# with old clients that signed `body=None` (no paper_mode key), accept
# either signature.
paper_mode = bool(raw.get("paper_mode", False))
signed_body = {"paper_mode": paper_mode} if paper_mode else None
verify_signed_request(
action=ACTION_SUBSCRIBE,
wallet=wallet,
timestamp_ms=body.timestamp,
signature=body.signature,
body=None,
body=signed_body,
)
result = await db.execute(
@@ -35,13 +49,22 @@ async def subscribe(body: SubscribeRequest, db: AsyncSession = Depends(get_db)):
now = datetime.now(timezone.utc).replace(tzinfo=None)
if sub is None:
sub = Subscription(wallet_address=wallet, active=True, subscribed_at=now)
sub = Subscription(
wallet_address=wallet,
active=True,
subscribed_at=now,
paper_mode=paper_mode,
)
db.add(sub)
else:
sub.active = True
if sub.subscribed_at is None:
sub.subscribed_at = now
# Re-subscribing is allowed to change paper mode (e.g. user wants to
# promote from paper to live). Otherwise leave existing flag alone.
if paper_mode != sub.paper_mode:
sub.paper_mode = paper_mode
await db.commit()
logger.info("Subscription activated for wallet %s", wallet)
return SubscribeResponse(status="ok", wallet=wallet)
logger.info("Subscription activated for %s (paper_mode=%s)", wallet, paper_mode)
return SubscribeResponse(status="ok", wallet=wallet, paper_mode=paper_mode)
app/api/telegram.py
+232
View File
@@ -0,0 +1,232 @@
"""
Telegram binding + preferences API.
All mutating endpoints require a signed envelope from the wallet (same EIP-191
flow as set_hl_api_key). Read endpoints are unsigned but require the wallet
in the path so other users' bindings stay private.
GET /api/telegram/{wallet}/status ← unsigned read
POST /api/telegram/{wallet}/init ← signed; returns binding code + deep link
POST /api/telegram/{wallet}/preferences ← signed; update toggles
POST /api/telegram/{wallet}/unbind ← signed; removes binding
POST /api/telegram/{wallet}/test ← signed; sends a self-test
"""
from __future__ import annotations
import logging
from typing import Optional
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel, Field
from sqlalchemy import select, update
from sqlalchemy.ext.asyncio import AsyncSession
from app.config import settings
from app.database import get_db
from app.models import TelegramBinding, Subscription
from app.services.signed_request import verify_signed_request
from app.services.telegram import send_test_message
from app.services.telegram_bot import issue_binding_code, unbind_wallet
router = APIRouter(prefix="/telegram", tags=["telegram"])
logger = logging.getLogger(__name__)
ACTION_TG_INIT = "telegram_init"
ACTION_TG_PREFS = "telegram_prefs"
ACTION_TG_UNBIND = "telegram_unbind"
ACTION_TG_TEST = "telegram_test"
def _require_tg_configured() -> None:
if not settings.telegram_bot_token or not settings.telegram_bot_username:
raise HTTPException(503, "Telegram alerts are not configured on this server")
# ── Schemas ──────────────────────────────────────────────────────────────
class SignedEnvelope(BaseModel):
wallet: str
timestamp: int
signature: str
class PrefsBody(SignedEnvelope):
alerts_enabled: Optional[bool] = None
alert_trump: Optional[bool] = None
alert_btc_bottom: Optional[bool] = None
alert_funding: Optional[bool] = None
alert_kol_divergence: Optional[bool] = None
min_confidence: Optional[int] = Field(None, ge=0, le=100)
mute_from_hour: Optional[int] = Field(None, ge=0, le=23)
mute_until_hour: Optional[int] = Field(None, ge=0, le=23)
class StatusResponse(BaseModel):
configured: bool # whether server has bot token
bot_username: Optional[str] = None
bound: bool
wallet_address: Optional[str] = None
tg_username: Optional[str] = None
chat_id: Optional[int] = None
alerts_enabled: Optional[bool] = None
alert_trump: Optional[bool] = None
alert_btc_bottom: Optional[bool] = None
alert_funding: Optional[bool] = None
alert_kol_divergence: Optional[bool] = None
min_confidence: Optional[int] = None
mute_from_hour: Optional[int] = None
mute_until_hour: Optional[int] = None
total_alerts_sent: Optional[int] = None
class InitResponse(BaseModel):
code: str
deep_link: str # t.me/<bot>?start=<code>
expires_in_seconds: int
# ── Endpoints ────────────────────────────────────────────────────────────
@router.get("/{wallet}/status", response_model=StatusResponse)
async def status(wallet: str, db: AsyncSession = Depends(get_db)) -> StatusResponse:
"""Public-by-wallet read. Returns whether server is configured AND
whether this wallet has bound a Telegram chat."""
wallet = wallet.lower().strip()
configured = bool(settings.telegram_bot_token and settings.telegram_bot_username)
b = (await db.execute(
select(TelegramBinding).where(TelegramBinding.wallet_address == wallet)
)).scalar_one_or_none()
if not b:
return StatusResponse(configured=configured,
bot_username=settings.telegram_bot_username or None,
bound=False)
return StatusResponse(
configured=configured,
bot_username=settings.telegram_bot_username or None,
bound=True,
wallet_address=b.wallet_address,
tg_username=b.tg_username,
chat_id=b.chat_id,
alerts_enabled=b.alerts_enabled,
alert_trump=b.alert_trump,
alert_btc_bottom=b.alert_btc_bottom,
alert_funding=b.alert_funding,
alert_kol_divergence=b.alert_kol_divergence,
min_confidence=b.min_confidence,
mute_from_hour=b.mute_from_hour,
mute_until_hour=b.mute_until_hour,
total_alerts_sent=b.total_alerts_sent,
)
@router.post("/{wallet}/init", response_model=InitResponse)
async def init_binding(
wallet: str, body: SignedEnvelope,
db: AsyncSession = Depends(get_db),
) -> InitResponse:
"""Generate a one-time code and the deep link the frontend renders.
Subscriber-gated — only paying wallets can receive alerts."""
_require_tg_configured()
wallet = wallet.lower().strip()
if wallet != body.wallet.lower().strip():
raise HTTPException(400, "Wallet mismatch")
verify_signed_request(
action=ACTION_TG_INIT, wallet=wallet,
timestamp_ms=body.timestamp, signature=body.signature, body=None,
)
sub = (await db.execute(
select(Subscription).where(Subscription.wallet_address == wallet)
)).scalar_one_or_none()
if not sub or not sub.active:
raise HTTPException(403, "Wallet must be subscribed to enable Telegram alerts")
code = issue_binding_code(wallet)
deep_link = f"https://t.me/{settings.telegram_bot_username}?start={code}"
return InitResponse(code=code, deep_link=deep_link, expires_in_seconds=600)
@router.post("/{wallet}/preferences", response_model=StatusResponse)
async def update_preferences(
wallet: str, body: PrefsBody,
db: AsyncSession = Depends(get_db),
) -> StatusResponse:
"""Toggle alert sources, confidence floor, mute hours. Signed.
Idempotent — only fields present in the body are updated."""
_require_tg_configured()
wallet = wallet.lower().strip()
if wallet != body.wallet.lower().strip():
raise HTTPException(400, "Wallet mismatch")
# Build a canonical body for signing: include only the fields the user
# is actually trying to change (so the signed payload matches what the
# frontend hashed).
signed_body = {k: v for k, v in body.model_dump(exclude={"wallet", "timestamp", "signature"}).items()
if v is not None}
verify_signed_request(
action=ACTION_TG_PREFS, wallet=wallet,
timestamp_ms=body.timestamp, signature=body.signature,
body=signed_body,
)
b = (await db.execute(
select(TelegramBinding).where(TelegramBinding.wallet_address == wallet)
)).scalar_one_or_none()
if not b:
raise HTTPException(404, "No Telegram binding for this wallet — bind via /start first")
values = {}
for f in ["alerts_enabled", "alert_trump", "alert_btc_bottom",
"alert_funding", "alert_kol_divergence", "min_confidence",
"mute_from_hour", "mute_until_hour"]:
v = getattr(body, f)
if v is not None:
values[f] = v
if values:
await db.execute(
update(TelegramBinding).where(TelegramBinding.id == b.id).values(**values)
)
await db.commit()
await db.refresh(b)
return await status(wallet, db)
@router.post("/{wallet}/unbind")
async def unbind(
wallet: str, body: SignedEnvelope,
db: AsyncSession = Depends(get_db),
) -> dict:
_require_tg_configured()
wallet = wallet.lower().strip()
if wallet != body.wallet.lower().strip():
raise HTTPException(400, "Wallet mismatch")
verify_signed_request(
action=ACTION_TG_UNBIND, wallet=wallet,
timestamp_ms=body.timestamp, signature=body.signature, body=None,
)
n = await unbind_wallet(wallet)
return {"removed": n}
@router.post("/{wallet}/test")
async def test(
wallet: str, body: SignedEnvelope,
db: AsyncSession = Depends(get_db),
) -> dict:
"""Send a sample alert to verify the binding works end-to-end."""
_require_tg_configured()
wallet = wallet.lower().strip()
if wallet != body.wallet.lower().strip():
raise HTTPException(400, "Wallet mismatch")
verify_signed_request(
action=ACTION_TG_TEST, wallet=wallet,
timestamp_ms=body.timestamp, signature=body.signature, body=None,
)
ok = await send_test_message(wallet)
if not ok:
raise HTTPException(400, "Test failed — bind via /start first, or check bot token")
return {"sent": True}
app/api/trades.py
+33 -1
View File
@@ -3,17 +3,31 @@ from typing import List
from fastapi import APIRouter, Depends, Query
from sqlalchemy import select
from sqlalchemy.orm import joinedload
from sqlalchemy.ext.asyncio import AsyncSession
from app.database import get_db
from app.models import BotTrade, iso_utc
from app.schemas import BotTrade as BotTradeSchema
from app.services.signed_request import verify_signed_request
router = APIRouter()
logger = logging.getLogger(__name__)
ACTION_VIEW_TRADES = "view_trades"
def _trade_to_schema(trade: BotTrade) -> BotTradeSchema:
# Join against trigger_post to surface the source tag. When a trade was
# opened by an ingested signal (VCP scanner, user's module, etc.) the
# source reveals WHICH module produced it — critical for "is module X
# actually making money?" attribution analysis.
trigger_source = None
if trade.trigger_post is not None:
trigger_source = trade.trigger_post.source
# Paper trades are tagged via hl_order_id at open time; that's the only
# stable signal we have to distinguish them in aggregate views.
is_paper = (trade.hl_order_id == "paper")
return BotTradeSchema(
id=trade.id,
asset=trade.asset,
@@ -25,18 +39,36 @@ def _trade_to_schema(trade: BotTrade) -> BotTradeSchema:
trigger_post_id=trade.trigger_post_id or 0,
opened_at=iso_utc(trade.opened_at) or "",
closed_at=iso_utc(trade.closed_at) or "",
trigger_source=trigger_source,
is_paper=is_paper,
)
@router.get("/trades", response_model=List[BotTradeSchema])
async def get_trades(
wallet: str = Query(..., description="Wallet address (lower-cased internally)"),
ts: int = Query(..., description="Signed timestamp (ms)"),
sig: str = Query(..., description="EIP-191 signature"),
limit: int = Query(default=20, ge=1, le=100),
page: int = Query(default=1, ge=1),
page: int = Query(default=1, ge=1),
db: AsyncSession = Depends(get_db),
):
wallet = wallet.lower().strip()
verify_signed_request(
action=ACTION_VIEW_TRADES,
wallet=wallet,
timestamp_ms=ts,
signature=sig,
body=None,
allow_replay=True,
)
offset = (page - 1) * limit
# joinedload pulls trigger_post in the same query — avoids N+1 lookups
# when serialising 100 trades.
result = await db.execute(
select(BotTrade)
.options(joinedload(BotTrade.trigger_post))
.where(BotTrade.wallet_address == wallet)
.where(BotTrade.closed_at.is_not(None))
.order_by(BotTrade.opened_at.desc())
.offset(offset)
app/api/user.py
+177 -5
View File
@@ -15,15 +15,30 @@ from app.schemas import (
SetSettingsRequest,
)
from app.services.crypto import encrypt_api_key
from app.services.hyperliquid import HyperliquidTrader
from app.services.signed_request import verify_signed_request
router = APIRouter()
logger = logging.getLogger(__name__)
# Action names — must match frontend exactly (used in the signed message)
ACTION_SET_API_KEY = "set_hl_api_key"
ACTION_SET_SETTINGS = "set_settings"
ACTION_VIEW_USER = "view_user"
ACTION_SET_API_KEY = "set_hl_api_key"
ACTION_SET_SETTINGS = "set_settings"
ACTION_VIEW_USER = "view_user"
ACTION_SET_MANUAL_WINDOW = "set_manual_window"
ACTION_SET_AUTO_TRADE = "set_auto_trade"
async def verify_hl_api_key_can_trade(api_key: str, account_address: str) -> None:
"""Fail before storing an unusable Hyperliquid API wallet key."""
try:
trader = HyperliquidTrader(
api_private_key=api_key,
account_address=account_address,
)
await trader.get_balance()
except Exception as exc:
raise HTTPException(422, f"Hyperliquid rejected this API key: {exc}")
def _trade_to_schema(trade: BotTrade) -> BotTradeSchema:
@@ -70,12 +85,14 @@ async def set_hl_api_key(
if sub is None:
raise HTTPException(404, "Wallet not subscribed. Subscribe first.")
await verify_hl_api_key_can_trade(api_key=api_key, account_address=wallet)
sub.hl_api_key = encrypt_api_key(api_key)
await db.commit()
masked = f"...{api_key[-6:]}"
logger.info("HL API key updated for wallet %s (masked: %s)", wallet, masked)
return SetApiKeyResponse(status="ok", masked_key=masked)
return SetApiKeyResponse(status="ok", masked_key=masked, verified=True)
@router.get("/user/{wallet}/public")
@@ -86,11 +103,22 @@ async def get_user_public(wallet: str, db: AsyncSession = Depends(get_db)):
result = await db.execute(select(Subscription).where(Subscription.wallet_address == wallet))
sub = result.scalar_one_or_none()
if sub is None:
return {"wallet_address": wallet, "active": False, "hl_api_key_set": False}
return {
"wallet_address": wallet, "active": False, "hl_api_key_set": False,
"paper_mode": False, "manual_window_until": None,
"circuit_breaker_tripped_at": None, "circuit_breaker_reason": None,
"auto_trade": False,
}
return {
"wallet_address": wallet,
"active": sub.active,
"hl_api_key_set": bool(sub.hl_api_key),
# Operational state shown on /signals page. Not sensitive.
"paper_mode": bool(sub.paper_mode),
"manual_window_until": iso_utc(sub.manual_window_until),
"circuit_breaker_tripped_at": iso_utc(sub.circuit_breaker_tripped_at),
"circuit_breaker_reason": sub.circuit_breaker_reason,
"auto_trade": bool(sub.auto_trade),
}
@@ -151,9 +179,12 @@ async def get_user(
stop_loss_pct=sub.stop_loss_pct,
min_confidence=sub.min_confidence,
daily_budget_usd=sub.daily_budget_usd,
sys2_leverage=sub.sys2_leverage,
sys2_mode=sub.sys2_mode,
active_from=iso_utc(sub.active_from),
active_until=iso_utc(sub.active_until),
),
manual_window_until=iso_utc(sub.manual_window_until),
)
@@ -190,6 +221,10 @@ async def set_user_settings(
raise HTTPException(422, "stop_loss_pct is required (0.1–50)")
if not (0 <= s.min_confidence <= 100):
raise HTTPException(422, "min_confidence must be 0–100")
if s.sys2_leverage is not None and not (1 <= s.sys2_leverage <= 10):
raise HTTPException(422, "sys2_leverage must be 1–10")
if s.sys2_mode is not None and s.sys2_mode not in ("standard", "aggressive"):
raise HTTPException(422, "sys2_mode must be 'standard' or 'aggressive'")
if s.daily_budget_usd is None or not (0 < s.daily_budget_usd <= 100000):
raise HTTPException(422, "daily_budget_usd is required (>0, ≤100,000)")
@@ -235,8 +270,145 @@ async def set_user_settings(
sub.stop_loss_pct = s.stop_loss_pct
sub.min_confidence = s.min_confidence
sub.daily_budget_usd = s.daily_budget_usd
sub.sys2_leverage = s.sys2_leverage
if s.sys2_mode is not None:
sub.sys2_mode = s.sys2_mode
sub.active_from = af
sub.active_until = au
await db.commit()
logger.info("Settings updated for %s: %s", wallet, s.model_dump())
return s
# ─── Manual window (convex-strategy "enable for N hours" override) ───────────
@router.post("/user/{wallet}/manual-window")
async def set_manual_window(
wallet: str,
request: Request,
db: AsyncSession = Depends(get_db),
):
"""Arm the bot for the next N hours, overriding the active_from/until schedule.
Body: { wallet, timestamp, signature, hours }
hours: integer 0–168. Pass 0 to clear (immediate disarm).
The bot's main gate (Subscription.active) still applies. This endpoint only
flips the schedule override; an inactive subscription stays paused.
"""
from datetime import datetime as _dt, timezone as _tz, timedelta as _td
wallet = wallet.lower().strip()
raw = await request.json()
# Manual auth payload — keep deliberately minimal so the signed string is
# short and easy to reason about.
body_wallet = (raw.get("wallet") or "").lower().strip()
timestamp = raw.get("timestamp")
signature = raw.get("signature")
hours_raw = raw.get("hours")
if body_wallet != wallet:
raise HTTPException(400, "Wallet mismatch")
if not isinstance(timestamp, int):
raise HTTPException(422, "timestamp (ms) required")
if not isinstance(signature, str) or not signature:
raise HTTPException(422, "signature required")
if not isinstance(hours_raw, int) or hours_raw < 0 or hours_raw > 168:
raise HTTPException(422, "hours must be 0–168")
verify_signed_request(
action=ACTION_SET_MANUAL_WINDOW,
wallet=wallet,
timestamp_ms=timestamp,
signature=signature,
body={"hours": hours_raw},
)
result = await db.execute(select(Subscription).where(Subscription.wallet_address == wallet))
sub = result.scalar_one_or_none()
if sub is None:
raise HTTPException(404, "Wallet not subscribed")
if hours_raw == 0:
sub.manual_window_until = None
new_until_iso = None
logger.info("Manual window cleared for %s", wallet)
else:
until = _dt.now(_tz.utc).replace(tzinfo=None) + _td(hours=hours_raw)
sub.manual_window_until = until
new_until_iso = iso_utc(until)
# Explicit re-arm clears any active circuit-breaker trip — human in
# the loop has acknowledged the risk and chosen to resume.
cb_cleared = False
if sub.circuit_breaker_tripped_at is not None:
sub.circuit_breaker_tripped_at = None
sub.circuit_breaker_reason = None
cb_cleared = True
logger.info("Manual window armed for %s: until %s (%dh)%s",
wallet, until, hours_raw, " — CB cleared" if cb_cleared else "")
await db.commit()
return {"manual_window_until": new_until_iso}
# ─── Master Auto-Trade switch (simplified operator model) ───────────────────
@router.post("/user/{wallet}/auto-trade")
async def set_auto_trade(
wallet: str,
request: Request,
db: AsyncSession = Depends(get_db),
):
"""Flip the ONE persistent Auto-Trade gate.
Body: { wallet, timestamp, signature, enabled }
enabled=false → signals are still ingested/shown but NO trade opens.
enabled=true → qualifying signals auto-open trades. Turning it ON also
acknowledges + clears a tripped circuit breaker
(human-in-the-loop), mirroring the old manual-window arm.
"""
wallet = wallet.lower().strip()
raw = await request.json()
body_wallet = (raw.get("wallet") or "").lower().strip()
timestamp = raw.get("timestamp")
signature = raw.get("signature")
enabled = raw.get("enabled")
if body_wallet != wallet:
raise HTTPException(400, "Wallet mismatch")
if not isinstance(timestamp, int):
raise HTTPException(422, "timestamp (ms) required")
if not isinstance(signature, str) or not signature:
raise HTTPException(422, "signature required")
if not isinstance(enabled, bool):
raise HTTPException(422, "enabled (bool) required")
verify_signed_request(
action=ACTION_SET_AUTO_TRADE,
wallet=wallet,
timestamp_ms=timestamp,
signature=signature,
body={"enabled": enabled},
)
result = await db.execute(select(Subscription).where(Subscription.wallet_address == wallet))
sub = result.scalar_one_or_none()
if sub is None:
raise HTTPException(404, "Wallet not subscribed")
sub.auto_trade = enabled
cb_cleared = False
if enabled and sub.circuit_breaker_tripped_at is not None:
# Turning Auto-Trade ON = explicit human ack → clear the breaker.
sub.circuit_breaker_tripped_at = None
sub.circuit_breaker_reason = None
cb_cleared = True
await db.commit()
logger.info("Auto-Trade %s for %s%s",
"ON" if enabled else "OFF", wallet,
" — CB cleared" if cb_cleared else "")
return {"auto_trade": enabled, "circuit_breaker_cleared": cb_cleared}
app/config.py
+31 -4
View File
@@ -11,10 +11,14 @@ class Settings(BaseSettings):
"?streams=btcusdt@kline_1m/ethusdt@kline_1m"
)
binance_rest_url: str = "https://data-api.binance.vision"
environment: str = "development"
environment: str = "production"
ai_api_key: str = ""
ai_base_url: str = "https://api.gptsapi.net/v1"
ai_model: str = "claude-haiku-4-5-20251001"
ai_base_url: str = "https://api.deepseek.com/v1"
ai_model: str = "deepseek-v4-pro" # batch / reanalysis (quality over speed)
ai_live_model: str = "deepseek-v4-flash" # live post analysis (latency-sensitive, ~2s)
# Native Anthropic API key — if set, takes priority over ai_api_key + ai_base_url.
# Get from https://console.anthropic.com → API Keys
anthropic_api_key: str = ""
# Hyperliquid — API wallet private key (NOT your MetaMask key)
# Created at https://app.hyperliquid.xyz/API and authorized by MetaMask
@@ -30,7 +34,30 @@ class Settings(BaseSettings):
# Generate with: openssl rand -hex 32
encryption_key: str = ""
model_config = {"env_file": ".env", "env_file_encoding": "utf-8"}
# Shared secret for /api/signals/ingest — external trading modules pass it
# in the X-Ingest-Key header. Empty (default) = ingest endpoint is REJECTED
# entirely until a key is configured (fail-closed).
ingest_api_key: str = ""
# Glassnode on-chain data. Used only by the BTC bottom-reversal state
# machine (MVRV-Z + STH-SOPR). Empty = scanner logs and skips fail-closed.
glassnode_api_key: str = ""
# Etherscan API key — free at etherscan.io/register → My API Keys.
# Used for KOL A-tier: fetch all ERC-20 token balances for a given ETH address.
# Empty = skip Ethereum wallet snapshots (only HL perp positions polled).
etherscan_api_key: str = ""
# ── Telegram push alerts ─────────────────────────────────────────────────
# Bot token from @BotFather (https://t.me/BotFather → /newbot). Free.
# Empty (default) = Telegram alerts disabled completely (bot loop skipped,
# notify_signal becomes a no-op, API endpoints return 503).
telegram_bot_token: str = ""
# Bot username (no @) — used by the Settings UI to render the deep link
# t.me/<username>?start=<code>. Example: "trumpalpha_bot".
telegram_bot_username: str = ""
model_config = {"env_file": ".env", "env_file_encoding": "utf-8", "extra": "ignore"}
settings = Settings()
app/main.py
+130 -10
View File
@@ -21,6 +21,13 @@ from app.api.trades import router as trades_router
from app.api.performance import router as performance_router
from app.api.subscribe import router as subscribe_router
from app.api.user import router as user_router
from app.api.funding_signal import router as funding_signal_router
from app.api.funding_reversal import router as funding_reversal_router
from app.api.telegram import router as telegram_router
from app.api.signals import router as signals_router
from app.api.positions import router as positions_router
from app.api.scanners import router as scanners_router
from app.api.kol import router as kol_router
logging.basicConfig(
level=logging.INFO,
@@ -30,17 +37,22 @@ logger = logging.getLogger(__name__)
from typing import Optional
_binance_task: Optional[asyncio.Task] = None
_telegram_task: Optional[asyncio.Task] = None
_scheduler: Optional[AsyncIOScheduler] = None
@asynccontextmanager
async def lifespan(app: FastAPI):
global _binance_task, _scheduler
global _binance_task, _telegram_task, _scheduler
# 1. Create DB tables (dev convenience; production uses Alembic)
async with engine.begin() as conn:
await conn.run_sync(Base.metadata.create_all)
logger.info("Database tables ensured.")
# 1. Dev convenience only. Production should rely on Alembic so schema
# ownership stays explicit and startup never mutates the DB implicitly.
if settings.environment == "development":
async with engine.begin() as conn:
await conn.run_sync(Base.metadata.create_all)
logger.info("Database tables ensured (development mode).")
else:
logger.info("Skipping create_all; expecting schema managed by Alembic.")
# 2. Backfill historical posts on startup (fast, no Claude API call)
asyncio.create_task(backfill_history(AsyncSessionLocal, limit=500))
@@ -56,6 +68,18 @@ async def lifespan(app: FastAPI):
# 3. Start Truth Social poller via APScheduler
_scheduler = AsyncIOScheduler()
# Signal monitor — polls every 5 minutes
from app.services.funding_signal import poll_funding_signal
_scheduler.add_job(
poll_funding_signal,
"interval",
minutes=5,
id="funding_signal_poll",
max_instances=1,
coalesce=True,
)
logger.info("Breakout signal monitor scheduled every 5 minutes.")
_scheduler.add_job(
poll_truth_social,
"interval",
@@ -84,12 +108,88 @@ async def lifespan(app: FastAPI):
coalesce=True,
next_run_time=_dt.now(_tz.utc) + _td(seconds=offset),
)
# HL <-> DB reconciliation — every 60s, detects state drift
# (manual closes on HL UI, liquidations, orphan positions). See
# app/services/reconciler.py. Critical for live trading safety.
from app.services.reconciler import reconcile_all_once, RECONCILE_INTERVAL_SECONDS
_scheduler.add_job(
reconcile_all_once,
"interval",
seconds=RECONCILE_INTERVAL_SECONDS,
id="hl_reconcile",
max_instances=1,
coalesce=True,
)
logger.info("HL <-> DB reconciler scheduled every %ds.", RECONCILE_INTERVAL_SECONDS)
# ── System-2 bottom-reversal state machine ─────────────────────────────
# Low-frequency, long-only: fires when ≥2 of 3 classic bottom signals
# agree — AHR999 < 0.45, price ≤ 200-week MA ×1.05, Pi Cycle Bottom
# (150d EMA ≤ 471d SMA × 0.745). Funding is a booster inside the scanner,
# not an independent entry source. See app/services/scanners/btc_bottom_reversal.py.
from app.services.scanners.btc_bottom_reversal import scan_once as btc_bottom_scan
_scheduler.add_job(
btc_bottom_scan, "cron", hour=0, minute=45,
id="btc_bottom_reversal_scan", max_instances=1, coalesce=True,
)
logger.info("BTC bottom-reversal scanner scheduled daily at 00:45 UTC.")
# ── BTC funding-rate reversal (hourly) ────────────────────────────────
# Mean-reversion play on extreme perp positioning. Independent of the
# bottom-reversal state machine but uses the same `evaluate_funding_reversal`
# algorithm. Runs every hour because HL funding settles hourly.
from app.services.scanners.funding_reversal import scan_once as funding_scan
_scheduler.add_job(
funding_scan, "cron", minute=7, # :07 every hour, away from other jobs
id="funding_reversal_scan", max_instances=1, coalesce=True,
)
logger.info("Funding reversal scanner scheduled hourly at :07.")
# ── KOL Substack poller (daily) ──────────────────────────────────────
# Hayes & co publish at most a few times a week. Daily poll is plenty;
# RSS dedupe by URL is idempotent if it ever fires twice.
from app.services.kol_substack import run_substack_poll
_scheduler.add_job(
run_substack_poll, "cron", hour=1, minute=15,
id="kol_substack_poll", max_instances=1, coalesce=True,
)
logger.info("KOL Substack poller scheduled daily at 01:15 UTC.")
# ── KOL A-tier: on-chain holdings snapshot (daily) ────────────────────
# Polls HL public API (free) for perp positions; Arkham (key optional)
# for full portfolio. Diffs against yesterday's snapshot → writes
# kol_holding_changes. Runs at 02:00 UTC, after Substack poll finishes.
from app.services.kol_onchain import run_onchain_poll
_scheduler.add_job(
run_onchain_poll, "cron", hour=2, minute=0,
id="kol_onchain_poll", max_instances=1, coalesce=True,
)
logger.info("KOL on-chain holdings poller scheduled daily at 02:00 UTC.")
# ── KOL talks-vs-trades divergence scan (daily) ───────────────────────
# Runs after the on-chain poll finishes. Matches post ticker signals vs
# holding changes for the same KOL+ticker within ±7 days.
from app.services.kol_divergence import run_divergence_scan
_scheduler.add_job(
run_divergence_scan, "cron", hour=2, minute=15,
id="kol_divergence_scan", max_instances=1, coalesce=True,
)
logger.info("KOL divergence scan scheduled daily at 02:15 UTC.")
_scheduler.start()
logger.info(
"Truth Social pollers scheduled every %ds (CNN + trumpstruth.org).",
settings.truth_social_poll_seconds,
)
# ── Telegram bot long-poll loop (optional) ────────────────────────────
# Only started if TELEGRAM_BOT_TOKEN is set. Handles /start CODE bindings
# and /stop, /status, /test commands. Survives transient network errors
# via internal back-off.
from app.services.telegram_bot import run_bot_loop
_telegram_task = asyncio.create_task(run_bot_loop(), name="telegram_bot")
logger.info("Telegram bot task created (will no-op if token missing).")
yield
# Shutdown
@@ -102,6 +202,12 @@ async def lifespan(app: FastAPI):
await _binance_task
except asyncio.CancelledError:
pass
if _telegram_task and not _telegram_task.done():
_telegram_task.cancel()
try:
await _telegram_task
except asyncio.CancelledError:
pass
await engine.dispose()
logger.info("Shutdown complete.")
@@ -114,11 +220,18 @@ app = FastAPI(
)
# CORS
allowed_origins = [
settings.frontend_url,
"http://localhost:3001",
"http://localhost:3000",
]
# In production we only allow the canonical frontend origin (FRONTEND_URL).
# In development we additionally permit the local Next dev server. NEVER
# permit "*" here — every endpoint either reads/writes user-personalised
# data or accepts signed envelopes, both of which require credentialled
# requests (and "*" is rejected by browsers in combination with credentials
# anyway).
allowed_origins = [settings.frontend_url]
if settings.environment == "development":
allowed_origins.extend([
"http://localhost:3000",
"http://localhost:3001",
])
app.add_middleware(
CORSMiddleware,
allow_origins=allowed_origins,
@@ -134,6 +247,13 @@ app.include_router(trades_router, prefix="/api")
app.include_router(performance_router, prefix="/api")
app.include_router(subscribe_router, prefix="/api")
app.include_router(user_router, prefix="/api")
app.include_router(funding_signal_router, prefix="/api")
app.include_router(funding_reversal_router, prefix="/api")
app.include_router(telegram_router, prefix="/api")
app.include_router(signals_router, prefix="/api")
app.include_router(positions_router, prefix="/api")
app.include_router(scanners_router, prefix="/api")
app.include_router(kol_router, prefix="/api")
@app.get("/api/health")
app/models.py
+285
View File
@@ -60,6 +60,7 @@ class Post(Base):
target_asset: Mapped[Optional[str]] = mapped_column(String(16), nullable=True)
category: Mapped[Optional[str]] = mapped_column(String(24), nullable=True)
expected_move_pct: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
invalidation_price: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
created_at: Mapped[datetime] = mapped_column(DateTime, nullable=False, default=utcnow)
trades: Mapped[List["BotTrade"]] = relationship("BotTrade", back_populates="trigger_post")
@@ -103,6 +104,54 @@ class BotTrade(Base):
size_usd: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
leverage: Mapped[Optional[int]] = mapped_column(Integer, nullable=True)
# ── Effective exit params, FROZEN at open ───────────────────────────────
# A trade's risk profile is a property of the trade, not of the mutable
# Subscription/category config. Without these, recovery.py rehydrates
# every open position with the USER's Trump settings — silently rewriting
# a 90-day reversal's stop to 1.5% / its max-hold to 7d on every restart.
# Stamped from the resolved snapshot at open; recovery reads these back.
eff_take_profit_pct: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
eff_stop_loss_pct: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
eff_trailing_stop_pct: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
eff_trailing_activate_pct: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
eff_max_hold_hours: Mapped[Optional[int]] = mapped_column(Integer, nullable=True)
eff_invalidation: Mapped[Optional[str]] = mapped_column(String(24), nullable=True)
eff_invalidation_price: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
eff_min_hold_until_ts: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
# ── System-2 staged de-risk (分段式减仓) ─────────────────────────────────
# A System-2 bottom trade is de-risked in stages as it moves against us:
# partial reduce-only closes at the early rungs, full close at the last.
# The trade stays OPEN (closed_at NULL) until the final rung / upside
# ladder / max-hold. Legacy + System-1 rows keep the defaults so all
# existing PnL math is unchanged (remaining=1.0, partial_pnl=0, steps=0).
realized_partial_pnl_usd: Mapped[float] = mapped_column(Float, nullable=False, default=0.0)
remaining_fraction: Mapped[float] = mapped_column(Float, nullable=False, default=1.0)
derisk_steps_done: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
# ── System-2 pyramiding (做对了往上加仓) ─────────────────────────────────
# base_size_usd: the ORIGINAL notional at open (immutable). size_usd grows
# as add-ons fill; entry_price becomes the blended average. addon_steps_done
# counts executed pyramids. Pyramiding only runs while derisk_steps_done==0
# (clean uptrend), so remaining_fraction stays 1.0 throughout.
base_size_usd: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
addon_steps_done: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
# Frozen System-2 risk mode for this trade ("standard"/"aggressive").
# NULL → legacy/standard. Recovery rebuilds mode-aware ladders from this.
sys2_mode: Mapped[Optional[str]] = mapped_column(String(16), nullable=True)
# Per-trade "Grow" switch. False (default): hold + protective de-risk /
# ratchet only — NO pyramiding. True: also scale INTO this winner on
# confirmed trend. User flips it per open position. De-risk + stop-loss
# run regardless (safety floor is never user-toggleable).
grow_mode: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
# Monotonic peak unrealised gain (%) seen by tp_sl_monitor. Persisted
# (throttled) so a restart doesn't reset the regime: without it a
# pyramided / in-profit System-2 trade would fall back to peak=0 →
# underwater de-risk regime on the next restart. Recovery seeds the
# monitor from this.
peak_gain_pct: Mapped[float] = mapped_column(Float, nullable=False, default=0.0)
trigger_post: Mapped[Optional["Post"]] = relationship("Post", back_populates="trades")
@@ -127,3 +176,239 @@ class Subscription(Base):
# Both stored as naive-UTC datetimes.
active_from: Mapped[Optional[datetime]] = mapped_column(DateTime, nullable=True)
active_until: Mapped[Optional[datetime]] = mapped_column(DateTime, nullable=True)
# ── Convex-strategy fields ──────────────────────────────────────────────
# Trailing-stop pair: when profit reaches `trailing_activate_at_pct`, switch
# from fixed TP to a trailing stop at `trailing_stop_pct` below the peak.
# Both None → legacy behaviour (fixed take_profit_pct closes the position).
trailing_stop_pct: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
trailing_activate_at_pct: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
# Max hold time (hours). 168 = 7 days, long enough for trend capture.
# Replaces the old hardcoded 1h cap.
max_hold_hours: Mapped[int] = mapped_column(Integer, nullable=False, default=168)
# One-shot "enable for the next N hours" override. When set and in the future,
# the bot trades regardless of the active_from/active_until schedule. NULL or
# past timestamp → fall back to the schedule.
manual_window_until: Mapped[Optional[datetime]] = mapped_column(DateTime, nullable=True)
# ── Phase 1: safety ─────────────────────────────────────────────────────
# Paper mode: trades are simulated end-to-end (entry price from Binance,
# exit from price_store at trigger time) but no Hyperliquid call is made.
# Lets you forward-test a new signal source for 30+ days without real money.
paper_mode: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
# Circuit breaker: set by services/circuit_breaker.py when daily DD or
# consecutive-loss threshold trips. Blocks new trades for 24h and nulls
# out any active manual_window. Cleared when user explicitly re-arms via
# POST /api/user/{wallet}/manual-window.
circuit_breaker_tripped_at: Mapped[Optional[datetime]] = mapped_column(DateTime, nullable=True)
circuit_breaker_reason: Mapped[Optional[str]] = mapped_column(String(32), nullable=True)
# ── Two-system separation ───────────────────────────────────────────────
# System 1 (Trump) and System 2 (reversal) must not starve or halt each
# other. sys2_budget_pct slices the daily budget; the sys2_cb_* columns
# are an independent circuit breaker so a Trump losing streak can't lock
# out a once-a-year reversal signal (and vice versa).
sys2_budget_pct: Mapped[float] = mapped_column(Float, nullable=False, default=0.7)
sys2_cb_tripped_at: Mapped[Optional[datetime]] = mapped_column(DateTime, nullable=True)
sys2_cb_reason: Mapped[Optional[str]] = mapped_column(String(32), nullable=True)
# ── System-2 dynamic leverage ───────────────────────────────────────────
# System 2 (bottom reversal) uses its OWN leverage, independent of the
# Trump `leverage` field. The user picks it freely; the value at signal
# time is frozen onto the trade. The protective stop in tp_sl_monitor is
# auto-scaled to this leverage so the position is de-risked INSIDE the
# exchange liquidation line — it is never liquidated by the exchange.
# NULL → fall back to SYS2_DEFAULT_LEVERAGE (signal_categories).
sys2_leverage: Mapped[Optional[int]] = mapped_column(Integer, nullable=True)
# System-2 risk mode: "standard" (tuned cycle-rider, low leverage) or
# "aggressive" (separately-funded high-risk sleeve: high leverage, heavier
# earlier pyramiding, wider peak-trail, lighter early de-risk). Both keep
# the never-exchange-liquidated + post-pyramid-breakeven invariants.
sys2_mode: Mapped[str] = mapped_column(String(16), nullable=False, default="standard")
# ── Master Auto-Trade switch (simplified operator model) ────────────────
# The ONE persistent gate the user controls. OFF (default, safe): signals
# are still scanned + ingested (shown in the feed) but NO trade is opened.
# ON: a qualifying signal auto-opens a trade. Replaces the old confusing
# scanner-toggle / timed manual-window / schedule trio. Flipping it ON also
# acknowledges + clears a tripped circuit breaker (human-in-the-loop).
auto_trade: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
class KolPost(Base):
"""A long-form post (Substack) or tweet from a tracked KOL, with inline
AI analysis. Standalone module — no FK into Trump posts/trades.
Dedupe key: (source, external_id). For Substack external_id is the post
URL; for Twitter it will be the tweet id.
"""
__tablename__ = "kol_posts"
__table_args__ = (UniqueConstraint("source", "external_id", name="uq_kol_post_src_extid"),)
id: Mapped[int] = mapped_column(Integer, primary_key=True, index=True)
kol_handle: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
source: Mapped[str] = mapped_column(String(16), nullable=False) # substack|twitter
external_id: Mapped[str] = mapped_column(String(512), nullable=False)
title: Mapped[Optional[str]] = mapped_column(String(512), nullable=True)
url: Mapped[str] = mapped_column(String(512), nullable=False)
published_at: Mapped[datetime] = mapped_column(DateTime, nullable=False, index=True)
raw_text: Mapped[str] = mapped_column(Text, nullable=False)
content_hash: Mapped[str] = mapped_column(String(64), nullable=False)
summary: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
tickers_json: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
analyzed_at: Mapped[Optional[datetime]] = mapped_column(DateTime, nullable=True)
analysis_model: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
analysis_version: Mapped[Optional[str]] = mapped_column(String(16), nullable=True)
created_at: Mapped[datetime] = mapped_column(DateTime, nullable=False, default=utcnow)
# ── KOL A-tier: on-chain holdings ────────────────────────────────────────────
class KolWallet(Base):
"""One tracked wallet address for a KOL. Many-to-one with handle."""
__tablename__ = "kol_wallets"
__table_args__ = (UniqueConstraint("chain", "address", name="uq_kol_wallet_chain_addr"),)
id: Mapped[int] = mapped_column(Integer, primary_key=True)
handle: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
chain: Mapped[str] = mapped_column(String(16), nullable=False) # ethereum|solana|hl
address: Mapped[str] = mapped_column(String(128), nullable=False)
label: Mapped[Optional[str]] = mapped_column(String(128), nullable=True) # e.g. "hot wallet"
source_url: Mapped[Optional[str]] = mapped_column(String(256), nullable=True) # Arkham entity URL
active: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True)
added_at: Mapped[datetime] = mapped_column(DateTime, nullable=False, default=utcnow)
snapshots: Mapped[List["KolHoldingSnapshot"]] = relationship(
"KolHoldingSnapshot", back_populates="wallet", order_by="KolHoldingSnapshot.snapshot_date.desc()"
)
changes: Mapped[List["KolHoldingChange"]] = relationship(
"KolHoldingChange", back_populates="wallet"
)
class KolHoldingSnapshot(Base):
"""Daily portfolio snapshot for one wallet. holdings_json is a list of
{ticker, amount, usd_value, chain} dicts. One row per (wallet, date)."""
__tablename__ = "kol_holdings_snapshots"
__table_args__ = (UniqueConstraint("wallet_id", "snapshot_date", name="uq_kol_snapshot_wallet_date"),)
id: Mapped[int] = mapped_column(Integer, primary_key=True)
wallet_id: Mapped[int] = mapped_column(Integer, ForeignKey("kol_wallets.id"), nullable=False, index=True)
snapshot_date: Mapped[str] = mapped_column(String(10), nullable=False) # YYYY-MM-DD UTC
holdings_json: Mapped[str] = mapped_column(Text, nullable=False) # JSON list
total_usd: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
source: Mapped[str] = mapped_column(String(16), nullable=False) # arkham|hl|manual
created_at: Mapped[datetime] = mapped_column(DateTime, nullable=False, default=utcnow)
wallet: Mapped["KolWallet"] = relationship("KolWallet", back_populates="snapshots")
class KolHoldingChange(Base):
"""A detected significant change between two consecutive daily snapshots."""
__tablename__ = "kol_holding_changes"
id: Mapped[int] = mapped_column(Integer, primary_key=True)
wallet_id: Mapped[int] = mapped_column(Integer, ForeignKey("kol_wallets.id"), nullable=False, index=True)
detected_at: Mapped[datetime] = mapped_column(DateTime, nullable=False, index=True, default=utcnow)
ticker: Mapped[str] = mapped_column(String(32), nullable=False)
change_type: Mapped[str] = mapped_column(String(16), nullable=False) # new_position|closed|increased|decreased
usd_before: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
usd_after: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
pct_change: Mapped[Optional[float]] = mapped_column(Float, nullable=True) # signed %
created_at: Mapped[datetime] = mapped_column(DateTime, nullable=False, default=utcnow)
wallet: Mapped["KolWallet"] = relationship("KolWallet", back_populates="changes")
class KolDivergence(Base):
"""Cross-signal: a (post, on-chain change) pair for the same KOL+ticker
within a ±7-day window.
signal_type='divergence' → KOL's public post contradicts their on-chain action
(says bullish, actually selling → smart-money caution)
signal_type='alignment' → post and chain agree → higher-conviction signal
direction='long'/'short' → net conclusion after resolving the pair
"""
__tablename__ = "kol_divergence"
__table_args__ = (
UniqueConstraint("post_id", "change_id", name="uq_kol_divergence_pair"),
)
id: Mapped[int] = mapped_column(Integer, primary_key=True)
handle: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
ticker: Mapped[str] = mapped_column(String(32), nullable=False)
# B-tier content side
post_id: Mapped[int] = mapped_column(Integer, ForeignKey("kol_posts.id"), nullable=False)
post_action: Mapped[str] = mapped_column(String(16), nullable=False)
post_conviction: Mapped[Optional[float]]= mapped_column(Float, nullable=True)
post_at: Mapped[datetime] = mapped_column(DateTime, nullable=False)
# A-tier on-chain side
change_id: Mapped[int] = mapped_column(Integer, ForeignKey("kol_holding_changes.id"), nullable=False)
onchain_action: Mapped[str] = mapped_column(String(16), nullable=False)
usd_before: Mapped[Optional[float]]= mapped_column(Float, nullable=True)
usd_after: Mapped[Optional[float]]= mapped_column(Float, nullable=True)
onchain_at: Mapped[datetime] = mapped_column(DateTime, nullable=False)
# Classification
signal_type: Mapped[str] = mapped_column(String(16), nullable=False) # divergence|alignment
direction: Mapped[Optional[str]] = mapped_column(String(8), nullable=True) # long|short
days_apart: Mapped[Optional[float]]= mapped_column(Float, nullable=True)
created_at: Mapped[datetime] = mapped_column(DateTime, nullable=False, default=utcnow)
post: Mapped["KolPost"] = relationship("KolPost")
change: Mapped["KolHoldingChange"] = relationship("KolHoldingChange")
class TelegramBinding(Base):
"""Wallet ↔ Telegram chat_id mapping plus per-source alert preferences.
Two flavours of binding co-exist in this table:
Free tier (walletless): user DM'd the bot `/start` with no code.
wallet_address is NULL. They get the same
push content as Pro users unless they tune
preferences in the bot.
Pro tier (wallet-linked): user generated a 6-char code in Settings,
replied `/start CODE` in the bot. wallet
is attached so the dashboard knows which
chat to disconnect / link / show status.
Uniqueness: chat_id is unique across all rows (table-level). wallet_address
is unique among non-NULL rows via the partial index in migration 021 —
NEVER use SQLAlchemy `unique=True` here because some dialects would emit a
plain UNIQUE that disallows multiple walletless rows.
Preferences (alert_*, min_confidence, mute_*) are now mutated exclusively
via the bot's /trump /btc /funding /kol /conf /quiet commands. The
/api/telegram/preferences endpoint exists for legacy reasons but no UI
surface calls it.
"""
__tablename__ = "telegram_bindings"
id: Mapped[int] = mapped_column(Integer, primary_key=True)
# NULL is allowed — see class docstring. Uniqueness for non-NULL values
# is enforced by the partial unique index defined in migration 021.
wallet_address: Mapped[Optional[str]] = mapped_column(String(64), nullable=True,
index=True)
chat_id: Mapped[int] = mapped_column(BigInteger, nullable=False,
unique=True, index=True)
tg_username: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
bound_at: Mapped[datetime] = mapped_column(DateTime, nullable=False, default=utcnow)
alerts_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True)
alert_trump: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True)
alert_btc_bottom: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True)
alert_funding: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True)
alert_kol_divergence: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
min_confidence: Mapped[int] = mapped_column(Integer, nullable=False, default=70)
# Quiet hours (UTC). Both null = always on.
mute_from_hour: Mapped[Optional[int]] = mapped_column(Integer, nullable=True)
mute_until_hour: Mapped[Optional[int]] = mapped_column(Integer, nullable=True)
last_alert_at: Mapped[Optional[datetime]] = mapped_column(DateTime, nullable=True)
total_alerts_sent: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
total_alerts_failed: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
app/schemas.py
+27 -3
View File
@@ -37,6 +37,7 @@ class TrumpPost(BaseModel):
target_asset: Optional[str] = None
category: Optional[str] = None
expected_move_pct: Optional[float] = None
invalidation_price: Optional[float] = None
model_config = {"from_attributes": True}
@@ -61,6 +62,13 @@ class BotTrade(BaseModel):
trigger_post_id: int
opened_at: str
closed_at: str
# Source tag of the originating signal (e.g. 'truth', 'breakout', 'my_strategy').
# Joined from posts.source on read; not stored on BotTrade itself.
# 'unknown' when the trigger post has been deleted or trigger_post_id is null.
trigger_source: Optional[str] = None
# True iff this was a paper trade (hl_order_id == 'paper'). Lets the UI
# tag the row so paper-mode P&L isn't mixed with real money in summaries.
is_paper: bool = False
model_config = {"from_attributes": True}
@@ -87,12 +95,17 @@ class SignedEnvelope(BaseModel):
class SubscribeRequest(SignedEnvelope):
pass
# Optional paper-mode flag. When true the bot writes simulated trades to
# the DB but never hits Hyperliquid — safe path for new users to try the
# system without risking funds. Defaults to false (live) for backwards
# compatibility with the existing signed-message format (body=None).
paper_mode: Optional[bool] = False
class SubscribeResponse(BaseModel):
status: str
wallet: str
status: str
wallet: str
paper_mode: bool = False
class SetApiKeyRequest(SignedEnvelope):
@@ -102,6 +115,7 @@ class SetApiKeyRequest(SignedEnvelope):
class SetApiKeyResponse(BaseModel):
status: str
masked_key: str # last 6 chars only, e.g. "...a1b2c3"
verified: bool = False
class UserSettings(BaseModel):
@@ -111,6 +125,13 @@ class UserSettings(BaseModel):
stop_loss_pct: Optional[float] = None
min_confidence: int
daily_budget_usd: Optional[float] = None
# System-2 (bottom reversal) leverage — independent of `leverage` (Trump).
# None → platform default (SYS2_DEFAULT_LEVERAGE). The protective stop is
# auto-scaled to this so the position is never exchange-liquidated.
sys2_leverage: Optional[int] = None
# System-2 risk mode: "standard" (default) or "aggressive" (separately
# funded high-risk/high-explosiveness sleeve). None → unchanged/standard.
sys2_mode: Optional[str] = None
# ISO-8601 UTC strings; both None = always on (Subscription.active still gates it).
active_from: Optional[str] = None
active_until: Optional[str] = None
@@ -128,3 +149,6 @@ class UserResponse(BaseModel):
hl_api_key_masked: Optional[str] = None
trades: list[BotTrade]
settings: UserSettings
# Convex-strategy: ISO-UTC timestamp until which the bot is manually armed.
# When null or in the past, the regular active_from/active_until schedule applies.
manual_window_until: Optional[str] = None
app/scrapers/trumpstruth.py
+12 -1
View File
@@ -53,7 +53,10 @@ async def _fetch_feed() -> Optional[str]:
resp.raise_for_status()
return resp.text
except Exception as exc:
logger.warning("Failed to fetch trumpstruth.org feed: %s", exc)
# Include type name — httpx often raises bare ConnectError/RemoteProtocolError
# with empty .args, which formats as just "Failed to fetch ..." with no body.
logger.warning("Failed to fetch trumpstruth.org feed: %s (%s)",
type(exc).__name__, exc)
return None
@@ -141,6 +144,14 @@ async def poll_trumpstruth(db_session_factory) -> None:
await manager.broadcast(_post_to_ws_payload(post))
logger.info("[trumpstruth] beat CNN — new post id=%d: %s",
post.id, post.text[:60])
# Telegram fan-out — matches truth_social.py. Without this,
# whichever poller wins the race determines whether users
# get pushed — flaky 50% delivery.
try:
from app.services.telegram import notify_signal
notify_signal(post)
except Exception as exc:
logger.warning("Telegram notify failed for post %d: %s", post.id, exc)
try:
from app.services.bot_engine import process_post
await process_post(post, db)
app/scrapers/truth_social.py
+40 -1
View File
@@ -63,7 +63,11 @@ async def _fetch_archive() -> Optional[list]:
resp.raise_for_status()
return resp.json()
except Exception as exc:
logger.error("Failed to fetch CNN archive: %s", exc)
# Include type name — httpx often raises bare ConnectError/TimeoutException
# with empty .args, which used to log as just "Failed to fetch CNN archive:"
# with no body, making outages impossible to diagnose.
logger.error("Failed to fetch CNN archive: %s (%s)",
type(exc).__name__, exc)
return None
@@ -80,6 +84,31 @@ async def _process_entry(entry: dict, db: AsyncSession) -> Optional[Post]:
published_at = _parse_dt(entry.get("created_at", ""))
# ── Deterministic entry pre-filter (saves AI spend + blocks 80% of noise) ──
# The 13-trade backtest showed AI confidence ≥ 85 still lets through pure
# rhetoric and second-derivative news. Hard-coded action-marker + future-
# tense + dedup check rejects those BEFORE the AI call. Failing posts are
# still saved to DB (so we have a record) but stamped as non-actionable.
from app.database import AsyncSessionLocal
from app.services.entry_filter import passes_entry_filter
filter_ok, filter_reason = await passes_entry_filter(text, AsyncSessionLocal)
if not filter_ok:
logger.info("Entry filter rejected post (id=%s): %s — %s",
entry.get("id"), filter_reason, text[:80])
# Insert a stub row so we don't keep re-fetching the same post from
# the upstream archive. Signal=hold, no AI call, no analysis.
stub = Post(
external_id=external_id, text=text, source="truth",
published_at=published_at,
sentiment="neutral", ai_confidence=0,
relevant=False, signal="hold",
prefilter_reason=filter_reason[:32],
analysis_version="entry_filter_rejected",
)
db.add(stub)
await db.flush()
return None # don't broadcast, don't trade
analysis = await analyze_post(text)
asset = analysis["asset"]
@@ -154,6 +183,9 @@ def _post_to_ws_payload(post: Post) -> dict:
"ai_confidence": post.ai_confidence,
"ai_reasoning": post.ai_reasoning,
"relevant": post.relevant,
"target_asset": post.target_asset,
"category": post.category,
"expected_move_pct": post.expected_move_pct,
"price_impact": price_impact,
},
}
@@ -187,6 +219,13 @@ async def poll_truth_social(db_session_factory) -> None:
for post in new_posts:
await manager.broadcast(_post_to_ws_payload(post))
logger.info("Saved new post id=%d: %s", post.id, post.text[:60])
# Telegram fan-out (fire-and-forget). Only fires if
# signal is buy/short; noise posts are filtered inside.
try:
from app.services.telegram import notify_signal
notify_signal(post)
except Exception as exc:
logger.warning("Telegram notify failed for post %d: %s", post.id, exc)
try:
from app.services.bot_engine import process_post
await process_post(post, db)
app/services/analysis.py
+62 -12
View File
@@ -32,6 +32,7 @@ from app.config import settings
logger = logging.getLogger(__name__)
_client: Optional[AsyncOpenAI] = None
_anthropic_client = None
ANALYSIS_VERSION = "v5-extreme-alpha"
@@ -373,6 +374,19 @@ def _fallback(prefilter: Optional[str] = None, reasoning: str = "") -> dict:
return out
def _use_anthropic() -> bool:
"""True if a native Anthropic API key is configured (takes priority over proxy)."""
return bool(settings.anthropic_api_key)
def _get_anthropic_client():
global _anthropic_client
if _anthropic_client is None:
import anthropic as _anthropic
_anthropic_client = _anthropic.AsyncAnthropic(api_key=settings.anthropic_api_key)
return _anthropic_client
def _get_client() -> AsyncOpenAI:
global _client
if _client is None:
@@ -393,13 +407,21 @@ def _liquidity_note(hour: int) -> str:
return "Off-peak hours, moderate liquidity"
async def analyze_post(text: str) -> dict:
async def analyze_post(text: str, model: Optional[str] = None) -> dict:
"""Score a Trump post and return signal + structured reasoning.
Args:
text: Raw post text (up to 2000 chars used).
model: Override the model to use. Defaults to settings.ai_live_model
for latency-sensitive live calls; pass settings.ai_model
explicitly for higher-quality batch reanalysis.
Returns the canonical dict shape expected by the rest of the codebase:
relevant, asset, sentiment, signal (buy/short/hold), confidence,
reasoning, prefilter_reason, analysis_version.
"""
if model is None:
model = settings.ai_live_model # fast flash for live posts
# ── Fast pre-filter (no AI call) ────────────────────────────────
stripped = text.strip()
if not stripped:
@@ -423,17 +445,45 @@ async def analyze_post(text: str) -> dict:
)
try:
client = _get_client()
response = await client.chat.completions.create(
model=settings.ai_model,
max_tokens=600,
temperature=0.1,
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": user_prompt},
],
)
raw = (response.choices[0].message.content or "").strip()
if _use_anthropic():
# ── Native Anthropic SDK path ────────────────────────────────
anthropic_client = _get_anthropic_client()
# Map OpenAI-style model name → Anthropic model name
model_name = model
if "haiku" in model_name.lower() and "claude-" not in model_name.lower():
model_name = "claude-haiku-4-5-20251001"
msg = await anthropic_client.messages.create(
model=model_name,
max_tokens=600,
temperature=0.1,
system=SYSTEM_PROMPT,
messages=[{"role": "user", "content": user_prompt}],
)
raw = (msg.content[0].text if msg.content else "").strip()
else:
# ── OpenAI-compatible proxy path (DeepSeek, gptsapi, etc.) ──
# Reasoning models (deepseek-v4-pro / R1) don't support
# temperature and need higher max_tokens for the thinking pass.
model_name = model
is_reasoning = any(x in model_name for x in ("pro", "reasoner", "r1", "think"))
kwargs: dict = {
"model": model_name,
"messages": [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": user_prompt},
],
}
if is_reasoning:
# Reasoning models: large token budget; omit temperature
kwargs["max_tokens"] = 4000
else:
kwargs["max_tokens"] = 1200
kwargs["temperature"] = 0.1
client = _get_client()
response = await client.chat.completions.create(**kwargs)
raw = (response.choices[0].message.content or "").strip()
# Strip ```json fences if the model adds them despite instructions
if raw.startswith("```"):
app/services/backtest.py
+376
View File
@@ -0,0 +1,376 @@
"""
Single-post backtest harness.
Given a Post with a directional signal (buy/short) and a target asset, fetch
the historical 1-minute candles for [published_at, published_at + max_hold_h]
from Binance and replay the new convex-strategy exit rules. Outputs what
WOULD have happened if we had been live at that moment with the current
trailing-stop / stop-loss / max-hold settings.
Why this exists:
The bot's m1h accuracy is 45.7%, but that was measured with a 1-hour
fixed-window snapshot — i.e. assuming we close at exactly the 1-hour mark.
The new exit logic (trailing stop, 7-day max hold) is FUNDAMENTALLY
different. Without backtesting we can't know whether changing exits also
changes the win rate / PnL profile. This harness lets us check.
Scope (deliberately narrow for the MVP):
- One post at a time. Batch runner sits on top.
- Uses 1m HIGH/LOW within each bar (worst-case path) — conservative.
- Ignores fees + slippage (caller is expected to subtract them).
- Assumes immediate fill at the candle's OPEN on entry.
- Does NOT re-evaluate regime gates (the caller decides whether to
include the post). This makes "what would have happened" cleaner.
The 1m granularity matters: with 5m or 1H bars you can't distinguish
"stop loss hit then bounced back" from "rallied straight up", and a
trailing-stop strategy lives or dies on that distinction.
"""
from __future__ import annotations
import logging
from dataclasses import dataclass, asdict
from datetime import datetime, timedelta, timezone
from typing import Optional
import httpx
logger = logging.getLogger(__name__)
BINANCE_KLINES_URL = "https://api.binance.com/api/v3/klines"
@dataclass
class BacktestResult:
post_id: int
asset: str
side: str # "long" | "short"
entry_price: float
exit_price: float
exit_reason: str # "trailing_stop" | "stop_loss" | "take_profit" | "max_hold"
hold_minutes: int
pnl_pct: float # net of nothing — apply your own fee model
peak_pct: float # best unrealised gain reached
trough_pct: float # worst unrealised gain reached
bars_evaluated: int
def to_dict(self) -> dict:
return asdict(self)
@dataclass
class BacktestParams:
"""Exit-rule snapshot used for the simulation."""
stop_loss_pct: float = 1.5
trailing_stop_pct: Optional[float] = 2.5
trailing_activate_at_pct: Optional[float] = 5.0
take_profit_pct: Optional[float] = None
max_hold_hours: int = 168
async def fetch_binance_1m(
symbol: str,
start: datetime,
end: datetime,
client: Optional[httpx.AsyncClient] = None,
) -> list[dict]:
"""Pull 1-minute candles from Binance spot for [start, end].
Binance caps a single call at 1000 candles → we page in chunks. Returns
candles in chronological order. Each candle: {time_ms, open, high, low, close}.
"""
own_client = client is None
if own_client:
client = httpx.AsyncClient(timeout=20)
out: list[dict] = []
cursor_ms = int(start.replace(tzinfo=timezone.utc).timestamp() * 1000)
end_ms = int(end.replace(tzinfo=timezone.utc).timestamp() * 1000)
try:
while cursor_ms < end_ms:
params = {
"symbol": symbol,
"interval": "1m",
"startTime": cursor_ms,
"endTime": end_ms,
"limit": 1000,
}
resp = await client.get(BINANCE_KLINES_URL, params=params)
resp.raise_for_status()
chunk = resp.json()
if not chunk:
break
for row in chunk:
out.append({
"time_ms": row[0],
"open": float(row[1]),
"high": float(row[2]),
"low": float(row[3]),
"close": float(row[4]),
})
# Next page starts after the last candle returned.
last_open = chunk[-1][0]
if last_open <= cursor_ms:
break
cursor_ms = last_open + 60_000
if len(chunk) < 1000:
break # final partial page
finally:
if own_client:
await client.aclose()
return out
def _asset_to_symbol(asset: str) -> str:
"""Map our internal asset code to a Binance spot symbol.
Most assets are simply {ASSET}USDT. TRUMP/HYPE/etc. may not exist on
Binance — those will fail at fetch time and we'll return None upstream.
"""
return f"{asset.upper()}USDT"
def simulate_exit(
candles: list[dict],
side: str,
params: BacktestParams,
) -> dict:
"""Walk the candles 1m at a time, applying the exit ladder.
Priority within a bar (worst-case ordering):
1. Stop loss — assume hit via LOW (long) / HIGH (short)
2. Trailing — only if armed; assume hit via the same worst-case extreme
3. Take profit — fixed TP if set
A real bar can't tell us whether the high or low printed first, so we
take the pessimistic path: STOP LOSS BEFORE PROFIT if both could have
triggered. This makes the backtest conservative.
"""
if not candles:
return {"reason": "no_data", "exit_price": 0.0, "bars": 0,
"peak_pct": 0.0, "trough_pct": 0.0, "pnl_pct": 0.0}
entry = candles[0]["open"]
if entry == 0:
return {"reason": "bad_entry", "exit_price": 0.0, "bars": 0,
"peak_pct": 0.0, "trough_pct": 0.0, "pnl_pct": 0.0}
is_long = side == "long"
peak_pct = 0.0
trough_pct = 0.0
armed = False
def pct(price: float) -> float:
raw = (price - entry) / entry
return (raw if is_long else -raw) * 100
for i, bar in enumerate(candles):
# Within-bar extremes in position's favoured direction
good_extreme = bar["high"] if is_long else bar["low"]
bad_extreme = bar["low"] if is_long else bar["high"]
good_pct = pct(good_extreme)
bad_pct = pct(bad_extreme)
# Update running peak / trough using extremes
if good_pct > peak_pct: peak_pct = good_pct
if bad_pct < trough_pct: trough_pct = bad_pct
# 1. Stop loss — pessimistic check first
if bad_pct <= -params.stop_loss_pct:
# Approximate exit at the SL trigger price (not the bar extreme)
sl_price = entry * (1 - params.stop_loss_pct / 100) if is_long \
else entry * (1 + params.stop_loss_pct / 100)
return {
"reason": "stop_loss",
"exit_price": sl_price,
"bars": i + 1,
"peak_pct": peak_pct,
"trough_pct": trough_pct,
"pnl_pct": -params.stop_loss_pct,
}
# 2. Trailing — arm if peak crossed activation
if (params.trailing_stop_pct is not None
and params.trailing_activate_at_pct is not None):
if not armed and peak_pct >= params.trailing_activate_at_pct:
armed = True
if armed:
# Drawdown from peak (using bad_extreme of this bar)
drawdown = peak_pct - bad_pct
if drawdown >= params.trailing_stop_pct:
# Exit at peak − trailing_stop_pct (approx)
exit_pct = peak_pct - params.trailing_stop_pct
ts_price = entry * (1 + exit_pct / 100) if is_long \
else entry * (1 - exit_pct / 100)
return {
"reason": "trailing_stop",
"exit_price": ts_price,
"bars": i + 1,
"peak_pct": peak_pct,
"trough_pct": trough_pct,
"pnl_pct": exit_pct,
}
# 3. Fixed TP
if params.take_profit_pct is not None and good_pct >= params.take_profit_pct:
tp_price = entry * (1 + params.take_profit_pct / 100) if is_long \
else entry * (1 - params.take_profit_pct / 100)
return {
"reason": "take_profit",
"exit_price": tp_price,
"bars": i + 1,
"peak_pct": peak_pct,
"trough_pct": trough_pct,
"pnl_pct": params.take_profit_pct,
}
# Walked the whole window without hitting any rule → close at last bar
last_close = candles[-1]["close"]
return {
"reason": "max_hold",
"exit_price": last_close,
"bars": len(candles),
"peak_pct": peak_pct,
"trough_pct": trough_pct,
"pnl_pct": pct(last_close),
}
async def backtest_post(
post_id: int,
params: Optional[BacktestParams] = None,
) -> Optional[BacktestResult]:
"""Backtest a single Post by ID.
Returns None if:
- Post not found
- signal is not buy/short
- target_asset is unknown / not on Binance
- Binance has no data for the window
"""
from app.database import AsyncSessionLocal
from app.models import Post
from sqlalchemy import select
params = params or BacktestParams()
async with AsyncSessionLocal() as db:
result = await db.execute(select(Post).where(Post.id == post_id))
post = result.scalar_one_or_none()
if post is None:
logger.warning("Backtest: post %d not found", post_id)
return None
if post.signal not in ("buy", "short"):
logger.warning("Backtest: post %d signal=%s not actionable", post_id, post.signal)
return None
asset = post.target_asset or post.price_impact_asset
if not asset:
logger.warning("Backtest: post %d has no asset", post_id)
return None
side = "long" if post.signal == "buy" else "short"
symbol = _asset_to_symbol(asset)
# Window: [published_at, published_at + max_hold_hours]
start = post.published_at
end = start + timedelta(hours=params.max_hold_hours)
# Don't backtest a window that hasn't fully elapsed yet
now_naive = datetime.now(timezone.utc).replace(tzinfo=None)
if end > now_naive:
end = now_naive
candles = await fetch_binance_1m(symbol, start, end)
if not candles:
logger.warning("Backtest: no Binance data for %s in [%s, %s]", symbol, start, end)
return None
sim = simulate_exit(candles, side, params)
if sim["reason"] in ("no_data", "bad_entry"):
return None
return BacktestResult(
post_id=post_id,
asset=asset,
side=side,
entry_price=candles[0]["open"],
exit_price=sim["exit_price"],
exit_reason=sim["reason"],
hold_minutes=sim["bars"],
pnl_pct=round(sim["pnl_pct"], 3),
peak_pct=round(sim["peak_pct"], 3),
trough_pct=round(sim["trough_pct"], 3),
bars_evaluated=sim["bars"],
)
async def backtest_batch(
limit: int = 50,
min_confidence: int = 80,
params: Optional[BacktestParams] = None,
) -> dict:
"""Backtest every directional post with confidence ≥ min_confidence.
Returns aggregate stats + per-trade results. Posts whose Binance data
is missing (TRUMP, niche perps) are skipped silently.
"""
from app.database import AsyncSessionLocal
from app.models import Post
from sqlalchemy import select, and_, or_
params = params or BacktestParams()
async with AsyncSessionLocal() as db:
rows = await db.execute(
select(Post.id).where(
and_(
Post.signal.in_(["buy", "short"]),
Post.ai_confidence >= min_confidence,
)
).order_by(Post.published_at.desc()).limit(limit)
)
post_ids = [r[0] for r in rows.all()]
results: list[BacktestResult] = []
skipped: list[int] = []
for pid in post_ids:
try:
r = await backtest_post(pid, params)
if r is None:
skipped.append(pid)
else:
results.append(r)
except Exception as exc:
logger.error("Backtest post %d failed: %s", pid, exc)
skipped.append(pid)
if not results:
return {"params": asdict(params), "results": [], "skipped": skipped,
"summary": {"trades": 0}}
pnls = [r.pnl_pct for r in results]
wins = [p for p in pnls if p > 0]
losses = [p for p in pnls if p <= 0]
by_reason: dict[str, int] = {}
for r in results:
by_reason[r.exit_reason] = by_reason.get(r.exit_reason, 0) + 1
summary = {
"trades": len(results),
"skipped": len(skipped),
"win_rate_pct": round(len(wins) / len(results) * 100, 1),
"avg_pnl_pct": round(sum(pnls) / len(pnls), 3),
"total_pnl_pct": round(sum(pnls), 3),
"best_pct": round(max(pnls), 3),
"worst_pct": round(min(pnls), 3),
"avg_win_pct": round(sum(wins) / len(wins), 3) if wins else 0.0,
"avg_loss_pct": round(sum(losses) / len(losses), 3) if losses else 0.0,
"exit_reasons": by_reason,
}
return {
"params": asdict(params),
"summary": summary,
"skipped": skipped,
"results": [r.to_dict() for r in results],
}
app/services/binance.py
+28 -2
View File
@@ -83,7 +83,21 @@ async def fetch_historical(asset: str, symbol: str, interval: str = "1m", limit:
async def run_binance_ws():
"""Connect to Binance kline stream with exponential back-off on disconnect."""
"""Connect to Binance kline stream with exponential back-off on disconnect.
Binance routinely cycles connections every ~24h, and intermediate proxies
can drop without sending a close frame — both surface here as exceptions.
The reconnect must be fast (sub-second) so price ticks aren't visibly
stuck on the dashboard during a hiccup.
Resilience defenses:
* ping_interval/ping_timeout — peer-of-record liveness check (websockets lib)
* close_timeout — bound how long we wait for a clean close handshake
* open_timeout — bound the initial connect handshake so a hung TCP doesn't
permanently park this task
* max_queue — drop frames if downstream can't keep up rather than back
the WS receive buffer up indefinitely
"""
# Pre-fill with historical data
await fetch_historical("BTC", "btcusdt", limit=500)
await fetch_historical("ETH", "ethusdt", limit=500)
@@ -96,15 +110,27 @@ async def run_binance_ws():
settings.binance_ws_url,
ping_interval=20,
ping_timeout=10,
close_timeout=5,
open_timeout=10,
max_queue=128,
) as ws:
backoff = 1 # reset on successful connection
logger.info("Binance WebSocket connected.")
async for message in ws:
await _process_message(message)
# If we exit the `async with` without an exception, the server
# closed normally — reconnect immediately rather than backing off.
logger.info("Binance WebSocket closed cleanly, reconnecting immediately.")
except asyncio.CancelledError:
logger.info("Binance WebSocket task cancelled.")
return
except Exception as exc:
logger.error("Binance WebSocket error: %s. Reconnecting in %ds.", exc, backoff)
# Use exception() once so a TRACE-back-with-context is logged the
# first time something interesting happens; subsequent same-error
# retries log compactly to avoid drowning the log.
logger.warning(
"Binance WebSocket error: %s (%s). Reconnecting in %ds.",
type(exc).__name__, exc, backoff,
)
await asyncio.sleep(backoff)
backoff = min(backoff * 2, 60)
app/services/bot_engine.py
+707 -63
View File
@@ -6,7 +6,7 @@ Iterates all active subscribers and executes trades on their behalf.
import asyncio
import logging
from datetime import datetime, timezone
from datetime import datetime, timezone, timedelta
from typing import Dict
from sqlalchemy import select, update
@@ -23,7 +23,9 @@ logger = logging.getLogger(__name__)
# Platform-wide thresholds (per-user values in Subscription override where applicable)
# No global confidence floor — users pick their own 0–100 threshold via settings.
MAX_HOLD_SECONDS = 3600 # force-close after 1 hour (shared across users)
# Per-user max hold lives on Subscription.max_hold_hours (default 168h = 7 days
# in the convex-strategy redesign). Old 1-hour cap killed every potential
# runner before it could compound, so it's been removed.
# Hyperliquid perp fees (mainnet, base tier).
# IOC orders always cross the book → taker fee both on open and close.
@@ -63,6 +65,17 @@ def _lock_for(trade_id: int) -> asyncio.Lock:
return lock
def _confidence_floor_for(sub: dict) -> int:
if sub.get("_is_system_2"):
from app.services.signal_categories import system2_min_confidence
return system2_min_confidence()
return sub["min_confidence"]
def _should_apply_schedule(sub: dict) -> bool:
return not bool(sub.get("_is_system_2"))
async def process_post(post: Post, db: AsyncSession) -> None:
"""
Entry point called by the scraper after a new post is saved.
@@ -91,6 +104,87 @@ async def process_post(post: Post, db: AsyncSession) -> None:
logger.info("Signal: post=%d asset=%s side=%s confidence=%d", post.id, asset, side, post.ai_confidence)
# ── System routing ─────────────────────────────────────────────────────
# Two independent trading systems share this execution layer but NOT
# their strategy logic. See app/services/signal_categories.py.
from app.services.signal_categories import (
get_exit_profile, get_stop_ladder, is_supported_trading_source,
is_system_2, system2_display_name,
)
if not is_supported_trading_source(post.source):
logger.info("Post %d skipped: unsupported trading source=%s", post.id, post.source)
return
sys2 = is_system_2(post.source)
if sys2:
# SYSTEM 2 — reversal/breakout. The Trump-tuned regime filter
# (recent-move ≤5%, vol-contraction) actively REJECTS valid reversal
# setups, so it's bypassed entirely. The signal's structural gates
# (RSI<30 ×4wk, 200d reclaim, funding extreme) ARE the regime check.
exit_profile = get_exit_profile(post.category)
if exit_profile.stop_ladder:
logger.info(
"%s [%s/%s]: bypassing Trump regime filter. "
"Exit = STAGED stop (no TP/trailing) base sl=%.1f%% "
"ladder=%s maxhold=%dh",
system2_display_name(), post.source, post.category,
exit_profile.stop_loss_pct, exit_profile.stop_ladder,
exit_profile.max_hold_hours,
)
else:
logger.info(
"%s [%s/%s]: bypassing Trump regime filter. "
"Exit profile sl=%.1f%% trail=%s@%s maxhold=%dh inval=%s",
system2_display_name(), post.source, post.category,
exit_profile.stop_loss_pct, exit_profile.trailing_stop_pct,
exit_profile.trailing_activate_at_pct, exit_profile.max_hold_hours,
exit_profile.invalidation,
)
else:
# SYSTEM 1 — Trump. REPOSITIONED: low-frequency, selective, tight
# stop, ≥30-min holds. See signal_categories.py.
exit_profile = None
from app.services.signal_categories import (
TRUMP_MIN_CONFIDENCE, TRUMP_COOLDOWN_HOURS,
)
# (a) Confidence floor — only the very top posts clear the bar.
if (post.ai_confidence or 0) < TRUMP_MIN_CONFIDENCE:
logger.info(
"Post %d skipped: Trump confidence %d < floor %d (low-freq mode)",
post.id, post.ai_confidence or 0, TRUMP_MIN_CONFIDENCE,
)
return
# (b) Cooldown — "don't trade every opportunity". If we opened ANY
# Trump trade in the last TRUMP_COOLDOWN_HOURS, skip this post.
# DB-backed so it survives restarts (same pattern as scanners).
from app.services.scanner_state import last_signal_at
from sqlalchemy import select as _sel, func as _fn
from app.models import Post as _Post, BotTrade as _BT
cutoff = datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(hours=TRUMP_COOLDOWN_HOURS)
recent_trump = await db.execute(
_sel(_fn.count(_BT.id))
.select_from(_BT)
.join(_Post, _Post.id == _BT.trigger_post_id)
.where(_Post.source == "truth", _BT.opened_at >= cutoff)
)
if (recent_trump.scalar() or 0) > 0:
logger.info(
"Post %d skipped: Trump cooldown active (a Trump trade opened "
"within the last %dh)", post.id, TRUMP_COOLDOWN_HOURS,
)
return
# (c) Regime filter — still applies; tuned for short-term behaviour.
from app.services.regime_filter import passes_regime_filter
regime_ok, regime_reasons = passes_regime_filter(asset, side)
for r in regime_reasons:
logger.info("Regime [%s]: %s", asset, r)
if not regime_ok:
logger.info("Post %d skipped: regime filter rejected (see ✗ lines above)", post.id)
return
result = await db.execute(select(Subscription).where(Subscription.active == True)) # noqa: E712
subscribers = result.scalars().all()
@@ -111,9 +205,80 @@ async def process_post(post: Post, db: AsyncSession) -> None:
daily_budget_usd=s.daily_budget_usd,
active_from=s.active_from,
active_until=s.active_until,
# Convex-strategy fields (default to legacy values if column null).
trailing_stop_pct=s.trailing_stop_pct,
trailing_activate_at_pct=s.trailing_activate_at_pct,
max_hold_hours=s.max_hold_hours or 168,
manual_window_until=s.manual_window_until,
# Phase 1 safety
paper_mode=bool(s.paper_mode),
circuit_breaker_tripped_at=s.circuit_breaker_tripped_at,
circuit_breaker_reason=s.circuit_breaker_reason,
# Two-system separation
sys2_budget_pct=(s.sys2_budget_pct if s.sys2_budget_pct is not None else 0.7),
sys2_cb_tripped_at=s.sys2_cb_tripped_at,
sys2_cb_reason=s.sys2_cb_reason,
sys2_leverage=s.sys2_leverage,
sys2_mode=s.sys2_mode,
auto_trade=bool(s.auto_trade),
)
for s in subscribers
]
# SYSTEM 2: override the user's Trump exit params with the category's
# exit profile. The user configures risk for their Trump scalp; the
# reversal system's risk is a property of the SIGNAL, not the user.
if sys2 and exit_profile is not None:
from app.services.signal_categories import (
sys2_effective_leverage, sys2_protective_stop_pct,
sys2_normalize_mode,
)
for snap in subs_snapshot:
mode = sys2_normalize_mode(snap.get("sys2_mode"))
snap["_sys2_mode"] = mode
# Dynamic System-2 leverage (independent of the Trump `leverage`);
# default depends on the risk mode (standard 2× / aggressive 8×).
lev = sys2_effective_leverage(snap.get("sys2_leverage"), mode)
# Protective stop auto-scaled INSIDE the liquidation line for THIS
# leverage → the position is de-risked by our monitor, never
# liquidated by the exchange.
prot = sys2_protective_stop_pct(lev)
snap["_is_system_2"] = True
snap["_category"] = post.category
snap["leverage"] = lev # HL opens at sys2 leverage
snap["take_profit_pct"] = None # pure trailing, no fixed TP
# Base catastrophic floor = leverage-aware protective stop. The
# upside ladder rungs (get_stop_ladder) still ratchet this UP as
# peak gain grows; they never loosen it.
snap["stop_loss_pct"] = prot
snap["trailing_activate_at_pct"] = exit_profile.trailing_activate_at_pct
snap["trailing_stop_pct"] = exit_profile.trailing_stop_pct
snap["max_hold_hours"] = exit_profile.max_hold_hours
# Carried through for the monitor (time-stop / invalidation).
snap["_sys2_time_stop_hours"] = exit_profile.time_stop_hours
snap["_sys2_invalidation"] = exit_profile.invalidation
snap["_sys2_invalidation_price"] = post.invalidation_price
logger.info(
"System-2 dynamic leverage: %dx → protective stop %.2f%% "
"(approx liquidation %.2f%%) for %s",
lev, prot, 100.0 / lev, snap["wallet"],
)
elif not sys2:
# SYSTEM 1 — Trump, repositioned. System-enforced now (not pure
# user params): tight SL always on, but TP/trailing suppressed for
# the first TRUMP_MIN_HOLD_MINUTES so we don't scalp out early.
# User still controls size / leverage / daily budget.
from app.services.signal_categories import (
TRUMP_STOP_LOSS_PCT, TRUMP_MAX_HOLD_HOURS, TRUMP_MIN_HOLD_MINUTES,
)
for snap in subs_snapshot:
snap["stop_loss_pct"] = TRUMP_STOP_LOSS_PCT
snap["max_hold_hours"] = TRUMP_MAX_HOLD_HOURS
snap["_min_hold_minutes"] = TRUMP_MIN_HOLD_MINUTES
# Keep user's take_profit_pct / trailing config — they still pick
# the profit target; we only gate WHEN it can fire.
post_id = post.id
post_confidence = post.ai_confidence or 0
@@ -135,30 +300,47 @@ async def _execute_for_subscriber(
if not sub["hl_api_key"]:
logger.warning("Subscriber %s has no HL API key, skipping", wallet)
return
# Required-setup guard: the bot is only allowed to trade when the user
# has explicitly set take-profit, stop-loss, and a daily budget. Prevents
# accidental trading on partially-configured accounts.
missing = [k for k in ("take_profit_pct", "stop_loss_pct", "daily_budget_usd")
if sub.get(k) is None]
# Required-setup guard. System 1 (Trump) needs the user's take-profit,
# stop-loss, and daily budget. System 2 (reversal) supplies its own
# stop-loss + (no) take-profit from the category profile — only the
# daily budget remains a user-required risk cap there.
if sub.get("_is_system_2"):
required = ("daily_budget_usd",)
else:
required = ("take_profit_pct", "stop_loss_pct", "daily_budget_usd")
missing = [k for k in required if sub.get(k) is None]
if missing:
logger.info("Sub %s skipped post %d: setup incomplete (missing %s)",
wallet, post_id, ", ".join(missing))
return
if post_confidence < sub["min_confidence"]:
confidence_floor = _confidence_floor_for(sub)
if post_confidence < confidence_floor:
logger.info("Sub %s filters out post %d: conf %d < user min %d",
wallet, post_id, post_confidence, sub["min_confidence"])
wallet, post_id, post_confidence, confidence_floor)
return
# Active-window check. Both values stored as naive-UTC.
af = sub.get("active_from")
au = sub.get("active_until")
if af is not None and au is not None:
now_utc_naive = datetime.now(timezone.utc).replace(tzinfo=None)
if now_utc_naive < af or now_utc_naive >= au:
logger.info("Sub %s skipped post %d: outside active window [%s, %s)",
wallet, post_id, af, au)
return
# ── Master Auto-Trade gate (D) ─────────────────────────────────────────
# The ONE user-facing switch. The signal has ALREADY been ingested as a
# Post by the ingest endpoint (so it shows in the feed regardless); here
# we only decide whether to OPEN a trade. OFF (default) = monitor-only.
# Replaces the old scanner-toggle / timed manual-window / schedule trio.
# NOTE: this gates NEW entries only — already-open positions keep their
# protective de-risk / stop (tp_sl_monitor runs independently of this).
if not sub.get("auto_trade"):
logger.info("Sub %s: Auto-Trade OFF — post %d recorded, no trade opened",
wallet, post_id)
return
# ── Circuit breaker gate (P1.1) ────────────────────────────────────────
# Checked BEFORE key decryption (cheap fast-fail). If tripped, the wallet
# has lost too much today or hit a losing streak — block until manual reset.
from app.services.circuit_breaker import is_tripped as _cb_tripped
_system = "sys2" if sub.get("_is_system_2") else "sys1"
tripped, cb_reason = _cb_tripped(sub, _system)
if tripped:
logger.info("Sub %s skipped post %d: %s", wallet, post_id, cb_reason)
return
try:
api_key = decrypt_api_key(sub["hl_api_key"])
@@ -166,49 +348,170 @@ async def _execute_for_subscriber(
logger.error("Cannot decrypt key for %s: %s", wallet, exc)
return
# ── Position sizing (C) ────────────────────────────────────────────────
# Score-based multiplier on the user's base size. Bigger bet when more
# confluences are in place. Computed BEFORE the lock so the value is
# available for both the budget check and the open call.
# System 2 has its OWN sizing — regime_filter's multiplier would shrink
# reversal bets (it penalises volatility expansion / prior movement,
# which are the SIGNAL for a reversal). See signal_categories.
if sub.get("_is_system_2"):
from app.services.signal_categories import system2_size_multiplier
size_mult = system2_size_multiplier(sub.get("_category"), post_confidence)
size_logic = f"sys2 cat={sub.get('_category')}"
else:
from app.services.regime_filter import calculate_size_multiplier
size_mult = calculate_size_multiplier(post_confidence, asset, side)
size_logic = "sys1 regime-scored"
sized_position_usd = round(sub["position_size_usd"] * size_mult, 2)
logger.info(
"Sub %s sizing [%s]: base=%.2f × %.2fx = %.2f (asset=%s, conf=%d)",
wallet, size_logic, sub["position_size_usd"], size_mult,
sized_position_usd, asset, post_confidence,
)
# Per-wallet lock wraps BOTH the budget re-check and the open, so two
# concurrent signals can't both pass the daily-budget gate before either
# trade is written to DB (TOCTOU race under asyncio.gather).
async with _wallet_lock(wallet):
# Re-check daily budget INSIDE the lock so it's atomic with the open.
daily_cap = sub.get("daily_budget_usd")
if daily_cap is not None and daily_cap > 0:
# Budget is SPLIT between the two systems so a Trump scalp can't eat
# the allocation reserved for a once-a-year reversal signal.
total_cap = sub.get("daily_budget_usd")
if total_cap is not None and total_cap > 0:
sys2_pct = sub.get("sys2_budget_pct", 0.7)
is_s2 = bool(sub.get("_is_system_2"))
daily_cap = total_cap * (sys2_pct if is_s2 else (1.0 - sys2_pct))
start_of_day = datetime.now(timezone.utc).replace(hour=0, minute=0, second=0, microsecond=0, tzinfo=None)
from app.models import Post as _P
from app.services.signal_categories import SYSTEM_2_SOURCES
async with AsyncSessionLocal() as budget_db:
spent_result = await budget_db.execute(
select(BotTrade).where(
select(BotTrade, _P.source)
.outerjoin(_P, _P.id == BotTrade.trigger_post_id)
.where(
BotTrade.wallet_address == wallet,
BotTrade.opened_at >= start_of_day,
)
)
spent_trades = spent_result.scalars().all()
spent = sum(
(t.size_usd if t.size_usd is not None else sub["position_size_usd"])
for t in spent_trades
# Only count spend from THIS system against THIS system's slice.
spent = 0.0
for t, src in spent_result.all():
t_is_s2 = (src or "").lower() in SYSTEM_2_SOURCES
if t_is_s2 != is_s2:
continue
spent += (t.size_usd if t.size_usd is not None else sub["position_size_usd"])
if spent + sized_position_usd > daily_cap:
logger.info("Sub %s [%s] daily budget reached: spent=%.2f + new=%.2f > cap=%.2f (%.0f%% of %.2f)",
wallet, _system, spent, sized_position_usd, daily_cap,
(sys2_pct if is_s2 else 1.0 - sys2_pct) * 100, total_cap)
return
# ── System-2 correlation / concentration cap ───────────────────────
# Inside the wallet lock so it's atomic with the open. The whole
# System-2 basket is correlated crypto-beta; cap CONCURRENT open
# positions + total open notional so a synchronised "crypto bottomed"
# week can't stack into one 10x leveraged macro bet.
if sub.get("_is_system_2"):
from app.services.signal_categories import (
SYS2_MAX_CONCURRENT, SYS2_MAX_OPEN_NOTIONAL_MULT, SYSTEM_2_SOURCES,
)
from app.models import Post as _P2
async with AsyncSessionLocal() as conc_db:
open_rows = await conc_db.execute(
select(BotTrade, _P2.source)
.outerjoin(_P2, _P2.id == BotTrade.trigger_post_id)
.where(
BotTrade.wallet_address == wallet,
BotTrade.closed_at.is_(None),
)
)
if spent + sub["position_size_usd"] > daily_cap:
logger.info("Sub %s daily budget reached: spent=%.2f + new=%.2f > cap=%.2f",
wallet, spent, sub["position_size_usd"], daily_cap)
open_sys2 = [
t for t, src in open_rows.all()
if (src or "").lower() in SYSTEM_2_SOURCES
]
open_count = len(open_sys2)
open_notional = sum(
(t.size_usd if t.size_usd is not None else sub["position_size_usd"])
for t in open_sys2
)
notional_ceiling = SYS2_MAX_OPEN_NOTIONAL_MULT * sub["position_size_usd"]
if open_count >= SYS2_MAX_CONCURRENT:
logger.info(
"Sub %s sys2 concentration cap: %d open positions ≥ max %d "
"(correlated crypto-beta — skipping post %d)",
wallet, open_count, SYS2_MAX_CONCURRENT, post_id)
return
if open_notional + sized_position_usd > notional_ceiling:
logger.info(
"Sub %s sys2 notional cap: open=%.2f + new=%.2f > ceiling=%.2f "
"(%.1f× base) — skipping post %d",
wallet, open_notional, sized_position_usd, notional_ceiling,
SYS2_MAX_OPEN_NOTIONAL_MULT, post_id)
return
# Each subscriber gets its own session — AsyncSession is NOT concurrency-safe.
async with AsyncSessionLocal() as db:
try:
trader = HyperliquidTrader(
api_private_key=api_key,
account_address=wallet,
leverage=sub["leverage"],
mainnet=settings.hl_mainnet,
)
# ── Paper-mode branch (P1.3) ──────────────────────────────
# Skip Hyperliquid entirely. Use the current Binance price as
# the synthetic fill. DB row gets hl_order_id="paper" so close
# logic + reconciliation know to skip HL too.
if sub["paper_mode"]:
from app.services.price_store import price_store
entry_price = price_store.latest_price(asset)
if not entry_price:
logger.warning("Paper mode: no price for %s, skipping", asset)
return
# Check DB (not HL) for already-open paper position on same asset.
open_dup = await db.execute(
select(BotTrade).where(
BotTrade.wallet_address == wallet,
BotTrade.asset == asset,
BotTrade.closed_at.is_(None),
)
)
if open_dup.scalar_one_or_none():
logger.info("Paper mode: %s already has open %s, skipping", wallet, asset)
return
result = {"fill_price": entry_price, "order_id": "paper"}
logger.info("[PAPER] Open %s %s for %s @ %.4f size=%.2f",
side, asset, wallet, entry_price, sized_position_usd)
else:
trader = HyperliquidTrader(
api_private_key=api_key,
account_address=wallet,
leverage=sub["leverage"],
mainnet=settings.hl_mainnet,
)
open_positions = await trader.get_open_positions()
if any(p.get('coin') == asset for p in open_positions):
logger.info("Subscriber %s already has open %s position, skipping", wallet, asset)
return
open_positions = await trader.get_open_positions()
if any(p.get('coin') == asset for p in open_positions):
logger.info("Subscriber %s already has open %s position, skipping", wallet, asset)
return
result = await trader.open_position(asset, side, sub["position_size_usd"])
result = await trader.open_position(asset, side, sized_position_usd)
entry_price = result.get('fill_price', 0.0)
# Resolve the FROZEN exit profile ONCE. Everything downstream
# (the watchdog AND recovery-after-restart) reads these — never
# the mutable Subscription/category config again.
import time as _time
eff_min_hold_ts = (
_time.time() + sub["_min_hold_minutes"] * 60
if sub.get("_min_hold_minutes") else None
)
eff = dict(
take_profit_pct = sub["take_profit_pct"],
stop_loss_pct = sub["stop_loss_pct"],
trailing_stop_pct = sub.get("trailing_stop_pct"),
trailing_activate_pct = sub.get("trailing_activate_at_pct"),
max_hold_hours = int(sub["max_hold_hours"]),
invalidation = sub.get("_sys2_invalidation"),
invalidation_price = sub.get("_sys2_invalidation_price"),
min_hold_until_ts = eff_min_hold_ts,
)
trade = BotTrade(
asset=asset,
side=side,
@@ -217,17 +520,40 @@ async def _execute_for_subscriber(
trigger_post_id=post_id,
opened_at=datetime.now(timezone.utc).replace(tzinfo=None),
hl_order_id=result.get('order_id'),
size_usd=sub["position_size_usd"],
size_usd=sized_position_usd,
base_size_usd=sized_position_usd, # immutable; pyramiding grows size_usd
sys2_mode=(_sys2_mode if sys2 else None),
leverage=sub["leverage"],
# Freeze the exit profile on the row.
eff_take_profit_pct = eff["take_profit_pct"],
eff_stop_loss_pct = eff["stop_loss_pct"],
eff_trailing_stop_pct = eff["trailing_stop_pct"],
eff_trailing_activate_pct = eff["trailing_activate_pct"],
eff_max_hold_hours = eff["max_hold_hours"],
eff_invalidation = eff["invalidation"],
eff_invalidation_price = eff["invalidation_price"],
eff_min_hold_until_ts = eff["min_hold_until_ts"],
)
db.add(trade)
await db.commit()
await db.refresh(trade)
logger.info("Opened %s %s for %s @ %.2f (trade_id=%d)",
side, asset, wallet, entry_price, trade.id)
logger.info("Opened %s %s for %s @ %.4f size=%.2f (trade_id=%d)",
side, asset, wallet, entry_price, sized_position_usd, trade.id)
# Register with the watchdog using the FROZEN profile.
from app.services.tp_sl_monitor import register_trade
_derisk = None
_addon = None
_peak_trail = None
_sys2_mode = sub.get("_sys2_mode", "standard")
if sys2:
from app.services.signal_categories import (
sys2_derisk_ladder, sys2_addon_ladder, sys2_peak_trail,
)
_derisk = sys2_derisk_ladder(sub["leverage"], _sys2_mode)
_addon = sys2_addon_ladder(_sys2_mode)
_peak_trail = sys2_peak_trail(_sys2_mode)
register_trade(
trade_id=trade.id,
wallet=wallet,
@@ -236,20 +562,235 @@ async def _execute_for_subscriber(
asset=asset,
side=side,
entry_price=entry_price,
take_profit_pct=sub["take_profit_pct"],
stop_loss_pct=sub["stop_loss_pct"],
take_profit_pct=eff["take_profit_pct"],
stop_loss_pct=eff["stop_loss_pct"],
trailing_stop_pct=eff["trailing_stop_pct"],
trailing_activate_at_pct=eff["trailing_activate_pct"],
invalidation=eff["invalidation"],
invalidation_price=eff.get("invalidation_price"),
min_hold_until_ts=eff["min_hold_until_ts"],
stop_ladder=get_stop_ladder(post.category) if sys2 else None,
derisk_ladder=_derisk,
derisk_done=0,
addon_ladder=_addon,
addon_done=0,
peak_trail=_peak_trail,
grow_mode=bool(getattr(trade, "grow_mode", False)),
)
# Max hold backstop (per-user for System 1, per-category for
# System 2 — already injected into the snapshot upstream).
max_hold_seconds = int(sub["max_hold_hours"]) * 3600
task = asyncio.create_task(_close_after_hold(
trade.id, api_key, sub["leverage"], asset, wallet
trade.id, api_key, sub["leverage"], asset, wallet, max_hold_seconds,
))
_background_tasks.add(task)
task.add_done_callback(_background_tasks.discard)
# System-2 time-stop: if the position is still ~flat after
# `time_stop_hours`, the thesis is slow/dead — close early
# instead of tying up capital for the full max-hold.
ts_hours = sub.get("_sys2_time_stop_hours")
if ts_hours:
ts_task = asyncio.create_task(_time_stop_check(
trade.id, api_key, sub["leverage"], asset, wallet,
int(ts_hours) * 3600,
))
_background_tasks.add(ts_task)
ts_task.add_done_callback(_background_tasks.discard)
except Exception as e:
logger.error("Trade execution failed for %s: %s", wallet, e)
async def partial_derisk(
trade_id: int,
api_key: str,
asset: str,
wallet: str,
step_idx: int,
frac_of_original: float,
reason: str = "derisk",
) -> bool:
"""Staged de-risk: partially close a System-2 trade, realise the slice's
PnL, and KEEP the trade open. Idempotent per step_idx (guarded by the
persisted `derisk_steps_done` counter under the per-trade lock).
Returns True if the step was executed (or already done), False on a
recoverable no-op so the monitor can retry next tick.
"""
async with _lock_for(trade_id):
async with AsyncSessionLocal() as db:
try:
row = await db.execute(select(BotTrade).where(BotTrade.id == trade_id))
trade = row.scalar_one_or_none()
if trade is None or trade.closed_at is not None:
return True # gone / already fully closed — nothing to do
if (trade.derisk_steps_done or 0) > step_idx:
return True # this step already executed (idempotent)
if (trade.derisk_steps_done or 0) != step_idx:
# Steps must run in order; let the monitor catch up.
return False
size_usd = trade.size_usd if trade.size_usd is not None else 0.0
rem_frac = trade.remaining_fraction if trade.remaining_fraction is not None else 1.0
if size_usd <= 0 or rem_frac <= 0:
return True
# frac of the CURRENT open position needed to shed
# `frac_of_original` of the ORIGINAL notional.
frac_of_current = max(0.0, min(1.0, frac_of_original / rem_frac))
if trade.hl_order_id == "paper":
from app.services.price_store import price_store
fill = price_store.latest_price(asset)
if not fill:
logger.error("Paper de-risk: no %s price, retry trade %d", asset, trade_id)
return False
closed_frac_of_current = frac_of_current
else:
trader = HyperliquidTrader(
api_private_key=api_key,
account_address=wallet,
leverage=(trade.leverage or 1),
mainnet=settings.hl_mainnet,
)
r = await trader.reduce_position(asset, frac_of_current)
if r.get("already_closed"):
# Position vanished (manual close / liquidation race).
# Let the reconciler / full-close path handle it.
logger.warning("De-risk: %s position gone for trade %d", asset, trade_id)
return True
fill = r.get("fill_price")
closed_frac_of_current = r.get("closed_fraction") or 0.0
if not fill or closed_frac_of_current <= 0:
return False # no fill — retry next tick
# Fraction of ORIGINAL notional actually shed this step.
closed_frac_of_original = closed_frac_of_current * rem_frac
slice_usd = size_usd * closed_frac_of_original
pct = ((fill - trade.entry_price) / trade.entry_price) if trade.entry_price else 0.0
signed_pct = pct if trade.side == "long" else -pct
# Round-trip taker fee on this slice (open-share + this close).
fees = slice_usd * HL_TAKER_FEE_RATE * 2
slice_pnl = slice_usd * signed_pct - fees
trade.realized_partial_pnl_usd = round((trade.realized_partial_pnl_usd or 0.0) + slice_pnl, 4)
trade.remaining_fraction = max(0.0, round(rem_frac - closed_frac_of_original, 6))
trade.derisk_steps_done = step_idx + 1
await db.commit()
logger.warning(
"De-risk step %d trade %d (%s): closed %.1f%% of original "
"@ %.2f, slice PnL %.2f, remaining %.1f%% (reason=%s)",
step_idx + 1, trade_id, asset, closed_frac_of_original * 100,
fill, slice_pnl, trade.remaining_fraction * 100, reason,
)
return True
except Exception as e:
logger.error("partial_derisk failed for trade %d step %d: %s",
trade_id, step_idx, e)
return False
async def pyramid_add(
trade_id: int,
api_key: str,
asset: str,
wallet: str,
step_idx: int,
frac_of_base: float,
reason: str = "pyramid",
) -> tuple:
"""Pyramiding: add to a CONFIRMED System-2 winner. Returns
(ok: bool, new_entry: float|None, ref_price: float|None).
Guards: only while derisk_steps_done==0 (clean uptrend), idempotent per
step_idx, structural confirmation re-checked at execution, per-trade
notional cap. On success blends the average entry and grows size_usd.
"""
async with _lock_for(trade_id):
async with AsyncSessionLocal() as db:
try:
row = await db.execute(select(BotTrade).where(BotTrade.id == trade_id))
trade = row.scalar_one_or_none()
if trade is None or trade.closed_at is not None:
return (True, None, None)
if (trade.derisk_steps_done or 0) > 0:
return (True, None, None) # went underwater — no pyramiding
if (trade.addon_steps_done or 0) > step_idx:
return (True, None, None) # already done (idempotent)
if (trade.addon_steps_done or 0) != step_idx:
return (False, None, None) # out of order — retry later
base = trade.base_size_usd or trade.size_usd or 0.0
if base <= 0:
return (True, None, None)
from app.services.signal_categories import SYS2_MAX_OPEN_NOTIONAL_MULT
add_usd = round(base * frac_of_base, 2)
cur_notional = trade.size_usd or base
if cur_notional + add_usd > base * SYS2_MAX_OPEN_NOTIONAL_MULT:
logger.warning("Pyramid: notional cap hit trade %d (%.2f+%.2f > %.1fx base)",
trade_id, cur_notional, add_usd, SYS2_MAX_OPEN_NOTIONAL_MULT)
trade.addon_steps_done = step_idx + 1 # stop retrying
await db.commit()
return (True, None, None)
# Structural confirmation — fetched ONCE here, not per tick.
from app.services.market_data import for_asset, drop_in_progress_bar
from app.services.bottom_indicators import trend_confirmed
from app.services.price_store import price_store
prov = for_asset(asset)
raw = await prov.fetch_1d(asset, days=260)
daily = drop_in_progress_bar(raw, "1d")
closes = [float(c["close"]) for c in daily if c.get("close")]
highs = [float(c["high"]) for c in daily if c.get("high")]
px = price_store.latest_price(asset)
if not px:
return (False, None, None)
if not trend_confirmed(closes, highs, px):
return (False, None, None) # not a confirmed uptrend yet
if trade.hl_order_id == "paper":
fill = px
actual_add_usd = add_usd # synthetic full fill
else:
trader = HyperliquidTrader(
api_private_key=api_key,
account_address=wallet,
leverage=(trade.leverage or 1),
mainnet=settings.hl_mainnet,
)
r = await trader.open_position(asset, trade.side, add_usd)
fill = r.get("fill_price")
filled_coins = float(r.get("size_coins") or 0.0)
if not fill or filled_coins <= 0:
return (False, None, None)
# Use the ACTUAL filled notional, not the intended amount —
# an IOC add can under-fill on thin/volatile books; assuming
# the full add_usd would corrupt the blended entry + size.
actual_add_usd = filled_coins * fill
old_notional = trade.size_usd or base
new_notional = old_notional + actual_add_usd
blended = (old_notional * trade.entry_price + actual_add_usd * fill) / new_notional
trade.entry_price = round(blended, 6)
trade.size_usd = round(new_notional, 2)
trade.addon_steps_done = step_idx + 1
await db.commit()
logger.warning(
"Pyramid add step %d trade %d (%s): +$%.2f filled @ %.2f → "
"notional $%.2f, blended entry %.4f (reason=%s)",
step_idx + 1, trade_id, asset, actual_add_usd, fill,
new_notional, blended, reason,
)
return (True, round(blended, 6), float(fill))
except Exception as e:
logger.error("pyramid_add failed trade %d step %d: %s",
trade_id, step_idx, e)
return (False, None, None)
async def close_and_finalize(
trade_id: int,
api_key: str,
@@ -297,23 +838,46 @@ async def close_and_finalize(
size_usd = sub.position_size_usd if sub else 20.0
trade_leverage = trade.leverage if trade.leverage is not None else leverage
trader = HyperliquidTrader(
api_private_key=api_key,
account_address=wallet,
leverage=trade_leverage,
mainnet=settings.hl_mainnet,
)
close_result = await trader.close_position(asset)
# ── Paper-mode close (P1.3) ──────────────────────────────
# No Hyperliquid call. Synthesize a fill at the current Binance
# price. Same downstream PnL math as a real trade.
if trade.hl_order_id == "paper":
from app.services.price_store import price_store
paper_exit = price_store.latest_price(asset)
if not paper_exit:
logger.error("Paper close: no price for %s, can't close trade %d",
asset, trade_id)
# Roll back the closed_at claim so we can retry later.
await db.execute(
update(BotTrade).where(BotTrade.id == trade_id).values(closed_at=None)
)
await db.commit()
return
close_result = {"fill_price": paper_exit, "already_closed": False}
else:
trader = HyperliquidTrader(
api_private_key=api_key,
account_address=wallet,
leverage=trade_leverage,
mainnet=settings.hl_mainnet,
)
close_result = await trader.close_position(asset)
# Detect "position was already closed externally" before we even tried
if close_result.get("already_closed"):
# PnL of the (now-gone) open remainder is unknown, BUT any
# PnL already BANKED by staged de-risk is real money — keep
# it in pnl_usd instead of discarding it as None, else
# analytics / "today realised" silently undercount.
banked = trade.realized_partial_pnl_usd or 0.0
logger.warning(
"Trade %d: no open %s position found on HL — "
"user likely closed it manually. Marking trade closed with no PnL.",
trade_id, asset,
"Trade %d: no open %s position found on HL — user likely "
"closed it manually. Marking closed; pnl=%s (banked de-risk only).",
trade_id, asset, round(banked, 2) if banked else None,
)
trade.exit_price = None
trade.pnl_usd = None
trade.pnl_usd = round(banked, 2) if banked else None
trade.remaining_fraction = 0.0
trade.hold_seconds = int(
(datetime.now(timezone.utc) -
trade.opened_at.replace(tzinfo=timezone.utc)).total_seconds()
@@ -334,13 +898,24 @@ async def close_and_finalize(
pct = ((exit_price - trade.entry_price) / trade.entry_price) if trade.entry_price else 0.0
signed_pct = pct if trade.side == 'long' else -pct
# size_usd is the NOTIONAL value — leverage affects margin, not PnL.
gross_pnl = size_usd * signed_pct
# Deduct round-trip taker fees (IOC crosses book on both open + close).
# Without this, displayed PnL overstates real returns by ~9 bps per trade.
fees_usd = size_usd * HL_TAKER_FEE_RATE * 2
pnl_usd = gross_pnl - fees_usd
# For a staged-de-risk trade only the REMAINING notional is
# closed here; earlier partial reduces already realised their
# slice into realized_partial_pnl_usd. Legacy/non-partial rows
# have remaining_fraction=1.0 + realized_partial=0.0 → identical
# math to before.
rem_frac = trade.remaining_fraction if trade.remaining_fraction is not None else 1.0
prior_pnl = trade.realized_partial_pnl_usd or 0.0
remaining_size = size_usd * rem_frac
gross_pnl = remaining_size * signed_pct
# Round-trip taker fees: the OPEN fee was paid on the full
# original notional; partial reduces already charged their
# close-side fee. Here charge the open-share for the remaining
# slice + this final close fee.
fees_usd = remaining_size * HL_TAKER_FEE_RATE * 2
pnl_usd = gross_pnl - fees_usd + prior_pnl
trade.exit_price = exit_price
trade.remaining_fraction = 0.0
trade.pnl_usd = round(pnl_usd, 2)
trade.hold_seconds = hold_secs
# closed_at already set by the atomic UPDATE
@@ -357,6 +932,24 @@ async def close_and_finalize(
gross_pnl, fees_usd, pnl_usd, reason,
)
# ── Circuit breaker check (P1.1) ───────────────────────────
# Run after PnL is written. If daily DD or consecutive-loss
# threshold hit, trip the breaker — that nulls manual_window
# and blocks new trades for CB_LOCKOUT_HOURS.
# Infer which system this trade belonged to from its trigger
# post's source, so the right (independent) breaker is checked.
from app.services.circuit_breaker import check_and_trip
from app.services.signal_categories import SYSTEM_2_SOURCES
_src_row = await db.execute(
select(Post.source).where(Post.id == trade.trigger_post_id)
)
_src = (_src_row.scalar_one_or_none() or "").lower()
_sys = "sys2" if _src in SYSTEM_2_SOURCES else "sys1"
cb_reason = await check_and_trip(wallet, db, _sys)
if cb_reason:
logger.warning("Circuit breaker [%s] tripped for %s: %s",
_sys, wallet, cb_reason)
except Exception as e:
logger.error("Failed to close trade %d: %s", trade_id, e)
# Rollback closed_at so recovery can retry this trade on next restart.
@@ -377,7 +970,58 @@ async def close_and_finalize(
async def _close_after_hold(
trade_id: int, api_key: str, leverage: int, asset: str, wallet: str
trade_id: int, api_key: str, leverage: int, asset: str, wallet: str,
max_hold_seconds: int,
) -> None:
await asyncio.sleep(MAX_HOLD_SECONDS)
"""Per-user max-hold backstop. Forces close after the configured duration.
The trailing-stop monitor will usually close the position long before this
fires, but for trades that drift sideways the cap prevents indefinite
funding-rate bleed.
"""
await asyncio.sleep(max_hold_seconds)
await close_and_finalize(trade_id, api_key, leverage, asset, wallet, reason="max_hold")
# Flat-zone band: a position whose |unrealised| is within this % at the
# time-stop checkpoint is considered "not working" and closed early.
_TIME_STOP_FLAT_BAND_PCT = 2.0
async def _time_stop_check(
trade_id: int, api_key: str, leverage: int, asset: str, wallet: str,
delay_seconds: int,
) -> None:
"""System-2 time-stop. After `delay_seconds`, if the trade is still open
AND roughly flat (|unrealised| < band), the thesis is slow/dead — close
it so capital isn't parked for the full multi-week max-hold on a dud.
A trade that's already moved (win or stopped) will be closed/gone by
now; the conditional UPDATE in close_and_finalize makes a late fire a
harmless no-op.
"""
await asyncio.sleep(delay_seconds)
from sqlalchemy import select
async with AsyncSessionLocal() as db:
row = await db.execute(select(BotTrade).where(BotTrade.id == trade_id))
trade = row.scalar_one_or_none()
if trade is None or trade.closed_at is not None:
return # already closed by trail / SL / invalidation
entry = trade.entry_price
from app.services.price_store import price_store
cur = price_store.latest_price(asset)
if not cur or not entry:
return # no price → don't act blindly; max-hold will catch it
raw = (cur - entry) / entry
signed_pct = (raw if trade.side == "long" else -raw) * 100
if abs(signed_pct) < _TIME_STOP_FLAT_BAND_PCT:
logger.info(
"Time-stop firing trade %d (%s %s): flat at %.2f%% after %dh",
trade_id, trade.side, asset, signed_pct, delay_seconds // 3600,
)
await close_and_finalize(
trade_id, api_key, leverage, asset, wallet, reason="time_stop",
)
app/services/bottom_indicators.py
+190
View File
@@ -0,0 +1,190 @@
"""Pure-price BTC bottom indicators (no on-chain data, no API keys).
Three independent, well-known bottom signals. Each is a simple, transparent
function of price history only — no Realized Cap, no MVRV, no paid feeds:
A. AHR999 — value/DCA index. < 0.45 = deep-value bottom zone.
B. 200-Week MA — BTC has bottomed near its 200WMA every cycle.
C. Pi Cycle Bot — 150d EMA vs 471d SMA × 0.745 crossover region.
The combiner fires only when at least 2 of the 3 agree ("三者为二"), which
historically pins genuine macro bottoms while rejecting single-indicator
noise. Long-only, low-frequency, stateless.
"""
from __future__ import annotations
import math
from dataclasses import dataclass
from datetime import date, datetime, timezone
# BTC genesis block: 2009-01-03.
_BTC_GENESIS = date(2009, 1, 3)
# AHR999 thresholds.
AHR999_BOTTOM = 0.45 # < this → deep-value / bottom zone (signal A)
AHR999_INVALIDATION = 1.2 # > this → no longer cheap, thesis dead
# 200WMA tolerance: price within +5% of the 200-week mean still counts.
WMA200_TOLERANCE = 1.05
# Pi Cycle Bottom multiplier (the canonical 0.745).
PI_CYCLE_MULT = 0.745
PI_CYCLE_EMA = 150
PI_CYCLE_SMA = 471
def _closes(candles: list[dict]) -> list[float]:
return [float(c["close"]) for c in candles if c.get("close")]
def _sma(values: list[float], n: int) -> float | None:
if len(values) < n:
return None
return sum(values[-n:]) / n
def _ema(values: list[float], n: int) -> float | None:
if len(values) < n:
return None
k = 2.0 / (n + 1)
# Seed with the SMA of the first n values, then walk forward.
ema = sum(values[:n]) / n
for v in values[n:]:
ema = v * k + ema * (1 - k)
return ema
def coin_age_days(today: date | None = None) -> int:
d = today or datetime.now(timezone.utc).date()
return max(1, (d - _BTC_GENESIS).days)
def ahr999(daily_closes: list[float], today: date | None = None) -> float | None:
"""AHR999 = (price / GM200) × (price / growth_val).
GM200 = geometric mean of the last 200 daily closes
growth_val = 10 ** (5.84 * log10(coin_age_days) - 17.01)
< 0.45 deep value · 0.45–1.2 DCA · > 1.2 expensive.
"""
if len(daily_closes) < 200:
return None
price = daily_closes[-1]
if price <= 0:
return None
window = daily_closes[-200:]
# Geometric mean via log-space (avoids overflow).
log_sum = sum(math.log(c) for c in window if c > 0)
gm200 = math.exp(log_sum / 200)
age = coin_age_days(today)
growth_val = 10 ** (5.84 * math.log10(age) - 17.01)
if gm200 <= 0 or growth_val <= 0:
return None
return (price / gm200) * (price / growth_val)
def below_200wma(
weekly_closes: list[float],
price: float | None = None,
) -> tuple[bool, float | None]:
"""True if price ≤ 200-week mean × 1.05. Returns (signal, wma200).
`price` is the reference price to compare. Pass the latest DAILY close so
all three indicators use the same "current price" — otherwise the last
weekly close can be up to 7 days stale and B would disagree with A/C right
at a fast-moving bottom. Falls back to the last weekly close if omitted.
"""
wma = _sma(weekly_closes, 200)
if wma is None or not weekly_closes:
return False, wma
px = price if price is not None else weekly_closes[-1]
return px <= wma * WMA200_TOLERANCE, wma
def pi_cycle_bottom(daily_closes: list[float]) -> tuple[bool, float | None, float | None]:
"""Pi Cycle Bottom: 150d EMA ≤ 471d SMA × 0.745.
Returns (signal, ema150, sma471_scaled).
"""
ema150 = _ema(daily_closes, PI_CYCLE_EMA)
sma471 = _sma(daily_closes, PI_CYCLE_SMA)
if ema150 is None or sma471 is None:
return False, ema150, None
scaled = sma471 * PI_CYCLE_MULT
return ema150 <= scaled, ema150, scaled
def trend_confirmed(
daily_closes: list[float],
daily_highs: list[float],
price: float,
sma_n: int = 200,
high_lookback: int = 20,
high_tol: float = 0.005,
) -> bool:
"""Structural confirmation that the bottom reversal has turned into an
actual uptrend — gate for pyramiding (add-to-winner):
price ≥ 200-day SMA (trend regime reclaimed)
AND price ≥ recent 20d high·(1−0.5%) (at/near a fresh local high)
Pure function (no I/O) so it's unit-testable; the caller fetches candles.
Conservative: both conditions must hold, evaluated at add-on time.
"""
sma = _sma(daily_closes, sma_n)
if sma is None or not daily_highs:
return False
recent_high = max(daily_highs[-high_lookback:]) if len(daily_highs) >= 1 else None
if recent_high is None or recent_high <= 0:
return False
return price >= sma and price >= recent_high * (1.0 - high_tol)
@dataclass(frozen=True)
class BottomConfluence:
fired: bool # ≥ 2 of 3 true
votes: int # 0..3
ahr999: float | None
a_value: bool # AHR999 < 0.45
b_200wma: bool # price ≤ 200WMA × 1.05
c_pi_cycle: bool # 150 EMA ≤ 471 SMA × 0.745
detail: dict
def bottom_confluence(
daily_closes: list[float],
weekly_closes: list[float],
today: date | None = None,
) -> BottomConfluence:
"""2-of-3 bottom confluence. Pure price, stateless."""
ah = ahr999(daily_closes, today)
a = ah is not None and ah < AHR999_BOTTOM
# All three indicators compare against the SAME current price (latest
# daily close), so a stale weekly close can't make B disagree with A/C.
cur_price = daily_closes[-1] if daily_closes else None
b, wma200 = below_200wma(weekly_closes, cur_price)
c, ema150, sma471s = pi_cycle_bottom(daily_closes)
votes = int(a) + int(b) + int(c)
detail = {
"ahr999": round(ah, 4) if ah is not None else None,
"ahr999_threshold": AHR999_BOTTOM,
"price": round(daily_closes[-1], 2) if daily_closes else None,
"wma200": round(wma200, 2) if wma200 is not None else None,
"wma200_band": round(wma200 * WMA200_TOLERANCE, 2) if wma200 is not None else None,
"pi_ema150": round(ema150, 2) if ema150 is not None else None,
"pi_sma471_scaled": round(sma471s, 2) if sma471s is not None else None,
"votes": votes,
"signals": {"ahr999_value": a, "below_200wma": b, "pi_cycle_bottom": c},
}
return BottomConfluence(
fired=votes >= 2,
votes=votes,
ahr999=ah,
a_value=a,
b_200wma=b,
c_pi_cycle=c,
detail=detail,
)
app/services/circuit_breaker.py
+201
View File
@@ -0,0 +1,201 @@
"""
Circuit breaker — autonomous "stop trading" trigger per wallet.
Two trip conditions, evaluated AFTER every closed trade:
1. Daily drawdown: cumulative PnL across all trades closed today (UTC)
drops below -CB_DAILY_DD_PCT × (subscription.position_size_usd × 10).
Default cap: -5% of "expected 10-trade notional" — i.e. if the user's
base bet is $20, the daily DD limit is -$10 net realized.
2. Consecutive losses: last CB_CONSEC_LOSSES closed trades for this wallet
all have pnl_usd < 0. Default = 3.
When tripped:
- manual_window_until is nulled (bot stops trading via manual override)
- circuit_breaker_tripped_at = now
- circuit_breaker_reason = "daily_dd" | "consecutive_losses"
- WS broadcast a 'circuit_breaker_tripped' event so the UI lights up red
Gate: `is_tripped()` is checked by _execute_for_subscriber BEFORE opening any
new trade. A tripped wallet stays blocked for CB_LOCKOUT_HOURS regardless of
schedule, so a buggy scanner or a black-swan move can't drain the account
overnight.
User unblock path:
POST /api/user/{wallet}/manual-window with any hours > 0 clears the trip.
This is a deliberate human-in-the-loop — the user must consciously re-arm.
"""
from __future__ import annotations
import logging
from datetime import datetime, timedelta, timezone
from typing import Optional
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.models import BotTrade, Subscription
logger = logging.getLogger(__name__)
# ─── Tunables ───────────────────────────────────────────────────────────────
CB_DAILY_DD_USD_PER_BASE = 0.5 # Daily DD limit as a multiple of base size.
# base $20 → DD limit -$10. Tune to taste.
CB_CONSEC_LOSSES = 3 # Consecutive losing trades that trip the CB.
CB_LOCKOUT_HOURS = 24 # Once tripped, blocked for this many hours
# unless the user explicitly clears it.
# ─── Trip detection ─────────────────────────────────────────────────────────
# Column indirection so one code path serves both independent breakers.
# system "sys1" → circuit_breaker_tripped_at / circuit_breaker_reason
# system "sys2" → sys2_cb_tripped_at / sys2_cb_reason
_CB_COLS = {
"sys1": ("circuit_breaker_tripped_at", "circuit_breaker_reason"),
"sys2": ("sys2_cb_tripped_at", "sys2_cb_reason"),
}
async def _trades_for_system(db: AsyncSession, wallet: str, since, system: str):
"""Closed, priced trades for `wallet` since `since`, filtered to the
given system by joining the trigger post's source.
sys2 = trigger_post.source in System-2 set; sys1 = everything else
(currently only "truth"). Trades with no trigger post → treated sys1.
"""
from app.models import Post
from app.services.signal_categories import SYSTEM_2_SOURCES
rows = await db.execute(
select(BotTrade, Post.source)
.outerjoin(Post, Post.id == BotTrade.trigger_post_id)
.where(
BotTrade.wallet_address == wallet,
BotTrade.closed_at >= since,
BotTrade.pnl_usd.is_not(None),
)
.order_by(BotTrade.closed_at.desc())
)
out = []
for trade, src in rows.all():
is_s2 = (src or "").lower() in SYSTEM_2_SOURCES
if (system == "sys2") == is_s2:
out.append(trade)
return out
async def check_and_trip(
wallet: str, db: AsyncSession, system: str = "sys1",
) -> Optional[str]:
"""Run trip conditions for `wallet` within ONE system. Independent per
system: a Trump losing streak can't lock out the reversal book.
Called from close_and_finalize after a trade's pnl is written, with the
system inferred from the closing trade's source.
"""
col_at, col_reason = _CB_COLS[system]
sub = (await db.execute(
select(Subscription).where(Subscription.wallet_address == wallet)
)).scalar_one_or_none()
if sub is None:
return None
tripped_at = getattr(sub, col_at)
if tripped_at is not None:
age_hrs = (datetime.now(timezone.utc).replace(tzinfo=None) - tripped_at).total_seconds() / 3600
if age_hrs < CB_LOCKOUT_HOURS:
return None # still in active lockout
reason: Optional[str] = None
now_naive = datetime.now(timezone.utc).replace(tzinfo=None)
start_of_day = now_naive.replace(hour=0, minute=0, second=0, microsecond=0)
today_trades = await _trades_for_system(db, wallet, start_of_day, system)
today_pnl = sum(t.pnl_usd or 0 for t in today_trades)
dd_limit = -CB_DAILY_DD_USD_PER_BASE * sub.position_size_usd * 10
if today_pnl < dd_limit:
reason = "daily_dd"
logger.warning("CB[%s] trip [daily_dd] %s: pnl=%.2f < %.2f",
system, wallet, today_pnl, dd_limit)
if reason is None:
# Consecutive losses within this system (all-time, most recent N).
all_sys = await _trades_for_system(
db, wallet, datetime(1970, 1, 1), system,
)
recent = all_sys[:CB_CONSEC_LOSSES]
if len(recent) >= CB_CONSEC_LOSSES and all((t.pnl_usd or 0) < 0 for t in recent):
reason = "consecutive_losses"
logger.warning("CB[%s] trip [consecutive_losses] %s: last %d all negative",
system, wallet, CB_CONSEC_LOSSES)
if reason is None:
return None
setattr(sub, col_at, now_naive)
setattr(sub, col_reason, reason)
# Only System 1 owns manual_window; nulling it on a sys2 trip would
# wrongly disable the Trump book too. Sys2 trip just blocks sys2 entries.
if system == "sys1":
sub.manual_window_until = None
await db.commit()
try:
from app.ws.manager import manager
await manager.broadcast({
"type": "circuit_breaker_tripped",
"wallet": wallet,
"system": system,
"reason": reason,
"tripped_at": now_naive.isoformat(),
"unlock_at": (now_naive + timedelta(hours=CB_LOCKOUT_HOURS)).isoformat(),
})
except Exception as exc:
logger.warning("CB broadcast failed: %s", exc)
return reason
# ─── Gate (called by _execute_for_subscriber) ───────────────────────────────
def is_tripped(sub_dict: dict, system: str = "sys1") -> tuple[bool, str]:
"""Returns (tripped, reason) for the given system's breaker.
`sub_dict` is the snapshot copy built in bot_engine."""
col_at, col_reason = _CB_COLS[system]
tripped_at = sub_dict.get(col_at)
if tripped_at is None:
return False, ""
age_hrs = (datetime.now(timezone.utc).replace(tzinfo=None) - tripped_at).total_seconds() / 3600
if age_hrs >= CB_LOCKOUT_HOURS:
return False, "" # lockout expired (auto-untrip on next refresh)
reason = sub_dict.get(col_reason) or "unknown"
remaining_hrs = CB_LOCKOUT_HOURS - age_hrs
return True, f"cb[{system}]:{reason} (unlock in {remaining_hrs:.1f}h)"
# ─── Manual reset (called from /manual-window endpoint) ─────────────────────
async def clear_trip(wallet: str, db: AsyncSession) -> bool:
"""Clear the trip state. Returns True if a trip was actually cleared.
Called when the user explicitly re-arms manual_window — that's the
human-in-the-loop unblock. Just letting LOCKOUT_HOURS expire works too,
but most users will hit the button.
"""
sub = (await db.execute(
select(Subscription).where(Subscription.wallet_address == wallet)
)).scalar_one_or_none()
if sub is None or sub.circuit_breaker_tripped_at is None:
return False
sub.circuit_breaker_tripped_at = None
sub.circuit_breaker_reason = None
await db.commit()
logger.info("CB cleared for %s", wallet)
return True
app/services/entry_filter.py
+347
View File
@@ -0,0 +1,347 @@
"""
Deterministic entry pre-filter — runs BEFORE the AI call.
Lesson from the 13-trade backtest: AI gives confidence 85+ to posts like
"I had excellent conversations with President X" and "The Strait is open again",
which are second-derivative news or pure rhetoric — they don't move markets.
The two posts that ACTUALLY produced 10%+ moves both contained explicit
"ACTION TAKEN" language: "STATEMENT OF PRESIDENT", "Military Success that we
have had". The losers all had future-tense or conversational language.
Rather than hoping the LLM internalises this distinction reliably, we enforce
it with a deterministic Python filter. Cheap, debuggable, no false positives
from a hallucinating model.
This filter runs BEFORE analyze_post() in the scraper, short-circuiting the
AI call when the text is obviously non-actionable. Saves API spend AND keeps
the bot from accumulating "confidence 85, no catalyst" trades.
Three independent gates (post must pass ALL):
1. ACTION verbs present — past-tense action OR formal statement marker
2. NO future-tense disqualifiers — "I will", "considering", "thinking about"
3. NOT a duplicate of a recent post on the same topic
"""
from __future__ import annotations
import logging
import re
from datetime import datetime, timedelta, timezone
from typing import Optional
logger = logging.getLogger(__name__)
# ─── Gate 1: explicit action markers ────────────────────────────────────────
# Lower-cased substring matches. A post must contain AT LEAST ONE of these.
ACTION_MARKERS = [
# Formal statement / announcement (past or imperative)
"statement of",
"statement by",
"executive order",
"announcing",
"today i signed",
"i have signed",
"i hereby",
"effective immediately",
"effective today",
# Past-tense action by the administration / military
"have agreed",
"has agreed",
"was signed",
"have signed",
"have implemented",
"have imposed",
"have authorized",
"we have had", # "Military Success that we have had..." (post 684)
"have successfully",
"just imposed",
"just signed",
# NOTE: "i have ordered" was in this list but produced false positives.
# Trump uses it for verbal directives ("ordered Navy to shoot boats")
# that don't move markets. Real catalysts use "I have signed" or "STATEMENT OF".
# If you need to catch a genuine order, list the specific action below:
"ordered the suspension of",
"ordered the cancellation of",
"ordered the deployment of",
"just announced",
# Asset / policy specific catalysts
"strategic reserve",
"tariff",
"sanctions on",
"ceasefire signed",
"rate cut",
"rate hike",
# Regulatory / legal
"sec approval",
"sec has approved",
"supreme court ruled",
"doj filed",
]
# ─── Gate 2: future-tense / hedge disqualifiers ─────────────────────────────
# If ANY of these appear in the first 200 chars, the post is intent not action.
FUTURE_TENSE_BLOCKERS = [
# Pure intent
"i will",
"we will",
"i'm thinking",
"considering",
"looking at",
"may consider",
"could be",
# Conversational (not action)
"had a conversation",
"had conversations",
"had a great call",
"had a great meeting",
"spoke with",
"talked with",
"met with",
"i just had excellent",
# Hedge / partial
"most points were agreed",
"we're close to",
"still working on",
]
# ─── Gate 3: deduplication ──────────────────────────────────────────────────
# Two layers:
#
# Layer A: TOPIC KEYWORDS — curated set of geopolitical / macro nouns. If two
# posts within DEDUP_WINDOW_HOURS share ≥1 topic keyword, they're flagged as
# being about the same event. This catches "Iran/Strait/Hormuz" being repeated
# 4 times across 5 days, which the original token-overlap dedup missed because
# the words ARE different (closed/opened/announced/transited) — only the
# topic is the same.
#
# Layer B (legacy): generic token-overlap fallback for posts that don't hit a
# curated topic. Kept as backup so we don't miss novel re-iterations.
DEDUP_WINDOW_HOURS = 48
DEDUP_KEYWORD_THRESHOLD = 3 # token-overlap backup threshold
# Curated list of "topic anchors". Lower-cased substring match. When a new
# post AND a prior post both contain one of these, we treat the new post as
# a probable update to an existing storyline, not a fresh catalyst.
#
# Tune this list against your backtest results: anything that produced repeat
# false-positives should be on here. Anything that produced a real catalyst
# (post 684 = "Pakistan" — yes, even though we list it, Pakistan being mentioned
# again within 48h would correctly be flagged as redundant) should NOT cause us
# to miss the FIRST occurrence — only subsequent ones.
TOPIC_KEYWORDS = {
# Middle East geopolitics (dominant source of FP in our 13-trade audit)
"iran", "iranian",
"hormuz", "strait of hormuz",
"strait of iran", # Trump-ism on post 62 — same physical geography as Hormuz
"the strait", # generic — catches "the Strait is open/closed" phrasing
"israel", "gaza", "lebanon", "hezbollah", "syria",
"yemen", "houthi",
# Russia / Ukraine
"ukraine", "russia", "putin", "zelensky", "kyiv",
# China
"china", "chinese", "xi jinping", "taiwan",
# North Korea
"north korea", "kim jong",
# Macro / monetary
"tariff", "tariffs",
"fed", "fomc", "powell", "interest rate", "rate cut", "rate hike",
"cpi", "inflation",
# Crypto-specific
"bitcoin", "btc", "ethereum", "crypto", "sec",
"strategic reserve",
# South Asia (post 684 family)
"pakistan", "india",
}
# Stop-words we ignore when comparing
_STOP_WORDS = {
"the","a","an","of","to","and","or","in","on","at","is","are","was",
"were","be","been","have","has","had","will","i","we","you","they",
"this","that","with","for","by","from","my","our","your","their",
"it","its","as","but","not","just","very","more","most","than","also",
"would","could","should","do","does","did","new","now","get","got",
"make","made","go","going","go","want","want","know","like","really",
"us","u","said","say","says",
}
def _significant_tokens(text: str) -> set[str]:
"""Lower-cased alphanumeric words ≥4 chars, with stop-words removed."""
raw = re.findall(r"[a-zA-Z]{4,}", text.lower())
return {w for w in raw if w not in _STOP_WORDS}
# How many leading characters to scan for "dominant" topics. Topics that only
# appear in the tail of a long post are background context, not the catalyst.
# A post that mentions Iran in passing 800 chars in shouldn't dedup against
# a different post whose CORE topic is Iran.
TOPIC_HEAD_CHARS = 200
def _topic_keywords_in(text: str) -> set[str]:
"""Return canonical TOPIC_KEYWORDS in the FIRST TOPIC_HEAD_CHARS of `text`.
Canonicalisation collapses substring matches: if both "hormuz" and
"strait of hormuz" match, only the longer one is kept. This prevents
inflated overlap counts where the same physical concept matches twice.
Limiting to the head ensures we match the post's DOMINANT topic, not
incidental mentions buried in the tail.
"""
t = text[:TOPIC_HEAD_CHARS].lower()
raw = {kw for kw in TOPIC_KEYWORDS if kw in t}
# Drop any keyword that is a proper substring of another matched keyword.
canonical = set(raw)
for a in raw:
for b in raw:
if a != b and a in b:
canonical.discard(a)
break
return canonical
# Threshold for dedup by topic. Requiring ≥2 shared CANONICAL topics avoids
# false positives where two posts both mention Iran but are about different
# events (post 684 = Pakistan-Iran context; post 416 = Iran sanctions —
# they share "iran" but nothing else). Two-topic overlap (e.g. Iran + Strait
# of Hormuz, or Bitcoin + Strategic Reserve) is a much stronger same-event
# signal.
DEDUP_TOPIC_MIN = 2
# ─── Public API ─────────────────────────────────────────────────────────────
def has_action_marker(text: str) -> tuple[bool, Optional[str]]:
"""True iff at least one ACTION_MARKERS substring appears in `text`.
Returns (ok, matched_marker_or_None).
"""
t = text.lower()
for m in ACTION_MARKERS:
if m in t:
return True, m
return False, None
def has_future_tense_blocker(text: str) -> tuple[bool, Optional[str]]:
"""True iff one of FUTURE_TENSE_BLOCKERS appears (in first 200 chars)."""
head = text[:200].lower()
for b in FUTURE_TENSE_BLOCKERS:
if b in head:
return True, b
return False, None
async def is_duplicate_of_recent(text: str, db_session_factory) -> tuple[bool, Optional[int]]:
"""True iff a post within DEDUP_WINDOW_HOURS appears to cover the same event.
Two-layer check:
A. Any shared TOPIC_KEYWORD (Iran, Pakistan, tariff, etc.) → duplicate
B. ≥3 shared significant tokens AND ≥50% overlap → duplicate (catches
novel topics not on the curated list)
Returns (is_dup, dup_post_id_or_None).
"""
from sqlalchemy import select
from app.models import Post
new_topics = _topic_keywords_in(text)
new_tokens = _significant_tokens(text)
cutoff = (datetime.now(timezone.utc) - timedelta(hours=DEDUP_WINDOW_HOURS)).replace(tzinfo=None)
async with db_session_factory() as db:
rows = await db.execute(
select(Post.id, Post.text).where(Post.published_at >= cutoff)
)
for pid, prior_text in rows.all():
# Layer A: canonical topic-keyword overlap (≥ DEDUP_TOPIC_MIN)
if new_topics:
prior_topics = _topic_keywords_in(prior_text)
if len(new_topics & prior_topics) >= DEDUP_TOPIC_MIN:
return True, pid
# Layer B: generic token-overlap fallback for novel topics
if len(new_tokens) >= DEDUP_KEYWORD_THRESHOLD:
prior_tokens = _significant_tokens(prior_text)
shared = new_tokens & prior_tokens
if len(shared) >= DEDUP_KEYWORD_THRESHOLD:
if len(shared) / max(len(new_tokens), 1) >= 0.5:
return True, pid
return False, None
async def passes_entry_filter(
text: str,
db_session_factory,
) -> tuple[bool, str]:
"""Master combiner. Returns (passed, reason_string).
On failure, `reason_string` explains which gate killed it. On success,
`reason_string` is empty.
Run this BEFORE analyze_post() to short-circuit the AI call on obvious
non-catalysts.
"""
ok_action, marker = has_action_marker(text)
if not ok_action:
return False, "no_action_marker (no past-tense / formal-statement keyword found)"
has_block, blocker = has_future_tense_blocker(text)
if has_block:
return False, f"future_tense_blocker: {blocker!r}"
is_dup, dup_id = await is_duplicate_of_recent(text, db_session_factory)
if is_dup:
return False, f"duplicate_of_post_{dup_id}"
return True, f"action_marker={marker!r}"
# ─── Backtest helper — synchronous version using a single text + history ────
def passes_entry_filter_sync(text: str, prior_texts: list[str]) -> tuple[bool, str]:
"""Pure-function variant for backtest replay: pass a list of prior post
texts (within the dedup window) instead of hitting the DB.
Useful for running the filter over the historical dataset to count how
many posts would have been EXECUTED vs FILTERED.
"""
ok_action, marker = has_action_marker(text)
if not ok_action:
return False, "no_action_marker"
has_block, blocker = has_future_tense_blocker(text)
if has_block:
return False, f"future_tense: {blocker}"
new_topics = _topic_keywords_in(text)
new_tokens = _significant_tokens(text)
for pt in prior_texts:
# Layer A: canonical topic overlap (≥ DEDUP_TOPIC_MIN)
if new_topics:
shared_topics = new_topics & _topic_keywords_in(pt)
if len(shared_topics) >= DEDUP_TOPIC_MIN:
return False, f"duplicate_topic: {','.join(sorted(shared_topics))}"
# Layer B: generic token overlap
if len(new_tokens) >= DEDUP_KEYWORD_THRESHOLD:
shared = new_tokens & _significant_tokens(pt)
if (len(shared) >= DEDUP_KEYWORD_THRESHOLD
and len(shared) / max(len(new_tokens), 1) >= 0.5):
return False, "duplicate_tokens"
return True, f"ok: {marker}"
app/services/funding_signal.py
+221
View File
@@ -0,0 +1,221 @@
"""
Breakout Signal Monitor — ETH + LINK (BTC as trend context)
Polls Binance every 5 minutes. When both conditions are met:
1. BB squeeze: BB width in bottom 20% of last 60 candles
2. Volume spike: current volume > 2.5x 20-period average
3. Taker buy ratio > 60%
4. Price closes above upper Bollinger Band
Broadcasts alert via WebSocket. Gated by user on/off toggle.
"""
import collections
import logging
from datetime import datetime, timezone
from typing import Deque, Optional
import httpx
from app.config import settings
logger = logging.getLogger(__name__)
# ── Config ────────────────────────────────────────────────────────────────────
WATCH_SYMBOLS = ["ETHUSDT", "LINKUSDT"]
CONTEXT_SYMBOL = "BTCUSDT"
ALL_SYMBOLS = [CONTEXT_SYMBOL] + WATCH_SYMBOLS
CANDLE_LIMIT = 200 # candles to fetch per poll (200 x 5m = ~16.7h history)
BB_PERIOD = 20
BB_SQUEEZE_PCT = 20 # bottom 20% of BB width history → squeeze
VOLUME_MULT = 2.5
TBR_THRESH = 0.60 # taker buy ratio threshold
BTC_TREND_MA = 288 # 24h trend (needs separate longer fetch for BTC)
# ── State ─────────────────────────────────────────────────────────────────────
_enabled: bool = False
_recent_signals: Deque[dict] = collections.deque(maxlen=50)
_last_fired: dict[str, Optional[datetime]] = {s: None for s in WATCH_SYMBOLS}
# ── Public API ────────────────────────────────────────────────────────────────
def set_enabled(value: bool) -> None:
global _enabled
_enabled = value
logger.info("Funding signal monitor: %s", "ENABLED" if value else "DISABLED")
def is_enabled() -> bool:
return _enabled
def get_recent_signals(limit: int = 20) -> list[dict]:
return list(reversed(list(_recent_signals)))[:limit]
def get_status() -> dict:
return {
"enabled": _enabled,
"symbols": WATCH_SYMBOLS,
"recent_signal_count": len(_recent_signals),
}
# ── Binance fetch ─────────────────────────────────────────────────────────────
async def _fetch_klines(client: httpx.AsyncClient, symbol: str, limit: int) -> list[list]:
url = f"{settings.binance_rest_url}/api/v3/klines"
try:
resp = await client.get(url, params={
"symbol": symbol, "interval": "5m", "limit": limit
}, timeout=15)
resp.raise_for_status()
return resp.json()
except Exception as exc:
logger.warning("Failed to fetch klines for %s: %s", symbol, exc)
return []
def _parse_candle(row: list) -> dict:
total_vol = float(row[5])
taker_buy = float(row[9])
return {
"time": row[0],
"open": float(row[1]),
"high": float(row[2]),
"low": float(row[3]),
"close": float(row[4]),
"volume": total_vol,
"tbr": taker_buy / total_vol if total_vol > 0 else 0.5,
}
# ── Indicators ────────────────────────────────────────────────────────────────
def _compute_signal(candles: list[dict]) -> Optional[dict]:
"""
Returns signal dict if breakout signal fires on the most recent candle,
else None. Requires at least 60 candles.
"""
# Drop the last (current incomplete) candle — always use completed candles
candles = candles[:-1]
if len(candles) < 60:
return None
closes = [c["close"] for c in candles]
volumes = [c["volume"] for c in candles]
tbrs = [c["tbr"] for c in candles]
# ── Bollinger Bands (last BB_PERIOD candles) ──────────────────────────────
window = closes[-BB_PERIOD:]
bb_mid = sum(window) / BB_PERIOD
variance = sum((x - bb_mid) ** 2 for x in window) / BB_PERIOD
bb_std = variance ** 0.5
bb_upper = bb_mid + 2 * bb_std
bb_width = (4 * bb_std) / bb_mid if bb_mid > 0 else 0
# BB width percentile over last 60 candles
widths = []
for i in range(len(candles) - 60, len(candles)):
w_window = closes[max(0, i - BB_PERIOD + 1): i + 1]
if len(w_window) < BB_PERIOD:
continue
m = sum(w_window) / len(w_window)
s = (sum((x - m) ** 2 for x in w_window) / len(w_window)) ** 0.5
widths.append((4 * s) / m if m > 0 else 0)
if not widths:
return None
rank = sum(1 for w in widths if w < bb_width) / len(widths) * 100
in_squeeze = rank < BB_SQUEEZE_PCT
# ── Volume ────────────────────────────────────────────────────────────────
vol_ma = sum(volumes[-BB_PERIOD - 1:-1]) / BB_PERIOD
vol_spike = volumes[-1] > VOLUME_MULT * vol_ma if vol_ma > 0 else False
# ── Taker buy ratio ───────────────────────────────────────────────────────
tbr_ok = tbrs[-1] > TBR_THRESH
# ── Price above upper BB ──────────────────────────────────────────────────
above_bb = closes[-1] > bb_upper
if in_squeeze and vol_spike and tbr_ok and above_bb:
return {
"bb_width_pct": round(rank, 1),
"vol_mult": round(volumes[-1] / vol_ma, 2) if vol_ma > 0 else 0,
"tbr": round(tbrs[-1], 3),
"close": closes[-1],
"bb_upper": round(bb_upper, 4),
}
return None
def _btc_trend(candles: list[dict]) -> Optional[bool]:
"""True = BTC above 24h MA, False = below, None = not enough data."""
if len(candles) < BTC_TREND_MA:
return None
closes = [c["close"] for c in candles]
ma = sum(closes[-BTC_TREND_MA:]) / BTC_TREND_MA
return closes[-1] > ma
# ── Main poll loop ────────────────────────────────────────────────────────────
async def poll_funding_signal() -> None:
"""Called by APScheduler every 5 minutes."""
from app.ws.manager import manager # avoid circular import at module load
async with httpx.AsyncClient(timeout=20) as client:
# Fetch BTC with longer history for 24h trend
btc_rows = await _fetch_klines(client, CONTEXT_SYMBOL, BTC_TREND_MA + 10)
btc_candles = [_parse_candle(r) for r in btc_rows]
btc_up = _btc_trend(btc_candles)
# Fetch ETH + LINK
for symbol in WATCH_SYMBOLS:
rows = await _fetch_klines(client, symbol, CANDLE_LIMIT)
if not rows:
continue
candles = [_parse_candle(r) for r in rows]
sig = _compute_signal(candles)
if sig is None:
continue
# Deduplicate: don't fire same symbol twice within 30 minutes
now = datetime.now(timezone.utc)
last = _last_fired.get(symbol)
if last and (now - last).total_seconds() < 1800:
logger.info("Signal for %s deduplicated (too soon)", symbol)
continue
_last_fired[symbol] = now
alert = {
"type": "funding_signal",
"symbol": symbol,
"time": now.isoformat(),
"close": sig["close"],
"tbr": sig["tbr"],
"vol_mult": sig["vol_mult"],
"bb_pct": sig["bb_width_pct"],
"bb_upper": sig["bb_upper"],
"btc_trend": "↑ uptrend" if btc_up else ("↓ downtrend" if btc_up is False else "unknown"),
"enabled": _enabled,
}
_recent_signals.append(alert)
if _enabled:
logger.info("🚨 SIGNAL %s | price=%.4f tbr=%.2f vol=%.1fx btc=%s",
symbol, sig["close"], sig["tbr"], sig["vol_mult"],
alert["btc_trend"])
await manager.broadcast(alert)
else:
logger.info("Signal %s (monitor OFF — not broadcast)", symbol)
app/services/hyperliquid.py
+74
View File
@@ -301,3 +301,77 @@ class HyperliquidTrader:
fill_price = float(filled_info.get("avgPx", mid) or mid)
logger.info("Closed %s position @ %.2f (size=%.5f)", coin, fill_price, total_sz)
return {"fill_price": fill_price, "already_closed": False}
async def reduce_position(self, asset: str, fraction: float) -> dict:
"""Partially close `fraction` (0<f<1) of the CURRENT open position
for `asset` with a reduce-only IOC order. Used by System-2 staged
de-risk. Returns {"fill_price": float, "closed_fraction": float,
"already_closed": bool}.
`closed_fraction` is the fraction of the position that was actually
closed (filled_size / pre-existing size) so the caller's PnL +
remaining bookkeeping stays exact even on partial fills.
"""
coin = asset.upper()
f = max(0.0, min(1.0, float(fraction)))
if f <= 0.0:
return {"fill_price": None, "closed_fraction": 0.0, "already_closed": False}
if f >= 1.0:
r = await self.close_position(asset)
r["closed_fraction"] = 0.0 if r.get("already_closed") else 1.0
return r
positions = await self.get_open_positions()
target = next((p for p in positions if p.get("coin") == coin), None)
if target is None:
logger.warning("reduce_position: no open %s on HL — already closed?", coin)
return {"fill_price": None, "closed_fraction": 0.0, "already_closed": True}
szi = float(target["szi"])
is_buy = szi < 0 # reducing a short → buy back
pre_size = abs(szi)
sz_dec = await self._get_sz_decimals(coin)
size = round(pre_size * f, sz_dec)
if size <= 0:
# Fraction rounds to nothing at this asset's size precision —
# treat as a no-op so the caller can advance the step without
# an erroneous PnL slice.
logger.warning("reduce_position: %s fraction %.3f rounds to 0 size (pre=%.6f)",
coin, f, pre_size)
return {"fill_price": None, "closed_fraction": 0.0, "already_closed": False}
mid = await self._mid_price(coin)
slippage = 0.01
raw_px = mid * (1 + slippage) if is_buy else mid * (1 - slippage)
if raw_px >= 10000:
limit_px = float(round(raw_px))
elif raw_px >= 1000:
limit_px = round(raw_px, 1)
elif raw_px >= 100:
limit_px = round(raw_px, 2)
else:
limit_px = round(raw_px, 3)
logger.info("Reducing %s by %.0f%%: size=%.6f @ limit %.2f (pre=%.6f)",
coin, f * 100, size, limit_px, pre_size)
result = await self._run(
self._exchange.order,
coin, is_buy, size, limit_px,
{"limit": {"tif": "Ioc"}},
reduce_only=True,
)
statuses = result.get("response", {}).get("data", {}).get("statuses", [{}])
status = statuses[0] if statuses else {}
if "error" in status:
raise ValueError(f"HL reduce rejected: {status['error']}")
filled_info = status.get("filled") or {}
total_sz = float(filled_info.get("totalSz", 0) or 0)
if total_sz <= 0:
raise ValueError(f"reduce_position {coin}: IOC returned 0 fill")
fill_price = float(filled_info.get("avgPx", mid) or mid)
closed_fraction = (total_sz / pre_size) if pre_size > 0 else 0.0
logger.info("Reduced %s: closed %.6f / %.6f (%.1f%%) @ %.2f",
coin, total_sz, pre_size, closed_fraction * 100, fill_price)
return {"fill_price": fill_price, "closed_fraction": closed_fraction,
"already_closed": False}
app/services/kol_analysis.py
+223
View File
@@ -0,0 +1,223 @@
"""KOL post → structured signal extractor.
Takes a long-form post (Substack essay) or tweet and returns:
- summary: one Chinese sentence on what this post is about
- tickers: list of {ticker, action, conviction, quote}
action ∈ bullish | bearish | buy | sell | mention
- buy/sell → KOL explicitly states they bought/sold or are entering/exiting
- bullish/bearish → directional view without an explicit position statement
- mention → ticker appears but no clear stance (don't flood with these)
conviction ∈ 0.0–1.0
- 0.8+ : explicit, repeated, with sizing / timing
- 0.5–0.7 : clear view, no commitment
- <0.5 : passing reference
Quote is the shortest verbatim sentence supporting the call.
Uses the same Anthropic client style as analysis.py. Designed to be reused
by the Trump-post AI signal module (TODO #4) — same JSON shape, just with
different KOL context strings.
"""
from __future__ import annotations
import json
import logging
from datetime import datetime, timezone
from typing import Optional
from openai import AsyncOpenAI
from app.config import settings
logger = logging.getLogger(__name__)
ANALYSIS_VERSION = "kol-v1"
ANTHROPIC_MODEL = "claude-haiku-4-5-20251001"
_anthropic_client = None
_openai_client: Optional[AsyncOpenAI] = None
def _use_anthropic() -> bool:
return bool(settings.anthropic_api_key)
def _anth():
global _anthropic_client
if _anthropic_client is None:
import anthropic as _a
_anthropic_client = _a.AsyncAnthropic(api_key=settings.anthropic_api_key)
return _anthropic_client
def _oai() -> AsyncOpenAI:
global _openai_client
if _openai_client is None:
_openai_client = AsyncOpenAI(
api_key=settings.ai_api_key,
base_url=settings.ai_base_url,
)
return _openai_client
SYSTEM_PROMPT = """You are an analyst extracting tradeable signals from crypto KOL posts.
The author is a known crypto KOL. Your job: distill what they said and which tokens they are talking about RIGHT NOW (not historical references).
Output **strict JSON only**, no markdown, no preface. Schema:
{
"summary": "<one sentence, ≤60 chars/字. If signal exists, state the author's current thesis. If no signal, describe the post topic. Match the post's primary language (中文文章用中文, English 用英文).>",
"tickers": [
{
"ticker": "<UPPERCASE symbol, e.g. BTC, ETH, HYPE, SOL>",
"action": "buy" | "sell" | "bullish" | "bearish" | "mention",
"conviction": <float 0.0-1.0>,
"quote": "<shortest verbatim sentence from the post supporting this call, ≤200 chars. Use the post's original language — do not translate.>"
}
]
}
Rules:
- If the post is macro commentary, news recap, or sponsored content with no specific token call, return tickers=[] and summary describing the topic.
- IGNORE historical price references ("BTC bottomed at $60k earlier this year") — these are context, not current calls.
- IGNORE advertising/sponsor sections — look for cues: "sponsor", "partner", "use code", "promo code", "this episode brought to you by", "ad", "广告", "赞助". Skip any ticker only mentioned inside such a section.
- buy/sell only when the author states a position action ("I bought", "we are long", "我们减仓了", "added to my bag"). Otherwise use bullish/bearish for directional views, or mention for passing references.
- Dedupe per ticker — at most one entry per symbol; pick the strongest action.
- Do NOT invent tickers. If you see "$XYZ" but unsure it's a real token, skip it.
- conviction: 0.8+ requires explicit + repeated + sized/timed view; 0.5-0.7 for clear directional view without commitment; <0.5 for passing references.
- Do not include fiat (USD/CNY/JPY) or stablecoins (USDT/USDC/DAI/FRAX) unless the post's main thesis is about them.
"""
USER_TEMPLATE = """Today is {today_utc}.
KOL handle: {handle}
Source: {source}
Title: {title}
Post body:
\"\"\"
{body}
\"\"\"
"""
def _truncate(text: str, max_chars: int = 24000) -> str:
"""Substack essays can be 50K+ chars. Haiku handles it but we cap to
control cost. Keep head + tail since conclusions often appear at the end."""
if len(text) <= max_chars:
return text
head = max_chars * 2 // 3
tail = max_chars - head
return text[:head] + "\n\n[...trimmed...]\n\n" + text[-tail:]
def _parse_json(raw: str) -> dict:
raw = raw.strip()
if raw.startswith("```"):
# strip fenced code block
raw = raw.split("\n", 1)[1] if "\n" in raw else raw
if raw.endswith("```"):
raw = raw.rsplit("```", 1)[0]
raw = raw.strip()
# Some models prepend "json" after the fence
if raw.startswith("json"):
raw = raw[4:].strip()
return json.loads(raw)
async def extract_kol_signal(
*,
handle: str,
source: str,
title: Optional[str],
body: str,
model: Optional[str] = None,
) -> dict:
"""Run the extractor. Returns {summary, tickers, model, version}.
Returns an empty-but-valid dict on parse/API failure rather than raising —
the caller stores the post regardless; an unanalyzed post can be retried.
"""
today_utc = datetime.now(timezone.utc).strftime("%Y-%m-%d")
user = USER_TEMPLATE.format(
today_utc=today_utc,
handle=handle,
source=source,
title=title or "",
body=_truncate(body),
)
use_anth = _use_anthropic()
if model is None:
# KOL analysis is a daily batch job, not latency-sensitive. Use the
# higher-quality `ai_model` (DeepSeek v4 Pro / reasoning) rather than
# the live `ai_live_model` (flash) reserved for Trump real-time path.
model = ANTHROPIC_MODEL if use_anth else settings.ai_model
try:
if use_anth:
msg = await _anth().messages.create(
model=model,
max_tokens=1500,
temperature=0.1,
system=SYSTEM_PROMPT,
messages=[{"role": "user", "content": user}],
)
raw = (msg.content[0].text if msg.content else "").strip()
else:
# OpenAI-compatible (DeepSeek). Reasoning models need higher tokens
# + no temperature; flash/chat models are fine with both.
is_reasoning = any(x in model for x in ("pro", "reasoner", "r1", "think"))
kwargs = {"model": model, "messages": [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": user},
], "max_tokens": 4000 if is_reasoning else 1500}
if not is_reasoning:
kwargs["temperature"] = 0.1
# JSON mode — DeepSeek + OpenAI both support response_format.
# Eliminates fenced/preface parse failures. Skipped for reasoning
# models (some don't accept response_format alongside reasoning).
if not is_reasoning:
kwargs["response_format"] = {"type": "json_object"}
resp = await _oai().chat.completions.create(**kwargs)
raw = (resp.choices[0].message.content or "").strip()
data = _parse_json(raw)
except Exception as e:
logger.warning("kol_analysis extract failed for %s: %s", handle, e)
return {"summary": None, "tickers": [], "model": model,
"version": ANALYSIS_VERSION, "error": str(e)}
# Normalize
tickers = data.get("tickers") or []
cleaned = []
for t in tickers:
if not isinstance(t, dict):
continue
sym = (t.get("ticker") or "").strip().upper()
if not sym or len(sym) > 12:
continue
action = (t.get("action") or "mention").lower()
if action not in {"buy", "sell", "bullish", "bearish", "mention"}:
action = "mention"
try:
conv = float(t.get("conviction") or 0)
except (TypeError, ValueError):
conv = 0.0
conv = max(0.0, min(1.0, conv))
cleaned.append({
"ticker": sym,
"action": action,
"conviction": round(conv, 2),
"quote": (t.get("quote") or "")[:200],
})
return {
"summary": (data.get("summary") or "").strip() or None,
"tickers": cleaned,
"model": model,
"version": ANALYSIS_VERSION,
}
app/services/kol_divergence.py
+285
View File
@@ -0,0 +1,285 @@
"""KOL talks-vs-trades cross-signal detector.
Compares B-tier content signals (Substack / Twitter post → ticker action)
against A-tier on-chain changes (wallet holding changes) for the same
KOL handle + ticker within a ±N-day window.
Two outcomes:
divergence — KOL says bullish but on-chain is selling (or vice versa).
On-chain action is the ground truth; word is noise/manipulation.
Conclusion = opposite of what was said.
alignment — Post and chain agree. Conviction is reinforced.
Conclusion = what was said (and done).
Logic:
post side → action ∈ {buy, bullish} = LONG intent
{sell, bearish} = SHORT intent
{mention} = skip (no clear view)
chain side → change_type ∈ {new_position, increased} = LONG action
{decreased, closed} = SHORT action
LONG intent + SHORT action → divergence, direction='short'
SHORT intent + LONG action → divergence, direction='long'
LONG intent + LONG action → alignment, direction='long'
SHORT intent + SHORT action → alignment, direction='short'
Run cadence: after each onchain poll (02:05 UTC). Also callable manually.
Idempotent: UniqueConstraint(post_id, change_id) prevents double-writes.
"""
from __future__ import annotations
import hashlib
import json
import logging
from collections import defaultdict
from datetime import datetime, timedelta, timezone
from typing import Optional
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.database import AsyncSessionLocal
from app.models import KolDivergence, KolHoldingChange, KolPost, KolWallet, Post, utcnow
logger = logging.getLogger(__name__)
# Match window: post and chain event must be within this many days of each other
WINDOW_DAYS = 7
# Only surface on-chain changes above this threshold (avoid noise from dust)
MIN_USD_CHANGE = 10_000
# Post actions that map to a directional view (skip 'mention')
_POST_LONG = {"buy", "bullish"}
_POST_SHORT = {"sell", "bearish"}
# On-chain actions that map to a direction
_CHAIN_LONG = {"new_position", "increased"}
_CHAIN_SHORT = {"decreased", "closed"}
# ── Telegram alert gating ────────────────────────────────────────────────────
# Only push divergences (the surprising "they say one thing, do another" case).
# Alignments are confirmations, not unique alpha — skip to keep volume sane.
ALERT_ON_TYPES = {"divergence"}
# Floor on the post's stated conviction. Without this, every "mention"-like
# soft view that happens to coincide with a buy/sell would page users.
ALERT_MIN_POST_CONVICTION = 0.5
# Minimum USD on the on-chain side to bother alerting. The 10K floor in
# MIN_USD_CHANGE keeps dust out of the table; this stricter floor avoids
# alerting on small rebalances by big wallets.
ALERT_MIN_USD = 50_000
def _post_for_divergence(handle: str, ticker: str, direction: str,
post_action: str, chain_action: str,
conviction: float, change_id: int,
usd_after: Optional[float],
days_apart: float) -> Post:
"""Build a Post row carrying a KOL divergence as an actionable signal.
`direction` is the CONCLUSION direction from _classify: 'long' → buy,
'short' → short. ai_confidence is derived from post conviction so the
user's min_confidence floor in TelegramBinding gates noise.
"""
signal = "buy" if direction == "long" else "short"
usd_str = f"${(usd_after or 0)/1000:.0f}K" if usd_after else "?"
text = (
f"KOL {handle} says {post_action.upper()} {ticker} — "
f"but on-chain shows {chain_action} ({usd_str}, Δ{days_apart:.1f}d). "
f"Following the chain (the trade, not the talk): {signal.upper()} {ticker}."
)
# external_id must be unique-per-source. Use the change_id as the dedupe
# key — at most one Post per (kol, ticker, chain-event).
ext = hashlib.md5(f"kol_divergence:{handle}:{ticker}:{change_id}".encode()).hexdigest()
return Post(
external_id=ext,
text=text,
source="kol_divergence",
published_at=datetime.now(timezone.utc).replace(tzinfo=None),
sentiment="bullish" if signal == "buy" else "bearish",
ai_confidence=int(round(conviction * 100)),
relevant=True,
signal=signal,
target_asset=ticker,
category=f"kol_divergence_{signal}",
analysis_version="kol_divergence_v1",
prefilter_reason="kol_divergence",
)
def _classify(post_action: str, chain_action: str) -> Optional[tuple[str, str]]:
"""Returns (signal_type, direction) or None if no meaningful pair."""
if post_action in _POST_LONG:
if chain_action in _CHAIN_LONG:
return "alignment", "long"
if chain_action in _CHAIN_SHORT:
return "divergence", "short"
elif post_action in _POST_SHORT:
if chain_action in _CHAIN_LONG:
return "divergence", "long"
if chain_action in _CHAIN_SHORT:
return "alignment", "short"
return None
async def run_divergence_scan(
lookback_days: int = 30,
) -> list[dict]:
"""Scan recent posts × on-chain changes, write new KolDivergence rows.
Returns list of newly written rows as dicts (for logging / API return).
Already-stored pairs are silently skipped (idempotent).
"""
since = datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(days=lookback_days)
results: list[dict] = []
# Posts we created this scan — notify_signal runs AFTER commit so the
# dispatcher's separate DB session can read the row.
alert_post_ids: list[int] = []
async with AsyncSessionLocal() as session:
# ── 1. Load recent posts that have directional ticker signals ──────
posts = (await session.execute(
select(KolPost)
.where(KolPost.published_at >= since)
.where(KolPost.tickers_json.is_not(None))
)).scalars().all()
# ── 2. Load recent on-chain changes (keyed by handle via wallet) ───
changes_rows = (await session.execute(
select(KolHoldingChange, KolWallet.handle)
.join(KolWallet, KolHoldingChange.wallet_id == KolWallet.id)
.where(KolHoldingChange.detected_at >= since)
)).all()
# Pre-load all existing (post_id, change_id) pairs so we can skip
# duplicates client-side instead of relying on a UniqueConstraint
# violation. Catching the violation would force a session rollback
# that wipes ALL pending writes (not just the offending one) — a
# subtle data-loss bug. Pre-check is cheap (the table stays small).
existing_pairs: set[tuple[int, int]] = {
(pid, cid)
for (pid, cid) in (await session.execute(
select(KolDivergence.post_id, KolDivergence.change_id)
)).all()
}
# Index changes by (handle, ticker) → list[KolHoldingChange]
chain_index: dict[tuple[str, str], list[tuple[KolHoldingChange, str]]] = defaultdict(list)
for change, handle in changes_rows:
# Skip dust moves
usd_delta = abs((change.usd_after or 0) - (change.usd_before or 0))
if usd_delta < MIN_USD_CHANGE and (change.usd_after or 0) < MIN_USD_CHANGE:
continue
chain_index[(handle, change.ticker.upper())].append((change, handle))
# ── 3. Match posts → changes ───────────────────────────────────────
for post in posts:
try:
tickers = json.loads(post.tickers_json or "[]")
except json.JSONDecodeError:
continue
for t in tickers:
post_action = (t.get("action") or "").lower()
if post_action not in (_POST_LONG | _POST_SHORT):
continue # skip 'mention'
ticker = (t.get("ticker") or "").upper()
if not ticker:
continue
candidates = chain_index.get((post.kol_handle, ticker), [])
for change, handle in candidates:
# Time-window check
post_dt = post.published_at
change_dt = change.detected_at
days_apart = abs((change_dt - post_dt).total_seconds()) / 86400
if days_apart > WINDOW_DAYS:
continue
result = _classify(post_action, change.change_type)
if result is None:
continue
signal_type, direction = result
# Idempotency: skip if (post_id, change_id) already stored
pair_key = (post.id, change.id)
if pair_key in existing_pairs:
continue
existing_pairs.add(pair_key)
conviction = float(t.get("conviction") or 0)
row = KolDivergence(
handle = post.kol_handle,
ticker = ticker,
post_id = post.id,
post_action = post_action,
post_conviction = conviction,
post_at = post_dt,
change_id = change.id,
onchain_action = change.change_type,
usd_before = change.usd_before,
usd_after = change.usd_after,
onchain_at = change_dt,
signal_type = signal_type,
direction = direction,
days_apart = round(days_apart, 2),
)
session.add(row)
# Emit a Post for Telegram fan-out — but only for the
# interesting case (divergences) with enough conviction
# and chain-size to be worth a push. See ALERT_* tunables.
if (signal_type in ALERT_ON_TYPES
and conviction >= ALERT_MIN_POST_CONVICTION
and (change.usd_after or 0) >= ALERT_MIN_USD):
alert_post = _post_for_divergence(
handle=post.kol_handle, ticker=ticker,
direction=direction, post_action=post_action,
chain_action=change.change_type,
conviction=conviction, change_id=change.id,
usd_after=change.usd_after, days_apart=days_apart,
)
session.add(alert_post)
await session.flush() # populate alert_post.id
alert_post_ids.append(alert_post.id)
info = {
"handle": post.kol_handle,
"ticker": ticker,
"signal_type": signal_type,
"direction": direction,
"post_action": post_action,
"onchain_action": change.change_type,
"days_apart": round(days_apart, 2),
"usd_after": change.usd_after,
}
results.append(info)
emoji = "⚠️" if signal_type == "divergence" else "✅"
logger.info(
"[kol_divergence] %s %s %s: says %s, onchain %s → %s (%s) Δ%.1fd",
emoji, post.kol_handle, ticker,
post_action, change.change_type,
signal_type, direction, days_apart,
)
await session.commit()
# Fire Telegram fan-out AFTER commit so _dispatch's own session can
# actually see the rows. _dispatch only needs the post_id, so we skip
# notify_signal's Post-object wrapper and schedule it directly.
if alert_post_ids:
try:
import asyncio
from app.services.telegram import _dispatch
for pid in alert_post_ids:
asyncio.create_task(_dispatch(pid))
except Exception as exc:
logger.warning("[kol_divergence] notify_signal failed: %s", exc)
logger.info("[kol_divergence] scan done: %d new pairs written, %d alerts queued",
len(results), len(alert_post_ids))
return results
app/services/kol_onchain.py
+531
View File
@@ -0,0 +1,531 @@
"""KOL A-tier: daily on-chain holdings snapshot + change detection.
Data sources (all free):
Hyperliquid public API — No key. Perp positions + mark prices for any
HL-listed asset (BTC/ETH/SOL/HYPE/...).
Endpoint: POST /info {"type":"clearinghouseState"}
Prices: POST /info {"type":"allMids"}
Etherscan API — Free key (etherscan.io/register, email only).
ERC-20 token balances for any Ethereum address.
Env: ETHERSCAN_API_KEY. Skip ETH wallets if unset.
CoinGecko (no key) — Fallback price for tokens not listed on HL.
Free tier: 30 req/min, batch by contract address.
Pricing priority per token:
1. HL allMids (by symbol, e.g. "ETH" → $2025)
2. CoinGecko by contract address (Ethereum only)
3. Skip (price unknown → usd_value=0, excluded from snapshot)
Daily flow (run_onchain_poll at 02:00 UTC):
For each HL wallet → fetch HL positions → snapshot → diff
For each ETH wallet → fetch Etherscan ERC-20 list → price each token → snapshot → diff
Diffs with usd_after > $50k (new) or ±25% change written to kol_holding_changes.
"""
from __future__ import annotations
import asyncio
import json
import logging
import time
from datetime import datetime, timezone
from typing import Optional
import httpx
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.config import settings
from app.database import AsyncSessionLocal
from app.models import KolHoldingChange, KolHoldingSnapshot, KolWallet, utcnow
logger = logging.getLogger(__name__)
HL_API_URL = "https://api.hyperliquid.xyz/info"
ETHERSCAN_API_URL = "https://api.etherscan.io/v2/api"
COINGECKO_URL = "https://api.coingecko.com/api/v3"
# Stablecoins — ignored in all snapshots
STABLES = {"USDC", "USDT", "DAI", "BUSD", "TUSD", "USDE", "FRAX", "LUSD", "CRVUSD"}
# Change detection thresholds
_NEW_POSITION_MIN_USD = 50_000 # new token > $50k → alert
_CHANGE_PCT_THRESHOLD = 25.0 # ±25% USD move → alert
_CLOSED_MIN_USD = 10_000 # closed position must have been > $10k to report
# ── Price layer ───────────────────────────────────────────────────────────────
_hl_prices: dict[str, float] = {} # symbol → USD, refreshed each poll
_hl_prices_fetched_at: float = 0.0
async def _get_hl_prices() -> dict[str, float]:
"""Fetch all HL mark prices (free, no auth). Cached for 5 min per poll cycle.
Returns empty dict on network failure — callers fall back to CoinGecko."""
global _hl_prices, _hl_prices_fetched_at
now = time.time()
if now - _hl_prices_fetched_at < 300 and _hl_prices:
return _hl_prices
try:
async with httpx.AsyncClient(timeout=10.0) as c:
r = await c.post(HL_API_URL, json={"type": "allMids"})
r.raise_for_status()
raw = r.json() # {"BTC": "74541.0", "ETH": "2025.0", ...}
_hl_prices = {sym: float(px) for sym, px in raw.items()}
_hl_prices_fetched_at = now
logger.info("[kol_onchain] HL prices loaded: %d assets", len(_hl_prices))
except Exception as e:
logger.warning("[kol_onchain] HL price fetch failed (%s) — falling back to CoinGecko", e)
# Don't overwrite a previous good cache on transient failure
if not _hl_prices:
_hl_prices = {}
return _hl_prices
# CoinGecko symbol→id map for common crypto assets (fallback when HL unavailable)
_CG_SYMBOL_IDS = {
"BTC": "bitcoin", "ETH": "ethereum", "SOL": "solana",
"BNB": "binancecoin", "AVAX": "avalanche-2", "MATIC": "matic-network",
"ARB": "arbitrum", "OP": "optimism", "LINK": "chainlink",
"UNI": "uniswap", "AAVE": "aave", "MKR": "maker",
"HYPE": "hyperliquid", "ENA": "ethena", "PENDLE": "pendle",
"WLD": "worldcoin-wld", "JUP": "jupiter-exchange-solana",
"WBTC": "wrapped-bitcoin", "STETH": "staked-ether",
"EETH": "ether-fi-staked-eth", "WEETH": "wrapped-eeth",
}
async def _coingecko_prices_by_symbol(symbols: list[str]) -> dict[str, float]:
"""Batch price lookup by CoinGecko coin ID. No key needed."""
ids_needed = {s: _CG_SYMBOL_IDS[s] for s in symbols if s in _CG_SYMBOL_IDS}
if not ids_needed:
return {}
try:
async with httpx.AsyncClient(timeout=15.0) as c:
r = await c.get(f"{COINGECKO_URL}/simple/price", params={
"ids": ",".join(ids_needed.values()),
"vs_currencies": "usd",
})
if r.status_code != 200:
return {}
data = r.json()
# Invert: coin_id → price, then map back to symbol
id_to_price = {v: data.get(v, {}).get("usd", 0) for v in ids_needed.values()}
return {sym: id_to_price[cid] for sym, cid in ids_needed.items() if id_to_price.get(cid)}
except Exception as e:
logger.warning("[kol_onchain] CoinGecko symbol lookup failed: %s", e)
return {}
async def _coingecko_prices_by_contract(
contract_addresses: list[str],
) -> dict[str, float]:
"""Batch price lookup by Ethereum contract address. No key needed.
Chunks into batches of 25 (CoinGecko free tier limit).
Returns {contract_addr_lower: usd_price}."""
if not contract_addresses:
return {}
result: dict[str, float] = {}
addrs_lower = [a.lower() for a in contract_addresses[:100]]
# CoinGecko free tier caps at ~25 addresses per request
chunk_size = 25
async with httpx.AsyncClient(timeout=15.0) as c:
for i in range(0, len(addrs_lower), chunk_size):
chunk = addrs_lower[i:i + chunk_size]
try:
r = await c.get(
f"{COINGECKO_URL}/simple/token_price/ethereum",
params={"contract_addresses": ",".join(chunk), "vs_currencies": "usd"},
)
if r.status_code != 200:
logger.warning("[kol_onchain] CoinGecko %s: %s", r.status_code, r.text[:80])
continue
data = r.json()
result.update({addr: info["usd"] for addr, info in data.items() if "usd" in info})
except Exception as e:
logger.warning("[kol_onchain] CoinGecko chunk failed: %s", e)
return result
# ── Hyperliquid: perp positions ───────────────────────────────────────────────
async def _fetch_hl_positions(address: str) -> tuple[list[dict], float]:
"""Query HL clearinghouseState. Returns (holdings, account_value_usd).
holdings: [{ticker, amount, usd_value, chain, side}]
"""
async with httpx.AsyncClient(timeout=15.0) as c:
r = await c.post(HL_API_URL, json={"type": "clearinghouseState", "user": address})
r.raise_for_status()
data = r.json()
prices = await _get_hl_prices()
holdings = []
for pos in data.get("assetPositions", []):
p = pos.get("position", {})
coin = p.get("coin", "")
if not coin:
continue
size = float(p.get("szi", 0))
if abs(size) < 1e-8:
continue
price = prices.get(coin, float(p.get("entryPx") or 0))
notional = abs(size) * price
if notional < 1:
continue
holdings.append({
"ticker": coin,
"amount": abs(size),
"usd_value": round(notional, 2),
"chain": "hl",
"side": "long" if size > 0 else "short",
})
margin = data.get("marginSummary", {})
account_value = float(margin.get("accountValue", 0))
return holdings, account_value
# ── Etherscan: ERC-20 balances ────────────────────────────────────────────────
async def _fetch_eth_holdings(address: str) -> tuple[list[dict], float]:
"""Fetch ERC-20 + ETH holdings via Etherscan free tier.
Free-tier strategy (no Pro needed):
Step 1: GET account/balance → native ETH amount
Step 2: GET account/tokentx (recent) → discover which ERC-20 contracts
the wallet has interacted with
Step 3: GET account/tokenbalance → current balance per contract
Step 4: Price via HL allMids, then CoinGecko by contract address fallback
Etherscan free limit: 5 req/s, ~100k req/day — well within budget for
daily snapshots of a handful of KOL wallets.
"""
if not settings.etherscan_api_key:
return [], 0.0
prices = await _get_hl_prices()
key = settings.etherscan_api_key
async with httpx.AsyncClient(timeout=20.0) as c:
# ── Step 1: ETH native balance ───────────────────────────────────
eth_r = await c.get(ETHERSCAN_API_URL, params={
"chainid": "1", "module": "account", "action": "balance",
"address": address, "tag": "latest", "apikey": key,
})
eth_data = eth_r.json()
# ── Step 2: ERC-20 tx history → unique (contract, symbol, decimals) ─
# offset=200 is enough to cover all tokens a KOL holds. Vitalik-scale
# addresses with 100k+ txs can still timeout — real KOL wallets are far smaller.
tx_r = await c.get(ETHERSCAN_API_URL, params={
"chainid": "1", "module": "account", "action": "tokentx",
"address": address, "page": "1", "offset": "200",
"sort": "desc", "apikey": key,
}, timeout=30.0)
tx_data = tx_r.json()
# Collect unique contracts from tx history
seen: dict[str, dict] = {} # contract → {symbol, decimals}
txs = tx_data.get("result") or []
if isinstance(txs, list):
for tx in txs:
contract = (tx.get("contractAddress") or "").lower()
if not contract or contract in seen:
continue
symbol = (tx.get("tokenSymbol") or "").upper()
if not symbol or symbol in STABLES:
continue
try:
decimals = int(tx.get("tokenDecimal") or 18)
except ValueError:
decimals = 18
seen[contract] = {"symbol": symbol, "decimals": decimals}
# ── Step 3: current balance per contract ─────────────────────────────
holdings: list[dict] = []
unpriced: list[str] = [] # contracts needing CoinGecko
contract_meta: dict[str, dict] = {}
# Rate-limit: Etherscan free = 5 req/s. Sleep 0.22s between calls to stay
# under burst limit even when looping 80 tokens × 4 wallets in one poll.
async with httpx.AsyncClient(timeout=20.0) as c:
for contract, meta in list(seen.items())[:80]: # cap at 80 tokens
await asyncio.sleep(0.22)
bal_r = await c.get(ETHERSCAN_API_URL, params={
"chainid": "1", "module": "account", "action": "tokenbalance",
"contractaddress": contract, "address": address,
"tag": "latest", "apikey": key,
})
bal_data = bal_r.json()
if bal_data.get("message") != "OK":
continue
raw = int(bal_data.get("result") or 0)
if raw == 0:
continue
balance = raw / (10 ** meta["decimals"])
symbol = meta["symbol"]
hl_price = prices.get(symbol)
if hl_price and hl_price > 0:
usd_value = round(balance * hl_price, 2)
if usd_value >= 500:
holdings.append({
"ticker": symbol, "amount": round(balance, 6),
"usd_value": usd_value, "chain": "ethereum",
"contract": contract,
})
else:
unpriced.append(contract)
contract_meta[contract] = {
"ticker": symbol, "amount": round(balance, 6),
"chain": "ethereum", "contract": contract,
}
# ── Step 4a: ETH native ──────────────────────────────────────────────
eth_raw = int(eth_data.get("result") or 0)
eth_balance = eth_raw / 1e18
if eth_balance > 0.001:
eth_price = prices.get("ETH", 0.0)
# Fallback: CoinGecko by symbol if HL unavailable
if not eth_price:
cg_sym = await _coingecko_prices_by_symbol(["ETH"])
eth_price = cg_sym.get("ETH", 0.0)
eth_usd = round(eth_balance * eth_price, 2)
if eth_usd >= 100:
holdings.append({
"ticker": "ETH", "amount": round(eth_balance, 6),
"usd_value": eth_usd, "chain": "ethereum",
})
# ── Step 4b: CoinGecko fallback for unpriced tokens ──────────────────
if unpriced:
try:
cg_prices = await _coingecko_prices_by_contract(unpriced)
for contract, price in cg_prices.items():
meta = contract_meta.get(contract)
if not meta or price <= 0:
continue
usd_value = round(meta["amount"] * price, 2)
if usd_value >= 500:
holdings.append({**meta, "usd_value": usd_value})
except Exception as e:
logger.warning("[kol_onchain] CoinGecko fallback failed: %s", e)
# Merge duplicate tickers (different contracts, same symbol) by summing USD value
merged: dict[str, dict] = {}
for h in holdings:
t = h["ticker"]
if t in merged:
merged[t]["usd_value"] = round(merged[t]["usd_value"] + h["usd_value"], 2)
merged[t]["amount"] = round(merged[t]["amount"] + h["amount"], 6)
else:
merged[t] = dict(h)
holdings = list(merged.values())
total_usd = round(sum(h["usd_value"] for h in holdings), 2)
logger.info("[kol_onchain] %s ETH holdings: %d tokens, $%.0f total",
address[:10], len(holdings), total_usd)
return holdings, total_usd
# ── Snapshot + diff ───────────────────────────────────────────────────────────
def _diff_holdings(
prev: list[dict], curr: list[dict], wallet_id: int,
) -> list[KolHoldingChange]:
# SUM by ticker — earlier snapshots may contain duplicate ticker rows
# (different contracts, same symbol e.g. two APE tokens). Dict comprehension
# would silently drop one and falsely flag +1900% diffs the next day.
def _sum_by_ticker(rows: list[dict]) -> dict[str, float]:
agg: dict[str, float] = {}
for h in rows:
t = h.get("ticker")
if not t:
continue
agg[t] = agg.get(t, 0.0) + float(h.get("usd_value") or 0)
return agg
prev_map = _sum_by_ticker(prev)
curr_map = _sum_by_ticker(curr)
now = utcnow()
changes = []
for ticker in set(prev_map) | set(curr_map):
before = prev_map.get(ticker, 0.0)
after = curr_map.get(ticker, 0.0)
if before == 0 and after >= _NEW_POSITION_MIN_USD:
changes.append(KolHoldingChange(
wallet_id=wallet_id, detected_at=now, ticker=ticker,
change_type="new_position", usd_before=0, usd_after=after,
))
elif after == 0 and before >= _CLOSED_MIN_USD:
changes.append(KolHoldingChange(
wallet_id=wallet_id, detected_at=now, ticker=ticker,
change_type="closed", usd_before=before, usd_after=0,
pct_change=-100.0,
))
elif before > 0 and after > 0:
pct = (after - before) / before * 100
if abs(pct) >= _CHANGE_PCT_THRESHOLD:
changes.append(KolHoldingChange(
wallet_id=wallet_id, detected_at=now, ticker=ticker,
change_type="increased" if pct > 0 else "decreased",
usd_before=before, usd_after=after, pct_change=round(pct, 1),
))
return changes
async def _snapshot_wallet(
session: AsyncSession,
wallet: KolWallet,
holdings: list[dict],
total_usd: float,
source: str,
) -> dict:
today = datetime.now(timezone.utc).strftime("%Y-%m-%d")
# Skip if already snapshotted today
existing = (await session.execute(
select(KolHoldingSnapshot).where(
KolHoldingSnapshot.wallet_id == wallet.id,
KolHoldingSnapshot.snapshot_date == today,
)
)).scalar_one_or_none()
if existing:
return {"wallet_id": wallet.id, "status": "already_snapshotted"}
# Previous snapshot for diff
prev_row = (await session.execute(
select(KolHoldingSnapshot)
.where(KolHoldingSnapshot.wallet_id == wallet.id)
.order_by(KolHoldingSnapshot.snapshot_date.desc())
.limit(1)
)).scalar_one_or_none()
prev_holdings = json.loads(prev_row.holdings_json) if prev_row else []
# Store snapshot (strip contract addresses to keep JSON lean)
clean_holdings = [{k: v for k, v in h.items() if k != "contract"} for h in holdings]
snap = KolHoldingSnapshot(
wallet_id=wallet.id,
snapshot_date=today,
holdings_json=json.dumps(clean_holdings, ensure_ascii=False),
total_usd=total_usd,
source=source,
)
session.add(snap)
# Detect and store changes
changes = _diff_holdings(prev_holdings, holdings, wallet.id)
for c in changes:
session.add(c)
logger.info(
"[kol_onchain] %s %s %s $%.0f→$%.0f",
wallet.handle, c.change_type, c.ticker,
c.usd_before or 0, c.usd_after or 0,
)
await session.flush()
return {
"handle": wallet.handle,
"chain": wallet.chain,
"holdings": len(holdings),
"total_usd": total_usd,
"changes": len(changes),
"source": source,
}
# ── Poll entry points ─────────────────────────────────────────────────────────
async def run_hl_poll() -> list[dict]:
"""Poll HL perp positions for all chain='hl' wallets."""
results = []
async with AsyncSessionLocal() as session:
wallets = (await session.execute(
select(KolWallet).where(KolWallet.chain == "hl", KolWallet.active == True)
)).scalars().all()
if not wallets:
logger.info("[kol_onchain] no HL wallets configured")
return []
for wallet in wallets:
try:
holdings, total = await _fetch_hl_positions(wallet.address)
stat = await _snapshot_wallet(session, wallet, holdings, total, "hl")
results.append(stat)
except Exception as e:
logger.warning("[kol_onchain] HL fetch failed %s: %s", wallet.handle, e)
results.append({"handle": wallet.handle, "error": str(e)})
await session.commit()
logger.info("[kol_onchain] HL poll done: %s", results)
return results
async def run_eth_poll() -> list[dict]:
"""Poll Ethereum ERC-20 holdings for all chain='ethereum' wallets.
No-op if ETHERSCAN_API_KEY not set."""
if not settings.etherscan_api_key:
logger.debug("[kol_onchain] ETHERSCAN_API_KEY not set — skipping ETH poll")
return []
results = []
async with AsyncSessionLocal() as session:
wallets = (await session.execute(
select(KolWallet).where(KolWallet.chain == "ethereum", KolWallet.active == True)
)).scalars().all()
if not wallets:
logger.info("[kol_onchain] no Ethereum wallets configured")
return []
# Pre-warm HL prices once for the whole batch
await _get_hl_prices()
for wallet in wallets:
try:
holdings, total = await _fetch_eth_holdings(wallet.address)
stat = await _snapshot_wallet(session, wallet, holdings, total, "etherscan")
results.append(stat)
except Exception as e:
logger.warning("[kol_onchain] ETH fetch failed %s: %s", wallet.handle, e)
results.append({"handle": wallet.handle, "error": str(e)})
await session.commit()
logger.info("[kol_onchain] ETH poll done: %s", results)
return results
async def run_onchain_poll() -> dict:
"""Combined entry point for APScheduler. Runs HL + ETH polls."""
hl = await run_hl_poll()
eth = await run_eth_poll()
return {"hl": hl, "eth": eth}
async def seed_wallets(entries: list[tuple]) -> int:
"""Bulk-insert (handle, chain, address, label, source_url) tuples, skip dupes."""
count = 0
async with AsyncSessionLocal() as session:
for handle, chain, address, label, source_url in entries:
existing = (await session.execute(
select(KolWallet).where(
KolWallet.chain == chain,
KolWallet.address == address.lower(),
)
)).scalar_one_or_none()
if existing:
continue
session.add(KolWallet(
handle=handle, chain=chain,
address=address.lower(),
label=label, source_url=source_url,
))
count += 1
await session.commit()
return count
app/services/kol_substack.py
+383
View File
@@ -0,0 +1,383 @@
"""KOL Substack RSS ingester.
Polls each tracked KOL's Substack feed, dedupes by URL, stores raw post,
then hands off to kol_analysis.extract_kol_signal and writes the result
back onto the same row.
Substack RSS embeds the full post HTML in <content:encoded>. We strip HTML
to plain text before storage + analysis. Hayes posts are typically 50K+
chars of body — the extractor truncates internally.
Daily cadence is plenty (Hayes posts ~monthly, Substack updates feed within
minutes of publish). Call run_substack_poll() from the APScheduler.
"""
from __future__ import annotations
import hashlib
import logging
import re
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime
from typing import Iterable, Optional
import feedparser
import httpx
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.database import AsyncSessionLocal
from app.models import KolPost, utcnow
from app.services import kol_analysis
logger = logging.getLogger(__name__)
# Curated B-tier KOL feeds. Handle is the canonical key. `source` is the
# DB column ("substack" | "podcast" | "blog"); empty defaults to "substack"
# for legacy entries. Twitter-only KOLs come in a separate ingester.
#
# When adding a new feed:
# 1. curl + grep '<item>' to confirm it returns entries.
# 2. Inspect entry summary/content length — AI extraction needs ≥300 chars
# of body per post or it just hallucinates a topic line. (Headlines-only
# feeds like Vitalik's blog need a follow-up HTML fetch, deferred.)
# 3. Add with a sensible handle + display_name.
KOL_FEEDS: list[dict] = [
# ── Substack essayists (long-form thesis pieces) ─────────────────────
{
"handle": "cryptohayes",
"display_name": "Arthur Hayes",
"feed_url": "https://cryptohayes.substack.com/feed",
},
# Placeholder VC (Joel Monegro / Chris Burniske). Token-focused VC, posts
# long-form thesis pieces every 1-3 months that map directly to their
# portfolio bets (Solana staking, L1 monetary premium, etc.).
{
"handle": "placeholder",
"display_name": "Placeholder VC",
"feed_url": "https://www.placeholder.vc/blog?format=rss",
},
# Dragonfly Capital research blog on Medium — free, active (10+ posts).
# dragonfly.xyz/blog/rss.xml returns 0 (paywall). medium.com/dragonfly-research
# is the team's public research arm: airdrops, DeFi, protocol deep-dives.
{
"handle": "dragonfly",
"display_name": "Dragonfly Capital",
"feed_url": "https://medium.com/feed/dragonfly-research",
},
# Andy Constan's Substack is paywalled (RSS returns 0). Keeping for any
# occasional public teaser. Forward Guidance podcast (Blockworks) features
# him weekly but is macro/equities-focused — not crypto-coin-specific enough
# to extract ticker signals from episode descriptions.
{
"handle": "dampedspring",
"display_name": "Damped Spring / Andy Constan",
"feed_url": "https://dampedspring.substack.com/feed",
},
# Nic Carter's Substack is paywalled (RSS returns 0). His Medium feed is
# FREE and active — different URL, same author, real content.
{
"handle": "niccarter",
"display_name": "Nic Carter (Castle Island)",
"feed_url": "https://medium.com/feed/@nic__carter",
},
# Delphi Digital podcast (Buzzsprout) — 478 episodes, active May 2025.
# Public, free. Episode descriptions name specific protocols / tokens with
# thesis framing — good extraction signal. delphidigital.io/feed returns 0.
{
"handle": "delphi",
"display_name": "Delphi Digital (Podcast)",
"feed_url": "https://rss.buzzsprout.com/2609274.rss",
},
# ── Newly added (verified live + active) ─────────────────────────────
# Anthony Pompliano — Pomp Investments. Active monthly+ on macro/crypto.
{
"handle": "pomp",
"display_name": "Anthony Pompliano (Pomp Letter)",
"feed_url": "https://pomp.substack.com/feed",
},
# The DeFi Edge — researcher who writes 1-2 deep dives per month on
# tokens / sectors. Real thesis + position-aware framing.
{
"handle": "thedefiedge",
"display_name": "The DeFi Edge",
"feed_url": "https://thedefiedge.com/feed/",
},
# Eugene Ng Ah Sio — trader/analyst, sporadic but specific.
{
"handle": "eugene",
"display_name": "Eugene Ng Ah Sio",
"feed_url": "https://eugene.substack.com/feed",
},
# ── DeFi journalism (Substack-style RSS) ─────────────────────────────
# The Defiant — Camila Russo's team. DeFi-focused news with frequent
# protocol + token mentions. Free RSS, ~100 entries.
{
"handle": "thedefiant",
"display_name": "The Defiant",
"feed_url": "https://www.thedefiant.io/api/feed",
"source": "blog",
},
# ── Major crypto podcasts (Megaphone / Simplecast RSS) ───────────────
# Show notes are 1-6K chars — long enough for AI to pull out tickers
# and theses. Bootstrap is capped at max_new=20/run so a 600-episode
# backlog spreads across ~30 days.
#
# Empire (Blockworks) — Jason Yanowitz + Santiago Roel Santos. Weekly
# crypto+macro interviews. Show notes name protocols + price calls.
{
"handle": "empire",
"display_name": "Empire Podcast (Blockworks)",
"feed_url": "https://feeds.megaphone.fm/empire",
"source": "podcast",
},
# 0xResearch (Blockworks) — Boccaccio + Dan Smith. Protocol research
# deep-dives, real revenue/usage discussion. Highest signal density of
# the Blockworks shows.
{
"handle": "0xresearch",
"display_name": "0xResearch (Blockworks)",
"feed_url": "https://feeds.megaphone.fm/0xresearch",
"source": "podcast",
},
# Lightspeed (Blockworks) — Mert Mumtaz (Helius CEO) + Garrett Harper.
# Solana ecosystem focus — SOL, JUP, JTO, PUMP, validator economics.
{
"handle": "lightspeed",
"display_name": "Lightspeed (Solana, Blockworks)",
"feed_url": "https://feeds.megaphone.fm/lightspeed",
"source": "podcast",
},
# Unchained — Laura Shin. Long interview format with founders and
# traders. Show notes are 6K+ chars (near-transcript).
{
"handle": "unchained",
"display_name": "Unchained (Laura Shin)",
"feed_url": "https://www.unchainedcrypto.com/feed/",
"source": "podcast",
},
# Bankless podcast — Ryan Sean Adams + David Hoffman. ETH-focused but
# covers all majors. 4K char show notes. Largest crypto-native podcast.
# NOTE: previous feed `simplecast.com/MLdpYXYI` was actually Robert
# Breedlove's "What is Money" show — wrong feed. libsyn is canonical.
{
"handle": "bankless",
"display_name": "Bankless Podcast",
"feed_url": "https://bankless.libsyn.com/rss",
"source": "podcast",
},
# Bell Curve (Multicoin) — Mike Ippolito + Jason Yanowitz + Myles Snider.
# 350 episodes, weekly macro+crypto roundup. Multicoin's portfolio shows
# up frequently (SOL, JTO, JUP, Helium, Render). 1.2K show notes.
{
"handle": "bellcurve",
"display_name": "Bell Curve (Multicoin)",
"feed_url": "https://feeds.megaphone.fm/bellcurve",
"source": "podcast",
},
# The Scoop (The Block) — Frank Chaparro interviews founders + traders.
# 110 episodes, ~700 char show notes. Strong on infrastructure/exchange
# deals (Hyperliquid, Coinbase, Binance dynamics).
{
"handle": "thescoop",
"display_name": "The Scoop (The Block)",
"feed_url": "https://feeds.megaphone.fm/the-scoop",
"source": "podcast",
},
# ── Research newsletters (long-form, high-signal) ────────────────────
# Reflexivity Research — Will Clemente + Sam Rule. On-chain BTC analysis
# and macro pieces. 20 entries, 8K char essays. Concrete on-chain calls.
{
"handle": "reflexivity",
"display_name": "Reflexivity Research (Will Clemente)",
"feed_url": "https://reflexivityresearch.substack.com/feed",
},
# TFTC — Marty Bent's "Bitcoin Brief" newsletter (also a podcast feed).
# 11K char issues, daily Bitcoin + policy. Pure BTC focus but covers
# legislation/macro that moves BTC.
{
"handle": "tftc",
"display_name": "TFTC / Bitcoin Brief (Marty Bent)",
"feed_url": "https://tftc.io/feed",
"source": "blog",
},
]
# Back-compat alias — older imports referenced SUBSTACK_KOLS.
SUBSTACK_KOLS = KOL_FEEDS
_TAG_RE = re.compile(r"<[^>]+>")
_WHITESPACE_RE = re.compile(r"[ \t]+")
_BLANKLINES_RE = re.compile(r"\n{3,}")
def _html_to_text(html: str) -> str:
"""Cheap HTML → text. Good enough for Substack which uses simple markup;
if we ever need real parsing, swap to bs4 (not currently a dep)."""
# Newlines for block-level closes so paragraphs survive
s = re.sub(r"</(p|div|h[1-6]|li|br)\s*>", "\n", html, flags=re.I)
s = re.sub(r"<br\s*/?>", "\n", s, flags=re.I)
s = _TAG_RE.sub("", s)
# HTML entities — feedparser usually decodes these but be safe
s = (s.replace("&amp;", "&").replace("&lt;", "<").replace("&gt;", ">")
.replace("&quot;", '"').replace("&#8217;", "'").replace("&#8220;", '"')
.replace("&#8221;", '"').replace("&nbsp;", " "))
s = _WHITESPACE_RE.sub(" ", s)
s = _BLANKLINES_RE.sub("\n\n", s)
return s.strip()
def _entry_body(entry) -> str:
"""Pull the richest body field available from a feedparser entry."""
if entry.get("content"):
# content is a list of {value, type}
return entry["content"][0].get("value", "") or ""
return entry.get("summary") or entry.get("description") or ""
def _parse_pub(entry) -> Optional[datetime]:
raw = entry.get("published") or entry.get("updated")
if not raw:
return None
try:
dt = parsedate_to_datetime(raw)
# Always normalize to naive UTC. Previously used .astimezone() which
# converts to *local* time → 8-hour skew when server runs in CST.
# Affects: divergence window matching, digest 'since' filter, UI display.
if dt.tzinfo:
dt = dt.astimezone(timezone.utc).replace(tzinfo=None)
return dt
except Exception:
return None
async def _fetch_feed(feed_url: str) -> list:
"""feedparser is sync; do the HTTP fetch through httpx for timeout
control + uniformity with the rest of the codebase, then hand bytes
to feedparser."""
async with httpx.AsyncClient(timeout=20.0, follow_redirects=True) as client:
r = await client.get(feed_url, headers={"User-Agent": "TrumpSignal/1.0 KOL-tracker"})
r.raise_for_status()
parsed = feedparser.parse(r.content)
return list(parsed.entries or [])
async def _ingest_kol(
session: AsyncSession,
kol: dict,
*,
analyze: bool = True,
max_new: int = 20,
) -> dict:
"""Ingest one KOL feed. max_new caps first-run cost for high-volume feeds
(e.g. Delphi podcast has 478 episodes). Subsequent runs only see truly new
entries so the cap rarely triggers after bootstrap."""
handle = kol["handle"]
feed_url = kol["feed_url"]
src = kol.get("source") or "substack" # substack | podcast | blog
stats = {"handle": handle, "source": src,
"new": 0, "skipped": 0, "analyzed": 0, "errors": 0}
try:
entries = await _fetch_feed(feed_url)
except Exception as e:
logger.warning("[kol_substack] fetch failed for %s: %s", handle, e)
stats["errors"] += 1
return stats
for entry in entries:
if stats["new"] >= max_new:
logger.info("[kol_substack] %s hit max_new=%d cap; rest deferred to next run",
handle, max_new)
break
# Podcast feeds (Buzzsprout, etc.) have no <link>; use enclosure URL or entry id.
url = entry.get("link")
if not url:
enclosures = entry.get("enclosures") or []
if enclosures:
url = enclosures[0].get("href")
if not url:
url = entry.get("id") # e.g. "Buzzsprout-19123172"
if not url:
continue
# Dedupe by (source, external_id=url). We also check against the
# legacy "substack" source so podcast/blog re-tags don't double-insert
# entries the old code already wrote.
existing = await session.execute(
select(KolPost).where(
KolPost.source.in_([src, "substack"]),
KolPost.external_id == url,
)
)
row = existing.scalar_one_or_none()
if row is not None:
stats["skipped"] += 1
continue
html = _entry_body(entry)
text = _html_to_text(html)
if not text:
continue
pub = _parse_pub(entry) or utcnow()
title = entry.get("title") or None
body_hash = hashlib.sha256(text.encode("utf-8")).hexdigest()
row = KolPost(
kol_handle=handle,
source=src,
external_id=url,
url=url,
title=title,
published_at=pub,
raw_text=text,
content_hash=body_hash,
)
session.add(row)
await session.flush() # get id for logging
stats["new"] += 1
logger.info("[kol_substack] new post %s id=%s title=%r", handle, row.id, title)
if analyze:
try:
result = await kol_analysis.extract_kol_signal(
handle=handle,
source=src,
title=title,
body=text,
)
if result.get("error"):
stats["errors"] += 1
else:
import json as _json
row.summary = result.get("summary")
row.tickers_json = _json.dumps(result.get("tickers") or [],
ensure_ascii=False)
row.analyzed_at = utcnow()
row.analysis_model = result.get("model")
row.analysis_version = result.get("version")
stats["analyzed"] += 1
except Exception as e:
logger.warning("[kol_substack] analysis failed for %s post %s: %s",
handle, row.id, e)
stats["errors"] += 1
await session.commit()
return stats
async def run_substack_poll(*, analyze: bool = True) -> list[dict]:
"""Poll every configured KOL feed once. Despite the legacy name this now
covers Substack essays, Medium blogs, and major crypto podcasts via RSS.
Returns per-KOL stats."""
results = []
async with AsyncSessionLocal() as session:
for kol in KOL_FEEDS:
stats = await _ingest_kol(session, kol, analyze=analyze)
results.append(stats)
logger.info("[kol_substack] poll done: %s", results)
return results
app/services/market_data.py
+360
View File
@@ -0,0 +1,360 @@
"""
Market data abstraction — pluggable candle sources.
Currently 2 providers:
- Binance : Free public REST, broad coverage of major coins.
- Hyperliquid : SAME venue as execution. Covers HL-native perps that
Binance doesn't list (TRUMP, HYPE, PURR, etc.) and
gives us mark-price-consistent data for those assets.
Routing rule:
- Assets in HL_NATIVE_ASSETS → Hyperliquid (no Binance pair exists)
- Everything else → Binance (better history, no rate friction)
Override per-call by selecting a provider explicitly:
await BinanceCandles().fetch_4h("BTC", days=30)
await HyperliquidCandles().fetch_4h("HYPE", days=30)
# Or auto-route:
src = for_asset("HYPE") # → HyperliquidCandles
candles = await src.fetch_4h("HYPE", days=30)
Normalized candle shape (returned by ALL providers):
{"time_ms": int, "open": float, "high": float, "low": float,
"close": float, "volume": float}
"""
from __future__ import annotations
import logging
from datetime import datetime, timezone
from typing import Protocol
import httpx
from app.config import settings
logger = logging.getLogger(__name__)
# ─── Provider protocol ──────────────────────────────────────────────────────
class CandleSource(Protocol):
name: str
async def fetch_4h(self, asset: str, days: int) -> list[dict]:
"""Last `days` worth of 4-hour candles for `asset`."""
...
async def fetch_1d(self, asset: str, days: int) -> list[dict]:
"""Last `days` daily candles. Used for SMA reclaim / VCP-Daily."""
...
async def fetch_1w(self, asset: str, weeks: int) -> list[dict]:
"""Last `weeks` weekly candles. Used for weekly-RSI reversal."""
...
async def fetch_1m(self, asset: str, start_ms: int, end_ms: int) -> list[dict]:
"""1-minute candles in [start_ms, end_ms]. Paginated internally."""
...
async def fetch_funding(self, asset: str, days: int) -> list[dict]:
"""Funding-rate history. List of {time_ms, rate}. HL-only for now —
Binance provides funding but the cross-venue rate differs, so we
defer to the execution venue (HL)."""
...
# ─── Binance ────────────────────────────────────────────────────────────────
class BinanceCandles:
name = "binance"
base_url = "https://api.binance.com/api/v3/klines"
@staticmethod
def _symbol(asset: str) -> str:
return f"{asset.upper()}USDT"
async def fetch_4h(self, asset: str, days: int) -> list[dict]:
end_ms = int(datetime.now(timezone.utc).timestamp() * 1000)
start_ms = end_ms - days * 24 * 3600 * 1000
async with httpx.AsyncClient(timeout=20) as client:
resp = await client.get(self.base_url, params={
"symbol": self._symbol(asset),
"interval": "4h",
"startTime": start_ms, "endTime": end_ms,
"limit": 1000,
})
resp.raise_for_status()
rows = resp.json()
return [self._normalize(r) for r in rows]
async def fetch_1d(self, asset: str, days: int) -> list[dict]:
return await self._fetch_simple(asset, "1d", days * 24 * 3600 * 1000)
async def fetch_1w(self, asset: str, weeks: int) -> list[dict]:
return await self._fetch_simple(asset, "1w", weeks * 7 * 24 * 3600 * 1000)
async def _fetch_simple(self, asset: str, interval: str, window_ms: int) -> list[dict]:
"""Single-call fetch for intervals where 1000 bars covers the window."""
end_ms = int(datetime.now(timezone.utc).timestamp() * 1000)
start_ms = end_ms - window_ms
async with httpx.AsyncClient(timeout=20) as client:
resp = await client.get(self.base_url, params={
"symbol": self._symbol(asset),
"interval": interval,
"startTime": start_ms, "endTime": end_ms,
"limit": 1000,
})
resp.raise_for_status()
rows = resp.json()
return [self._normalize(r) for r in rows]
async def fetch_1m(self, asset: str, start_ms: int, end_ms: int) -> list[dict]:
# Binance caps each call at 1000 candles — paginate forward.
out: list[dict] = []
cursor = start_ms
async with httpx.AsyncClient(timeout=20) as client:
while cursor < end_ms:
resp = await client.get(self.base_url, params={
"symbol": self._symbol(asset),
"interval": "1m",
"startTime": cursor, "endTime": end_ms,
"limit": 1000,
})
resp.raise_for_status()
chunk = resp.json()
if not chunk:
break
out.extend(self._normalize(r) for r in chunk)
last_open = chunk[-1][0]
if last_open <= cursor:
break
cursor = last_open + 60_000
if len(chunk) < 1000:
break
return out
async def fetch_funding(self, asset: str, days: int) -> list[dict]:
"""Binance perp funding. Format: list of {time_ms, rate}.
Binance returns rate per 8h funding cycle (matches HL convention).
Note: this is Binance's perp funding, NOT HL's. For HL-traded
positions, prefer HyperliquidCandles.fetch_funding().
"""
end_ms = int(datetime.now(timezone.utc).timestamp() * 1000)
start_ms = end_ms - days * 24 * 3600 * 1000
url = "https://fapi.binance.com/fapi/v1/fundingRate"
async with httpx.AsyncClient(timeout=20) as client:
resp = await client.get(url, params={
"symbol": self._symbol(asset),
"startTime": start_ms, "endTime": end_ms,
"limit": 1000,
})
resp.raise_for_status()
rows = resp.json()
return [
{"time_ms": r["fundingTime"], "rate": float(r["fundingRate"])}
for r in rows
]
@staticmethod
def _normalize(row) -> dict:
return {
"time_ms": row[0],
"open": float(row[1]),
"high": float(row[2]),
"low": float(row[3]),
"close": float(row[4]),
"volume": float(row[5]),
}
# ─── Hyperliquid ────────────────────────────────────────────────────────────
class HyperliquidCandles:
"""HL public /info endpoint — same data the HL UI uses.
Endpoint:
POST https://api.hyperliquid.xyz/info
body { "type": "candleSnapshot",
"req": { "coin": "SOL", "interval": "4h",
"startTime": ms, "endTime": ms } }
Returns array of {t, T, s, i, o, c, h, l, v, n} — we normalize to our
standard shape. Useful for HL-native perps Binance doesn't list.
"""
name = "hyperliquid"
def __init__(self, mainnet: bool | None = None):
use_mainnet = settings.hl_mainnet if mainnet is None else mainnet
self.base_url = (
"https://api.hyperliquid.xyz/info" if use_mainnet
else "https://api.hyperliquid-testnet.xyz/info"
)
async def fetch_4h(self, asset: str, days: int) -> list[dict]:
return await self._fetch_window(asset, "4h", days * 24 * 3600 * 1000)
async def fetch_1d(self, asset: str, days: int) -> list[dict]:
return await self._fetch_window(asset, "1d", days * 24 * 3600 * 1000)
async def fetch_1w(self, asset: str, weeks: int) -> list[dict]:
return await self._fetch_window(asset, "1w", weeks * 7 * 24 * 3600 * 1000)
async def fetch_1m(self, asset: str, start_ms: int, end_ms: int) -> list[dict]:
# HL returns ALL candles in the window in one response — no pagination
# needed for typical scanner windows. For multi-day 1m calls HL may
# truncate; the caller should keep windows under ~24h for 1m data.
return await self._fetch(asset.upper(), "1m", start_ms, end_ms)
async def _fetch_window(self, asset: str, interval: str, window_ms: int) -> list[dict]:
end_ms = int(datetime.now(timezone.utc).timestamp() * 1000)
start_ms = end_ms - window_ms
return await self._fetch(asset.upper(), interval, start_ms, end_ms)
async def _fetch(self, coin: str, interval: str, start_ms: int, end_ms: int) -> list[dict]:
async with httpx.AsyncClient(timeout=20) as client:
resp = await client.post(self.base_url, json={
"type": "candleSnapshot",
"req": {
"coin": coin, "interval": interval,
"startTime": start_ms, "endTime": end_ms,
},
})
resp.raise_for_status()
rows = resp.json() or []
return [self._normalize(r) for r in rows]
async def fetch_funding(self, asset: str, days: int) -> list[dict]:
"""HL funding history — HOURLY cadence (1 cycle per hour).
IMPORTANT: HL's /info endpoint caps fundingHistory at 500 rows per
response. 500 rows × 1h cadence = 20.8 days, so a single call CAN'T
return a full 30-day window. We page backwards from `endTime` until
we cover `days` worth of history (or HL runs out of data).
Returns chronologically-sorted list of {time_ms, rate}, deduplicated.
"""
end_ms = int(datetime.now(timezone.utc).timestamp() * 1000)
start_ms = end_ms - days * 24 * 3600 * 1000
cursor = end_ms
collected: dict[int, float] = {} # time_ms → rate (dedup by time)
async with httpx.AsyncClient(timeout=20) as client:
# Safety cap: at most 10 pages (5000 rows ≈ 208 days) — way more
# than any caller could reasonably want, prevents runaway loops
# if HL returns inconsistent data.
for _ in range(10):
if cursor <= start_ms:
break
resp = await client.post(self.base_url, json={
"type": "fundingHistory",
"coin": asset.upper(),
"startTime": start_ms,
"endTime": cursor,
})
resp.raise_for_status()
chunk = resp.json() or []
if not chunk:
break
for r in chunk:
t = r["time"]
if t not in collected:
collected[t] = float(r["fundingRate"])
oldest = min(r["time"] for r in chunk)
if oldest <= start_ms or len(chunk) < 500:
# Either we reached the start of our window, or HL gave
# us a partial page (no more data behind it).
break
cursor = oldest - 1
return [
{"time_ms": t, "rate": rate}
for t, rate in sorted(collected.items())
]
@staticmethod
def _normalize(row: dict) -> dict:
# HL candle keys: t=open_ms, o=open, h/l, c, v
return {
"time_ms": row["t"],
"open": float(row["o"]),
"high": float(row["h"]),
"low": float(row["l"]),
"close": float(row["c"]),
"volume": float(row["v"]),
}
# ─── Routing ────────────────────────────────────────────────────────────────
# HL-native perps that DO NOT have a Binance spot pair. The scanner / backtest
# auto-route these to HL. Add more as you discover them — the list is the
# only place to maintain provider preference.
HL_NATIVE_ASSETS = {
"HYPE", "PURR", "JEFF", "VAPOR",
"PIP", "OMNIX", "PYTH",
# NOTE: TRUMP is now listed on Binance too — using Binance for that gets
# cleaner history. SUI is on both.
}
# Reversal-strategy basket. Major-cap only (no shitcoins), 1 HL-native.
# Used by all three reversal scanners (weekly RSI, SMA reclaim, funding extreme).
REVERSAL_BASKET = ["BTC", "ETH", "SOL", "BNB", "LINK", "AAVE", "DOGE", "HYPE"]
# Bar durations in seconds — used by drop_in_progress_bar() to know whether
# the last candle in a response is still open. Crypto exchanges return the
# CURRENT (in-progress) bar in `klines` responses; for daily/weekly logic
# we usually want the most recent CLOSED bar instead.
INTERVAL_SECONDS = {
"1m": 60, "5m": 300, "15m": 900,
"1h": 3600, "4h": 14400, "8h": 28800,
"1d": 86400, "1w": 604800,
}
def drop_in_progress_bar(candles: list[dict], interval: str) -> list[dict]:
"""Return `candles` minus the last entry if it's still an open bar.
Daily / weekly bars on Binance & HL are returned with `time_ms = open_time`.
The bar's close time is open_time + interval_seconds. If now < close_time,
the bar hasn't closed yet — using its volume/close is misleading (volume
is partial, close is current spot mid-bar).
Use this in scanners that care about CONFIRMED signals (SMA reclaim,
weekly RSI). Use raw candles only when you want to react intra-bar
(rare and usually wrong for position trading).
"""
if not candles:
return candles
dur_s = INTERVAL_SECONDS.get(interval)
if dur_s is None:
return candles # unknown interval — be conservative, return as-is
bar_close_ms = candles[-1]["time_ms"] + dur_s * 1000
now_ms = int(datetime.now(timezone.utc).timestamp() * 1000)
if now_ms < bar_close_ms:
return candles[:-1]
return candles
_BINANCE_SINGLETON = BinanceCandles()
_HL_SINGLETON = HyperliquidCandles()
def for_asset(asset: str) -> CandleSource:
"""Pick the right provider. HL-native → HL, otherwise Binance.
Always returns SOMETHING — caller doesn't need to handle None. If the
asset doesn't actually trade on the chosen venue, the underlying HTTP
call will return an empty list and the caller falls back to its
"no data" path.
"""
return _HL_SINGLETON if asset.upper() in HL_NATIVE_ASSETS else _BINANCE_SINGLETON
app/services/onchain_data.py
+177
View File
@@ -0,0 +1,177 @@
"""On-chain metrics for the BTC bottom-reversal scanner.
Two providers, same interface (fetch_mvrv_z_score / fetch_sth_sopr → list[float]):
BitcoinDataClient — DEFAULT. bitcoin-data.com public API. FREE, no key.
Returns PRE-COMPUTED MVRV-Z and STH-SOPR (we don't
need realized-cap or local math — realized cap is an
on-chain quantity that CANNOT be derived from price
data, so a vendor is unavoidable; this is the free one).
Free tier: 10 requests/HOUR. The scanner runs once a
day and needs 2 calls — well within budget — but we
add a 6-hour in-process cache as a safety net against
restarts / manual re-runs hammering the limit.
GlassnodeClient — Optional paid fallback. Used automatically if
GLASSNODE_API_KEY is set (higher resolution / SLA).
get_onchain_client() picks the right one. btc_bottom_reversal.py only
depends on the interface, never the concrete class.
"""
from __future__ import annotations
import json
import logging
import os
import time
from datetime import datetime, timezone
from typing import Optional
import httpx
from app.config import settings
logger = logging.getLogger(__name__)
# ─── Provider 1: bitcoin-data.com (free, default) ───────────────────────────
BITCOIN_DATA_BASE = "https://bitcoin-data.com/v1"
# Endpoint → (path, json value key). Both return a full daily history list
# of {"d","unixTs",<key>} when called with no query params.
_BD_ENDPOINTS = {
"mvrv_z": ("/mvrv-zscore", "mvrvZscore"),
"sth_sopr": ("/sth-sopr", "sthSopr"),
}
# Fresh-fetch window: don't re-hit the API if the on-disk copy is younger
# than this. The scanner runs daily; 18h keeps it to ~1 fetch/day/metric
# (2 req/day total, vs the 10 req/HOUR free cap).
_BD_FRESH_S = 18 * 3600
# How stale on-disk data may be and still be USABLE when the API is down /
# rate-limited. A daily MVRV-Z / STH-SOPR barely moves over 2 days, so
# serving 48h-old data beats skipping a scan or, worse, acting on nothing.
_BD_STALE_OK_S = 48 * 3600
# Disk cache survives restarts — critical given the tight free rate limit.
_BD_CACHE_DIR = os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(__file__))), ".cache")
_BD_CACHE_FILE = os.path.join(_BD_CACHE_DIR, "onchain_bitcoin_data.json")
def _load_disk_cache() -> dict:
try:
with open(_BD_CACHE_FILE) as f:
return json.load(f)
except (FileNotFoundError, json.JSONDecodeError, OSError):
return {}
def _save_disk_cache(cache: dict) -> None:
try:
os.makedirs(_BD_CACHE_DIR, exist_ok=True)
tmp = _BD_CACHE_FILE + ".tmp"
with open(tmp, "w") as f:
json.dump(cache, f)
os.replace(tmp, _BD_CACHE_FILE) # atomic
except OSError as exc:
logger.warning("onchain disk-cache write failed: %s", exc)
class BitcoinDataClient:
"""Free, key-less. Returns pre-computed metric series (already the
Z-score / SOPR value — no local computation needed). Disk-cached so the
10 req/hour free cap and process restarts can't starve the scanner."""
async def _series(self, metric: str, days: int) -> list[float]:
now = time.time()
cache = _load_disk_cache()
entry = cache.get(metric) # {"ts": epoch, "values": [...]}
age = (now - entry["ts"]) if entry else None
# Fresh enough → no network call at all.
if entry and age is not None and age < _BD_FRESH_S:
vals = entry["values"]
return vals[-days:] if days else vals
path, key = _BD_ENDPOINTS[metric]
try:
async with httpx.AsyncClient(timeout=20) as client:
resp = await client.get(f"{BITCOIN_DATA_BASE}{path}")
if resp.status_code == 429:
raise RuntimeError("rate_limited")
resp.raise_for_status()
rows = resp.json() or []
values = [float(r[key]) for r in rows if r.get(key) is not None]
if not values:
raise RuntimeError("empty_response")
cache[metric] = {"ts": now, "values": values}
_save_disk_cache(cache)
return values[-days:] if days else values
except Exception as exc:
# Network/limit failure: fall back to disk if it's still usable.
if entry and age is not None and age < _BD_STALE_OK_S:
logger.warning("bitcoin-data.com %s fetch failed (%s) — "
"serving disk cache aged %.1fh",
metric, exc, age / 3600)
vals = entry["values"]
return vals[-days:] if days else vals
raise RuntimeError(
f"bitcoin-data.com {metric} unavailable ({exc}) and no usable cache"
) from exc
async def fetch_mvrv_z_score(self, asset: str = "BTC", days: int = 730) -> list[float]:
if asset.upper() != "BTC":
return [] # bitcoin-data.com is BTC-only
return await self._series("mvrv_z", days)
async def fetch_sth_sopr(self, asset: str = "BTC", days: int = 30) -> list[float]:
if asset.upper() != "BTC":
return []
return await self._series("sth_sopr", days)
# ─── Provider 2: Glassnode (paid, optional) ─────────────────────────────────
GLASSNODE_BASE_URL = "https://api.glassnode.com/v1/metrics"
class GlassnodeClient:
def __init__(self, api_key: Optional[str] = None):
self.api_key = api_key if api_key is not None else settings.glassnode_api_key
async def _get_series(self, path: str, asset: str, days: int) -> list[dict]:
if not self.api_key:
raise RuntimeError("GLASSNODE_API_KEY is not configured")
end = int(datetime.now(timezone.utc).timestamp())
start = end - days * 86_400
async with httpx.AsyncClient(timeout=20) as client:
resp = await client.get(
f"{GLASSNODE_BASE_URL}{path}",
params={"a": asset, "api_key": self.api_key,
"s": start, "u": end, "i": "24h"},
)
resp.raise_for_status()
return resp.json() or []
async def fetch_mvrv_z_score(self, asset: str = "BTC", days: int = 730) -> list[float]:
rows = await self._get_series("/market/mvrv_z_score", asset, days)
return [float(r["v"]) for r in rows if r.get("v") is not None]
async def fetch_sth_sopr(self, asset: str = "BTC", days: int = 30) -> list[float]:
rows = await self._get_series("/indicators/sopr_less_155", asset, days)
return [float(r["v"]) for r in rows if r.get("v") is not None]
# ─── Factory ────────────────────────────────────────────────────────────────
def get_onchain_client():
"""Glassnode if a key is configured (paid, higher SLA), else the free
bitcoin-data.com client. Both expose the same async interface."""
if settings.glassnode_api_key:
logger.info("On-chain provider: Glassnode (paid key configured)")
return GlassnodeClient()
logger.info("On-chain provider: bitcoin-data.com (free, no key)")
return BitcoinDataClient()
app/services/reanalyze.py
+179
View File
@@ -0,0 +1,179 @@
"""
Batch re-analyzer — runs Claude analysis on posts that were backfilled
without AI scoring (ai_confidence == 0).
Designed to run as a one-shot background task. Processes posts oldest-first,
1 per second, so 479 posts ≈ 8 minutes. Progress is logged at every batch.
Usage (via dev endpoint POST /api/dev/reanalyze):
curl -X POST "http://localhost:8000/api/dev/reanalyze?limit=500&dry_run=false"
"""
import asyncio
import logging
from datetime import datetime, timezone
from typing import Optional
from sqlalchemy import select
logger = logging.getLogger(__name__)
# Global so the HTTP endpoint can check progress without storing state externally.
_reanalyze_state: dict = {
"running": False,
"processed": 0,
"updated": 0,
"errors": 0,
"total": 0,
"started_at": None,
"finished_at": None,
}
def get_state() -> dict:
return dict(_reanalyze_state)
async def reanalyze_unscored(
db_session_factory,
limit: int = 500,
dry_run: bool = False,
delay_secs: float = 1.0,
legacy_signals: bool = False,
model: Optional[str] = None,
) -> dict:
"""
Find all posts with ai_confidence == 0, run analyze_post on each,
and write results back to the DB.
Args:
limit: Maximum number of posts to process in this run.
dry_run: If True, analyze but do NOT write to DB (for testing).
delay_secs: Pause between API calls to avoid rate-limiting.
Returns:
Summary dict with processed/updated/errors counts.
"""
global _reanalyze_state
if _reanalyze_state["running"]:
logger.warning("reanalyze already in progress, skipping")
return _reanalyze_state
_reanalyze_state = {
"running": True,
"processed": 0,
"updated": 0,
"errors": 0,
"total": 0,
"started_at": datetime.now(timezone.utc).isoformat(),
"finished_at": None,
}
from app.models import Post
from app.services.analysis import analyze_post
try:
# ── Fetch posts needing (re-)analysis ────────────────────────────
# Two cases:
# 1. Never analyzed (ai_confidence == 0)
# 2. Analyzed before target_asset column was added (has buy/short
# signal but target_asset IS NULL). Pass legacy_signals=True
# to target ONLY these, bypassing unscored posts.
from sqlalchemy import or_, and_
if legacy_signals:
condition = and_(
Post.signal.in_(["buy", "short"]),
Post.target_asset == None, # noqa: E711
)
else:
# Use analysis_version IS NULL (not ai_confidence==0) so that
# posts already analyzed as "hold" (conf=0, version set) are not
# re-run every time. Only truly unanalyzed posts are targeted.
condition = or_(
Post.analysis_version == None, # noqa: E711
and_(
Post.signal.in_(["buy", "short"]),
Post.target_asset == None, # noqa: E711
)
)
async with db_session_factory() as db:
rows = await db.execute(
select(Post.id, Post.text)
.where(condition)
.order_by(Post.published_at.asc())
.limit(limit)
)
posts_to_do = rows.all() # list of (id, text) tuples
_reanalyze_state["total"] = len(posts_to_do)
logger.info("Reanalyze: %d posts queued (limit=%d, dry_run=%s)",
len(posts_to_do), limit, dry_run)
if not posts_to_do:
_reanalyze_state["running"] = False
_reanalyze_state["finished_at"] = datetime.now(timezone.utc).isoformat()
return _reanalyze_state
# ── Process each post ─────────────────────────────────────────────
for post_id, text in posts_to_do:
_reanalyze_state["processed"] += 1
try:
analysis = await analyze_post(text, model=model) # caller decides quality vs speed
if not dry_run:
async with db_session_factory() as db:
result = await db.execute(
select(Post).where(Post.id == post_id)
)
post = result.scalar_one_or_none()
if post is None:
continue
post.sentiment = analysis["sentiment"]
post.signal = analysis.get("signal")
post.ai_confidence = analysis["confidence"]
post.ai_reasoning = analysis.get("reasoning")
post.prefilter_reason = analysis.get("prefilter_reason")
post.analysis_version = analysis.get("analysis_version")
post.relevant = analysis["relevant"]
post.price_impact_asset = analysis["asset"] if analysis["relevant"] else None
post.target_asset = analysis.get("target_asset")
post.category = analysis.get("category")
post.expected_move_pct = analysis.get("expected_move_pct")
await db.commit()
_reanalyze_state["updated"] += 1
if _reanalyze_state["processed"] % 20 == 0:
logger.info(
"Reanalyze progress: %d/%d processed, %d updated, %d errors",
_reanalyze_state["processed"],
_reanalyze_state["total"],
_reanalyze_state["updated"],
_reanalyze_state["errors"],
)
except Exception as exc:
_reanalyze_state["errors"] += 1
logger.error("Reanalyze error on post %d: %s", post_id, exc)
# Rate-limit: wait between API calls
if delay_secs > 0:
await asyncio.sleep(delay_secs)
except Exception as exc:
logger.error("Reanalyze fatal error: %s", exc)
finally:
_reanalyze_state["running"] = False
_reanalyze_state["finished_at"] = datetime.now(timezone.utc).isoformat()
logger.info(
"Reanalyze finished: %d processed, %d updated, %d errors",
_reanalyze_state["processed"],
_reanalyze_state["updated"],
_reanalyze_state["errors"],
)
return _reanalyze_state
app/services/reconciler.py
+191
View File
@@ -0,0 +1,191 @@
"""
Hyperliquid <-> DB state reconciliation (P1.2).
Runs every RECONCILE_INTERVAL_SECONDS. For each subscription with a saved HL
API key (and NOT in paper mode), fetches actual open positions from HL and
compares them to BotTrade rows with closed_at IS NULL.
Two drift cases:
1. DB has open trade, HL has no matching position
→ User closed manually on HL UI, or HL liquidated, or our open_position
call failed silently after writing the row. Either way the row is stale.
Action: mark the DB row closed with pnl_usd=NULL, reason="closed_externally".
2. HL has open position, DB has no matching open trade
→ A trade exists on HL that we didn't open (legacy position, or stale
wallet shared across systems). We can't safely manage it — log a
warning + WS broadcast for the user to handle manually.
NOTE: this does NOT close stray HL positions. Auto-closing positions we didn't
open is exactly the kind of "helpful" behaviour that costs users money. We
report, the human decides.
Cost note:
HL's user_state endpoint is cheap. 1 call per active subscriber per
RECONCILE_INTERVAL. With 10 subscribers at 60s interval that's 600 calls/hr,
well inside HL's free tier.
"""
from __future__ import annotations
import asyncio
import logging
from datetime import datetime, timezone
from typing import Optional
from sqlalchemy import select, update
from app.config import settings
from app.database import AsyncSessionLocal
from app.models import BotTrade, Subscription
from app.services.crypto import decrypt_api_key
from app.services.hyperliquid import HyperliquidTrader
logger = logging.getLogger(__name__)
RECONCILE_INTERVAL_SECONDS = 60
# ─── Single-wallet reconcile ────────────────────────────────────────────────
async def _reconcile_wallet(sub: Subscription) -> dict:
"""Compare HL state vs DB state for one subscription.
Returns a small summary dict for logging / WS broadcast. Never raises —
network / HL errors are logged and counted but don't halt the cycle.
"""
summary = {
"wallet": sub.wallet_address, "orphan_hl": [], "orphan_db": [],
"marked_closed": 0, "error": None,
}
# Paper-mode subs aren't on HL at all — nothing to reconcile.
if sub.paper_mode:
return summary
try:
api_key = decrypt_api_key(sub.hl_api_key)
except Exception as exc:
summary["error"] = f"decrypt_failed: {exc}"
return summary
try:
trader = HyperliquidTrader(
api_private_key=api_key,
account_address=sub.wallet_address,
leverage=sub.leverage,
mainnet=settings.hl_mainnet,
)
hl_positions = await trader.get_open_positions()
except Exception as exc:
summary["error"] = f"hl_query_failed: {exc}"
return summary
hl_assets_open = {p["coin"]: p for p in hl_positions}
async with AsyncSessionLocal() as db:
# All DB rows we believe are still open for this wallet.
rows = await db.execute(
select(BotTrade).where(
BotTrade.wallet_address == sub.wallet_address,
BotTrade.closed_at.is_(None),
)
)
db_open_trades = rows.scalars().all()
db_assets_open = {t.asset: t for t in db_open_trades}
# Case 1: DB open trade, HL has nothing. Stale row → mark closed.
now_naive = datetime.now(timezone.utc).replace(tzinfo=None)
for asset, trade in db_assets_open.items():
# Paper trades never have HL positions; they're managed by the
# normal close path. Skip here.
if trade.hl_order_id == "paper":
continue
if asset not in hl_assets_open:
# Preserve any de-risk PnL ALREADY banked on this trade — the
# open remainder's PnL is unknown (closed externally) but the
# banked slice is real money on HL; setting pnl_usd=None would
# silently undercount realized PnL in analytics / KPIs.
banked = trade.realized_partial_pnl_usd or 0.0
preserved_pnl = round(banked, 2) if banked else None
# Atomic claim — same conditional UPDATE pattern as close_and_finalize.
claimed = await db.execute(
update(BotTrade)
.where(BotTrade.id == trade.id)
.where(BotTrade.closed_at.is_(None))
.values(closed_at=now_naive, pnl_usd=preserved_pnl,
remaining_fraction=0.0)
)
if claimed.rowcount > 0:
summary["marked_closed"] += 1
summary["orphan_db"].append({"asset": asset, "trade_id": trade.id})
# Stop the TP/SL watcher for this trade
from app.services.tp_sl_monitor import unregister
unregister(trade.id)
logger.warning("Reconcile: trade %d (%s %s) closed externally — marked.",
trade.id, trade.side, asset)
await db.commit()
# Case 2: HL has position we don't know about. Report only — don't auto-trade.
for asset, hl_pos in hl_assets_open.items():
if asset not in db_assets_open:
summary["orphan_hl"].append({
"asset": asset, "szi": hl_pos["szi"],
"entry": hl_pos["entry_px"], "uPnL": hl_pos["unrealized_pnl"],
})
logger.warning("Reconcile: %s has unmanaged HL position %s (szi=%s, entry=%s)",
sub.wallet_address, asset, hl_pos["szi"], hl_pos["entry_px"])
return summary
# ─── Periodic top-level task ────────────────────────────────────────────────
async def reconcile_all_once() -> None:
"""One scan over every subscription. Wired into the APScheduler at startup."""
async with AsyncSessionLocal() as db:
rows = await db.execute(
select(Subscription).where(
Subscription.active == True, # noqa: E712
Subscription.hl_api_key.is_not(None),
)
)
subs = rows.scalars().all()
if not subs:
return
# Snapshot the fields we read so we don't hold the session across HL calls.
snapshots = list(subs)
n_changed = 0
n_orphans = 0
n_errors = 0
for sub in snapshots:
summary = await _reconcile_wallet(sub)
n_changed += summary["marked_closed"]
n_orphans += len(summary["orphan_hl"])
if summary["error"]:
n_errors += 1
# Broadcast significant events so the UI shows a warning banner.
if summary["marked_closed"] or summary["orphan_hl"]:
try:
from app.ws.manager import manager
await manager.broadcast({
"type": "reconcile_drift",
"wallet": summary["wallet"],
"orphan_hl": summary["orphan_hl"],
"marked_closed": summary["marked_closed"],
"at": datetime.now(timezone.utc).isoformat() + "Z",
})
except Exception as exc:
logger.warning("reconcile WS broadcast failed: %s", exc)
if n_changed or n_orphans or n_errors:
logger.info("Reconcile cycle: %d subs scanned, %d trades marked closed, %d HL orphans, %d errors",
len(snapshots), n_changed, n_orphans, n_errors)
app/services/recovery.py
+76 -7
View File
@@ -11,15 +11,21 @@ from datetime import datetime, timezone
from sqlalchemy import select
from app.database import AsyncSessionLocal
from app.models import BotTrade, Subscription
from app.models import BotTrade, Post, Subscription
logger = logging.getLogger(__name__)
async def rehydrate_open_trades() -> None:
# Imported locally to avoid circular imports at module load
from app.services.bot_engine import MAX_HOLD_SECONDS, close_and_finalize
from app.services.bot_engine import close_and_finalize
from app.services.crypto import decrypt_api_key
from app.services.signal_categories import (
get_stop_ladder as _get_stop_ladder,
sys2_derisk_ladder as _sys2_derisk_ladder,
sys2_addon_ladder as _sys2_addon_ladder,
sys2_peak_trail as _sys2_peak_trail,
)
from app.services.tp_sl_monitor import register_trade
async with AsyncSessionLocal() as db:
@@ -53,7 +59,29 @@ async def rehydrate_open_trades() -> None:
# Fall back to current Subscription only for legacy rows (pre-migration 005).
trade_leverage = t.leverage if t.leverage is not None else sub.leverage
# Re-register TP/SL watcher
# Re-register from the trade's FROZEN exit profile (eff_* columns),
# NOT the live Subscription. The trade may be a 90-day System-2
# reversal; using sub.* would rehydrate it with the user's Trump
# stop (1.5%) and 7-day max-hold — silently rewriting its risk on
# every restart. eff_* is stamped at open and never changes.
#
# Legacy rows (opened before this migration) have NULL eff_* —
# fall back to the Subscription so they still get *some* watcher.
eff_sl = t.eff_stop_loss_pct if t.eff_stop_loss_pct is not None else sub.stop_loss_pct
eff_tp = t.eff_take_profit_pct # may legitimately be None (sys2 pure-trail)
eff_tr = t.eff_trailing_stop_pct if t.eff_trailing_stop_pct is not None else sub.trailing_stop_pct
eff_tra = t.eff_trailing_activate_pct if t.eff_trailing_activate_pct is not None else sub.trailing_activate_at_pct
eff_mh = t.eff_max_hold_hours if t.eff_max_hold_hours is not None else (sub.max_hold_hours or 168)
# NOTE: peak_gain_pct is now RESTORED from the row (throttled
# persistence) so a pyramided / in-profit System-2 trade keeps its
# regime across restarts. min_hold still resets on rehydrate, but
# it only matters in the first 30 min of a Trump trade; a restart
# inside that window is rare and the downside (TP can fire early)
# is bounded.
post_res = await db.execute(
select(Post).where(Post.id == t.trigger_post_id)
)
trigger_post = post_res.scalar_one_or_none()
register_trade(
trade_id=t.id,
wallet=t.wallet_address,
@@ -62,14 +90,55 @@ async def rehydrate_open_trades() -> None:
asset=t.asset,
side=t.side,
entry_price=t.entry_price,
take_profit_pct=sub.take_profit_pct,
stop_loss_pct=sub.stop_loss_pct,
take_profit_pct=eff_tp,
stop_loss_pct=eff_sl,
trailing_stop_pct=eff_tr,
trailing_activate_at_pct=eff_tra,
invalidation=t.eff_invalidation,
invalidation_price=(
t.eff_invalidation_price
if t.eff_invalidation_price is not None
else (trigger_post.invalidation_price if trigger_post else None)
),
min_hold_until_ts=t.eff_min_hold_until_ts,
stop_ladder=_get_stop_ladder(
trigger_post.category if trigger_post else None
),
# Restore staged de-risk: rebuild the ladder from the trade's
# FROZEN leverage and skip steps already executed before the
# restart (derisk_steps_done is persisted on the row).
# Restore staged de-risk/pyramid/peak-trail with the trade's
# FROZEN risk mode + leverage, skipping steps already executed
# before the restart (persisted on the row).
derisk_ladder=(
_sys2_derisk_ladder(t.leverage or 1, t.sys2_mode)
if _get_stop_ladder(trigger_post.category if trigger_post else None)
else None
),
derisk_done=(t.derisk_steps_done or 0),
addon_ladder=(
_sys2_addon_ladder(t.sys2_mode)
if _get_stop_ladder(trigger_post.category if trigger_post else None)
else None
),
addon_done=(t.addon_steps_done or 0),
# Restore the monotonic peak so a pyramided / in-profit trade
# doesn't fall back to the underwater de-risk regime on restart.
initial_peak=(t.peak_gain_pct or 0.0),
peak_trail=(
_sys2_peak_trail(t.sys2_mode)
if _get_stop_ladder(trigger_post.category if trigger_post else None)
else None
),
grow_mode=bool(getattr(t, "grow_mode", False)),
)
# Compute remaining hold; if already exceeded, close immediately
# Remaining hold from the trade's FROZEN max_hold (e.g. 2160h for
# an sma_reclaim), not the user's Trump setting.
opened_aware = t.opened_at.replace(tzinfo=timezone.utc)
elapsed = (now - opened_aware).total_seconds()
remaining = MAX_HOLD_SECONDS - elapsed
max_hold_seconds = int(eff_mh) * 3600
remaining = max_hold_seconds - elapsed
from app.services.bot_engine import _background_tasks
app/services/regime_filter.py
+221
View File
@@ -0,0 +1,221 @@
"""
Regime filter — gates entries to convex setups only.
The Trump-signal bot was originally designed to fire on every high-confidence
post. For a convex (asymmetric-payoff) strategy that's wrong: you want to
trade RARELY but with very high quality. This module decides "is the market
in a regime where this signal has a real shot at a big move?"
Four gates, in order of likely impact:
1. Volatility contraction — only trade assets that have been quiet recently.
A coin that already moved 10% today is not setting up; it has set up.
2. Recent-move filter — don't chase. If the asset moved >5% in the last 24h,
stand down. The fat tail comes AFTER consolidation, not after a run.
3. Thin-liquidity hours — UTC 03–09 is Asia thin book. Wider spreads, more
noise stop-outs, fewer institutional bids.
4. BTC counter-trend — if BTC is moving hard against the signal direction,
alts get dragged. Don't fight beta.
Each gate returns a bool + reason. The combiner short-circuits on the first
failure for clear logging.
All thresholds are intentionally configurable in one place. Tune them in
backtest, don't sprinkle magic numbers across the codebase.
"""
from __future__ import annotations
import logging
import statistics
from datetime import datetime, timezone
from typing import Optional, Tuple
from app.services.price_store import price_store
logger = logging.getLogger(__name__)
# ─── Tunables (one place to change) ─────────────────────────────────────────
ATR_RECENT_WINDOW_HOURS = 24 # "recent" volatility window
ATR_BASELINE_WINDOW_HOURS = 168 # 7-day baseline to compare against
ATR_CONTRACTION_RATIO = 0.7 # recent must be ≤ 70% of baseline → "contracted"
RECENT_MOVE_WINDOW_HOURS = 24
RECENT_MOVE_MAX_PCT = 5.0 # block entries if asset already moved >5%
BTC_COUNTER_TREND_HOURS = 6
BTC_COUNTER_TREND_PCT = 3.0 # block if BTC moved >3% against signal
THIN_LIQUIDITY_HOURS_UTC = range(3, 9) # 03:00–08:59 UTC
# ─── Individual gates ───────────────────────────────────────────────────────
def _atr_proxy(asset: str, hours: int) -> Optional[float]:
"""Average true-range proxy using 1-min candles in price_store.
Returns the mean of (high - low) / close over the window, or None if
we don't have enough data yet. Cheap; recomputes on every call.
"""
asset = asset.upper()
buf = price_store._candles.get(asset)
if not buf:
return None
n_minutes = hours * 60
candles = list(buf)[-n_minutes:]
if len(candles) < n_minutes // 2: # need at least half the window populated
return None
ranges = [
(c["high"] - c["low"]) / c["close"]
for c in candles
if c["close"] > 0
]
if not ranges:
return None
return statistics.fmean(ranges)
def is_volatility_contracted(asset: str) -> Tuple[bool, str]:
"""True if recent ATR ≤ ATR_CONTRACTION_RATIO × baseline ATR.
Returns (ok, reason). On insufficient data we PASS (None means "we don't
know, don't block") to avoid breaking the bot on cold-start. Document this
asymmetry in the regime log.
"""
recent = _atr_proxy(asset, ATR_RECENT_WINDOW_HOURS)
base = _atr_proxy(asset, ATR_BASELINE_WINDOW_HOURS)
if recent is None or base is None or base == 0:
return True, f"vol_contraction: insufficient data ({asset}), passing"
ratio = recent / base
ok = ratio <= ATR_CONTRACTION_RATIO
return ok, f"vol_contraction: ratio={ratio:.2f} (threshold {ATR_CONTRACTION_RATIO})"
def is_not_already_moving(asset: str) -> Tuple[bool, str]:
"""Block entries when the asset already ran in the last 24h."""
asset = asset.upper()
buf = price_store._candles.get(asset)
if not buf or len(buf) < RECENT_MOVE_WINDOW_HOURS * 60:
return True, f"recent_move: insufficient data ({asset}), passing"
candles = list(buf)[-RECENT_MOVE_WINDOW_HOURS * 60:]
first_close = candles[0]["close"]
last_close = candles[-1]["close"]
if first_close == 0:
return True, "recent_move: bad first close, passing"
move_pct = abs(last_close - first_close) / first_close * 100
ok = move_pct <= RECENT_MOVE_MAX_PCT
return ok, f"recent_move: {move_pct:.2f}% in {RECENT_MOVE_WINDOW_HOURS}h (max {RECENT_MOVE_MAX_PCT}%)"
def is_not_thin_liquidity_hour() -> Tuple[bool, str]:
"""Block during Asia thin-book hours (UTC 03–09)."""
hour_utc = datetime.now(timezone.utc).hour
ok = hour_utc not in THIN_LIQUIDITY_HOURS_UTC
return ok, f"thin_liquidity: UTC {hour_utc:02d}h"
def is_btc_not_countertrend(side: str) -> Tuple[bool, str]:
"""Block if BTC is moving hard against the signal direction.
side = 'long' or 'short'. Long signal + BTC dumping > 3% → block.
"""
buf = price_store._candles.get("BTC")
if not buf or len(buf) < BTC_COUNTER_TREND_HOURS * 60:
return True, "btc_trend: insufficient BTC data, passing"
candles = list(buf)[-BTC_COUNTER_TREND_HOURS * 60:]
first_close = candles[0]["close"]
last_close = candles[-1]["close"]
if first_close == 0:
return True, "btc_trend: bad first close, passing"
btc_change_pct = (last_close - first_close) / first_close * 100
# Long signal but BTC dumping → bad. Short signal but BTC pumping → bad.
countertrend = (
(side == "long" and btc_change_pct < -BTC_COUNTER_TREND_PCT) or
(side == "short" and btc_change_pct > BTC_COUNTER_TREND_PCT)
)
ok = not countertrend
return ok, f"btc_trend: BTC {btc_change_pct:+.2f}% in {BTC_COUNTER_TREND_HOURS}h vs {side}"
# ─── Master combiner ────────────────────────────────────────────────────────
def passes_regime_filter(asset: str, side: str) -> Tuple[bool, list[str]]:
"""Run every gate; return (passed_all, list_of_reasons).
Gates are evaluated in cheap-to-expensive order so we short-circuit on
the first failure when iterating callers want fast-rejection paths.
"""
reasons: list[str] = []
checks = [
is_not_thin_liquidity_hour(),
is_not_already_moving(asset),
is_volatility_contracted(asset),
is_btc_not_countertrend(side),
]
passed_all = True
for ok, reason in checks:
reasons.append(("✓ " if ok else "✗ ") + reason)
if not ok:
passed_all = False
return passed_all, reasons
# ─── Position sizing (C) ────────────────────────────────────────────────────
def calculate_size_multiplier(
ai_confidence: int,
asset: str,
side: str,
) -> float:
"""Score-based multiplier on `position_size_usd`.
Each green box adds a point; more points → bigger bet. This is the
Druckenmiller "when you have conviction, swing for the fences" idea
expressed numerically. Capped at 4× so a single misread doesn't blow
the daily budget.
Scoring:
+1 if AI confidence ≥ 90
+1 if volatility is contracted (we're at the start of the spring, not the end)
+1 if asset hasn't already run today (still room to go)
+1 if BTC is supportive of the direction (beta in our favour)
"""
score = 0
if ai_confidence >= 90:
score += 1
vc_ok, _ = is_volatility_contracted(asset)
if vc_ok:
score += 1
nm_ok, _ = is_not_already_moving(asset)
if nm_ok:
score += 1
bt_ok, _ = is_btc_not_countertrend(side)
if bt_ok:
score += 1
multipliers = {0: 0.5, 1: 1.0, 2: 1.5, 3: 2.5, 4: 4.0}
return multipliers[score]
# ─── Manual-window helper (D) ───────────────────────────────────────────────
def is_in_manual_window(manual_window_until: Optional[datetime]) -> bool:
"""True if a manual-enable timestamp is set and still in the future.
`manual_window_until` is stored as naive-UTC in DB.
"""
if manual_window_until is None:
return False
now_naive = datetime.now(timezone.utc).replace(tzinfo=None)
return manual_window_until > now_naive
app/services/scanner_state.py
+175
View File
@@ -0,0 +1,175 @@
"""
Scanner runtime state + control plane.
One module to answer: "what scanners are alive, when did each last fire, and
how do I turn one off without restarting the server?"
Why this exists (3 problems it solves at once):
1. KILL SWITCH — Production safety. If a scanner is misbehaving (false
fires, hammering an API, leaking memory) the operator needs to disable
it WITHOUT redeploying. Per-scanner toggle + "stop all" both required.
2. COOLDOWN PERSISTENCE — Previously each scanner kept _last_signal_at as
a process-local dict. A backend restart wiped that dict, so a scanner
that fired yesterday would happily re-fire today. Now cooldown is read
from the posts table (source-of-truth) via `last_signal_at()`.
3. OBSERVABILITY — Operator needs to see "is my RSI scanner alive?". The
in-memory state tracker records every run (success / fire / error) so
a GET /api/scanners endpoint can show the whole fleet at a glance.
State is intentionally process-local (no DB writes). After restart all
scanners come back ENABLED — same as the rest of the system. Cooldowns
survive restart because they're DB-derived. If you want a "permanently
disabled" scanner, comment it out of main.py instead.
"""
from __future__ import annotations
import logging
from dataclasses import dataclass, field, asdict
from datetime import datetime, timedelta, timezone
from typing import Optional
logger = logging.getLogger(__name__)
# ─── State container ────────────────────────────────────────────────────────
@dataclass
class ScannerState:
name: str
enabled: bool = True
last_run_at: Optional[datetime] = None
last_status: str = "never_ran" # never_ran | ok | fired | error
last_message: Optional[str] = None
last_fired_at: Optional[datetime] = None
consecutive_errors: int = 0
total_runs: int = 0
total_fires: int = 0
def to_dict(self) -> dict:
d = asdict(self)
for k in ("last_run_at", "last_fired_at"):
if d[k] is not None:
# last_run_at is tz-aware (datetime.now(timezone.utc)), so
# isoformat() already emits "...+00:00". Appending "Z" would
# produce the invalid "...+00:00Z" which JS Date can't parse
# (renders as "NaNd ago" in the UI). Just use isoformat().
d[k] = d[k].isoformat()
return d
_STATES: dict[str, ScannerState] = {}
# ─── Registration (called once per scanner at import time) ──────────────────
def register(name: str) -> ScannerState:
if name not in _STATES:
_STATES[name] = ScannerState(name=name)
return _STATES[name]
# ─── Toggle controls ────────────────────────────────────────────────────────
def is_enabled(name: str) -> bool:
"""Return False ONLY if explicitly disabled. Unknown scanners default
to enabled — a scanner shouldn't silently refuse to run because its
state wasn't registered (defensive)."""
s = _STATES.get(name)
return s.enabled if s else True
def set_enabled(name: str, enabled: bool) -> Optional[ScannerState]:
s = _STATES.get(name)
if s is None:
return None
s.enabled = enabled
logger.info("Scanner %s %s", name, "ENABLED" if enabled else "DISABLED")
return s
def disable_all() -> int:
"""Kill switch. Returns count of scanners that were running and got
flipped off. Already-disabled scanners are skipped."""
n = 0
for s in _STATES.values():
if s.enabled:
s.enabled = False
n += 1
if n:
logger.warning("Scanner kill switch: disabled %d scanners", n)
return n
def enable_all() -> int:
n = 0
for s in _STATES.values():
if not s.enabled:
s.enabled = True
n += 1
return n
def get_all() -> list[ScannerState]:
return list(_STATES.values())
# ─── Per-run telemetry (called by scanners after each scan) ─────────────────
def record_run(name: str, status: str, message: Optional[str] = None) -> None:
"""status: 'ok' (ran, no signal), 'fired' (emitted a signal), 'error'."""
s = register(name)
s.last_run_at = datetime.now(timezone.utc)
s.last_status = status
s.last_message = message[:240] if message else None
s.total_runs += 1
if status == "fired":
s.total_fires += 1
s.last_fired_at = s.last_run_at
s.consecutive_errors = 0
elif status == "error":
s.consecutive_errors += 1
else:
s.consecutive_errors = 0
# ─── DB-backed cooldown (survives restart) ──────────────────────────────────
async def last_signal_at(source: str, target_asset: str) -> Optional[datetime]:
"""Most recent post with the given source+target_asset. Returns naive UTC.
Used by scanners INSTEAD of in-memory cooldown trackers. The DB is the
authoritative record of "did this scanner already fire?" — surviving
restarts, multi-process deploys, and even DB migrations.
"""
from sqlalchemy import select, func
from app.database import AsyncSessionLocal
from app.models import Post
async with AsyncSessionLocal() as db:
result = await db.execute(
select(func.max(Post.published_at)).where(
Post.source == source,
Post.target_asset == target_asset.upper(),
)
)
ts = result.scalar_one_or_none()
return ts
async def in_cooldown(source: str, target_asset: str, cooldown_days: int) -> bool:
"""Convenience wrapper. True iff we fired the same {source, asset} within
the last `cooldown_days`."""
last = await last_signal_at(source, target_asset)
if last is None:
return False
age = datetime.now(timezone.utc).replace(tzinfo=None) - last
return age < timedelta(days=cooldown_days)
app/services/scanners/__init__.py
View File
app/services/scanners/_archive/README.md
+12
View File
@@ -0,0 +1,12 @@
# Archived scanners (not in the live path)
These were generic System-2 scanners superseded by the focused
`btc_bottom_reversal` state machine. Nothing imports them; they are NOT
scheduled and do NOT register with scanner_state. Kept for reference/history.
- vcp_breakout.py — volatility-contraction breakout (bidirectional)
- weekly_rsi_reversal.py — weekly RSI extreme + recovery (bidirectional)
To revive: re-add an APScheduler job in app/main.py and a
scanner_state.register() call, and ensure the source tag is in
signal_categories.SYSTEM_2_SOURCES.
app/services/scanners/_archive/vcp_breakout.py
+251
View File
@@ -0,0 +1,251 @@
"""
VCP-style breakout scanner — example of an external signal module.
What this scans for (Minervini-light, adapted for crypto perps):
1. Volatility contraction: 7-day ATR ≤ 50% of 30-day ATR
(the price has been getting tighter — supply is drying up)
2. Breakout confirmation: current 4h close > max(prior 30-day highs)
(a real break of the range, not just touching the top)
3. Volume confirmation: last 24h volume ≥ 1.5× 30-day average
(institutional participation, not retail noise)
4. Cooldown: no signal for the same asset in the past 48h
(avoid stacking)
When all 4 conditions hit, the scanner POSTs to /api/signals/ingest. From
there the trade goes through the same convex-strategy pipeline as Trump
signals — regime filter, sizing, trailing stop, all of it.
Why this module exists:
Trump signals have 1-2 true catalysts per month. Adding a TA-based source
that finds setups continuously means the bot has SOMETHING to act on most
weeks, instead of waiting for geopolitical fireworks. The user's hypothesis
is that consolidation breakouts have asymmetric upside; the existing
trailing-stop logic is purpose-built to capture that.
Tunables (don't hardcode in caller — change them HERE if you want to test):
ATR_RECENT_DAYS, ATR_BASELINE_DAYS, ATR_RATIO_MAX,
BREAKOUT_LOOKBACK_DAYS, VOLUME_MULT_MIN, COOLDOWN_HOURS.
Asset list:
Default = SOL only. Extend ASSETS_TO_SCAN with more once you've watched
this run for a week and confirmed it doesn't fire-hose noise.
"""
from __future__ import annotations
import logging
import statistics
from datetime import datetime, timedelta, timezone
from typing import Optional
import httpx
from app.config import settings
from app.services.market_data import for_asset, drop_in_progress_bar
from app.services import scanner_state
logger = logging.getLogger(__name__)
SCANNER_NAME = "vcp_breakout"
scanner_state.register(SCANNER_NAME)
# ─── Tunables ───────────────────────────────────────────────────────────────
ATR_RECENT_DAYS = 7
ATR_BASELINE_DAYS = 30
ATR_RATIO_MAX = 0.5 # recent ATR must be ≤ 50% of baseline
BREAKOUT_LOOKBACK_DAYS = 30 # break of this many days' high
VOLUME_MULT_MIN = 1.5 # current 24h vol vs 30-day avg
COOLDOWN_HOURS = 48 # don't re-signal same asset within this window
# Assets to scan. Provider selection (Binance vs HL) is automatic via
# market_data.for_asset() — HL-native perps like TRUMP/HYPE route to HL
# automatically. Add to HL_NATIVE_ASSETS in market_data.py if needed.
ASSETS_TO_SCAN = [
"SOL",
# "ETH", "LINK", "AVAX", # Binance has them
# "TRUMP", "HYPE", # HL-native (routed automatically)
]
# Cooldown via scanner_state.in_cooldown — DB-backed, restart-safe.
# Data fetching is delegated to market_data.for_asset() — see that module
# for Binance vs Hyperliquid routing.
# ─── Signal logic ───────────────────────────────────────────────────────────
def _atr_proxy(candles: list[dict]) -> Optional[float]:
"""Mean of (high-low)/close — simple ATR proxy, dimensionless."""
ranges = [(c["high"] - c["low"]) / c["close"] for c in candles if c["close"] > 0]
return statistics.fmean(ranges) if ranges else None
def evaluate_breakout(candles: list[dict]) -> tuple[bool, dict]:
"""Returns (is_signal, debug_info). All 4 gates must pass.
Pure function — no side effects, fully testable.
"""
if len(candles) < ATR_BASELINE_DAYS * 6:
return False, {"reason": "insufficient_data", "bars": len(candles)}
# 1. Volatility contraction
recent_bars = candles[-ATR_RECENT_DAYS * 6:]
baseline_bars = candles[-ATR_BASELINE_DAYS * 6:]
atr_recent = _atr_proxy(recent_bars)
atr_baseline = _atr_proxy(baseline_bars)
if not atr_recent or not atr_baseline:
return False, {"reason": "atr_calc_failed"}
atr_ratio = atr_recent / atr_baseline
if atr_ratio > ATR_RATIO_MAX:
return False, {"reason": "no_contraction", "atr_ratio": round(atr_ratio, 3)}
# 2. Range break — UP through prior high (long) or DOWN through prior
# low (short). Lookback excludes the current bar.
lookback = candles[-BREAKOUT_LOOKBACK_DAYS * 6 : -1]
if not lookback:
return False, {"reason": "no_lookback"}
prior_high = max(c["high"] for c in lookback)
prior_low = min(c["low"] for c in lookback)
current_close = candles[-1]["close"]
if current_close > prior_high:
direction, ref, gap = "buy", prior_high, (current_close - prior_high) / prior_high * 100
elif current_close < prior_low:
direction, ref, gap = "short", prior_low, (prior_low - current_close) / prior_low * 100
else:
return False, {"reason": "no_range_break", "close": current_close,
"prior_high": prior_high, "prior_low": prior_low}
# 3. Volume confirmation (same for both directions — a real break needs
# participation regardless of which way it goes)
recent_vol_24h = sum(c["volume"] for c in candles[-6:])
baseline_avg_24h = sum(c["volume"] for c in candles[-180:]) / 30
if baseline_avg_24h <= 0:
return False, {"reason": "no_volume_data"}
vol_ratio = recent_vol_24h / baseline_avg_24h
if vol_ratio < VOLUME_MULT_MIN:
return False, {"reason": "weak_volume", "vol_ratio": round(vol_ratio, 2)}
return True, {
"direction": direction,
"atr_ratio": round(atr_ratio, 3),
"vol_ratio": round(vol_ratio, 2),
"break_ref": round(ref, 4),
"break_at": round(current_close, 4),
"headroom_pct": round(gap, 2),
}
def _confidence_from(debug: dict) -> int:
"""Map the signal's quality into a 0-100 confidence used by sizing.
The tighter the contraction and the bigger the volume spike, the higher
the confidence. Capped at 95 — leave headroom so AI signals at 100 stay
distinguishable.
"""
score = 70 # baseline for "all 4 gates passed"
# Tighter contraction → higher confidence
if debug.get("atr_ratio", 1) < 0.35: score += 10
elif debug.get("atr_ratio", 1) < 0.45: score += 5
# Bigger volume spike → higher confidence
if debug.get("vol_ratio", 0) >= 3.0: score += 10
elif debug.get("vol_ratio", 0) >= 2.0: score += 5
return min(score, 95)
# ─── Ingest call ────────────────────────────────────────────────────────────
async def _emit_signal(asset: str, debug: dict) -> None:
"""POST the signal to /api/signals/ingest. Treats the local server as the
target — same machine, no network hop in dev.
"""
if not settings.ingest_api_key:
logger.warning("VCP: signal would fire on %s but INGEST_API_KEY is empty", asset)
return
confidence = _confidence_from(debug)
direction = debug["direction"]
way = "breakout ↑" if direction == "buy" else "breakdown ↓"
rel = "above" if direction == "buy" else "below"
payload = {
"source": "breakout",
"external_id": f"vcp-{asset}-{direction}-{datetime.now(timezone.utc).strftime('%Y%m%d-%H%M')}",
"text": (
f"VCP {way} on {asset}: ATR ratio {debug['atr_ratio']} "
f"(7d/30d), {debug['vol_ratio']}× normal volume, "
f"close ${debug['break_at']} {rel} {BREAKOUT_LOOKBACK_DAYS}-day "
f"level ${debug['break_ref']} ({debug['headroom_pct']}%)"
),
"signal": direction,
"target_asset": asset,
"confidence": confidence,
"category": "vcp_breakout",
"expected_move_pct": 5.0, # historical VCP median expansion
}
async with httpx.AsyncClient(timeout=10) as client:
resp = await client.post(
"http://localhost:8000/api/signals/ingest",
json=payload,
headers={"X-Ingest-Key": settings.ingest_api_key},
)
if resp.status_code >= 400:
logger.error("VCP ingest failed (%d): %s", resp.status_code, resp.text[:200])
else:
logger.info("VCP signal emitted for %s, confidence=%d: %s",
asset, confidence, resp.json())
# ─── Scheduler entry point ──────────────────────────────────────────────────
async def scan_once() -> None:
"""Single scan pass over ASSETS_TO_SCAN. Called every 30 minutes.
Kill-switch + DB-backed cooldown. Drops in-progress 4h bar before
evaluating — without that, the breakout test ran against a partial bar
and could flicker FIRE / no-FIRE within a 4h window.
"""
if not scanner_state.is_enabled(SCANNER_NAME):
logger.debug("VCP scanner disabled — skipping run")
return
fired_any = False
error_msg: Optional[str] = None
for asset in ASSETS_TO_SCAN:
# Cooldown — DB-backed, restart-safe.
cooldown_days = COOLDOWN_HOURS / 24
if await scanner_state.in_cooldown("breakout", asset, cooldown_days):
continue
provider = for_asset(asset)
try:
candles = await provider.fetch_4h(asset, days=ATR_BASELINE_DAYS + 1)
except Exception as exc:
error_msg = f"{asset}: {exc}"
logger.error("VCP fetch failed for %s via %s: %s", asset, provider.name, exc)
continue
candles = drop_in_progress_bar(candles, "4h")
if not candles:
logger.warning("VCP: %s returned 0 candles from %s — symbol may not exist",
asset, provider.name)
continue
is_signal, debug = evaluate_breakout(candles)
logger.info("VCP scan %s [%s]: %s — %s", asset, provider.name,
"FIRE" if is_signal else "no", debug)
if is_signal:
await _emit_signal(asset, debug)
fired_any = True
if error_msg:
scanner_state.record_run(SCANNER_NAME, "error", error_msg)
elif fired_any:
scanner_state.record_run(SCANNER_NAME, "fired")
else:
scanner_state.record_run(SCANNER_NAME, "ok")
app/services/scanners/_archive/weekly_rsi_reversal.py
+264
View File
@@ -0,0 +1,264 @@
"""
Weekly RSI extreme + recovery scanner.
What it catches:
Severe weekly oversold conditions that flush capitulating holders, followed
by the first sign of strength. Historical examples this would have hit:
BTC 2022-11 ($15.5k bottom)
ETH 2022-06 ($900 bottom)
SOL 2022-12 ($8 bottom)
AAVE 2022-06 ($45 bottom)
Trigger logic (intentionally strict):
PRE-CONDITION: ≥ 4 consecutive completed weekly bars with RSI(14) < 30
(this is the capitulation phase — sustained extreme weakness)
TRIGGER: current completed week's RSI(14) >= 35
(i.e. the FIRST recovery week — RSI has lifted off the floor)
COOLDOWN: 60 days — these events happen ~once per year per asset.
No re-firing on the same setup.
Companion exit parameters (passed to the bot via signal payload — wider stops
to match the longer holding period and higher single-trade conviction):
SL = 8%
TRAILING_ACTIVATE = 15%
TRAILING_STOP = 6%
MAX_HOLD = 60 days
This is a high-conviction, low-frequency signal. Expect 0-3 fires per asset
per year. Don't tune it for higher frequency — that defeats the point.
"""
from __future__ import annotations
import logging
from datetime import datetime, timedelta, timezone
from typing import Optional
import httpx
from app.config import settings
from app.services.market_data import REVERSAL_BASKET, for_asset, drop_in_progress_bar
from app.services import scanner_state
SCANNER_NAME = "weekly_rsi_reversal"
scanner_state.register(SCANNER_NAME)
logger = logging.getLogger(__name__)
# ─── Tunables ───────────────────────────────────────────────────────────────
RSI_PERIOD = 14
RSI_OVERSOLD = 30 # "deeply oversold" floor (long setup)
RSI_RECOVERY_TRIGGER = 35 # bounce-off line for the long
RSI_OVERBOUGHT = 70 # "euphoric" ceiling (short setup)
RSI_ROLLOVER_TRIGGER = 65 # roll-off line for the short
WEEKS_OF_OVERSOLD = 4 # consecutive extreme weeks required (both sides)
WEEKS_TO_FETCH = 40 # enough history for stable Wilder's RSI
COOLDOWN_DAYS = 60
# Exit profile passed to /signals/ingest. Caller can override on the bot side.
PAYLOAD_CONFIDENCE = 88 # high conviction — top of the regime-filter score
PAYLOAD_EXPECTED_MOVE = 30.0 # historical median move after capitulation
# Cooldown is now DB-backed via scanner_state.in_cooldown() — survives
# restart. No more in-memory _last_signal_at dict.
# ─── RSI math (Wilder's smoothed) ───────────────────────────────────────────
def calculate_rsi(closes: list[float], period: int = 14) -> list[Optional[float]]:
"""Wilder's RSI. Returns a list aligned with `closes` where the first
`period` entries are None (not enough history yet).
Implementation note: the first valid RSI uses a SIMPLE mean of the first
`period` gains/losses, subsequent values use exponential smoothing with
alpha = 1/period. This matches TradingView, Binance UI, and most charting
tools — the "Cutler's RSI" (pure SMA) variant would give different results.
"""
n = len(closes)
rsi: list[Optional[float]] = [None] * n
if n < period + 1:
return rsi
gains = [max(closes[i] - closes[i - 1], 0) for i in range(1, n)]
losses = [max(closes[i - 1] - closes[i], 0) for i in range(1, n)]
# First valid RSI = simple mean over first `period` deltas
avg_gain = sum(gains[:period]) / period
avg_loss = sum(losses[:period]) / period
if avg_loss == 0:
rsi[period] = 100.0
else:
rs = avg_gain / avg_loss
rsi[period] = 100 - 100 / (1 + rs)
# Subsequent values use Wilder's smoothing
for i in range(period + 1, n):
avg_gain = (avg_gain * (period - 1) + gains[i - 1]) / period
avg_loss = (avg_loss * (period - 1) + losses[i - 1]) / period
if avg_loss == 0:
rsi[i] = 100.0
else:
rs = avg_gain / avg_loss
rsi[i] = 100 - 100 / (1 + rs)
return rsi
# ─── Signal logic ───────────────────────────────────────────────────────────
def evaluate_rsi_reversal(weekly_candles: list[dict]) -> tuple[bool, dict]:
"""Pure function — no side effects, fully testable.
Returns (is_signal, debug_info).
"""
if len(weekly_candles) < RSI_PERIOD + WEEKS_OF_OVERSOLD + 2:
return False, {"reason": "insufficient_data", "bars": len(weekly_candles)}
closes = [c["close"] for c in weekly_candles]
rsi = calculate_rsi(closes, RSI_PERIOD)
# The MOST RECENT weekly bar is `closes[-1]`. We treat it as the "current"
# recovery candidate. Look BACKWARDS for WEEKS_OF_OVERSOLD prior bars all
# with RSI < RSI_OVERSOLD.
current_rsi = rsi[-1]
if current_rsi is None:
return False, {"reason": "rsi_not_computable"}
window = rsi[-(WEEKS_OF_OVERSOLD + 1):-1] # the N weeks before current
if any(r is None for r in window):
return False, {"reason": "history_gaps_in_window"}
this_close = weekly_candles[-1]["close"]
prev_close = weekly_candles[-2]["close"]
# ── LONG: capitulation bottom ──────────────────────────────────────────
# Sustained RSI<30, now lifting ≥35, price turning up.
if (current_rsi >= RSI_RECOVERY_TRIGGER
and all(r < RSI_OVERSOLD for r in window)
and this_close > prev_close):
low_w = min(c["low"] for c in weekly_candles[-(WEEKS_OF_OVERSOLD + 1):])
return True, {
"direction": "buy",
"current_rsi": round(current_rsi, 1),
"window_rsi": [round(r, 1) for r in window],
"move_pct": round((this_close - low_w) / low_w * 100, 2) if low_w > 0 else 0.0,
"trigger_close": round(this_close, 4),
}
# ── SHORT: euphoric top ────────────────────────────────────────────────
# Sustained RSI>70, now rolling ≤65, price turning down. Symmetric mirror.
if (current_rsi <= RSI_ROLLOVER_TRIGGER
and all(r > RSI_OVERBOUGHT for r in window)
and this_close < prev_close):
high_w = max(c["high"] for c in weekly_candles[-(WEEKS_OF_OVERSOLD + 1):])
return True, {
"direction": "short",
"current_rsi": round(current_rsi, 1),
"window_rsi": [round(r, 1) for r in window],
"move_pct": round((high_w - this_close) / high_w * 100, 2) if high_w > 0 else 0.0,
"trigger_close": round(this_close, 4),
}
# Neither side qualified — report the closest miss for the log.
return False, {
"reason": "no_setup",
"current_rsi": round(current_rsi, 1),
"window_rsi": [round(r, 1) for r in window],
}
# ─── Ingest emission ────────────────────────────────────────────────────────
async def _emit_signal(asset: str, debug: dict) -> None:
if not settings.ingest_api_key:
logger.warning("RSI-reversal would fire on %s but INGEST_API_KEY empty", asset)
return
direction = debug["direction"]
side_word = "capitulation bottom" if direction == "buy" else "euphoric top"
payload = {
"source": "rsi_reversal",
"external_id": f"rsi-{asset}-{direction}-{datetime.now(timezone.utc).strftime('%Y%W')}",
"text": (
f"Weekly RSI {side_word} on {asset}: RSI {debug['current_rsi']} after "
f"{WEEKS_OF_OVERSOLD}wk extreme (window: {debug['window_rsi']}). "
f"{debug['move_pct']}% move off the extreme @ ${debug['trigger_close']}."
),
"signal": direction,
"target_asset": asset,
"confidence": PAYLOAD_CONFIDENCE,
"category": "rsi_extreme_reversal",
"expected_move_pct": PAYLOAD_EXPECTED_MOVE,
}
async with httpx.AsyncClient(timeout=10) as client:
resp = await client.post(
"http://localhost:8000/api/signals/ingest",
json=payload,
headers={"X-Ingest-Key": settings.ingest_api_key},
)
if resp.status_code >= 400:
logger.error("RSI-reversal ingest failed (%d): %s", resp.status_code, resp.text[:200])
else:
logger.info("RSI-reversal signal emitted for %s: %s", asset, resp.json())
# ─── Scheduler entry point ──────────────────────────────────────────────────
async def scan_once() -> None:
"""One scan pass over REVERSAL_BASKET. Called by APScheduler.
Kill-switch aware: short-circuits if the operator disabled this scanner.
Cooldown is DB-backed (queries posts table), so restarts don't reset
state.
"""
if not scanner_state.is_enabled(SCANNER_NAME):
logger.debug("RSI-reversal scanner disabled — skipping run")
return
fired_any = False
error_msg: Optional[str] = None
for asset in REVERSAL_BASKET:
# Cooldown — DB-backed.
if await scanner_state.in_cooldown("rsi_reversal", asset, COOLDOWN_DAYS):
continue
provider = for_asset(asset)
try:
candles = await provider.fetch_1w(asset, weeks=WEEKS_TO_FETCH)
except Exception as exc:
error_msg = f"{asset}: {exc}"
logger.error("RSI-reversal fetch failed for %s via %s: %s",
asset, provider.name, exc)
continue
candles = drop_in_progress_bar(candles, "1w")
if not candles:
logger.warning("RSI-reversal: %s returned 0 weekly bars from %s",
asset, provider.name)
continue
is_signal, debug = evaluate_rsi_reversal(candles)
if is_signal:
logger.info("RSI-reversal scan %s [%s]: FIRE — %s", asset, provider.name, debug)
await _emit_signal(asset, debug)
fired_any = True
else:
logger.debug("RSI-reversal scan %s [%s]: no — %s", asset, provider.name, debug)
if error_msg:
scanner_state.record_run(SCANNER_NAME, "error", error_msg)
elif fired_any:
scanner_state.record_run(SCANNER_NAME, "fired")
else:
scanner_state.record_run(SCANNER_NAME, "ok")
app/services/scanners/btc_bottom_reversal.py
+148
View File
@@ -0,0 +1,148 @@
"""BTC bottom-reversal long scanner — 2-of-3 price confluence.
Simple, transparent, no on-chain data and no API keys. Fires only when at
least TWO of these three classic bottom signals agree:
A. AHR999 < 0.45 — deep-value / bottom zone
B. price ≤ 200-week MA ×1.05 — every cycle has bottomed near the 200WMA
C. Pi Cycle Bottom — 150d EMA ≤ 471d SMA × 0.745
Long only, low frequency, stateless. No tight stop (real macro bottoms wick
15–30%); the position is invalidated when AHR999 climbs back above 1.2 — i.e.
BTC is no longer cheap and the bottom thesis is spent.
"""
from __future__ import annotations
import logging
from datetime import datetime, timezone
import httpx
from app.config import settings
from app.services import scanner_state
from app.services.market_data import for_asset, drop_in_progress_bar
from app.services.bottom_indicators import bottom_confluence
SCANNER_NAME = "btc_bottom_reversal"
scanner_state.register(SCANNER_NAME)
logger = logging.getLogger(__name__)
ASSET = "BTC"
COOLDOWN_DAYS = 30
# 3 votes → max conviction; 2 votes → solid. Scale confidence with agreement.
CONFIDENCE_BY_VOTES = {2: 88, 3: 94}
EXPECTED_MOVE_PCT = 25.0
# Pi Cycle needs 471 daily closes; +buffer for the dropped in-progress bar.
DAILY_LOOKBACK_DAYS = 520
# 200-week MA needs 200 weekly closes; +buffer.
WEEKLY_LOOKBACK_WEEKS = 210
async def evaluate_once() -> tuple[bool, dict]:
provider = for_asset(ASSET)
raw_daily = await provider.fetch_1d(ASSET, days=DAILY_LOOKBACK_DAYS)
daily = drop_in_progress_bar(raw_daily, "1d")
raw_weekly = await provider.fetch_1w(ASSET, weeks=WEEKLY_LOOKBACK_WEEKS)
weekly = drop_in_progress_bar(raw_weekly, "1w")
if not daily:
return False, {"reason": "missing_btc_daily"}
if not weekly:
return False, {"reason": "missing_btc_weekly"}
daily_closes = [float(c["close"]) for c in daily if c.get("close")]
weekly_closes = [float(c["close"]) for c in weekly if c.get("close")]
conf = bottom_confluence(daily_closes, weekly_closes)
debug: dict = dict(conf.detail)
if not conf.fired:
debug["reason"] = "confluence_not_met"
return False, debug
confidence = CONFIDENCE_BY_VOTES.get(conf.votes, 88)
debug["confidence"] = confidence
# No tight stop. Thesis is invalidated when BTC is no longer cheap; the
# 200WMA band is a useful structural reference for the exit monitor.
debug["invalidation_price"] = debug.get("wma200")
return True, debug
def _summary(debug: dict) -> str:
s = debug.get("signals", {})
on = [k for k, v in s.items() if v]
return (
f"BTC bottom confluence ({debug.get('votes')}/3): {', '.join(on)}. "
f"AHR999 {debug.get('ahr999')} (<{debug.get('ahr999_threshold')}), "
f"price {debug.get('price')}, 200WMA {debug.get('wma200')}, "
f"Pi 150EMA {debug.get('pi_ema150')} vs 471SMA×0.745 "
f"{debug.get('pi_sma471_scaled')}. No tight stop; "
f"invalidate if AHR999 > 1.2."
)
async def _emit_signal(debug: dict) -> bool:
"""POST the signal to the ingest endpoint. Returns True iff a Post was
(idempotently) created. False means the signal was NOT recorded — caller
must NOT treat the run as 'fired' (cooldown is keyed off the DB Post)."""
if not settings.ingest_api_key:
logger.error("BTC bottom-reversal would fire but INGEST_API_KEY empty "
"— signal NOT recorded, check deploy env")
return False
payload = {
"source": "btc_bottom_reversal",
"external_id": f"btc-bottom-{datetime.now(timezone.utc).strftime('%Y%m%d')}",
"text": _summary(debug),
"signal": "buy",
"target_asset": ASSET,
"confidence": debug["confidence"],
"category": "btc_bottom_reversal_long",
"expected_move_pct": EXPECTED_MOVE_PCT,
"invalidation_price": debug.get("invalidation_price"),
}
async with httpx.AsyncClient(timeout=10) as client:
resp = await client.post(
"http://localhost:8000/api/signals/ingest",
json=payload,
headers={"X-Ingest-Key": settings.ingest_api_key},
)
if resp.status_code >= 400:
logger.error("BTC bottom-reversal ingest failed (%d): %s",
resp.status_code, resp.text[:200])
return False
logger.info("BTC bottom-reversal signal emitted: %s", resp.json())
return True
async def scan_once() -> None:
if not scanner_state.is_enabled(SCANNER_NAME):
logger.debug("BTC bottom-reversal scanner disabled — skipping run")
return
if await scanner_state.in_cooldown("btc_bottom_reversal", ASSET, COOLDOWN_DAYS):
logger.debug("BTC bottom-reversal cooldown active")
scanner_state.record_run(SCANNER_NAME, "ok", "cooldown")
return
try:
should_fire, debug = await evaluate_once()
except Exception as exc:
logger.error("BTC bottom-reversal scan failed: %s", exc)
scanner_state.record_run(SCANNER_NAME, "error", str(exc))
return
if should_fire:
logger.info("BTC bottom-reversal FIRE — %s", debug)
emitted = await _emit_signal(debug)
if emitted:
scanner_state.record_run(SCANNER_NAME, "fired")
else:
# Not recorded → no DB Post → cooldown won't arm. Surface this as
# an error so a misconfigured deploy is obvious, not a silent
# daily "fired" that never actually trades.
scanner_state.record_run(SCANNER_NAME, "error", "ingest_failed_or_key_missing")
else:
logger.info("BTC bottom-reversal no — %s", debug)
scanner_state.record_run(SCANNER_NAME, "ok")
app/services/scanners/funding_reversal.py
+379
View File
@@ -0,0 +1,379 @@
"""
Funding rate extreme + reversal scanner.
What it catches:
Crowded perp positioning that's about to unwind. When one side has been
paying ridiculous funding for weeks, the eventual unwind ("the squeeze")
is the cleanest single-day move you'll find. Examples:
BTC 2024-08 funding deeply +ve for 30d → 14% flush down within a week
SOL 2023-09 funding deeply -ve for 30d → 60% rally over the next month
ETH 2024-Q4 funding +3.5% on 30d → 12% pullback
Trigger logic:
PRE-CONDITION: Sum of last 30 days of funding rate (per 8h cycle)
exceeds ±FUNDING_EXTREME_PCT. The sign tells us which
side is overcrowded:
sum > +3% → longs have been paying — bearish setup
(signal = short)
sum < -3% → shorts paying — bullish setup
(signal = buy)
TRIGGER: Funding direction has STARTED to mean-revert
(last 3 funding cycles closer to zero than the 30d avg)
AND price has begun moving in the contrarian direction
(last 7 days price move at least 3% in that direction).
COOLDOWN: 14 days. Funding extremes can persist for weeks but
each unwind is a distinct event.
Companion exits — tighter than RSI/SMA because funding moves play out faster
(days, not weeks):
SL = 4%
TRAILING_ACTIVATE = 10%
TRAILING_STOP = 5%
MAX_HOLD = 30 days
This is the highest-frequency of the three reversal signals — expect 3-5
fires per asset per year — and the most "alpha-like" (most market participants
ignore funding entirely).
"""
from __future__ import annotations
import logging
import statistics
from datetime import datetime, timedelta, timezone
from typing import Optional
import httpx
from app.config import settings
from app.services import scanner_state
from app.services.market_data import REVERSAL_BASKET, for_asset, drop_in_progress_bar
# Promoted from booster → standalone scanner. Still importable by
# btc_bottom_reversal for confluence boost; ALSO emits its own signals via
# scan_once() registered with the APScheduler in app/main.py.
SCANNER_NAME = "funding_reversal"
scanner_state.register(SCANNER_NAME)
ASSET = "BTC" # BTC-only for now; extend to ETH/SOL after results validate
# How much funding+price history to load each tick.
# - Binance: 8h cadence → 30d ≈ 90 cycles
# - HL: 1h cadence → capped at 500 rows ≈ 20.8d (handled inside evaluate)
FUNDING_DAYS_LOOKBACK = 30
PRICE_DAYS_LOOKBACK = 35 # need 7d confirm + buffer for in-progress bar
logger = logging.getLogger(__name__)
# ─── Tunables ───────────────────────────────────────────────────────────────
FUNDING_HISTORY_DAYS = 30
FUNDING_EXTREME_THRESHOLD = 0.03 # 3% cumulative — extreme one-sided pressure
# CRITICAL: "recent" is in HOURS, not CYCLES. Binance funding is 8h-cadence
# (3 cycles/day) but HL is 1h-cadence (24 cycles/day) — a fixed "3 cycles
# lookback" means 24h on Binance but only 3h on HL. We pick cycles dynamically
# based on the response's actual cadence so 24h of funding history is always
# the comparison window.
FUNDING_REVERSAL_LOOKBACK_HOURS = 24
FUNDING_REVERSAL_RATIO = 0.5 # recent avg must be ≤ 50% of 30d-avg in magnitude
PRICE_CONFIRM_DAYS = 7
PRICE_CONFIRM_PCT = 3.0 # price moved 3%+ in contrarian direction
COOLDOWN_DAYS = 14
PAYLOAD_CONFIDENCE = 82 # slightly lower than RSI/SMA — funding can chop
PAYLOAD_EXPECTED_MOVE = 10.0
EMIT_STANDALONE_SIGNALS = True # promoted from booster → standalone signal source
# Cooldown via scanner_state.in_cooldown — DB-backed, restart-safe.
# ─── Signal logic ───────────────────────────────────────────────────────────
def _detect_cadence_hours(funding: list[dict]) -> float:
"""Infer the funding cadence (hours between cycles) from the response.
Binance returns ~8h intervals; HL returns ~1h. We can't assume — the
response itself is the source of truth. Average over the most recent 10
intervals to smooth out occasional gaps.
"""
if len(funding) < 2:
return 8.0 # safe Binance default
sample = funding[-min(11, len(funding)):]
diffs = [sample[i]["time_ms"] - sample[i - 1]["time_ms"] for i in range(1, len(sample))]
if not diffs:
return 8.0
return statistics.fmean(diffs) / 3_600_000.0 # ms → hours
MIN_COVERAGE_DAYS = 14 # below this, the signal is statistically too noisy
def evaluate_funding_reversal(
funding_history: list[dict],
daily_candles: list[dict],
) -> tuple[bool, dict]:
"""Pure function. Returns (is_signal, debug).
Debug contains `direction` key on real signals — 'buy' or 'short'.
Cross-venue safety: HL caps fundingHistory at 500 rows (~20.8 days for
1h cadence), Binance can give a full 30 days. We compute the actual
coverage span from the data and scale the cumulative-funding threshold
proportionally — so 3% over 30 days and 2.08% over 20.8 days are
treated as equivalent in daily-average terms.
"""
if not funding_history or len(funding_history) < 2:
return False, {"reason": "insufficient_funding_history",
"cycles": len(funding_history)}
cadence_h = _detect_cadence_hours(funding_history)
# Actual time span the response covers — bounded by HL's 500-row cap
# for HL, or by what we asked for on Binance.
span_ms = funding_history[-1]["time_ms"] - funding_history[0]["time_ms"]
actual_days = span_ms / 86_400_000
if actual_days < MIN_COVERAGE_DAYS:
return False, {"reason": "insufficient_coverage_days",
"actual_days": round(actual_days, 1),
"min_required": MIN_COVERAGE_DAYS}
rates = [f["rate"] for f in funding_history]
cumulative_funding = sum(rates)
avg_full_window = statistics.fmean(rates)
# Scale the extreme threshold by ACTUAL coverage so cross-venue results
# are comparable. Binance: 30d → threshold ×1. HL: 20.8d → threshold ×0.69.
scaled_threshold = FUNDING_EXTREME_THRESHOLD * (actual_days / FUNDING_HISTORY_DAYS)
# "Recent" lookback in TIME units, not cycles. 24h of cycles regardless
# of venue. Min 3 to avoid pathological cases (very short history).
recent_cycles = max(3, int(FUNDING_REVERSAL_LOOKBACK_HOURS / cadence_h))
recent_cycles = min(recent_cycles, len(rates)) # don't over-slice
avg_recent_cycle = statistics.fmean(rates[-recent_cycles:])
# 2. Identify direction (which side is overcrowded)
if cumulative_funding > scaled_threshold:
# Longs have been paying → expect SHORT squeeze
direction = "short"
elif cumulative_funding < -scaled_threshold:
# Shorts have been paying → expect LONG rally
direction = "buy"
else:
return False, {
"reason": "no_extreme",
"cumulative_funding_pct": round(cumulative_funding * 100, 3),
"scaled_threshold_pct": round(scaled_threshold * 100, 3),
"coverage_days": round(actual_days, 1),
}
# 3. Funding must be MEAN-REVERTING (recent cycles softer than 30d avg)
# Use absolute magnitude — what matters is "the pressure is easing",
# not the direction of zero-crossing.
if abs(avg_full_window) == 0:
return False, {"reason": "degenerate_avg"}
revert_ratio = abs(avg_recent_cycle) / abs(avg_full_window)
if revert_ratio > FUNDING_REVERSAL_RATIO:
return False, {
"reason": "funding_still_extreme",
"revert_ratio": round(revert_ratio, 2),
"needed_below": FUNDING_REVERSAL_RATIO,
"direction": direction,
}
# 4. Price confirms the contrarian move (last 7 days)
if not daily_candles or len(daily_candles) < PRICE_CONFIRM_DAYS + 1:
return False, {"reason": "insufficient_price_history",
"have": len(daily_candles), "need": PRICE_CONFIRM_DAYS + 1}
first_close = daily_candles[-(PRICE_CONFIRM_DAYS + 1)]["close"]
last_close = daily_candles[-1]["close"]
if first_close == 0:
return False, {"reason": "bad_first_close"}
pct_change = (last_close - first_close) / first_close * 100
if direction == "buy" and pct_change < PRICE_CONFIRM_PCT:
return False, {"reason": "price_not_yet_recovering",
"pct_7d": round(pct_change, 2), "direction": direction}
if direction == "short" and pct_change > -PRICE_CONFIRM_PCT:
return False, {"reason": "price_not_yet_falling",
"pct_7d": round(pct_change, 2), "direction": direction}
boost = {
"direction": direction,
"cum_funding_30d_pct": round(cumulative_funding * 100, 3),
"recent_avg_cycle_pct": round(avg_recent_cycle * 100, 4),
"revert_ratio": round(revert_ratio, 2),
"price_7d_pct": round(pct_change, 2),
"trigger_close": round(last_close, 4),
}
if not EMIT_STANDALONE_SIGNALS:
return False, {"reason": "boost_only", "boost": boost}
return True, boost
# ─── Standalone scanner plumbing ────────────────────────────────────────────
async def _fetch_inputs() -> tuple[list[dict], list[dict]]:
"""Pull funding history + daily candles for ASSET.
We bypass provider.fetch_1d in favor of fapi.binance.com/fapi/v1/klines
because (a) funding lives on the futures venue anyway — same liquidity
pool, same trade flow — and (b) the spot api.binance.com host is
occasionally geo-blocked, while fapi is more reliably reachable.
"""
provider = for_asset(ASSET)
funding = await provider.fetch_funding(ASSET, days=FUNDING_DAYS_LOOKBACK)
end_ms = int(datetime.now(timezone.utc).timestamp() * 1000)
start_ms = end_ms - PRICE_DAYS_LOOKBACK * 24 * 3600 * 1000
async with httpx.AsyncClient(timeout=20) as client:
resp = await client.get(
"https://fapi.binance.com/fapi/v1/klines",
params={"symbol": f"{ASSET}USDT", "interval": "1d",
"startTime": start_ms, "endTime": end_ms, "limit": 1000},
)
resp.raise_for_status()
raw_daily = [
{"time_ms": r[0], "open": float(r[1]), "high": float(r[2]),
"low": float(r[3]), "close": float(r[4]), "volume": float(r[5])}
for r in resp.json()
]
daily = drop_in_progress_bar(raw_daily, "1d")
return funding, daily
def _summary(debug: dict) -> str:
direction = debug.get("direction", "?").upper()
cum = debug.get("cum_funding_30d_pct")
recent = debug.get("recent_avg_cycle_pct")
revert = debug.get("revert_ratio")
p7d = debug.get("price_7d_pct")
return (
f"BTC funding extreme reversal — {direction}. "
f"30d cumulative funding {cum}% (crowded {'longs' if direction == 'SHORT' else 'shorts'}); "
f"last-24h cycle avg {recent}% (revert ratio {revert}); "
f"price 7d {p7d:+.2f}% confirms the unwind."
)
async def _emit_signal(debug: dict) -> bool:
"""POST to the ingest endpoint. Idempotent via external_id (per-day)."""
if not settings.ingest_api_key:
logger.error("Funding reversal would fire but INGEST_API_KEY empty — not recorded")
return False
direction = debug["direction"] # 'buy' or 'short'
expected_move = PAYLOAD_EXPECTED_MOVE if direction == "buy" else -PAYLOAD_EXPECTED_MOVE
payload = {
"source": "funding_reversal",
"external_id": f"funding-{ASSET}-{direction}-{datetime.now(timezone.utc).strftime('%Y%m%d')}",
"text": _summary(debug),
"signal": direction,
"target_asset": ASSET,
"confidence": PAYLOAD_CONFIDENCE,
"category": f"funding_reversal_{direction}",
"expected_move_pct": expected_move,
"invalidation_price": debug.get("trigger_close"),
}
async with httpx.AsyncClient(timeout=10) as client:
resp = await client.post(
"http://localhost:8000/api/signals/ingest",
json=payload,
headers={"X-Ingest-Key": settings.ingest_api_key},
)
if resp.status_code >= 400:
logger.error("Funding reversal ingest failed (%d): %s",
resp.status_code, resp.text[:200])
return False
logger.info("Funding reversal signal emitted: %s", resp.json())
return True
async def scan_once() -> None:
"""Hourly tick. Idempotent (cooldown-gated, external_id-dedupe at ingest)."""
if not scanner_state.is_enabled(SCANNER_NAME):
logger.debug("Funding reversal scanner disabled — skipping")
return
if await scanner_state.in_cooldown("funding_reversal", ASSET, COOLDOWN_DAYS):
logger.debug("Funding reversal cooldown active (%dd)", COOLDOWN_DAYS)
scanner_state.record_run(SCANNER_NAME, "ok", "cooldown")
return
try:
funding, daily = await _fetch_inputs()
fired, debug = evaluate_funding_reversal(funding, daily)
except Exception as exc:
# Always include exception type — httpx errors often have empty .args
# which formatted as just "Funding reversal scan failed:" before.
logger.exception("Funding reversal scan failed: %s (%s)",
type(exc).__name__, exc)
scanner_state.record_run(SCANNER_NAME, "error",
f"{type(exc).__name__}: {exc}"[:200])
return
if fired:
logger.info("Funding reversal FIRE — %s", debug)
emitted = await _emit_signal(debug)
scanner_state.record_run(SCANNER_NAME, "fired" if emitted else "error",
None if emitted else "ingest_failed")
else:
logger.info("Funding reversal no — %s", debug.get("reason"))
scanner_state.record_run(SCANNER_NAME, "ok", debug.get("reason"))
# ─── Read API helper — current snapshot for the BTC page tab ────────────────
async def get_current_snapshot() -> dict:
"""Live read for the frontend BTC page funding tab. Returns the latest
funding rate, the 24h running average, cumulative 30d sum, and the verdict
of evaluate_funding_reversal() against current data. Cheap to call — only
network cost is two market_data fetches the scanner would do anyway."""
try:
funding, daily = await _fetch_inputs()
except Exception as exc:
logger.exception("funding snapshot fetch failed")
return {"ok": False, "error": f"{type(exc).__name__}: {exc}"}
if not funding:
return {"ok": False, "error": "no_funding_data"}
cadence_h = _detect_cadence_hours(funding)
rates = [f["rate"] for f in funding]
cum_30d_pct = sum(rates) * 100
span_days = (funding[-1]["time_ms"] - funding[0]["time_ms"]) / 86_400_000
latest = rates[-1] * 100
# Last-24h equivalent average per-cycle rate (in %)
recent_n = max(3, int(24 / cadence_h)) if cadence_h > 0 else 24
recent_n = min(recent_n, len(rates))
last_24h_avg = (sum(rates[-recent_n:]) / recent_n) * 100
fired, debug = evaluate_funding_reversal(funding, daily)
# 7-day funding history for the sparkline (truncate to keep payload small)
history = [
{"t": int(f["time_ms"]), "rate_pct": round(f["rate"] * 100, 5)}
for f in funding[-int(min(len(funding), 24 * 7 / max(cadence_h, 0.5))) :]
]
return {
"ok": True,
"asset": ASSET,
"cadence_hours": round(cadence_h, 2),
"coverage_days": round(span_days, 1),
"latest_rate_pct": round(latest, 5),
"last_24h_avg_pct": round(last_24h_avg, 5),
"cum_30d_pct": round(cum_30d_pct, 3),
"extreme_threshold_pct": round(FUNDING_EXTREME_THRESHOLD * 100, 3),
"signal_fired": fired,
"debug": debug,
"history": history,
}
app/services/scanners/sma_reclaim.py
+146
View File
@@ -0,0 +1,146 @@
"""
200-day SMA Reclaim scanner.
What it catches:
The moment a price that has been LIVING BELOW its 200-day moving average
for a sustained period climbs back ABOVE it on real volume. Historically
one of the most reliable "trend has changed" markers in any market —
hedge fund books, retail TA tools, momentum quants, everyone watches it.
Examples this would have caught:
BTC 2023-01 (~$22k, after the FTX flush)
BTC 2024-09 (after Q3 chop)
ETH 2023-01 (~$1500)
SOL 2023-02 (~$24, after FTX)
Trigger logic:
PRE-CONDITION: For the past DAYS_BELOW_REQUIRED days, daily close has been
BELOW the rolling 200-day SMA. (proves we're reversing a
sustained downtrend, not crossing a flat MA in chop)
TRIGGER: Today's close > 200-day SMA, AND
Today's volume > 1.3 × 30-day avg volume.
COOLDOWN: 30 days — false reclaims and shake-outs happen, don't
re-fire on noise.
Companion exit profile:
SL = 6%
TRAILING_ACTIVATE = 12%
TRAILING_STOP = 5%
MAX_HOLD = 90 days
The 90-day max-hold matches the holding period needed for a real trend
change to play out (~3 months is the historical median for a confirmed
200d-SMA-reclaim trend run).
"""
from __future__ import annotations
import logging
from datetime import datetime, timedelta, timezone
from typing import Optional
import httpx
from app.config import settings
from app.services.market_data import REVERSAL_BASKET, for_asset, drop_in_progress_bar
# LIBRARY MODULE — NOT a standalone scanner. evaluate_sma_reclaim() is
# imported by btc_bottom_reversal.py as the price-reclaim entry gate. It
# deliberately does NOT register with scanner_state: no UI toggle, no
# schedule. (Old standalone scan_once/_emit_signal removed — see git log.)
logger = logging.getLogger(__name__)
# ─── Tunables ───────────────────────────────────────────────────────────────
SMA_PERIOD = 200
DAYS_BELOW_REQUIRED = 30 # how long the asset must have been under SMA
VOLUME_LOOKBACK_DAYS = 30
VOLUME_MULT_MIN = 1.3
DAYS_TO_FETCH = 260 # SMA(200) + 30d-below check + safety margin
COOLDOWN_DAYS = 30
PAYLOAD_CONFIDENCE = 85
PAYLOAD_EXPECTED_MOVE = 20.0 # historical median 90-day run after reclaim
# Cooldown via scanner_state.in_cooldown — DB-backed, restart-safe.
# ─── Signal logic ───────────────────────────────────────────────────────────
def evaluate_sma_reclaim(daily_candles: list[dict]) -> tuple[bool, dict]:
"""Pure function. Returns (is_signal, debug).
Expects `daily_candles` ordered chronologically (oldest first), each
having keys close, volume.
"""
if len(daily_candles) < SMA_PERIOD + DAYS_BELOW_REQUIRED + 2:
return False, {"reason": "insufficient_data", "bars": len(daily_candles)}
closes = [c["close"] for c in daily_candles]
volumes = [c["volume"] for c in daily_candles]
# Rolling 200-day SMA at each bar from index SMA_PERIOD-1 onwards
smas: list[Optional[float]] = [None] * len(closes)
running_sum = sum(closes[:SMA_PERIOD])
smas[SMA_PERIOD - 1] = running_sum / SMA_PERIOD
for i in range(SMA_PERIOD, len(closes)):
running_sum += closes[i] - closes[i - SMA_PERIOD]
smas[i] = running_sum / SMA_PERIOD
today_close = closes[-1]
today_sma = smas[-1]
if today_sma is None:
return False, {"reason": "sma_not_computable"}
# Bottom-reversal mode is LONG-only:
# reclaim (long): was BELOW the SMA for N days, today closes ABOVE
# We explicitly do not trade symmetric short breakdowns here. Crypto
# top-calling is a different strategy with different risk.
reclaimed = today_close > today_sma
brokedown = today_close < today_sma
if brokedown:
return False, {
"reason": "shorts_disabled",
"close": round(today_close, 4),
"sma": round(today_sma, 4),
}
if not reclaimed:
return False, {"reason": "on_sma_no_cross",
"close": round(today_close, 4), "sma": round(today_sma, 4)}
# Prior DAYS_BELOW_REQUIRED bars must ALL be on the OPPOSITE side of the
# SMA from today (a real regime flip, not chop around a flat MA).
streak = 0
for i in range(2, DAYS_BELOW_REQUIRED + 2):
sma_at = smas[-i]
if sma_at is None:
return False, {"reason": "sma_history_incomplete"}
prior_on_wrong_side = closes[-i] >= sma_at
if prior_on_wrong_side:
return False, {"reason": "regime_period_too_short", "broke_at_day": i}
streak += 1
# Volume confirmation: today >= VOLUME_MULT_MIN × 30-day avg
avg_vol_30d = sum(volumes[-(VOLUME_LOOKBACK_DAYS + 1):-1]) / VOLUME_LOOKBACK_DAYS
if avg_vol_30d <= 0:
return False, {"reason": "no_volume_baseline"}
vol_ratio = volumes[-1] / avg_vol_30d
if vol_ratio < VOLUME_MULT_MIN:
return False, {"reason": "weak_volume", "vol_ratio": round(vol_ratio, 2)}
return True, {
"direction": "buy",
"close": round(today_close, 4),
"sma_200": round(today_sma, 4),
"gap_pct": round(abs(today_close - today_sma) / today_sma * 100, 2),
"streak_days": streak,
"vol_ratio": round(vol_ratio, 2),
}
app/services/signal_categories.py
+430
View File
@@ -0,0 +1,430 @@
"""
Two trading systems, one execution layer.
This module is the formal boundary between:
SYSTEM 1 — Trump (event-driven scalp)
source == "truth". Immediate market reaction to a post. Tight stops,
short hold, time-stop if no move. Uses the USER's configured exit
params (take_profit_pct / stop_loss_pct / trailing_*). Goes through
the Trump-tuned regime filter (thin-liquidity hours, recent-move,
vol-contraction).
SYSTEM 2 — Bitcoin Bottom (low-frequency bottom-reversal long)
source == "btc_bottom_reversal". Multi-week holds, wide trailing,
signal-invalidation exits.
Exit params come from THIS module (per category), NOT the user.
BYPASSES the Trump regime filter — those gates actively reject valid
reversal setups (a reclaim day IS a >5% move; reversals happen on
volatility EXPANSION not contraction).
The two systems share only the low-level execution plumbing (HL connector,
position monitor, reconciler, DB). Everything strategic is separated here.
"""
from __future__ import annotations
from dataclasses import dataclass
from typing import Optional
# Sources that belong to System 2. Everything else is either System 1 ("truth")
# or unsupported for live trading.
SYSTEM_2_SOURCES = {
"btc_bottom_reversal",
}
SYSTEM_1_SOURCES = {"truth"}
SUPPORTED_TRADING_SOURCES = SYSTEM_1_SOURCES | SYSTEM_2_SOURCES
def is_system_2(source: str) -> bool:
"""True if this signal should run through the reversal pipeline."""
return (source or "").lower() in SYSTEM_2_SOURCES
def is_supported_trading_source(source: str) -> bool:
"""Fail closed for unknown ingest sources instead of treating them as Trump."""
return (source or "").lower() in SUPPORTED_TRADING_SOURCES
def system2_display_name() -> str:
return "Bitcoin Bottom"
def system2_min_confidence() -> int:
return 85
# ── System 1 (Trump) — REPOSITIONED ─────────────────────────────────────────
# Originally "high-frequency, many small bets". Repositioned per operator
# decision to: LOW frequency, SELECTIVE, tight stop, ≥30-min holds.
#
# - Don't trade every post. A Trump trade puts the system on cooldown so
# the next N hours of posts are ignored — only the FIRST qualifying
# high-conviction post in a quiet window gets acted on.
# - Raise the confidence floor: only the very top posts clear the bar.
# - Tight stop, ALWAYS active (capital protection — a post that triggers
# an immediate -1.5% means we read it wrong, get out).
# - But on the winning side, hold ≥30 min: suppress take-profit / trailing
# exits until the floor elapses so we don't scalp out of a developing
# move. Asymmetric by design: losers cut fast, winners get room.
TRUMP_MIN_CONFIDENCE = 88 # was effectively the user's 70-85 setting
TRUMP_COOLDOWN_HOURS = 12 # after a Trump trade, ignore further Trump
# signals this long (the "don't trade every
# opportunity" lever)
TRUMP_MIN_HOLD_MINUTES = 30 # no TP / trailing exit before this; hard SL
# stays active throughout
TRUMP_STOP_LOSS_PCT = 1.5 # tight, kept
TRUMP_MAX_HOLD_HOURS = 6 # backstop force-close
@dataclass(frozen=True)
class CategoryExit:
"""Exit profile for a System-2 signal category.
These OVERRIDE the user's per-subscription exit params. Rationale: the
user picks risk for the Trump scalp; the reversal system's risk is a
property of the SIGNAL TYPE (an RSI capitulation reversal needs 60 days
to play out no matter what the user set their Trump stop-loss to).
Fields:
stop_loss_pct : hard stop, % adverse in position direction
trailing_activate_at_pct : peak gain that arms the trailing stop
trailing_stop_pct : retrace-from-peak that closes once armed
max_hold_hours : backstop force-close
time_stop_hours : if position is still ~flat after this many
hours, the thesis is slow/dead — close.
None = no time stop (pure trend capture).
invalidation : symbolic fast-exit condition. Interpreted
by tp_sl_monitor. None = stop-loss only.
"below_entry" → exit if price crosses back
through the entry (signal thesis dead).
"""
stop_loss_pct: float
trailing_activate_at_pct: Optional[float]
trailing_stop_pct: Optional[float]
max_hold_hours: int
time_stop_hours: Optional[int] = None
invalidation: Optional[str] = None
# Staged stop-loss ladder (System-2, optional). A list of
# (peak_gain_trigger_pct, new_stop_floor_pct) rungs. As the trade's PEAK
# gain crosses each trigger, the stop FLOOR ratchets up to the rung's
# value (signed gain% vs entry: negative = still a loss, 0 = breakeven,
# positive = locked-in profit). The position is NEVER closed for hitting
# a profit target — it only ever exits when price falls back to the
# current staged floor (or max-hold). This is a pure staged stop, not a
# take-profit and not a from-peak trailing stop. When set, it OVERRIDES
# take_profit_pct + trailing_* for that trade.
stop_ladder: Optional[list] = None
# ── Per-category exit profiles ──────────────────────────────────────────────
# Tuned to each signal's natural time horizon. These are STARTING points —
# refine against forward-test data, not intuition.
_CATEGORY_EXITS: dict[str, CategoryExit] = {
# Weekly RSI capitulation reversal — slowest, biggest target.
"rsi_extreme_reversal": CategoryExit(
stop_loss_pct=8.0, trailing_activate_at_pct=15.0,
trailing_stop_pct=6.0, max_hold_hours=1440, # 60 days
),
# 200d SMA reclaim — trend change. Fast exit if it loses the SMA again.
"sma_reclaim": CategoryExit(
stop_loss_pct=6.0, trailing_activate_at_pct=12.0,
trailing_stop_pct=5.0, max_hold_hours=2160, # 90 days
invalidation="below_entry", # reclaim failed → thesis dead
),
# Funding extreme unwind — faster, days not weeks.
"funding_extreme_reversal": CategoryExit(
stop_loss_pct=4.0, trailing_activate_at_pct=10.0,
trailing_stop_pct=5.0, max_hold_hours=720, # 30 days
time_stop_hours=240, # 10d flat → squeeze didn't materialise
),
# BTC bottom-reversal long — 2-of-3 price confluence (AHR999 + 200WMA +
# Pi Cycle Bottom). NO take-profit, NO from-peak trailing. The ONLY exit
# is a STAGED stop-loss ("阶段止损") that ratchets up as the trade's peak
# gain crosses each rung, plus the 120-day max-hold backstop.
#
# The BASE catastrophic floor is NOT fixed here — it is derived per-trade
# from the chosen System-2 leverage (sys2_protective_stop_pct), so it
# always sits inside the exchange liquidation line. These rungs only
# ratchet that floor UP as peak gain grows (they never loosen it):
# peak ≥ 20% → stop -12%
# peak ≥ 40% → stop 0% (breakeven — free trade)
# peak ≥ 70% → stop +25% (lock profit)
# peak ≥ 110% → stop +55%
# peak ≥ 160% → stop +95%
#
# Rungs are deliberately far apart so normal post-bottom volatility does
# not knock it out, and it never sells just because a target was hit.
"btc_bottom_reversal_long": CategoryExit(
stop_loss_pct=35.0, # fallback only; bot_engine overrides per-lev
trailing_activate_at_pct=None,
trailing_stop_pct=None,
max_hold_hours=12960, # 18 months — a cycle bull runs 6–18mo;
# the ratchet/peak-trail is the real exit,
# the clock is just a far backstop.
invalidation=None,
stop_ladder=[
(20.0, -12.0),
(40.0, 0.0),
(70.0, 25.0),
(110.0, 55.0),
(160.0, 95.0),
],
),
# VCP breakout — short-term continuation.
"vcp_breakout": CategoryExit(
stop_loss_pct=3.0, trailing_activate_at_pct=6.0,
trailing_stop_pct=2.5, max_hold_hours=168, # 7 days
time_stop_hours=48,
),
}
# Fallback for a System-2 signal whose category isn't explicitly mapped —
# conservative medium profile so an un-mapped scanner doesn't trade huge.
_SYSTEM_2_DEFAULT = CategoryExit(
stop_loss_pct=5.0, trailing_activate_at_pct=10.0,
trailing_stop_pct=4.0, max_hold_hours=336, # 14 days
time_stop_hours=120,
)
def get_exit_profile(category: Optional[str]) -> CategoryExit:
"""Resolve a System-2 category to its exit profile. Unknown → default."""
if not category:
return _SYSTEM_2_DEFAULT
return _CATEGORY_EXITS.get(category.lower(), _SYSTEM_2_DEFAULT)
# ── System-2 dynamic leverage ───────────────────────────────────────────────
# The user picks System-2 leverage freely. The protective stop is then
# auto-scaled to stay INSIDE the exchange liquidation line, so the position
# is de-risked by our own monitor and is never liquidated by the exchange.
#
# liquidation distance (price move) ≈ 100 / leverage (%)
# protective full-exit stop = 85% of that (15% maint/fee/funding buffer)
# …but never wider than SYS2_MAX_STOP_PCT — the bottom thesis only needs to
# tolerate a ~30% wick; risking more buys nothing.
#
# Net effect:
# lev ≤ 2x → stop = 35% (full bottom-wick tolerance, the original design)
# lev = 3x → stop ≈ 28%
# lev = 5x → stop ≈ 17%
# lev = 10x→ stop ≈ 8.5% (you WILL get shaken out by a normal wick — shown
# to the user as the explicit cost of high leverage)
SYS2_DEFAULT_LEVERAGE = 2
SYS2_MIN_LEVERAGE = 1
SYS2_MAX_LEVERAGE = 10
SYS2_MAX_STOP_PCT = 35.0
SYS2_LIQ_BUFFER = 0.85 # exit at 85% of the way to liquidation
# ── System-2 risk MODE ──────────────────────────────────────────────────────
# Two opt-in profiles, frozen onto the trade at signal time. STANDARD is the
# tuned cycle-rider (low leverage, survive bull corrections). AGGRESSIVE is a
# separately-funded high-risk/high-explosiveness sleeve: high leverage, heavier
# + earlier pyramiding, wider peak-trail, lighter early de-risk. BOTH keep the
# two safety invariants: (1) final de-risk rung is a full close INSIDE the
# liquidation line — never exchange-liquidated; (2) post-pyramid breakeven
# floor — a winner can't become a loser.
SYS2_MODE_STANDARD = "standard"
SYS2_MODE_AGGRESSIVE = "aggressive"
SYS2_MODES = (SYS2_MODE_STANDARD, SYS2_MODE_AGGRESSIVE)
# Default leverage when the user hasn't set an explicit sys2_leverage.
SYS2_MODE_DEFAULT_LEV = {SYS2_MODE_STANDARD: 2, SYS2_MODE_AGGRESSIVE: 8}
def sys2_normalize_mode(mode: Optional[str]) -> str:
m = (mode or "").strip().lower()
return m if m in SYS2_MODES else SYS2_MODE_STANDARD
def sys2_effective_leverage(value: Optional[int],
mode: Optional[str] = SYS2_MODE_STANDARD) -> int:
"""Resolve + clamp System-2 leverage. None → the mode's default
(standard 2×, aggressive 8×). Always clamped to [1, 10]."""
m = sys2_normalize_mode(mode)
default = SYS2_MODE_DEFAULT_LEV[m]
if value is None:
return default
try:
v = int(value)
except (TypeError, ValueError):
return default
return max(SYS2_MIN_LEVERAGE, min(SYS2_MAX_LEVERAGE, v))
def sys2_protective_stop_pct(leverage: int) -> float:
"""Protective full-exit distance (positive %), auto-scaled to leverage so
it always triggers INSIDE the exchange liquidation line."""
lev = max(SYS2_MIN_LEVERAGE, min(SYS2_MAX_LEVERAGE, int(leverage)))
liq_distance_pct = 100.0 / lev
return round(min(SYS2_MAX_STOP_PCT, SYS2_LIQ_BUFFER * liq_distance_pct), 2)
def sys2_approx_liquidation_pct(leverage: int) -> float:
"""Rough exchange liquidation distance (positive %) for UX display."""
lev = max(SYS2_MIN_LEVERAGE, min(SYS2_MAX_LEVERAGE, int(leverage)))
return round(100.0 / lev, 2)
# Staged de-risk ("分段式减仓"): instead of one full exit at the protective
# stop, scale OUT in three steps as the trade moves against us. The final
# step is exactly the Phase-1 protective level P (inside liquidation), so the
# never-exchange-liquidated guarantee is preserved — we just bleed out in
# thirds rather than all at once.
#
# −0.60·P → close 1/3 of the original notional
# −0.80·P → close another 1/3
# −1.00·P → close the remaining 1/3 (full exit; same safety as Phase 1)
SYS2_DERISK_FRACTIONS = (0.60, 0.80, 1.00)
# Pyramiding ("做对了往上加仓"): when the bottom call is RIGHT and the trend
# confirms, scale INTO the winner to amplify the move. Mirror of the de-risk
# ladder. Conservative sizing (user-selected): adds at most +0.6× the base
# notional. Each rung requires BOTH a peak-gain trigger AND a structural
# trend confirmation (price ≥ 200d SMA AND at a fresh local high), checked at
# execution time. Pyramiding is DISABLED once any de-risk step has fired
# (a trade that went underwater is not a clean uptrend to add into).
# Extended for a cycle bull: deeper continuation rungs so a multi-x move is
# actually scaled. Each add still needs structural confirmation (price ≥ 200d
# SMA AND a fresh high) so the deep rungs only fire in a genuine sustained
# uptrend, never on a chop fakeout. Total adds ≤ +0.75× base — still well
# inside the 8× notional cap; amplification stays conservative.
SYS2_ADDON_PEAK_TRIGGERS = (25.0, 50.0, 85.0, 140.0, 220.0) # peak-gain % vs blended entry
SYS2_ADDON_FRACTIONS = (0.30, 0.20, 0.10, 0.10, 0.05) # of ORIGINAL base notional
# AGGRESSIVE sleeve: earlier + heavier adds (≤ +1.50× base), so a clean
# multi-x run is meaningfully compounded. Still gated by the same structural
# confirmation (≥200d SMA + fresh high), still inside the per-trade 8×
# notional cap, still floored at breakeven once any add fills.
SYS2_AGGR_ADDON_PEAK_TRIGGERS = (15.0, 35.0, 60.0, 100.0, 160.0)
SYS2_AGGR_ADDON_FRACTIONS = (0.50, 0.40, 0.30, 0.20, 0.10)
# AGGRESSIVE de-risk: shed less early (¼/¼) so a normal correction doesn't
# gut the runner; final rung still a FULL close at the protective level.
SYS2_AGGR_DERISK_STEP_FRACS = (0.25, 0.25, 0.50)
SYS2_STD_DERISK_STEP_FRACS = (1.0 / 3.0, 1.0 / 3.0, 1.0 / 3.0)
# Peak-% trailing for the parabolic top. Below the start threshold the fixed
# stop_ladder rungs govern; at/above it the floor also trails the PEAK PRICE
# by at most SYS2_PEAK_TRAIL_DD (price drawdown, scale-invariant) so a +500%
# move isn't capped at the +95% rung, while a normal ~25–30% bull correction
# still doesn't knock it out. Floor = max(rung floor, peak-trail floor).
SYS2_PEAK_TRAIL_START_PCT = 80.0 # activate once peak gain ≥ +80%
SYS2_PEAK_TRAIL_DD = 0.30 # give back at most 30% of peak PRICE
# AGGRESSIVE: let it run longer before the trailing top kicks in, and give
# back more, so a multi-x parabolic isn't cut early by a mid-run shakeout.
SYS2_AGGR_PEAK_TRAIL_START_PCT = 60.0
SYS2_AGGR_PEAK_TRAIL_DD = 0.42
def sys2_addon_ladder(mode: Optional[str] = SYS2_MODE_STANDARD) -> list:
"""Pyramiding rungs: (peak_gain_trigger_pct, frac_of_base, is_last)."""
if sys2_normalize_mode(mode) == SYS2_MODE_AGGRESSIVE:
trigs, fracs = SYS2_AGGR_ADDON_PEAK_TRIGGERS, SYS2_AGGR_ADDON_FRACTIONS
else:
trigs, fracs = SYS2_ADDON_PEAK_TRIGGERS, SYS2_ADDON_FRACTIONS
n = len(trigs)
return [(trigs[i], fracs[i], i == n - 1) for i in range(n)]
def sys2_peak_trail(mode: Optional[str] = SYS2_MODE_STANDARD) -> tuple:
"""(activate_peak_gain_pct, price_drawdown_frac) for the parabolic-top
trailing floor. Scale-invariant: works the same at +120% or +900%."""
if sys2_normalize_mode(mode) == SYS2_MODE_AGGRESSIVE:
return (SYS2_AGGR_PEAK_TRAIL_START_PCT, SYS2_AGGR_PEAK_TRAIL_DD)
return (SYS2_PEAK_TRAIL_START_PCT, SYS2_PEAK_TRAIL_DD)
def sys2_derisk_ladder(leverage: int,
mode: Optional[str] = SYS2_MODE_STANDARD) -> list:
"""Downside staged de-risk rungs for the given leverage.
Returns a list of (threshold_signed_pct, frac_of_ORIGINAL_to_close,
is_final), ordered by increasing adversity. threshold is NEGATIVE
(a loss in position direction). The executor converts frac-of-original
into frac-of-current using the trade's remaining_fraction.
"""
p = sys2_protective_stop_pct(leverage)
steps = (SYS2_AGGR_DERISK_STEP_FRACS
if sys2_normalize_mode(mode) == SYS2_MODE_AGGRESSIVE
else SYS2_STD_DERISK_STEP_FRACS)
return [
(round(-SYS2_DERISK_FRACTIONS[0] * p, 4), steps[0], False),
(round(-SYS2_DERISK_FRACTIONS[1] * p, 4), steps[1], False),
(round(-SYS2_DERISK_FRACTIONS[2] * p, 4), steps[2], True),
]
def get_stop_ladder(category: Optional[str]) -> Optional[list]:
"""Staged stop-loss ladder for a category, or None if it uses trailing.
Sorted ascending by trigger so the monitor can walk it cheaply. The
ladder is a CODE constant (not frozen per-trade): if it's retuned, open
positions adopt the new rungs on the next price tick / restart. That is
intentional — staged-stop levels are a property of the strategy, not of
an individual fill.
"""
prof = get_exit_profile(category)
if not prof.stop_ladder:
return None
return sorted(prof.stop_ladder, key=lambda r: r[0])
# ── System-2 position sizing ────────────────────────────────────────────────
# CRITICAL: System 2 must NOT use regime_filter.calculate_size_multiplier.
# That function asks "is volatility contracted? has price NOT moved?" — both
# FALSE during a reversal (vol expands, price already moved), which would
# SHRINK our rarest, highest-conviction trades. Sizing here is a function of
# (category conviction × signal confidence), nothing else.
# Base conviction per category. Rarer + cleaner setup → bigger base bet.
_CATEGORY_SIZE_BASE: dict[str, float] = {
"rsi_extreme_reversal": 2.5, # ~1-2×/yr/asset, deepest capitulation
"sma_reclaim": 2.0, # clean trend-change marker
"funding_extreme_reversal": 1.4, # more frequent, choppier unwinds
"btc_bottom_reversal_long": 2.3, # 2-of-3 price confluence (AHR999/200WMA/Pi)
"vcp_breakout": 1.0, # continuation, lowest conviction
}
_SYSTEM_2_SIZE_BASE_DEFAULT = 1.2
SYSTEM_2_SIZE_CAP = 4.0
# ── System-2 correlation / concentration cap ────────────────────────────────
# Every asset in REVERSAL_BASKET is high-beta crypto. When the market bottoms,
# RSI/SMA/funding reversals fire on BTC + ETH + SOL in the SAME week — these
# are NOT independent bets, they're one "crypto reversed" thesis. With Fix #1
# sizing each up to 4x, 3 correlated positions = ~10x effective exposure to a
# single macro call. Treat the whole System-2 book as one correlated bucket
# and cap it.
#
# - SYS2_MAX_CONCURRENT: at most this many open System-2 positions at once.
# Beyond this you're not diversifying, just leveraging the same thesis.
# - SYS2_MAX_OPEN_NOTIONAL_MULT: total open System-2 notional must stay
# under this × the wallet's base position_size_usd × default-ish size.
# Acts as a $ ceiling independent of how many positions.
SYS2_MAX_CONCURRENT = 3
SYS2_MAX_OPEN_NOTIONAL_MULT = 8.0 # × base position_size_usd
def system2_size_multiplier(category: Optional[str], confidence: int) -> float:
"""Position multiplier for a System-2 signal.
base(category) × confidence_scale, capped at SYSTEM_2_SIZE_CAP.
confidence_scale: 70→1.0, 85→1.3, 95→1.5, 100→1.6 (linear, floored at 1.0).
A scanner that emits confidence 88 for an rsi_extreme_reversal therefore
sizes 2.5 × 1.36 ≈ 3.4×. Tune the base table against forward-test data.
"""
base = _CATEGORY_SIZE_BASE.get((category or "").lower(), _SYSTEM_2_SIZE_BASE_DEFAULT)
conf_scale = 1.0 + max(0, confidence - 70) / 50.0
return round(min(base * conf_scale, SYSTEM_2_SIZE_CAP), 2)
# ── BTC bottom-reversal ─────────────────────────────────────────────────────
# The old MVRV-Z / STH-SOPR / drawdown state machine was REMOVED. It depended
# on paid on-chain data and was over-engineered. The strategy is now a pure
# 2-of-3 price confluence (AHR999 + 200-Week MA + Pi Cycle Bottom), implemented
# in app/services/bottom_indicators.py and driven by the btc_bottom_reversal
# scanner. There is no state machine and no on-chain dependency.
app/services/telegram.py
+280
View File
@@ -0,0 +1,280 @@
"""
Telegram push alerts — send + signal dispatcher.
This module is fire-and-forget by design. `notify_signal()` returns immediately
to the caller (signal ingestion path) and dispatches in a background task. A
DB failure or Telegram API error MUST NOT block a signal from being saved.
The bot token is loaded from `settings.telegram_bot_token`. If empty, every
function in this module degrades into a no-op (and logs once at module load).
notify_signal(post) ← called from /api/signals/ingest
send_test_message(wallet) ← called from /api/telegram/test
send_message(chat_id, text) ← low-level escape hatch
Source → user-toggle mapping:
truth → alert_trump
btc_bottom_reversal → alert_btc_bottom
funding_reversal → alert_funding
kol_divergence (future) → alert_kol_divergence
"""
from __future__ import annotations
import asyncio
import logging
from datetime import datetime, timezone
from typing import Optional
import httpx
from sqlalchemy import select, update
from app.config import settings
from app.database import AsyncSessionLocal as async_session
from app.models import Post, TelegramBinding
logger = logging.getLogger(__name__)
TG_API = "https://api.telegram.org/bot{token}/{method}"
# Telegram caps a single message at 4096 chars; we render way below this.
MAX_LEN = 3500
# ── Source → preference column mapping ────────────────────────────────────
def _pref_column_for_source(source: str) -> Optional[str]:
"""Which user-toggle column gates this source. None → unknown source,
don't send."""
if source == "truth":
return "alert_trump"
if source == "btc_bottom_reversal":
return "alert_btc_bottom"
if source == "funding_reversal":
return "alert_funding"
if source == "kol_divergence":
return "alert_kol_divergence"
return None
def _is_in_mute_window(now_hour: int, mute_from: Optional[int],
mute_until: Optional[int]) -> bool:
"""Both null → never mute. start<until → mute inside [start, until).
start>until → window wraps midnight (e.g. 23..7 → mute 23, 0–6)."""
if mute_from is None or mute_until is None:
return False
if mute_from == mute_until:
return False
if mute_from < mute_until:
return mute_from <= now_hour < mute_until
# wraps midnight
return now_hour >= mute_from or now_hour < mute_until
# ── Formatting ────────────────────────────────────────────────────────────
def _signal_emoji(post: Post) -> str:
if post.signal == "buy":
return "🟢"
if post.signal == "short":
return "🔴"
return "⚪"
def _source_label(source: str) -> str:
return {
"truth": "Trump · Truth Social",
"btc_bottom_reversal": "BTC · Macro Bottom",
"funding_reversal": "BTC · Funding Reversal",
"kol_divergence": "KOL · Talks vs Trades",
}.get(source, source)
def format_post(post: Post) -> str:
"""Render a Post into a single Telegram message (HTML parse mode).
Keep it scannable: heading, one-line verdict, the underlying text, link."""
emoji = _signal_emoji(post)
src = _source_label(post.source)
sig = (post.signal or "noise").upper()
asset = post.target_asset or "?"
conf = post.ai_confidence or 0
# Heading: emoji + asset + direction + confidence
head = f"{emoji} <b>{asset} · {sig}</b> · conf <b>{conf}</b>"
sub = f"<i>{src}</i>"
body = (post.text or "").strip()
if len(body) > 600:
body = body[:600].rstrip() + "…"
# Move size hint if present (BTC bottom & funding emit expected_move_pct)
extra = ""
if post.expected_move_pct is not None:
extra = f"\n📐 expected move: <b>{post.expected_move_pct:+.1f}%</b>"
if post.invalidation_price is not None:
extra += f"\n🛑 invalidation @ <code>{post.invalidation_price:g}</code>"
# Deep-link back to the dashboard. Frontend URL comes from settings.
fe = (settings.frontend_url or "").rstrip("/")
link = ""
if fe:
# Use the section that matches the source
path = {
"truth": "/en/trump",
"btc_bottom_reversal": "/en/btc",
"funding_reversal": "/en/btc",
"kol_divergence": "/en/kol",
}.get(post.source, "/en")
link = f'\n\n<a href="{fe}{path}">→ open in dashboard</a>'
msg = f"{head}\n{sub}\n\n{body}{extra}{link}"
return msg[:MAX_LEN]
# ── Low-level HTTP ────────────────────────────────────────────────────────
async def send_message(chat_id: int, text: str, *,
parse_mode: str = "HTML",
disable_preview: bool = True) -> bool:
"""Single HTTP POST to Telegram Bot API. Returns True on 200, False on
any failure (caller decides whether to bump the failure counter)."""
token = settings.telegram_bot_token
if not token:
return False
url = TG_API.format(token=token, method="sendMessage")
try:
async with httpx.AsyncClient(timeout=10) as client:
r = await client.post(url, json={
"chat_id": chat_id, "text": text,
"parse_mode": parse_mode,
"disable_web_page_preview": disable_preview,
})
if r.status_code != 200:
logger.warning("Telegram sendMessage failed chat=%d status=%d body=%s",
chat_id, r.status_code, r.text[:200])
return False
return True
except Exception as exc:
logger.warning("Telegram sendMessage exception chat=%d: %s", chat_id, exc)
return False
# ── Dispatcher ────────────────────────────────────────────────────────────
async def _dispatch(post_id: int) -> None:
"""Fan-out a single Post to every eligible subscriber. Always runs in
its own DB session so signal ingestion's session is unaffected."""
if not settings.telegram_bot_token:
return
async with async_session() as db:
post = await db.get(Post, post_id)
if not post:
logger.warning("Telegram dispatch: post id=%d not found", post_id)
return
# Only fan out for actionable signals (not NOISE / null)
if not post.signal or post.signal not in ("buy", "short"):
return
pref_col = _pref_column_for_source(post.source)
if pref_col is None:
logger.debug("Telegram: unknown source %r — not fanning out", post.source)
return
# Build the query: alerts_enabled + the relevant per-source flag.
# min_confidence applies to every source — scanner-emitted signals
# carry their own confidence in the Post.ai_confidence column.
col = getattr(TelegramBinding, pref_col)
base_filters = [
TelegramBinding.alerts_enabled.is_(True),
col.is_(True),
]
# Only apply confidence gate when the post has a real score (> 0).
# Scanner-generated signals (funding, BTC) always carry a score, but
# a Truth-Social post might be dispatched before Claude scores it (score=0).
# In that edge case we let it through so no alert is silently swallowed.
if post.ai_confidence and post.ai_confidence > 0:
base_filters.append(TelegramBinding.min_confidence <= post.ai_confidence)
q = select(TelegramBinding).where(*base_filters)
bindings = (await db.execute(q)).scalars().all()
if not bindings:
return
text = format_post(post)
now = datetime.now(timezone.utc)
hour = now.hour
sent = 0
for b in bindings:
if _is_in_mute_window(hour, b.mute_from_hour, b.mute_until_hour):
continue
ok = await send_message(b.chat_id, text)
# Update audit counters per binding. Single UPDATE per row keeps
# us out of trouble if one user blocks the bot.
if ok:
await db.execute(
update(TelegramBinding)
.where(TelegramBinding.id == b.id)
.values(
last_alert_at=now,
total_alerts_sent=TelegramBinding.total_alerts_sent + 1,
)
)
sent += 1
else:
await db.execute(
update(TelegramBinding)
.where(TelegramBinding.id == b.id)
.values(
total_alerts_failed=TelegramBinding.total_alerts_failed + 1,
)
)
await db.commit()
logger.info("Telegram fan-out: post=%d source=%s sent=%d/%d",
post_id, post.source, sent, len(bindings))
def notify_signal(post: Post) -> None:
"""Fire-and-forget. Schedules `_dispatch(post.id)` on the running loop
and returns immediately. Safe to call from any async context — falls
back to a no-op if telegram is disabled or no event loop is running.
We pass post.id rather than the post object because the caller's DB
session might close before our background task runs."""
if not settings.telegram_bot_token:
return
if not post or not post.id:
return
try:
asyncio.create_task(_dispatch(post.id))
except RuntimeError:
# No running loop — extremely unusual in our FastAPI context.
logger.warning("notify_signal: no running event loop, skipping post=%s", post.id)
async def send_test_message(wallet: str) -> bool:
"""Triggered from the Settings UI to verify the binding works end-to-end."""
if not settings.telegram_bot_token:
return False
async with async_session() as db:
b = (await db.execute(
select(TelegramBinding).where(TelegramBinding.wallet_address == wallet.lower())
)).scalar_one_or_none()
if not b:
return False
return await send_message(
b.chat_id,
"✅ <b>Trump Alpha connected.</b>\n\n"
"You'll get push alerts here when signals fire. "
"Use /trump /btc /funding /kol /conf /quiet in this chat to tune "
"which sources and thresholds apply to you.",
)
if not settings.telegram_bot_token:
logger.info("Telegram bot token not set — push alerts disabled.")
app/services/telegram_bot.py
+564
View File
@@ -0,0 +1,564 @@
"""
Telegram bot worker — long-polls getUpdates and handles user commands.
Commands supported:
/start CODE → bind this chat_id to the wallet that owns CODE
/start → friendly hello with instructions
/stop → disable alerts (keeps the binding so re-enable is one tap)
/status → show binding status and current preferences
/test → send self a test message to verify formatting
This runs as a single background asyncio task started from main.py lifespan.
Only one instance should run at a time — Telegram's long-poll model assumes
exactly one consumer per bot token. If you horizontally scale the API, switch
to webhook mode (see set_webhook in the Bot API).
The one-time binding codes live in a process-local dict with a 10-minute
TTL. On API restart all pending codes evaporate (the user re-clicks Connect).
This is intentional — codes never touch the DB so a compromised dump can't
hijack future bindings.
"""
from __future__ import annotations
import asyncio
import logging
import secrets
import string
from datetime import datetime, timedelta, timezone
from typing import Optional
import httpx
from sqlalchemy import select, update
from app.config import settings
from app.database import AsyncSessionLocal as async_session
from app.models import TelegramBinding, Subscription
from app.services.telegram import send_message, TG_API
logger = logging.getLogger(__name__)
# ── One-time binding codes ────────────────────────────────────────────────
_CODE_TTL_SECONDS = 10 * 60
_CODE_ALPHABET = string.ascii_uppercase + string.digits # 36^6 ≈ 2.1B
_CODE_LEN = 6
# code → (wallet_lower, expires_at)
_pending_codes: dict[str, tuple[str, datetime]] = {}
def _purge_expired_codes() -> None:
now = datetime.now(timezone.utc)
expired = [c for c, (_, exp) in _pending_codes.items() if exp < now]
for c in expired:
_pending_codes.pop(c, None)
def issue_binding_code(wallet: str) -> str:
"""Generate a fresh 6-char code for a wallet. If the same wallet calls
twice within the TTL, the old code is invalidated — only the latest
code works (prevents stale shared links)."""
_purge_expired_codes()
wallet_l = wallet.lower()
# Invalidate any previous code for this wallet
for c, (w, _) in list(_pending_codes.items()):
if w == wallet_l:
_pending_codes.pop(c, None)
code = "".join(secrets.choice(_CODE_ALPHABET) for _ in range(_CODE_LEN))
exp = datetime.now(timezone.utc) + timedelta(seconds=_CODE_TTL_SECONDS)
_pending_codes[code] = (wallet_l, exp)
return code
def _consume_code(code: str) -> Optional[str]:
"""Resolve and remove a code. Returns the wallet, or None if invalid/expired."""
_purge_expired_codes()
rec = _pending_codes.pop(code.upper(), None)
if rec is None:
return None
return rec[0]
# ── Command handlers ──────────────────────────────────────────────────────
HELP_TEXT = (
"👋 <b>Trump Alpha bot</b>\n\n"
"Push alerts for high-conviction crypto signals — Trump posts, BTC bottom "
"reversals, funding extremes, KOL divergence.\n\n"
"<b>Just press /start</b> — you're subscribed. No wallet needed.\n\n"
"<b>Configure:</b>\n"
" /trump on|off Trump · Truth Social posts\n"
" /btc on|off BTC · macro bottom\n"
" /funding on|off BTC · funding reversal\n"
" /kol on|off KOL · talks vs trades\n"
" /conf 0-100 min AI confidence (Trump only)\n"
" /quiet 23 7 mute hours UTC (or 'off' to disable)\n\n"
"<b>Status &amp; control:</b>\n"
" /status — show your preferences\n"
" /stop — pause all alerts\n"
" /test — send a sample alert\n\n"
"<b>Pro features</b> (auto-trade on Hyperliquid):\n"
"Open the dashboard → <i>Settings</i> → <i>Connect Telegram</i> to link "
"your wallet."
)
# Default preferences for a freshly-created walletless binding.
_DEFAULT_PREFS = {
"alerts_enabled": True,
"alert_trump": True,
"alert_btc_bottom": True,
"alert_funding": True,
"alert_kol_divergence": False,
"min_confidence": 70,
}
async def _ensure_binding(chat_id: int, username: Optional[str]) -> TelegramBinding:
"""Create or refresh a walletless binding for this chat. Returns the
binding (always with id set)."""
async with async_session() as db:
existing = (await db.execute(
select(TelegramBinding).where(TelegramBinding.chat_id == chat_id)
)).scalar_one_or_none()
if existing:
# Refresh username + re-enable alerts on subsequent /start
await db.execute(
update(TelegramBinding)
.where(TelegramBinding.id == existing.id)
.values(tg_username=username, alerts_enabled=True)
)
await db.commit()
return existing
b = TelegramBinding(
wallet_address=None,
chat_id=chat_id,
tg_username=username,
**_DEFAULT_PREFS,
)
db.add(b)
await db.commit()
await db.refresh(b)
return b
async def _cmd_start(chat_id: int, username: Optional[str], arg: str) -> None:
"""
/start → free-tier walletless binding (anyone can subscribe)
/start CODE → upgrade to wallet-linked binding (Pro features)
"""
arg = arg.strip()
# No arg → free-tier subscribe. Always idempotent.
if not arg:
b = await _ensure_binding(chat_id, username)
if b.wallet_address:
await send_message(
chat_id,
"✅ Already linked — you're getting alerts.\n"
"Send /status to see your settings, /help for commands.",
)
else:
await send_message(
chat_id,
"🎉 <b>You're subscribed!</b>\n\n"
"You'll get alerts for: Trump posts, BTC bottom signals, "
"funding reversals. KOL divergence is off by default (noisier).\n\n"
"Type /help to see every command, or /test to preview an alert.",
)
return
# Arg present → wallet-binding flow (Pro)
wallet = _consume_code(arg)
if not wallet:
await send_message(
chat_id,
"❌ <b>Invalid or expired code.</b>\n\n"
"The 6-char code only comes from the dashboard's "
"<i>Settings → Connect Telegram</i> panel — and it expires after "
"10 minutes.\n\n"
"Don't need wallet features? Just send /start (no code) — alerts "
"are free for everyone.",
)
return
async with async_session() as db:
sub = (await db.execute(
select(Subscription).where(Subscription.wallet_address == wallet)
)).scalar_one_or_none()
if not sub or not sub.active:
await send_message(
chat_id,
"⚠️ <b>This wallet isn't subscribed yet.</b>\n\n"
"Open Settings on the dashboard, click <i>Start trading</i>, "
"then come back and re-connect.",
)
return
existing_by_wallet = (await db.execute(
select(TelegramBinding).where(TelegramBinding.wallet_address == wallet)
)).scalar_one_or_none()
existing_by_chat = (await db.execute(
select(TelegramBinding).where(TelegramBinding.chat_id == chat_id)
)).scalar_one_or_none()
# Edge: chat already bound to a DIFFERENT wallet — reject loudly.
if existing_by_chat and existing_by_chat.wallet_address and \
existing_by_chat.wallet_address != wallet:
await send_message(
chat_id,
"❌ This Telegram account is already bound to a different "
"wallet. Run /stop on that wallet first (or unbind via the "
"dashboard).",
)
return
if existing_by_wallet and existing_by_wallet.chat_id != chat_id:
# Wallet was previously bound to a different chat — move it here.
# Also clean up any walletless binding for this chat to avoid a
# duplicate row.
if existing_by_chat:
from sqlalchemy import delete as _delete
await db.execute(_delete(TelegramBinding).where(
TelegramBinding.id == existing_by_chat.id))
await db.execute(
update(TelegramBinding)
.where(TelegramBinding.id == existing_by_wallet.id)
.values(chat_id=chat_id, tg_username=username,
alerts_enabled=True,
bound_at=datetime.now(timezone.utc))
)
elif existing_by_chat:
# Free user is upgrading to wallet-linked — just attach the wallet
# to their existing walletless binding, keep their preferences.
await db.execute(
update(TelegramBinding)
.where(TelegramBinding.id == existing_by_chat.id)
.values(wallet_address=wallet, tg_username=username,
alerts_enabled=True,
bound_at=datetime.now(timezone.utc))
)
else:
# Brand-new wallet + chat.
db.add(TelegramBinding(
wallet_address=wallet, chat_id=chat_id,
tg_username=username, **_DEFAULT_PREFS,
))
await db.commit()
short = wallet[:6] + "…" + wallet[-4:]
await send_message(
chat_id,
f"✅ <b>Wallet linked: {short}</b>\n\n"
"Pro features unlocked. Your existing alert preferences are kept. "
"Manage everything from the dashboard's Settings page or via bot "
"commands. Send /status to see your current setup.",
)
# ── Preference commands ──────────────────────────────────────────────────
async def _set_pref(chat_id: int, field: str, value: bool | int) -> bool:
"""Returns True if the binding existed and was updated."""
async with async_session() as db:
r = await db.execute(
update(TelegramBinding)
.where(TelegramBinding.chat_id == chat_id)
.values(**{field: value})
)
await db.commit()
return r.rowcount > 0
def _parse_on_off(arg: str) -> Optional[bool]:
a = arg.strip().lower()
if a in ("on", "yes", "enable", "1", "true"): return True
if a in ("off", "no", "disable", "0", "false"): return False
return None
async def _cmd_toggle(chat_id: int, field: str, label: str, arg: str) -> None:
v = _parse_on_off(arg)
if v is None:
await send_message(chat_id,
f"Usage: <code>{label.lower().split()[0]} on</code> or "
f"<code>{label.lower().split()[0]} off</code>")
return
ok = await _set_pref(chat_id, field, v)
if not ok:
await send_message(chat_id, "No binding here. Send /start first.")
return
state = "🟢 ON" if v else "🔴 OFF"
await send_message(chat_id, f"{state} · {label}")
async def _cmd_conf(chat_id: int, arg: str) -> None:
try:
n = int(arg.strip())
if not 0 <= n <= 100:
raise ValueError
except ValueError:
await send_message(chat_id,
"Usage: <code>/conf 70</code> (0–100). Only Trump posts above "
"this AI confidence will trigger alerts.")
return
ok = await _set_pref(chat_id, "min_confidence", n)
if not ok:
await send_message(chat_id, "No binding here. Send /start first.")
return
await send_message(chat_id, f"Min confidence set to <b>{n}</b>.")
async def _cmd_quiet(chat_id: int, arg: str) -> None:
a = arg.strip().lower()
if a in ("off", "none", "disable", ""):
async with async_session() as db:
r = await db.execute(
update(TelegramBinding)
.where(TelegramBinding.chat_id == chat_id)
.values(mute_from_hour=None, mute_until_hour=None)
)
await db.commit()
if not r.rowcount:
await send_message(chat_id, "No binding here. Send /start first.")
return
await send_message(chat_id, "🔔 Quiet hours disabled.")
return
parts = arg.split()
if len(parts) != 2:
await send_message(chat_id,
"Usage: <code>/quiet 23 7</code> (UTC, e.g. mute 23:00–07:00) "
"or <code>/quiet off</code>.")
return
try:
a_h, b_h = int(parts[0]), int(parts[1])
if not (0 <= a_h <= 23 and 0 <= b_h <= 23):
raise ValueError
except ValueError:
await send_message(chat_id, "Hours must be integers 0–23.")
return
async with async_session() as db:
r = await db.execute(
update(TelegramBinding)
.where(TelegramBinding.chat_id == chat_id)
.values(mute_from_hour=a_h, mute_until_hour=b_h)
)
await db.commit()
if not r.rowcount:
await send_message(chat_id, "No binding here. Send /start first.")
return
await send_message(chat_id, f"🌙 Quiet hours: {a_h:02d}:00–{b_h:02d}:00 UTC.")
async def _cmd_stop(chat_id: int) -> None:
async with async_session() as db:
r = await db.execute(
update(TelegramBinding)
.where(TelegramBinding.chat_id == chat_id)
.values(alerts_enabled=False)
)
await db.commit()
if r.rowcount:
await send_message(chat_id,
"🔕 Alerts paused. Send /start any time to re-enable.")
else:
await send_message(chat_id,
"You don't have an active binding here. Send /start to set one up.")
async def _cmd_status(chat_id: int) -> None:
async with async_session() as db:
b = (await db.execute(
select(TelegramBinding).where(TelegramBinding.chat_id == chat_id)
)).scalar_one_or_none()
if not b:
await send_message(chat_id,
"Not subscribed yet. Send /start to begin (no wallet required).")
return
if b.wallet_address:
short = b.wallet_address[:6] + "…" + b.wallet_address[-4:]
tier_line = f"Tier: <b>Pro</b> · wallet <code>{short}</code>"
else:
tier_line = "Tier: <b>Free</b> · no wallet linked"
on = "🟢 ON" if b.alerts_enabled else "🔴 OFF"
srcs = []
if b.alert_trump: srcs.append("Trump")
if b.alert_btc_bottom: srcs.append("BTC bottom")
if b.alert_funding: srcs.append("Funding")
if b.alert_kol_divergence: srcs.append("KOL")
src_line = ", ".join(srcs) or "(none — alerts will be silent)"
mute = ""
if b.mute_from_hour is not None and b.mute_until_hour is not None:
mute = f"\n🌙 Quiet hours: {b.mute_from_hour:02d}–{b.mute_until_hour:02d} UTC"
await send_message(
chat_id,
f"📡 <b>Status</b>\n\n"
f"{tier_line}\n"
f"Alerts: {on}\n"
f"Sources: {src_line}\n"
f"Min confidence: {b.min_confidence}{mute}\n\n"
f"Sent: {b.total_alerts_sent} · Failed: {b.total_alerts_failed}\n\n"
f"Toggle anything with /trump /btc /funding /kol /conf /quiet — "
f"send /help for the full list.",
)
async def _cmd_test(chat_id: int) -> None:
async with async_session() as db:
b = (await db.execute(
select(TelegramBinding).where(TelegramBinding.chat_id == chat_id)
)).scalar_one_or_none()
if not b:
await send_message(chat_id, "Send /start first to subscribe (no wallet needed).")
return
await send_message(
chat_id,
"🟢 <b>BTC · BUY</b> · conf <b>88</b>\n"
"<i>Trump · Truth Social</i>\n\n"
"BITCOIN is the FUTURE of money. America will LEAD the world in "
"crypto. BIG things coming very soon!\n\n"
"(this is a test message — your actual alerts will look like this)",
)
# ── Long-poll loop ────────────────────────────────────────────────────────
async def _handle_message(msg: dict) -> None:
chat = msg.get("chat") or {}
chat_id = chat.get("id")
if not chat_id:
return
text = (msg.get("text") or "").strip()
if not text:
return
from_user = msg.get("from") or {}
username = from_user.get("username")
# Parse first word as command
parts = text.split(maxsplit=1)
cmd = parts[0].lower()
arg = parts[1] if len(parts) > 1 else ""
# Normalize /command@botname (group-chat syntax) → /command
if "@" in cmd:
cmd = cmd.split("@", 1)[0]
try:
if cmd == "/start":
await _cmd_start(chat_id, username, arg)
elif cmd in ("/stop", "/pause"):
await _cmd_stop(chat_id)
elif cmd in ("/status", "/me"):
await _cmd_status(chat_id)
elif cmd == "/test":
await _cmd_test(chat_id)
elif cmd in ("/help", "/?"):
await send_message(chat_id, HELP_TEXT)
# ── source toggles ───────────────────────────────────────────
elif cmd == "/trump":
await _cmd_toggle(chat_id, "alert_trump", "Trump alerts", arg)
elif cmd == "/btc":
await _cmd_toggle(chat_id, "alert_btc_bottom", "BTC bottom alerts", arg)
elif cmd == "/funding":
await _cmd_toggle(chat_id, "alert_funding", "Funding reversal alerts", arg)
elif cmd == "/kol":
await _cmd_toggle(chat_id, "alert_kol_divergence", "KOL divergence alerts", arg)
# ── numeric / range prefs ────────────────────────────────────
elif cmd == "/conf":
await _cmd_conf(chat_id, arg)
elif cmd == "/quiet":
await _cmd_quiet(chat_id, arg)
else:
if cmd.startswith("/"):
await send_message(chat_id, "Unknown command. Send /help for the list.")
except Exception:
logger.exception("Telegram handler failed for chat=%s text=%r", chat_id, text[:80])
async def run_bot_loop() -> None:
"""Long-poll forever. Idempotent on offsets — Telegram serves the same
update twice only if we don't acknowledge it. Sleep on errors to avoid
hot-looping if the network is down."""
token = settings.telegram_bot_token
if not token:
logger.info("Telegram bot loop not started — TELEGRAM_BOT_TOKEN empty.")
return
logger.info("Telegram bot loop starting (long-poll mode).")
offset: Optional[int] = None
backoff = 1.0
while True:
try:
url = TG_API.format(token=token, method="getUpdates")
params: dict = {"timeout": 25}
if offset is not None:
params["offset"] = offset
async with httpx.AsyncClient(timeout=35) as client:
r = await client.get(url, params=params)
if r.status_code != 200:
logger.warning("Telegram getUpdates HTTP %d: %s", r.status_code, r.text[:200])
await asyncio.sleep(backoff)
backoff = min(backoff * 2, 30)
continue
backoff = 1.0
data = r.json()
for upd in data.get("result", []):
offset = upd["update_id"] + 1
msg = upd.get("message") or upd.get("edited_message")
if msg:
await _handle_message(msg)
except asyncio.CancelledError:
logger.info("Telegram bot loop cancelled.")
raise
except httpx.ReadTimeout:
# Normal — long-poll returned with no updates within timeout. Just retry.
# (Previously caught by the bare Exception below and logged with an
# empty message — "error: — sleeping" — which was opaque.)
continue
except (httpx.ConnectError, httpx.ConnectTimeout, httpx.RemoteProtocolError) as exc:
# Network-level error — expected on transient connectivity issues.
# Compact log to avoid spamming when the network is down.
logger.warning(
"Telegram bot loop network error: %s (%s) — sleeping %.1fs",
type(exc).__name__, exc, backoff,
)
await asyncio.sleep(backoff)
backoff = min(backoff * 2, 30)
except Exception:
# Anything else is unexpected — log the full traceback so we
# can actually diagnose it. Previously this silently swallowed
# exceptions with no message body.
logger.exception("Telegram bot loop unexpected error — sleeping %.1fs", backoff)
await asyncio.sleep(backoff)
backoff = min(backoff * 2, 30)
# ── Admin helper ──────────────────────────────────────────────────────────
async def unbind_wallet(wallet: str) -> int:
"""Detach a wallet from its Telegram binding (called by /api/telegram/unbind).
Sets wallet_address = NULL rather than deleting the row. The chat stays
subscribed at the free tier — matches the UI's promise ("Your free
subscription stays active in the bot"). User wanting a full disconnect
sends /stop in the bot.
"""
async with async_session() as db:
r = await db.execute(
update(TelegramBinding)
.where(TelegramBinding.wallet_address == wallet.lower())
.values(wallet_address=None)
)
await db.commit()
return r.rowcount
app/services/tp_sl_monitor.py
+397 -23
View File
@@ -1,21 +1,39 @@
"""
Take-profit / stop-loss monitor.
Take-profit / stop-loss / trailing-stop monitor.
Subscribes to the Binance price stream; for every open trade that has TP/SL set,
closes the HL position as soon as the live mark price crosses the threshold.
Subscribes to the Binance price stream; for every open trade, closes the HL
position when the live mark price crosses one of:
1. Fixed stop-loss (always active).
2. Fixed take-profit (optional — set to None for trend capture).
3. Trailing stop (optional — once peak profit ≥ trailing_activate_at_pct,
close if price drops trailing_stop_pct from the peak).
The trailing branch is the centrepiece of the convex-payoff redesign:
fixed TP at +2% kills every potential runner before it starts. Instead we
let unrealised PnL grow, watching the peak, and only exit on a real
retracement.
Lifecycle:
- bot_engine.open_position calls register_trade(...) after each new trade
- price callback in binance.py calls on_price_tick() once per second
- when TP or SL hits, we enqueue close_and_finalize(...) and de-register
- binance.py's price callback calls on_price_tick() once per second
- when a rule fires, _fire_close(...) calls bot_engine.close_and_finalize
"""
import asyncio
import logging
from dataclasses import dataclass
import time
from dataclasses import dataclass, field
from typing import Dict, Optional
logger = logging.getLogger(__name__)
# Throttled peak persistence: write peak_gain_pct to the DB at most once per
# PEAK_FLUSH_SECS per trade, and only after it advances ≥ PEAK_FLUSH_DELTA pp.
# Peak is monotonic so writes are rare after the initial run-up; this is
# enough to keep the post-restart regime correct without per-tick I/O.
PEAK_FLUSH_DELTA = 1.0
PEAK_FLUSH_SECS = 30.0
@dataclass
class WatchedTrade:
@@ -26,14 +44,83 @@ class WatchedTrade:
asset: str
side: str # "long" | "short"
entry_price: float
take_profit_pct: Optional[float]
stop_loss_pct: Optional[float]
take_profit_pct: Optional[float] # legacy fixed TP (may be None)
stop_loss_pct: Optional[float] # always required in practice
trailing_stop_pct: Optional[float] # trail distance, e.g. 2.5
trailing_activate_at_pct: Optional[float] # activation threshold, e.g. 5.0
# System-2 only. "below_entry": exit immediately if price crosses back
# through entry BEFORE trailing arms — the reversal thesis (e.g. SMA
# reclaim) is dead, no reason to wait for the wide hard stop. None for
# System-1 / unset.
invalidation: Optional[str] = None
invalidation_price: Optional[float] = None
# Staged stop-loss ladder (System-2, optional). List of
# (peak_gain_trigger_pct, stop_floor_pct) sorted ascending. When set,
# this trade has NO take-profit and NO from-peak trailing: the only
# market exit is "price fell back to the highest staged floor that the
# peak gain has unlocked". See signal_categories.get_stop_ladder.
stop_ladder: Optional[list] = None
# Staged DOWNSIDE de-risk ladder (System-2, optional). List of
# (threshold_signed_pct_negative, frac_of_original, is_final) ordered by
# increasing adversity. While the trade is underwater (peak below the
# first stop_ladder trigger) it is scaled OUT in partial reduces at each
# rung; the final rung is a full close at the protective level (inside
# liquidation). See signal_categories.sys2_derisk_ladder.
derisk_ladder: Optional[list] = None
derisk_done: int = 0
derisk_in_flight: bool = field(default=False)
# Pyramiding (做对了往上加仓): add to a confirmed winner. List of
# (peak_gain_trigger_pct, frac_of_base, is_last). Only active in the
# profit regime AND while derisk_done==0. Each add blends entry up; the
# monitor then floors the stop at breakeven (eff_stop ≥ 0) so a pyramided
# winner can never become a loser.
addon_ladder: Optional[list] = None
addon_done: int = 0
addon_in_flight: bool = field(default=False)
# Per-trade "Grow" switch. Pyramiding only runs when True. De-risk +
# ratchet stop are unaffected (safety floor is always on). Toggled live
# by the user via POST /positions/{id}/grow.
grow_mode: bool = field(default=False)
# Parabolic-top trailing: (activate_peak_gain_pct, price_drawdown_frac).
# In the profit regime, once peak ≥ activate, the floor also trails the
# peak PRICE by at most drawdown_frac (scale-invariant). None = use only
# the fixed stop_ladder rungs. See signal_categories.sys2_peak_trail.
peak_trail: Optional[tuple] = None
# Throttled persistence of peak_gain_pct (monotonic). peak_persisted is
# the last value written to the DB; peak_persist_ts the last write time.
peak_persisted: float = field(default=0.0)
peak_persist_ts: float = field(default=0.0)
# System-1 (Trump) min-hold floor. Epoch seconds; before this time,
# take-profit and trailing exits are SUPPRESSED (don't scalp out of a
# developing move). Hard stop-loss + invalidation always fire — capital
# protection isn't deferrable. None = no floor (System 2 / legacy).
min_hold_until_ts: Optional[float] = None
# Runtime state — peak unrealised gain seen so far, in %. Used by the
# trailing branch. Long: peak = max(over time) of pct_gain. Short: same,
# but pct_gain is already signed correctly by the caller.
peak_gain_pct: float = field(default=0.0)
trailing_armed: bool = field(default=False)
# Buffer below entry before "below_entry" invalidation fires. Entry price is
# only a PROXY for the true thesis-invalidation level (e.g. the 200d SMA for
# an sma_reclaim). 0.75% is roughly one normal noise band on a major coin —
# big enough to survive the fill-tick wobble, small enough to still cut a
# failed reclaim well before the wide hard stop.
INVALIDATION_BUFFER_PCT = 0.75
# trade_id -> WatchedTrade
_watched: Dict[int, WatchedTrade] = {}
# Strong references to fire-close tasks to prevent GC before completion
# Strong refs to fire-close tasks (prevent GC before completion)
_background_tasks: set = set()
@@ -45,18 +132,58 @@ def register_trade(
asset: str,
side: str,
entry_price: float,
take_profit_pct: Optional[float],
stop_loss_pct: Optional[float],
take_profit_pct: Optional[float],
stop_loss_pct: Optional[float],
trailing_stop_pct: Optional[float] = None,
trailing_activate_at_pct: Optional[float] = None,
invalidation: Optional[str] = None,
invalidation_price: Optional[float] = None,
min_hold_until_ts: Optional[float] = None,
stop_ladder: Optional[list] = None,
derisk_ladder: Optional[list] = None,
derisk_done: int = 0,
addon_ladder: Optional[list] = None,
addon_done: int = 0,
initial_peak: float = 0.0,
peak_trail: Optional[tuple] = None,
grow_mode: bool = False,
) -> None:
if take_profit_pct is None and stop_loss_pct is None:
return # nothing to watch
# Nothing to monitor → skip. We require AT LEAST a stop-loss for any
# registered trade; protocol still works without TP if trailing or a
# staged-stop / de-risk ladder is set.
if (stop_loss_pct is None
and take_profit_pct is None
and trailing_stop_pct is None
and not stop_ladder
and not derisk_ladder):
return
_watched[trade_id] = WatchedTrade(
trade_id=trade_id, wallet=wallet, api_key=api_key, leverage=leverage,
asset=asset, side=side, entry_price=entry_price,
take_profit_pct=take_profit_pct, stop_loss_pct=stop_loss_pct,
take_profit_pct=take_profit_pct,
stop_loss_pct=stop_loss_pct,
trailing_stop_pct=trailing_stop_pct,
trailing_activate_at_pct=trailing_activate_at_pct,
invalidation=invalidation,
invalidation_price=invalidation_price,
min_hold_until_ts=min_hold_until_ts,
stop_ladder=stop_ladder,
derisk_ladder=derisk_ladder,
derisk_done=derisk_done or 0,
addon_ladder=addon_ladder,
addon_done=addon_done or 0,
peak_gain_pct=max(0.0, float(initial_peak or 0.0)),
peak_persisted=max(0.0, float(initial_peak or 0.0)),
peak_trail=peak_trail,
grow_mode=bool(grow_mode),
)
logger.info(
"Watching trade %d (%s %s @ %.4f, sl=%s, tp=%s, trail=%s/@%s)",
trade_id, side, asset, entry_price,
stop_loss_pct, take_profit_pct,
trailing_stop_pct, trailing_activate_at_pct,
)
logger.info("TP/SL watching trade %d (%s %s @ %.2f, tp=%s, sl=%s)",
trade_id, side, asset, entry_price, take_profit_pct, stop_loss_pct)
def unregister(trade_id: int) -> None:
@@ -64,21 +191,163 @@ def unregister(trade_id: int) -> None:
def on_price_tick(asset: str, price: float) -> None:
"""Called from binance.py on every price update. Fires close for any trade that hit TP/SL."""
"""Called from binance.py on every price update.
Iterates watched trades on this asset, evaluates each rule, and triggers
a close on the first match. Rule evaluation order: stop-loss → trailing
→ fixed TP. (Stop-loss first so a sudden gap kills the position before
we celebrate hitting trailing.)
"""
if not _watched:
return
triggered = []
derisk_actions = [] # (wt, step_idx, frac_of_original)
addon_actions = [] # (wt, step_idx, frac_of_base)
for tid, wt in list(_watched.items()):
if wt.asset != asset:
continue
pct = (price - wt.entry_price) / wt.entry_price
signed_pct = pct if wt.side == 'long' else -pct # gain in position's direction
pct_x100 = signed_pct * 100
# Convert raw price → signed gain in position's direction, in %.
raw_pct = (price - wt.entry_price) / wt.entry_price
signed_pct = (raw_pct if wt.side == "long" else -raw_pct) * 100
if wt.take_profit_pct is not None and pct_x100 >= wt.take_profit_pct:
triggered.append((wt, "take_profit"))
elif wt.stop_loss_pct is not None and pct_x100 <= -wt.stop_loss_pct:
# Update peak BEFORE evaluating trailing — a tick that pushes the peak
# AND retraces in the same step should still arm trailing at the new peak.
if signed_pct > wt.peak_gain_pct:
wt.peak_gain_pct = signed_pct
# Throttled monotonic persistence so a restart keeps the regime.
now_s = time.time()
if (wt.peak_gain_pct - wt.peak_persisted >= PEAK_FLUSH_DELTA
and now_s - wt.peak_persist_ts >= PEAK_FLUSH_SECS):
wt.peak_persisted = wt.peak_gain_pct
wt.peak_persist_ts = now_s
_pt = asyncio.create_task(
_persist_peak(wt.trade_id, wt.peak_gain_pct))
_background_tasks.add(_pt)
_pt.add_done_callback(_background_tasks.discard)
# 0. System-2 self-contained exit model: NO take-profit, NO from-peak
# trailing. Two regimes, fully replacing branches 1/1b/2/3:
# • Underwater → STAGED DE-RISK ("分段式减仓"): partial reduces
# at each downside rung, full close at the final (protective,
# inside-liquidation) rung.
# • In profit → STAGED STOP: floor ratchets up as peak gain
# crosses each upside rung; full close when price falls back.
if wt.stop_ladder or wt.derisk_ladder:
first_up = (min(t for t, _ in wt.stop_ladder)
if wt.stop_ladder else None)
in_profit_regime = (first_up is not None
and wt.peak_gain_pct >= first_up)
if wt.derisk_ladder and not in_profit_regime:
ladder = wt.derisk_ladder
# Gap protection: blown straight through to the final
# protective level → full close NOW regardless of step
# bookkeeping (still inside the exchange liquidation line).
if signed_pct <= ladder[-1][0]:
triggered.append((wt, "staged_stop"))
continue
if (not wt.derisk_in_flight
and 0 <= wt.derisk_done < len(ladder)):
thr, frac, is_final = ladder[wt.derisk_done]
if signed_pct <= thr:
if is_final:
triggered.append((wt, "staged_stop"))
continue
# Partial reduce — spawn async, guard re-entry until
# it completes (done-callback clears the flag).
wt.derisk_in_flight = True
derisk_actions.append((wt, wt.derisk_done, frac))
continue # underwater: no profit-ratchet / TP / trailing
# Profit regime — ratchet floor (base + unlocked upside rungs).
eff_stop = -wt.stop_loss_pct if wt.stop_loss_pct is not None else float("-inf")
if wt.stop_ladder:
for trig, floor in wt.stop_ladder: # sorted ascending
if wt.peak_gain_pct >= trig and floor > eff_stop:
eff_stop = floor
# Parabolic-top trailing: once deep in profit, also trail the PEAK
# PRICE by ≤ drawdown_frac (scale-invariant — self-adjusts to a
# +120% or +900% move so the fixed top rung doesn't cap it, while
# a normal ~25–30% bull pullback still doesn't trip it).
if wt.peak_trail is not None:
start_pp, dd = wt.peak_trail
if wt.peak_gain_pct >= start_pp:
trail_floor = ((1.0 + wt.peak_gain_pct / 100.0)
* (1.0 - dd) - 1.0) * 100.0
if trail_floor > eff_stop:
eff_stop = trail_floor
# Pyramided → never let a winner become a loser: blended position
# is floored at breakeven once any add-on has filled.
if wt.addon_done > 0 and eff_stop < 0.0:
eff_stop = 0.0
if signed_pct <= eff_stop:
triggered.append((wt, "staged_stop"))
continue
# Pyramiding — add to a confirmed winner. Gated by the per-trade
# Grow switch (off by default — user opts a position in). Evaluated
# ONLY after the stop check above passed (never add on a tick that's
# also exiting). Peak-gain triggered; only while no de-risk has
# fired. Structural confirmation is re-checked in the executor.
if (wt.grow_mode and wt.addon_ladder and wt.derisk_done == 0
and not wt.addon_in_flight
and 0 <= wt.addon_done < len(wt.addon_ladder)):
a_trig, a_frac, _a_last = wt.addon_ladder[wt.addon_done]
if wt.peak_gain_pct >= a_trig:
wt.addon_in_flight = True
addon_actions.append((wt, wt.addon_done, a_frac))
continue
# 1. Stop loss — first priority.
if wt.stop_loss_pct is not None and signed_pct <= -wt.stop_loss_pct:
triggered.append((wt, "stop_loss"))
continue
# 1b. Signal invalidation (System-2 only). Prefer the real thesis
# level when available (e.g. 200d SMA reclaim price). Fall back
# to entry-buffer proxy for legacy rows.
if wt.invalidation == "below_entry" and not wt.trailing_armed:
if wt.invalidation_price is not None:
invalidation_hit = (
(wt.side == "long" and price <= wt.invalidation_price) or
(wt.side == "short" and price >= wt.invalidation_price)
)
if invalidation_hit:
triggered.append((wt, "invalidation"))
continue
elif signed_pct <= -INVALIDATION_BUFFER_PCT:
triggered.append((wt, "invalidation"))
continue
# Min-hold gate (System-1 Trump). Before the floor elapses we let a
# winner develop — suppress TP + trailing. SL / invalidation above
# already ran, so downside is still protected; we only defer the
# PROFIT-side exits here.
if wt.min_hold_until_ts is not None:
import time as _t
if _t.time() < wt.min_hold_until_ts:
continue # still in min-hold; skip TP/trailing this tick
# 2. Trailing stop — only once armed.
if (wt.trailing_stop_pct is not None
and wt.trailing_activate_at_pct is not None):
if not wt.trailing_armed and wt.peak_gain_pct >= wt.trailing_activate_at_pct:
wt.trailing_armed = True
logger.info(
"Trade %d: trailing armed at peak %.2f%% (asset=%s)",
wt.trade_id, wt.peak_gain_pct, asset,
)
if wt.trailing_armed:
drawdown_from_peak = wt.peak_gain_pct - signed_pct
if drawdown_from_peak >= wt.trailing_stop_pct:
triggered.append((wt, "trailing_stop"))
continue
# 3. Fixed TP — legacy / fallback when no trailing config.
if wt.take_profit_pct is not None and signed_pct >= wt.take_profit_pct:
triggered.append((wt, "take_profit"))
continue
for wt, reason in triggered:
unregister(wt.trade_id)
@@ -86,10 +355,115 @@ def on_price_tick(asset: str, price: float) -> None:
_background_tasks.add(task)
task.add_done_callback(_background_tasks.discard)
# Partial de-risk steps — trade stays registered (it's not closed).
for wt, step_idx, frac in derisk_actions:
task = asyncio.create_task(_fire_partial(wt, step_idx, frac))
_background_tasks.add(task)
task.add_done_callback(_background_tasks.discard)
# Pyramid add-ons — trade stays registered (size grows, entry blends).
for wt, step_idx, frac in addon_actions:
task = asyncio.create_task(_fire_pyramid(wt, step_idx, frac))
_background_tasks.add(task)
task.add_done_callback(_background_tasks.discard)
async def _fire_partial(wt: WatchedTrade, step_idx: int, frac: float) -> None:
"""Execute one staged de-risk partial reduce. The trade stays open and
registered; on success advance derisk_done, always clear the in-flight
guard so the next rung can fire on a later tick."""
from app.services.bot_engine import partial_derisk
ok = False
try:
ok = await partial_derisk(
trade_id=wt.trade_id,
api_key=wt.api_key,
asset=wt.asset,
wallet=wt.wallet,
step_idx=step_idx,
frac_of_original=frac,
reason="derisk",
)
except Exception as exc:
logger.error("De-risk partial failed for trade %d step %d: %s",
wt.trade_id, step_idx, exc)
finally:
# Re-fetch: the trade may have been closed/unregistered meanwhile.
live = _watched.get(wt.trade_id)
if live is not None:
if ok and live.derisk_done == step_idx:
live.derisk_done = step_idx + 1
live.derisk_in_flight = False
async def _fire_pyramid(wt: WatchedTrade, step_idx: int, frac: float) -> None:
"""Execute one pyramid add-on. On success re-base the in-memory entry to
the blended average and reset peak to the post-add gain so the ratchet /
regime use the NEW entry (and we stay in the profit regime). Always clear
the in-flight guard."""
from app.services.bot_engine import pyramid_add
ok, new_entry, ref_price = False, None, None
try:
ok, new_entry, ref_price = await pyramid_add(
trade_id=wt.trade_id,
api_key=wt.api_key,
asset=wt.asset,
wallet=wt.wallet,
step_idx=step_idx,
frac_of_base=frac,
reason="pyramid",
)
except Exception as exc:
logger.error("Pyramid add failed for trade %d step %d: %s",
wt.trade_id, step_idx, exc)
finally:
live = _watched.get(wt.trade_id)
if live is not None:
if ok and live.addon_done == step_idx:
# Step is RESOLVED (added, notional-capped, or already-done).
# Advance regardless of whether an add actually filled, else
# the monitor re-schedules this step every tick forever.
if new_entry:
live.entry_price = new_entry
if ref_price:
raw = (ref_price - new_entry) / new_entry
g = (raw if live.side == "long" else -raw) * 100.0
else:
g = 0.0
# Stay in the profit regime; ratchet re-accumulates here.
first_up = (min(t for t, _ in live.stop_ladder)
if live.stop_ladder else 0.0)
live.peak_gain_pct = max(g, first_up)
live.addon_done = step_idx + 1
live.addon_in_flight = False
async def _persist_peak(trade_id: int, peak: float) -> None:
"""Monotonic best-effort write of peak_gain_pct. Race-safe: only raises
the stored value (WHERE peak_gain_pct < :peak), never lowers it."""
try:
from sqlalchemy import update
from app.database import AsyncSessionLocal
from app.models import BotTrade
async with AsyncSessionLocal() as db:
await db.execute(
update(BotTrade)
.where(BotTrade.id == trade_id)
.where(BotTrade.peak_gain_pct < peak)
.values(peak_gain_pct=peak)
)
await db.commit()
except Exception as exc: # never let persistence break the monitor
logger.debug("peak persist failed trade %d: %s", trade_id, exc)
async def _fire_close(wt: WatchedTrade, reason: str) -> None:
from app.services.bot_engine import close_and_finalize
try:
logger.info(
"Closing trade %d on %s (peak=%.2f%%, reason=%s)",
wt.trade_id, wt.asset, wt.peak_gain_pct, reason,
)
await close_and_finalize(
trade_id=wt.trade_id,
api_key=wt.api_key,
app/ws/manager.py
+43 -7
View File
@@ -1,3 +1,22 @@
"""
WebSocket connection manager.
One global broadcaster that fans out every message to every connected client.
Critical that broadcast NEVER blocks the caller — Binance kline ticks arrive
every ~500 ms and the broadcast is inline with the WS read loop. A single
slow / half-closed client could otherwise stall the kline pipeline, miss the
keepalive ping, and tear the upstream Binance socket down (this was the root
cause of the "no close frame received or sent" reconnect storms in prod logs).
Two defences here:
* Per-client send wrapped in `asyncio.wait_for(..., timeout=PER_CLIENT_SEND_TIMEOUT)`.
* All clients dispatched in parallel via `asyncio.gather()` so one slow
client can't delay anyone else.
"""
from __future__ import annotations
import asyncio
import json
import logging
from typing import List
@@ -6,6 +25,10 @@ from fastapi import WebSocket
logger = logging.getLogger(__name__)
# Per-client send timeout. 2 s is plenty for a healthy TCP send of our tiny
# JSON payloads; anything slower means a half-closed or zombie client.
PER_CLIENT_SEND_TIMEOUT = 2.0
class ConnectionManager:
def __init__(self):
@@ -25,15 +48,28 @@ class ConnectionManager:
if not self.active_connections:
return
payload = json.dumps(message)
dead: List[WebSocket] = []
for connection in self.active_connections:
# Snapshot the list — disconnect() mutates active_connections and we
# don't want "list changed during iteration" when we prune.
connections = list(self.active_connections)
async def _send_one(ws: WebSocket):
try:
await connection.send_text(payload)
await asyncio.wait_for(ws.send_text(payload),
timeout=PER_CLIENT_SEND_TIMEOUT)
return None
except asyncio.TimeoutError:
logger.warning("WebSocket send timed out (>%.1fs) — pruning client",
PER_CLIENT_SEND_TIMEOUT)
return ws
except Exception as exc:
logger.warning("Failed to send to WebSocket: %s", exc)
dead.append(connection)
for ws in dead:
await self.disconnect(ws)
logger.warning("WebSocket send failed: %s — pruning client", exc)
return ws
# Fan out concurrently — one slow client can't stall the others.
results = await asyncio.gather(*[_send_one(ws) for ws in connections])
for dead_ws in results:
if dead_ws is not None:
await self.disconnect(dead_ws)
manager = ConnectionManager()