fix: pre-launch hardening — HYPE price feed, KOL wallet cleanup, Telegram Trump alert, rate limiting, brittle test
Batch of the pre-launch audit campaign (BUG-01…14 plus three new features): Pricing / TP-SL protection - Add app/services/hl_price_feed.py: supplemental HL allMids poller for HL-native assets (HYPE, PURR) not listed on Binance. Pumps price_store + tp_sl_monitor.on_price_tick so bot trades on these assets keep full stop-loss / take-profit / trailing protection instead of max-hold only. - Wire feed into main.py lifespan (startup task + graceful shutdown cancel). Telegram - Add format_trump_mention + PATH B in _dispatch: crypto-relevant Trump posts with no directional signal (relevant=True, signal=hold) now alert the public channel only (no per-subscriber noise). - Rate limiter (slowapi) on the API; assorted bot/digest fixes. KOL on-chain - seed_kol_wallets.py: KOL_FEEDS coverage cross-check; reversibly deactivate orphaned wallets (handle not in KOL_FEEDS → can never produce divergence) so the scanner stops burning cycles on them. Tests / misc - Fix brittle test_macro_ahr999_uses_same_formula_as_scanner: mock now uses realistic ms timestamps so the in-progress-day drop fires, matching the fetcher's bar count (was 0.3179 vs 0.3178 off-by-one). - Refresh stale notify_signal comment in truth_social.py. Frontend reduce-action type fix lives in the sibling repo. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
k
@@ -96,7 +96,7 @@ linked) = Trump auto-trade + /adopt manage-only flow for sys2.
|
||||
|
||||
- **Python 3.9+** / FastAPI / async SQLAlchemy 2.x / APScheduler
|
||||
- **DB**: SQLite dev, **Postgres prod**. All schema lives in
|
||||
`alembic/versions/NNN_*.py`, ordered. Currently at head **024**.
|
||||
`alembic/versions/NNN_*.py`, ordered. Currently at head **026**.
|
||||
- **AI**: DeepSeek via OpenAI-compatible API (`AI_BASE_URL`, `AI_MODEL`).
|
||||
- Live scoring uses `AI_LIVE_MODEL` (~2s, latency-critical)
|
||||
- Batch / reanalysis uses `AI_MODEL` (quality, ~10s)
|
||||
@@ -106,6 +106,9 @@ linked) = Trump auto-trade + /adopt manage-only flow for sys2.
|
||||
`tp_sl_monitor` per-tick evaluator.
|
||||
- **Telegram**: long-poll mode (single instance), HTML messages, inline
|
||||
keyboards. `telegram.py` send/edit/answer + `telegram_bot.py` commands.
|
||||
Public channel broadcast (`TELEGRAM_PUBLIC_CHANNEL_ID` env) sends a
|
||||
sanitised `format_public_post` version (no execution details, tier label
|
||||
instead of raw confidence) after every per-user fan-out in `_dispatch`.
|
||||
|
||||
---
|
||||
|
||||
@@ -165,7 +168,8 @@ app/
|
||||
├── config.py Pydantic Settings — reads .env
|
||||
└── main.py FastAPI lifespan, scheduler setup, route mount
|
||||
|
||||
alembic/versions/ Migrations (numbered NNN). Latest = 024
|
||||
alembic/versions/ Migrations (numbered NNN). Latest = 026
|
||||
026 = composite index (wallet_address, closed_at) on bot_trades
|
||||
scripts/ One-shot ops
|
||||
├── preflight.py Pre-launch readiness gate (env / DB / TG / AI)
|
||||
├── launch_smoke.py End-to-end smoke (14 checks against running API)
|
||||
@@ -304,6 +308,8 @@ minimalism — don't extend them.
|
||||
7. Add a `/yoursource on|off` command in `telegram_bot.py`.
|
||||
8. If sys2: extend `signal_categories._CATEGORY_EXITS` if it needs a custom
|
||||
exit profile (otherwise default works).
|
||||
9. Add the deep-link path in `telegram.format_post` AND `telegram.format_public_post`
|
||||
(both the per-user private alert and the public channel use the same path map).
|
||||
|
||||
### Add a new bot command
|
||||
|
||||
@@ -399,29 +405,108 @@ this for you" to "you opened it, bot manages your discipline".
|
||||
- **Adopted trades have `hl_order_id` starting with `"adopted:"`.** Distinct
|
||||
from auto-opened (HL order id integer) and paper (`"paper"` literal).
|
||||
Useful for telemetry filtering.
|
||||
- **`telegram.send_message` accepts `int | str` for `chat_id`.** Intentional.
|
||||
Integer = private chat, string = public channel username (e.g. `"@trumpalpha"`).
|
||||
The public channel uses the string form; per-user alerts still pass integers.
|
||||
- **`format_public_post` deliberately omits `expected_move_pct`,
|
||||
`invalidation_price`, and `/adopt` CTA.** Execution-sensitive data stays
|
||||
private (per-user). The public version shows confidence tier (HIGH/MED/LOW)
|
||||
instead of the raw score for readability.
|
||||
- **`_adopt_locks` in adoption.py looks like it should have a cap like
|
||||
`_wallet_open_locks` (512).** It doesn't yet — see Known Issues below.
|
||||
|
||||
---
|
||||
|
||||
## Open known issues (not blocking launch but worth fixing later)
|
||||
|
||||
- **Trump daily budget split**: `_execute_for_subscriber` still computes
|
||||
`daily_cap = total × (1 - sys2_pct)` for sys1 — i.e. Trump gets only 30%
|
||||
of the configured daily budget by default. With sys2 manage-only there's
|
||||
no reason to split. Fix: when not sys2, use full `total_cap`. (Low-prio:
|
||||
users who notice can just set `sys2_budget_pct = 0` in DB.)
|
||||
- **HL high-leverage adoption**: if a user has a 25× position on HL, our
|
||||
`sys2_protective_stop_pct` clamps to 10× (the strategy's design max),
|
||||
which gives a stop FURTHER from entry than the real liquidation point.
|
||||
Position could get HL-liquidated before our stop fires. Fix: reject
|
||||
adoptions where actual HL leverage > `SYS2_MAX_LEVERAGE`.
|
||||
- ~~**`_time_stop_check` tasks not rehydrated on restart** (BUG-01, FIXED 2026-05-29):~~
|
||||
`recovery.py` now imports `_time_stop_check` + `_background_tasks` from
|
||||
`bot_engine` and `get_exit_profile` from `signal_categories`. After each
|
||||
`register_trade()` call for a sys2 trade, it checks `exit_profile.time_stop_hours`,
|
||||
computes remaining seconds, and reschedules the task. Elapsed windows (backend
|
||||
was down longer than the time-stop period) fire immediately with `delay_seconds=0`.
|
||||
|
||||
- **Rate limit bypass via proxy x-forwarded-for deletion** (BUG-02, HIGH):
|
||||
`app/api/proxy/[...path]/route.ts` (frontend) strips `x-forwarded-for`
|
||||
before forwarding. Backend `slowapi` sees the Next.js server IP and applies
|
||||
one rate-limit bucket to ALL users. Fix: relay the real client IP from
|
||||
`req.headers.get('x-forwarded-for')` or `x-real-ip` in the proxy.
|
||||
|
||||
- ~~**`close_and_finalize` double-failure leaves DB/HL state inconsistent**
|
||||
(BUG-03, MITIGATED 2026-05-29):~~ Full two-phase-commit is out of scope.
|
||||
Mitigation: `reconciler._reconcile_wallet` now runs a "ghost position" pass —
|
||||
queries DB-closed trades from the last 2h and cross-checks against HL open
|
||||
positions. Mismatches are logged at ERROR level and broadcast via WS
|
||||
(`reconcile_drift.ghost_positions`). Manual close required on HL UI.
|
||||
|
||||
- ~~**`_adopt_locks` has no capacity cap** (BUG-04, FIXED 2026-05-29):~~
|
||||
`adoption._adopt_locks` is now an `OrderedDict` capped at `_ADOPT_LOCK_MAX=512`
|
||||
with LRU eviction — matches the `_WALLET_LOCK_MAX` pattern in `bot_engine`.
|
||||
|
||||
- ~~**Reconciler runs wallets sequentially** (BUG-05, FIXED 2026-05-29):~~
|
||||
`reconcile_all_once` now fans out with `asyncio.gather` + `asyncio.Semaphore(10)`.
|
||||
Worst-case tail latency is `ceil(N/10) × 30s` instead of `N × 30s`.
|
||||
|
||||
- ~~**`/stop` also silently disables daily digest** (BUG-07, FIXED 2026-05-29):~~
|
||||
`send_daily_digest` no longer filters on `alerts_enabled` — digest and
|
||||
real-time alerts are independent. `/stop` reply updated to say "Real-time
|
||||
alerts paused … send `/digest off` to stop the daily brief separately".
|
||||
|
||||
- ~~**`binance.py` ASSET_MAP only had BTC + ETH** (BUG-08, CRITICAL, FIXED 2026-05-29):~~
|
||||
Trump's AI can set `target_asset` to SOL/TRUMP/BNB/DOGE/LINK/AAVE.
|
||||
`ASSET_MAP` now covers all those; `BINANCE_WS_URL` is derived from it
|
||||
automatically so the stream list and routing table can never diverge.
|
||||
**Still missing**: HYPE (HL-native, not on Binance) — trades on HYPE fall
|
||||
back to max-hold only until a supplemental HL price feed is added.
|
||||
|
||||
- ~~**`_close_locks` leaked one entry per concurrent-close loser** (BUG-11, FIXED 2026-05-29):~~
|
||||
`close_and_finalize` now pops `_close_locks[trade_id]` in the
|
||||
`rowcount == 0` early-return path, not just on the success paths.
|
||||
|
||||
- ~~**`signed_request._seen` O(n) purge at 5000 entries** (BUG-12, FIXED 2026-05-29):~~
|
||||
Threshold reduced to `_SEEN_PURGE_THRESHOLD = 1000` and the purge now
|
||||
builds an `expired` list in one pass rather than calling `dict.pop` in a loop.
|
||||
|
||||
- ~~**`_get_max_leverage` makes a fresh HL `meta()` call on every trade open**
|
||||
(BUG-10, FIXED 2026-05-29):~~ `_MAX_LEV_CACHE` added (same 300s TTL as
|
||||
`_SZ_DECIMALS_CACHE`). Cache miss populates ALL coins from the single
|
||||
`meta()` response, so concurrent opens pay one API call, not N.
|
||||
|
||||
- ~~**Trump daily budget split**~~ (FIXED 2026-05-29): System-2 is manage-only
|
||||
so the `sys2_pct` reservation no longer makes sense for auto-opens.
|
||||
`daily_cap` for System-1 (Trump) is now `total_cap × 1.0` instead of
|
||||
`total_cap × (1 - sys2_pct)`. The split logic remains intact so it would
|
||||
work correctly if sys2 auto-open is ever re-enabled via ADR.
|
||||
|
||||
- ~~**HL high-leverage adoption**~~ (BUG-09, FIXED 2026-05-29): `adopt_position`
|
||||
now rejects positions where `leverage > SYS2_MAX_LEVERAGE` with error code
|
||||
`leverage_too_high`. Previously the protective stop was computed for 10×
|
||||
but applied to a 25× position — the stop was OUTSIDE the liquidation band.
|
||||
|
||||
- **`adopt:choose:BTC` callback may show stale prices** if user takes >60s
|
||||
to tap (HL fees, partial fills can change entry/size). adopt_position
|
||||
re-reads HL at mode-tap time so the FROZEN BotTrade is always fresh,
|
||||
but the picker label could be outdated.
|
||||
|
||||
- **Telegram bot offset on restart**: `getUpdates` may replay last 24h of
|
||||
messages on backend restart. Stale `/adopt` could re-fire. User can
|
||||
/release to recover.
|
||||
|
||||
- ~~**`pyramid_add` double-add on DB commit failure** (BUG-13, FIXED 2026-05-29):~~
|
||||
Same pattern as BUG-03 / partial_derisk. HL `open_position` succeeds but the
|
||||
subsequent `db.commit()` of `addon_steps_done` fails → retrigger sees same
|
||||
`step_idx` and double-adds. Fix: pre-claim `addon_steps_done = step_idx + 1`
|
||||
with a conditional UPDATE (WHERE `addon_steps_done == step_idx`) BEFORE calling
|
||||
HL. If `rowcount == 0`, return early. Second UPDATE writes `entry_price` and
|
||||
`size_usd` after fill confirmation.
|
||||
|
||||
- ~~**`price_impact_monitor` measured wrong asset** (BUG-14, FIXED 2026-05-29):~~
|
||||
`truth_social.py` passed `analysis["asset"]` (BTC/ETH sentiment proxy) to
|
||||
`register_post` instead of `target_asset` (SOL/TRUMP/etc. — the perp we actually
|
||||
trade). Impact % was measuring BTC/ETH not the traded coin. Fix: introduced
|
||||
`tracked_asset = analysis.get("target_asset") or asset` and used it for
|
||||
`price_at_post`, `price_impact_asset`, and `register_post(asset=…)`.
|
||||
|
||||
---
|
||||
|
||||
## Repos in this project
|
||||
|
||||
@@ -0,0 +1,24 @@
|
||||
"""Add composite index ix_bot_trades_wallet_closed
|
||||
|
||||
Revision ID: 026
|
||||
Revises: 025
|
||||
Create Date: 2026-05-28
|
||||
"""
|
||||
from alembic import op
|
||||
|
||||
revision = '026'
|
||||
down_revision = '025'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_index(
|
||||
'ix_bot_trades_wallet_closed',
|
||||
'bot_trades',
|
||||
['wallet_address', 'closed_at'],
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index('ix_bot_trades_wallet_closed', table_name='bot_trades')
|
||||
@@ -8,13 +8,14 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from app.database import get_db
|
||||
from app.models import BotTrade
|
||||
from app.schemas import BotPerformance
|
||||
from app.services.signed_request import verify_signed_request
|
||||
from app.services.signed_request import verify_signed_request_any
|
||||
|
||||
router = APIRouter()
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
PERIOD_DAYS = 30
|
||||
ACTION_VIEW_PERFORMANCE = "view_performance"
|
||||
ACTION_VIEW_USER = "view_user"
|
||||
|
||||
|
||||
@router.get("/performance", response_model=BotPerformance)
|
||||
@@ -25,8 +26,8 @@ async def get_performance(
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
wallet = wallet.lower().strip()
|
||||
verify_signed_request(
|
||||
action=ACTION_VIEW_PERFORMANCE,
|
||||
verify_signed_request_any(
|
||||
actions=[ACTION_VIEW_PERFORMANCE, ACTION_VIEW_USER],
|
||||
wallet=wallet,
|
||||
timestamp_ms=ts,
|
||||
signature=sig,
|
||||
|
||||
@@ -35,7 +35,7 @@ from app.database import get_db
|
||||
from app.models import BotTrade, Subscription, iso_utc
|
||||
from app.services.crypto import decrypt_api_key
|
||||
from app.services.price_store import price_store
|
||||
from app.services.signed_request import verify_signed_request
|
||||
from app.services.signed_request import verify_signed_request, verify_signed_request_any
|
||||
|
||||
router = APIRouter()
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -150,8 +150,8 @@ async def get_open_positions(
|
||||
):
|
||||
"""Live open positions for the wallet, with mark-to-market PnL."""
|
||||
wallet = wallet.lower().strip()
|
||||
verify_signed_request(
|
||||
action=ACTION_VIEW_POSITIONS,
|
||||
verify_signed_request_any(
|
||||
actions=[ACTION_VIEW_POSITIONS, ACTION_VIEW_USER],
|
||||
wallet=wallet,
|
||||
timestamp_ms=ts,
|
||||
signature=sig,
|
||||
@@ -187,8 +187,8 @@ async def get_today_stats(
|
||||
one-shot count for opens.
|
||||
"""
|
||||
wallet = wallet.lower().strip()
|
||||
verify_signed_request(
|
||||
action=ACTION_VIEW_POSITIONS,
|
||||
verify_signed_request_any(
|
||||
actions=[ACTION_VIEW_POSITIONS, ACTION_VIEW_USER],
|
||||
wallet=wallet,
|
||||
timestamp_ms=ts,
|
||||
signature=sig,
|
||||
@@ -242,6 +242,7 @@ async def get_today_stats(
|
||||
ACTION_CLOSE_TRADE = "close_trade"
|
||||
ACTION_SET_GROW = "set_trade_grow"
|
||||
ACTION_VIEW_POSITIONS = "view_positions"
|
||||
ACTION_VIEW_USER = "view_user"
|
||||
ACTION_ADOPT_POSITION = "adopt_position"
|
||||
ACTION_RELEASE_TRADE = "release_trade"
|
||||
|
||||
|
||||
+14
-1
@@ -1,7 +1,12 @@
|
||||
import logging
|
||||
from typing import List, Optional
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Request
|
||||
from fastapi.responses import Response
|
||||
from slowapi import Limiter
|
||||
from slowapi.util import get_remote_address
|
||||
|
||||
limiter = Limiter(key_func=get_remote_address)
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
@@ -66,16 +71,24 @@ def _post_to_schema(post: Post) -> TrumpPost:
|
||||
|
||||
|
||||
@router.get("/posts", response_model=List[TrumpPost])
|
||||
@limiter.limit("60/minute")
|
||||
async def get_posts(
|
||||
request: Request,
|
||||
limit: int = Query(default=20, ge=1, le=500),
|
||||
page: int = Query(default=1, ge=1),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
response: Response = None,
|
||||
):
|
||||
offset = (page - 1) * limit
|
||||
result = await db.execute(
|
||||
select(Post).order_by(Post.published_at.desc()).offset(offset).limit(limit)
|
||||
)
|
||||
posts = result.scalars().all()
|
||||
# Posts are scraped every 5s but rarely change once written — allow CDN/browser
|
||||
# to cache for 30s. stale-while-revalidate=60 means stale content is served
|
||||
# while a fresh fetch happens in the background (no loading flash).
|
||||
if response is not None:
|
||||
response.headers["Cache-Control"] = "public, max-age=30, stale-while-revalidate=60"
|
||||
return [_post_to_schema(p) for p in posts]
|
||||
|
||||
|
||||
|
||||
+17
-3
@@ -2,7 +2,12 @@ import logging
|
||||
from typing import List
|
||||
|
||||
import httpx
|
||||
from fastapi import APIRouter, HTTPException, Query
|
||||
from fastapi import APIRouter, HTTPException, Query, Request
|
||||
from fastapi.responses import Response
|
||||
from slowapi import Limiter
|
||||
from slowapi.util import get_remote_address
|
||||
|
||||
limiter = Limiter(key_func=get_remote_address)
|
||||
|
||||
from app.config import settings
|
||||
from app.schemas import Candle
|
||||
@@ -48,10 +53,13 @@ async def fetch_binance_candles(asset: str, tf: str, limit: int) -> List[Candle]
|
||||
|
||||
|
||||
@router.get("/prices/{asset}", response_model=List[Candle])
|
||||
@limiter.limit("30/minute")
|
||||
async def get_prices(
|
||||
request: Request,
|
||||
asset: str,
|
||||
tf: str = Query(default="4H"),
|
||||
limit: int = Query(default=200, ge=1, le=1000),
|
||||
response: Response = None,
|
||||
):
|
||||
asset = asset.upper()
|
||||
if asset not in VALID_ASSETS:
|
||||
@@ -59,12 +67,18 @@ async def get_prices(
|
||||
if tf not in VALID_TIMEFRAMES:
|
||||
raise HTTPException(status_code=400, detail=f"Timeframe must be one of {VALID_TIMEFRAMES}")
|
||||
|
||||
# 1m: use in-memory store (updated in real-time)
|
||||
# 1m: use in-memory store (updated in real-time) — don't cache at CDN level
|
||||
if tf == "1m":
|
||||
candles = price_store.get_candles(asset, "1m", limit)
|
||||
if response is not None:
|
||||
response.headers["Cache-Control"] = "public, max-age=5, stale-while-revalidate=10"
|
||||
return [Candle(**c) for c in candles]
|
||||
|
||||
# All other timeframes: fetch directly from Binance
|
||||
# Larger timeframes change infrequently — safe to cache longer
|
||||
if response is not None:
|
||||
ttl = 60 if tf in ("5m", "15m") else 300 # 1m for short TFs, 5m for 1H+
|
||||
response.headers["Cache-Control"] = f"public, max-age={ttl}, stale-while-revalidate={ttl * 2}"
|
||||
|
||||
try:
|
||||
return await fetch_binance_candles(asset, tf, limit)
|
||||
except Exception as exc:
|
||||
|
||||
+4
-3
@@ -9,12 +9,13 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from app.database import get_db
|
||||
from app.models import BotTrade, iso_utc
|
||||
from app.schemas import BotTrade as BotTradeSchema
|
||||
from app.services.signed_request import verify_signed_request
|
||||
from app.services.signed_request import verify_signed_request_any
|
||||
|
||||
router = APIRouter()
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
ACTION_VIEW_TRADES = "view_trades"
|
||||
ACTION_VIEW_USER = "view_user"
|
||||
|
||||
|
||||
def _trade_to_schema(trade: BotTrade) -> BotTradeSchema:
|
||||
@@ -54,8 +55,8 @@ async def get_trades(
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
wallet = wallet.lower().strip()
|
||||
verify_signed_request(
|
||||
action=ACTION_VIEW_TRADES,
|
||||
verify_signed_request_any(
|
||||
actions=[ACTION_VIEW_TRADES, ACTION_VIEW_USER],
|
||||
wallet=wallet,
|
||||
timestamp_ms=ts,
|
||||
signature=sig,
|
||||
|
||||
+4
-3
@@ -231,8 +231,8 @@ async def set_user_settings(
|
||||
raise HTTPException(422, "min_confidence must be 0–100")
|
||||
if s.sys2_leverage is not None and not (1 <= s.sys2_leverage <= 10):
|
||||
raise HTTPException(422, "sys2_leverage must be 1–10")
|
||||
if s.sys2_mode is not None and s.sys2_mode not in ("standard", "aggressive"):
|
||||
raise HTTPException(422, "sys2_mode must be 'standard' or 'aggressive'")
|
||||
if s.sys2_mode is not None and s.sys2_mode not in ("standard", "aggressive", ""):
|
||||
raise HTTPException(422, "sys2_mode must be 'standard', 'aggressive', or '' (reset)")
|
||||
if s.daily_budget_usd is not None and not (0 < s.daily_budget_usd <= 100000):
|
||||
raise HTTPException(422, "daily_budget_usd must be >0 and ≤100,000 if provided")
|
||||
|
||||
@@ -280,7 +280,8 @@ async def set_user_settings(
|
||||
sub.daily_budget_usd = s.daily_budget_usd
|
||||
sub.sys2_leverage = s.sys2_leverage
|
||||
if s.sys2_mode is not None:
|
||||
sub.sys2_mode = s.sys2_mode
|
||||
# Empty string = reset to default; otherwise store the chosen mode.
|
||||
sub.sys2_mode = "standard" if s.sys2_mode == "" else s.sys2_mode
|
||||
sub.active_from = af
|
||||
sub.active_until = au
|
||||
if s.trump_enabled is not None:
|
||||
|
||||
+11
-4
@@ -6,10 +6,10 @@ class Settings(BaseSettings):
|
||||
frontend_url: str = "http://localhost:3001"
|
||||
truth_social_rss: str = "https://truthsocial.com/@realDonaldTrump.rss"
|
||||
truth_social_poll_seconds: int = 15
|
||||
binance_ws_url: str = (
|
||||
"wss://data-stream.binance.vision/stream"
|
||||
"?streams=btcusdt@kline_1m/ethusdt@kline_1m"
|
||||
)
|
||||
# WS URL is now built programmatically in binance.py from ASSET_MAP so the
|
||||
# stream list and the routing table are always in sync. This setting is kept
|
||||
# for external callers that still reference it; binance.py ignores it.
|
||||
binance_ws_url: str = "wss://data-stream.binance.vision/stream" # base only
|
||||
binance_rest_url: str = "https://data-api.binance.vision"
|
||||
environment: str = "production"
|
||||
ai_api_key: str = ""
|
||||
@@ -38,6 +38,9 @@ class Settings(BaseSettings):
|
||||
# in the X-Ingest-Key header. Empty (default) = ingest endpoint is REJECTED
|
||||
# entirely until a key is configured (fail-closed).
|
||||
ingest_api_key: str = ""
|
||||
# Base URL for internal signal ingest (used by scanners). In Docker, set to
|
||||
# the service name (e.g. http://api:8000); bare-metal defaults to localhost.
|
||||
ingest_base_url: str = "http://localhost:8000"
|
||||
|
||||
# Glassnode on-chain data. Used only by the BTC bottom-reversal state
|
||||
# machine (MVRV-Z + STH-SOPR). Empty = scanner logs and skips fail-closed.
|
||||
@@ -56,6 +59,10 @@ class Settings(BaseSettings):
|
||||
# Bot username (no @) — used by the Settings UI to render the deep link
|
||||
# t.me/<username>?start=<code>. Example: "trumpalpha_bot".
|
||||
telegram_bot_username: str = ""
|
||||
# Public broadcast channel chat_id (e.g. "-1001234567890" or "@trumpalpha").
|
||||
# When set, every actionable signal is also posted here in sanitised form
|
||||
# (no trading details, no /adopt CTA). Leave empty to disable.
|
||||
telegram_public_channel_id: str = ""
|
||||
|
||||
model_config = {"env_file": ".env", "env_file_encoding": "utf-8", "extra": "ignore"}
|
||||
|
||||
|
||||
+28
-5
@@ -3,8 +3,11 @@ import logging
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
from apscheduler.schedulers.asyncio import AsyncIOScheduler
|
||||
from fastapi import FastAPI, WebSocket, WebSocketDisconnect
|
||||
from fastapi import FastAPI, Request, WebSocket, WebSocketDisconnect
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
from slowapi import Limiter, _rate_limit_exceeded_handler
|
||||
from slowapi.errors import RateLimitExceeded
|
||||
from slowapi.util import get_remote_address
|
||||
|
||||
from app.config import settings
|
||||
from app.database import AsyncSessionLocal, engine
|
||||
@@ -12,6 +15,7 @@ from app.models import Base
|
||||
from app.scrapers.truth_social import poll_truth_social, backfill_history
|
||||
from app.scrapers.trumpstruth import poll_trumpstruth
|
||||
from app.services.binance import run_binance_ws
|
||||
from app.services.hl_price_feed import run_hl_price_feed
|
||||
from app.ws.manager import manager
|
||||
|
||||
# Import all routers
|
||||
@@ -37,14 +41,15 @@ logging.basicConfig(
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
from typing import Optional
|
||||
_binance_task: Optional[asyncio.Task] = None
|
||||
_telegram_task: Optional[asyncio.Task] = None
|
||||
_scheduler: Optional[AsyncIOScheduler] = None
|
||||
_binance_task: Optional[asyncio.Task] = None
|
||||
_hl_price_task: Optional[asyncio.Task] = None
|
||||
_telegram_task: Optional[asyncio.Task] = None
|
||||
_scheduler: Optional[AsyncIOScheduler] = None
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
global _binance_task, _telegram_task, _scheduler
|
||||
global _binance_task, _hl_price_task, _telegram_task, _scheduler
|
||||
|
||||
# 1. Dev convenience only. Production should rely on Alembic so schema
|
||||
# ownership stays explicit and startup never mutates the DB implicitly.
|
||||
@@ -67,6 +72,10 @@ async def lifespan(app: FastAPI):
|
||||
_binance_task = asyncio.create_task(run_binance_ws(), name="binance_ws")
|
||||
logger.info("Binance WebSocket task started.")
|
||||
|
||||
# 3b. Supplemental HL price feed for assets not on Binance (HYPE, PURR, …)
|
||||
_hl_price_task = asyncio.create_task(run_hl_price_feed(), name="hl_price_feed")
|
||||
logger.info("HL supplemental price feed task started.")
|
||||
|
||||
# 3. Start Truth Social poller via APScheduler
|
||||
_scheduler = AsyncIOScheduler()
|
||||
# Signal monitor — polls every 5 minutes
|
||||
@@ -251,6 +260,12 @@ async def lifespan(app: FastAPI):
|
||||
await _binance_task
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
if _hl_price_task and not _hl_price_task.done():
|
||||
_hl_price_task.cancel()
|
||||
try:
|
||||
await _hl_price_task
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
if _telegram_task and not _telegram_task.done():
|
||||
_telegram_task.cancel()
|
||||
try:
|
||||
@@ -261,12 +276,20 @@ async def lifespan(app: FastAPI):
|
||||
logger.info("Shutdown complete.")
|
||||
|
||||
|
||||
# Rate limiter — keyed by client IP (X-Forwarded-For → remote_addr fallback).
|
||||
# Public read endpoints: 60 req/min. Signed mutations: 20 req/min (enforced
|
||||
# per-route). Limits are generous enough for normal use but block scrapers
|
||||
# and accidental polling loops.
|
||||
limiter = Limiter(key_func=get_remote_address, default_limits=["60/minute"])
|
||||
|
||||
app = FastAPI(
|
||||
title="TrumpSignal API",
|
||||
version="1.0.0",
|
||||
description="Crypto trading signals derived from Trump's Truth Social posts.",
|
||||
lifespan=lifespan,
|
||||
)
|
||||
app.state.limiter = limiter
|
||||
app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler)
|
||||
|
||||
# CORS
|
||||
# In production we only allow the canonical frontend origin (FRONTEND_URL).
|
||||
|
||||
@@ -8,6 +8,7 @@ from sqlalchemy import (
|
||||
DateTime,
|
||||
Float,
|
||||
ForeignKey,
|
||||
Index,
|
||||
Integer,
|
||||
String,
|
||||
Text,
|
||||
@@ -84,6 +85,12 @@ class Candle(Base):
|
||||
|
||||
class BotTrade(Base):
|
||||
__tablename__ = "bot_trades"
|
||||
__table_args__ = (
|
||||
# Composite index for the two most common query patterns:
|
||||
# circuit_breaker: WHERE wallet_address=? AND closed_at>=? AND pnl_usd IS NOT NULL
|
||||
# trades API: WHERE wallet_address=? ORDER BY closed_at DESC
|
||||
Index("ix_bot_trades_wallet_closed", "wallet_address", "closed_at"),
|
||||
)
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True, index=True)
|
||||
asset: Mapped[str] = mapped_column(String(8), nullable=False)
|
||||
@@ -242,6 +249,12 @@ class Subscription(Base):
|
||||
# acknowledges + clears a tripped circuit breaker (human-in-the-loop).
|
||||
auto_trade: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
|
||||
|
||||
# ── Module on/off toggles ───────────────────────────────────────────────
|
||||
# Both default False so newly-subscribed users start with the bot idle
|
||||
# until they explicitly enable a module and save settings.
|
||||
trump_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
|
||||
macro_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
|
||||
|
||||
|
||||
class KolPost(Base):
|
||||
"""A long-form post (Substack) or tweet from a tracked KOL, with inline
|
||||
|
||||
@@ -131,6 +131,7 @@ class UserSettings(BaseModel):
|
||||
sys2_leverage: Optional[int] = None
|
||||
# System-2 risk mode: "standard" (default) or "aggressive" (separately
|
||||
# funded high-risk/high-explosiveness sleeve). None → unchanged/standard.
|
||||
# "standard" | "aggressive" | "" (empty string = reset to default "standard")
|
||||
sys2_mode: Optional[str] = None
|
||||
# ISO-8601 UTC strings; both None = always on (Subscription.active still gates it).
|
||||
active_from: Optional[str] = None
|
||||
|
||||
@@ -112,12 +112,19 @@ async def _process_entry(entry: dict, db: AsyncSession) -> Optional[Post]:
|
||||
analysis = await analyze_post(text)
|
||||
|
||||
asset = analysis["asset"]
|
||||
# `tracked_asset`: the asset whose price impact we measure and display.
|
||||
# Use target_asset (the perp we actually trade — may be SOL/TRUMP/etc.)
|
||||
# when available; fall back to the sentiment asset (BTC/ETH) otherwise.
|
||||
# Bug fix: previously always used `asset` (BTC/ETH), which measured the
|
||||
# wrong price move when the bot traded a different perp.
|
||||
tracked_asset = analysis.get("target_asset") or asset
|
||||
|
||||
# Only capture the price AT post time. The m5/m15/m1h peaks are filled in
|
||||
# asynchronously by price_impact_monitor as the windows elapse — avoids
|
||||
# recording 0.00% because future candles don't exist yet at entry time.
|
||||
price_at_post = None
|
||||
if asset and analysis["relevant"]:
|
||||
price_at_post = price_store.get_price_at(asset, published_at)
|
||||
if tracked_asset and analysis["relevant"]:
|
||||
price_at_post = price_store.get_price_at(tracked_asset, published_at)
|
||||
|
||||
post = Post(
|
||||
external_id=external_id,
|
||||
@@ -131,8 +138,8 @@ async def _process_entry(entry: dict, db: AsyncSession) -> Optional[Post]:
|
||||
prefilter_reason=analysis.get("prefilter_reason"),
|
||||
analysis_version=analysis.get("analysis_version"),
|
||||
relevant=analysis["relevant"],
|
||||
# `asset` (BTC/ETH only) feeds the existing price_impact tracker.
|
||||
price_impact_asset=asset if analysis["relevant"] else None,
|
||||
# Track the actually-traded asset (target_asset ?? sentiment_asset).
|
||||
price_impact_asset=tracked_asset if analysis["relevant"] else None,
|
||||
price_impact_m5=None, # filled by price_impact_monitor after 5 m
|
||||
price_impact_m15=None, # filled by price_impact_monitor after 15 m
|
||||
price_impact_m1h=None, # filled by price_impact_monitor after 1 h
|
||||
@@ -146,11 +153,11 @@ async def _process_entry(entry: dict, db: AsyncSession) -> Optional[Post]:
|
||||
await db.flush()
|
||||
|
||||
# Register with the live peak tracker so it starts watching immediately.
|
||||
if asset and analysis["relevant"] and price_at_post:
|
||||
if tracked_asset and analysis["relevant"] and price_at_post:
|
||||
from app.services.price_impact_monitor import register_post
|
||||
register_post(
|
||||
post_id=post.id,
|
||||
asset=asset,
|
||||
asset=tracked_asset,
|
||||
signal=analysis.get("signal"),
|
||||
entry_price=price_at_post,
|
||||
published_at=published_at,
|
||||
@@ -219,8 +226,9 @@ async def poll_truth_social(db_session_factory) -> None:
|
||||
for post in new_posts:
|
||||
await manager.broadcast(_post_to_ws_payload(post))
|
||||
logger.info("Saved new post id=%d: %s", post.id, post.text[:60])
|
||||
# Telegram fan-out (fire-and-forget). Only fires if
|
||||
# signal is buy/short; noise posts are filtered inside.
|
||||
# Telegram fan-out (fire-and-forget). _dispatch filters
|
||||
# internally: buy/short → per-subscriber + public channel;
|
||||
# relevant-but-hold → public channel only; noise → dropped.
|
||||
try:
|
||||
from app.services.telegram import notify_signal
|
||||
notify_signal(post)
|
||||
|
||||
@@ -31,9 +31,10 @@ from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from collections import OrderedDict
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timezone
|
||||
from typing import Dict, Optional
|
||||
from typing import Optional
|
||||
|
||||
from sqlalchemy import select, update
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
@@ -66,14 +67,28 @@ ADOPTED_CATEGORY = "btc_bottom_reversal_long"
|
||||
# (each row would race to reduce the position). Process-local; with the
|
||||
# atomic duplicate check inside the lock, multi-process deploys are still
|
||||
# safe (losers would just see "already_adopted" on the second attempt).
|
||||
_adopt_locks: Dict[str, asyncio.Lock] = {}
|
||||
#
|
||||
# Capacity-capped at _ADOPT_LOCK_MAX (matches _WALLET_LOCK_MAX in bot_engine)
|
||||
# to prevent unbounded growth if many unique wallets pass through. LRU
|
||||
# eviction: evicting an in-flight lock is safe — the holder already holds a
|
||||
# reference; new adopt for that wallet just gets a fresh lock.
|
||||
_ADOPT_LOCK_MAX = 512
|
||||
_adopt_locks: OrderedDict[str, asyncio.Lock] = OrderedDict()
|
||||
|
||||
|
||||
def _wallet_adopt_lock(wallet: str) -> asyncio.Lock:
|
||||
lock = _adopt_locks.get(wallet)
|
||||
if lock is None:
|
||||
if len(_adopt_locks) >= _ADOPT_LOCK_MAX:
|
||||
# Evict LRU entry. The evicted lock object stays alive as long as
|
||||
# any coroutine holds a reference to it — eviction only means it
|
||||
# won't be reused, not that it's freed mid-acquire.
|
||||
_adopt_locks.popitem(last=False)
|
||||
lock = asyncio.Lock()
|
||||
_adopt_locks[wallet] = lock
|
||||
else:
|
||||
# Promote to MRU position.
|
||||
_adopt_locks.move_to_end(wallet)
|
||||
return lock
|
||||
|
||||
|
||||
@@ -311,6 +326,20 @@ async def _adopt_locked(wallet_l: str, asset_u: str, mode_n: str) -> AdoptionRes
|
||||
lev_info = target.get("leverage") or {}
|
||||
leverage = int(lev_info.get("value") or sub.leverage or 2)
|
||||
|
||||
# BUG-09 guard: sys2_protective_stop_pct clamps leverage to SYS2_MAX_LEVERAGE
|
||||
# when computing the stop distance. If the actual position is above that cap
|
||||
# the computed stop is WIDER than the real liquidation band — the user could
|
||||
# get HL-liquidated before our stop fires. Reject up front with a clear
|
||||
# error so the user knows to reduce leverage on HL before adopting.
|
||||
if leverage > SYS2_MAX_LEVERAGE:
|
||||
raise AdoptionError(
|
||||
"leverage_too_high",
|
||||
f"{asset_u} position is at {leverage}× leverage. "
|
||||
f"The bot's System-2 strategy supports up to {SYS2_MAX_LEVERAGE}×. "
|
||||
f"Reduce leverage on Hyperliquid to ≤{SYS2_MAX_LEVERAGE}× first, "
|
||||
"then try /adopt again.",
|
||||
)
|
||||
|
||||
# Build the System-2 exit profile against the ACTUAL HL leverage so the
|
||||
# protective stop is always inside the real liquidation line. Identical
|
||||
# math to bot_engine's old sys2 open path; reused so behaviour stays
|
||||
|
||||
@@ -307,8 +307,9 @@ HARD CONSTRAINTS on the JSON:
|
||||
|
||||
|
||||
USER_PROMPT_TEMPLATE = """\
|
||||
Trump just posted on Truth Social. Score it for an immediate BTC/ETH
|
||||
trade per your system rules.
|
||||
Trump just posted on Truth Social. Score it for an immediate crypto
|
||||
trade per your system rules. The tradeable universe includes BTC, ETH,
|
||||
SOL, named alts, and meme coins — not just BTC/ETH.
|
||||
|
||||
Current UTC hour: {hour} ({liquidity_note})
|
||||
|
||||
@@ -344,6 +345,9 @@ HL_PERPS = {
|
||||
"BTC", "ETH", "SOL", "AVAX", "ARB", "OP", "DOGE", "SHIB", "PEPE",
|
||||
"WIF", "TRUMP", "SUI", "APT", "INJ", "ATOM", "ADA", "MATIC",
|
||||
"FARTCOIN", "BNB", "LINK", "LTC", "XRP",
|
||||
# Added 2025-2026: Hyperliquid native + high-volume memes/alts
|
||||
"HYPE", "BONK", "RENDER", "FET", "TAO", "JUP", "PYTH", "TIA",
|
||||
"SEI", "STRK", "MANTA", "ALT", "PIXEL", "PORTAL", "PENDLE",
|
||||
}
|
||||
|
||||
# Chain → its native token's HL ticker. Used when a meme isn't on HL —
|
||||
|
||||
+39
-8
@@ -11,11 +11,38 @@ from app.ws.manager import manager
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
ASSET_MAP = {
|
||||
"btcusdt": "BTC",
|
||||
"ethusdt": "ETH",
|
||||
# symbol (lowercase Binance ticker) → asset (HL perp name, uppercase).
|
||||
#
|
||||
# WHY THIS MATTERS: tp_sl_monitor.on_price_tick(asset, price) is the sole
|
||||
# mechanism that fires stop-loss, take-profit, and trailing-stop for live
|
||||
# trades. If an asset is missing here, every trade on that asset silently
|
||||
# loses all TP/SL protection — only the asyncio max-hold timer remains.
|
||||
#
|
||||
# Trump's AI picks target_asset from the full HL perp universe. The assets
|
||||
# below cover the realistic set (BTC/ETH always, SOL/TRUMP/BNB/DOGE/LINK/AAVE
|
||||
# for direct-named or thematic posts). HL-native assets (HYPE, PURR) are NOT
|
||||
# on Binance; those require a separate HL price feed (see BUG-08 note in CLAUDE.md).
|
||||
# Until that feed is added, HYPE trades fall back to max-hold only.
|
||||
ASSET_MAP: dict[str, str] = {
|
||||
"btcusdt": "BTC",
|
||||
"ethusdt": "ETH",
|
||||
"solusdt": "SOL",
|
||||
"trumpusdt": "TRUMP",
|
||||
"bnbusdt": "BNB",
|
||||
"dogeusdt": "DOGE",
|
||||
"linkusdt": "LINK",
|
||||
"aaveusdt": "AAVE",
|
||||
}
|
||||
|
||||
# Build the combined-stream WS URL from ASSET_MAP so the two are always in sync.
|
||||
# Format: wss://...?streams=sym1@kline_1m/sym2@kline_1m/...
|
||||
_BINANCE_WS_BASE = "wss://data-stream.binance.vision/stream"
|
||||
BINANCE_WS_URL = (
|
||||
_BINANCE_WS_BASE
|
||||
+ "?streams="
|
||||
+ "/".join(f"{sym}@kline_1m" for sym in ASSET_MAP)
|
||||
)
|
||||
|
||||
|
||||
async def _process_message(raw: str):
|
||||
try:
|
||||
@@ -98,16 +125,20 @@ async def run_binance_ws():
|
||||
* max_queue — drop frames if downstream can't keep up rather than back
|
||||
the WS receive buffer up indefinitely
|
||||
"""
|
||||
# Pre-fill with historical data
|
||||
await fetch_historical("BTC", "btcusdt", limit=500)
|
||||
await fetch_historical("ETH", "ethusdt", limit=500)
|
||||
# Pre-fill historical candles for every tracked asset so price_store has
|
||||
# a warm baseline before the first live tick arrives.
|
||||
await asyncio.gather(*(
|
||||
fetch_historical(asset, symbol, limit=500)
|
||||
for symbol, asset in ASSET_MAP.items()
|
||||
), return_exceptions=True)
|
||||
|
||||
backoff = 1
|
||||
while True:
|
||||
try:
|
||||
logger.info("Connecting to Binance WebSocket: %s", settings.binance_ws_url)
|
||||
logger.info("Connecting to Binance WebSocket (%d assets): %s",
|
||||
len(ASSET_MAP), BINANCE_WS_URL)
|
||||
async with websockets.connect(
|
||||
settings.binance_ws_url,
|
||||
BINANCE_WS_URL,
|
||||
ping_interval=20,
|
||||
ping_timeout=10,
|
||||
close_timeout=5,
|
||||
|
||||
+116
-23
@@ -46,12 +46,19 @@ _background_tasks: set = set()
|
||||
# Per-wallet locks for the open-position critical section.
|
||||
# Prevents two concurrent signals from both passing the daily-budget check
|
||||
# before either trade is written to DB (TOCTOU race).
|
||||
# Cap at 512 wallets; evict the oldest entry when full (simple FIFO approximation
|
||||
# — a wallet that hasn't traded in a long time doesn't need a warm lock).
|
||||
_WALLET_LOCK_MAX = 512
|
||||
_wallet_open_locks: Dict[str, asyncio.Lock] = {}
|
||||
|
||||
|
||||
def _wallet_lock(wallet: str) -> asyncio.Lock:
|
||||
lock = _wallet_open_locks.get(wallet)
|
||||
if lock is None:
|
||||
if len(_wallet_open_locks) >= _WALLET_LOCK_MAX:
|
||||
# Evict the first (oldest) key — dict preserves insertion order in Python 3.7+
|
||||
oldest = next(iter(_wallet_open_locks))
|
||||
del _wallet_open_locks[oldest]
|
||||
lock = asyncio.Lock()
|
||||
_wallet_open_locks[wallet] = lock
|
||||
return lock
|
||||
@@ -375,13 +382,21 @@ async def _execute_for_subscriber(
|
||||
# trade is written to DB (TOCTOU race under asyncio.gather).
|
||||
async with _wallet_lock(wallet):
|
||||
# Re-check daily budget INSIDE the lock so it's atomic with the open.
|
||||
# Budget is SPLIT between the two systems so a Trump scalp can't eat
|
||||
# the allocation reserved for a once-a-year reversal signal.
|
||||
# System-2 is now MANAGE-ONLY (no auto-open), so the budget split is
|
||||
# only meaningful when a sys2 auto-open path could actually run. For
|
||||
# System-1 (Trump), give the full daily budget — sys2_pct is no longer
|
||||
# "reserved" for auto-opens that can't happen. Users who want a tighter
|
||||
# Trump cap can lower daily_budget_usd directly.
|
||||
total_cap = sub.get("daily_budget_usd")
|
||||
if total_cap is not None and total_cap > 0:
|
||||
sys2_pct = sub.get("sys2_budget_pct", 0.7)
|
||||
is_s2 = bool(sub.get("_is_system_2"))
|
||||
daily_cap = total_cap * (sys2_pct if is_s2 else (1.0 - sys2_pct))
|
||||
# sys2 auto-opens are disabled; no System-2 trades will ever run
|
||||
# this code path (process_post returns early for sys2 sources).
|
||||
# Keep the split logic here so it would still work correctly if
|
||||
# sys2 auto-open is ever re-enabled via an ADR, but for sys1
|
||||
# (Trump) use the full cap rather than the 30% remainder.
|
||||
daily_cap = total_cap * (sys2_pct if is_s2 else 1.0)
|
||||
start_of_day = datetime.now(timezone.utc).replace(hour=0, minute=0, second=0, microsecond=0, tzinfo=None)
|
||||
from app.models import Post as _P
|
||||
from app.services.signal_categories import SYSTEM_2_SOURCES
|
||||
@@ -404,7 +419,7 @@ async def _execute_for_subscriber(
|
||||
if spent + sized_position_usd > daily_cap:
|
||||
logger.info("Sub %s [%s] daily budget reached: spent=%.2f + new=%.2f > cap=%.2f (%.0f%% of %.2f)",
|
||||
wallet, _system, spent, sized_position_usd, daily_cap,
|
||||
(sys2_pct if is_s2 else 1.0 - sys2_pct) * 100, total_cap)
|
||||
(sys2_pct if is_s2 else 100.0), total_cap)
|
||||
return
|
||||
|
||||
# ── System-2 correlation / concentration cap ───────────────────────
|
||||
@@ -685,12 +700,35 @@ async def partial_derisk(
|
||||
# `frac_of_original` of the ORIGINAL notional.
|
||||
frac_of_current = max(0.0, min(1.0, frac_of_original / rem_frac))
|
||||
|
||||
# ── Pre-claim the step BEFORE touching HL (BUG-03 analog) ───
|
||||
# If we call HL first and THEN commit, a db.commit() failure
|
||||
# leaves derisk_steps_done unchanged → next tick re-triggers
|
||||
# reduce_position on a SMALLER position → double-reduce.
|
||||
# Pre-claiming burns the step on DB failure but that's safer
|
||||
# than over-reducing a live position. A burned step leaves
|
||||
# the position slightly larger than the ladder intended but
|
||||
# the NEXT rung still fires correctly.
|
||||
claim = await db.execute(
|
||||
update(BotTrade)
|
||||
.where(BotTrade.id == trade_id)
|
||||
.where(BotTrade.derisk_steps_done == step_idx)
|
||||
.values(derisk_steps_done=step_idx + 1)
|
||||
)
|
||||
await db.commit()
|
||||
if claim.rowcount == 0:
|
||||
# Another coroutine already advanced past this step.
|
||||
return True
|
||||
|
||||
if trade.hl_order_id == "paper":
|
||||
from app.services.price_store import price_store
|
||||
fill = price_store.latest_price(asset)
|
||||
if not fill:
|
||||
logger.error("Paper de-risk: no %s price, retry trade %d", asset, trade_id)
|
||||
return False
|
||||
logger.error("Paper de-risk: no %s price, skip trade %d step %d",
|
||||
asset, trade_id, step_idx)
|
||||
# Step already claimed — don't retry (would skip the
|
||||
# next rung). Remaining_fraction stays at pre-reduce
|
||||
# value; reconciler will catch the drift if material.
|
||||
return True
|
||||
closed_frac_of_current = frac_of_current
|
||||
else:
|
||||
trader = HyperliquidTrader(
|
||||
@@ -708,7 +746,16 @@ async def partial_derisk(
|
||||
fill = r.get("fill_price")
|
||||
closed_frac_of_current = r.get("closed_fraction") or 0.0
|
||||
if not fill or closed_frac_of_current <= 0:
|
||||
return False # no fill — retry next tick
|
||||
# HL returned no fill — position is unchanged on HL but
|
||||
# derisk_steps_done is already incremented. Log loudly
|
||||
# so the operator can investigate; returning True skips
|
||||
# a double-reduce at the cost of this rung being silent.
|
||||
logger.error(
|
||||
"De-risk step %d trade %d: HL returned no fill "
|
||||
"(step already claimed). Position unchanged on HL.",
|
||||
step_idx, trade_id,
|
||||
)
|
||||
return True
|
||||
|
||||
# Fraction of ORIGINAL notional actually shed this step.
|
||||
closed_frac_of_original = closed_frac_of_current * rem_frac
|
||||
@@ -719,16 +766,27 @@ async def partial_derisk(
|
||||
fees = slice_usd * HL_TAKER_FEE_RATE * 2
|
||||
slice_pnl = slice_usd * signed_pct - fees
|
||||
|
||||
trade.realized_partial_pnl_usd = round((trade.realized_partial_pnl_usd or 0.0) + slice_pnl, 4)
|
||||
trade.remaining_fraction = max(0.0, round(rem_frac - closed_frac_of_original, 6))
|
||||
trade.derisk_steps_done = step_idx + 1
|
||||
await db.commit()
|
||||
# Second DB write: update PnL + remaining fraction now that
|
||||
# we know the actual fill. derisk_steps_done is already committed.
|
||||
async with AsyncSessionLocal() as post_db:
|
||||
post_row = await post_db.execute(
|
||||
select(BotTrade).where(BotTrade.id == trade_id)
|
||||
)
|
||||
post_trade = post_row.scalar_one_or_none()
|
||||
if post_trade is not None and post_trade.closed_at is None:
|
||||
post_trade.realized_partial_pnl_usd = round(
|
||||
(post_trade.realized_partial_pnl_usd or 0.0) + slice_pnl, 4
|
||||
)
|
||||
post_trade.remaining_fraction = max(
|
||||
0.0, round((post_trade.remaining_fraction or 1.0) - closed_frac_of_original, 6)
|
||||
)
|
||||
await post_db.commit()
|
||||
|
||||
logger.warning(
|
||||
"De-risk step %d trade %d (%s): closed %.1f%% of original "
|
||||
"@ %.2f, slice PnL %.2f, remaining %.1f%% (reason=%s)",
|
||||
"@ %.2f, slice PnL %.2f (reason=%s)",
|
||||
step_idx + 1, trade_id, asset, closed_frac_of_original * 100,
|
||||
fill, slice_pnl, trade.remaining_fraction * 100, reason,
|
||||
fill, slice_pnl, reason,
|
||||
)
|
||||
return True
|
||||
except Exception as e:
|
||||
@@ -799,6 +857,20 @@ async def pyramid_add(
|
||||
if not trend_confirmed(closes, highs, px):
|
||||
return (False, None, None) # not a confirmed uptrend yet
|
||||
|
||||
# Pre-claim the step BEFORE touching HL (BUG-03 analog).
|
||||
# Prevents double-add when HL open_position succeeds but the
|
||||
# subsequent DB commit fails — the next retrigger sees the step
|
||||
# already claimed and returns early instead of re-opening.
|
||||
claim = await db.execute(
|
||||
update(BotTrade)
|
||||
.where(BotTrade.id == trade_id)
|
||||
.where(BotTrade.addon_steps_done == step_idx)
|
||||
.values(addon_steps_done=step_idx + 1)
|
||||
)
|
||||
await db.commit()
|
||||
if claim.rowcount == 0:
|
||||
return (True, None, None) # already claimed by another coroutine
|
||||
|
||||
if trade.hl_order_id == "paper":
|
||||
fill = px
|
||||
actual_add_usd = add_usd # synthetic full fill
|
||||
@@ -822,9 +894,15 @@ async def pyramid_add(
|
||||
old_notional = trade.size_usd or base
|
||||
new_notional = old_notional + actual_add_usd
|
||||
blended = (old_notional * trade.entry_price + actual_add_usd * fill) / new_notional
|
||||
trade.entry_price = round(blended, 6)
|
||||
trade.size_usd = round(new_notional, 2)
|
||||
trade.addon_steps_done = step_idx + 1
|
||||
# addon_steps_done already committed via pre-claim above
|
||||
await db.execute(
|
||||
update(BotTrade)
|
||||
.where(BotTrade.id == trade_id)
|
||||
.values(
|
||||
entry_price=round(blended, 6),
|
||||
size_usd=round(new_notional, 2),
|
||||
)
|
||||
)
|
||||
await db.commit()
|
||||
|
||||
logger.warning(
|
||||
@@ -884,6 +962,9 @@ async def close_and_finalize(
|
||||
# Either someone else closed it, or (automated path) the
|
||||
# user released it between the price-tick capture and
|
||||
# this close attempt. Either way: bail without touching HL.
|
||||
# Clean up the lock we just created so _close_locks doesn't
|
||||
# accumulate one entry per losing-racer per trade_id.
|
||||
_close_locks.pop(trade_id, None)
|
||||
return
|
||||
|
||||
# Reload the row we just claimed
|
||||
@@ -1001,15 +1082,27 @@ async def close_and_finalize(
|
||||
# Run after PnL is written. If daily DD or consecutive-loss
|
||||
# threshold hit, trip the breaker — that nulls manual_window
|
||||
# and blocks new trades for CB_LOCKOUT_HOURS.
|
||||
# Infer which system this trade belonged to from its trigger
|
||||
# post's source, so the right (independent) breaker is checked.
|
||||
# Infer which system this trade belonged to so the right
|
||||
# independent circuit breaker is checked.
|
||||
#
|
||||
# Three hl_order_id shapes:
|
||||
# - integer string → System-1 auto-open (Trump)
|
||||
# - "paper" → paper-mode (System-1 or legacy sys2)
|
||||
# - "adopted:<ts>" → System-2 adopted position
|
||||
# Adopted trades have trigger_post_id=NULL, so querying
|
||||
# Post.source returns nothing and would incorrectly identify
|
||||
# them as sys1. Check the hl_order_id prefix FIRST.
|
||||
from app.services.circuit_breaker import check_and_trip
|
||||
from app.services.signal_categories import SYSTEM_2_SOURCES
|
||||
_src_row = await db.execute(
|
||||
select(Post.source).where(Post.id == trade.trigger_post_id)
|
||||
)
|
||||
_src = (_src_row.scalar_one_or_none() or "").lower()
|
||||
_sys = "sys2" if _src in SYSTEM_2_SOURCES else "sys1"
|
||||
_is_adopted = (trade.hl_order_id or "").startswith("adopted:")
|
||||
if _is_adopted or trade.sys2_mode is not None:
|
||||
_sys = "sys2"
|
||||
else:
|
||||
_src_row = await db.execute(
|
||||
select(Post.source).where(Post.id == trade.trigger_post_id)
|
||||
)
|
||||
_src = (_src_row.scalar_one_or_none() or "").lower()
|
||||
_sys = "sys2" if _src in SYSTEM_2_SOURCES else "sys1"
|
||||
cb_reason = await check_and_trip(wallet, db, _sys)
|
||||
if cb_reason:
|
||||
logger.warning("Circuit breaker [%s] tripped for %s: %s",
|
||||
|
||||
@@ -61,12 +61,17 @@ _CB_COLS = {
|
||||
}
|
||||
|
||||
|
||||
async def _trades_for_system(db: AsyncSession, wallet: str, since, system: str):
|
||||
async def _trades_for_system(db: AsyncSession, wallet: str, since, system: str, limit: int | None = None):
|
||||
"""Closed, priced trades for `wallet` since `since`, filtered to the
|
||||
given system by joining the trigger post's source.
|
||||
|
||||
sys2 = trigger_post.source in System-2 set; sys1 = everything else
|
||||
(currently only "truth"). Trades with no trigger post → treated sys1.
|
||||
sys2 = trigger_post.source in System-2 set OR adopted trade (identified
|
||||
by hl_order_id starting with "adopted:" or sys2_mode being non-NULL).
|
||||
sys1 = everything else (currently only "truth").
|
||||
|
||||
Important: adopted trades have trigger_post_id=NULL so their Post join
|
||||
returns NULL source — they must be identified by the trade row itself,
|
||||
not the linked post.
|
||||
"""
|
||||
from app.models import Post
|
||||
from app.services.signal_categories import SYSTEM_2_SOURCES
|
||||
@@ -80,10 +85,17 @@ async def _trades_for_system(db: AsyncSession, wallet: str, since, system: str):
|
||||
BotTrade.pnl_usd.is_not(None),
|
||||
)
|
||||
.order_by(BotTrade.closed_at.desc())
|
||||
.limit(limit * 4 if limit else None) # fetch a small multiple to account for system filtering
|
||||
)
|
||||
out = []
|
||||
for trade, src in rows.all():
|
||||
is_s2 = (src or "").lower() in SYSTEM_2_SOURCES
|
||||
# Adopted trades: trigger_post_id=NULL, hl_order_id starts with "adopted:"
|
||||
# or sys2_mode is set. Check trade row directly before falling back to source.
|
||||
is_s2 = (
|
||||
(trade.hl_order_id or "").startswith("adopted:")
|
||||
or trade.sys2_mode is not None
|
||||
or (src or "").lower() in SYSTEM_2_SOURCES
|
||||
)
|
||||
if (system == "sys2") == is_s2:
|
||||
out.append(trade)
|
||||
return out
|
||||
@@ -124,9 +136,10 @@ async def check_and_trip(
|
||||
system, wallet, today_pnl, dd_limit)
|
||||
|
||||
if reason is None:
|
||||
# Consecutive losses within this system (all-time, most recent N).
|
||||
# Consecutive losses within this system — only need the most recent
|
||||
# CB_CONSEC_LOSSES trades; no reason to scan full history.
|
||||
all_sys = await _trades_for_system(
|
||||
db, wallet, datetime(1970, 1, 1), system,
|
||||
db, wallet, datetime(1970, 1, 1), system, limit=CB_CONSEC_LOSSES,
|
||||
)
|
||||
recent = all_sys[:CB_CONSEC_LOSSES]
|
||||
if len(recent) >= CB_CONSEC_LOSSES and all((t.pnl_usd or 0) < 0 for t in recent):
|
||||
|
||||
@@ -0,0 +1,139 @@
|
||||
"""Supplemental price feed for HL-native assets not listed on Binance.
|
||||
|
||||
Polls HL allMids endpoint every 2 seconds and pumps price_store +
|
||||
tp_sl_monitor for assets in HL_PRICE_ASSETS. Runs alongside run_binance_ws().
|
||||
|
||||
WHY THIS EXISTS:
|
||||
tp_sl_monitor.on_price_tick(asset, price) is the sole mechanism that fires
|
||||
stop-loss, take-profit, and trailing-stop for live bot trades.
|
||||
HL-native assets (HYPE, PURR, …) are not listed on Binance, so the
|
||||
Binance WebSocket in binance.py cannot provide their prices.
|
||||
Without this feed, any bot trade on an HL-native asset silently loses ALL
|
||||
TP/SL protection — only the max_hold asyncio timer remains as an exit.
|
||||
|
||||
COVERAGE:
|
||||
• Binance WS → BTC, ETH, SOL, TRUMP, BNB, DOGE, LINK, AAVE (ASSET_MAP)
|
||||
• This feed → HYPE, PURR (HL_PRICE_ASSETS)
|
||||
|
||||
Adding a new HL-native asset:
|
||||
1. Add its symbol (uppercase) to HL_PRICE_ASSETS below.
|
||||
2. Verify the symbol appears in HL allMids by running:
|
||||
curl -s -X POST https://api.hyperliquid.xyz/info \\
|
||||
-H 'Content-Type: application/json' \\
|
||||
-d '{"type":"allMids"}' | python3 -m json.tool | grep HYPE
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import time
|
||||
from typing import Optional
|
||||
|
||||
import httpx
|
||||
|
||||
from app.services.price_store import price_store
|
||||
from app.ws.manager import manager
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
HL_API_URL = "https://api.hyperliquid.xyz/info"
|
||||
|
||||
# HL-native assets not covered by the Binance WS. Add new ones here as needed.
|
||||
HL_PRICE_ASSETS: frozenset[str] = frozenset({"HYPE", "PURR"})
|
||||
|
||||
# Poll cadence — 2 s gives ~same freshness as a Binance 1-min kline close
|
||||
# without hammering the free HL REST endpoint (30 req/min well within limit).
|
||||
_POLL_INTERVAL = 2.0
|
||||
|
||||
# Exponential backoff caps: start at 5 s, max at 60 s, reset on clean loop.
|
||||
_BACKOFF_START = 5
|
||||
_BACKOFF_MAX = 60
|
||||
|
||||
|
||||
async def run_hl_price_feed() -> None:
|
||||
"""Long-running loop. Retries with exponential backoff on connection failure.
|
||||
Designed to be created once as an asyncio task at startup and cancelled at
|
||||
shutdown — matches the run_binance_ws() lifecycle pattern."""
|
||||
logger.info("HL supplemental price feed starting for %s", sorted(HL_PRICE_ASSETS))
|
||||
backoff = _BACKOFF_START
|
||||
while True:
|
||||
try:
|
||||
await _poll_loop()
|
||||
except asyncio.CancelledError:
|
||||
logger.info("HL price feed cancelled — shutting down.")
|
||||
return
|
||||
except Exception as exc:
|
||||
logger.warning(
|
||||
"HL price feed outer error: %s — retrying in %ds", exc, backoff
|
||||
)
|
||||
await asyncio.sleep(backoff)
|
||||
backoff = min(backoff * 2, _BACKOFF_MAX)
|
||||
else:
|
||||
backoff = _BACKOFF_START # reset on clean exit (shouldn't happen normally)
|
||||
|
||||
|
||||
async def _poll_loop() -> None:
|
||||
"""Inner loop: tick every 2 s. Individual tick failures are swallowed so
|
||||
a transient HTTP 5xx doesn't break the whole loop."""
|
||||
while True:
|
||||
try:
|
||||
await _tick()
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception as exc:
|
||||
# Transient: network blip, HL rate limit, malformed JSON.
|
||||
logger.debug("HL price tick error (suppressed): %s", exc)
|
||||
await asyncio.sleep(_POLL_INTERVAL)
|
||||
|
||||
|
||||
async def _tick() -> None:
|
||||
"""Single price fetch + dispatch cycle for all HL_PRICE_ASSETS."""
|
||||
now_ms = int(time.time() * 1000)
|
||||
|
||||
async with httpx.AsyncClient(timeout=4.0) as c:
|
||||
r = await c.post(HL_API_URL, json={"type": "allMids"})
|
||||
r.raise_for_status()
|
||||
mids: dict = r.json() # {"BTC": "74541.0", "HYPE": "13.5", …}
|
||||
|
||||
for asset in HL_PRICE_ASSETS:
|
||||
raw: Optional[str] = mids.get(asset)
|
||||
if raw is None:
|
||||
# Asset not listed on HL yet — skip silently.
|
||||
continue
|
||||
try:
|
||||
price = float(raw)
|
||||
except (TypeError, ValueError):
|
||||
logger.warning("HL price feed: bad value for %s: %r", asset, raw)
|
||||
continue
|
||||
if price <= 0:
|
||||
continue
|
||||
|
||||
# Synthetic 1-minute candle aligned to the current minute bucket.
|
||||
# All OHLC set to the mid price (no spread data from allMids).
|
||||
# price_store.update() handles the same-bucket replace logic.
|
||||
candle = {
|
||||
"time": (now_ms // 60_000) * 60_000,
|
||||
"open": price,
|
||||
"high": price,
|
||||
"low": price,
|
||||
"close": price,
|
||||
"volume": 0.0,
|
||||
}
|
||||
price_store.update(asset, candle)
|
||||
|
||||
# TP/SL + trailing-stop evaluation on every tick.
|
||||
from app.services.tp_sl_monitor import on_price_tick
|
||||
on_price_tick(asset, price)
|
||||
|
||||
# Price-impact peak tracker for Trump-post windows.
|
||||
from app.services.price_impact_monitor import on_price_tick as pi_tick
|
||||
pi_tick(asset, price, price, price)
|
||||
|
||||
# Live price broadcast to the frontend dashboard.
|
||||
await manager.broadcast({
|
||||
"type": "price",
|
||||
"asset": asset,
|
||||
"price": price,
|
||||
"time": now_ms,
|
||||
})
|
||||
@@ -12,6 +12,7 @@ Both are read from config (hl_account_address, hl_api_private_key).
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import time
|
||||
from functools import partial
|
||||
from typing import Optional
|
||||
|
||||
@@ -28,6 +29,17 @@ HL_TESTNET_URL = "https://api.hyperliquid-testnet.xyz"
|
||||
# Fetched dynamically via meta() but cached here as fallback
|
||||
SZ_DECIMALS_FALLBACK = {"BTC": 5, "ETH": 4}
|
||||
|
||||
# Module-level cache: coin → (szDecimals, expiry_timestamp)
|
||||
_SZ_DECIMALS_CACHE: dict[str, tuple[int, float]] = {}
|
||||
_SZ_DECIMALS_TTL = 300 # 5 minutes
|
||||
|
||||
# Max-leverage cache: coin → (maxLeverage, expiry_timestamp).
|
||||
# meta() is a full-universe call (~1 KB JSON), cheap but wasteful on every
|
||||
# trade open when the leverage caps change at most once a month. 5-min TTL
|
||||
# matches _SZ_DECIMALS_TTL so a single meta() response can refresh both.
|
||||
_MAX_LEV_CACHE: dict[str, tuple[int, float]] = {}
|
||||
_MAX_LEV_TTL = 300 # 5 minutes
|
||||
|
||||
|
||||
class HyperliquidTrader:
|
||||
def __init__(
|
||||
@@ -65,11 +77,18 @@ class HyperliquidTrader:
|
||||
return await loop.run_in_executor(None, partial(fn, *args, **kwargs))
|
||||
|
||||
async def _get_sz_decimals(self, coin: str) -> int:
|
||||
now = time.monotonic()
|
||||
cached = _SZ_DECIMALS_CACHE.get(coin)
|
||||
if cached is not None and now < cached[1]:
|
||||
return cached[0]
|
||||
try:
|
||||
meta = await self._run(self._info.meta)
|
||||
for u in meta.get("universe", []):
|
||||
if u.get("name") == coin:
|
||||
return int(u.get("szDecimals", SZ_DECIMALS_FALLBACK.get(coin, 4)))
|
||||
name = u.get("name")
|
||||
val = int(u.get("szDecimals", SZ_DECIMALS_FALLBACK.get(name, 4)))
|
||||
_SZ_DECIMALS_CACHE[name] = (val, now + _SZ_DECIMALS_TTL)
|
||||
if coin in _SZ_DECIMALS_CACHE:
|
||||
return _SZ_DECIMALS_CACHE[coin][0]
|
||||
except Exception:
|
||||
pass
|
||||
return SZ_DECIMALS_FALLBACK.get(coin, 4)
|
||||
@@ -77,14 +96,26 @@ class HyperliquidTrader:
|
||||
async def _get_max_leverage(self, coin: str) -> int:
|
||||
"""Hyperliquid caps max leverage per asset (BTC/ETH 50×, SOL 20×,
|
||||
memes typically 3-5×). Querying meta() returns each asset's
|
||||
`maxLeverage`. We cache nothing — meta() is fast and HL can change
|
||||
tiers. Returns a conservative 3 if lookup fails (memes default)."""
|
||||
`maxLeverage`. Returns a conservative 3 if lookup fails (memes default).
|
||||
|
||||
Results are cached for _MAX_LEV_TTL seconds (same TTL as szDecimals).
|
||||
meta() returns the full universe in one call so we populate all coins
|
||||
on a cache miss to amortise the cost across concurrent opens.
|
||||
"""
|
||||
import time as _time
|
||||
now = _time.time()
|
||||
cached = _MAX_LEV_CACHE.get(coin)
|
||||
if cached is not None and cached[1] > now:
|
||||
return cached[0]
|
||||
try:
|
||||
meta = await self._run(self._info.meta)
|
||||
expiry = now + _MAX_LEV_TTL
|
||||
for u in meta.get("universe", []):
|
||||
if u.get("name") == coin:
|
||||
ml = int(u.get("maxLeverage", 3))
|
||||
return max(1, ml)
|
||||
name = u.get("name")
|
||||
if name:
|
||||
_MAX_LEV_CACHE[name] = (max(1, int(u.get("maxLeverage", 3))), expiry)
|
||||
if coin in _MAX_LEV_CACHE:
|
||||
return _MAX_LEV_CACHE[coin][0]
|
||||
except Exception as exc:
|
||||
logger.warning("_get_max_leverage failed for %s: %s", coin, exc)
|
||||
return 3 # safe fallback for unknown / illiquid coin
|
||||
|
||||
@@ -4,8 +4,9 @@ Takes a long-form post (Substack essay) or tweet and returns:
|
||||
- summary: one Chinese sentence on what this post is about
|
||||
- tickers: list of {ticker, action, conviction, quote}
|
||||
|
||||
action ∈ bullish | bearish | buy | sell | mention
|
||||
action ∈ bullish | bearish | buy | sell | reduce | mention
|
||||
- buy/sell → KOL explicitly states they bought/sold or are entering/exiting
|
||||
- reduce → KOL is partially exiting / taking profits on an existing long
|
||||
- bullish/bearish → directional view without an explicit position statement
|
||||
- mention → ticker appears but no clear stance (don't flood with these)
|
||||
|
||||
@@ -63,32 +64,46 @@ def _oai() -> AsyncOpenAI:
|
||||
return _openai_client
|
||||
|
||||
|
||||
SYSTEM_PROMPT = """You are an analyst extracting tradeable signals from crypto KOL posts.
|
||||
SYSTEM_PROMPT = """You are an analyst extracting tradeable signals from crypto KOL (Key Opinion Leader) posts.
|
||||
|
||||
The author is a known crypto KOL. Your job: distill what they said and which tokens they are talking about RIGHT NOW (not historical references).
|
||||
The author is a known crypto KOL. Your job: distill what they said and which tokens they are talking about RIGHT NOW (not historical references). Pay special attention to DIVERGENCE between what they say publicly and what they imply about their actual position — this is the platform's highest-conviction signal.
|
||||
|
||||
Output **strict JSON only**, no markdown, no preface. Schema:
|
||||
|
||||
{
|
||||
"summary": "<one sentence, ≤60 chars/字. If signal exists, state the author's current thesis. If no signal, describe the post topic. Match the post's primary language (中文文章用中文, English 用英文).>",
|
||||
"summary": "<one sentence in ENGLISH, ≤80 chars. State the author's current market thesis if they have one, or describe the post topic if no clear signal. Always English regardless of the post's original language.>",
|
||||
"tickers": [
|
||||
{
|
||||
"ticker": "<UPPERCASE symbol, e.g. BTC, ETH, HYPE, SOL>",
|
||||
"action": "buy" | "sell" | "bullish" | "bearish" | "mention",
|
||||
"conviction": <float 0.0-1.0>,
|
||||
"quote": "<shortest verbatim sentence from the post supporting this call, ≤200 chars. Use the post's original language — do not translate.>"
|
||||
"ticker": "<UPPERCASE symbol, e.g. BTC, ETH, HYPE, SOL>",
|
||||
"action": "buy" | "sell" | "bullish" | "bearish" | "reduce" | "mention",
|
||||
"conviction": <float 0.0-1.0>,
|
||||
"timeframe": "immediate" | "days" | "weeks" | "months" | "unspecified",
|
||||
"stance_change": <true if this contradicts or reverses what the author said in earlier parts of the same post, or walks back a previously stated position | false>,
|
||||
"quote": "<shortest verbatim sentence from the post supporting this call, ≤200 chars. Keep original language — do not translate.>"
|
||||
}
|
||||
]
|
||||
],
|
||||
"talks_vs_trades_flag": <true if you detect a mismatch between the KOL's stated bullish/bearish narrative and signals of the opposite actual position (e.g. claiming bullish while describing reducing size, or bearish while mentioning recent buys) | false>
|
||||
}
|
||||
|
||||
Rules:
|
||||
- Always return summary in ENGLISH regardless of the post language.
|
||||
- If the post is macro commentary, news recap, or sponsored content with no specific token call, return tickers=[] and summary describing the topic.
|
||||
- IGNORE historical price references ("BTC bottomed at $60k earlier this year") — these are context, not current calls.
|
||||
- IGNORE advertising/sponsor sections — look for cues: "sponsor", "partner", "use code", "promo code", "this episode brought to you by", "ad", "广告", "赞助". Skip any ticker only mentioned inside such a section.
|
||||
- buy/sell only when the author states a position action ("I bought", "we are long", "我们减仓了", "added to my bag"). Otherwise use bullish/bearish for directional views, or mention for passing references.
|
||||
- action values:
|
||||
"buy"/"sell" → author explicitly states a position action ("I bought", "we are long", "我们减仓了", "added to my bag", "taking profits", "已建仓")
|
||||
"reduce" → author is partially exiting or taking profits on a long-held position (distinct from "sell" which implies closing)
|
||||
"bullish"/"bearish" → directional view without explicit position statement
|
||||
"mention" → ticker appears but no clear stance
|
||||
- Dedupe per ticker — at most one entry per symbol; pick the strongest action.
|
||||
- Do NOT invent tickers. If you see "$XYZ" but unsure it's a real token, skip it.
|
||||
- conviction: 0.8+ requires explicit + repeated + sized/timed view; 0.5-0.7 for clear directional view without commitment; <0.5 for passing references.
|
||||
- timeframe: "immediate" = author is acting now or within 24h; "days" = 1-7 days; "weeks" = 1-4 weeks; "months" = 1+ months; "unspecified" = no timeframe given.
|
||||
- talks_vs_trades_flag: set true when narrative reads one way but position signals read the other. Examples:
|
||||
- Author writes a bullish thesis but mentions "reducing", "taking profits", "trimming", "risk management"
|
||||
- Author is bearish on macro but "accumulating at these levels"
|
||||
- High-conviction public call ("this is THE entry") but with very low disclosed position size
|
||||
- Previously loudly bullish post, but this post avoids reaffirming the position
|
||||
- Do not include fiat (USD/CNY/JPY) or stablecoins (USDT/USDC/DAI/FRAX) unless the post's main thesis is about them.
|
||||
"""
|
||||
|
||||
@@ -194,6 +209,8 @@ async def extract_kol_signal(
|
||||
# Normalize
|
||||
tickers = data.get("tickers") or []
|
||||
cleaned = []
|
||||
valid_actions = {"buy", "sell", "reduce", "bullish", "bearish", "mention"}
|
||||
valid_timeframes = {"immediate", "days", "weeks", "months", "unspecified"}
|
||||
for t in tickers:
|
||||
if not isinstance(t, dict):
|
||||
continue
|
||||
@@ -201,23 +218,32 @@ async def extract_kol_signal(
|
||||
if not sym or len(sym) > 12:
|
||||
continue
|
||||
action = (t.get("action") or "mention").lower()
|
||||
if action not in {"buy", "sell", "bullish", "bearish", "mention"}:
|
||||
if action not in valid_actions:
|
||||
action = "mention"
|
||||
try:
|
||||
conv = float(t.get("conviction") or 0)
|
||||
except (TypeError, ValueError):
|
||||
conv = 0.0
|
||||
conv = max(0.0, min(1.0, conv))
|
||||
timeframe = (t.get("timeframe") or "unspecified").lower()
|
||||
if timeframe not in valid_timeframes:
|
||||
timeframe = "unspecified"
|
||||
stance_change = bool(t.get("stance_change", False))
|
||||
cleaned.append({
|
||||
"ticker": sym,
|
||||
"action": action,
|
||||
"conviction": round(conv, 2),
|
||||
"timeframe": timeframe,
|
||||
"stance_change": stance_change,
|
||||
"quote": (t.get("quote") or "")[:200],
|
||||
})
|
||||
|
||||
talks_vs_trades = bool(data.get("talks_vs_trades_flag", False))
|
||||
|
||||
return {
|
||||
"summary": (data.get("summary") or "").strip() or None,
|
||||
"tickers": cleaned,
|
||||
"talks_vs_trades_flag": talks_vs_trades,
|
||||
"model": model,
|
||||
"version": ANALYSIS_VERSION,
|
||||
}
|
||||
|
||||
@@ -202,6 +202,85 @@ KOL_FEEDS: list[dict] = [
|
||||
"feed_url": "https://tftc.io/feed",
|
||||
"source": "blog",
|
||||
},
|
||||
# ── Macro research & on-chain analysis (deep, independent thinkers) ───
|
||||
# Checkmate (James Check) — ex-Glassnode lead analyst. Combines on-chain
|
||||
# data with macro (bonds, energy, commodities). Very active (2-3x/week).
|
||||
# Concrete BTC cycle calls with high conviction.
|
||||
{
|
||||
"handle": "checkmate",
|
||||
"display_name": "Checkmate (James Check)",
|
||||
"feed_url": "https://newsletter.checkonchain.com/feed",
|
||||
"source": "blog",
|
||||
},
|
||||
# CoinMetrics "State of the Network" — weekly data-driven research.
|
||||
# Covers specific protocols/tokens with on-chain metrics. Stable Tuesday
|
||||
# cadence, institutional quality.
|
||||
{
|
||||
"handle": "coinmetrics",
|
||||
"display_name": "CoinMetrics (State of the Network)",
|
||||
"feed_url": "https://coinmetrics.substack.com/feed",
|
||||
},
|
||||
# Willy Woo — one of the most recognized BTC on-chain analysts. Models
|
||||
# like NVT, difficulty ribbon. Clear directional calls on BTC cycle phase.
|
||||
{
|
||||
"handle": "willywoo",
|
||||
"display_name": "Willy Woo (Bitcoin Vector)",
|
||||
"feed_url": "https://willywoo.substack.com/feed",
|
||||
},
|
||||
# Glassnode Insights — institutional on-chain research. Monthly deep
|
||||
# dives on holder behavior, market structure, accumulation/distribution.
|
||||
{
|
||||
"handle": "glassnode",
|
||||
"display_name": "Glassnode Research",
|
||||
"feed_url": "https://insights.glassnode.com/rss/",
|
||||
"source": "blog",
|
||||
},
|
||||
# Lyn Alden — independent macro analyst. Analyzes BTC through the lens of
|
||||
# global liquidity, fiscal dominance, sovereign debt. Infrequent (monthly)
|
||||
# but extremely high quality. Mostly BTC-only signals.
|
||||
{
|
||||
"handle": "lynalden",
|
||||
"display_name": "Lyn Alden",
|
||||
"feed_url": "https://www.lynalden.com/feed/",
|
||||
"source": "blog",
|
||||
},
|
||||
# Bitcoin Magazine — high-frequency news + institutional adoption analysis.
|
||||
# AI will extract tickers from policy/corporate BTC reserve articles.
|
||||
{
|
||||
"handle": "bitcoinmag",
|
||||
"display_name": "Bitcoin Magazine",
|
||||
"feed_url": "https://bitcoinmagazine.com/feed",
|
||||
"source": "blog",
|
||||
},
|
||||
# ── Derivatives positioning & institutional weeklies (2026-05-29) ────
|
||||
# Deribit Insights / Block Scholes — weekly options & derivatives data
|
||||
# (gamma exposure, IV skew, BTC/ETH positioning). Fills the derivatives
|
||||
# coverage gap — no other feed in this list tracks options flow.
|
||||
{
|
||||
"handle": "deribit",
|
||||
"display_name": "Deribit Insights (Block Scholes)",
|
||||
"feed_url": "https://insights.deribit.com/feed/",
|
||||
"source": "blog",
|
||||
},
|
||||
# Bitfinex Alpha — weekly BTC structure analysis (leverage, ETF flows,
|
||||
# on-chain profit-taking). Institutional voice, weekly cadence, concrete
|
||||
# directional framing. Complements Reflexivity's monthly cadence.
|
||||
{
|
||||
"handle": "bitfinex",
|
||||
"display_name": "Bitfinex Alpha",
|
||||
"feed_url": "https://blog.bitfinex.com/feed/",
|
||||
"source": "blog",
|
||||
},
|
||||
# Forward Guidance (Blockworks) — Felix Jauvin macro-interview podcast.
|
||||
# Fed policy, AI/liquidity, rates ↔ crypto. Same Blockworks quality bar
|
||||
# as Empire / Bell Curve / 0xResearch. Replaces dampedspring as the
|
||||
# primary macro signal source (her substack is paywalled).
|
||||
{
|
||||
"handle": "forwardguidance",
|
||||
"display_name": "Forward Guidance (Blockworks)",
|
||||
"feed_url": "https://feeds.megaphone.fm/forwardguidance",
|
||||
"source": "podcast",
|
||||
},
|
||||
]
|
||||
|
||||
# Back-compat alias — older imports referenced SUBSTACK_KOLS.
|
||||
|
||||
@@ -28,6 +28,7 @@ from datetime import datetime, timedelta, timezone
|
||||
from typing import Any, Optional
|
||||
|
||||
import httpx
|
||||
from app.services.bottom_indicators import ahr999 as compute_ahr999
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -119,17 +120,20 @@ def _parse_farside_latest_total(html: str) -> dict:
|
||||
|
||||
|
||||
# ── 1. AHR999 ───────────────────────────────────────────────────────────────
|
||||
# Formula: AHR999 = (price / 200d MA) × (price / age_fit_price)
|
||||
# age_fit_price = 10 ** (5.84 * log10(days_since_2009_01_03) - 17.01)
|
||||
# Below 0.45 historically marks accumulation zones; above 1.2 marks
|
||||
# "expensive" regime that invalidates a bottom thesis.
|
||||
|
||||
_AHR999_GENESIS = datetime(2009, 1, 3, tzinfo=timezone.utc)
|
||||
# IMPORTANT: this macro-panel AHR999 MUST stay formula-identical to the
|
||||
# live BTC bottom scanner, otherwise the displayed value can disagree with the
|
||||
# actual trigger logic. Reuse the canonical implementation from
|
||||
# app.services.bottom_indicators instead of re-implementing it here.
|
||||
|
||||
|
||||
@_none_on_fail("ahr999")
|
||||
async def fetch_ahr999() -> dict:
|
||||
"""Compute AHR999 from the last 200 daily BTC closes (Binance fapi)."""
|
||||
"""Compute the SAME AHR999 used by the BTC bottom scanner.
|
||||
|
||||
Raw input source stays the same (Binance daily BTCUSDT closes); only the
|
||||
formula source-of-truth is centralized so the UI cannot drift from the
|
||||
trading logic.
|
||||
"""
|
||||
end_ms = int(datetime.now(timezone.utc).timestamp() * 1000)
|
||||
start_ms = end_ms - 260 * 24 * 3600 * 1000 # extra buffer after dropping in-progress day
|
||||
async with httpx.AsyncClient(timeout=DEFAULT_TIMEOUT) as c:
|
||||
@@ -144,12 +148,12 @@ async def fetch_ahr999() -> dict:
|
||||
if len(closes) < 200:
|
||||
return {"value": None, "raw": {"error": "insufficient candles", "have": len(closes)}}
|
||||
price = closes[-1]
|
||||
ma200 = sum(closes[-200:]) / 200
|
||||
|
||||
days = (datetime.now(timezone.utc) - _AHR999_GENESIS).total_seconds() / 86400
|
||||
ma200 = sum(closes[-200:]) / 200 # kept in raw for operator intuition only
|
||||
days = (datetime.now(timezone.utc) - datetime(2009, 1, 3, tzinfo=timezone.utc)).total_seconds() / 86400
|
||||
age_fit = 10 ** (5.84 * math.log10(days) - 17.01)
|
||||
|
||||
ahr = (price / ma200) * (price / age_fit)
|
||||
ahr = compute_ahr999(closes)
|
||||
if ahr is None:
|
||||
return {"value": None, "raw": {"error": "ahr999 computation returned null"}}
|
||||
return {
|
||||
"value": round(ahr, 4),
|
||||
"raw": {"price": price, "ma200": round(ma200, 2),
|
||||
|
||||
@@ -1,6 +1,11 @@
|
||||
"""
|
||||
历史帖子价格回溯
|
||||
对 DB 中没有价格数据的帖子，从 Binance 拉历史 K 线计算涨跌幅
|
||||
Historical price backfill for already-relevant posts.
|
||||
|
||||
Important safety rule:
|
||||
This job only fills missing price fields. It must NEVER rewrite the post's
|
||||
semantic classification (`relevant`, `sentiment`, signal direction) from a
|
||||
keyword heuristic, otherwise historical truth rows become a mixed dataset of
|
||||
AI-scored rows + script-guessed rows.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
@@ -53,17 +58,21 @@ def _pct_change(klines: list, from_ms: int, delta_minutes: int) -> Optional[floa
|
||||
|
||||
|
||||
async def backfill_price_impact(db_session_factory, asset: str = "BTC") -> None:
|
||||
"""
|
||||
对 DB 中 relevant=True 但没有价格数据的帖子补充价格回溯。
|
||||
对 relevant=False 的帖子，做简单判断（标题含 crypto/bitcoin/btc 关键词则标记为相关）。
|
||||
"""Fill price-impact fields for posts that are already relevant.
|
||||
|
||||
The semantic label must come from the original analyzer, not from this
|
||||
backfill job. This task only enriches rows with price-at-post and
|
||||
subsequent returns.
|
||||
"""
|
||||
symbol = "BTCUSDT" if asset == "BTC" else "ETHUSDT"
|
||||
|
||||
async with db_session_factory() as db:
|
||||
# 拿所有没有价格数据的帖子
|
||||
# Only touch rows whose semantic classification already exists.
|
||||
result = await db.execute(
|
||||
select(Post)
|
||||
.where(Post.relevant == True)
|
||||
.where(Post.price_at_post == None)
|
||||
.where(Post.price_impact_asset == asset)
|
||||
.order_by(Post.published_at.asc())
|
||||
)
|
||||
posts = result.scalars().all()
|
||||
@@ -72,18 +81,6 @@ async def backfill_price_impact(db_session_factory, asset: str = "BTC") -> None:
|
||||
if not posts:
|
||||
return
|
||||
|
||||
# 关键词判断是否与加密相关（快速，不用 Claude API）
|
||||
crypto_keywords = [
|
||||
"bitcoin", "btc", "crypto", "cryptocurrency", "blockchain",
|
||||
"ethereum", "eth", "digital currency", "defi", "coinbase",
|
||||
"sec", "regulation", "tariff", "dollar", "inflation", "fed",
|
||||
"economy", "market", "trade", "sanctions", "iran", "china",
|
||||
]
|
||||
|
||||
def is_relevant(text: str) -> bool:
|
||||
t = text.lower()
|
||||
return any(kw in t for kw in crypto_keywords)
|
||||
|
||||
saved = 0
|
||||
errors = 0
|
||||
|
||||
@@ -95,18 +92,6 @@ async def backfill_price_impact(db_session_factory, asset: str = "BTC") -> None:
|
||||
|
||||
start_ms = int(published_at.timestamp() * 1000)
|
||||
|
||||
# 判断相关性（用关键词，不消耗 Claude API）
|
||||
relevant = is_relevant(post.text)
|
||||
sentiment = "neutral"
|
||||
if relevant:
|
||||
t = post.text.lower()
|
||||
bullish_kw = ["great", "win", "winning", "strong", "best", "love", "beautiful", "tremendous", "amazing", "pro-crypto", "bitcoin reserve"]
|
||||
bearish_kw = ["bad", "terrible", "war", "crisis", "sanction", "ban", "regulate", "crack", "fraud", "scam"]
|
||||
if any(k in t for k in bullish_kw):
|
||||
sentiment = "bullish"
|
||||
elif any(k in t for k in bearish_kw):
|
||||
sentiment = "bearish"
|
||||
|
||||
# 拉 Binance 历史价格
|
||||
klines = await _fetch_klines(symbol, start_ms, limit=FETCH_WINDOW_MINUTES)
|
||||
|
||||
@@ -120,13 +105,10 @@ async def backfill_price_impact(db_session_factory, asset: str = "BTC") -> None:
|
||||
result = await db.execute(select(Post).where(Post.id == post.id))
|
||||
p = result.scalar_one_or_none()
|
||||
if p:
|
||||
p.relevant = relevant
|
||||
p.sentiment = sentiment
|
||||
p.price_impact_asset = asset if relevant else None
|
||||
p.price_at_post = price_at_post
|
||||
p.price_impact_m5 = m5 if relevant else None
|
||||
p.price_impact_m15 = m15 if relevant else None
|
||||
p.price_impact_m1h = m1h if relevant else None
|
||||
p.price_impact_m5 = m5
|
||||
p.price_impact_m15 = m15
|
||||
p.price_impact_m1h = m1h
|
||||
await db.commit()
|
||||
saved += 1
|
||||
|
||||
|
||||
+90
-14
@@ -31,7 +31,7 @@ from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Optional
|
||||
|
||||
from sqlalchemy import select, update
|
||||
@@ -47,6 +47,16 @@ logger = logging.getLogger(__name__)
|
||||
|
||||
RECONCILE_INTERVAL_SECONDS = 60
|
||||
|
||||
# Max concurrent HL API calls per reconcile cycle. Each call can take up to
|
||||
# ~30s on timeout; with 10 in flight the worst-case tail latency for N wallets
|
||||
# is ceil(N/10) × 30s instead of N × 30s.
|
||||
_RECONCILE_CONCURRENCY = 10
|
||||
|
||||
# How far back to look for the BUG-03 ghost-position check (DB closed but HL
|
||||
# still open). 2 hours covers most close_and_finalize failure windows without
|
||||
# scanning the entire trades history.
|
||||
_GHOST_LOOKBACK_HOURS = 2
|
||||
|
||||
|
||||
# ─── Single-wallet reconcile ────────────────────────────────────────────────
|
||||
|
||||
@@ -59,7 +69,7 @@ async def _reconcile_wallet(sub: Subscription) -> dict:
|
||||
"""
|
||||
summary = {
|
||||
"wallet": sub.wallet_address, "orphan_hl": [], "orphan_db": [],
|
||||
"marked_closed": 0, "error": None,
|
||||
"marked_closed": 0, "ghost_positions": [], "error": None,
|
||||
}
|
||||
|
||||
# Paper-mode subs aren't on HL at all — nothing to reconcile.
|
||||
@@ -142,6 +152,44 @@ async def _reconcile_wallet(sub: Subscription) -> dict:
|
||||
logger.warning("Reconcile: %s has unmanaged HL position %s (szi=%s, entry=%s)",
|
||||
sub.wallet_address, asset, hl_pos["szi"], hl_pos["entry_px"])
|
||||
|
||||
# Case 3 (BUG-03 mitigation): DB says closed recently but HL still shows
|
||||
# the position open — the "ghost" state caused by close_and_finalize
|
||||
# writing closed_at to DB while the HL market order fails. We can't
|
||||
# auto-re-close (risk of double-execution) so we alert loudly and let
|
||||
# the human decide. Look back _GHOST_LOOKBACK_HOURS only; older discrepancies
|
||||
# are likely stale HL data or positions the user re-opened manually.
|
||||
ghost_cutoff = datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(
|
||||
hours=_GHOST_LOOKBACK_HOURS
|
||||
)
|
||||
async with AsyncSessionLocal() as db:
|
||||
ghost_rows = (await db.execute(
|
||||
select(BotTrade).where(
|
||||
BotTrade.wallet_address == sub.wallet_address,
|
||||
BotTrade.closed_at.is_not(None),
|
||||
BotTrade.closed_at >= ghost_cutoff,
|
||||
)
|
||||
)).scalars().all()
|
||||
|
||||
for trade in ghost_rows:
|
||||
# Paper trades and adopted/released trades have no HL position to check.
|
||||
if trade.hl_order_id == "paper" or trade.released_at is not None:
|
||||
continue
|
||||
if trade.asset in hl_assets_open:
|
||||
ghost_entry = {
|
||||
"asset": trade.asset,
|
||||
"trade_id": trade.id,
|
||||
"closed_at": trade.closed_at.isoformat() if trade.closed_at else None,
|
||||
"hl_szi": hl_assets_open[trade.asset].get("szi"),
|
||||
}
|
||||
summary["ghost_positions"].append(ghost_entry)
|
||||
logger.error(
|
||||
"GHOST POSITION [BUG-03]: trade %d (%s %s) has closed_at=%s in DB "
|
||||
"but HL still shows an open position (szi=%s). "
|
||||
"Manual close required on HL UI.",
|
||||
trade.id, trade.side, trade.asset, trade.closed_at,
|
||||
hl_assets_open[trade.asset].get("szi"),
|
||||
)
|
||||
|
||||
return summary
|
||||
|
||||
|
||||
@@ -149,7 +197,12 @@ async def _reconcile_wallet(sub: Subscription) -> dict:
|
||||
|
||||
|
||||
async def reconcile_all_once() -> None:
|
||||
"""One scan over every subscription. Wired into the APScheduler at startup."""
|
||||
"""One scan over every subscription. Wired into the APScheduler at startup.
|
||||
|
||||
Runs all per-wallet checks concurrently (up to _RECONCILE_CONCURRENCY in
|
||||
flight at once) so tail latency scales as ceil(N / concurrency) × per-call
|
||||
timeout rather than N × per-call timeout.
|
||||
"""
|
||||
async with AsyncSessionLocal() as db:
|
||||
rows = await db.execute(
|
||||
select(Subscription).where(
|
||||
@@ -165,30 +218,53 @@ async def reconcile_all_once() -> None:
|
||||
# Snapshot the fields we read so we don't hold the session across HL calls.
|
||||
snapshots = list(subs)
|
||||
|
||||
sem = asyncio.Semaphore(_RECONCILE_CONCURRENCY)
|
||||
|
||||
async def _bounded(sub: Subscription) -> dict:
|
||||
async with sem:
|
||||
return await _reconcile_wallet(sub)
|
||||
|
||||
results = await asyncio.gather(
|
||||
*[_bounded(sub) for sub in snapshots],
|
||||
return_exceptions=True,
|
||||
)
|
||||
|
||||
n_changed = 0
|
||||
n_orphans = 0
|
||||
n_ghosts = 0
|
||||
n_errors = 0
|
||||
for sub in snapshots:
|
||||
summary = await _reconcile_wallet(sub)
|
||||
for sub, result in zip(snapshots, results):
|
||||
if isinstance(result, BaseException):
|
||||
n_errors += 1
|
||||
logger.error("Reconcile: unexpected exception for %s: %s",
|
||||
sub.wallet_address, result)
|
||||
continue
|
||||
|
||||
summary = result
|
||||
n_changed += summary["marked_closed"]
|
||||
n_orphans += len(summary["orphan_hl"])
|
||||
n_ghosts += len(summary.get("ghost_positions", []))
|
||||
if summary["error"]:
|
||||
n_errors += 1
|
||||
|
||||
# Broadcast significant events so the UI shows a warning banner.
|
||||
if summary["marked_closed"] or summary["orphan_hl"]:
|
||||
if summary["marked_closed"] or summary["orphan_hl"] or summary.get("ghost_positions"):
|
||||
try:
|
||||
from app.ws.manager import manager
|
||||
await manager.broadcast({
|
||||
"type": "reconcile_drift",
|
||||
"wallet": summary["wallet"],
|
||||
"orphan_hl": summary["orphan_hl"],
|
||||
"marked_closed": summary["marked_closed"],
|
||||
"at": datetime.now(timezone.utc).isoformat() + "Z",
|
||||
"type": "reconcile_drift",
|
||||
"wallet": summary["wallet"],
|
||||
"orphan_hl": summary["orphan_hl"],
|
||||
"marked_closed": summary["marked_closed"],
|
||||
"ghost_positions": summary.get("ghost_positions", []),
|
||||
"at": datetime.now(timezone.utc).isoformat() + "Z",
|
||||
})
|
||||
except Exception as exc:
|
||||
logger.warning("reconcile WS broadcast failed: %s", exc)
|
||||
|
||||
if n_changed or n_orphans or n_errors:
|
||||
logger.info("Reconcile cycle: %d subs scanned, %d trades marked closed, %d HL orphans, %d errors",
|
||||
len(snapshots), n_changed, n_orphans, n_errors)
|
||||
if n_changed or n_orphans or n_ghosts or n_errors:
|
||||
logger.info(
|
||||
"Reconcile cycle: %d subs scanned, %d trades marked closed, "
|
||||
"%d HL orphans, %d ghost positions, %d errors",
|
||||
len(snapshots), n_changed, n_orphans, n_ghosts, n_errors,
|
||||
)
|
||||
|
||||
@@ -18,13 +18,14 @@ logger = logging.getLogger(__name__)
|
||||
|
||||
async def rehydrate_open_trades() -> None:
|
||||
# Imported locally to avoid circular imports at module load
|
||||
from app.services.bot_engine import close_and_finalize
|
||||
from app.services.bot_engine import close_and_finalize, _time_stop_check, _background_tasks
|
||||
from app.services.crypto import decrypt_api_key
|
||||
from app.services.signal_categories import (
|
||||
get_stop_ladder as _get_stop_ladder,
|
||||
sys2_derisk_ladder as _sys2_derisk_ladder,
|
||||
sys2_addon_ladder as _sys2_addon_ladder,
|
||||
sys2_peak_trail as _sys2_peak_trail,
|
||||
get_exit_profile as _get_exit_profile,
|
||||
)
|
||||
from app.services.tp_sl_monitor import register_trade
|
||||
|
||||
@@ -183,5 +184,52 @@ async def rehydrate_open_trades() -> None:
|
||||
_background_tasks.add(task)
|
||||
task.add_done_callback(_background_tasks.discard)
|
||||
|
||||
# ── System-2 time-stop rehydration ──────────────────────────────
|
||||
# _time_stop_check is a background task created at open time that
|
||||
# closes the trade if it's still flat (|unrealised| < 2%) after
|
||||
# `time_stop_hours`. It is NOT stored in DB — it must be rebuilt
|
||||
# here on every restart or open sys2 trades silently lose this guard.
|
||||
# btc_bottom_reversal_long has time_stop_hours=None (no time stop).
|
||||
if _stop_ladder and t.sys2_mode:
|
||||
_exit_profile = _get_exit_profile(_cat_for_ladders)
|
||||
ts_hours = _exit_profile.time_stop_hours
|
||||
if ts_hours:
|
||||
ts_elapsed_h = elapsed / 3600
|
||||
ts_remaining_s = max(0.0, ts_hours - ts_elapsed_h) * 3600
|
||||
if ts_remaining_s > 0:
|
||||
ts_task = asyncio.create_task(_time_stop_check(
|
||||
trade_id=t.id,
|
||||
api_key=api_key,
|
||||
leverage=trade_leverage,
|
||||
asset=t.asset,
|
||||
wallet=t.wallet_address,
|
||||
delay_seconds=int(ts_remaining_s),
|
||||
))
|
||||
_background_tasks.add(ts_task)
|
||||
ts_task.add_done_callback(_background_tasks.discard)
|
||||
logger.info(
|
||||
"Rehydrated time-stop for trade %d: %.1fh remaining",
|
||||
t.id, ts_remaining_s / 3600,
|
||||
)
|
||||
else:
|
||||
# Time-stop window already elapsed while backend was down.
|
||||
# Fire it now — close_and_finalize is idempotent (WHERE
|
||||
# closed_at IS NULL) so a trade that already closed via
|
||||
# another path is a safe no-op.
|
||||
logger.info(
|
||||
"Trade %d time-stop elapsed during downtime — checking now",
|
||||
t.id,
|
||||
)
|
||||
ts_task = asyncio.create_task(_time_stop_check(
|
||||
trade_id=t.id,
|
||||
api_key=api_key,
|
||||
leverage=trade_leverage,
|
||||
asset=t.asset,
|
||||
wallet=t.wallet_address,
|
||||
delay_seconds=0,
|
||||
))
|
||||
_background_tasks.add(ts_task)
|
||||
ts_task.add_done_callback(_background_tasks.discard)
|
||||
|
||||
await db.commit()
|
||||
logger.info("Rehydrated %d open trades.", len(open_trades))
|
||||
|
||||
@@ -105,7 +105,7 @@ async def _emit_signal(debug: dict) -> bool:
|
||||
}
|
||||
async with httpx.AsyncClient(timeout=10) as client:
|
||||
resp = await client.post(
|
||||
"http://localhost:8000/api/signals/ingest",
|
||||
f"{settings.ingest_base_url}/api/signals/ingest",
|
||||
json=payload,
|
||||
headers={"X-Ingest-Key": settings.ingest_api_key},
|
||||
)
|
||||
|
||||
@@ -285,7 +285,7 @@ async def _emit_signal(debug: dict) -> bool:
|
||||
}
|
||||
async with httpx.AsyncClient(timeout=10) as client:
|
||||
resp = await client.post(
|
||||
"http://localhost:8000/api/signals/ingest",
|
||||
f"{settings.ingest_base_url}/api/signals/ingest",
|
||||
json=payload,
|
||||
headers={"X-Ingest-Key": settings.ingest_api_key},
|
||||
)
|
||||
|
||||
@@ -52,15 +52,22 @@ def build_message(action: str, wallet: str, timestamp_ms: int, body_hash: str) -
|
||||
# In-memory; fine for a single-process deploy. For multi-worker, move to Redis.
|
||||
_seen: dict[str, float] = {} # sha256(signature) → expiry epoch seconds
|
||||
|
||||
# Purge when the dict exceeds this size. At 5 minutes TTL and ~10 req/s peak
|
||||
# that's ~3000 entries at saturation; 1000 gives comfortable headroom while
|
||||
# keeping purge cost tiny (<1 ms for 1000-entry linear scan).
|
||||
_SEEN_PURGE_THRESHOLD = 1000
|
||||
|
||||
|
||||
def _cache_put(sig: str, ttl: float) -> bool:
|
||||
"""Return True if sig was new, False if replay."""
|
||||
now = time.time()
|
||||
# lazy purge
|
||||
if len(_seen) > 5000:
|
||||
for k, v in list(_seen.items()):
|
||||
if v < now:
|
||||
_seen.pop(k, None)
|
||||
# Lazy purge: only scan when we've crossed the threshold, not every call.
|
||||
# Scanning 1000 vs 5000 entries keeps individual request latency negligible
|
||||
# even under burst traffic.
|
||||
if len(_seen) >= _SEEN_PURGE_THRESHOLD:
|
||||
expired = [k for k, v in _seen.items() if v < now]
|
||||
for k in expired:
|
||||
del _seen[k]
|
||||
key = hashlib.sha256(sig.encode("utf-8")).hexdigest()
|
||||
if key in _seen and _seen[key] > now:
|
||||
return False
|
||||
@@ -105,3 +112,35 @@ def verify_signed_request(
|
||||
if not allow_replay:
|
||||
if not _cache_put(signature, ttl=MAX_SKEW_SECONDS):
|
||||
raise HTTPException(401, "Signature already used (replay blocked)")
|
||||
|
||||
|
||||
def verify_signed_request_any(
|
||||
*,
|
||||
actions: list[str],
|
||||
wallet: str,
|
||||
timestamp_ms: int,
|
||||
signature: str,
|
||||
body: Optional[Any],
|
||||
allow_replay: bool = False,
|
||||
) -> None:
|
||||
"""Accept any one of several signed read-actions.
|
||||
|
||||
Useful for wallet-owned read endpoints where the frontend may already hold
|
||||
a fresh cached envelope for a sibling read action such as `view_user`.
|
||||
"""
|
||||
last_exc: Optional[HTTPException] = None
|
||||
for action in actions:
|
||||
try:
|
||||
verify_signed_request(
|
||||
action=action,
|
||||
wallet=wallet,
|
||||
timestamp_ms=timestamp_ms,
|
||||
signature=signature,
|
||||
body=body,
|
||||
allow_replay=allow_replay,
|
||||
)
|
||||
return
|
||||
except HTTPException as exc:
|
||||
last_exc = exc
|
||||
if last_exc is not None:
|
||||
raise last_exc
|
||||
|
||||
+137
-9
@@ -146,16 +146,100 @@ def format_post(post: Post) -> str:
|
||||
return msg[:MAX_LEN]
|
||||
|
||||
|
||||
def format_trump_mention(post: Post) -> str:
|
||||
"""Public-channel-only alert for Trump posts that are crypto-relevant
|
||||
but don't carry a clear directional signal (signal=hold, relevant=True).
|
||||
|
||||
These posts can still move markets (e.g. "I support crypto development")
|
||||
so we surface them as low-urgency monitoring alerts. Never sent per-user
|
||||
to avoid subscriber spam; only broadcast to the public channel.
|
||||
"""
|
||||
body = (post.text or "").strip()
|
||||
if len(body) > 300:
|
||||
body = body[:300].rstrip() + "…"
|
||||
|
||||
fe = (settings.frontend_url or "").rstrip("/")
|
||||
link = f'\n\n<a href="{fe}/en/trump">→ view on TrumpAlpha</a>' if fe else ""
|
||||
|
||||
msg = (
|
||||
f"📢 <b>Trump mentioned crypto</b>\n"
|
||||
f"<i>Trump · Truth Social — no clear trade signal</i>\n\n"
|
||||
f"{body}"
|
||||
f"{link}"
|
||||
)
|
||||
return msg[:MAX_LEN]
|
||||
|
||||
|
||||
def format_public_post(post: Post) -> str:
|
||||
"""Render a sanitised version of a Post for the public broadcast channel.
|
||||
|
||||
Intentionally omits:
|
||||
- expected_move_pct / invalidation_price (execution-sensitive data)
|
||||
- /adopt CTA (requires a private bot session)
|
||||
- Any wallet address or subscriber-only details
|
||||
|
||||
Keeps: signal direction, asset, confidence tier, source label, post
|
||||
excerpt (≤ 280 chars), and a link back to the relevant dashboard section.
|
||||
"""
|
||||
emoji = _signal_emoji(post)
|
||||
src = _source_label(post.source)
|
||||
sig = (post.signal or "noise").upper()
|
||||
asset = post.target_asset or "?"
|
||||
conf = post.ai_confidence or 0
|
||||
|
||||
# Confidence tier label instead of raw number (less intimidating publicly)
|
||||
if conf >= 80:
|
||||
conf_label = "HIGH"
|
||||
elif conf >= 60:
|
||||
conf_label = "MED"
|
||||
elif conf > 0:
|
||||
conf_label = "LOW"
|
||||
else:
|
||||
conf_label = "—"
|
||||
|
||||
head = f"{emoji} <b>{asset} · {sig}</b> · conf <b>{conf_label}</b>"
|
||||
sub = f"<i>{src}</i>"
|
||||
|
||||
body = (post.text or "").strip()
|
||||
if len(body) > 280:
|
||||
body = body[:280].rstrip() + "…"
|
||||
|
||||
# KOL divergence: strip wallet address from the text if present
|
||||
# (the raw text shouldn't contain one, but belt-and-suspenders)
|
||||
if post.source == "kol_divergence" and post.ai_reasoning:
|
||||
reason = post.ai_reasoning.strip()
|
||||
if len(reason) > 200:
|
||||
reason = reason[:200].rstrip() + "…"
|
||||
body = reason
|
||||
|
||||
fe = (settings.frontend_url or "").rstrip("/")
|
||||
link = ""
|
||||
if fe:
|
||||
path = {
|
||||
"truth": "/en/trump",
|
||||
"btc_bottom_reversal": "/en/macro",
|
||||
"funding_reversal": "/en/macro",
|
||||
"kol_divergence": "/en/kol",
|
||||
}.get(post.source, "/en")
|
||||
link = f'\n\n<a href="{fe}{path}">→ view on TrumpAlpha</a>'
|
||||
|
||||
msg = f"{head}\n{sub}\n\n{body}{link}"
|
||||
return msg[:MAX_LEN]
|
||||
|
||||
|
||||
# ── Low-level HTTP ────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
async def send_message(chat_id: int, text: str, *,
|
||||
async def send_message(chat_id: int | str, text: str, *,
|
||||
parse_mode: str = "HTML",
|
||||
disable_preview: bool = True,
|
||||
reply_markup: Optional[dict] = None) -> bool:
|
||||
"""Single HTTP POST to Telegram Bot API. Returns True on 200, False on
|
||||
any failure (caller decides whether to bump the failure counter).
|
||||
|
||||
`chat_id` may be an integer user/group ID or a string channel username
|
||||
such as ``"@trumpalpha"`` (with or without the leading @).
|
||||
|
||||
`reply_markup` is the standard Bot-API inline keyboard / reply keyboard
|
||||
payload, e.g.
|
||||
{"inline_keyboard": [
|
||||
@@ -179,12 +263,12 @@ async def send_message(chat_id: int, text: str, *,
|
||||
async with httpx.AsyncClient(timeout=10) as client:
|
||||
r = await client.post(url, json=payload)
|
||||
if r.status_code != 200:
|
||||
logger.warning("Telegram sendMessage failed chat=%d status=%d body=%s",
|
||||
logger.warning("Telegram sendMessage failed chat=%s status=%d body=%s",
|
||||
chat_id, r.status_code, r.text[:200])
|
||||
return False
|
||||
return True
|
||||
except Exception as exc:
|
||||
logger.warning("Telegram sendMessage exception chat=%d: %s", chat_id, exc)
|
||||
logger.warning("Telegram sendMessage exception chat=%s: %s", chat_id, exc)
|
||||
return False
|
||||
|
||||
|
||||
@@ -246,7 +330,20 @@ async def answer_callback(callback_query_id: str, text: str = "",
|
||||
|
||||
async def _dispatch(post_id: int) -> None:
|
||||
"""Fan-out a single Post to every eligible subscriber. Always runs in
|
||||
its own DB session so signal ingestion's session is unaffected."""
|
||||
its own DB session so signal ingestion's session is unaffected.
|
||||
|
||||
Two distinct paths:
|
||||
|
||||
PATH A — actionable signal (signal=buy/short):
|
||||
• Per-subscriber fan-out filtered by alert preference + min_confidence
|
||||
• Public channel broadcast (format_public_post)
|
||||
|
||||
PATH B — crypto mention (source=truth, relevant=True, signal=hold):
|
||||
• Public channel ONLY (format_trump_mention) — no per-subscriber push
|
||||
to avoid noise for individual users who subscribed for trade signals.
|
||||
• These posts are crypto-relevant but lack a clear directional call.
|
||||
They can still move markets so we surface them as monitoring alerts.
|
||||
"""
|
||||
if not settings.telegram_bot_token:
|
||||
return
|
||||
|
||||
@@ -256,10 +353,32 @@ async def _dispatch(post_id: int) -> None:
|
||||
logger.warning("Telegram dispatch: post id=%d not found", post_id)
|
||||
return
|
||||
|
||||
# Only fan out for actionable signals (not NOISE / null)
|
||||
if not post.signal or post.signal not in ("buy", "short"):
|
||||
is_actionable = post.signal in ("buy", "short")
|
||||
is_trump_mention = (
|
||||
post.source == "truth"
|
||||
and bool(post.relevant)
|
||||
and post.signal == "hold"
|
||||
)
|
||||
|
||||
if not is_actionable and not is_trump_mention:
|
||||
# Non-relevant noise — nothing to send.
|
||||
return
|
||||
|
||||
# ── PATH B: crypto mention (public channel only) ──────────────────
|
||||
if is_trump_mention and not is_actionable:
|
||||
channel_id = settings.telegram_public_channel_id
|
||||
if channel_id:
|
||||
mention_text = format_trump_mention(post)
|
||||
ok = await send_message(channel_id, mention_text)
|
||||
if ok:
|
||||
logger.info("Telegram mention alert: post=%d sent to %s",
|
||||
post_id, channel_id)
|
||||
else:
|
||||
logger.warning("Telegram mention alert: post=%d send FAILED to %s",
|
||||
post_id, channel_id)
|
||||
return
|
||||
|
||||
# ── PATH A: actionable signal (buy/short) ─────────────────────────
|
||||
pref_col = _pref_column_for_source(post.source)
|
||||
if pref_col is None:
|
||||
logger.debug("Telegram: unknown source %r — not fanning out", post.source)
|
||||
@@ -282,9 +401,6 @@ async def _dispatch(post_id: int) -> None:
|
||||
q = select(TelegramBinding).where(*base_filters)
|
||||
bindings = (await db.execute(q)).scalars().all()
|
||||
|
||||
if not bindings:
|
||||
return
|
||||
|
||||
text = format_post(post)
|
||||
now = datetime.now(timezone.utc)
|
||||
hour = now.hour
|
||||
@@ -318,6 +434,18 @@ async def _dispatch(post_id: int) -> None:
|
||||
logger.info("Telegram fan-out: post=%d source=%s sent=%d/%d",
|
||||
post_id, post.source, sent, len(bindings))
|
||||
|
||||
# ── Public channel broadcast ──────────────────────────────────────
|
||||
channel_id = settings.telegram_public_channel_id
|
||||
if channel_id:
|
||||
public_text = format_public_post(post)
|
||||
ok = await send_message(channel_id, public_text)
|
||||
if ok:
|
||||
logger.info("Telegram public channel: post=%d sent to %s",
|
||||
post_id, channel_id)
|
||||
else:
|
||||
logger.warning("Telegram public channel: post=%d send FAILED to %s",
|
||||
post_id, channel_id)
|
||||
|
||||
|
||||
def notify_signal(post: Post) -> None:
|
||||
"""Fire-and-forget. Schedules `_dispatch(post.id)` on the running loop
|
||||
|
||||
@@ -377,8 +377,12 @@ async def _cmd_stop(chat_id: int) -> None:
|
||||
)
|
||||
await db.commit()
|
||||
if r.rowcount:
|
||||
await send_message(chat_id,
|
||||
"🔕 Alerts paused. Send /start any time to re-enable.")
|
||||
await send_message(
|
||||
chat_id,
|
||||
"🔕 Real-time alerts paused. Send /start any time to re-enable.\n\n"
|
||||
"Your daily brief is unaffected — send <code>/digest off</code> "
|
||||
"to stop that separately.",
|
||||
)
|
||||
else:
|
||||
await send_message(chat_id,
|
||||
"You don't have an active binding here. Send /start to set one up.")
|
||||
|
||||
@@ -413,10 +413,14 @@ async def send_daily_digest(current_hour: Optional[int] = None) -> int:
|
||||
|
||||
sent = 0
|
||||
async with async_session() as db:
|
||||
# NOTE: alerts_enabled is intentionally NOT gated here. /stop pauses
|
||||
# real-time signal alerts but the daily digest is a separate feature
|
||||
# (/digest on|off). Filtering on alerts_enabled would silently kill
|
||||
# the digest for any user who typed /stop, contradicting the bot's
|
||||
# own message ("Alerts paused — send /start to re-enable").
|
||||
bindings = (await db.execute(
|
||||
select(TelegramBinding).where(
|
||||
TelegramBinding.digest_enabled.is_(True),
|
||||
TelegramBinding.alerts_enabled.is_(True),
|
||||
TelegramBinding.digest_hour_utc == hour,
|
||||
or_(
|
||||
TelegramBinding.last_digest_sent_at.is_(None),
|
||||
|
||||
@@ -187,7 +187,18 @@ def register_trade(
|
||||
|
||||
|
||||
def unregister(trade_id: int) -> None:
|
||||
_watched.pop(trade_id, None)
|
||||
wt = _watched.pop(trade_id, None)
|
||||
# Flush the latest peak to DB on unregister so a restart between the last
|
||||
# throttled write and this point doesn't lose progress.
|
||||
if wt is not None and wt.peak_gain_pct > wt.peak_persisted:
|
||||
task = asyncio.create_task(_persist_peak(trade_id, wt.peak_gain_pct))
|
||||
_background_tasks.add(task)
|
||||
task.add_done_callback(_background_tasks.discard)
|
||||
# Clean up the per-trade close lock from bot_engine to prevent memory leak.
|
||||
# Trades closed via reconciler bypass close_and_finalize, so the lock would
|
||||
# otherwise remain in the dict forever.
|
||||
from app.services.bot_engine import _close_locks
|
||||
_close_locks.pop(trade_id, None)
|
||||
|
||||
|
||||
def on_price_tick(asset: str, price: float) -> None:
|
||||
|
||||
@@ -0,0 +1,509 @@
|
||||
"""
|
||||
X (Twitter) post semantic analysis — KOL real-time signals.
|
||||
|
||||
X posts are fundamentally different from Substack long-form articles:
|
||||
- 280 chars: conclusion IS the content, no room for hedging
|
||||
- Latency matters: "just bought SOL" is a minutes-level signal
|
||||
- Noise ratio is extreme: 95%+ of posts are irrelevant
|
||||
- Retweet/quote patterns must be detected and handled differently
|
||||
- Position statements ("added", "trimmed", "exit") are high-value
|
||||
- Thread context matters: a post may continue a thought from above
|
||||
|
||||
Design philosophy:
|
||||
- Strict NOISE default (opposite of buy/sell). Most X posts should
|
||||
be filtered out. The cost of false-positive on X is high because
|
||||
KOLs tweet constantly.
|
||||
- Three tiers of output:
|
||||
TRADE_SIGNAL → actionable now (position statement + specific asset)
|
||||
DIRECTIONAL → clear view, no explicit position action
|
||||
NOISE → everything else (no tickers, no stance, filler)
|
||||
- talks_vs_trades_flag: X posts often reveal real positions
|
||||
("taking profits" while publicly bullish = divergence)
|
||||
|
||||
Output feeds into kol_divergence.py for cross-referencing with on-chain.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
from typing import Optional
|
||||
|
||||
from openai import AsyncOpenAI
|
||||
from app.config import settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
ANALYSIS_VERSION = "x-v1"
|
||||
ANTHROPIC_MODEL = "claude-haiku-4-5-20251001"
|
||||
|
||||
_anthropic_client = None
|
||||
_openai_client: Optional[AsyncOpenAI] = None
|
||||
|
||||
|
||||
def _use_anthropic() -> bool:
|
||||
return bool(settings.anthropic_api_key)
|
||||
|
||||
|
||||
def _anth():
|
||||
global _anthropic_client
|
||||
if _anthropic_client is None:
|
||||
import anthropic as _a
|
||||
_anthropic_client = _a.AsyncAnthropic(api_key=settings.anthropic_api_key)
|
||||
return _anthropic_client
|
||||
|
||||
|
||||
def _oai() -> AsyncOpenAI:
|
||||
global _openai_client
|
||||
if _openai_client is None:
|
||||
_openai_client = AsyncOpenAI(
|
||||
api_key=settings.ai_api_key,
|
||||
base_url=settings.ai_base_url,
|
||||
)
|
||||
return _openai_client
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────
|
||||
# SYSTEM PROMPT
|
||||
# ─────────────────────────────────────────────────────────────────────
|
||||
SYSTEM_PROMPT = """\
|
||||
You are a signal extraction system for a crypto trading platform. You read
|
||||
X (Twitter) posts from known crypto KOLs and decide whether they contain
|
||||
a tradeable signal.
|
||||
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
NOISE IS THE DEFAULT — internalize this
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
KOLs post 20-50 times per day. Most of it is:
|
||||
- gm / wagmi / vibes / lifestyle
|
||||
- macro commentary with no specific call
|
||||
- reactions to news without a position
|
||||
- vague encouragement ("BTC is king")
|
||||
- promotional content / sponsored
|
||||
- replies to followers (context-dependent, usually not standalone)
|
||||
- jokes, memes, personal life
|
||||
|
||||
ALL of the above → tier: "noise", tickers: []
|
||||
|
||||
Only extract signals when the post contains CLEAR, EXPLICIT information
|
||||
about what the KOL is doing or believes RIGHT NOW. Ambiguity → noise.
|
||||
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
THREE SIGNAL TIERS
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
TIER 1 — TRADE_SIGNAL (rarest, highest value)
|
||||
The KOL states an ACTUAL POSITION ACTION in this post:
|
||||
"just bought", "added more", "long from here", "trimming my",
|
||||
"sold my", "exit", "took profits", "cutting", "building position",
|
||||
"DCA'd", "我建仓了", "已经上车", "减仓了", "清仓了"
|
||||
Must reference a SPECIFIC asset (not just "the market").
|
||||
Conviction ≥ 0.7 required for TRADE_SIGNAL.
|
||||
|
||||
TIER 2 — DIRECTIONAL (moderate value)
|
||||
The KOL expresses a SPECIFIC, CURRENT directional view on a named asset:
|
||||
"SOL looking strong here", "ETH is going to $5k", "BTC at support",
|
||||
"I think [TICKER] breaks down", "watching [TICKER] for entry"
|
||||
No explicit position action, but a clear current opinion.
|
||||
Conviction 0.4–0.7 typical.
|
||||
|
||||
TIER 3 — NOISE (default)
|
||||
Everything else. When in doubt, NOISE.
|
||||
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
POST TYPE DETECTION — apply before scoring
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
Detect what kind of post this is before scoring content:
|
||||
|
||||
"original" — KOL's own thought, highest signal value
|
||||
"reply" — reply to another user (context missing → usually noise
|
||||
unless the post is fully self-contained)
|
||||
"retweet" — RT of someone else's post without commentary → NOISE
|
||||
"quote" — RT with commentary → score the commentary only,
|
||||
ignore the original post content
|
||||
"thread_cont"— continues a thread (may lack context → lower conviction)
|
||||
|
||||
Retweets without commentary are ALWAYS noise regardless of content.
|
||||
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
TALKS VS TRADES FLAG — the platform's core value
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
Set talks_vs_trades_flag=true when you detect a MISMATCH between:
|
||||
- What the KOL says publicly (bullish/bearish narrative)
|
||||
- What they reveal about their ACTUAL position in the same post
|
||||
|
||||
Classic divergence patterns in X posts:
|
||||
1. "BTC is going to $200k 🚀" + "took some chips off the table"
|
||||
→ Narrative: ultra bullish. Action: selling. FLAG.
|
||||
2. "This market is trash, crypto is dead" + "accumulated more $ETH"
|
||||
→ Narrative: bearish. Action: buying. FLAG.
|
||||
3. "Stay strong, don't sell!" + reveals they have <5% allocation
|
||||
→ Narrative: encouraging others to hold. Position: minimal. FLAG.
|
||||
4. "Not financial advice but..." followed by extreme conviction call
|
||||
→ Common hedge used right before a large directional call. Reduce
|
||||
conviction by 0.1. Not a flag by itself.
|
||||
5. "I was wrong about X, but now I think Y"
|
||||
→ Explicit reversal/stance_change. Mark stance_change=true.
|
||||
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
CRYPTO SLANG DECODER — handle these correctly
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
"aping in" / "aped" → bought (often large, impulsive)
|
||||
"degen'd" → took a risky position
|
||||
"paper hands" → sold too early (about someone else, not actionable)
|
||||
"diamond hands" → holding through drawdown (not a new position)
|
||||
"ngmi" → bearish on a token or person
|
||||
"wagmi" → general optimism, NOT a signal
|
||||
"rekt" → lost money, position closed
|
||||
"accumulated" / "acc" → bought (ongoing or past)
|
||||
"trimmed" / "trim" → partially sold
|
||||
"bags" / "holding bags"→ holding a position
|
||||
"flipped" → changed direction (stance_change=true)
|
||||
"this is the one" / "THE entry" → high conviction call
|
||||
"under the radar" → bullish on an obscure asset
|
||||
"cook" → doing something well, bullish signal about a project
|
||||
"cooked" (about a token) → bearish, likely dying/failed
|
||||
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
EXAMPLES — calibrate against these
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
[TRADE_SIGNAL — fire]
|
||||
|
||||
POST: "Aped into $SOL at $145. Full size. We go."
|
||||
→ tier: "trade_signal", action: "buy", asset: SOL, conviction: 0.92,
|
||||
timeframe: "immediate", talks_vs_trades_flag: false
|
||||
|
||||
POST: "Trimming half my $ETH here. Not selling the thesis but locking
|
||||
profits at 3x. Will re-add lower."
|
||||
→ tier: "trade_signal", action: "reduce", asset: ETH, conviction: 0.85,
|
||||
timeframe: "immediate", talks_vs_trades_flag: false
|
||||
|
||||
POST: "This is cooked. Sold everything. Moving to stable until macro clears."
|
||||
→ tier: "trade_signal", action: "sell", asset: null (multiple/all),
|
||||
conviction: 0.88, timeframe: "immediate"
|
||||
|
||||
[DIRECTIONAL — extract but lower weight]
|
||||
|
||||
POST: "SOL is setting up for a breakout. $180 is the line to watch."
|
||||
→ tier: "directional", action: "bullish", asset: SOL, conviction: 0.58,
|
||||
timeframe: "days"
|
||||
|
||||
POST: "ETH dominance will crush alts this cycle. The flippening is real."
|
||||
→ tier: "directional", action: "bullish", asset: ETH, conviction: 0.52,
|
||||
timeframe: "months"
|
||||
|
||||
[NOISE — output noise, empty tickers]
|
||||
|
||||
POST: "gm everyone 🌅"
|
||||
→ tier: "noise", tickers: []
|
||||
|
||||
POST: "The market is wild lmao. Stay safe everyone."
|
||||
→ tier: "noise", tickers: []
|
||||
|
||||
POST: "Interesting take by @SomeAnalyst on the Fed" [with RT of article]
|
||||
→ tier: "noise" (retweet of someone else, no original content)
|
||||
|
||||
POST: "Not financial advice but BTC is really interesting here 👀"
|
||||
→ tier: "noise" (too vague, "interesting" is not a stance)
|
||||
|
||||
POST: "Just attended [Conference]. Amazing speakers. Building in crypto
|
||||
is incredible."
|
||||
→ tier: "noise" (no market signal)
|
||||
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
OUTPUT FORMAT (strict JSON, no markdown)
|
||||
═══════════════════════════════════════════════════════════════════════
|
||||
{
|
||||
"post_type": "original" | "reply" | "retweet" | "quote" | "thread_cont",
|
||||
"tier": "trade_signal" | "directional" | "noise",
|
||||
"summary": "<one sentence in ENGLISH, ≤80 chars. State what the KOL
|
||||
is saying/doing. 'Noise: gm post' if tier is noise.>",
|
||||
"tickers": [
|
||||
{
|
||||
"ticker": "<UPPERCASE symbol>",
|
||||
"action": "buy" | "sell" | "reduce" | "bullish" | "bearish" | "mention",
|
||||
"conviction": <float 0.0-1.0>,
|
||||
"timeframe": "immediate" | "hours" | "days" | "weeks" | "months" | "unspecified",
|
||||
"stance_change": <true if this reverses a previously stated position in this post>,
|
||||
"quote": "<verbatim phrase from the post supporting this, ≤100 chars>"
|
||||
}
|
||||
],
|
||||
"talks_vs_trades_flag": <true | false>,
|
||||
"has_price_target": <true if a specific price level is mentioned>,
|
||||
"price_targets": [
|
||||
{ "ticker": "SOL", "price": 180, "direction": "up" | "down" }
|
||||
],
|
||||
"sentiment": "bullish" | "bearish" | "neutral",
|
||||
"reasoning": "<one sentence: why this tier? What specific phrase drove the decision?>"
|
||||
}
|
||||
|
||||
HARD RULES:
|
||||
• tier == "noise" → tickers MUST be [], talks_vs_trades_flag MUST be false
|
||||
• tier == "trade_signal" → at least one ticker with action in {buy, sell, reduce}
|
||||
AND conviction ≥ 0.7
|
||||
• post_type == "retweet" (RT without commentary) → tier MUST be "noise"
|
||||
• If no explicit price target → price_targets: []
|
||||
• conviction < 0.4 → tier cannot be "trade_signal"
|
||||
"""
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────
|
||||
# USER PROMPT TEMPLATE
|
||||
# ─────────────────────────────────────────────────────────────────────
|
||||
USER_TEMPLATE = """\
|
||||
KOL handle: @{handle}
|
||||
Follower tier: {follower_tier}
|
||||
Post time (UTC): {posted_at}
|
||||
Current UTC hour: {hour} ({liquidity_note})
|
||||
|
||||
POST:
|
||||
\"\"\"{text}\"\"\"
|
||||
|
||||
{thread_context}
|
||||
Score this post. Default to NOISE unless there is explicit, specific signal."""
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────
|
||||
# Helpers
|
||||
# ─────────────────────────────────────────────────────────────────────
|
||||
def _liquidity_note(hour: int) -> str:
|
||||
if 3 <= hour < 9:
|
||||
return "Asia overnight — thin liquidity, be extra strict"
|
||||
if 13 <= hour < 21:
|
||||
return "US session — normal strictness"
|
||||
return "Off-peak hours"
|
||||
|
||||
|
||||
def _follower_tier(follower_count: Optional[int]) -> str:
|
||||
if follower_count is None:
|
||||
return "unknown"
|
||||
if follower_count >= 500_000:
|
||||
return "mega (500k+)"
|
||||
if follower_count >= 100_000:
|
||||
return "large (100k-500k)"
|
||||
if follower_count >= 20_000:
|
||||
return "mid (20k-100k)"
|
||||
return "small (<20k)"
|
||||
|
||||
|
||||
_FALLBACK = {
|
||||
"post_type": "original",
|
||||
"tier": "noise",
|
||||
"summary": None,
|
||||
"tickers": [],
|
||||
"talks_vs_trades_flag": False,
|
||||
"has_price_target": False,
|
||||
"price_targets": [],
|
||||
"sentiment": "neutral",
|
||||
"reasoning": "",
|
||||
"model": None,
|
||||
"version": ANALYSIS_VERSION,
|
||||
"error": None,
|
||||
}
|
||||
|
||||
VALID_ACTIONS = {"buy", "sell", "reduce", "bullish", "bearish", "mention"}
|
||||
VALID_TIMEFRAMES = {"immediate", "hours", "days", "weeks", "months", "unspecified"}
|
||||
VALID_TIERS = {"trade_signal", "directional", "noise"}
|
||||
VALID_POST_TYPES = {"original", "reply", "retweet", "quote", "thread_cont"}
|
||||
|
||||
|
||||
async def analyze_x_post(
|
||||
*,
|
||||
handle: str,
|
||||
text: str,
|
||||
posted_at: Optional[str] = None,
|
||||
follower_count: Optional[int] = None,
|
||||
thread_context: Optional[str] = None,
|
||||
model: Optional[str] = None,
|
||||
) -> dict:
|
||||
"""Score an X post from a crypto KOL.
|
||||
|
||||
Args:
|
||||
handle: Twitter/X handle (without @)
|
||||
text: Post text (already stripped of URLs if desired)
|
||||
posted_at: ISO UTC timestamp of the post
|
||||
follower_count: Follower count for context (affects tier label)
|
||||
thread_context: Optional: preceding posts in thread, ≤500 chars
|
||||
model: Override model. Defaults to ai_model (quality path).
|
||||
|
||||
Returns dict with: post_type, tier, summary, tickers, talks_vs_trades_flag,
|
||||
has_price_target, price_targets, sentiment, reasoning, model, version.
|
||||
Errors return a noise-safe fallback (never raises).
|
||||
"""
|
||||
stripped = (text or "").strip()
|
||||
if not stripped:
|
||||
out = dict(_FALLBACK)
|
||||
out["error"] = "empty post"
|
||||
return out
|
||||
|
||||
# Fast pre-filter: bare RT with no added text is always noise
|
||||
low = stripped.lower()
|
||||
if low.startswith("rt @") and len(stripped) < 300:
|
||||
out = dict(_FALLBACK)
|
||||
out["post_type"] = "retweet"
|
||||
out["reasoning"] = "Pre-filtered: bare retweet with no commentary."
|
||||
return out
|
||||
|
||||
hour = datetime.now(timezone.utc).hour
|
||||
thread_block = (
|
||||
f"Thread context (preceding posts):\n\"\"\"\n{thread_context[:500]}\n\"\"\""
|
||||
if thread_context else ""
|
||||
)
|
||||
|
||||
user_prompt = USER_TEMPLATE.format(
|
||||
handle=handle,
|
||||
follower_tier=_follower_tier(follower_count),
|
||||
posted_at=posted_at or "unknown",
|
||||
hour=hour,
|
||||
liquidity_note=_liquidity_note(hour),
|
||||
text=stripped[:1000],
|
||||
thread_context=thread_block,
|
||||
)
|
||||
|
||||
use_anth = _use_anthropic()
|
||||
if model is None:
|
||||
# X analysis is near-real-time but less latency-critical than Trump.
|
||||
# Use the quality model for better signal/noise discrimination.
|
||||
model = ANTHROPIC_MODEL if use_anth else settings.ai_model
|
||||
|
||||
try:
|
||||
if use_anth:
|
||||
msg = await _anth().messages.create(
|
||||
model=model,
|
||||
max_tokens=800,
|
||||
temperature=0.1,
|
||||
system=SYSTEM_PROMPT,
|
||||
messages=[{"role": "user", "content": user_prompt}],
|
||||
)
|
||||
raw = (msg.content[0].text if msg.content else "").strip()
|
||||
else:
|
||||
is_reasoning = any(x in model for x in ("pro", "reasoner", "r1", "think"))
|
||||
kwargs: dict = {
|
||||
"model": model,
|
||||
"messages": [
|
||||
{"role": "system", "content": SYSTEM_PROMPT},
|
||||
{"role": "user", "content": user_prompt},
|
||||
],
|
||||
"max_tokens": 2000 if is_reasoning else 800,
|
||||
}
|
||||
if not is_reasoning:
|
||||
kwargs["temperature"] = 0.1
|
||||
kwargs["response_format"] = {"type": "json_object"}
|
||||
resp = await _oai().chat.completions.create(**kwargs)
|
||||
raw = (resp.choices[0].message.content or "").strip()
|
||||
|
||||
# Strip fences
|
||||
if raw.startswith("```"):
|
||||
lines = raw.split("\n")
|
||||
raw = "\n".join(lines[1:-1] if lines[-1].strip() == "```" else lines[1:])
|
||||
|
||||
result = json.loads(raw)
|
||||
|
||||
except json.JSONDecodeError as exc:
|
||||
logger.error("x_analysis JSON parse error for @%s: %s", handle, exc)
|
||||
out = dict(_FALLBACK)
|
||||
out["error"] = f"parse_error: {exc}"
|
||||
return out
|
||||
except Exception as exc:
|
||||
logger.error("x_analysis API error for @%s: %s", handle, exc)
|
||||
out = dict(_FALLBACK)
|
||||
out["error"] = f"api_error: {exc}"
|
||||
return out
|
||||
|
||||
# ── Normalize ────────────────────────────────────────────────────
|
||||
post_type = (result.get("post_type") or "original").lower()
|
||||
if post_type not in VALID_POST_TYPES:
|
||||
post_type = "original"
|
||||
|
||||
tier = (result.get("tier") or "noise").lower()
|
||||
if tier not in VALID_TIERS:
|
||||
tier = "noise"
|
||||
|
||||
# Enforce: retweet → noise
|
||||
if post_type == "retweet":
|
||||
tier = "noise"
|
||||
|
||||
# Normalize tickers
|
||||
raw_tickers = result.get("tickers") or []
|
||||
cleaned_tickers = []
|
||||
if tier != "noise":
|
||||
for t in raw_tickers:
|
||||
if not isinstance(t, dict):
|
||||
continue
|
||||
sym = (t.get("ticker") or "").strip().upper()
|
||||
if not sym or len(sym) > 12:
|
||||
continue
|
||||
action = (t.get("action") or "mention").lower()
|
||||
if action not in VALID_ACTIONS:
|
||||
action = "mention"
|
||||
try:
|
||||
conv = float(t.get("conviction") or 0)
|
||||
except (TypeError, ValueError):
|
||||
conv = 0.0
|
||||
conv = max(0.0, min(1.0, conv))
|
||||
timeframe = (t.get("timeframe") or "unspecified").lower()
|
||||
if timeframe not in VALID_TIMEFRAMES:
|
||||
timeframe = "unspecified"
|
||||
cleaned_tickers.append({
|
||||
"ticker": sym,
|
||||
"action": action,
|
||||
"conviction": round(conv, 2),
|
||||
"timeframe": timeframe,
|
||||
"stance_change": bool(t.get("stance_change", False)),
|
||||
"quote": (t.get("quote") or "")[:100],
|
||||
})
|
||||
|
||||
# Enforce: trade_signal requires ≥1 ticker with buy/sell/reduce + conviction ≥ 0.7
|
||||
if tier == "trade_signal":
|
||||
strong = [
|
||||
t for t in cleaned_tickers
|
||||
if t["action"] in {"buy", "sell", "reduce"} and t["conviction"] >= 0.7
|
||||
]
|
||||
if not strong:
|
||||
tier = "directional" # downgrade rather than drop
|
||||
|
||||
# Enforce: noise → empty tickers
|
||||
if tier == "noise":
|
||||
cleaned_tickers = []
|
||||
|
||||
sentiment = (result.get("sentiment") or "neutral").lower()
|
||||
if sentiment not in ("bullish", "bearish", "neutral"):
|
||||
sentiment = "neutral"
|
||||
|
||||
talks_vs_trades = bool(result.get("talks_vs_trades_flag", False))
|
||||
if tier == "noise":
|
||||
talks_vs_trades = False
|
||||
|
||||
# Price targets
|
||||
has_price_target = bool(result.get("has_price_target", False))
|
||||
raw_pts = result.get("price_targets") or []
|
||||
price_targets = []
|
||||
for pt in raw_pts:
|
||||
if not isinstance(pt, dict):
|
||||
continue
|
||||
ticker = (pt.get("ticker") or "").upper()
|
||||
try:
|
||||
price = float(pt.get("price") or 0)
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
direction = (pt.get("direction") or "up").lower()
|
||||
if ticker and price > 0 and direction in ("up", "down"):
|
||||
price_targets.append({"ticker": ticker, "price": price, "direction": direction})
|
||||
if not price_targets:
|
||||
has_price_target = False
|
||||
|
||||
return {
|
||||
"post_type": post_type,
|
||||
"tier": tier,
|
||||
"summary": (result.get("summary") or "").strip() or None,
|
||||
"tickers": cleaned_tickers,
|
||||
"talks_vs_trades_flag": talks_vs_trades,
|
||||
"has_price_target": has_price_target,
|
||||
"price_targets": price_targets,
|
||||
"sentiment": sentiment,
|
||||
"reasoning": str(result.get("reasoning", ""))[:500],
|
||||
"model": model,
|
||||
"version": ANALYSIS_VERSION,
|
||||
"error": None,
|
||||
}
|
||||
+2
-1
@@ -30,7 +30,8 @@ services:
|
||||
ENCRYPTION_KEY: ${ENCRYPTION_KEY}
|
||||
AI_API_KEY: ${AI_API_KEY}
|
||||
AI_BASE_URL: ${AI_BASE_URL:-https://api.gptsapi.net/v1}
|
||||
AI_MODEL: ${AI_MODEL:-claude-sonnet-4-6}
|
||||
AI_MODEL: ${AI_MODEL:-deepseek-v4-pro}
|
||||
INGEST_BASE_URL: http://localhost:8000
|
||||
HL_MAINNET: ${HL_MAINNET:-true}
|
||||
ENVIRONMENT: ${ENVIRONMENT:-production}
|
||||
healthcheck:
|
||||
|
||||
@@ -22,3 +22,4 @@ redis==7.0.1
|
||||
apscheduler==3.11.2
|
||||
python-dotenv==1.2.1
|
||||
cryptography==46.0.7
|
||||
slowapi==0.1.9
|
||||
|
||||
@@ -1,8 +1,12 @@
|
||||
"""
|
||||
把 AI 信号写回数据库。
|
||||
跳过纯RT/URL帖子和已有signal的帖子。
|
||||
并发执行提速，自动限速避免API超限。
|
||||
用法: python scripts/backfill_signals.py [--limit 500] [--overwrite]
|
||||
Backfill AI analysis onto historical Truth posts.
|
||||
|
||||
Safety rules:
|
||||
* Only re-analyze `source='truth'` rows. Technical/scanner posts already
|
||||
carry their own signal payloads and must not be sent through the Trump
|
||||
text analyzer.
|
||||
* Persist the full analysis payload so history doesn't become a mixed schema
|
||||
of old partial rows and new v5 rows.
|
||||
"""
|
||||
import asyncio
|
||||
import argparse
|
||||
@@ -38,8 +42,11 @@ async def process_one(post: Post, semaphore: asyncio.Semaphore, overwrite: bool)
|
||||
p.relevant = analysis["relevant"]
|
||||
p.prefilter_reason = analysis.get("prefilter_reason")
|
||||
p.analysis_version = analysis.get("analysis_version")
|
||||
if analysis["relevant"] and analysis["asset"] and not p.price_impact_asset:
|
||||
p.price_impact_asset = analysis["asset"]
|
||||
p.price_impact_asset = analysis["asset"] if analysis["relevant"] else None
|
||||
p.target_asset = analysis.get("target_asset")
|
||||
p.category = analysis.get("category")
|
||||
p.expected_move_pct = analysis.get("expected_move_pct")
|
||||
p.invalidation_price = analysis.get("invalidation_price")
|
||||
await db.commit()
|
||||
|
||||
sig = analysis["signal"]
|
||||
@@ -49,7 +56,10 @@ async def process_one(post: Post, semaphore: asyncio.Semaphore, overwrite: bool)
|
||||
async def main(limit: int, overwrite: bool):
|
||||
async with AsyncSessionLocal() as db:
|
||||
result = await db.execute(
|
||||
select(Post).order_by(Post.published_at.desc()).limit(limit)
|
||||
select(Post)
|
||||
.where(Post.source == "truth")
|
||||
.order_by(Post.published_at.desc())
|
||||
.limit(limit)
|
||||
)
|
||||
posts = result.scalars().all()
|
||||
|
||||
|
||||
+74
-12
@@ -13,7 +13,7 @@ What it does:
|
||||
* KOL Substack / podcast / blog polls
|
||||
* KOL on-chain snapshots (HL perps + Etherscan ERC-20 balances)
|
||||
* KOL divergence detector
|
||||
* BTC bottom-reversal + funding-reversal scanners
|
||||
* Optional: BTC bottom-reversal + funding-reversal scanners
|
||||
|
||||
What it INTENTIONALLY DOES NOT touch:
|
||||
* posts where source IN ('truth', 'btc_bottom_reversal', 'funding_reversal',
|
||||
@@ -33,6 +33,10 @@ Usage:
|
||||
DATABASE_URL='sqlite+aiosqlite:///./trumpsignal.db' \\
|
||||
venv/bin/python scripts/launch_seed.py --dry-run
|
||||
|
||||
# Seed only (pure pre-launch warmup, no truncation / deletion):
|
||||
DATABASE_URL='sqlite+aiosqlite:///./trumpsignal.db' \\
|
||||
venv/bin/python scripts/launch_seed.py --seed-only
|
||||
|
||||
# Execute:
|
||||
DATABASE_URL='sqlite+aiosqlite:///./trumpsignal.db' \\
|
||||
venv/bin/python scripts/launch_seed.py --yes
|
||||
@@ -198,7 +202,29 @@ async def wipe_phase(dry_run: bool) -> None:
|
||||
print(green(" ✓ wipe committed"))
|
||||
|
||||
|
||||
async def seed_real_data() -> None:
|
||||
async def abort_if_live_user_state() -> bool:
|
||||
"""Refuse destructive launch seeding if the DB already looks user-live."""
|
||||
async with AsyncSessionLocal() as db:
|
||||
active_subs = (await db.execute(
|
||||
select(func.count(Subscription.id)).where(Subscription.active == True)
|
||||
)).scalar() or 0
|
||||
tg_bindings = (await db.execute(
|
||||
select(func.count(TelegramBinding.id))
|
||||
)).scalar() or 0
|
||||
if active_subs or tg_bindings:
|
||||
print(red(
|
||||
"Refusing to run launch_seed on a DB that already has live user state "
|
||||
f"(active_subscriptions={active_subs}, telegram_bindings={tg_bindings})."
|
||||
))
|
||||
print(yellow(
|
||||
"Use --dry-run to inspect only. If you truly intend to clean this DB, "
|
||||
"do it manually with a one-off migration/backup plan."
|
||||
))
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
async def seed_real_data(*, exercise_scanners: bool = False) -> None:
|
||||
"""Re-fetch every upstream source. All are idempotent on (source, external_id)."""
|
||||
print(bold("\n── SEED (re-running upstream fetches) ──"))
|
||||
|
||||
@@ -237,15 +263,21 @@ async def seed_real_data() -> None:
|
||||
except Exception as e:
|
||||
print(red(f" ✗ FAILED: {type(e).__name__}: {e}"))
|
||||
|
||||
print("\n [5/5] BTC bottom + funding reversal scanners (exercise the path)...")
|
||||
from app.services.scanners.btc_bottom_reversal import scan_once as btc_scan
|
||||
from app.services.scanners.funding_reversal import scan_once as funding_scan
|
||||
for name, fn in [("btc_bottom", btc_scan), ("funding_reversal", funding_scan)]:
|
||||
try:
|
||||
await fn()
|
||||
print(green(f" ✓ {name} scan completed (fire conditional on market state)"))
|
||||
except Exception as e:
|
||||
print(red(f" ✗ {name} FAILED: {type(e).__name__}: {e}"))
|
||||
if exercise_scanners:
|
||||
print("\n [5/5] BTC bottom + funding reversal scanners (exercise the path)...")
|
||||
from app.services.scanners.btc_bottom_reversal import scan_once as btc_scan
|
||||
from app.services.scanners.funding_reversal import scan_once as funding_scan
|
||||
for name, fn in [("btc_bottom", btc_scan), ("funding_reversal", funding_scan)]:
|
||||
try:
|
||||
await fn()
|
||||
print(green(f" ✓ {name} scan completed (fire conditional on market state)"))
|
||||
except Exception as e:
|
||||
print(red(f" ✗ {name} FAILED: {type(e).__name__}: {e}"))
|
||||
else:
|
||||
print(yellow(
|
||||
"\n [5/5] Scanner exercise skipped by default. "
|
||||
"Use --exercise-scanners only if you explicitly want signal emission side-effects."
|
||||
))
|
||||
|
||||
|
||||
async def main() -> int:
|
||||
@@ -253,16 +285,46 @@ async def main() -> int:
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
p.add_argument("--dry-run", action="store_true",
|
||||
help="show what would be deleted, no DB writes")
|
||||
p.add_argument("--seed-only", action="store_true",
|
||||
help="run upstream fetch/warmup only; skip all wipe/truncation steps")
|
||||
p.add_argument("--yes", action="store_true",
|
||||
help="actually perform the wipe (required without --dry-run)")
|
||||
p.add_argument("--no-seed", action="store_true",
|
||||
help="skip the upstream re-fetch step")
|
||||
p.add_argument("--exercise-scanners", action="store_true",
|
||||
help="also run standalone scanners during seed (may emit signals/alerts)")
|
||||
args = p.parse_args()
|
||||
|
||||
if args.seed_only and args.dry_run:
|
||||
print(red("Choose either --dry-run or --seed-only, not both."))
|
||||
return 2
|
||||
|
||||
if args.seed_only and args.no_seed:
|
||||
print(red("--seed-only conflicts with --no-seed."))
|
||||
return 2
|
||||
|
||||
if args.seed_only:
|
||||
before = await report_counts("BEFORE")
|
||||
await seed_real_data(exercise_scanners=args.exercise_scanners)
|
||||
after = await report_counts("AFTER")
|
||||
|
||||
print(bold("\n── DELTA ──"))
|
||||
for k in sorted(before):
|
||||
if k.startswith("_"): continue
|
||||
d = after[k] - before[k]
|
||||
arrow = green(f"+{d}") if d > 0 else (red(str(d)) if d < 0 else " ·")
|
||||
print(f" {k:25s} {before[k]:6d} → {after[k]:6d} ({arrow})")
|
||||
|
||||
print(bold(green("\n✓ seed-only warmup complete. Next: start backend and run scripts/launch_smoke.py.")))
|
||||
return 0
|
||||
|
||||
if not args.dry_run and not args.yes:
|
||||
print(red("Refusing to run without --yes (or use --dry-run to preview)."))
|
||||
return 2
|
||||
|
||||
if not args.dry_run and await abort_if_live_user_state():
|
||||
return 2
|
||||
|
||||
before = await report_counts("BEFORE")
|
||||
await wipe_phase(dry_run=args.dry_run)
|
||||
|
||||
@@ -271,7 +333,7 @@ async def main() -> int:
|
||||
return 0
|
||||
|
||||
if not args.no_seed:
|
||||
await seed_real_data()
|
||||
await seed_real_data(exercise_scanners=args.exercise_scanners)
|
||||
|
||||
after = await report_counts("AFTER")
|
||||
|
||||
|
||||
@@ -73,6 +73,16 @@ SEED_WALLETS: list[dict] = [
|
||||
"label": "Arthur Hayes (secondary)",
|
||||
"source_url": "https://etherscan.io/address/0x534a0076fb7c2b1f83fa21497429ad7ad3bd7587",
|
||||
},
|
||||
# ⚠️ ORPHANED: the two entries below have VERIFIED on-chain attribution
|
||||
# but their `handle` does NOT match any handle in KOL_FEEDS, so the
|
||||
# divergence scanner has no post-side data to join against — they produce
|
||||
# ZERO divergence detections today. They are kept (a) because the wallet
|
||||
# attribution is sound and (b) so that if/when X/Twitter ingestion or a
|
||||
# matching RSS feed is added under these handles, the wallets light up
|
||||
# automatically. The cross-check in main() prints a warning for these.
|
||||
# • andrewkang — publishes on X/@Rewkang only (no Substack) → needs
|
||||
# X ingestion before post-side data exists.
|
||||
# • murad — not currently in KOL_FEEDS at all.
|
||||
{
|
||||
"handle": "andrewkang",
|
||||
"chain": "ethereum",
|
||||
@@ -152,6 +162,57 @@ async def main() -> int:
|
||||
|
||||
print()
|
||||
print(f"Inserted {inserted} wallets, skipped {skipped} existing.")
|
||||
|
||||
# ── Coverage cross-check ─────────────────────────────────────────────
|
||||
# A wallet is only useful for divergence detection if its handle has a
|
||||
# matching post-side feed in KOL_FEEDS. Flag any orphans loudly, and
|
||||
# report which feeds still have no wallet at all (the real coverage gap).
|
||||
try:
|
||||
from app.services.kol_substack import KOL_FEEDS
|
||||
feed_handles = {f["handle"] for f in KOL_FEEDS}
|
||||
except Exception as exc: # pragma: no cover - diagnostic only
|
||||
print(f"\n(could not import KOL_FEEDS for cross-check: {exc})")
|
||||
return 0
|
||||
|
||||
wallet_handles = {e["handle"] for e in SEED_WALLETS}
|
||||
orphaned = sorted(wallet_handles - feed_handles)
|
||||
feeds_without_wallet = sorted(feed_handles - wallet_handles)
|
||||
|
||||
print()
|
||||
print(f"Coverage: {len(wallet_handles - set(orphaned))}/{len(feed_handles)} "
|
||||
f"KOL_FEEDS handles have ≥1 wallet.")
|
||||
|
||||
# ── Orphan reconciliation ────────────────────────────────────────────
|
||||
# An orphaned wallet (handle ∉ KOL_FEEDS) can NEVER produce a divergence
|
||||
# because there's no post-side data to join against — but kol_onchain
|
||||
# still burns a scan cycle (HL clearinghouseState + Etherscan) on it every
|
||||
# run. Park such rows as active=False so the scanner skips them. This is
|
||||
# REVERSIBLE: the row + its verified attribution stay in the DB, and the
|
||||
# moment a matching feed (e.g. X/Twitter ingestion) is added, flip it back.
|
||||
if orphaned:
|
||||
async with AsyncSessionLocal() as session:
|
||||
res = await session.execute(
|
||||
select(KolWallet).where(
|
||||
KolWallet.handle.in_(orphaned),
|
||||
KolWallet.active.is_(True),
|
||||
)
|
||||
)
|
||||
rows = res.scalars().all()
|
||||
for row in rows:
|
||||
row.active = False
|
||||
await session.commit()
|
||||
deactivated = len(rows)
|
||||
print()
|
||||
print(f"⚠️ ORPHANED wallets (handle not in KOL_FEEDS → ZERO divergence) "
|
||||
f"— deactivated {deactivated} so the scanner skips them:")
|
||||
for h in orphaned:
|
||||
print(f" {h} — re-activate once a feed under this handle exists.")
|
||||
if feeds_without_wallet:
|
||||
print()
|
||||
print(f"📭 {len(feeds_without_wallet)} feeds have NO wallet (no on-chain "
|
||||
f"divergence possible):")
|
||||
print(" " + ", ".join(feeds_without_wallet))
|
||||
|
||||
print()
|
||||
print("Next step: edit SEED_WALLETS above and re-run. Each new wallet")
|
||||
print("MUST cite a public attestation in source_url — see the docstring.")
|
||||
|
||||
@@ -5,6 +5,8 @@ from app.services.macro.fetchers import (
|
||||
_latest_closed_daily_point,
|
||||
_parse_farside_latest_total,
|
||||
)
|
||||
from app.services.bottom_indicators import ahr999 as scanner_ahr999
|
||||
from app.services.macro import fetchers
|
||||
|
||||
|
||||
def test_drop_in_progress_daily_klines_removes_today_open_bar():
|
||||
@@ -48,3 +50,48 @@ def test_parse_farside_latest_total_uses_newest_date_not_first_row():
|
||||
|
||||
assert parsed["value"] == 321_000_000.0
|
||||
assert parsed["raw"]["date"] == "25 May 2026"
|
||||
|
||||
|
||||
def test_macro_ahr999_uses_same_formula_as_scanner(monkeypatch):
|
||||
# Build 300 daily bars with realistic OPEN timestamps so the fetcher's
|
||||
# in-progress-day drop (_drop_in_progress_daily_klines) actually fires on
|
||||
# the last bar (today's 00:00 UTC). With fake integer timestamps the drop
|
||||
# is a no-op, so the fetcher would compute AHR999 over one MORE bar than
|
||||
# the scanner below — producing a spurious 4th-decimal mismatch (the
|
||||
# fetcher and scanner share the exact same ahr999() function, so identical
|
||||
# inputs must yield identical output).
|
||||
now = datetime.now(timezone.utc)
|
||||
midnight_ms = int(
|
||||
now.replace(hour=0, minute=0, second=0, microsecond=0).timestamp() * 1000
|
||||
)
|
||||
day_ms = 86_400_000
|
||||
# Row i opens at (299 - i) days before today's midnight; the last row
|
||||
# (i=299) opens at today's midnight → still in progress → dropped.
|
||||
closes = [
|
||||
[midnight_ms - (299 - i) * day_ms, 0, 0, 0, str(50_000 + i)]
|
||||
for i in range(300)
|
||||
]
|
||||
|
||||
class _Resp:
|
||||
def raise_for_status(self):
|
||||
return None
|
||||
|
||||
def json(self):
|
||||
return closes
|
||||
|
||||
class _Client:
|
||||
async def __aenter__(self):
|
||||
return self
|
||||
|
||||
async def __aexit__(self, exc_type, exc, tb):
|
||||
return None
|
||||
|
||||
async def get(self, *_args, **_kwargs):
|
||||
return _Resp()
|
||||
|
||||
monkeypatch.setattr(fetchers.httpx, "AsyncClient", lambda *a, **kw: _Client())
|
||||
|
||||
result = __import__("asyncio").run(fetchers.fetch_ahr999())
|
||||
expected = scanner_ahr999([float(r[4]) for r in closes[:-1]])
|
||||
|
||||
assert result["value"] == round(expected or 0, 4)
|
||||
|
||||
Reference in New Issue
Block a user