Z/trumpsignal-backend
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(ai): v5 extreme-alpha prompt — checklist gate + drop sell signal

Browse Source 
Why
  v4 was firing buy/short on 13% of posts, but only 9% of those had a
  ≥1% move within the hour. Median move on 'actionable' was 0.298% vs
  0.258% on 'hold' — a 1.15× signal-to-noise ratio (random would be 1.0).
  The model was confabulating transmission chains to please the user
  rather than holding when uncertain.

  Separately: 'sell' meant 'close longs / de-risk' in the prompt but
  was traded as 'open short' by bot_engine.py, producing systematically
  negative results on sell signals (27% win rate vs 57% on real shorts).

What changed
  • analysis.py rewritten as v5-extreme-alpha:
    - Asymmetric error costs framing (false positive = -$30, FN = $0)
    - 7-item checklist that MUST all pass before buy/short
    - Only 4 named transmission paths (a/b/c/d); anything else = HOLD
    - 5 positive + 5 negative few-shot examples
    - UTC hour injected with liquidity context (Asia thin → stricter)
    - Adversarial steelman self-check before final output
    - confidence < 80 + checklist failure both force-collapse to HOLD
      in code, regardless of what the model returns (defense-in-depth)
    - 'sell' removed from output schema entirely
  • bot_engine.py: stop trading 'sell' signals (treat as hold)
  • Case-insensitive normalization on checklist values so model
    returning 'None'/'True' (capitalized) doesn't slip through

Expected impact (to validate over next 2-3 weeks of new posts)
  • actionable rate: 13% → 2-4%
  • signal/hold MFE ratio: 1.15× → 3-5×
  • ≥1% hit rate among actionable: 9% → 40-60%

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
k
2026-05-08 15:00:44 +08:00
parent 4ffcb442fe
commit 747158b5ed
2 changed files with 276 additions and 106 deletions
Download Patch File Download Diff File Expand all files Collapse all files
app/services/analysis.py
+272 -104
View File
@@ -1,5 +1,29 @@
"""
Trump Truth Social → BTC/ETH signal scorer.
v5-extreme-alpha: rewritten for high-precision, low-recall trading.
Design notes (read before tweaking):
- Default is HOLD. The cost of a false positive (-$30 with 20× leverage) is
much higher than a missed signal (0). Prompt is explicitly biased that way.
- Output schema is buy / short / hold ONLY. "sell" was previously emitted
to mean "close longs" and silently mistraded as "open short" by the bot.
Removed entirely.
- 4 named transmission paths only. If the AI cannot fit the post into one
of (a/b/c/d), the answer is HOLD — no fancy multi-step storytelling.
- Confidence < 80 is forced to HOLD inside the prompt. Padding to 70 to
"play it safe with a soft signal" produces the same downstream outcome
as just outputting hold, so we don't let the AI hedge.
- Few-shot examples are concrete posts that did or didn't move BTC, drawn
from the press cards on the landing page (Mar 2025 reserve announcement,
repeated rhetoric, geopolitical noise, etc.).
- The current UTC hour is injected so the AI can apply a tighter filter
during Asia thin-liquidity hours (UTC 03-09).
"""
import json
import logging
from datetime import datetime, timezone
from typing import Optional
from openai import AsyncOpenAI
@@ -9,127 +33,198 @@ logger = logging.getLogger(__name__)
_client: Optional[AsyncOpenAI] = None
SYSTEM_PROMPT = """You are an expert macro trader analyzing Trump's Truth Social posts for real-time crypto trading signals.
ANALYSIS_VERSION = "v5-extreme-alpha"
CORE ASSUMPTION: Treat every post as BREAKING NEWS. Do not assume anything is already priced in — you have no knowledge of what happened before or after this post. Your job is to evaluate the post content only.
=== BULLISH TRANSMISSION CHAINS (BTC/ETH go UP) ===
• Ceasefire / peace deal / de-escalation / conflict ending
→ war-risk premium unwinds → oil stabilizes → USD softens → risk-on → BTC/ETH rally
• Pro-crypto executive action / legislation / reserve
→ regulatory tailwind + direct demand → BTC up immediately
• Dollar weakness (deficit spending, tariff concessions, Fed rate cut signals)
→ hard asset bid → BTC as digital gold
• Trade deal / tariff reduction announcement
→ global growth outlook improves → risk appetite rises → BTC/ETH up
• Strait of Hormuz open / oil supply stable
→ energy prices moderate → inflation fears drop → risk-on
# ────────────────────────────────────────────────────────────────────
# SYSTEM PROMPT
# ────────────────────────────────────────────────────────────────────
SYSTEM_PROMPT = """\
You are the news-analyst desk at a quant crypto fund. You read every Trump
Truth Social post and decide whether it justifies an IMMEDIATE trade.
=== BEARISH TRANSMISSION CHAINS (BTC/ETH go DOWN) ===
• War escalation / military strikes / ceasefire violation
→ risk-off panic → crypto liquidations → sharp BTC/ETH drop
• Strait of Hormuz threatened / blocked / mined
→ oil price spike → inflation surge → Fed tightening fears → risk-off
• SEC/DOJ crypto enforcement / new crypto regulation
→ direct selling pressure → BTC/ETH down
• Tariff escalation / new sanctions / trade war
→ growth fears + USD safe-haven bid → crypto outflows
• New military front / ally breakdown / nuclear threat
→ extreme risk-off → everything sells
Your job is NOT to find "interesting" posts. Your job is to find posts where
a sophisticated trader would aggressively click the trade button within 30
seconds of seeing them. Everything else is HOLD.
=== WHAT IS NOT RELEVANT ===
• Retweets with only a URL and no added comment (RT: https://...)
• Media/political attacks with no policy content
• Personal endorsements, rallies, awards, domestic court cases
• Pure domestic politics (abortion, immigration) with no macro angle
• UFO files, sports, entertainment, holidays
═══════════════════════════════════════════════════════════════════════
ASYMMETRIC ERROR COSTS — internalize this before everything else
═══════════════════════════════════════════════════════════════════════
• False positive (you say buy/short, no real move): -$30 of real money lost
• False negative (you say hold, real catalyst): $0 missed (we didn't
trade — that's fine)
=== SIGNAL TAXONOMY (CRITICAL — do not conflate) ===
Each signal is a distinct trading action. Choose exactly one.
You are punished for false positives. You are NOT punished for false
negatives. **DEFAULT IS ALWAYS HOLD.** When in doubt, HOLD. When the
catalyst is "interesting but not specific", HOLD. When you would not
personally bet your salary on the call, HOLD.
• "buy" — OPEN A NEW LONG. Use when the post is a FRESH bullish catalyst that
is likely to push price UP in the next 5–60 minutes. Example:
"Confirmed ceasefire with Iran", "Crypto reserve executive order signed."
→ Expected move: price rises. Directional accuracy = price > entry.
═══════════════════════════════════════════════════════════════════════
THE ONLY 4 TRANSMISSION PATHS
═══════════════════════════════════════════════════════════════════════
A post can only justify a signal if you can write a chain ≤ 15 words
using ONE of these paths:
• "sell" — CLOSE AN EXISTING LONG / DE-RISK. Use when the post WEAKENS a prior
bullish thesis but is NOT strong enough to justify an active short.
Think: "take profit / step aside." Examples:
"Ceasefire talks delayed", "Hinting at new tariffs (vague)",
"Crypto-friendly nominee withdrawn." Ambiguous bearish.
→ Use ONLY if you would exit longs but not open shorts.
→ Directional accuracy = price < entry (same as short, but lower conviction).
(a) DIRECT_CRYPTO — post explicitly names BTC / ETH / crypto / SEC / CFTC
/ Strategic Reserve / a specific coin policy.
(b) USD_RATES — concrete tariff order, Fed rate signal, debt deal,
USD-strengthening or weakening fiscal action.
(c) GEOPOLITICAL — confirmed military strike, ceasefire signed/broken
with NAMED parties, Strait of Hormuz status, sanctions
enacted (not threatened, not "considering", ENACTED).
(d) REGULATORY — SEC/DOJ enforcement filed/settled against a named
crypto entity, executive order signed.
• "short"— OPEN A NEW SHORT. Use only for a FRESH bearish catalyst strong enough
that you would actively bet on price falling. Examples:
"Military strike on Iran launched", "Strait of Hormuz mined",
"SEC lawsuit against major exchange announced."
→ Expected move: price drops hard. Reserve for high-conviction bearish.
If your transmission chain requires path (e) "well, this might cause
markets to get nervous about uncertainty" → REJECT, that is HOLD.
If you find yourself writing "→ ... → ... → ... → BTC" with 4+ steps
→ REJECT, that is confabulation, output HOLD.
• "hold" — NO TRADE. Post is irrelevant, or macro-relevant but ambiguous in
direction, or confidence too low to act. DEFAULT when in doubt.
═══════════════════════════════════════════════════════════════════════
CHECKLIST (must answer ALL 7 before allowing buy/short)
═══════════════════════════════════════════════════════════════════════
1. SPECIFICITY: Does the post name a specific entity, number, date,
or order? (Vague rhetoric → no.)
2. NEW INFO: Is this NEW information, not something Trump has said
in the past 30 days? (Repeats are priced in.)
3. TRANSMISSION: Can you write the chain in ≤ 15 words using exactly
ONE of (a/b/c/d)?
4. EXECUTION ≤ 24h: Is the action effective today/tomorrow, OR is it
just "I will" / "considering" / "thinking about"?
(Future intent ≠ tradeable. HOLD on future-tense.)
5. NOT DOMESTIC: Is this US-only domestic politics (elections, courts,
opponents, monuments, agency disputes, immigration,
medical claims)? If yes → HOLD regardless.
6. NOT RT: Does the post start with "RT @" or "RT:" or is it just
a URL with no commentary? If yes → HOLD.
7. SURPRISE TEST: Would a Bloomberg desk lead with this as a HEADLINE?
If your honest answer is "this is just Trump being
Trump" → HOLD.
Decision tree:
1. Is there a concrete macro/crypto event? NO → hold.
2. Is the direction clear? NO → hold.
3. BULLISH: confidence ≥ 60 → buy. Else → hold.
4. BEARISH: strong/explicit (war, ban, sanctions, enforcement) → short.
weak/vague/partial walkback of bullish → sell.
otherwise → hold.
A single FAIL on any item → HOLD. There are no exceptions, no "well
mostly yes". Asymmetric costs (see top) make it cheap to over-reject.
NEVER use "sell" for a strong bearish event — that is "short".
NEVER use "short" for a vague/ambiguous negative — that is "sell" or "hold".
═══════════════════════════════════════════════════════════════════════
CONFIDENCE CALIBRATION (read before assigning a number)
═══════════════════════════════════════════════════════════════════════
90-100 — "Bet the desk." All 7 checklist items pass STRONGLY. Concrete
order/EO/sanction taking effect <24h on a major crypto-relevant
entity. Should fire ≤ 1×/week historically.
80-89 — "I'd trade this." All 7 checklist items pass. Catalyst is
specific and not yet priced in. Should fire ≤ 1×/2 weeks.
≤ 79 — HOLD. The trading bot's minimum threshold is 80, so anything
below 80 produces zero downstream action. Do NOT pad to 70-79
as a "soft signal" — that wastes your output. Just emit HOLD.
=== CONFIDENCE CALIBRATION ===
80-100: Explicit crypto/monetary policy action; confirmed ceasefire with named parties; Strait of Hormuz status confirmed; named trade deal signed
60-79: Clear new geopolitical event with specific details (named countries, named actions); direct risk-on/off chain
40-59: Probable macro relevance but vague details or highly indirect effect
20-39: Possible relevance, highly uncertain
0-19: Mark as not relevant instead
═══════════════════════════════════════════════════════════════════════
EXAMPLES — calibrate against these
═══════════════════════════════════════════════════════════════════════
=== SIGNAL CALIBRATION (critical — read before deciding) ===
DEFAULT is HOLD. Only 5–10% of posts should receive buy or short.
The overwhelming majority of Trump's posts are domestic politics, rhetoric, personal attacks, or vague boasts — these are HOLD.
[REAL CATALYST — should fire]
Use "buy" only when ALL of the following are true:
1. The event is CONCRETE (named countries, named action, specific policy)
2. The macro transmission chain to BTC/ETH is DIRECT and SHORT (< 2 steps)
3. Your confidence is ≥ 75
POST: "Today I am announcing a Strategic Crypto Reserve including Bitcoin,
Solana, XRP, Ethereum. Effective immediately by Executive Order."
ANSWER: signal=buy, conf=95
WHY: (a) DIRECT_CRYPTO. Specific entities, immediate execution, concrete
EO. All 7 checklist items pass strongly. This is the textbook hit.
Use "short" only when ALL of the following are true:
1. A hard bearish event is CONFIRMED (not speculated): verified military strike, enacted tariff, filed lawsuit
2. The transmission chain to BTC/ETH down is DIRECT
3. Your confidence is ≥ 80
POST: "Effective midnight Eastern time, 25% tariffs on all Chinese imports.
Signed today. No exceptions."
ANSWER: signal=short, conf=88
WHY: (b) USD_RATES. Concrete order, immediate execution, named country and
percentage. Strong USD bid + risk-off → BTC down.
Use "sell" only for mild walkback of a prior bullish event — NOT for strong bearish news.
POST: "Just spoke with Israel and Iran. Ceasefire signed and effective in
6 hours. All hostilities to cease."
ANSWER: signal=buy, conf=85
WHY: (c) GEOPOLITICAL. Named parties, signed (not "discussing"), hard
deadline. Risk-on unwind → BTC up.
When in doubt between buy and hold → choose HOLD.
When in doubt between short and sell → choose SELL or HOLD, not short.
[NOT A CATALYST — must be HOLD]
Return ONLY valid JSON. No markdown.
POST: "BITCOIN is the FUTURE of money. America LEADS the world in crypto!"
ANSWER: signal=hold, conf=0
WHY: Pure rhetoric, no specific action. Trump has said variations of this
30+ times. Priced in. Fails checklist #1 (specificity) and #2 (newness).
Return ONLY valid JSON. No markdown."""
POST: "I am thinking very seriously about new tariffs on China. We will see."
ANSWER: signal=hold, conf=0
WHY: Future-tense, "thinking about". Fails checklist #4 (execution ≤ 24h).
USER_PROMPT_TEMPLATE = """Analyze this Trump Truth Social post for BTC/ETH trading signals.
Treat it as breaking news — evaluate only what is written.
POST: "Iran tankers approaching US coasts. Texas Louisiana Alaska. NOT GOOD!"
ANSWER: signal=hold, conf=0
WHY: Vague threat description. No specific action announced. Trump
frequently makes such claims without follow-through. Fails #1 and #4.
POST: "Crooked Hillary will go to JAIL for what she's done. SAD!"
ANSWER: signal=hold, conf=0
WHY: Domestic politics. Fails #5.
POST: "RT @realDonaldTrump: The wall is being built faster than ever!"
ANSWER: signal=hold, conf=0
WHY: Retweet. Fails #6.
POST: "The Fed must CUT rates NOW. They are STUPID and WEAK!"
ANSWER: signal=hold, conf=0
WHY: Rhetoric / Fed criticism, no concrete fiscal action. Trump complains
about Fed weekly. Priced in. Fails #1 and #2.
═══════════════════════════════════════════════════════════════════════
ADVERSARIAL SELF-CHECK (before final output)
═══════════════════════════════════════════════════════════════════════
Before you commit to buy/short, ask yourself: would a contrarian quant
who has read 10,000 Trump posts actually click TRADE on this? Steelman
the case for HOLD in one sentence. If the steelman is convincing,
DOWNGRADE TO HOLD.
═══════════════════════════════════════════════════════════════════════
OUTPUT FORMAT (strict JSON, no markdown)
═══════════════════════════════════════════════════════════════════════
{
"checklist": {
"specificity": "<the specific entity/number/date OR 'none'>",
"novelty": "<'new' | 'repeated' | 'unsure'>",
"transmission": "<'a' | 'b' | 'c' | 'd' | 'none'>",
"execution": "<'<24h' | 'future' | 'none'>",
"domestic": <true|false>,
"is_rt": <true|false>,
"surprise": "<'headline' | 'noise'>"
},
"relevant": <true|false>,
"asset": "BTC" | "ETH" | null,
"sentiment": "bullish" | "bearish" | "neutral",
"signal": "buy" | "short" | "hold",
"confidence": <0-100>,
"reasoning": "<EITHER the ≤15-word transmission chain (if buy/short),
OR which checklist item failed (if hold). One sentence.>"
}
HARD CONSTRAINTS on the JSON:
• If signal == "buy" or "short", then confidence MUST be ≥ 80.
Lower than 80? Change signal to "hold" and confidence to 0.
• If transmission == "none" OR domestic == true OR is_rt == true OR
execution == "future" OR execution == "none", then signal MUST be
"hold" and confidence MUST be 0.
• Never emit "sell" — that signal type does not exist anymore.
"""
USER_PROMPT_TEMPLATE = """\
Trump just posted on Truth Social. Score it for an immediate BTC/ETH
trade per your system rules.
Current UTC hour: {hour} ({liquidity_note})
POST TEXT:
{text}
Respond with JSON:
{{
"relevant": <true if post contains ANY concrete macro/geopolitical/crypto information>,
"asset": "BTC" | "ETH" | "BOTH" | null,
"sentiment": "bullish" | "bearish" | "neutral",
"signal": "buy" | "sell" | "short" | "hold",
"confidence": <0-100>,
"reasoning": "<What specific event is described → macro transmission chain → expected market direction → WHY this signal and NOT the adjacent one (e.g. 'short not sell because strike is confirmed', 'sell not short because only a walkback of prior bullish headline')>"
}}
Run the 7-item checklist out loud in the JSON `checklist` field, then
decide. Default to HOLD. Only buy/short if every gate passes."""
If post is only a URL, only a retweet link, or purely personal/domestic with zero macro angle: set relevant=false, signal=hold, confidence=0."""
ANALYSIS_VERSION = "v4-selective"
# ────────────────────────────────────────────────────────────────────
# Implementation
# ────────────────────────────────────────────────────────────────────
_FALLBACK = {
"relevant": False,
"asset": None,
@@ -159,47 +254,120 @@ def _get_client() -> AsyncOpenAI:
return _client
def _liquidity_note(hour: int) -> str:
"""Inject session context so AI can apply tighter filters during
Asia thin-liquidity hours when crypto reactions are smaller."""
if 3 <= hour < 9:
return "Asia overnight, thin liquidity — be EXTRA strict, only the cleanest signals fire"
if 13 <= hour < 21:
return "US session, deepest liquidity — normal strictness applies"
return "Off-peak hours, moderate liquidity"
async def analyze_post(text: str) -> dict:
# Fast pre-filter: pure RT/URL with no content
"""Score a Trump post and return signal + structured reasoning.
Returns the canonical dict shape expected by the rest of the codebase:
relevant, asset, sentiment, signal (buy/short/hold), confidence,
reasoning, prefilter_reason, analysis_version.
"""
# ── Fast pre-filter (no AI call) ────────────────────────────────
stripped = text.strip()
if not stripped:
return _fallback("empty", "Pre-filtered: empty post body.")
if stripped.startswith("RT: https://") and len(stripped) < 60:
return _fallback("rt_only", "Pre-filtered: retweet with no added commentary.")
if stripped.startswith("https://") and " " not in stripped:
return _fallback("url_only", "Pre-filtered: bare URL with no text content.")
if not stripped:
return _fallback("empty", "Pre-filtered: empty post body.")
# New v5 prefilter: strict RT-prefix, even with extra text. Pure retweets
# of self / others rarely contain new tradeable info.
low = stripped.lower()
if low.startswith("rt @") and len(stripped) < 200:
return _fallback("short_rt", "Pre-filtered: short retweet, unlikely to contain new catalyst.")
# ── Full AI scoring ─────────────────────────────────────────────
hour = datetime.now(timezone.utc).hour
user_prompt = USER_PROMPT_TEMPLATE.format(
hour=hour,
liquidity_note=_liquidity_note(hour),
text=text[:2000],
)
try:
client = _get_client()
response = await client.chat.completions.create(
model=settings.ai_model,
max_tokens=450,
max_tokens=600,
temperature=0.1,
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": USER_PROMPT_TEMPLATE.format(text=text[:2000])},
{"role": "user", "content": user_prompt},
],
)
raw = response.choices[0].message.content.strip()
raw = (response.choices[0].message.content or "").strip()
# Strip ```json fences if the model adds them despite instructions
if raw.startswith("```"):
lines = raw.split("\n")
raw = "\n".join(lines[1:-1] if lines[-1].strip() == "```" else lines[1:])
result = json.loads(raw)
# ── Normalize + enforce hard constraints ─────────────────────
sentiment = result.get("sentiment", "neutral")
if sentiment not in ("bullish", "bearish", "neutral"):
sentiment = "neutral"
signal = result.get("signal", "hold")
if signal not in ("buy", "sell", "short", "hold"):
# v5: drop "sell" entirely. If the model still emits it (during the
# transition before retraining), coerce to hold.
if signal == "sell":
signal = "hold"
if signal not in ("buy", "short", "hold"):
signal = "hold"
confidence = int(result.get("confidence", 0))
confidence = int(result.get("confidence", 0) or 0)
confidence = max(0, min(100, confidence))
relevant = bool(result.get("relevant", False))
# Hard floor: signal != hold MUST have confidence ≥ 80. Otherwise
# the bot's downstream subscribers (whose min_confidence defaults
# to 60-70) would still fire on weak signals. Force-collapse here.
if signal in ("buy", "short") and confidence < 80:
signal = "hold"
confidence = 0
# Checklist gate — if any of these failed, the model should already
# have set hold, but enforce it as a defense-in-depth check.
# Normalize values: model occasionally returns "None"/"Future"/"True"
# (capitalized) or string "true" instead of JSON true. Lowercase
# everything before comparing.
checklist = result.get("checklist") or {}
def _norm(v):
return v.strip().lower() if isinstance(v, str) else v
def _is_truthy(v):
v = _norm(v)
return v is True or v in ("true", "yes", "1")
transmission = _norm(checklist.get("transmission"))
execution = _norm(checklist.get("execution"))
bad_checks = (
transmission in (None, "none", "")
or _is_truthy(checklist.get("domestic"))
or _is_truthy(checklist.get("is_rt"))
or execution in (None, "none", "future", "")
)
if signal in ("buy", "short") and bad_checks:
logger.info(
"Coerced signal to hold due to failed checklist gate: %s",
checklist,
)
signal = "hold"
confidence = 0
relevant = bool(result.get("relevant", False)) or signal in ("buy", "short")
asset = result.get("asset")
if asset == "BOTH":