improve signed reads, crypto hardening, and scraper transport
This commit is contained in:
k
@@ -105,7 +105,13 @@ linked) = Trump auto-trade + /adopt manage-only flow for sys2.
|
||||
- **Trading**: Hyperliquid SDK; API-wallet keys are envelope-encrypted with
|
||||
`ENCRYPTION_KEY` (KEK), per-user DEK derivation via `crypto.py`.
|
||||
- **Prices**: Two feeds:
|
||||
- `binance.py` WebSocket → BTC, ETH, SOL, TRUMP, BNB, DOGE, LINK, AAVE
|
||||
- `binance.py` WebSocket → 30 mainstream perps (BTC, ETH, SOL, TRUMP, BNB,
|
||||
DOGE, LINK, AAVE + AVAX/ARB/OP/SUI/APT/INJ/ATOM/XRP/LTC/ADA/MATIC/SHIB/
|
||||
PEPE/WIF/BONK/TAO/JUP/RENDER/FET/TIA/SEI/PENDLE). The WS URL is built FROM
|
||||
`ASSET_MAP` so the two never drift. Any asset NOT in ASSET_MAP loses
|
||||
TP/SL/trailing protection (only max-hold remains) — see the ASSET_MAP
|
||||
docstring. `tp_sl_monitor.register_trade` logs an ERROR if a trade opens on
|
||||
an uncovered asset.
|
||||
- `hl_price_feed.py` polls HL `allMids` every 2s → HYPE, PURR (HL-native assets not on Binance)
|
||||
Both pump `price_store` + `tp_sl_monitor` on every tick.
|
||||
- **Telegram**: long-poll mode (single instance), HTML messages, inline
|
||||
@@ -173,7 +179,9 @@ app/
|
||||
│ │ candles for [published_at, +max_hold_h] and
|
||||
│ │ replays current exit rules. Conservative (uses
|
||||
│ │ HIGH/LOW within bar). No fees. Batch runner on top.
|
||||
│ ├── crypto.py HL API-key envelope encryption
|
||||
│ ├── crypto.py HL API-key envelope encryption. enc:v2 =
|
||||
│ │ PBKDF2-salted (H4 fix); enc:v1 read-compat.
|
||||
│ │ scripts/reencrypt_keys.py upgrades stored rows.
|
||||
│ ├── scanner_state.py In-memory toggle + observability for scanners
|
||||
│ ├── macro/
|
||||
│ │ ├── fetchers.py 8 macro indicator HTTP fetchers (each @_none_on_fail)
|
||||
@@ -190,12 +198,32 @@ app/
|
||||
│ │ feed for X-only KOLs (andrewkang, murad).
|
||||
│ ├── kol_onchain.py HL public API + Etherscan diff
|
||||
│ ├── kol_divergence.py Cross-ref talks vs trades within ±7d
|
||||
│ ├── kol_analysis.py AI ticker/direction/conviction extract (Substack)
|
||||
│ ├── kol_analysis.py AI ticker/direction/conviction extract (Substack).
|
||||
│ │ `_derive_tier()` maps its conviction + talks-vs-
|
||||
│ │ trades score → the SAME trade_signal/directional/
|
||||
│ │ noise tiers x_analysis emits, so non-Twitter posts
|
||||
│ │ get tier set in kol_substack (SIGNAL/VIEW badges +
|
||||
│ │ "Signals only" filter work for blog/substack/pod).
|
||||
│ ├── bottom_indicators.py AHR999 / Pi Cycle / 200WMA math
|
||||
│ ├── funding_signal.py Real-time funding extreme detector
|
||||
│ └── signed_request.py EIP-191 signature verification (+ replay cache)
|
||||
│ ├── signed_request.py EIP-191 signature verification (+ replay cache).
|
||||
│ │ signed_read_creds / optional_signed_read_creds:
|
||||
│ │ header-based (X-Sig-Ts/X-Sig-Sig) creds for read
|
||||
│ │ endpoints, query fallback deprecated (C3 fix).
|
||||
│ └── http_client.py Shared pooled httpx.AsyncClient (keep-alive).
|
||||
│ Hot paths (scrapers, telegram send/poll,
|
||||
│ hl_price_feed, binance REST, x_poster) use
|
||||
│ get_client() with per-request timeout instead of
|
||||
│ new-client-per-call. Closed in lifespan shutdown.
|
||||
├── scrapers/
|
||||
│ ├── truth_social.py CNN archive poller (5s interval) — primary
|
||||
│ ├── truth_social.py CNN archive poller (5s interval) — primary.
|
||||
│ │ Conditional GET (ETag/Last-Modified → 304 skips
|
||||
│ │ the 30k-post download), batch dedup (1 IN-query
|
||||
│ │ per poll instead of 50 SELECTs), per-post
|
||||
│ │ commit+dispatch so an actionable post never
|
||||
│ │ waits behind older entries' AI analysis.
|
||||
│ │ dispatch_post() is the shared WS/TG/X/trade
|
||||
│ │ fan-out used by BOTH pollers.
|
||||
│ └── trumpstruth.py trumpstruth.org RSS fallback poller. Same post id
|
||||
│ hash → automatic dedup. Whoever sees first wins.
|
||||
│ Offset by half the interval so the two pollers
|
||||
@@ -225,11 +253,14 @@ scripts/ One-shot ops
|
||||
│ Idempotent (INSERT OR IGNORE). Run once at first deploy.
|
||||
├── rescore_v5.py Re-score every Post with current AI prompt
|
||||
├── backfill_signals.py Fill in signal for posts missing it
|
||||
├── reencrypt_keys.py Upgrade stored HL keys to enc:v2 (+KEK rotation
|
||||
│ via OLD_ENCRYPTION_KEY). Idempotent; --dry-run.
|
||||
└── verify_sys2_lifecycle.py Manual System-2 lifecycle walk-through
|
||||
|
||||
tests/ pytest, 83 tests, fast (<3s total)
|
||||
tests/ pytest, 112 tests, fast (<3s total)
|
||||
├── test_adoption.py Adoption + release flow (snapshot-style, no real HL/AI)
|
||||
├── test_telegram_digest.py Daily digest formatting
|
||||
├── test_kol_tier.py kol_analysis._derive_tier (non-Twitter tier mapping)
|
||||
├── test_kol_x.py X ingest: dedup / mapping / no-op (mocked fetch+AI)
|
||||
├── test_ratelimit.py Rate limit coverage (BUG-02 fix)
|
||||
├── test_bottom_reversal_strategy.py btc_bottom_reversal 2-of-3 logic
|
||||
@@ -292,6 +323,8 @@ minimalism — don't extend them.
|
||||
└─ adoption.adopt_position(wallet, asset, mode):
|
||||
a. Per-wallet asyncio lock acquired
|
||||
b. Pre-flight: no_subscription / no_hl_key / paper_mode /
|
||||
macro_disabled (Subscription.macro_enabled must be ON — /adopt is the
|
||||
sys2 management entry point, so the Macro Vibes toggle gates it) /
|
||||
circuit_breaker (sys2 CB still gates adopt!) /
|
||||
already_adopted / concurrency_cap (3)
|
||||
c. Re-read HL state inside lock (fresh entry/size/lev)
|
||||
@@ -432,7 +465,7 @@ uvicorn app.main:app --reload # dev only; prod uses --workers 1, no --reload
|
||||
|
||||
```bash
|
||||
source venv/bin/activate
|
||||
python -m pytest tests/ -q # 79 tests, ~2s
|
||||
python -m pytest tests/ -q # 112 tests, ~2s
|
||||
python scripts/preflight.py # env + DB + TG + AI auth checks
|
||||
python scripts/launch_smoke.py # 14 end-to-end checks vs running API
|
||||
```
|
||||
@@ -498,7 +531,20 @@ this for you" to "you opened it, bot manages your discipline".
|
||||
(Trump) reads full `daily_budget_usd`. Don't read it for new code.
|
||||
- **Adopted trades have `hl_order_id` starting with `"adopted:"`.** Distinct
|
||||
from auto-opened (HL order id integer) and paper (`"paper"` literal).
|
||||
Useful for telemetry filtering.
|
||||
Useful for telemetry filtering — AND it's the canonical sys2 marker the
|
||||
daily-budget query uses, because adopted trades have `trigger_post_id=NULL`
|
||||
so source-based classification fails (M1 fix). The `/trades` serializer also
|
||||
reports `trigger_source="adopted"` from this prefix.
|
||||
- **`/signals/accuracy` is scoped to production sources + buy/short.** It
|
||||
intentionally restricts to `SUPPORTED_TRADING_SOURCES` and the CURRENT
|
||||
buy/short vocabulary — retired/test sources (rsi_reversal, sma_reclaim,
|
||||
breakout, phase1, `test`) and the legacy `sell` signal are excluded so the
|
||||
public accuracy scoreboard reflects what the live bot actually trades.
|
||||
- **macro_enabled vs Telegram alerts are TWO separate switches.** Turning off
|
||||
Macro Vibes (`Subscription.macro_enabled`) gates sys2 *management* — it now
|
||||
blocks `/adopt` (macro_disabled). It does NOT silence Telegram alerts; those
|
||||
are controlled independently by the per-source `TelegramBinding` preference
|
||||
columns. "Alerts on, don't auto-manage" is a deliberately supported combo.
|
||||
- **`telegram.send_message` accepts `int | str` for `chat_id`.** Intentional.
|
||||
Integer = private chat, string = public channel username (e.g. `"@trumpalpha"`).
|
||||
- **`format_public_post` deliberately omits `expected_move_pct`,
|
||||
@@ -527,11 +573,29 @@ this for you" to "you opened it, bot manages your discipline".
|
||||
- **`/adopt` picker label** can show stale price if user waits >60s to tap
|
||||
(frozen BotTrade is always fresh, but the Telegram picker label may be outdated).
|
||||
|
||||
**Deferred security items (need interface change or DB migration):**
|
||||
- **C3**: Read endpoints pass `ts`/`sig` as URL query params → access log exposure.
|
||||
- **H4**: HL API key KEK uses single unsalted SHA-256. Needs re-encryption migration.
|
||||
- **M1**: Adopted sys2 positions miscounted against sys1 daily budget.
|
||||
- **M5**: `GET /telegram/{wallet}/status` unauthenticated — exposes chat_id/tg_username.
|
||||
- ~~**M1 adopted positions miscounted against sys1 budget**~~ **FIXED 2026-06-09**:
|
||||
the daily-budget query in `bot_engine` now treats any trade whose
|
||||
`hl_order_id` starts with `"adopted:"` as sys2, regardless of the NULL
|
||||
`trigger_post_id`. Previously the outerjoin to Post yielded src=NULL →
|
||||
classified as sys1 → every adopted macro position inflated the Trump
|
||||
scalp budget and could prematurely trip `budget_reached`.
|
||||
|
||||
- **Funding-reversal `/adopt` uses the btc_bottom_reversal exit profile**
|
||||
(`ADOPTED_CATEGORY` is fixed). This is BY DESIGN, not a bug: adopt is
|
||||
ASSET-based (the user opens any position on HL and adopts it) — the bot has
|
||||
no reliable link back to which signal motivated it, and the sys2 ladder is
|
||||
direction/horizon-agnostic. Changing this needs a source-tracking mechanism
|
||||
at adopt time (an ADR), not a one-line tweak.
|
||||
|
||||
**Deferred security items — ALL RESOLVED 2026-06-12:**
|
||||
- ~~**C3**~~ FIXED: signed reads now send `X-Sig-Ts` / `X-Sig-Sig` HEADERS
|
||||
(see `signed_read_creds` in signed_request.py). Legacy `?ts=&sig=` query
|
||||
params still accepted (deprecated) for old clients.
|
||||
- ~~**H4**~~ FIXED: keys now encrypt as `enc:v2` (PBKDF2-HMAC-SHA256, per-blob
|
||||
salt, 600k iters). v1 blobs still decrypt; run `scripts/reencrypt_keys.py`
|
||||
ONCE in prod (after DB backup) to upgrade stored rows.
|
||||
- ~~**M5**~~ Was already fixed: unauthenticated `/telegram/{wallet}/status`
|
||||
returns only `configured`/`bound` booleans; full details require a signed read.
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user