KOL feeds: fix dead/blocked sources, drop stale feeds (29→25)
Feed-health pass over KOL_FEEDS: - raoulpal: stale Substack (last 2024-05) → Real Vision podcast feed - dampedspring: paywalled (0 entries) → free "Damped Spring 101" Substack - unchained: Cloudflare 403 → canonical Megaphone podcast feed - lynalden: Cloudflare 202 → FeedBurner mirror - glassnode: recovered via httpx http2=True (was 403 on HTTP/1.1) - browser User-Agent + Accept headers on feed fetch - removed dead feeds with no active replacement: placeholder, dragonfly, niccarter, eugene - pin h2==4.3.0 (required by http2=True) All 25 remaining feeds verified fetching real body content; newest post per feed ≤88d. Bundles in-flight KOL-module work already in the working tree (kol_x ingest, migration 027, tests). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
k
@@ -43,6 +43,16 @@ logger = logging.getLogger(__name__)
|
||||
# of body per post or it just hallucinates a topic line. (Headlines-only
|
||||
# feeds like Vitalik's blog need a follow-up HTML fetch, deferred.)
|
||||
# 3. Add with a sensible handle + display_name.
|
||||
# 4. ⚠️ The KOL feed COUNT is hardcoded in the FRONTEND for SEO/marketing
|
||||
# (it can't read this list cross-repo). If len(KOL_FEEDS) changes, update
|
||||
# every "N KOL feeds" mention in the frontend repo:
|
||||
# app/layout.tsx (JSON-LD ×2), app/page.tsx (×3 incl. metric),
|
||||
# app/[locale]/kol/page.tsx (×4), app/[locale]/kol/KolPageClient.tsx
|
||||
# (subtitle "and N more" = count-3), app/[locale]/glossary/page.tsx,
|
||||
# public/llms.txt + llms-full.txt, app/opengraph-image.tsx.
|
||||
# Currently len(KOL_FEEDS) == 25. (X-only KOLs live in kol_x.X_KOLS.)
|
||||
# (2026-06-09: dropped from 29 → 25; removed placeholder, dragonfly,
|
||||
# niccarter, eugene as dead feeds. Frontend count mentions need updating.)
|
||||
KOL_FEEDS: list[dict] = [
|
||||
# ── Substack essayists (long-form thesis pieces) ─────────────────────
|
||||
{
|
||||
@@ -50,38 +60,40 @@ KOL_FEEDS: list[dict] = [
|
||||
"display_name": "Arthur Hayes",
|
||||
"feed_url": "https://cryptohayes.substack.com/feed",
|
||||
},
|
||||
# Placeholder VC (Joel Monegro / Chris Burniske). Token-focused VC, posts
|
||||
# long-form thesis pieces every 1-3 months that map directly to their
|
||||
# portfolio bets (Solana staking, L1 monetary premium, etc.).
|
||||
# Raoul Pal — Real Vision / Global Macro Investor founder. "Short Excerpts
|
||||
# Raoul Pal — his public Substack (raoulpal.substack.com/feed) went STALE
|
||||
# (last post 2024-05). Replaced 2026-06-09 with the Real Vision official
|
||||
# podcast feed (feeds.megaphone.fm/realvision) — daily, free, 2000+ eps,
|
||||
# full episode descriptions with macro/crypto thesis. It's the whole Real
|
||||
# Vision channel (not Raoul-only), but high signal + fresh. Same canonical
|
||||
# handle as his X (@RaoulGMI in kol_x) so long-form + real-time aggregate.
|
||||
{
|
||||
"handle": "placeholder",
|
||||
"display_name": "Placeholder VC",
|
||||
"feed_url": "https://www.placeholder.vc/blog?format=rss",
|
||||
"handle": "raoulpal",
|
||||
"display_name": "Raoul Pal (Real Vision)",
|
||||
"feed_url": "https://feeds.megaphone.fm/realvision",
|
||||
"source": "podcast",
|
||||
},
|
||||
# Dragonfly Capital research blog on Medium — free, active (10+ posts).
|
||||
# dragonfly.xyz/blog/rss.xml returns 0 (paywall). medium.com/dragonfly-research
|
||||
# is the team's public research arm: airdrops, DeFi, protocol deep-dives.
|
||||
{
|
||||
"handle": "dragonfly",
|
||||
"display_name": "Dragonfly Capital",
|
||||
"feed_url": "https://medium.com/feed/dragonfly-research",
|
||||
},
|
||||
# Andy Constan's Substack is paywalled (RSS returns 0). Keeping for any
|
||||
# occasional public teaser. Forward Guidance podcast (Blockworks) features
|
||||
# him weekly but is macro/equities-focused — not crypto-coin-specific enough
|
||||
# to extract ticker signals from episode descriptions.
|
||||
# REMOVED 2026-06-09 — dead feeds, no active replacement exists:
|
||||
# • placeholder (Placeholder VC) — last post 2025-09, blog is their only
|
||||
# public source, no newsletter. ~266d stale.
|
||||
# • dragonfly (Dragonfly Research, Medium) — last post 2025-03 (~454d).
|
||||
# dragonfly.xyz has no working RSS (JS-rendered, all endpoints empty);
|
||||
# Haseeb's medium/@hosseeb is even older (2024).
|
||||
# If either resumes regular publishing with a real RSS, re-add here.
|
||||
# Andy Constan — his old Substack (dampedspring.substack.com) returned 0
|
||||
# entries (paywalled). Replaced 2026-06-09 with his FREE long-form Substack
|
||||
# "Damped Spring 101" (dampedspring101.substack.com), which he's publicly
|
||||
# committed to keeping free — active, real macro essays (liquidity, rates,
|
||||
# positioning). His deepest paid research stays gated, but this carries
|
||||
# genuine thesis content suitable for ticker/direction extraction.
|
||||
{
|
||||
"handle": "dampedspring",
|
||||
"display_name": "Damped Spring / Andy Constan",
|
||||
"feed_url": "https://dampedspring.substack.com/feed",
|
||||
},
|
||||
# Nic Carter's Substack is paywalled (RSS returns 0). His Medium feed is
|
||||
# FREE and active — different URL, same author, real content.
|
||||
{
|
||||
"handle": "niccarter",
|
||||
"display_name": "Nic Carter (Castle Island)",
|
||||
"feed_url": "https://medium.com/feed/@nic__carter",
|
||||
"feed_url": "https://dampedspring101.substack.com/feed",
|
||||
},
|
||||
# REMOVED 2026-06-09 — niccarter (Nic Carter / Castle Island). Substack
|
||||
# paywalled (0 entries); Medium feed went stale (last 2025-07, ~316d).
|
||||
# No active free replacement. Re-add if he resumes a public RSS.
|
||||
# Delphi Digital podcast (Buzzsprout) — 478 episodes, active May 2025.
|
||||
# Public, free. Episode descriptions name specific protocols / tokens with
|
||||
# thesis framing — good extraction signal. delphidigital.io/feed returns 0.
|
||||
@@ -104,12 +116,8 @@ KOL_FEEDS: list[dict] = [
|
||||
"display_name": "The DeFi Edge",
|
||||
"feed_url": "https://thedefiedge.com/feed/",
|
||||
},
|
||||
# Eugene Ng Ah Sio — trader/analyst, sporadic but specific.
|
||||
{
|
||||
"handle": "eugene",
|
||||
"display_name": "Eugene Ng Ah Sio",
|
||||
"feed_url": "https://eugene.substack.com/feed",
|
||||
},
|
||||
# REMOVED 2026-06-09 — eugene (Eugene Ng Ah Sio). Substack effectively
|
||||
# dormant: only 5 entries, last 2025-05 (~399d). No alternative source.
|
||||
# ── DeFi journalism (Substack-style RSS) ─────────────────────────────
|
||||
# The Defiant — Camila Russo's team. DeFi-focused news with frequent
|
||||
# protocol + token mentions. Free RSS, ~100 entries.
|
||||
@@ -149,12 +157,15 @@ KOL_FEEDS: list[dict] = [
|
||||
"feed_url": "https://feeds.megaphone.fm/lightspeed",
|
||||
"source": "podcast",
|
||||
},
|
||||
# Unchained — Laura Shin. Long interview format with founders and
|
||||
# traders. Show notes are 6K+ chars (near-transcript).
|
||||
# Unchained — Laura Shin. Long interview format with founders and traders.
|
||||
# The website RSS (unchainedcrypto.com/feed) is Cloudflare-blocked (403,
|
||||
# even HTTP/2 + browser UA). Switched 2026-06-09 to the canonical Megaphone
|
||||
# podcast feed (resolved via iTunes lookup id=1123922160) — 1100+ episodes,
|
||||
# active daily, 1.5-2K char show notes naming protocols/tokens.
|
||||
{
|
||||
"handle": "unchained",
|
||||
"display_name": "Unchained (Laura Shin)",
|
||||
"feed_url": "https://www.unchainedcrypto.com/feed/",
|
||||
"feed_url": "https://feeds.megaphone.fm/LSHML4761942757",
|
||||
"source": "podcast",
|
||||
},
|
||||
# Bankless podcast — Ryan Sean Adams + David Hoffman. ETH-focused but
|
||||
@@ -240,8 +251,12 @@ KOL_FEEDS: list[dict] = [
|
||||
# but extremely high quality. Mostly BTC-only signals.
|
||||
{
|
||||
"handle": "lynalden",
|
||||
# lynalden.com/feed is Cloudflare-blocked (202, no body). Switched
|
||||
# 2026-06-09 to her FeedBurner mirror, which serves fine. Note: bodies
|
||||
# are article EXCERPTS (~400 chars) not full text, but enough for the
|
||||
# AI to extract the BTC/macro thesis + direction.
|
||||
"display_name": "Lyn Alden",
|
||||
"feed_url": "https://www.lynalden.com/feed/",
|
||||
"feed_url": "https://feeds.feedburner.com/lynalden",
|
||||
"source": "blog",
|
||||
},
|
||||
# Bitcoin Magazine — high-frequency news + institutional adoption analysis.
|
||||
@@ -336,8 +351,19 @@ async def _fetch_feed(feed_url: str) -> list:
|
||||
"""feedparser is sync; do the HTTP fetch through httpx for timeout
|
||||
control + uniformity with the rest of the codebase, then hand bytes
|
||||
to feedparser."""
|
||||
async with httpx.AsyncClient(timeout=20.0, follow_redirects=True) as client:
|
||||
r = await client.get(feed_url, headers={"User-Agent": "TrumpSignal/1.0 KOL-tracker"})
|
||||
# http2=True matters: some CDN-fronted feeds (e.g. Glassnode) return 403
|
||||
# to HTTP/1.1 requests but serve fine over HTTP/2 (curl defaults to h2,
|
||||
# which is why they worked manually but not here). Requires the `h2` pkg.
|
||||
async with httpx.AsyncClient(timeout=20.0, follow_redirects=True, http2=True) as client:
|
||||
# Use a real browser UA. Several feeds (Glassnode, others behind a CDN)
|
||||
# return 403/202 to non-browser agents. A plain bot UA was silently
|
||||
# losing those feeds. This recovers them without per-feed special-casing.
|
||||
r = await client.get(feed_url, headers={
|
||||
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
|
||||
"AppleWebKit/537.36 (KHTML, like Gecko) "
|
||||
"Chrome/124.0 Safari/537.36",
|
||||
"Accept": "application/rss+xml, application/atom+xml, application/xml, text/xml, */*",
|
||||
})
|
||||
r.raise_for_status()
|
||||
parsed = feedparser.parse(r.content)
|
||||
return list(parsed.entries or [])
|
||||
@@ -439,6 +465,9 @@ async def _ingest_kol(
|
||||
row.analyzed_at = utcnow()
|
||||
row.analysis_model = result.get("model")
|
||||
row.analysis_version = result.get("version")
|
||||
# Extended analysis fields (migration 027)
|
||||
row.post_type = result.get("post_type")
|
||||
row.talks_vs_trades_flag = bool(result.get("talks_vs_trades_flag", False))
|
||||
stats["analyzed"] += 1
|
||||
except Exception as e:
|
||||
logger.warning("[kol_substack] analysis failed for %s post %s: %s",
|
||||
@@ -452,11 +481,15 @@ async def _ingest_kol(
|
||||
async def run_substack_poll(*, analyze: bool = True) -> list[dict]:
|
||||
"""Poll every configured KOL feed once. Despite the legacy name this now
|
||||
covers Substack essays, Medium blogs, and major crypto podcasts via RSS.
|
||||
Returns per-KOL stats."""
|
||||
Returns per-KOL stats.
|
||||
|
||||
Each KOL gets its own session so a commit failure for one does not leave
|
||||
a dirty session that breaks subsequent KOLs in the same run.
|
||||
"""
|
||||
results = []
|
||||
async with AsyncSessionLocal() as session:
|
||||
for kol in KOL_FEEDS:
|
||||
for kol in KOL_FEEDS:
|
||||
async with AsyncSessionLocal() as session:
|
||||
stats = await _ingest_kol(session, kol, analyze=analyze)
|
||||
results.append(stats)
|
||||
results.append(stats)
|
||||
logger.info("[kol_substack] poll done: %s", results)
|
||||
return results
|
||||
|
||||
Reference in New Issue
Block a user