KOL feeds: fix dead/blocked sources, drop stale feeds (29→25)

Feed-health pass over KOL_FEEDS:
- raoulpal: stale Substack (last 2024-05) → Real Vision podcast feed
- dampedspring: paywalled (0 entries) → free "Damped Spring 101" Substack
- unchained: Cloudflare 403 → canonical Megaphone podcast feed
- lynalden: Cloudflare 202 → FeedBurner mirror
- glassnode: recovered via httpx http2=True (was 403 on HTTP/1.1)
- browser User-Agent + Accept headers on feed fetch
- removed dead feeds with no active replacement: placeholder,
  dragonfly, niccarter, eugene
- pin h2==4.3.0 (required by http2=True)

All 25 remaining feeds verified fetching real body content; newest
post per feed ≤88d. Bundles in-flight KOL-module work already in the
working tree (kol_x ingest, migration 027, tests).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
k
2026-06-09 22:55:16 +08:00
parent 213bb911e3
commit 54884f3e24
38 changed files with 2340 additions and 322 deletions
+28 -8
View File
@@ -29,6 +29,7 @@ so a coalesced cron or worker restart can't double-send within a day.
from __future__ import annotations
import html
import logging
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
@@ -138,26 +139,43 @@ async def build_global_digest(now: Optional[datetime] = None) -> GlobalDigest:
)
# ── KOL: last 24h posts + divergences ────────────────────────────
# Exclude twitter noise (gm / RT / jokes) from the count. Substack
# posts have tier=NULL (kept); twitter signal posts have tier!='noise'
# (kept); twitter noise is dropped. Without this filter a KOL's gm/RT
# spam inflates the daily "N KOL posts" stat. Mirrors the /kol/posts
# signals_only filter and the frontend.
kol_posts = (await db.execute(
select(KolPost).where(KolPost.published_at >= cutoff_24h)
select(KolPost).where(
KolPost.published_at >= cutoff_24h,
(KolPost.tier.is_(None)) | (KolPost.tier != "noise"),
)
)).scalars().all()
# action lives inside tickers_json; easier: classify on summary text.
# The structured action is on KolDivergence rows; for the bare count
# we just split posts by whether divergence rows reference them.
# Count bullish/bearish/neutral at POST level, not divergence-pair level.
# A single post can match multiple on-chain events (one per ticker/wallet),
# producing several KolDivergence rows. Counting rows would inflate the
# numbers — a post mentioning 3 tickers was previously counted 3× (bug).
bullish = bearish = neutral = 0
divergence_rows = (await db.execute(
select(KolDivergence)
.where(KolDivergence.post_at >= cutoff_24h)
.order_by(KolDivergence.created_at.desc())
)).scalars().all()
# Deduplicate by post_id; for posts with mixed directions take "long"
# first (a bullish call that also hedges short is net bullish).
post_direction: dict[int, str] = {}
for d in divergence_rows:
if d.direction == "long":
if d.post_id not in post_direction:
post_direction[d.post_id] = d.direction
elif d.direction == "long":
post_direction[d.post_id] = "long" # long overrides neutral
for direction in post_direction.values():
if direction == "long":
bullish += 1
elif d.direction == "short":
elif direction == "short":
bearish += 1
else:
neutral += 1
# Posts not yet classified into divergences just count as neutral.
# Posts not yet cross-referenced with on-chain data count as neutral.
neutral += max(0, len(kol_posts) - (bullish + bearish + neutral))
divergences_24h = sum(
@@ -168,7 +186,9 @@ async def build_global_digest(now: Optional[datetime] = None) -> GlobalDigest:
if d.signal_type != "divergence":
continue
# "Hayes publicly bullish BTC, on-chain selling"
sample_divergence = (
# Escape — ticker is AI-extracted (kol_analysis) so not guaranteed
# HTML-safe; the digest is sent with parse_mode=HTML.
sample_divergence = html.escape(
f"{d.handle} publicly {d.post_action} {d.ticker}, "
f"on-chain {d.onchain_action}"
)