KOL feeds: fix dead/blocked sources, drop stale feeds (29→25)
Feed-health pass over KOL_FEEDS: - raoulpal: stale Substack (last 2024-05) → Real Vision podcast feed - dampedspring: paywalled (0 entries) → free "Damped Spring 101" Substack - unchained: Cloudflare 403 → canonical Megaphone podcast feed - lynalden: Cloudflare 202 → FeedBurner mirror - glassnode: recovered via httpx http2=True (was 403 on HTTP/1.1) - browser User-Agent + Accept headers on feed fetch - removed dead feeds with no active replacement: placeholder, dragonfly, niccarter, eugene - pin h2==4.3.0 (required by http2=True) All 25 remaining feeds verified fetching real body content; newest post per feed ≤88d. Bundles in-flight KOL-module work already in the working tree (kol_x ingest, migration 027, tests). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -29,6 +29,7 @@ so a coalesced cron or worker restart can't double-send within a day.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import html
|
||||
import logging
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timedelta, timezone
|
||||
@@ -138,26 +139,43 @@ async def build_global_digest(now: Optional[datetime] = None) -> GlobalDigest:
|
||||
)
|
||||
|
||||
# ── KOL: last 24h posts + divergences ────────────────────────────
|
||||
# Exclude twitter noise (gm / RT / jokes) from the count. Substack
|
||||
# posts have tier=NULL (kept); twitter signal posts have tier!='noise'
|
||||
# (kept); twitter noise is dropped. Without this filter a KOL's gm/RT
|
||||
# spam inflates the daily "N KOL posts" stat. Mirrors the /kol/posts
|
||||
# signals_only filter and the frontend.
|
||||
kol_posts = (await db.execute(
|
||||
select(KolPost).where(KolPost.published_at >= cutoff_24h)
|
||||
select(KolPost).where(
|
||||
KolPost.published_at >= cutoff_24h,
|
||||
(KolPost.tier.is_(None)) | (KolPost.tier != "noise"),
|
||||
)
|
||||
)).scalars().all()
|
||||
# action lives inside tickers_json; easier: classify on summary text.
|
||||
# The structured action is on KolDivergence rows; for the bare count
|
||||
# we just split posts by whether divergence rows reference them.
|
||||
# Count bullish/bearish/neutral at POST level, not divergence-pair level.
|
||||
# A single post can match multiple on-chain events (one per ticker/wallet),
|
||||
# producing several KolDivergence rows. Counting rows would inflate the
|
||||
# numbers — a post mentioning 3 tickers was previously counted 3× (bug).
|
||||
bullish = bearish = neutral = 0
|
||||
divergence_rows = (await db.execute(
|
||||
select(KolDivergence)
|
||||
.where(KolDivergence.post_at >= cutoff_24h)
|
||||
.order_by(KolDivergence.created_at.desc())
|
||||
)).scalars().all()
|
||||
# Deduplicate by post_id; for posts with mixed directions take "long"
|
||||
# first (a bullish call that also hedges short is net bullish).
|
||||
post_direction: dict[int, str] = {}
|
||||
for d in divergence_rows:
|
||||
if d.direction == "long":
|
||||
if d.post_id not in post_direction:
|
||||
post_direction[d.post_id] = d.direction
|
||||
elif d.direction == "long":
|
||||
post_direction[d.post_id] = "long" # long overrides neutral
|
||||
for direction in post_direction.values():
|
||||
if direction == "long":
|
||||
bullish += 1
|
||||
elif d.direction == "short":
|
||||
elif direction == "short":
|
||||
bearish += 1
|
||||
else:
|
||||
neutral += 1
|
||||
# Posts not yet classified into divergences just count as neutral.
|
||||
# Posts not yet cross-referenced with on-chain data count as neutral.
|
||||
neutral += max(0, len(kol_posts) - (bullish + bearish + neutral))
|
||||
|
||||
divergences_24h = sum(
|
||||
@@ -168,7 +186,9 @@ async def build_global_digest(now: Optional[datetime] = None) -> GlobalDigest:
|
||||
if d.signal_type != "divergence":
|
||||
continue
|
||||
# "Hayes publicly bullish BTC, on-chain selling"
|
||||
sample_divergence = (
|
||||
# Escape — ticker is AI-extracted (kol_analysis) so not guaranteed
|
||||
# HTML-safe; the digest is sent with parse_mode=HTML.
|
||||
sample_divergence = html.escape(
|
||||
f"{d.handle} publicly {d.post_action} {d.ticker}, "
|
||||
f"on-chain {d.onchain_action}"
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user