KOL feeds: fix dead/blocked sources, drop stale feeds (29→25)
Feed-health pass over KOL_FEEDS: - raoulpal: stale Substack (last 2024-05) → Real Vision podcast feed - dampedspring: paywalled (0 entries) → free "Damped Spring 101" Substack - unchained: Cloudflare 403 → canonical Megaphone podcast feed - lynalden: Cloudflare 202 → FeedBurner mirror - glassnode: recovered via httpx http2=True (was 403 on HTTP/1.1) - browser User-Agent + Accept headers on feed fetch - removed dead feeds with no active replacement: placeholder, dragonfly, niccarter, eugene - pin h2==4.3.0 (required by http2=True) All 25 remaining feeds verified fetching real body content; newest post per feed ≤88d. Bundles in-flight KOL-module work already in the working tree (kol_x ingest, migration 027, tests). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
+17
-2
@@ -31,6 +31,7 @@ from typing import Optional
|
||||
from fastapi import APIRouter, Depends, Header, HTTPException
|
||||
from pydantic import BaseModel, Field, field_validator
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.config import settings
|
||||
@@ -174,8 +175,22 @@ async def ingest_signal(
|
||||
prefilter_reason="external_signal", # bypasses entry-filter audit
|
||||
)
|
||||
db.add(post)
|
||||
await db.commit()
|
||||
await db.refresh(post)
|
||||
try:
|
||||
await db.commit()
|
||||
await db.refresh(post)
|
||||
except IntegrityError:
|
||||
# Concurrent request beat us to the INSERT — fetch the existing row.
|
||||
await db.rollback()
|
||||
existing2 = await db.execute(select(Post).where(Post.external_id == ext_id_hashed))
|
||||
prior2 = existing2.scalar_one_or_none()
|
||||
if prior2:
|
||||
return SignalIngestResponse(
|
||||
status="duplicate",
|
||||
post_id=prior2.id,
|
||||
dedup_against=prior2.id,
|
||||
note=f"concurrent ingest: external_id {body.external_id!r} already exists",
|
||||
)
|
||||
raise # unexpected — re-raise if we still can't find the row
|
||||
|
||||
logger.info(
|
||||
"Ingested signal: source=%s id=%s → post_id=%d, %s/%s conf=%d category=%s",
|
||||
|
||||
Reference in New Issue
Block a user