# Backend deployment + uptime monitoring

This folder collects the moving pieces that turn the backend from "runs on
my laptop" into "runs 24/7 with auto-restart and external alerting".

## What the bot needs to be reliable

1. **Auto-restart on crash** — handled by systemd (`Restart=always`) or
   Docker Compose (`restart: unless-stopped`). Pick one.
2. **Health endpoints** — exposed by `app/main.py`:
   - `GET /api/health` (shallow): 200 as long as the FastAPI process is alive.
     Used by Docker's internal `HEALTHCHECK`.
   - `GET /api/health/deep`: 200 only if DB ping works AND at least one
     Truth Social scraper has polled in the last 90s. Used by external
     uptime monitors.
3. **Dual-source scraper** — `truth_social.py` (CNN archive) +
   `trumpstruth.py` (RSS). Both insert into `posts` with the same
   `external_id` hash, so duplicates are dropped. Whoever sees the post
   first wins.
4. **External uptime monitor** — pings the deep health endpoint from
   outside the box. If we lose internet or the box is on fire, we get
   alerted. Recommended: UptimeRobot (free tier covers this).

---

## Option A — Docker Compose (recommended)

`docker-compose.yml` is already configured with:

```yaml
restart: unless-stopped
healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:8000/api/health"]
  interval: 15s
  timeout: 5s
  retries: 5
  start_period: 30s
```

Deploy steps on a fresh server:

```bash
git clone <repo>
cd backend
cp .env.example .env
# Fill in: SERVICE_USER_POSTGRES, SERVICE_PASSWORD_POSTGRES, FRONTEND_URL,
#          ENCRYPTION_KEY, AI_API_KEY, HL_MAINNET=true
docker compose up -d
docker compose logs -f api    # tail logs
```

To verify auto-restart works:

```bash
docker compose kill api       # simulate crash
sleep 12                      # wait for compose to relaunch
curl http://localhost:8000/api/health   # should return {"status":"ok"}
```

---

## Option B — systemd (bare-metal, no Docker)

Use `trumpsignal-api.service` from this folder.

```bash
# 1. Code lives at /opt/trumpsignal/backend
sudo useradd -m -d /opt/trumpsignal trumpsignal
sudo -u trumpsignal git clone <repo> /opt/trumpsignal/backend
cd /opt/trumpsignal/backend
sudo -u trumpsignal python3.11 -m venv venv
sudo -u trumpsignal venv/bin/pip install -r requirements.txt

# 2. Configure
sudo -u trumpsignal cp .env.example .env
sudo -u trumpsignal nano .env       # fill in secrets

# 3. Install + start the service
sudo cp deploy/trumpsignal-api.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now trumpsignal-api

# 4. Tail logs
sudo journalctl -u trumpsignal-api -f
```

Test auto-restart:

```bash
sudo systemctl kill --signal=SIGKILL trumpsignal-api
sleep 12
curl http://localhost:8000/api/health   # should return {"status":"ok"}
```

---

## UptimeRobot setup

1. Sign up at https://uptimerobot.com (free tier = 50 monitors, 5-min interval).
2. Add new monitor:
   - **Type:** HTTP(s)
   - **URL:** `https://api.trump-signal.bitnews.day/api/health/deep`
     (replace with your real domain)
   - **Monitoring interval:** 5 min
   - **Alert contacts:** add your email and/or Telegram
3. UptimeRobot will alert you when the deep healthcheck returns 503 — i.e.
   when both scrapers have been silent for >90s, OR when the DB is
   unreachable.

### Why deep, not shallow

The shallow `/api/health` returns 200 even if the scraper has died but the
FastAPI process is still serving HTTP. A monitor pointed at the shallow
endpoint would happily report "all green" while the bot silently misses
posts. Always point external monitors at `/api/health/deep`.

---

## Reading the deep healthcheck

```json
{
  "status": "ok",
  "now": "2026-04-25T07:35:01.123456+00:00",
  "db_ok": true,
  "db_error": null,
  "scrapers": [
    { "name": "cnn",         "last_poll": "2026-04-25T07:34:58Z", "age_sec": 3,  "last_error": null },
    { "name": "trumpstruth", "last_poll": "2026-04-25T07:34:52Z", "age_sec": 9,  "last_error": null }
  ],
  "freshest_age_sec": 3,
  "problems": []
}
```

`status: "degraded"` + HTTP 503 means at least one item in `problems` is
non-empty. The most common failure modes:

- `db: <error>` — Postgres/SQLite can't be reached. Check `DATABASE_URL`.
- `scrapers: all stale (freshest=Ns)` — both upstream feeds silent. Check
  network egress, then check if both `ix.cnn.io` and `trumpstruth.org` are
  responding from the box: `curl -sI https://ix.cnn.io/data/truth-social/truth_archive.json`

---

## Burst-protection note (systemd)

The unit file caps restarts at 5 within 60s, so a hard misconfiguration
(missing env var, etc.) stops the boot loop instead of thrashing forever.
After fixing the underlying issue:

```bash
sudo systemctl reset-failed trumpsignal-api
sudo systemctl start trumpsignal-api
```