Recurring tasks — heartbeats + the system-health dashboard

Every cron, polling loop, scheduled retry, or other repeating background task on this platform reports a heartbeat to a single system_heartbeats table on every fire. The maintenance dashboard at /dashboard/admin/system-health reads those heartbeats and flags any task that's stale, errored, or never fired.

This is the contract: if a task runs on a cadence, it must record a heartbeat. Otherwise it's invisible to ops and silently dying is the most likely failure mode.

Why this matters

The Node process can be alive while the cron loop has died — an unhandled exception inside setInterval, an await import() that started failing after a deploy, or a Promise chain that swallowed its own error. From outside, the box looks healthy: HTTP /health returns 200, memory is stable. But work isn't getting done.

Heartbeats catch this without external monitoring infrastructure. The dashboard reads last_run_at, compares to expected_interval_seconds, and flags > 1.5× as yellow, > 3× as red. A task that hasn't fired in 45 minutes when it should fire every 15 = red, regardless of whether the rest of the platform looks fine.

The contract

For any task that fires on a cadence:

1. Call recordHeartbeat() (or wrap in withHeartbeat()) on every fire — both success AND error paths.
2. Use a stable, unique name — snake_case, namespaced if helpful (e.g. cafe_contest_tick, welcome_drip_tick, season_auto_render_scan).
3. Pass expectedIntervalSeconds — what cadence are you firing on? The dashboard staleness check needs this.
4. Surface useful counters in metadata — they appear under "Show metadata" in the dashboard for click-through inspection (e.g. { transitioned: 2, emailed: 5, errors: 0 }).
5. Add a friendly label to CRON_LABELS in apps/backend/src/services/system-health.ts so the dashboard renders "Cafe contest cron (15m)" instead of cafe_contest_tick.

That's it. The dashboard picks up the new task automatically once it fires once.

Two patterns

Pattern A — manual control (use when the task has nuanced statuses)

import { recordHeartbeat } from './services/system-heartbeats.js'

const INTERVAL_MS = 15 * 60 * 1000

setInterval(async () => {
  let result: any = null
  let errMsg: string | null = null
  try {
    result = await runTheTaskLogic()
  } catch (e: any) {
    errMsg = e?.message || String(e)
    console.warn('[my-task] failed:', errMsg)
  }
  await recordHeartbeat({
    name: 'my_task_tick',
    status: errMsg
      ? 'error'
      : (result.errors?.length ? 'partial' : 'ok'),
    error: errMsg || result?.errors?.[0]?.message || null,
    expectedIntervalSeconds: INTERVAL_MS / 1000,
    metadata: result
      ? { processed: result.count, errors: result.errors?.length || 0 }
      : undefined,
  })
}, INTERVAL_MS)

The 'partial' status is for tasks that completed but had per-item errors (e.g. emailed 48 of 50 users with 2 Resend hiccups). Yellow on the dashboard.

Pattern B — withHeartbeat() wrapper (use when the task is a simple ok-or-throw)

import { withHeartbeat } from './services/system-heartbeats.js'

setInterval(async () => {
  try {
    await withHeartbeat(
      { name: 'my_task_tick', expectedIntervalSeconds: 900 },
      async () => {
        const result = await runTheTaskLogic()
        return { processed: result.count }  // surfaces as metadata
      },
    )
  } catch {
    // withHeartbeat already recorded the error and re-threw; outer
    // catch is just to keep setInterval running.
  }
}, 15 * 60 * 1000)

Use this for the simpler case — it can't forget the error path.

Adding new check types

Beyond cron heartbeats, apps/backend/src/services/system-health.ts supports three other check categories. To add a new one:

External service probe

Add to checkExternalServices(). Conventions:

• Yellow when env vars are unset for a feature-flagged service (e.g. Bluesky without creds — that's "off as expected", not broken).
• Green when probe succeeds, red when it fails with creds set.
• 5-second timeout via probeUrl() so a hanging service can't stall the dashboard.
• Use the cheapest auth'd endpoint the service exposes — Stripe /v1/balance, OpenAI /v1/models, Resend /v1/domains, etc. Don't burn quota.

Data-state check

Add to checkDataState(). Conventions:

• Read-only queries. No writes ever from the dashboard.
• Time-aware: "this week's contest exists?" makes sense on Monday 10am, may be expected-missing on Sunday 11pm. Use unknown (gray) for "not enough info to judge" rather than red.
• Single targeted query, not a sweep. The dashboard runs every check on every page load; broad scans don't belong here.

Config-presence panel

Add to REQUIRED_ENV in checkConfigPresence(). Each entry is { name, label, severity } where severity is what the missing var produces (red for unrecoverable, yellow for "feature is off"). Lighter than a probe — just env-presence.

When NOT to use this

• One-shot jobs (a migration script, a backfill you ran once). Heartbeats are for repeating work.
• Per-request work (an HTTP route's handler logic). That's monitored at the request level, not by heartbeat staleness.
• Sub-second loops (e.g. a polling loop inside a single request). Heartbeats are for tasks where "didn't fire for an hour" is meaningful.

Existing tasks (as of 2026-05-08)

In addition to true cron heartbeats, system_heartbeats is also used by some services as a key-value persistence mechanism for ephemeral state that needs to survive process restarts:

These rows write last_status: 'ok' purely to satisfy the table's check constraint — the dashboard's cron-staleness check ignores them because they aren't in the CRON_LABELS map. Keeping them in the same table avoids a separate system_state migration.

When you add a new recurring task, add a row to the cron table above so the next contributor knows what's already wired.