Your scheduled jobs are failing silently: monitoring webhook deliveries

Ask a candid backend engineer whether last night’s scheduled jobs ran, and you’ll often get a shrug. Classic cron is the least observable thing in most stacks: it appends to a syslog nobody reads, exits silently when your script has a bug, and has no concept of “failure” beyond a nonzero exit code nobody is subscribed to.

Scheduled webhooks change the situation structurally — because an HTTP call leaves a much richer trail than a shell script.

The delivery record is the audit trail

Every scheduled delivery in Recurrence Cloud is recorded in the activity trail:

• Timestamp of every attempt (first fire and every retry)
• HTTP status code your receiver returned
• Response body, truncated for inspection
• Duration / latency of the call
• Retry status — pending, exhausted, or resolved

“Did the 3am sync run?” becomes a query, not an archaeology dig. You can view the activity per webhook URL via the dashboard or pull it via the API:

curl https://api.recurrence.cloud/v1/webhooks/<id>/activity \
  -H "Authorization: Bearer $API_TOKEN"

Retries with receipts

A delivery that gets a non-2xx (or times out) is retried on a backoff schedule until it succeeds or exhausts its attempts. Crucially, the record of each attempt persists — so a job that ultimately succeeded on the third try still leaves evidence in the trail that the first two attempts failed with a 500 and a connection reset.

That distinction matters for humans (“was this just a blip?” — now you know) and for alerting: a single transient failure and a chronic 20-minute outage look identical in a health dashboard but nothing alike in an activity trail.

Wire the delivery stream into your existing metrics stack — integrations for tools like Sentry and Datadog let scheduled failures land where your team already works:

• Failure-rate spikes per webhook URL
• p95 latency of receiver endpoints by schedule
• Retry exhaustion counts as the “job never completed” metric

Receiver-side discipline

The trail is only as truthful as your status codes. Three rules:

1. Acknowledge only completed work. Return 202 for accepted-and-queued, 500 for genuine failure — never a blanket 200 at the top of the handler.
2. Idempotency keys. Retries will happen; make redeliveries safe by keying on the event identity.
3. Fail loudly inside, quietly outside. Catch, log with context (schedule name, subject, event id), and return a real error — so the trail captures what your code knew.

A minimal honest handler:

app.post("/jobs", async (c) => {
  const claims = await verifyToken(c.req); // 401 on bad signature
  try {
    await processBatch(claims.event);
    return c.json({ ok: true });
  } catch (err) {
    logger.error({ err, subject: claims.event.subject }, "job failed");
    return c.json({ ok: false }, 500); // schedules a retry
  }
});

A monitoring checklist for scheduled work

• Every schedule has a named owner and appears in the activity trail
• Alerts fire on retry exhaustion, not on first failure
• Delivery latency is tracked per webhook URL (receivers degrading slowly is a classic pre-outage signal)
• A runbook links from each schedule: what its payload means, what to do when it fails

Reliability is the payoff of the pattern; verify the inbound side and batch intelligently to keep the volume manageable. For the full tour, see the scheduled webhook guide.