Webhook Reliability Monitoring#

If your endpoint returns 502 for ten minutes, the failed payment, the new order and the cancelled subscription are gone, and nothing in your logs says so. Assert posts to the endpoint on a schedule and tells you when it stops answering.

The Problem#

Four failures that leave no trace on your side:

• Missed events: the provider posts, your endpoint is down, the event is gone
• Processing failures: the handler returns 200 and drops the payload
• Queue buildup: events pile up behind a worker that stopped
• Retry exhaustion: the provider runs out of retries and stops trying

You call your own APIs, so you notice when they break. Nobody calls a webhook endpoint on your behalf, so the first sign is missing data days later.

How Assert Helps#

Monitor Endpoint Availability#

Check that your webhook endpoint answers:

Monitor: Stripe Webhook Endpoint
URL: POST /webhooks/stripe
Headers:
  Content-Type: application/json
Body: {"type": "ping"}
Assertions:
  ✓ Status code == 200 or 400
  ✓ Response time < 1000ms

Monitor Processing Health#

Reachable is not the same as working. If you expose a health endpoint, watch the queue behind it:

Monitor: Webhook Queue Health
URL: GET /api/webhooks/health
Assertions:
  ✓ $.queue_size < 100
  ✓ $.processing == true
  ✓ $.last_processed within 5 minutes
  ✓ $.error_rate < 0.05

Monitor Multiple Endpoints#

Different services, different endpoints:

Monitors:
  ✓ POST /webhooks/stripe
  ✓ POST /webhooks/github
  ✓ POST /webhooks/slack
  ✓ POST /webhooks/shopify

Common Webhook Sources#

Payment Providers#

E-commerce#

Developer Tools#

Communication#

Real-World Example#

The Scenario#

A marketplace takes order webhooks from Shopify. Its endpoint starts returning intermittent 502s, Shopify retries a few times and gives up, and the orders never land. Customers get their products; the marketplace has no record of the sale.

The Assert Solution#

Monitor 1: Webhook Endpoint

URL: POST /webhooks/shopify
Headers: X-Shopify-Topic: orders/create
Body: {"test": true}
Assertions:
  ✓ Status code == 200
  ✓ Response time < 2000ms
Interval: 1 minute

Monitor 2: Order Processing Health

URL: GET /api/orders/sync-status
Assertions:
  ✓ $.pending_webhooks < 50
  ✓ $.last_processed within 5 minutes
  ✓ $.success_rate > 0.99
Interval: 2 minutes

The Outcome#

The endpoint monitor caught the 502s inside a minute, and Assert alerted once with the failing response attached. The marketplace traced it to a memory leak in the webhook handler. The fix went out before the backlog grew and not one order went missing.

What to Assert#

Response Codes#

What your endpoint returns decides whether the provider retries:

✓ 200 — Event processed successfully
✓ 202 — Event accepted for processing
✓ 400 — Invalid payload (your code, not downtime)
✗ 500 — Server error (problem!)
✗ 502 — Gateway error (problem!)
✗ 503 — Service unavailable (problem!)

Response Time#

Webhook providers have timeout limits:

Set assertions below these thresholds:

Assertion: Response time < 3000ms

Health Metrics#

If you have a health endpoint:

// GET /api/webhooks/health
{
  "status": "healthy",
  "queue_size": 12,
  "processing_rate": 150,
  "error_rate": 0.001,
  "last_event": "2024-12-11T14:30:00Z"
}

Assertions:

• $.status equals "healthy"
• $.queue_size less than threshold
• $.error_rate less than 0.05
• $.last_event within expected timeframe

Best Practices#

Respond Fast, Process Later#

Verify the signature, queue the event, return 200. Do the work afterwards. Shopify gives you five seconds and doesn't care what your job queue is doing.

Watch the Queue Too#

A healthy endpoint and a stuck worker look identical from the outside:

Endpoint Up + Queue Growing = Problem
Endpoint Up + Queue Stable = Healthy

Use Signature Verification#

Most providers sign webhooks. Verify in your handler:

// Stripe example
const event = stripe.webhooks.constructEvent(
  body,
  signature,
  webhookSecret
);

Implement Idempotency#

Providers retry webhooks. Use event IDs so your handler skips what it has already processed:

if (await wasProcessed(event.id)) {
  return res.status(200).send('Already processed');
}

Log Everything#

The payload is gone once you've dropped it, so write down what arrived:

console.log('Webhook received:', {
  type: event.type,
  id: event.id,
  timestamp: new Date().toISOString()
});

Alert Configuration#

Assert opens one incident per failing monitor, attaches the response that failed, and closes it when the endpoint answers again.

Critical (Immediate)#

Condition: Endpoint returns 5xx
Action: Page on-call engineer
Reason: Events are being dropped

Warning (Slack)#

Condition: Response time > 2000ms
Action: Notify #engineering
Reason: Risk of timeout, needs investigation

Monitoring (Email)#

Condition: Queue size > threshold
Action: Email team
Reason: Processing may be falling behind

Getting Started#

List the services that send you events, then build one monitor per endpoint that posts a test payload to it. Assert on the response code and the response time, using the provider's own timeout as your ceiling. If you expose a health endpoint, add a second monitor for the queue behind it, and send the 5xx alerts to PagerDuty.

Related Use Cases#

• Payment API Monitoring: the payment side of those events
• Third-party API Monitoring: the vendor APIs sending them