Notify

CampusCritique needed a dozen kinds of notification the week Connect launched: booking confirmed, reminder two hours before, session rescheduled, refund processed, review ready, payout sent. Each had to reach a student and a mentor, in-app, by email and by push, without a slow email ever failing a payment webhook.

The first version lived inside the Next.js app: an email call at the end of each API route. It had the obvious problems. A Resend timeout inside the Cashfree webhook handler meant the gateway saw a 500 and retried the payment callback. Nobody could say whether a given reminder had gone out, because the only record was a console line on Vercel. And every new notification meant another route learning about SMTP, retry loops and HTML templates.

So I pulled it out into its own service, built as a product rather than a helper: any application posts an event, Notify decides who gets what on which channel, and every attempt is a row you can query. It has run in production since 18 May 2026 and now serves two tenants, CampusCritique and this site’s contact form.

Package com.npaas.notify, nine tables (tenants, tenant_api_keys, notification_events, notification_rules, notification_templates, notification_jobs, notification_delivery_attempts, in_app_notifications, push_subscriptions), six controllers under /api/v1 (events, jobs, templates, in-app notifications, push subscriptions, metrics), eleven test classes covering the renderer, job creation, delivery, recovery, the request-size filter and the API-key filter.

Ingest. A product authenticates with a tenant-scoped API key sent as X-Notify-Api-Key. The key is stored as a SHA-256 hash; the raw key is printed exactly once by an internal CLI (admin:create-api-key) and never again. The request carries an event type, a recipient, a payload and an idempotency key. If the same tenant sends the same idempotency key twice, the second call returns the first event with duplicate: true and nothing else happens.

Commit, then publish. The event row is written with status RECEIVED, marked QUEUED, and the transaction commits. Only then, in an afterCommit hook, is the event published to the notify.events exchange. If the broker is down the publish fails quietly, the client still gets its 202, and a recovery scheduler republishes any event still QUEUED after 120 seconds. There is no distributed transaction anywhere in the service.

Fan out. A RabbitMQ consumer looks up the tenant’s enabled rules for that event type (a rule is an event type paired with a channel) and creates one job per channel, rendering the tenant’s template for that event and channel at creation time so the job carries its final subject and body. Templates use {{placeholder}} substitution from the payload; most of the 25 migrations are rules and templates for the CampusCritique tenant.

Deliver. A scheduled worker runs every five seconds. It first returns any job stuck in PROCESSING for more than two minutes to PENDING, then claims up to fifty due jobs one at a time with SELECT ... FOR UPDATE. Each claim, each success and each failure is its own transaction (REQUIRES_NEW), so a batch of fifty emails is never one transaction and one failure cannot roll back forty-nine deliveries that already happened. The handler runs on a separate thread with a 20-second timeout. Every attempt is recorded with the provider, the attempt number and the error.

Retry with judgement. Failures are classified at the source. A Resend 5xx, a timeout or a network error is retryable and schedules the next attempt sixty seconds later, up to three attempts. A 422 for a bad address, a missing recipient field or a disabled channel fails the job immediately, because retrying cannot change the answer. Failed jobs are listed at GET /api/v1/jobs/failed with their last error.

The simulator below runs one connect.session_booked event through the same states the service uses, with the production settings. Flip the switches to take the broker down, redeliver a message, or make the email provider misbehave, then step through what each loop does.

• Commit, then publish, then sweep. The outbox pattern without an outbox table: the event row itself is the outbox, and QUEUED older than two minutes is the signal to republish. No distributed transaction, no lost events, and a broker outage costs at most a couple of minutes of latency.
• Claim and finalise one job at a time. A batch of fifty emails is never one transaction. The first version (17 May) delivered a whole batch inside one @Transactional method, which meant one failing push could roll back the rows for emails Resend had already accepted, and the next tick would send them again. The hardening commit the next morning split it into per-job claim and finalise, before the first production event.
• Idempotent at every hop. Idempotency key at ingest, existsByEventIdAndChannel plus a unique constraint at fan-out, FOR UPDATE at claim. Each hop can be retried by the layer above it without double-sending.
• Classify failures where they happen. The email and push handlers throw DeliveryException(message, retryable). The delivery service does not guess; it retries exactly what the handler says can change on a retry.
• Off by default. Email and push are disabled until their credentials exist. Enabling email without SMTP or Resend settings fails startup with a clear message (EmailConfigurationValidator) instead of failing silently at 2am.
• Small surface, checked early. A request-size filter caps event bodies (64 KB default, 1 MB hard ceiling), validates the charset and caches the body so it can be read once. No stack traces in responses. CORS is an allow-list. The API-key filter runs before Spring Security’s own chain and only the health endpoints are anonymous.
• Templates in the database, per tenant, versioned by migration. Product teams change copy by shipping a migration, which is reviewed like code. The render-test endpoint lets them see the output for a sample payload before it goes live.

Twenty-five of 1,346 jobs failed, all on email and push, none in-app: 10 of 284 emails (96.6%) and 15 of 325 pushes (95.6%). The likely causes, to be confirmed from the attempt log before the fix ships: bounced or unverified addresses on email, and expired browser subscriptions on push. The push case is the interesting one, because the handler classes every Firebase error as retryable, so an expired token burns three attempts sixty seconds apart before the job fails. If all 15 push failures went that route, they alone account for 30 of the 32 retries in the table above.

The fix is small and is the next change: treat Firebase’s UNREGISTERED and INVALID_ARGUMENT as non-retryable, delete the subscription row on the first one, and expose a per-recipient last failure so a product can prompt the user to re-subscribe. The before and after will be published here from the same query that produced the numbers above.

The second tenant now exists: the contact form on this site is tenant portfolio, with its own API key, rules and templates, and GET /api/v1/metrics (totals, per channel, per day, median ingest to delivered) feeds the live reliability ledger. For the first message through it, the endpoint reported a median ingest-to-delivered time of 7.1 seconds across the two channels (14 Sep 2026): one worker tick plus one Resend round-trip.

Further out: a dead-letter queue for events whose rules produce no jobs, per-tenant rate limits, an SMS or WhatsApp channel, and Micrometer metrics into Grafana so the ledger does not have to be a bespoke endpoint.