What 1,346 notification jobs taught me about async delivery

Notify is the notification service I built for CampusCritique and now run for two tenants. Between 18 May and 14 September 2026 it ingested 712 events, fanned them out into 1,346 delivery jobs across in-app, email and push, and delivered 1,321 of them. Twenty-five failed. This is what the numbers taught me, in the order I learned it.

The first mistake was synchronous

The week Connect launched, CampusCritique needed a dozen kinds of notification: booking confirmed, reminder two hours before, session rescheduled, refund processed, review ready, payout sent. The obvious build is to send the email inside the request that caused it. It is also how a slow SMTP call ends up failing a payment webhook, and how a retry sends the same email twice.

So the first decision was that a product never sends anything. It posts an event with a tenant id, an event type, an idempotency key, a recipient and a payload, and gets a 202 back. Everything after that is Notify’s problem.

Commit first, publish second

The event is written to Postgres with status QUEUED before it is published to RabbitMQ. Not after, not in the same step. If the broker is down, the row still exists.

The gap between the commit and the publish is real and I did not paper over it. A scheduler runs every sixty seconds and republishes any event that has sat in QUEUED for more than two minutes. That is an outbox pattern without an outbox table, and it is enough at this scale: the event row already holds everything the broker message would.

Idempotent at every step, or it is not idempotent

A broker can deliver a message twice. A client can retry a POST. A worker can crash after sending an email and before marking the job sent. Each of those is a duplicate waiting to happen, and each needs its own guard:

• The ingest endpoint keys events on the tenant plus the idempotency key the caller supplies. A retry returns the existing event.
• Job creation is createInitialJobIfMissing. A redelivered broker message finds the jobs already there and does nothing.
• Delivery claims one job at a time and finalises it on its own. One failed email cannot roll back a batch of emails that already went out.

I stopped thinking of idempotency as a property of the API and started thinking of it as a property of every boundary a message crosses.

Retries need a failure class

Three attempts with sixty-second backoff, then FAILED. That policy is only sensible if the worker knows which failures are worth retrying. A rate limit or a 5xx from the provider is. A bounced address or a missing recipient field is not, and retrying it three times just delays the truth. Every attempt is written to its own table with the provider, the attempt number, the status and the error, which is what let me write the next paragraph.

The 25 that failed

Of the 1,346 jobs, in-app delivered 712 of 712. Email delivered 284 of 294. Push delivered 325 of 340. All twenty-five failures are on channels that leave the building: expired push subscriptions and addresses that bounced. The retry policy cannot fix either. What it can do is make them visible, and GET /api/v1/jobs/failed exists so a tenant can see exactly which ones. The next release adds cleanup of dead push subscriptions and a bounce handler, and the before and after will go on the case study page.

The retry rate is the number I watch more than the success rate: 1,378 attempts for 1,346 jobs means 32 retries in four months. If that climbs, something upstream changed.

The second tenant found what the first never would

For four months Notify had one tenant, which meant “multi-tenant” was a claim, not a fact. Making the contact form on this site the second tenant took a migration for the rules and templates and one API key. It also found two bugs in an afternoon. The email sender name was a global setting, so a message from my portfolio arrived from “CampusCritique”. And the authentication filter kept its own list of protected paths, separate from the security configuration, so the new metrics endpoint answered 403 to a valid key until both lists agreed.

Neither would have surfaced with one tenant. The lesson generalises: a second real user of any boundary is worth more than a week of imagining one.

What I would tell myself in May

Persist before you publish. Assume every message arrives twice. Classify failures before you retry them. Log every attempt, because the failures are the story. And get a second tenant, customer or caller as early as you can stand it.

The numbers here come from the production database on 14 September 2026 and from the metrics endpoint that now feeds the live ledger. The full case study, with the evaluation table and its sources, is at /work/notify.