Inside the engineering decisions that reduced rider dispatch latency from seconds to milliseconds in uEngage Flash.
By uEngage Engineering
Infrastructure · Rider Dispatch Platform ·
Every Delivery Starts With a Race
Most people think a food delivery begins when the kitchen starts preparing the order.
It doesn't.
It begins the moment the customer taps "Place Order."
Behind that single click, an invisible race starts.
When an order enters our system, it isn't sent to just one logistics provider. Our POS and Delivery Management System (DMS) partners broadcast the same delivery request to multiple logistics platforms simultaneously.
Each platform races to assign a rider.
The first platform to confirm the assignment wins the order.
Everyone else loses.
The difference between winning and losing isn't measured in minutes.
It's measured in milliseconds.
"In rider dispatch, being one second late doesn't make you slower it makes you invisible."
For a long time, this race wasn't something we thought much about.
Our dispatch pipeline was fast enough.
Orders flowed smoothly.
Partners received updates quickly.
Everything worked exactly as designed.
Then our platform started growing.
More restaurant brands joined.
Order volumes increased.
Promotional campaigns generated larger traffic spikes.
Peak-hour dispatch became significantly more demanding.
And suddenly, the race we used to win consistently became harder to finish first.
That's when we realised we weren't facing a scaling problem.
We were facing a latency problem.
Why Milliseconds Matter in Food Delivery
Unlike many software systems, delivery platforms don't operate in isolation.
Every delivery request exists inside a competitive ecosystem.
When multiple logistics providers receive the same assignment request, the fastest confirmation usually secures the delivery.
That means dispatch speed directly affects:
Rider allocation success
Delivery efficiency
Restaurant experience
Platform competitiveness
A delay of a few hundred milliseconds might sound insignificant.
In rider dispatch, it can determine whether the order is yours or someone else's.
Average performance isn't enough.
Consistency matters more.
Because customers don't experience averages.
They experience the slowest moment in the journey.
This is why engineering teams often pay close attention to tail latency the slower end of response times that can have an outsized impact on real-world performance.
Reducing average latency is valuable.
Reducing unpredictable latency is what wins races.
When Growth Exposed a Hidden Bottleneck
Ironically, our biggest challenge appeared because the platform was becoming more successful.
The busiest days became our slowest.
Every Wednesday, promotional campaigns from multiple restaurant brands would drive sudden spikes in delivery requests.
Those traffic spikes exposed a design decision that had quietly worked for years.
At the time, whenever an order changed state such as:
Rider Assigned
Picked Up
Delivered
Cancelled
our platform immediately sent webhook updates to downstream partners.
Those webhook calls happened synchronously inside the request path.
At first, this architecture made perfect sense.
It was simple.
Easy to reason about.
Reliable under normal traffic.
Until partner APIs slowed down.
A delayed webhook didn't just affect one delivery update.
It occupied a request thread.
That thread delayed the next request.
Which delayed another.
And another.
Eventually, a single slow downstream dependency started slowing an entire chain of rider assignment requests.
The component responsible for confirming deliveries in milliseconds had become the bottleneck preventing them.
"Our servers weren't running out of capacity. They were running out of time."
The Problem Wasn't Throughput Anymore
Initially, we assumed we needed to process more events.
But production metrics kept pointing somewhere else.
The system wasn't failing because it couldn't handle volume.
It was failing because critical events were waiting too long before they could be processed.
That's an important distinction.
Throughput answers the question:
"How much work can the system process?"
Latency answers a different question:
"How quickly can this specific event be completed?"
In a rider dispatch platform, those aren't the same thing.
A system capable of processing millions of events per hour can still lose rider assignments if the one event that matters waits a few extra seconds.
We weren't trying to process more work.
We were trying to process the right work faster.
And that changed how we approached every engineering decision that followed.
Success Forced Us to Rethink Everything
At this point, it became clear that incremental tuning wouldn't be enough.
We needed to understand why our architecture behaved the way it did before deciding how to improve it.
Like most engineering teams, we didn't jump directly to the final solution.
Instead, we went through several iterations.
Each one solved a real problem.
Each one improved production.
And each one taught us something new.
Looking back, none of those decisions were mistakes.
They were necessary steps that eventually led us to rethink the architecture itself.
"Sometimes the biggest engineering breakthrough isn't finding a better optimisation. It's recognising that you're optimising the wrong design."
Perfect. This is where the story gets interesting.
The developer's original version is technically correct, but it reads like a sprint retrospective. Let's transform it into an engineering story where every decision has a reason, a trade-off, and a lesson.
Three Solutions That Improved the System But Didn't Win the Race
Engineering rarely gets the perfect solution on the first attempt.
Neither did we.
When we first started investigating our dispatch delays, there wasn't one obvious bug to fix or one service that had failed. The platform was functioning exactly as it had been designed to.
The challenge was that our workload had evolved.
The architecture hadn't.
So we did what every engineering team does: we improved the system one bottleneck at a time.
Each iteration solved a real problem.
Each one made the platform stronger.
And each one exposed the next limitation.
Looking back, none of these solutions was a mistake.
They were stepping stones that eventually led us to redesign the architecture altogether.
"Every iteration bought us time. None of them bought us the finish line."
Attempt One: Moving Work Off the Request Path
The first bottleneck was easy to identify.
Every rider status update triggered a synchronous webhook call.
If a downstream partner responded slowly, our request thread stayed occupied until the webhook completed.
Under normal traffic, this wasn't an issue.
During peak campaigns, it became one.
Instead of confirming rider assignments in milliseconds, the platform spent valuable time waiting for external systems to respond.
The solution seemed straightforward.
Rather than executing webhook calls inside the request lifecycle, we moved them into an asynchronous event pipeline powered by Apache Kafka running on Amazon MSK.
The request no longer waited for partner APIs.
Instead, it published an event to Kafka and immediately returned control to the application.
Webhook processing happened independently in the background.
The impact was immediate.
Application threads stopped blocking.
CPU utilisation became healthier.
Peak-hour stability improved significantly.
For the first time in weeks, the platform felt comfortable under load.
It was a meaningful improvement.
But it wasn't the solution.
The bottleneck had simply changed shape.
Instead of requests waiting on webhook execution, events were now waiting inside Kafka queues.
We had successfully increased throughput.
We hadn't reduced dispatch latency.
"We removed the traffic jam from the highway only to find it waiting at the toll booth."
What We Learned
Moving work asynchronously is excellent for scalability.
But asynchronous systems still introduce waiting time.
In a dispatch race where the first confirmation wins, even a one-second queue delay is enough to lose the order.
Attempt Two: Giving High-Volume Brands Their Own Lane
Our production metrics revealed something interesting.
The latency wasn't evenly distributed.
Most restaurants generated predictable traffic.
A handful of high-volume brands generated the majority of dispatch events during peak hours.
Those brands were effectively competing with every other event entering the same pipeline.
So we asked ourselves a simple question.
What if our busiest customers didn't have to wait behind everyone else?
Instead of processing every webhook through a shared queue, we isolated high-volume brands into dedicated Kafka topics and consumers.
Think of it like opening an express lane on a busy highway.
Large restaurant chains could move independently without being slowed down by unrelated traffic.
The results were encouraging.
Dispatch performance improved.
Allocation success increased.
Peak-hour delays became less frequent.
For a while, it looked like we had solved the problem.
Then traffic kept growing.
As order volumes increased, the dedicated queues eventually experienced the same behaviour as the shared queue before them.
The congestion had moved.
Again.
We hadn't removed the architectural limit.
We had simply postponed it.
"A wider road still has a speed limit."
What We Learned
Partitioning improves fairness.
It doesn't eliminate latency.
If the underlying processing model remains the same, increasing traffic eventually recreates the same bottleneck.
Attempt Three: Prioritising the Event That Decides Everything
By this stage, one thing had become obvious.
Not every event carried the same business value.
Some events were informational.
Others were operational.
Only one determined whether we won the dispatch race.
Rider assignment.
Everything else could tolerate a small delay.
Rider assignment couldn't.
So instead of prioritising brands, we prioritised event types.
Assignment events received their own dedicated queues.
Dedicated consumers.
Dedicated processing path.
On paper, this looked like the logical conclusion of everything we had learned so far.
Critical work would no longer compete with less important updates.
The architecture became cleaner.
More specialised.
More optimised.
And yet...
The production metric we cared about barely moved.
That was the moment we stopped asking:
"How do we optimise this architecture?"
and started asking:
"Is this architecture capable of solving the problem at all?"
Because no matter how carefully we partitioned the workload, every improvement was still based on the same core assumption.
We were optimising traffic flow.
The race itself hadn't changed.
"Sometimes the bottleneck isn't inside the queue. It's the idea behind the queue."
What We Learned
Prioritisation makes critical work faster.
But if the architecture still allows unpredictable waiting, critical work can still lose the race.
The Pattern We Couldn't Ignore
By now, three things had become clear.
Our architecture had improved dramatically.
So had our operational metrics.
Yet the dispatch race remained inconsistent.
Why?
Because every optimisation had focused on improving average performance.
The race wasn't decided by the average.
It was decided by the outliers.
The slowest requests.
The unexpected delays.
The duplicate events.
The edge cases.
That's when we realised we had been solving the symptoms rather than the system itself.
The real challenge wasn't simply making event processing faster.
It was making rider assignment fast, deterministic, and exactly-once.
Everything we had built so far solved one piece of that puzzle.
None of it solved all three.
"Fast isn't enough if the same event can be processed twice. Correct isn't enough if it arrives too late. We needed both."
The Question That Changed Everything
Once we stopped looking at queues and started looking at the race itself, the problem became surprisingly simple.
We didn't need more queues.
We didn't need more consumers.
We didn't need another optimisation.
We needed something our architecture had never truly had before.
A single source of truth.
A way to guarantee that one and only one consumer could win the race for each assignment.
In other words...
We didn't need another lane.
We needed a referee.
That realisation changed everything.
Perfect. This is the heart of the article.
From here onward, we stop talking about what failed and start talking about what finally worked.
This section should feel like the "aha!" moment the point where the engineering team stopped chasing symptoms and started solving the root cause.
The Turning Point: We Didn't Need Another Lane. We Needed a Referee.
By the time we'd completed three iterations of our dispatch pipeline, one thing had become impossible to ignore.
Every optimisation had made the system better.
None of them had made the system fundamentally different.
We had reduced blocking.
Partitioned workloads.
Prioritised critical events.
Yet during peak traffic, we still lost dispatch races that we should have won.
At that point, we stopped asking:
"How can we make our queues faster?"
Instead, we asked a different question.
"Why are we still depending on the queue to decide the winner?"
That shift in thinking changed everything.
Because the problem was never just about latency.
It was about certainty.
A rider assignment has only one correct outcome.
One confirmation.
One winner.
One successful dispatch.
Everything else is duplicate work.
Our architecture needed a way to guarantee that outcome without relying on application logic scattered across multiple consumers.
In other words, the race didn't need another optimisation.
It needed a referee.
"Fast systems win races. Deterministic systems win them consistently."
Understanding the Real Problem
Once we stepped back from the implementation details, the architecture exposed two challenges that had been hiding in plain sight.
1. Tail Latency Was Deciding the Race
Most engineering dashboards celebrate average performance.
But averages don't win rider assignments.
Imagine two dispatch pipelines.
Pipeline A processes most events in 80 milliseconds, but occasionally spikes to three seconds.
Pipeline B consistently processes events in 150 milliseconds.
On paper, Pipeline A looks faster.
In production, Pipeline B wins more races.
Why?
Because rider assignment isn't decided by averages.
It's decided by the slowest requests.
Those unpredictable delays known as tail latency were quietly determining whether we secured the delivery.
Reducing average latency wasn't enough anymore.
We had to make latency predictable.
2. Duplicate Processing Couldn't Be Left to Chance
There was another challenge hiding beneath the surface.
Delivery ecosystems are noisy.
Orders can be retried.
Events can arrive almost simultaneously.
Partners may send updates more than once.
Without proper safeguards, the same rider assignment could be processed multiple times.
That's not just inefficient.
It creates operational risk.
Duplicate confirmations.
Conflicting updates.
Unnecessary webhook calls.
Potential inconsistencies across downstream systems.
The dispatch platform didn't just need to be faster.
It needed to guarantee that every assignment was processed exactly once.
"Correctness is just as important as speed. Customers only see one delivery we should process it exactly once."
Designing a Different Kind of Pipeline
At this point, it became clear that further tuning wouldn't solve the problem.
We needed a different architecture.
Not an improved version of the old one.
A fundamentally simpler one.
Before replacing anything, we built the new pipeline alongside the existing Kafka implementation.
Both systems consumed the same production events.
Both processed real traffic.
Both generated comparable metrics.
Instead of relying on synthetic benchmarks or staging environments, we let production data decide which architecture performed better.
That decision removed assumptions from the process.
The numbers would tell us when the new design was ready.
Building the New Dispatch Pipeline
The final architecture wasn't built around adding more components.
It was built around reducing uncertainty.
Three technologies became the foundation of the new system.
Amazon SQS FIFO: One Queue. One Source of Truth.
The first decision was replacing multiple Kafka topics with a single Amazon SQS FIFO queue.
FIFO First In, First Out provided two characteristics that aligned perfectly with rider dispatch.
Ordering
Events belonging to the same workflow would always be processed in sequence.
No unexpected reordering.
No race conditions caused by event timing.
Just predictable execution.
Deduplication
SQS FIFO can automatically discard duplicate messages within a defined deduplication window.
Instead of building custom deduplication logic inside the application, the messaging infrastructure handled it for us.
That simplified the consumer significantly.
Messages that failed processing weren't silently discarded.
They retried automatically.
If repeated attempts still failed, they moved into a Dead Letter Queue (DLQ), where engineers could inspect and recover them without affecting live traffic.
The queue wasn't just transporting messages anymore.
It was actively enforcing reliability.
"Infrastructure should guarantee correctness whenever possible. Application code shouldn't have to reinvent it."
Redis Locks: Choosing One Winner
Message ordering solved part of the problem.
Exactly-once processing required another layer.
Before any consumer processed a rider assignment, it attempted to acquire a Redis lock using a unique key based on the task and status.
Something like:
FlashTaskStatusUpdate:18219757:cancel
The first consumer successfully acquiring that lock became the only worker allowed to process the event.
Every subsequent consumer attempting the same assignment immediately recognised that another worker had already claimed ownership.
Instead of processing duplicate confirmations, they simply stepped aside.
One assignment.
One confirmation.
One winner.
Exactly as intended.
The lock operation itself completed in approximately one millisecond, making it effectively invisible compared to overall dispatch latency.
That tiny Redis operation became the referee our previous architecture had been missing.
Long Polling: The Optimisation We Almost Missed
By this stage, the new pipeline was already producing more stable results.
Yet one metric refused to improve.
Average dispatch lag.
The queue itself wasn't the issue anymore.
Our consumers were.
They repeatedly checked the queue for new messages, even when it was empty.
Those frequent polling cycles introduced unnecessary waiting time between message arrival and message consumption.
The fix turned out to be remarkably simple.
Instead of continuously polling, we enabled long polling.
Consumers now held their connection open for up to 20 seconds, allowing Amazon SQS to return new messages immediately when they became available.
That single configuration change transformed the system.
Median dispatch lag dropped from multiple seconds to tens of milliseconds.
Sometimes the biggest performance improvements don't come from writing new code.
They come from understanding how the platform already works.
"Our biggest optimisation wasn't another service. It was changing how we listened."
Running Two Pipelines. Trusting the Data.
Even after the architecture looked promising, we weren't ready to migrate production traffic immediately.
Large infrastructure changes deserve evidence, not optimism.
So we ran both pipelines side by side.
Kafka continued processing live events.
The new SQS pipeline processed the same workload simultaneously.
Every event carried structured metadata.
Trace ID
Queue lag
Processing duration
Source pipeline
That visibility allowed us to compare the two systems objectively.
Not through theory.
Through production behaviour.
The new pipeline consistently demonstrated:
Lower latency
More predictable performance
Cleaner processing
Fewer duplicate events
Only after those metrics remained stable did we begin retiring the Kafka consumers.
One at a time.
No big-bang migration.
No risky overnight switch.
Just gradual confidence built on observable production data.
Engineering Isn't About Loving Technology
One of the most valuable lessons from this rebuild wasn't technical at all.
Many of us genuinely enjoyed working with Kafka.
It had served us well.
It solved real problems.
But engineering decisions shouldn't be based on familiarity.
They should be based on outcomes.
For this workload, Amazon SQS FIFO combined with Redis locks proved to be:
Simpler
More deterministic
Easier to operate
More cost-efficient
Better suited to rider dispatch
Replacing a favourite technology isn't admitting it was wrong.
It's recognising that different problems deserve different tools.
"Good engineering isn't about defending yesterday's architecture. It's about choosing today's best solution."
Perfect. This is the conclusion, but I don't want it to feel like a conclusion. I want readers to finish the article thinking:
"That wasn't a blog. That was an engineering case study."
This section should leave them with data, lessons, and philosophy the kind of ending Stripe or Cloudflare engineering blogs use.
Measuring Before Migrating: Why Observability Came First
One of the easiest mistakes during an infrastructure migration is assuming the new architecture is better simply because it's newer.
We didn't want assumptions.
We wanted evidence.
Before migrating a single production workload, we invested heavily in observability.
Every event flowing through both pipelines carried structured metadata, allowing us to understand exactly what happened at every stage of processing.
For every rider assignment, we recorded information such as:
Trace ID
Event source
Queue wait time (
lag_ms)Processing duration
Consumer status
Pipeline origin
Instead of relying on dashboards that only showed averages, we could trace an individual event from the moment it entered the system until the webhook reached our downstream partners.
That visibility became our confidence.
It told us not only whether the new pipeline worked but why it worked.
"Observability didn't just help us monitor the migration. It gave us the confidence to complete it."
Migrating Without Pressing the Big Red Button
Infrastructure migrations rarely fail because of technology.
They fail because of deployment strategy.
Rather than replacing the old system overnight, we introduced the new architecture gradually.
Both pipelines continued processing live production traffic.
Kafka remained active.
Amazon SQS FIFO processed the exact same events in parallel.
This gave us something invaluable.
A real-world comparison.
Every latency measurement.
Every duplicate event.
Every retry.
Every queue delay.
Everything could be compared against production not theory.
Once the data consistently showed that the new architecture was outperforming the legacy pipeline, we began retiring Kafka consumers one service at a time.
Each deployment was small.
Observable.
Reversible.
No overnight migration.
No high-risk cutover.
Just a series of deliberate engineering decisions supported by production evidence.
Sometimes the safest migration is also the slowest one.
And that's perfectly okay.
The Results
Ultimately, engineering decisions should be measured by outcomes not intentions.
Once the new dispatch architecture became the primary production pipeline, the improvements were measurable across every metric that mattered.
Dispatch Latency
The metric we cared about most improved dramatically.
Median end-to-end dispatch lag: ~76 milliseconds
95th percentile (p95): ~171 milliseconds
Those aren't benchmark results from a staging environment.
They're production measurements captured while writing this article.
For a dispatch platform competing in millisecond races, reducing tail latency wasn't just a technical achievement.
It directly increased our ability to win rider assignments consistently.
Queue Health
Healthy queues should spend their time processing not waiting.
Under the new architecture:
Average queue age remained below one second.
Peak-hour traffic created only brief and manageable spikes.
Backlogs no longer accumulated during promotional campaigns.
The queue became predictable instead of reactive.
Exactly-Once Processing
Reliability improved just as much as performance.
Combining:
Amazon SQS FIFO deduplication
Redis-based idempotency locks
Dead Letter Queue protection
allowed every rider assignment to be processed exactly once.
No duplicate confirmations.
No unnecessary retries.
No competing consumers racing to perform the same work.
Correctness became part of the infrastructure not just the application.
Operational Stability
Perhaps the most satisfying metric wasn't visible on any dashboard.
It was the absence of incidents.
Peak traffic that previously generated operational alerts became... uneventful.
No emergency debugging.
No queue firefighting.
No anxious monitoring every Wednesday evening.
Sometimes, the best engineering achievement is silence.
"Great infrastructure isn't exciting in production. It's predictable."
Lower Infrastructure Costs
Performance wasn't the only unexpected outcome.
Operating costs improved as well.
Our previous architecture relied on a provisioned Amazon MSK (Kafka) cluster.
That meant paying for brokers continuously, regardless of whether traffic was high or low.
The new design behaves differently.
Amazon SQS scales automatically with demand.
During quieter periods, it consumes almost no infrastructure resources.
As legacy Kafka consumers were retired, the dedicated MSK cluster could eventually be removed altogether.
The result?
A simpler architecture that was:
Faster
Easier to operate
More reliable
And significantly more cost-efficient
Performance improvements are valuable.
Performance improvements that also reduce operational cost are even better.
What This Rebuild Taught Us
Every engineering project leaves behind lessons that extend beyond the codebase.
This one certainly did.
1. Optimising the Wrong Architecture Still Leads to the Wrong Destination
Each of our early improvements solved a real bottleneck.
None solved the underlying problem.
Incremental optimisation is valuable.
But only when the underlying architecture is capable of delivering the outcome you're chasing.
2. Tail Latency Matters More Than Average Latency
Customers don't experience averages.
Neither do dispatch races.
The slowest requests often determine the real user experience.
Reducing variability turned out to be just as important as reducing speed.
3. Infrastructure Should Handle Infrastructure Problems
Message ordering.
Deduplication.
Retries.
Exactly-once guarantees.
Whenever these responsibilities can move into infrastructure, application code becomes simpler, safer, and easier to maintain.
4. Measure Everything Before Changing Anything
Migration decisions shouldn't depend on confidence.
They should depend on evidence.
Production observability gave us the ability to compare two architectures objectively and migrate only when the data supported it.
5. Don't Fall in Love With Technology
One of the hardest parts of this project wasn't writing code.
It was letting go of a technology we genuinely enjoyed using.
Kafka solved many of our earlier problems.
It simply wasn't the right solution for this one.
Engineering isn't about proving your favourite technology is always correct.
It's about choosing the right tool for today's constraints.
"Good engineers don't defend old decisions. They improve them."
Final Thoughts
Every engineering team eventually reaches a point where incremental improvements stop delivering meaningful results.
That's usually the moment when the real work begins.
For us, rebuilding the rider dispatch pipeline wasn't about replacing Kafka.
It wasn't about introducing Redis.
And it certainly wasn't about adopting another AWS service.
It was about rethinking the assumptions behind the architecture itself.
The outcome wasn't simply a faster dispatch pipeline.
It was a system that's easier to reason about, more predictable under load, and built around the realities of modern food delivery.
At uEngage, this is the kind of engineering we aspire to build.
Not infrastructure that's merely fast.
Infrastructure that's measurable.
Reliable.
Simple.
And ready to scale with the restaurants that depend on it.
Because in the end, customers don't remember queues, brokers, or distributed locks.
They remember one thing.
Whether their food arrived on time.
Key Takeaways
? Rider dispatch is a race measured in milliseconds, not minutes.
? Optimising latency requires reducing unpredictability, not just improving averages.
? Exactly-once processing is essential for reliable dispatch systems.
? Amazon SQS FIFO, Redis locks, and long polling created a simpler, faster, and more deterministic architecture.
? Observability should come before migration, not after.
? The best engineering decisions are guided by production data, not personal preference.
