Skip to content
VVS

Orders, stock, and failure

An event-driven e-commerce backend built to practise the parts of distributed systems that tutorials skip: what happens when a message arrives twice, a dependency goes down, or a cache key goes cold under load.

Role
Sole developer
Timeline
Feb 2026 to Jun 2026
Type
Personal project, runs locally with Docker Compose
Stack
Java 17, Spring Boot 3.2, Kafka, PostgreSQL, Redis, Resilience4j, Eureka, Zipkin, k6
01

One order, three services

Placing an order touches the order service, the product catalogue and inventory. Each service owns its own PostgreSQL database, so there is no shared transaction to lean on. If inventory runs out after the order is written, something has to undo the order; if a confirmed order is cancelled, something has to give the stock back.

I used a choreographed saga: services react to each other's events on Kafka, and each step has a compensating action. Around it sit the protections a real service needs: a circuit breaker on the one synchronous call, a dead-letter path for bad messages, a cache that does not stampede, and a rate limiter.

02

Explore the system

Step through four scenarios to see which services and events take part, and click a component to read what it does. Seven Spring Boot applications in total: the gateway, Eureka and five domain services.

1 / 6

Swipe sideways to see the whole diagram.

price lookupClientk6 or curlAPI GatewaySpring Cloud GatewayEurekaservice registryuser-serviceown PostgreSQLproduct-serviceown PostgreSQLorder-serviceown PostgreSQLinventory-serviceown PostgreSQLnotification-serviceKafka consumerKafkasaga topics + .DLTRediscache + rate limitZipkintracing
HTTP Kafka event active in this stepdashed: discovery, or not yet wired

Step 1 · Order succeeds

POST /orders arrives at the gateway, which spends one token from this client's bucket.

Selected · order-service

Creates orders as PENDING and publishes order-created. Its one synchronous call, the product price lookup, is wrapped in retry, circuit breaker and a thread-pool bulkhead, with a single typed fallback.

Click any box in the diagram to read about it.

Illustrative· walkthrough of the documented design
03

Making the saga safe

Idempotent inventory
Every order's outcome is stored in a reservation row keyed uniquely by orderId. A redelivered order-created replays the stored outcome instead of reserving twice; a duplicate cancel on a released reservation does nothing.
Idempotent orders
State transitions only fire from PENDING, so a duplicate reserved or failed event is a no-op.
All or nothing
Inventory reserves every line or none. A partial reservation is rolled back before inventory-failed is published.
No overselling
The stock ledger has an optimistic lock, and stock is split into available and reserved buckets so a reservation can be reversed exactly.
Poison messages
Consumers wrap deserialisers so a bad payload becomes a handled error. Failing records are retried twice, one second apart, then routed to a .DLT topic.
Stampede protection
On a cache miss, one reader takes a Redis lock with a per-holder token, re-checks the cache, loads and repopulates. Others wait briefly, or read the database if the holder is slow.
04

Measured behaviour

k6 runs on one machine (Ryzen 9 7940HX, Docker Desktop, k6 v2.0.0, 13 Jun 2026, single host). The gateway limits each client IP to 10 requests per second, which would turn a single-host load test into a test of the limiter, so the heavy tests hit services directly and the limiter is measured on its own. Single-host numbers say how each layer behaves, not how the system scales.

Product reads, cached

Peak virtual users
200
Requests
42,921
Throughput
214.2 req/s
p50 / p95 / p99
1.06 / 2.02 / 2.39 ms
Errors
0 of 42,921
Cache hit rate
about 100% (one hot key, warmed)

One hot key, warmed first. This measures the cache path, not database reads.

Order creation, product service healthy

Target rate
ramp to 150 req/s
Achieved
50.7 req/s average over the ramp
Orders created
7,626, all 201
p95 latency
9.03 ms
Bulkhead rejections
0
Dropped iterations
98 (k6 VU pool limit)

The 98 dropped iterations were k6 running out of virtual users, not server errors.

Forced circuit-breaker trip

Calls while product-service down
25
Breaker state reached
OPEN at 50% failure rate
Fast-failed by fallback
25 not-permitted calls

After product-service restarted, the breaker did not close on its own. See below.

Gateway rate limiter

Burst of requests
60
Served
39
Rejected with 429
21
Measured· eCOMMERCE-MICROSERVICES docs/BENCHMARKS.md
05

Bugs found on the way

Getting the benchmark run to work at all surfaced two real bugs on the saga path.

A bean that vanished
Adding a KafkaTemplate<String, byte[]> for the dead-letter path made Spring Boot skip its default KafkaTemplate<String, Object>, because the auto-configuration condition matches the raw type and ignores generics. Order and inventory services failed to start. The fix declares the JSON template explicitly.
A null from a builder
Lombok's @Builder ignores field initialisers unless marked @Builder.Default, so an order's item list was null and every order creation returned 500. One annotation fixed it.
06

What is not done

Breaker recovery
After product-service restarted with a new container IP, order-service's load balancer cache kept the old instance, so every half-open probe failed and the breaker re-opened for minutes. Restarting order-service fixed it. I documented it rather than fixing it; the follow-up is a shorter cache TTL or eager refresh.
Dead-letter path
Wired and configured, but not re-captured in the latest benchmark run.
Notifications
The notification service listens for user-created events that nothing publishes yet.
Deployment
Runs locally with Docker Compose. It has not been deployed to a cloud provider or Kubernetes.
Public README
The top of the repository README predates the Kafka saga and still describes an older design. The architecture and benchmark docs are current.