Customer Data Synchronization Architecture

1. Purpose

This document describes the ingestion and cache-refresh architecture for synchronizing customer profiles and mandates from a central banking system into an investment platform.

The central banking system is the authoritative source of customer data. It emits an event for every update, but an individual event is treated as a change notification rather than a complete representation of customer state. The investment platform accumulates related notifications, fetches the latest authoritative snapshot, stores a durable local projection, and refreshes a low-latency cache.

A successful customer login is an additional high-priority synchronization trigger because mandates should be available as soon as possible after login.

2. Requirements and assumptions

Functional requirements

Synchronize customer profiles and mandates almost in real time.
Accumulate bursts of granular events for the same customer.
Fetch the complete fresh state from the central customer API.
Maintain a durable local customer projection.
Load or refresh mandate data in cache when a customer logs in.
Prevent stale snapshots from replacing newer customer state.
Recover from duplicate, delayed, missing and out-of-order events.

Working assumptions

The central system can identify a customer in every event.
Preferably, events and snapshot responses contain a source-generated monotonically increasing version.
The central customer API can return a complete customer and mandate snapshot.
Kafka provides durable internal event transport.
The system accepts bounded eventual consistency between the central system and the investment platform.
The exact event rates, latency SLOs, cache freshness threshold and central API capacity have not yet been defined.

3. Core architectural decision

Central events are treated as invalidation signals, not state-transfer events.

When several events arrive for the same customer, the platform records that the customer is dirty through the highest observed source version. It then performs one snapshot fetch after a short accumulation window. This avoids reconstructing customer state from incomplete events and reduces redundant calls to the central API.

Correctness does not depend solely on Kafka ordering. It comes from:

Durable per-customer synchronization state.
Source versions.
Idempotent state transitions.
Conditional projection writes.
Worker leases and fencing.
Periodic reconciliation.

4. Component architecture

flowchart TD
    Central["Central customer system"] --> Adapter["Source consumer + Kafka producer<br/>Customer event adapter"]
    Adapter --> Changes["Kafka topic<br/>customer-change"]
    Changes --> Collector["Kafka consumer<br/>Change collector"]
    Collector --> State["Database<br/>Sync state + outbox"]
    State --> Scheduler["Scheduler + Kafka producer<br/>Outbox publisher"]
    Scheduler --> Commands["Kafka topic<br/>customer-sync-command"]
    Commands --> Worker["Kafka consumer + API client<br/>Snapshot worker"]
    Worker --> API["REST API<br/>Central customer snapshot API"]
    Worker --> Projection["Database<br/>Customer projection + outbox"]
    Projection --> Publisher["Kafka producer<br/>Projection outbox publisher"]
    Publisher --> Updated["Kafka topic<br/>customer-projection-updated"]
    Updated --> CacheLoader["Kafka consumer<br/>Cache loader"]
    CacheLoader --> Cache["Redis/cache<br/>Customer mandates"]

5. Component responsibilities

Component	Component type	Responsibility
Central customer system	External authoritative system	Maintains customer profiles and mandates and emits granular change notifications.
Customer event adapter	Source consumer and Kafka producer	Consumes the source transport, validates and normalizes events, and publishes canonical events keyed by `customerId`.
`customer-change`	Kafka topic	Retains the canonical customer-change stream and preserves ordering within a customer partition.
Change collector	Kafka consumer	Accumulates notifications and atomically advances the highest observed source version.
Sync-state store	Database	Stores observed/synchronized versions, status, priority, debounce deadline, attempts and processing lease.
Sync outbox	Database table	Atomically records sync commands alongside state changes to avoid a database/Kafka dual write.
Sync scheduler	Scheduler	Selects eligible pending customers and records sync commands in the outbox.
Sync outbox publisher	Kafka producer	Reliably publishes committed sync commands.
`customer-sync-command`	Kafka topic	Carries customer snapshot-fetch commands keyed by `customerId`.
Snapshot worker	Kafka consumer and REST client	Claims a fenced lease, fetches the authoritative snapshot, validates it and updates the projection conditionally.
Central customer snapshot API	REST API	Returns the complete customer profile, mandates and source version.
Customer projection	Database	Provides a durable, versioned local view of customer state.
Projection outbox publisher	Kafka producer	Publishes committed projection changes without a dual-write gap.
`customer-projection-updated`	Kafka topic	Notifies cache loaders and other downstream services of a committed projection version.
Cache loader	Kafka consumer	Writes the latest customer and mandate projection to the cache idempotently.
Mandate cache	Redis or equivalent	Serves low-latency customer and mandate reads. It is not the durable source of truth.

6. Canonical change event

{
  "eventId": "evt-9182",
  "customerId": "cust-123",
  "sourceVersion": 481,
  "changeType": "MANDATE_UPDATED",
  "occurredAt": "2026-08-30T20:15:31.123Z",
  "receivedAt": "2026-08-30T20:15:31.340Z",
  "correlationId": "corr-82",
  "source": "BANK_CUSTOMER_MASTER"
}

customerId is the Kafka key. Events belonging to one customer therefore enter the same partition, while different customers can be processed concurrently.

The source adapter acknowledges the original message only after the normalized event has been durably accepted. If the source transport and Kafka cannot participate in one transaction, the adapter must tolerate redelivery and use stable event identifiers.

7. Event accumulation

The change collector does not invoke the central API for every event. It performs an atomic update of a synchronization-state record.

{
  "customerId": "cust-123",
  "latestObservedVersion": 481,
  "lastSyncedVersion": 477,
  "status": "PENDING",
  "priority": "NORMAL",
  "firstPendingAt": "2026-08-30T20:15:31Z",
  "lastEventAt": "2026-08-30T20:15:31Z",
  "eligibleAt": "2026-08-30T20:15:31.500Z",
  "attemptCount": 0
}

For every event:

latestObservedVersion = max(existingVersion, event.sourceVersion)
lastEventAt = max(existingLastEventAt, event.occurredAt)
status = PENDING, unless a synchronization is already running

This makes duplicate and out-of-order notifications harmless. Deduplication by eventId can reduce repeated processing, but correctness still comes from source-version comparisons.

Debounce and maximum wait

A short debounce window collapses a burst of events into one snapshot fetch. A maximum wait prevents continuous traffic from postponing synchronization forever.

eligibleAt = min(
    lastEventAt + debounceWindow,
    firstPendingAt + maximumWait
)

Illustrative values, subject to measurement:

Debounce window: 300 milliseconds.
Maximum wait: 2 seconds.

Critical change types, such as mandate revocation, can bypass the normal debounce by setting eligibleAt to the current time and assigning critical priority.

8. Scheduling and transactional outbox

The scheduler finds records where:

status = PENDING
eligibleAt <= now
nextAttemptAt <= now

It atomically claims the record and writes a sync command into an outbox in the same database transaction.

{
  "syncId": "sync-772",
  "customerId": "cust-123",
  "targetVersion": 481,
  "reason": "EVENT_ACCUMULATION",
  "priority": "NORMAL",
  "requestedAt": "2026-08-30T20:15:31.500Z"
}

The outbox is needed because independently updating the database and publishing to Kafka creates two failure gaps:

The database says SCHEDULED, but Kafka publication fails.
Kafka receives a command, but the state transition fails.

The outbox publisher retries publication from durable database state. Duplicate Kafka publication remains possible, so the downstream worker must be idempotent.

9. Snapshot synchronization

The snapshot worker consumes a command and:

Claims the customer by an atomic state transition.
Creates a lease containing an owner, expiry and fencing token.
Fetches the complete customer snapshot from the central API.
Validates customer identity, schema and source version.
Maps the central model into the investment-domain projection.
Writes the projection only if the fetched version is newer.
Records a projection update in an outbox transactionally.
Updates synchronization state based on the fetched and observed versions.

Example snapshot:

{
  "customerId": "cust-123",
  "sourceVersion": 483,
  "profile": {},
  "mandates": [],
  "relationships": [],
  "eligibility": {}
}

It is valid for a command targeting version 481 to receive version 483. The newer snapshot covers the requested change and additional changes that occurred before the fetch.

10. Concurrency and race handling

Event arrives during an active fetch

Worker begins fetching target version 481
Collector observes event version 482
Central API returns snapshot version 481

After writing version 481, the worker compares it with latestObservedVersion. Because version 482 has been observed, the customer remains pending and another fetch is scheduled.

lastSyncedVersion = 481
latestObservedVersion = 482
status = PENDING

If the API instead returns version 483:

lastSyncedVersion = 483
latestObservedVersion = 482
status = IDLE

Two workers process the same customer

Kafka partitioning normally serializes commands for the same customer, but rebalancing, retries and expired leases can still create overlapping work. A conditional lease claim and fencing token prevent an obsolete worker from committing ownership-sensitive state.

Slow worker finishes after a newer worker

Projection writes are conditional:

apply snapshot only when fetchedVersion > storedVersion

Therefore, a slow response for version 481 cannot overwrite an already committed version 483.

Synchronization state machine

stateDiagram-v2
    [*] --> IDLE
    IDLE --> PENDING: Change or login received
    PENDING --> SCHEDULED: Eligible for synchronization
    SCHEDULED --> RUNNING: Worker claims fenced lease
    RUNNING --> IDLE: Snapshot covers observed version
    RUNNING --> PENDING: A newer version remains
    RUNNING --> RETRY: Transient failure
    RETRY --> SCHEDULED: Backoff completed
    RUNNING --> FAILED: Non-retryable failure
    FAILED --> PENDING: Repaired or replayed

A successful login emits an additional event:

{
  "eventId": "login-9821",
  "customerId": "cust-123",
  "sessionId": "session-781",
  "loggedInAt": "2026-08-30T21:30:00Z",
  "reason": "CUSTOMER_LOGIN"
}

Authentication credentials and sensitive tokens must not be placed in this event.

flowchart TD
    Login["Successful login"] --> LoginProducer["Kafka producer<br/>Authentication service"]
    LoginProducer --> LoginTopic["Kafka topic<br/>customer-login"]
    LoginTopic --> LoginConsumer["Kafka consumer<br/>Login sync trigger"]
    LoginConsumer --> State["Sync-state database<br/>HIGH priority; eligible now"]
    State --> Command["Kafka topic<br/>customer-sync-command"]
    Command --> Worker["Snapshot worker"]
    Worker --> CentralAPI["Central customer API"]
    Worker --> Projection["Durable customer projection"]
    Projection --> Updated["Kafka topic<br/>customer-projection-updated"]
    Updated --> Cache["Mandate cache"]

The login handler reuses the same synchronization pipeline instead of creating a second fetch implementation. It records:

status = PENDING
priority = HIGH
reason = LOGIN
eligibleAt = now

Before scheduling a central API request, the handler checks whether:

The local projection is already sufficiently fresh.
A synchronization is already pending or running.
A recent login has already caused a refresh.

Multiple tabs, retries or repeated logins should therefore result in one in-flight synchronization per customer rather than multiple central API calls.

Important asynchronous limitation

Publishing a login event does not guarantee that mandates are ready immediately:

Login succeeds
→ event is published
→ consumer schedules work
→ worker calls central API
→ projection commits
→ cache consumer updates Redis

Two approaches are possible:

Asynchronous cache warming

Complete login immediately and populate the cache in the background. This minimizes authentication latency, but the first mandate request may arrive before the cache is ready.

Bounded readiness wait

If the first screen requires mandates, a cache miss may join the existing in-flight synchronization for a bounded duration. It should not independently issue another central API call. After the timeout, the system follows an explicit business rule, such as showing a loading state or failing closed for a sensitive action.

12. Cache consistency

The cache is a disposable serving layer, not the durable local source of truth.

Recommended order:

Fetch the central snapshot.
Conditionally commit the durable projection.
Commit a projection-outbox record in the same transaction.
Publish customer-projection-updated.
Let the cache consumer write the corresponding version to Redis.

The cache entry should contain the source version and refresh time. A cache consumer must reject an event older than the cached version.

If the cache is unavailable, synchronization can still complete successfully because the projection and outbox are durable. The cache can be rebuilt by replaying projection-update events or scanning the projection store.

13. Fault tolerance

Failure	Treatment
Duplicate source event	`max(sourceVersion)` state update is idempotent.
Out-of-order event	Older versions cannot reduce `latestObservedVersion`.
Kafka redelivery	Consumers use idempotent transitions and stable identifiers.
Malformed event	Quarantine it with validation reason and correlation metadata; do not block the partition indefinitely.
Central API timeout	Retry with exponential backoff and jitter.
Central API overload	Bound concurrency, apply rate limits and open a circuit breaker.
Worker crash	Lease expires and another worker reclaims the customer.
Old worker resumes	Fencing token and conditional writes reject obsolete ownership.
Projection write failure	Do not advance `lastSyncedVersion`. Retry safely.
Projection committed before worker crash	Redelivery repeats an idempotent version-conditional write.
Outbox publication failure	Retry from the durable outbox.
Redis failure	Preserve the durable projection and rebuild the cache later.
Lost source notification	Periodic reconciliation restores convergence.
Poison customer record	Isolate after controlled retries, alert and support explicit replay.

Retries must be bounded and include jitter to prevent synchronized retry storms. Operational tooling should distinguish retryable failures from schema, validation or business-data errors.

14. Data consistency model

The architecture uses bounded eventual consistency between the central system and the investment platform.

The central system is authoritative.
The local projection is a versioned, eventually consistent copy.
Kafka events indicate that synchronization is required.
Source versions define freshness and prevent stale overwrites.
Reconciliation provides eventual convergence when notifications are missed.

End-to-end exactly-once processing is not claimed. Kafka exactly-once features cannot create one atomic transaction across the central REST API, the projection database and Redis. Instead, the design uses at-least-once delivery with idempotent effects.

For financially sensitive actions, such as placing an order under a mandate, the allowed staleness and possible synchronous revalidation remain business decisions.

15. Scalability and backpressure

customerId should be the key for change events, sync commands and projection-update events. This serializes conflicting work per customer while allowing independent customers to run concurrently.

Scaling mechanisms include:

Increase Kafka partitions to increase customer-level parallelism.
Scale collectors and workers horizontally within consumer groups.
Accumulate event bursts to reduce central API calls.
Limit snapshot-worker concurrency to central API capacity.
Apply per-source bulkheads and rate limits.
Pause or slow scheduling when the central API circuit is open.
Consider a bulk snapshot API if individual fetch capacity is insufficient.
Detect hot partitions and customers that produce disproportionate traffic.

The safe concurrency level is constrained by central API capacity, projection database capacity and required freshness—not merely the number of worker instances.

An initial capacity relationship is:

required concurrent requests
    = required snapshot requests per second
    × average API latency in seconds

Concrete sizing cannot be completed until event rate, accumulation ratio, API latency and source-system limits are known.

16. System performance and observability

Important technical metrics:

Kafka consumer lag by partition.
Event-to-projection latency at p50, p95 and p99.
Projection-to-cache latency.
Oldest pending customer age.
Number of customers in each synchronization state.
Snapshot API latency, throughput, timeout and error rate.
Event-to-fetch reduction ratio achieved by accumulation.
Worker concurrency, saturation and retry volume.
Conditional-write rejection count.
Lease expiration and fencing rejection count.
Outbox age and unpublished record count.
Cache hit rate and stale-entry count.
Reconciliation mismatch count.

Distributed traces should carry customerId in a privacy-safe form, eventId, syncId, correlation ID and source version through the adapter, collector, scheduler, worker, projection and cache update.

Alerts should emphasize user-visible freshness and oldest pending age rather than queue depth alone.

17. Consensus and coordination boundary

The application does not implement Raft, Paxos or another consensus algorithm. Kafka, Kubernetes and the selected distributed database may use consensus internally for controller election, metadata or replication.

At the application level:

Kafka partition ownership provides efficient work assignment.
Database conditional updates provide atomic state transitions.
Leases and fencing tokens prevent obsolete workers from committing.
Source versions determine data freshness.

Kafka ownership is not treated as a correctness guarantee because rebalances and long-running external calls can leave an old worker executing. The conditional write and fencing layer remains necessary.

If the system later needs a singleton reconciliation coordinator, it should use an established consensus-backed lease mechanism rather than implement consensus internally.

18. Principal trade-offs

Event accumulation versus freshness

A longer accumulation window reduces central API load.
A shorter window improves freshness.
A maximum wait prevents starvation.
Critical mandate events and login events can bypass the normal delay.

Asynchronous warming keeps login fast.
It cannot guarantee that mandates are ready for the first request.
A bounded wait improves immediate readiness but couples user latency to synchronization and central API performance.

Local projection versus direct central reads

A durable projection improves availability, performance and auditability.
It introduces eventual consistency and synchronization complexity.
Direct reads offer fresher data but increase latency and central-system dependency.

Source versions versus timestamps

A monotonic source version provides reliable ordering.
Timestamps can be affected by clock skew, precision and equal timestamps.
If versions are unavailable, the system needs a weaker and explicitly documented convergence strategy.

Kafka ordering versus database correctness

Per-customer partitioning reduces concurrent conflicts.
Rebalancing and retries mean ordering alone is insufficient.
Conditional writes add database work but provide durable stale-write protection.

Outbox versus direct publication

Direct publication is simpler but creates dual-write failure gaps.
An outbox provides reliable publication but introduces another table, publisher and operational backlog.

Cache updated by worker versus cache consumer

Direct cache update can reduce latency.
Event-driven cache loading isolates Redis failure and supports replay.
The durable projection must remain the locally authoritative state in either approach.

At-least-once versus exactly-once claims

At-least-once with idempotent effects works across Kafka, REST and database boundaries.
End-to-end exactly-once would be misleading because the external API and cache do not share one transaction.

19. Summary

The design treats central customer events as invalidation signals. Events for the same customer are partitioned and accumulated, and a worker fetches the latest authoritative snapshot. Durable synchronization state records the latest observed and successfully synchronized versions. Conditional writes, leases, fencing and reconciliation protect the system from duplicates, out-of-order delivery, overlapping workers and missed events.

A successful login produces a high-priority synchronization request that bypasses normal accumulation but reuses the same pipeline. This warms the mandate cache efficiently and deduplicates repeated logins. Because Kafka processing is asynchronous, immediate mandate readiness requires either acceptance of a loading interval or a bounded wait on the existing synchronization.

The architecture strongly demonstrates fault tolerance, concurrency handling, eventual consistency and horizontal scalability. Consensus is deliberately delegated to proven infrastructure rather than implemented in the application.

20. Uncovered items for later discussion

The following topics remain deliberately open and should be addressed in later design sessions:

Exact functional boundaries between customer profiles, parties, mandates, portfolios and orders.
Concrete event volume, customer count, burst size and payload size.
End-to-end freshness SLOs for ordinary changes, login refreshes and mandate revocations.
Maximum capacity and rate limits of the central customer API.
Required Kafka partition count, worker count and autoscaling policy.
Behaviour when the first mandate request arrives before asynchronous cache warming finishes.
Whether order placement must synchronously revalidate a mandate.
Fail-open versus fail-closed behaviour when the central system is unavailable.
Whether mandate revocation must invalidate authorization immediately.
Source behaviour when no monotonic version is available.
Read-your-own-writes requirements after a customer changes information centrally.
Cross-entity consistency between customer, mandate, party and portfolio projections.
Multi-region topology, regional ownership and cross-region replication.
Disaster recovery targets, including RPO, RTO and Kafka/database restoration.
Kafka cluster and projection-database failure scenarios.
Full reconciliation strategy, scheduling, partitioning and source comparison mechanism.
Schema evolution and backward/forward compatibility for events and APIs.
GDPR deletion, consent withdrawal, data minimization and retention.
Authentication, authorization, encryption, secrets management and PII-safe observability.
Audit requirements for customer and mandate changes.
Cache TTL, freshness threshold, eviction policy and rebuild strategy.
Hot-customer and hot-partition mitigation.
Large customer snapshots, pagination and partial API responses.
Central API bulk-fetch capability and whether it is needed for backlog recovery.
Deployment compatibility, rolling upgrades and Kafka-rebalance handling.
Chaos testing, load testing and failure-injection strategy.
Precise monitoring thresholds, alert ownership and operational runbooks.
Singleton coordination needs, such as reconciliation leadership, and the consensus-backed mechanism to use.
Business handling of malformed or internally inconsistent customer data.
Data migration and initial population before real-time change processing begins.