TobussSystems in Practice

System Design: Multi-Region Payee Store & DynamoDB Cheat Sheet

4 min read

A practical reference covering the design of a scalable, multi-region payee storage system, followed by a deep dive into DynamoDB — the data store that makes the client-scoped access pattern fast.

Designing a Multi-Region Payee Store

Problem: store payees for millions of clients (each with hundreds of payees) across multiple geographic regions with high availability and low latency.

Access pattern analysis (drives everything):

  • List all payees for a client — very frequent
  • Get a specific payee — on every payment
  • Add / update / deactivate a payee — moderate
  • Cross-client queries — rare, admin only

Key insight: almost all access is scoped to a single client → partition by client_id.

Storage choice: a partitioned NoSQL store (DynamoDB, Cosmos DB, Cassandra) over a single relational DB, because:

  • Horizontal scaling without re-sharding
  • Client-scoped queries are O(1) partition reads
  • Built-in geo-replication

Multi-region strategy — Primary-write, Multi-read:

  • Each client has a home region. Writes go to the home region (strong consistency).
  • Reads are served locally from replicas (eventual consistency acceptable for UI).
  • Avoids write conflicts — simpler than multi-master, much less error-prone for payments.
  • Multi-master only if global write latency is a hard requirement — adds significant conflict resolution complexity.

Security: encrypt account numbers / IBAN at field level. Tokenise where possible. Enforce row-level isolation (one client cannot see another’s payees). Full audit trail — soft-delete only, never hard-delete.

  Client App
      │
      ▼
  API Gateway (auth, rate limiting)
      │
      ▼
  Payee Service (stateless, horizontally scaled)
      │
      ├──► Redis Cache (key: client_id, TTL: 30s)
      │         │ cache miss
      ▼         ▼
  Partitioned NoSQL DB
  ┌────────────────────────────────┐
  │  PK: client_id  SK: payee_id  │
  │  client_123 | payee_001       │
  │  client_123 | payee_002       │
  │  client_456 | payee_001       │
  └────────────────────────────────┘
       EU Region (primary writes)
            │  async replication
       ┌────┴────┐
       ▼         ▼
    US Region   APAC Region  (local reads)

  Trade-offs:
  Partition by client_id  →  easy scale, no cross-client queries
  NoSQL                   →  scalability over joins/transactions
  Primary-write region    →  simple consistency, easier compliance
  Eventual reads          →  slight staleness acceptable for payee list UI
// DynamoDB table design (AWS SDK v2)
// PK = clientId (partition key), SK = payeeId (sort key)

// List all payees for a client — single partition read, O(1)
QueryRequest listPayees = QueryRequest.builder()
    .tableName("Payees")
    .keyConditionExpression("clientId = :cid")
    .expressionAttributeValues(Map.of(":cid", AttributeValue.fromS(clientId)))
    .build();
QueryResponse response = dynamoDb.query(listPayees);

// Get one payee — GetItem, single key lookup
GetItemRequest getPayee = GetItemRequest.builder()
    .tableName("Payees")
    .key(Map.of(
        "clientId", AttributeValue.fromS(clientId),
        "payeeId",  AttributeValue.fromS(payeeId)
    ))
    .consistentRead(true) // strong consistency for payment flows
    .build();

// Deactivate a payee — soft delete via status update
UpdateItemRequest deactivate = UpdateItemRequest.builder()
    .tableName("Payees")
    .key(Map.of(
        "clientId", AttributeValue.fromS(clientId),
        "payeeId",  AttributeValue.fromS(payeeId)
    ))
    .updateExpression("SET #s = :inactive, updatedAt = :now")
    .expressionAttributeNames(Map.of("#s", "status"))
    .expressionAttributeValues(Map.of(
        ":inactive", AttributeValue.fromS("INACTIVE"),
        ":now",      AttributeValue.fromS(Instant.now().toString())
    ))
    .conditionExpression("clientId = :cid") // ownership guard
    .expressionAttributeValues(Map.of(":cid", AttributeValue.fromS(clientId)))
    .build();

DynamoDB Deep Dive

DynamoDB is a fully managed, distributed key-value and document database designed for massive scale, single-digit millisecond latency, and high availability. Mental model: “a globally scalable hash table with optional sorting.”

Core concepts:

  • Table — equivalent to a SQL table
  • Item — equivalent to a row; stored as JSON-like attributes; schema-flexible except for keys
  • Attributes — String, Number, Boolean, List, Map

Key types:

  • Partition key only (simple key) — good for one item per key lookups
  • Partition key + Sort key (composite) — allows multiple items per partition, range queries, ordered data. This is the payee model: PK=clientId, SK=payeeId.

Scaling: DynamoDB hashes the partition key and distributes across physical partitions automatically. Hot partition problem: if many clients use the same PK (e.g. “ALL_PAYEES”), all traffic hits one partition. Fix: choose high-cardinality keys (clientId, orderId).

Query model — intentionally limited:

  • GetItem, Query (PK + optional SK conditions), BatchGet — all fast
  • Scan — reads every item; avoid at scale
  • No joins, no ad-hoc queries by design

Secondary indexes:

  • GSI (Global Secondary Index) — different PK/SK, eventually consistent, costs extra writes. Use for “find payee across clients” scenarios.
  • LSI (Local Secondary Index) — same PK, different SK, strongly consistent, must be defined at table creation.

DynamoDB is great for: user profiles, sessions, payments, payees, idempotency keys, event metadata. DynamoDB is bad for: ad-hoc queries, joins, reporting, analytics, full-text search — pair with Elasticsearch/Athena for those.

  Partition key hashing:
  clientId "client_123" → hash → physical partition 3
  clientId "client_456" → hash → physical partition 7
  clientId "client_789" → hash → physical partition 1
                                   (even distribution ✅)

  Hot partition — bad key design:
  PK = "ALL_PAYEES" → hash → always partition 5 → overloaded ❌

  Composite key layout:
  PK (clientId)  | SK (payeeId)   | name    | status
  ───────────────────────────────────────────────────
  client_123     | payee_001      | Alice   | ACTIVE
  client_123     | payee_002      | Bob     | ACTIVE
  client_123     | payee_003      | Carol   | INACTIVE
  client_456     | payee_001      | Dave    | ACTIVE

  GSI for reverse lookup (payeeId → clientId):
  GSI-PK (payeeId) | GSI-SK (clientId)
  payee_001        | client_123
  payee_001        | client_456
// DynamoDB table provisioning (CloudFormation / CDK concept)
// PK = clientId, SK = payeeId
// GSI: PK = payeeId (find which clients share a payee)

// Consistency choice at read time
GetItemRequest strongRead = GetItemRequest.builder()
    .tableName("Payees")
    .key(key)
    .consistentRead(true)   // strong — use for payment flows
    .build();

GetItemRequest eventualRead = GetItemRequest.builder()
    .tableName("Payees")
    .key(key)
    .consistentRead(false)  // eventual (default) — fine for UI lists
    .build();

// Conditional write — add payee only if it doesn't already exist
PutItemRequest addPayee = PutItemRequest.builder()
    .tableName("Payees")
    .item(Map.of(
        "clientId",  AttributeValue.fromS(clientId),
        "payeeId",   AttributeValue.fromS(UUID.randomUUID().toString()),
        "name",      AttributeValue.fromS(name),
        "iban",      AttributeValue.fromS(encryptedIban),  // encrypted at field level
        "status",    AttributeValue.fromS("ACTIVE"),
        "createdAt", AttributeValue.fromS(Instant.now().toString())
    ))
    .conditionExpression("attribute_not_exists(clientId) AND attribute_not_exists(payeeId)")
    .build();

// GSI query — find all clients that have a given payee (admin use case)
QueryRequest gsiQuery = QueryRequest.builder()
    .tableName("Payees")
    .indexName("PayeeId-ClientId-GSI")
    .keyConditionExpression("payeeId = :pid")
    .expressionAttributeValues(Map.of(":pid", AttributeValue.fromS(payeeId)))
    .build();