Designing an Event-Driven Legal Intelligence Platform

Legal and regulatory platforms ingest content from external publishers, transform it into a consistent format, enrich it with links and metadata, and make it searchable for legal professionals.

That creates several distributed-systems questions:

• How should independent enrichment steps run in parallel?
• How do we prevent redelivery from producing duplicate documents?
• Should complete documents travel through Kafka?
• How do we join enrichment results that arrive at different times?
• Where do Kafka’s exactly-once guarantees end when S3 is involved?

This reference architecture answers those questions using Java 21, Spring Boot 3, Apache Kafka, Kafka Streams, Azure Event Grid, Amazon S3, OpenTelemetry and Kubernetes.

Note: This is a reference design created for demonstration and learning. It does not describe a real project I worked on.

System Overview

Two vendors publish legal-content notifications through Azure Event Grid. An ingestion adapter authenticates and normalizes them, then publishes canonical events to Kafka.

Independent services perform the enrichment:

• Content Enricher: Fetches the document, transforms it and stores it in S3.
• Link Enricher: Resolves and classifies links in the vendor payload.
• Metadata Generator: Extracts language, jurisdiction, entities, tags and document type.
• Statistics Engine: Calculates ingestion rates, processing lag and operational counts.

High-Level Architecture

Vendor A ─┐
          ├──► Azure Event Grid ──► Ingestion Adapter
Vendor B ─┘                              │
                                        ▼
                              Kafka: raw-content
                                  │           │
                           ┌──────┘           └──────┐
                           ▼                         ▼
                   Content Enricher           Link Enricher
                           │                         │
                           ▼                         ▼
                  enriched-content             link-events
                           │                         │
                           ▼                         │
                  Metadata Generator                │
                           │                         │
                           ▼                         ▼
                   metadata-events          fully-enriched join

                   Documents and metadata → Amazon S3
                   Traces and metrics → OpenTelemetry

The central design decision is to keep large document payloads out of Kafka. Kafka transports compact events and object references; S3 stores document bodies.

Component Responsibilities

Component	Responsibility	Output
Ingestion Adapter	Authenticate, validate and normalize vendor events	raw-content
Content Enricher	Fetch and transform documents; store them in S3	enriched-content
Link Enricher	Resolve and classify document links	link-events
Metadata Generator	Extract structured legal metadata	metadata-events
Readiness Topology	Join required enrichment results	fully-enriched
Statistics Topology	Compute operational aggregates	Named state stores/topics

Kafka Topic Design

All domain topics are keyed by contentId so events for one document remain ordered within each consumer group.

raw-content          12 partitions, 7-day retention
enriched-content     12 partitions, 14-day retention
link-events          12 partitions, 3-day retention
metadata-events      12 partitions, 30-day retention
fully-enriched       12 partitions

raw-content-dlq
enriched-content-dlq

Inputs to a Kafka Streams join must be co-partitioned: compatible partition counts, key types and partitioning logic. This is why link-events also has 12 partitions here.

The numbers are capacity assumptions—not universal defaults. Choose them from expected throughput, key distribution, processing time and scaling requirements.

Ingestion Adapter

The adapter acknowledges Event Grid only after Kafka accepts the normalized event.

1. Authenticate and validate the event
2. Deserialize the vendor envelope
3. Map it to RawContentEvent
4. Check idempotency using eventId or contentId + version
5. Publish to raw-content
6. Return a successful HTTP response

Using only contentId as the idempotency key would incorrectly discard legitimate later versions of the same document.

Azure Event Grid uses exponential backoff and, by default, attempts delivery for up to 24 hours or 30 attempts. The subscription can configure time-to-live and maximum attempts within supported limits.

Parallel Enrichment with Consumer Groups

The enrichers subscribe to raw-content under different groups:

raw-content
   ├── group: content-enricher ──► Content Enricher
   └── group: link-enricher    ──► Link Enricher

Kafka delivers each record independently to both groups. Within a group, useful parallelism is bounded by the partition count. Deploying 20 consumers against 12 partitions leaves at least eight idle.

Content Enricher

RawContentEvent
      ▼
Fetch document from vendor API
      ▼
Transform to canonical format
      ▼
PUT versioned object into S3
      ▼
Publish EnrichedContentEvent with S3 key

A starting consumer configuration could be:

enable.auto.commit=false
isolation.level=read_committed
max.poll.records=50
max.poll.interval.ms=300000

These values require load testing. Increasing max.poll.interval.ms prevents some rebalances but does not solve unbounded work. Other options include reducing max.poll.records, pausing assigned partitions or separating slow fetching from the Kafka poll loop.

After bounded retries are exhausted, publish a structured DLQ record containing the original topic, partition, offset, event ID, content version, failure classification, attempt count and trace identifier.

Link Enricher

Java 21 virtual threads suit blocking link-resolution calls, but they do not remove downstream limits. Cap outbound concurrency with a semaphore or bulkhead.

try (var executor = Executors.newVirtualThreadPerTaskExecutor()) {
    var futures = links.stream()
        .map(link -> executor.submit(() -> resolve(link)))
        .toList();

    var results = futures.stream()
        .map(Future::join)
        .toList();
}

Do not rely exclusively on HTTP HEAD; some servers reject it or implement it differently from GET. Use a controlled fallback, strict timeouts, redirect limits and host-level concurrency limits.

Metadata Generator and S3 Layout

The Metadata Generator consumes enriched-content, retrieves the document and applies language detection, named-entity recognition, jurisdiction extraction, taxonomy tagging and document classification.

s3://legal-intelligence-content/
├── raw/{vendorId}/{contentId}/{version}/payload.json
└── enriched/{contentId}/{version}/
    ├── document.pdf
    └── metadata.json

Including the version prevents a late retry from silently overwriting newer content. S3 Versioning can add recovery protection.

Joining Enrichment Results

Parallel work creates a state question: when is a document fully ready? Represent that state explicitly rather than inferring it from timing.

KStream<String, EnrichedContentEvent> content =
    builder.stream("enriched-content");

KStream<String, LinkEvent> links =
    builder.stream("link-events");

content.join(
        links,
        FullyEnrichedEvent::of,
        JoinWindows.ofTimeDifferenceAndGrace(
            Duration.ofMinutes(10),
            Duration.ofMinutes(2)
        )
    )
    .to("fully-enriched");

The window and grace period must reflect realistic event-time skew. If one result may arrive hours later, use a readiness aggregate or workflow state store instead of a short stream-stream join.

Operational Statistics

KStream<String, RawContentEvent> raw =
    builder.stream("raw-content");

raw.groupBy((contentId, event) -> event.getVendorId())
   .windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(1)))
   .count(Materialized.as("ingestion-rate-by-vendor"));

Use separate schemas or named stores for unrelated statistics. Mixing counts, link health and latency values in one topic complicates governance.

A KTable becomes queryable through interactive queries only after it is materialized as a named state store and the application exposes routing to the instance hosting the key.

Claim-Check Pattern

The platform uses claim check for large documents:

1. Store the document in S3.
2. Publish a compact event with its key, version and checksum.
3. Let authorized consumers retrieve it directly.

Coordinate S3 retention with Kafka replay requirements. Otherwise, replayed events may reference objects that no longer exist. Apply encryption, least-privilege IAM and checksum validation.

Exactly-Once: Where the Boundary Ends

Kafka transactions can atomically combine consumed offsets with records written to Kafka topics. They do not automatically include an S3 upload or vendor API call.

consume Kafka → write S3 → publish Kafka event

A crash after the S3 write but before the Kafka transaction commits can repeat the S3 operation.

The practical design is therefore to:

• use deterministic, versioned object keys;
• attach checksums and content versions;
• make retries idempotent;
• use Kafka transactions for Kafka-to-Kafka atomicity;
• reconcile incomplete workflow states;
• never claim that Kafka transactions cover S3 side effects.

S3 Consistency

Amazon S3 provides strong read-after-write consistency for PUT, GET, DELETE and LIST operations.

The Content Enricher must complete its S3 PUT before publishing EnrichedContentEvent. After the successful write returns, the Metadata Generator can read the object without relying on an eventual-consistency delay.

Kafka ordering does not create S3 consistency. The guarantee comes from S3’s model plus correct application sequencing.

Observability

Producers should inject OpenTelemetry context into Kafka headers and consumers should extract it. Useful telemetry includes:

• consumer lag and oldest-record age;
• end-to-end enrichment latency;
• vendor API latency, retries and circuit state;
• S3 request latency and failures;
• DLQ publication and replay counts;
• join-window misses;
• rebalance frequency and duration.

Avoid sampling.probability: 1.0 in production without understanding the cost. Tail-based sampling can preserve slow and failed traces while controlling volume.

Kubernetes Scaling and Backpressure

Lag-based scaling helps only while unassigned partitions remain. A robust policy should consider lag growth, oldest-record age, processing latency, dependency limits, error rates and rebalance disruption.

Scale up responsively, but scale down conservatively to avoid continuous rebalancing.

Key Failure Scenarios

Failure	Expected response
Event Grid redelivery	Deduplicate by event ID or content ID plus version
Vendor API outage	Bounded retries, circuit breaker and DLQ
Crash after S3 write	Retry safely with deterministic versioned keys
Enrichment results arrive far apart	Retain state longer or use a readiness aggregate
Poison event	Route to DLQ and advance the main partition deliberately
Consumer lag grows	Scale only to useful partition parallelism and protect dependencies
Incompatible schema	Reject it in CI through Schema Registry checks

Final Design Principles

The architecture has five important boundaries:

1. Event Grid is the vendor-facing delivery boundary.
2. Kafka is the internal event and ordering boundary.
3. S3 is the durable document-content boundary.
4. The fully enriched event is the business-readiness boundary.
5. Kafka exactly-once semantics stop at non-Kafka side effects unless those systems explicitly participate.

The strongest design is not the one with the most services or topics. It is the one that makes completion, retries, ownership and consistency boundaries explicit.

References

• Azure Event Grid delivery and retry
• Amazon S3 consistency model
• Apache Kafka delivery semantics
• OpenTelemetry context propagation

---

Follow Systems in Practice for more articles about Java, Kafka, cloud architecture and reliable distributed systems.