TobussSystems in Practice

Designing a Global Legal-Intelligence Platform on AWS

13 min read

Part of seriesLegal Intelligence

In the previous article, we defined the business problem behind a global legal-intelligence platform.

Legislation arrives from external vendors. Editorial systems provide commentary and news. A publication can have many editions, each containing a hierarchy of legal provisions. HTML content, document metadata and the table of contents must describe the same edition, while citations and editorial relationships connect that edition to a much larger legal knowledge graph.

The existing process reloads an entire edition for every small change. That approach is expensive, slow and vulnerable to partial updates.

The architectural goal is different:

Accumulate related source events, calculate a node-level change plan, stage only the affected content and graph elements, and publish the new edition only when its HTML, hierarchy and TOC are mutually consistent.

This article develops that solution using AWS-managed services.

The Consistency Model Comes First

The system contains two different consistency boundaries.

Edition-critical data

The following must agree before a new edition becomes visible:

  • HTML content;
  • document metadata;
  • the provision hierarchy;
  • the table of contents;
  • additions, moves and tombstones;
  • the edition’s active-version pointer.

Graph enrichment

The following may be completed asynchronously after the edition is published:

  • citations to other publications;
  • links between editions;
  • commentary and news relationships;
  • backlinks;
  • cross-jurisdiction references;
  • search and recommendation indexes.

Blocking publication until every external citation has been resolved would make the platform unavailable whenever a referenced publication is missing or ambiguous. Publishing a broken internal hierarchy would be equally unacceptable.

The system therefore provides atomic visibility for an edition and eventual enrichment for its external relationships.

AWS Architecture with Neo4j

flowchart TD
    A["IBM MQ<br/>legislation"] --> B["Legislation adapter<br/>ECS on Fargate"]
    C["Event Grid<br/>editorial"] --> D["API Gateway and SQS"]
    D --> O["Editorial adapter<br/>ECS on Fargate"]
    B --> E["Amazon MSK<br/>canonical events"]
    O --> E
    E --> F["Change Aggregator"]
    F --> G["DynamoDB<br/>update state"]
    G --> H["Step Functions<br/>edition workflow"]
    H --> I["S3<br/>versioned HTML"]
    H --> J["Neo4j<br/>versioned graph"]
    I --> K["TOC Generator"]
    J --> K
    K --> L["Validate and publish"]
    L --> M["Citation and link resolver"]
    M --> N["Neo4j and OpenSearch"]

The primary services are:

Responsibility AWS service
Run vendor and jurisdiction adapters Amazon ECS on AWS Fargate
Receive Event Grid webhooks Amazon API Gateway
Durable ordered event stream Amazon MSK
Accumulate changes and coordinate editions Amazon DynamoDB
Debounce short event bursts Amazon SQS delayed messages
Orchestrate a publication AWS Step Functions Standard Workflows
Store immutable HTML and TOC artifacts Amazon S3
Store the legal knowledge graph Neo4j deployed in an AWS Region
Resolve textual citation candidates Amazon OpenSearch Service
Publish domain events Amazon EventBridge
Handle failed asynchronous work Amazon SQS dead-letter queues
Encrypt keys and credentials AWS KMS and AWS Secrets Manager
Observe the platform Amazon CloudWatch and AWS X-Ray/OpenTelemetry

Connecting the External Vendors

The two source systems use different technologies and should not leak their formats into the domain model.

IBM MQ legislation feed

An adapter running on ECS Fargate connects to IBM MQ using the vendor client. Connectivity can use a site-to-site VPN or AWS Direct Connect when the queue manager is outside AWS.

The adapter:

  1. consumes an IBM MQ message;
  2. validates its source schema;
  3. maps source identifiers to canonical identifiers;
  4. publishes a canonical event to Amazon MSK;
  5. acknowledges the MQ message only after MSK confirms the write.

The adapter must assume redelivery. A stable source message ID or vendor event ID is therefore required for deduplication.

Microsoft Event Grid editorial feed

Event Grid can deliver events to an external HTTPS webhook, so Amazon API Gateway can expose the ingestion endpoint. Microsoft documents both webhook delivery and at-least-once delivery with retries.

API Gateway authenticates and rate-limits requests before placing accepted events on an SQS ingress queue. The editorial adapter consumes that queue and publishes canonical events to MSK. The endpoint must also implement Event Grid’s validation handshake.

Because Event Grid can redeliver an event, its event ID must remain part of the canonical envelope.

A Canonical Event Contract

Country and vendor adapters translate incoming records into a shared contract:

{
  "eventId": "evt-83471",
  "source": "EDITORIAL",
  "jurisdiction": "IE",
  "publicationId": "ie-oireachtas-2014-38",
  "editionId": "consolidated-2026-08-28",
  "sourceSequence": "9182",
  "changeType": "NODE_CHANGED",
  "affectedNodeIds": ["section-14"],
  "sourceObjectUri": "vendor://publication/42/edition/18",
  "occurredAt": "2026-08-28T15:30:12Z",
  "schemaVersion": 1
}

The adapters own:

  • transport integration;
  • source-schema validation;
  • country-specific parsing;
  • identifier normalization;
  • local hierarchy mapping;
  • citation extraction hints;
  • source provenance.

The core update workflow only understands the canonical model. Adding a country should require a new adapter and jurisdiction configuration, not new branches throughout the platform.

Why Amazon MSK Is the Main Event Backbone

Edition processing requires durable replay, partition ordering and multiple independent consumers. Amazon MSK provides managed Apache Kafka for that workload.

Canonical change events are keyed by:

jurisdiction + publicationId + editionId

All changes for one edition therefore enter the same Kafka partition and can be observed in order. Different editions remain independently processable. AWS also notes that an MSK source preserves ordering by limiting consumption to one consumer per partition where ordering is required (AWS documentation).

Suggested topics include:

legal.source-events
legal.edition-plans
legal.edition-events
legal.link-resolution
legal.unresolved-references
legal.dead-letter

Source sequence numbers are compared only within their own source feed. An IBM MQ sequence and an Event Grid sequence do not share a meaningful global order.

Accumulating Small Changes

An editorial operation may produce several events within seconds. Starting a publication workflow for each event would reproduce the original amplification problem.

The Change Aggregator consumes the canonical stream and writes an edition-update record to DynamoDB. Every incoming event:

  1. is deduplicated;
  2. adds its affected node IDs to the pending set;
  3. advances the source-specific high-water mark;
  4. moves the edition’s quiet-window deadline;
  5. sends a delayed check message to Amazon SQS.

SQS supports delayed delivery for up to 15 minutes, which is sufficient for a short debounce window (AWS documentation). When a delayed check arrives, the worker compares its deadline with the current DynamoDB deadline.

  • If a newer event extended the deadline, the message is stale and is ignored.
  • If the quiet window has expired, the aggregator freezes the pending changes into an immutable update plan.

No timer needs to be cancelled. Duplicate delayed messages are harmless.

DynamoDB as the Edition Coordinator

DynamoDB is not merely a cache in this design. It is the authoritative workflow and publication-state store.

A single-table layout could use:

PK = EDITION#<jurisdiction>#<publicationId>#<editionId>

with records such as:

Sort key Purpose
STATE Current edition state and active version
EVENT#<source>#<eventId> Event deduplication record
PLAN#<updateId> Frozen node-level update plan
STEP#<updateId>#CONTENT HTML writer status
STEP#<updateId>#GRAPH Neo4j writer status
STEP#<updateId>#TOC TOC Generator status
STEP#<updateId>#VALIDATION Validation result

The STATE record might contain:

{
  "activeVersion": 17,
  "pendingVersion": 18,
  "updateId": "update-173",
  "status": "STAGING",
  "lockVersion": 29,
  "quietUntil": "2026-08-28T15:31:00Z"
}

DynamoDB conditional writes ensure that only the worker holding the expected lockVersion can advance the edition. A conditional write succeeds only when its condition evaluates to true (AWS documentation).

DynamoDB transactions can atomically freeze the plan, change the edition state and record the workflow start inside DynamoDB. They cannot create a transaction spanning S3 and Neo4j—and the architecture does not pretend otherwise.

Creating the Node-Level Update Plan

The planner compares the incoming edition representation with the currently active version.

Every stable legal node carries independent hashes:

contentHash
metadataHash
hierarchyHash

This produces an immutable plan:

{
  "updateId": "update-173",
  "baseVersion": 17,
  "targetVersion": 18,
  "added": ["section-14A"],
  "contentChanged": ["section-14"],
  "metadataChanged": [],
  "moved": ["schedule-2"],
  "deleted": ["section-9"],
  "affectedTocRoots": ["part-3"],
  "planHash": "sha256:..."
}

Stable canonical node IDs are essential. A renumbered provision should be represented as a move or alias when its legal identity survives, rather than automatically being treated as an unrelated deletion and addition.

If a vendor supplies only a complete edition snapshot, the adapter may still download that snapshot to a staging area. The downstream plan should nevertheless contain only the differences.

Orchestrating the Edition Update

Each frozen plan starts an AWS Step Functions Standard Workflow named with the updateId.

Standard Workflows are appropriate because they are durable, auditable and follow an exactly-once workflow execution model unless retries are configured (AWS documentation). The individual writers must still be idempotent because retries, timeouts and lost responses remain possible at service boundaries.

flowchart TD
    A["Plan frozen"] --> B["Stage content"]
    A --> C["Stage graph"]
    B --> D{"Both staged?"}
    C --> D
    D -->|Yes| E["Generate TOC"]
    E --> F["Validate"]
    F -->|Valid| G["Publish"]
    F -->|Invalid| H["Keep current version"]
    G --> I["Resolve links"]

The workflow uses a parallel state for HTML and graph staging. It does not start the TOC Generator until both are available, because the TOC must be derived from one complete target hierarchy.

Storing Versioned HTML in Amazon S3

Each HTML node is written under an immutable edition prefix:

s3://legal-content/
  IE/
    ie-oireachtas-2014-38/
      consolidated-2026-08-28/
        version-18/
          nodes/section-14.html
          nodes/section-14A.html
          manifest.json

Unchanged nodes do not need to be copied if the edition manifest supports structural sharing and resolves them from the base version. Alternatively, a compaction process can materialize complete edition snapshots asynchronously for simpler serving.

S3 provides strong read-after-write consistency for object PUT and DELETE operations (AWS documentation). S3 Versioning can also retain object versions for recovery from accidental overwrite or deletion (AWS documentation).

The content writer records the object key, version ID, ETag and content hash in its Step Functions result and DynamoDB step record.

Neo4j remains the system of record for document metadata, hierarchies, editions, citations and editorial relationships. It can run in an AWS Region through Neo4j AuraDB where a managed service is acceptable, or as a customer-managed Neo4j Enterprise deployment when infrastructure ownership and network controls require it.

Because Neo4j is not an AWS service, the overall design is more accurately described as AWS-hosted with Neo4j, rather than entirely AWS-native. This does not change the publication model or the surrounding AWS services.

The graph distinguishes stable legal identities from versioned representations:

flowchart TD
    W["LegalWork"] -->|HAS_EDITION| E["Edition v18"]
    E -->|CONTAINS| P["Provision version"]
    P -->|VERSION_OF| I["Stable provision identity"]
    P -->|PARENT_OF| C["Child provision"]
    M["Commentary"] -->|ANNOTATES| I
    P -->|CITES| X["Target identity"]

Version 18 nodes and relationships are inserted with publicationStatus = STAGED. Existing version 17 graph elements remain untouched. The Graph Writer performs the changes for one update plan in one or more bounded Neo4j transactions, depending on the number of affected nodes.

Every write is idempotent on updateId + targetVersion + canonicalNodeId. Neo4j constraints should enforce the uniqueness of stable legal identities and versioned node identities. A transaction inside Neo4j protects graph mutations, but it does not extend to S3 or DynamoDB.

The Separate TOC Generator

The TOC Generator remains an independent microservice running on ECS Fargate.

It receives:

{
  "updateId": "update-173",
  "publicationId": "ie-oireachtas-2014-38",
  "editionId": "consolidated-2026-08-28",
  "targetVersion": 18,
  "graphSnapshot": "version-18",
  "affectedTocRoots": ["part-3"]
}

It reads the staged hierarchy, regenerates only affected TOC subtrees and writes an immutable TOC artifact to S3:

toc/version-18/toc.json

Its idempotency key is:

updateId + targetVersion

The same request must always produce or return the same staged artifact. The TOC Generator does not publish the edition; it only reports completion and its output hash.

Validation Before Publication

The validator checks the target version across all representations:

  • every TOC entry resolves to a staged or inherited provision;
  • every visible provision requiring content resolves to an HTML object;
  • every parent-child edge is valid;
  • no active relationship targets a tombstoned node;
  • node order is complete and deterministic;
  • content and metadata hashes match the frozen plan;
  • S3, Neo4j and TOC artifacts all carry the same updateId and target version;
  • the edition has exactly one root hierarchy;
  • stable node IDs are unique within their legal scope.

Validation produces a signed or hashed report stored in S3 and referenced from DynamoDB.

The Publication Barrier

There is no distributed transaction across DynamoDB, S3 and Neo4j. Instead, the system provides atomic visibility through one small manifest record.

Readers begin every request by resolving:

{
  "publicationId": "ie-oireachtas-2014-38",
  "editionId": "consolidated-2026-08-28",
  "activeVersion": 18,
  "publishedByUpdateId": "update-173",
  "publicationStatus": "CORE_PUBLISHED"
}

The workflow changes activeVersion from 17 to 18 using a DynamoDB conditional update:

Publish only if:
activeVersion = 17
pendingVersion = 18
updateId = update-173
validationStatus = PASSED

Until that single conditional update succeeds, readers continue using version 17 from every store. After it succeeds, they use version 18 from every store.

This is not atomic storage. It is atomic publication, which is the guarantee the customer actually needs.

Publication emits an EditionPublished event through Amazon EventBridge and Amazon MSK:

{
  "publicationId": "ie-oireachtas-2014-38",
  "editionId": "consolidated-2026-08-28",
  "version": 18,
  "addedNodeIds": ["section-14A"],
  "modifiedNodeIds": ["section-14"],
  "deletedNodeIds": ["section-9"],
  "movedNodeIds": ["schedule-2"]
}

The Link Resolver then:

  1. extracts citations from changed HTML nodes;
  2. parses them using jurisdiction-specific rules;
  3. queries canonical identifiers and aliases in Neo4j;
  4. uses OpenSearch for names, abbreviations and textual candidate retrieval;
  5. ranks candidates deterministically;
  6. writes versioned relationships to Neo4j;
  7. finds inbound relationships affected by deleted or moved nodes;
  8. re-resolves only that impact set.

OpenSearch remains a derived candidate-search index rather than the source of truth. Neo4j owns resolved relationships, their temporal validity, provenance and resolution status.

Every relationship records:

source text
source vendor
resolver version
confidence
validFrom and validTo
resolution status

Possible statuses are:

UNRESOLVED
AMBIGUOUS
RESOLVED
STALE
INVALID

Low-confidence or ambiguous references enter an SQS review queue. They are never silently converted into authoritative legal relationships.

Material Arriving Before Its Target

Commentary and news can arrive before the legislation they reference.

The resolver stores the source content and creates an unresolved-reference record containing its normalized clues. When a new legal work, alias or provision is published, the resolver searches the unresolved index for possible matches and retries only relevant references.

This prevents data loss without delaying publication of either document.

Deletions and Historical Editions

A deleted provision should initially become a tombstone rather than being physically removed.

The tombstone records:

  • the last active edition;
  • the effective end date;
  • whether the provision was repealed, renumbered or replaced;
  • successor and predecessor identities;
  • the update that created the tombstone.

Historical editions continue pointing to their historical provision versions. Current-edition traversal excludes inactive nodes. Physical cleanup, if legally permitted, occurs only after the configured retention period.

Failure Handling

Failure System response
Duplicate MQ or Event Grid event Ignore using the source event ID
Event arrives after the plan is frozen Add it to the next edition update
S3 succeeds but Neo4j fails Keep the current edition active and retry Neo4j
TOC generation fails Keep the target version staged and retry the TOC step
Validation fails Mark the update failed; do not change activeVersion
Workflow response is lost Read DynamoDB state using the updateId
Publication request is repeated Conditional update returns the existing outcome
Citation target is missing Store the unresolved reference and retry later
Link resolver fails Send work to an SQS DLQ without rolling back the edition
Staged update is abandoned Expire it through a controlled cleanup workflow

Step Functions redrive can resume an unsuccessful Standard Workflow while preserving the results of successful steps, which is useful for operational recovery (AWS documentation).

Scaling Across Countries

The platform scales along natural boundaries:

  • MSK partitions distribute editions independently.
  • ECS services scale by queue depth and consumer lag.
  • S3 scales independently for HTML and TOC artifacts.
  • Neo4j handles graph traversal and relationship storage.
  • OpenSearch provides jurisdiction-aware textual candidate lookup.
  • Step Functions isolates every frozen edition update as an auditable execution.

Jurisdiction adapter packages contain:

identifier rules
hierarchy mappings
citation grammars
language analyzers
effective-date rules
source precedence
vendor schema mappings

A country is onboarded through its adapter, configuration and conformance tests. The edition workflow itself remains unchanged.

For data residency requirements, jurisdictions can be assigned to regional deployment cells. Each cell owns its MSK cluster, DynamoDB tables, S3 buckets, Neo4j deployment and OpenSearch domain. A global catalogue contains only the identifiers and routing metadata permitted to cross regional boundaries.

Security and Auditability

Legal content needs a complete chain of custody.

The system should retain:

  • the original vendor event in S3;
  • its canonical transformation;
  • the adapter and schema versions;
  • the frozen update plan;
  • hashes of input and output artifacts;
  • Step Functions execution history;
  • validation reports;
  • the identity that performed manual resolution;
  • the update responsible for every graph relationship.

S3 Object Lock can be considered for records that must be retained in a write-once model. AWS KMS encrypts AWS-managed data stores, and Neo4j encryption must be configured according to the selected deployment model. Secrets Manager stores vendor and Neo4j credentials, while private networking keeps service traffic off the public internet where supported.

What This Design Deliberately Avoids

The architecture does not attempt to:

  • run a distributed ACID transaction across all stores;
  • rebuild an edition for every event;
  • update published graph nodes in place;
  • block core publication on every external citation;
  • assume source events arrive exactly once;
  • compare unrelated vendor sequence numbers;
  • use an LLM as the authoritative citation resolver;
  • embed every country’s rules in the core workflow.

Final Design Principle

The architecture can be summarized in one sentence:

Amazon MSK and DynamoDB turn heterogeneous vendor events into an immutable edition plan; Step Functions stages versioned HTML, graph and TOC results; and one conditional publication pointer exposes the complete edition before asynchronous services enrich its wider legal relationships.

The most important decision is not the choice between S3, Neo4j or DynamoDB.

It is recognizing that storage completion and business publication are different events.

Once that boundary is explicit, the platform can update a single paragraph without rebuilding an entire act, preserve historical legal meaning and grow across jurisdictions without allowing its content, navigation and knowledge graph to drift apart.