# 5-Tier Medallion Architecture > Data platform layers optimized for autonomous clinical AI: Bronze raw logs to Platinum agent control states - **Category**: scaling-agentic-ai-healthcare-lakehouse - **Canonical URL**: https://designpattern.fyi/scaling-agentic-ai-healthcare-lakehouse/5-tier-medallion-architecture/ --- ## Description Data platform layers optimized for autonomous clinical AI: Bronze raw logs to Platinum agent control states ## Additional Notes # The 5-Tier Medallion Architecture for Clinical AI The classic three-tier Medallion pattern (Bronze-Silver-Gold) is extended by two additional tiers — **Diamond** and **Platinum** — that represent a paradigm shift:

Data is no longer prepared exclusively for human analysts. It must also be optimized for autonomous AI systems that read, reason, and act.

--- ## Bronze Layer — Raw Append-Only Ledger The Bronze Layer is the immutable landing zone for all raw hospital data streams. It captures: - Device telemetry, EHR event payloads, ADT messages, DICOM metadata, lab instrument feeds, pharmacy dispensing events, and third-party data connectors. - **Exactly as generated — no transformation, no filtering.** ### Apache-Powered Execution

🚀 Apache Kafka

High-throughput, low-latency event streaming from edge devices and HL7 FHIR API sources. Partitioning by department enables parallel consumption. Modern deployments use KRaft mode, eliminating ZooKeeper for simplified operations.

⚡ Apache Flink / Spark Structured Streaming

Micro-batch ingestion jobs consume from Kafka topics and append raw JSON/HL7 payloads as immutable Parquet/Iceberg files on the object store, with exactly-once processing semantics.

💾 Apache Hadoop HDFS / S3-compatible Object Store

Acts as the underlying distributed storage layer. All raw files are append-only. Portable across AWS S3, Azure ADLS Gen2, GCS, and on-premises MinIO/Ceph via the S3A connector.

🧊 Apache Iceberg Table Format

Provides ACID-compliant snapshot isolation, time-travel queries, and schema evolution without rewriting historical data.

🔒 Optimistic Concurrency Control (OCC)

Implemented via Iceberg's transaction model to handle thousands of simultaneous writer streams without lock contention.

🛡️ On-Premise PII Redaction Sidecar

A Gemma or lightweight open-weight LLM runs as a sidecar to the Flink/Spark ingestion consumers, so redaction happens inline before each record is committed to Iceberg — no raw text field is ever written to durable storage unredacted. See Data Protection Enforcement below for what the redaction step itself covers.

### Kubernetes Deployment Kafka brokers run as StatefulSets with persistent volume claims (PVCs) on NVMe storage, managed by the Strimzi Kafka Operator. Spark Structured Streaming applications run as Kubernetes Deployments via the Spark Operator, auto-scaling executor pods based on Kafka consumer lag metrics exposed through KEDA. ### Data Protection Enforcement

🛡️ PII Redaction at the Edge

A Gemma or lightweight open-weight LLM sidecar container performs permanent, irreversible PII Redaction on raw text fields before any payload leaves the hospital network perimeter. This is the first line of defense — direct identifiers (names, SSNs, MRNs) are stripped at ingestion time using NER-based detection.

🔒 Masking for Device Metadata

Masking is applied to device telemetry metadata fields (serial numbers, location codes) that could serve as quasi-identifiers. Format-preserving placeholders maintain structural consistency for downstream parsing without exposing device-to-patient mappings.

### Rationale

Infinite replayability and absolute audit lineage. If a downstream parsing rule fails, the platform replays data from any historical snapshot without hitting physical edge devices again.

--- ## Silver Layer — Conformed Operational State The Silver Layer cleans, normalizes, deduplicates, and structures raw Bronze data into an enterprise-wide schema.

Key characteristics:

Single source of truth for operational and analytical workloads
De-identification enforced here — all PHI has been stripped at the edge
Silver enforces the schema contract and serves as the platform compliance boundary

### Apache-Powered Execution

⚙️ Apache Spark ELT Jobs

Parse raw JSON/HL7/FHIR payloads into structured Parquet tables with Iceberg's schema enforcement. Delta detection via MERGE INTO operations deduplicates late-arriving events.

📋 Apache Hive Metastore

Centralized schema registry for all Silver tables, consumed by Spark, Flink, Trino, and ML pipelines uniformly across clouds.

🔐 Apache Ranger

Row-level and column-level access policies enforce multi-tenant isolation using Attribute-Based Access Control (ABAC). Partition key (/TenantID/DepartmentID) ensures strict isolation.

🔍 Apache Atlas

Tracks full data lineage from Bronze ingest through every Spark transformation. PHI classification tags propagate automatically to all derived columns.

📝 CQRS Write-Side Pattern

The Silver layer operates as the Command (Write) side. Long-term clinical states are written to Iceberg tables; operational real-time states are written to a low-latency distributed database (Cassandra or CockroachDB/YugabyteDB).

### Data Protection Enforcement

✂️ PII Redaction Pipeline

PII Redaction — permanent, irreversible removal of all direct identifiers — is enforced at the Silver boundary via NER-based detection (Microsoft Presidio, spaCy, or Google Sensitive Data Protection). Every record entering Silver passes through the redaction pipeline; no raw patient names, SSNs, emails, or phone numbers survive.

🔒 Anonymization

Anonymization via generalization (age→age bands, zip→region prefix) and suppression (removing quasi-identifiers) is applied to structured fields, then checked against the re-identification risk test set out in GDPR Recital 26 — whether re-identification is achievable by "means reasonably likely to be used." The bar for that test is deliberately conservative: research on quasi-identifiers has shown that age, zip code, and gender alone can uniquely re-identify a large majority of the US population, which is why all three are generalized or suppressed rather than just one.

🏷️ Pseudonymization

Pseudonymization replaces patient identifiers with HMAC-based artificial aliases under strict cryptographic governance. Keys are stored separately in a dedicated secrets manager — never in the same database as pseudonymized data. This enables longitudinal tracking across encounters without exposing identity.

### Kubernetes Deployment Spark ELT jobs are managed by Apache Airflow's KubernetesExecutor — each DAG task spawns a dedicated Kubernetes pod, providing complete workload isolation. Hive Metastore runs as a Kubernetes Deployment backed by a PostgreSQL StatefulSet. --- ## Gold Layer — Dimensional Analytics The Gold Layer transforms relational Silver data into highly aggregated, domain-specific KPIs and dimensional models.

Designed for human consumption via:

BI dashboards
Executive reporting
SQL-based operational analytics

### Apache-Powered Execution

📊 Apache Spark SQL

Executes dimensional modeling transformations — building Star Schemas, Fact tables (patient encounters, device events, claims) and Dimension tables.

🧹 Apache Iceberg OPTIMIZE + VACUUM

Automated Spark jobs compact small files and expire old snapshots on a 15-minute schedule, maintaining fast OLAP query performance.

🔍 Apache Trino

Federated SQL query engine deployed on Kubernetes, enabling sub-second analytical queries across Gold Iceberg tables without moving data.

📈 Apache Superset

Open-source BI dashboarding deployed on Kubernetes, connecting to Trino for interactive dashboards — a cloud-agnostic alternative to proprietary tools.

⏰ Apache Airflow

Orchestrates Gold transformation DAGs, SLA monitoring, and automated data quality checks using Great Expectations.

### Data Protection Enforcement

🎭 Masking

Masking applies format-preserving placeholders to any residual analytics fields that need structural consistency. Insurance member IDs become ****-****-1234, partial dates preserve month granularity. Static masking enables consistent downstream lookups in star-schema rollups.

🔑 Tokenization

Tokenization swaps any residual identifiers with opaque, vault-backed tokens. The vault (HashiCorp Vault / AWS Secrets Manager) is the only authoritative mapping — tokens carry no structural resemblance to originals. Every detokenize operation is logged for SOC 2 accountability.

🏷️ Pseudonymization

Pseudonymization via deterministic HMAC enables cross-record joins in dimensional models without exposing identity. The same patient across encounters resolves to the same pseudonym — enabling ALOS, readmission, and cost-per-case analytics — while the re-identification key stays locked in a separate secrets manager.

Why Gold gets "Full Access": By the time data reaches Gold, it has been through PII Redaction at Bronze, Anonymization + Pseudonymization at Silver, and Masking + Tokenization in the Silver→Gold ELT pipeline. The KPIs, star-schema rollups, and dimensional models served from Gold contain zero raw PII — that's why downstream consumers (including Claude Cowork) can access Gold data freely.

### Rationale

All clinical KPIs are served from Gold:

ALOS, readmission rates, device utilization, cost per case
Business executives, department heads, and quality teams access long-term operational trends without incurring heavy compute costs on raw transactional tables

--- ## Diamond Layer — Semantic Spine & Knowledge Store The Diamond Layer converts structured enterprise data and unstructured clinical assets into formats optimized for Large Language Models and ML inference. It serves as the organization's Feature Store, RAG knowledge engine, and Ontological GraphRAG Gateway. ### Apache-Powered Execution

🧠 Apache Spark NLP / Apache OpenNLP

Extract semantic chunks from clinical text at scale. Named Entity Recognition (NER) tags clinical entities (diagnosis, medication, procedure) before embedding.

➡️ Apache Arrow

High-performance in-memory columnar format used for zero-copy transfer of feature vectors between Spark feature computation and the vector store ingestion pipeline.

🍽️ Feast (Feature Store)

Manages clinical ML features — patient risk scores, device health indices, lab trend vectors — with versioned, point-in-time-correct retrieval.

🔍 Weaviate / Qdrant / Milvus

Vector databases store high-density embeddings. Hybrid Search combines dense vector similarity with BM25 sparse lexical matching, merged via Reciprocal Rank Fusion (RRF).

🤖 Embedding Models

Deployed as Kubernetes services using vLLM or Triton Inference Server, supporting BioBERT, Clinical-BERT, or BioMedLM models.

🕸️ Temporal Knowledge Graph (Neo4j / Apache AGE)

Maps absolute relational concepts. Nodes represent patients, specimens, or facilities; edges represent deterministic medical relationships and clinical timelines.

🏷️ Unified Ontological Mesh

Every ingested chunk is deterministically tagged against SNOMED-CT (clinical terms), LOINC (laboratory observations), ICD-11 (disease classification), RxNorm (pharmaceuticals), and HGNC (genomic nomenclature).

📊 Apache Iceberg (Feature History Table)

Maintains a versioned history of all feature values for offline ML training, enabling time-travel audits.

### Data Protection Enforcement

✂️ PII Redaction on Clinical Text

PII Redaction via production-grade NER models (BioBERT, Clinical-BERT, Microsoft Presidio) strips all direct identifiers from clinical notes, radiology reports, and pathology narratives before they enter the chunking and embedding pipeline. Regex alone cannot reliably detect names, organizations, or context-sensitive PII — NER is required.

🔒 Anonymization on Structured Fields

Anonymization via generalization and suppression is applied to structured metadata accompanying embedded chunks — patient demographics, encounter dates, and location fields are generalized (age→decade bands, dates→month precision, locations→region level) before they become vector-store metadata.

🧪 Synthetic Data for Testing

Synthetic Data generation (via Faker, SDV/CTGAN, or LLM-based schema synthesis) produces statistically faithful test datasets with zero real PII for prompt engineering, RAG evaluation, embedding quality testing, and QA against the Diamond index. If the purpose is testing — don't use real data at all.

Why Diamond gets "Full Access": The Diamond layer's document library and vector index contain only de-identified clinical text (NER-redacted), anonymized structured metadata, and synthetic test data. Protocols, guidelines, and SOPs are inherently non-PHI documents. This is why Claude Cowork can freely organize, deduplicate, and curate Diamond content — there is no recoverable PII to protect against.

### Rationale

Diamond is built for machine consumption. Passing relational rows or star schemas directly to an LLM context window produces high token overhead, hallucinations, and logic errors. Decoupling relational database design from semantic reasoning lowers token costs, improves response precision, and enables deterministic RAG retrieval.

--- ## Platinum Layer — Agent Control Plane, State Ledger & xAI Runtime The Platinum Layer is the operational hub for all autonomous multi-agent systems. It governs how AI agents interact with the data platform, enforces compliance boundaries, manages memory states, and provides an immutable audit log. It is also the primary Explainable AI (xAI) runtime, where every reasoning step, memory access, and tool invocation is captured as a verifiable, human-readable execution trajectory. ### Apache-Powered Execution

📡 Apache Kafka (Agent Event Bus)

All agent tool calls, LLM responses, Chain-of-Thought (CoT) thinking traces, reflection loops, and state transitions are published as structured events to a dedicated Kafka topic.

🧊 Apache Iceberg (Agent Audit Ledger)

Every Kafka agent event is micro-batched into an append-only Iceberg table. Compliance teams and clinicians can query execution trajectories with full time-travel capability.

📦 Minimal Explanation Packet (MEP)

Produces a structured MEP containing: (1) trace ID, (2) memory reads with similarity scores, (3) extracted CoT steps, (4) tool call I/O, and (5) post-generation critique results.

⚡ Apache Flink CEP Engine

Complex Event Processing on the agent event stream detects anomalous behavior (excessive calls, goal drift, policy violations) and triggers automated circuit breakers.

📋 Tool Contract Registry

A versioned registry of all MCP tool contracts (OpenAPI schemas). Agents reference tool versions by contract ID, ensuring deterministic behavior and auditability.

💬 Multi-Agent Communication

NATS JetStream is the underlying message transport — the durable queue that carries supervisor/worker handoffs and distributed task assignments between agent pods. A2A (below) sits above it as the protocol layer: agent discovery, capability negotiation, and task semantics. NATS moves the bytes; A2A defines what they mean.

🔍 OpenTelemetry Collector

Traces every agent interaction across microservices — LLM calls, tool invocations, memory accesses — exporting to Tempo for visualization in Grafana.

🛡️ Cross-Cutting Guardrails

NeMo GuardRails runs as a sidecar container, enforcing content safety, PHI detection, prompt injection protection, and hallucination detection before output delivery.

📖 CQRS Read-Side (PostgreSQL + Redis)

Serves the read side of agent state for rapid HITL checkpoint querying, session replay UI, and operational dashboards.

### Data Protection Enforcement

🔑 Tokenization in Audit Ledger

All patient references in the Platinum audit ledger use vault-backed opaque tokens — no raw patient identifiers appear in execution traces, MEPs, or CoT logs. The token vault is isolated from the ledger itself, with every detokenize operation independently logged for SOC 2 accountability.

📊 Differential Privacy on Aggregates

Differential Privacy adds mathematically calibrated noise to aggregated metrics exported from the Platinum layer (agent performance statistics, population-level outcome correlations). This provides formal privacy guarantees that satisfy GDPR Recital 26's "all means reasonably likely" standard.

### Agent Gateway, MCP & A2A — Agentic Connectivity Layer The Platinum layer's multi-agent ecosystem requires a unified connectivity fabric that manages all agentic traffic — LLM calls, tool invocations, agent-to-agent messaging, and HTTP API requests — through a single, auditable data plane.

🌐 Agent Gateway

Agent Gateway is the open-source gateway for AI traffic — LLM, MCP, A2A, and HTTP in one data plane. In the healthcare lakehouse, it serves as the unified connectivity layer that routes all agentic traffic through centralized policy enforcement, rate limiting, and observability. Deployed as a Kubernetes service, Agent Gateway provides a single control point for managing which agents can call which LLMs, invoke which MCP tools, and communicate with which other agents — with full audit logging to the Platinum Iceberg ledger.

🔗 MCP (Model Context Protocol)

MCP provides standardized tool connectivity for agents — wrapping SMART on FHIR endpoints, internal APIs, and external services into a uniform tool interface. Agent Gateway provides MCP server management and routing, enabling clinical agents to discover and invoke tools across departments through a single gateway. The Tool Contract Registry (Platinum) versions every MCP schema, ensuring deterministic behavior and auditability.

🤝 A2A (Agent-to-Agent Protocol)

Google's open Agent-to-Agent (A2A) protocol enables structured inter-agent communication with agent discovery, capability negotiation, and task delegation. In the Platinum layer, A2A enables cross-department agent collaboration — a pharmacy agent delegating a drug-interaction check to a lab agent, or a triage agent escalating a STEMI case to the cardiology supervisor agent — with Agent Gateway managing discovery, routing, and access control.

The Agent Connectivity Stack: Agent Gateway + MCP + A2A together form a complete agentic connectivity layer — Agent Gateway is the data plane (routing, policy, observability), MCP is the tool interface (how agents invoke capabilities), and A2A is the communication protocol (how agents talk to each other). All three are open-source, and all traffic flows through the same audit pipeline that feeds the Platinum Iceberg ledger.

### Rationale

Platinum provides the operational control plane that makes Agentic AI safe, auditable, and explainable in clinical settings. Decoupling agent state from individual instances and persisting all execution events to an immutable Iceberg ledger guarantees full traceability for regulatory compliance.

--- ## Claude Cowork: Desktop Productivity Layer Claude Cowork is Anthropic's desktop agentic system for knowledge work — file organization, report assembly, spreadsheet analysis, and document synthesis. It operates on local files, folders, and connected applications, completing multi-step tasks with user-defined goals and approval gates. ### Critical Compliance Boundary

Important: Cowork is explicitly excluded from Anthropic's Business Associate Agreement (BAA) coverage. This is a standing exclusion, not a gap that will be resolved with future features. Cowork must never see PHI under any circumstance — only de-identified or aggregated data from Gold/Diamond layers.

### Cowork's Role at Each Medallion Layer

Layer	Cowork Access	What It Actually Does
Bronze	No	Raw device telemetry and PHI-bearing streams never enter a Cowork-accessible folder.
Silver	Indirectly, on exports only	Data stewards export de-identified schema samples; Cowork checks for null-policy violations, format drift, and flags anomalies. Never has direct folder/connector access to live Silver stores.
Gold	Yes, heavily	Transforms exported Gold-layer CSV/Excel extracts (KPIs, star-schema rollups) into narrative reports, slide decks, and recurring briefings for department analysts and quality officers.
Diamond	Yes, as curation assistant	Before documents (protocols, guidelines, SOPs) are embedded into the vector index, Cowork organizes, deduplicates, standardizes formatting, and flags stale versions. Does not run the embedding pipeline itself.
Platinum	Documentation only	Drafts and versions tool-contract documentation, Supervisor-Critic evaluation rubrics, and audit-report summaries from exported ledger extracts — never executes governed clinical actions.

### Foundation Setup Requirements Before any department uses Cowork, implement these platform-wide controls: 1. **Internal MCP Connector Strategy** — Build one internal MCP connector exposing read-only endpoints against Gold and Diamond layers (e.g., `get_kpi_export(department, date_range)`, `search_guidelines(query, department)`). This connector calls through existing Apache Trino/vector-search services and inherits Apache Ranger's row/column/department policies. 2. **Department Folder Scoping** — Create shared-drive folder trees per department containing only: - Weekly/monthly Gold KPI exports (CSV/XLSX) written by existing Airflow DAGs - Department document libraries (protocols, guidelines, policy PDFs) - A `working/` subfolder Cowork is allowed to write into 3. **Private Plugins per Department** — Bundle skills (templates, tone, department conventions), connectors, and sub-agents into one install per department so users get context-specific tools by default. 4. **Approval-Required Default** — Keep approval-before-action on for anything leaving the department. Only relax for narrowly scoped, non-PHI, fully reversible tasks. 5. **BAA Boundary as Hard Constraint** — Treat the BAA exclusion as a design constraint, not a hygiene issue. Cowork must never see PHI — not "PHI with compensating controls," not "PHI with extra logging." The de-identification must be complete and verified before data lands in any Cowork-accessible folder. 6. **SIEM Integration** — Route Cowork's OpenTelemetry activity stream (tool calls, file access, approval states) into existing SIEM/GRC tooling and review on the same cadence as other access logs. Note that this doesn't resolve the BAA coverage question — it's for operational monitoring only. --- ## Edge De-identification & Multi-Tiered Memory When orchestrating agents across a multi-tenant enterprise ecosystem, real-time compliance is managed by containerized Pre-LLM and Post-LLM Hooks. Deployed as sidecars or microservices, these hooks intercept payloads directly at the MCP Protocol Layer boundary: - **PII Redaction**: Executes permanent, irreversible erasure of direct identifiers from unstructured text inputs during Pre-LLM ingestion. - **Tokenization**: Swaps sensitive clinical fields with format-preserving random tokens, managed through an isolated, vault-backed microservice restricted by OPA sidecars. - **Pseudonymization**: Replaces patient identifiers with artificial aliases under strict cryptographic governance, allowing longitudinal tracking without exposing identity. - **Masking**: Applies structurally consistent placeholders to preserve database formats for non-production environments. - **Anonymization**: Performs irreversible data destruction on demographics, protecting downstream aggregated research datasets. - **Synthetic Data Generation**: Programmatically outputs statistically faithful, zero-PII datasets to execute prompt engineering and QA testing safely. ### Multi-Tiered Persistent Memory Architecture To execute complex, long-running clinical workflows across shifts, sessions, and physical locations, agents utilize a multi-tiered memory infrastructure running on Kubernetes: - **Working Memory**: Fast, ephemeral workspace mapped to the agent pod’s active in-context window. Tracks active conversation threads and short-term reasoning traces. - **Episodic Memory**: A timestamped, persistent record log stored in a distributed document tier, preserving clinical actions across session boundaries. - **Semantic Memory**: Generalized organizational rules, local medical knowledge bases, and clinician preferences, kept factually consistent over time. - **Procedural Memory**: Enforced standard operating procedures (SOPs), specialized clinical routing rules, and skill sets (e.g. SKILL.md libraries) stored as config maps or mounted volumes. - **Vector & Graph Stores**: Milvus/Qdrant databases execute semantic similarity searches, while temporal graph databases (Neo4j) manage entity relationships and multi-hop reasoning.