What is a Medallion Architecture? (2024)

What is a medallion architecture?

A medallion architecture is a data design pattern used to logically organize data in a lakehouse, with the goal of incrementally and progressively improving the structure and quality of data as it flows through each layer of the architecture (from Bronze ⇒ Silver ⇒ Gold layer tables). Medallion architectures are sometimes also referred to as "multi-hop" architectures.

What is a Medallion Architecture? (1)

Big Book of Data Engineering: 2nd EditionThe latest technical guidance for building real-time data pipelinesDownload nowThe Data Lakehouse Platform for Dummies How to power all your data and AI on one platformGet the eBookLakehouse FundamentalsGet up to speed on Lakehouse by taking this free on-demand trainingStart now

Building data pipelines with medallion architecture

Databricks provides tools like Delta Live Tables(DLT) that allow users to instantly build data pipelines with Bronze, Silver and Gold tables from just a few lines of code. And, with streaming tables and materialized views, users can create streamingDLT pipelines built on Apache Spark™️ Structured Streaming that are incrementally refreshed and updated. For more details, see Databricks documentation on combining streaming tables and materialized views in a single pipeline.

Bronze layer (raw data)

The Bronze layer is where we land all the data from external source systems. The table structures in this layer correspond to the source system table structures "as-is," along with any additional metadata columns that capture the load date/time, process ID, etc. The focus in this layer is quick Change Data Capture and the ability to provide an historical archive of source (cold storage), data lineage, auditability, reprocessing if needed without rereading the data from the source system.

Silver layer (cleansed and conformed data)

In the Silver layer of the lakehouse, the data from the Bronze layer is matched, merged, conformed and cleansed ("just-enough") so that the Silver layer can provide an "Enterprise view" of all its key business entities, concepts and transactions. (e.g. master customers, stores, non-duplicated transactions and cross-reference tables).

The Silver layer brings the data from different sources into an Enterprise view and enables self-service analytics for ad-hoc reporting, advanced analytics and ML. It serves as a source for Departmental Analysts, Data Engineers and Data Scientists to further create projects and analysis to answer business problems via enterprise and departmental data projects in the Gold Layer.

In the lakehouse data engineering paradigm, typically the ELT methodology is followed vs. ETL - which means only minimal or "just-enough" transformations and data cleansing rules are applied while loading the Silver layer. Speed and agility to ingest and deliver the data in the data lake is prioritized, and a lot of project-specific complex transformations and business rules are applied while loading the data from the Silver to Gold layer. From a data modeling perspective, the Silver Layer has more 3rd-Normal Form like data models. Data Vault-like, write-performant data models can be used in this layer.

Gold layer (curated business-level tables)

Data in the Gold layer of the lakehouse is typically organized in consumption-ready "project-specific" databases. The Gold layer is for reporting and uses more de-normalized and read-optimized data models with fewer joins. The final layer of data transformations and data quality rules are applied here. Final presentation layer of projects such as Customer Analytics, Product Quality Analytics, Inventory Analytics, Customer Segmentation, Product Recommendations, Marking/Sales Analytics etc. fit in this layer. We see a lot of Kimball style star schema-based data models or Inmon style Data marts fit in this Gold Layer of the lakehouse.

So you can see that the data is curated as it moves through the different layers of a lakehouse. In some cases, we also see that lot of Data Marts and EDWs from the traditional RDBMS technology stack are ingested into the lakehouse, so that for the first time Enterprises can do "pan-EDW" advanced analytics and ML - which was just not possible or too cost prohibitive to do on a traditional stack. (e.g. IoT/Manufacturing data is tied with Sales and Marketing data for defect analysis or health care genomics, EMR/HL7 clinical data markets are tied with financial claims data to create a Healthcare Data Lake for timely and improved patient care analytics.)

Benefits of a lakehouse architecture

  • Simple data model
  • Easy to understand and implement
  • Enables incremental ETL
  • Can recreate your tables from raw data at any time
  • ACID transactions, time travel

A quick primer on lakehouses

A lakehouse is a data platform architecture paradigm that combines the best features of data lakes and data warehouses. A modern lakehouse is a highly scalable and performant data platform hosting both raw and prepared data sets for quick business consumption and to drive advanced business insights and decisions. It breaks data silos and allows seamless, secure data access to authorized users across the enterprise on one platform.

What is a Medallion Architecture? (5)

Medallion architecture and data mesh

The Medallion architecture is compatible with the concept of a data mesh. Bronze and silver tables can be joined together in a "one-to-many" fashion, meaning that the data in a single upstream table could be used to generate multiple downstream tables.

[ Try Databricks for free today ]

Additional Resources

  • What is a Data Lakehouse?
  • Data Lakehouse Platform by Databricks
  • How Incremental ETL Makes Life Simpler With Data Lakes - Databricks Blog
  • Simplify Your Medallion Architecture with Delta Lake's CDF Feature

Back to Glossary

What is a Medallion Architecture? (2024)

FAQs

What is a medallion in architecture? ›

A medallion is a round or oval ornament that frames a sculptural or pictorial decoration in any context, but typically a façade, an interior, a monument, or a piece of furniture or equipment.

What is a medallion design? ›

an oval or circular design, portrait, relief carving, etc. resembling a medal in shape and used as a decorative form in architecture, textiles, etc.

What are the benefits of medallion architecture? ›

This architecture guarantees atomicity, consistency, isolation, and durability as data passes through multiple layers of validations and transformations before being stored in a layout optimized for efficient analytics.

What is medallion architecture in Azure? ›

The Medallion architecture is compatible with the concept of a data mesh. Bronze and silver tables can be joined together in a "one-to-many" fashion, meaning that the data in a single upstream table could be used to generate multiple downstream tables.

What is the purpose of a medallion? ›

A medallion signature guarantee is a special stamp that's used when you transfer securities. It confirms that the signature authorizing the transfer is genuine and that the signer has the legal capacity and authority to sign the document.

What do medallions represent? ›

Designers are donning their own medallions to invoke protection, guidance, luck, or as reminders of where they're going or what they've overcome. “I love protective medallions because it reminds us to have inner strength and that we are warriors,” said Joy Smith, designer of Communion by Joy.

Who invented medallion architecture? ›

The term “Medallion Data Architecture” was first coined by Databricks where they describe data at different stages of processing as being “bronze,” “silver” or “gold” level data. The term has since been used by Microsoft to describe stages of processing within Microsoft Fabric.

Are medallions in style? ›

One such element is the ceiling medallion, a decorative piece that has been used for centuries to add elegance and grandeur to interior spaces. In recent years, antique ceiling medallions have made a comeback in modern interior design, bringing a touch of old-world charm to contemporary spaces.

What is the medallion architecture in snowflake? ›

The Medallion architecture consists of three main layers: Bronze, Silver, and Gold. It aims to incrementally and progressively improve the structure and quality of data as it flows through each layer of the architecture.

What are the cons of medallion architecture? ›

Medallion Architecture Disadvantages

Medallion architecture provides a framework for data cleaning, not data architecture. Uses large amounts of storage: though, as many have proclaimed, "storage is cheap," a medallion architecture effectively triples the amount of storage used in a data lake.

What is the benefit of medallion status? ›

Flight Benefits

With Medallion Status, you're eligible for Unlimited Complimentary Upgrades for yourself and a companion to Delta Comfort+® and First Class — anywhere the products are sold.

What is the benefits of an architect? ›

Pros and Cons of Architecture:
Pros of Being an ArchitectCons of Being an Architect
Collaborative EndeavorsBalancing Art and Budget
Satisfaction in Helping Clients Resolve ChallengesThe Stress of Meeting Client Expectations
Opportunities for Innovation and Technological GrowthChallenges in Balancing Work-Life
5 more rows
Nov 12, 2023

What is landing zone in medallion architecture? ›

The architecture consists of three main layers: Bronze, Silver, and Gold. The Bronze layer is where all the data from external source systems is landed. You can also refer to this layer or zone as landing zone.

What is the difference between medallion and data mesh? ›

Can you spot the difference? In a nutshell: Medallion Architecture is an architectural pattern for a better organization of data in the lake. Data Mesh is an organizational pattern changing the way organization see and extract value from data.

What is the difference between a coin and a medallion? ›

A medallion is made to commemorate an event, person, or a place. It is round in shape like a coin, but it does not have a monetary value. A medallion cannot be spent. It can still be made by a sovereign government, like the Eisenhower Medal made by the US Mint, but more than likely it is made by a private mint.

What is another name for a medallion? ›

What is another word for medallion?
decorationmedal
pendantaward
badgehonorUS
ribbonhonourUK
stargong
81 more rows

References

Top Articles
Latest Posts
Article information

Author: Ray Christiansen

Last Updated:

Views: 6167

Rating: 4.9 / 5 (69 voted)

Reviews: 84% of readers found this page helpful

Author information

Name: Ray Christiansen

Birthday: 1998-05-04

Address: Apt. 814 34339 Sauer Islands, Hirtheville, GA 02446-8771

Phone: +337636892828

Job: Lead Hospitality Designer

Hobby: Urban exploration, Tai chi, Lockpicking, Fashion, Gunsmithing, Pottery, Geocaching

Introduction: My name is Ray Christiansen, I am a fair, good, cute, gentle, vast, glamorous, excited person who loves writing and wants to share my knowledge and understanding with you.