Data Mesh: Concepts and best practices for implementing a Product-centric Data Architecture

Navigating the Evolution of Data Architecture

Navigating the evolution of data architecture

Photo by Shubham Dhage on Unsplash

Welcome to the world of data mesh! As the name suggests, data mesh is all about breaking up the monolithic data architecture and spreading data access and management across different teams and services. But why would you want to do that, you might ask? Well, for starters, data mesh architecture helps create a more resilient and scalable system by distributing data ownership and reducing dependencies between services. It also promotes data autonomy, allowing teams to make decisions about their data domains without having to go through a centralized gatekeeper. If you are familiar with microservices, I believe the best analogy for data mesh is microservices. Similar to microservices, data mesh tries to break down the central code monolith and distribute it for better management and scalability. However, unlike microservices, data mesh is still in its infancy.

But before you get too excited and start tearing down your data architecture, it’s important to note that data mesh is not a one-size-fits-all solution. It does come with its own set of challenges and limitations, which we’ll dive into later on in the article.

In this article, I’ll give you a crash course on the evolution of different generations of data architecture, the failure symptoms in the existing big data platforms and why the data mesh became an appealing solution, the benefits of mesh architecture and its history, and the key concepts of this architecture. I’ll also take a look at best practices for implementing data mesh, such as communicating and collaborating across teams and managing data governance. And of course, I’ll touch on the challenges and limitations you might encounter along the way.

Different Generations of Data Architecture

There is a common classification of data architecture in terms of different generations. The following are the common generations of data architecture that have been observed in the industry:

1. First Generation: This is the earliest form of data architecture and it is characterized by data being stored in a central location and accessed through simple queries. It often involves the use of a single database management system.

2. Second Generation: With the increase of data volumes and complexity, there was a need for a more advanced way of storing and managing data, this is where the second generation data architecture comes in. This generation is characterized by the use of data warehousing, where data is extracted, transformed, and loaded into a central repository. This generation also includes the use of extract, transform, load (ETL) processes to manage and organize data.

3. Third Generation: The third generation data architecture is characterized by the use of big data platforms such as Hadoop and Spark, which are capable of handling large volumes of data in a distributed and parallel manner. This generation also includes the use of data lakes as a central repository for storing all types of data in their raw form.

4. Fourth Generation: This is the most recent generation of data architecture, characterized by the use of cloud-based storage and processing, the use of data mesh, data catalogs and data governance practices, the emergence of real-time data processing and analytics, the adoption of multi-modal approach, and the use of machine learning and AI techniques to manage and extract insights from data.

Failure Symptoms in Big Data Platforms and the Need for Data Mesh Architecture

Here, I provide some of the failure symptoms that companies face with big data platforms, and which Data Mesh as a concept aims to address.

1. Difficulty in scaling: Companies struggle to scale data sourcing from diverse set of domains and corners of an organization that’s producing data with large volumes and rapid pace, as well as scaling consumption of the data, to run experiments quickly and improve operations or optimize the experience of customers.

2. Failure to materialize value from data: Despite collecting large amounts of data, companies are unable to realize measurable and transformational metrics from the data they collect, and this failure to materialize any value from the data is one of the key symptoms of failure.

3. Getting stuck in building the next big platform: Companies often resort to building the next big platform to address the issues of scaling data sourcing and consumption, but ultimately fail to materialize any value from the data they collect.

4. Lack of data culture and failing to be data-driven: Despite large investments in data and AI, the results of becoming data-driven and competing on data are low. Companies may not have data culture in their organization, and are unable to develop a sense of data ownership and understanding of the business data throughout the organization, that leads to a lack of experimentation and failure to materialize the value of the data they collect.

5. Difficulty in coordinating and communicating with cross-functional teams: Large data platform projects often require coordination and communication between teams, and often, failures occur when these activities are not done efficiently.

Benefits of Data Mesh Architecture

From the contents of the previous section, I guess you can infer the benefit of data mesh architecture. However, I believe, it is important to consolidate, organize and present them as bullet points in a separate section. So, here, I try to look at some of the most commonly cited benefits of data mesh.

1. Scalability: One of the key benefits of data mesh architecture is that it allows for better scalability compared to monolithic architectures. By breaking up data systems into smaller, independently managed services, it’s easier to add new services or scale existing ones as the data volume or complexity increases.

2. Improved Development Velocity: Another benefit is the improved development velocity, with a Data Mesh architecture, teams are able to work more independently, and are not constrained by a centralized data management system. This allows teams to make data-related decisions and innovate more quickly, resulting in faster development and deployment of new features and services.

3. Increased Resilience: With data mesh architecture, teams are more autonomous, this autonomy allows them to make decisions about the data they own which leads to increased resilience. In case of service failure, another team can take over the data domain and continue to serve it, thus avoiding a complete system failure.

4. Better Data Governance: Data mesh architecture emphasizes on decentralization of data and its management, this leads to better understanding of data ownership and reduces the risk of data breaches or compliance issues.

5. Flexibility: With Data Mesh architecture, teams have more autonomy over the data domains they own and can decide what technology stack to use to build their services. This allows teams to select the best tool for the job, resulting in more flexible and efficient systems.

6. Improved Data Quality: With Data Mesh architecture, teams are responsible for the data domains they own, this leads to better data quality as teams are more invested in the data they use. Teams will ensure that the data is accurate and consistent, because their services are dependent on that data.

7. Better Access to Data: Data Mesh architecture is based on decentralization of data, this means that teams have more direct access to the data they need, rather than having to request it from a centralized system. This leads to faster and more efficient data access and processing.

8. Better Alignment with Business Needs: Data Mesh architecture encourages teams to take ownership of their data domains, and to align their data management and usage with the specific needs of their business unit or department. This leads to a better alignment of data usage with business goals, resulting in more impactful data-driven decisions.

History of Data Mesh Architecture

Now that we’ve talked about the evolution of different generations of data architecture, the problems with existing big data platforms, and the benefits of the data mesh architecture, it’s worth taking a look at the history of this architecture. Who first observed these issues and came up with the idea, and who expanded and evangelized this solution?

The term “Data Mesh” was first coined by Christian Posta and Michael Bryzek, in 2018, in an article for the O’Reilly Media. They observed that many organizations were struggling with scaling their data infrastructure, despite investing heavily in big data platforms. This led them to the idea of a data mesh, which aims to break up monolithic data systems into smaller, independently-managed services. In previous section, I gave you a rundown of all the issues that organizations struggle with.

In the following years, more and more practitioners and thought leaders in the field of data architecture and engineering started to adopt and expand on the idea of data mesh. They recognized that it can provide a number of benefits, including increased scalability and resilience, as well as improved development velocity.

In 2020, the book “Data Mesh: Beyond Monolithic Data Architecture” written by Zhamak Dehghani was published and it became a reference for understanding Data Mesh. The book provides a comprehensive introduction to the concept of data mesh, including the benefits, patterns, principles, and challenges of implementing a data mesh architecture.

Since then, Data Mesh has been gaining more traction and is being adopted by more companies. Thought leaders and practitioners in the field continue to share their experiences and best practices with others in the community through presentations, articles, and meetups, helping to further the understanding and adoption of this approach.

Key Concepts in Data Mesh Architecture

Key Concepts in Data Mesh Architecture (Image by author)

Before we dive into the nitty-gritty details of data mesh architecture, it’s important to understand some of the key concepts that underpin it.

First up is product-centric data architecture. This concept is all about aligning data architecture with product development, rather than aligning it with the traditional backend and frontend boundaries. In other words, instead of having a centralized data management team that controls access to data for all services, product teams take ownership of their data domains and are responsible for managing and maintaining it. This allows for faster and more autonomous decision-making, as well as better alignment between data and product goals. In other words, data will be treated as a product. If you think of it that way, then all the metrics we use to measure the success of a product could be extended to data. For example, how many people used the data. How many downstream process are built off of our data product.

Another key concept is data autonomy. This means that teams have autonomy over the data domains they own, and can make decisions about how to access, manage, and evolve the data without having to go through a centralized gatekeeper. This helps reduce dependencies between services and allows teams to move faster.

You may also have heard of “backend for frontend” (BFF). This refers to a pattern where each frontend service has its own backend service that is responsible for handling data and business logic specific to that frontend. This helps reduce the complexity of the main backend services and allows frontend teams to have more autonomy over their data.

Domain-Driven Design (DDD) is a key concept in data mesh. DDD is a software development approach that focuses on creating an understanding of the problem domain and creating a model that matches it. By using DDD, teams can better understand the data domains they own, allowing them to make better decisions about how to manage and evolve the data. One of the benefits of this design approach is that the data is managed by people who are familiar with that data. Compare it with a centralized data warehouse that data engineers have sometime no clue about the context of a particular data and its importance and unique conditions.

Event-driven architecture is another key concept in data mesh. This approach involves using events to communicate between services and propagate changes in the data. This allows for a more decoupled and scalable system, as well as better visibility into what’s happening in the system.

Finally, Self-Sufficient Services is another key aspect, it means that each service should be able to operate independently, being able to find the data it needs, manage its own state and be able to evolve the data.

Got all that? Great! With these key concepts in mind, we’re now ready to dive into the specifics of how to implement data mesh architecture. I hope this section gives a good overview of the key concepts.

Implementing Data Mesh Architecture

Implementing a data mesh architecture may seem like a big undertaking, but don’t let that discourage you! With a little planning and organization, you can break it down into manageable steps that will lead to a more scalable and resilient data infrastructure.

Identifying and grouping data domains

One of the first steps in implementing a data mesh is to identify and group the different data domains within your organization. These are the different areas of your business that have distinct data needs, such as customer data, product data, and financial data.

Assigning ownership of data domains to cross-functional teams

Once you’ve identified these domains, you can assign ownership of them to cross-functional teams. This way, the teams responsible for the domain can take ownership of the data and decisions about how it’s used.

Decentralizing data access and management

Another important aspect of data mesh is decentralizing data access and management. This means moving away from a monolithic data service and instead creating many smaller, self-sufficient services. This allows for increased scalability and improved development velocity.

Implementing BFFs and self-sufficient services

One way to accomplish this is by implementing BFFs and self-sufficient services, which can provide the data and functionality needed by a specific application or service.

Implementing event-driven architecture

Event-driven architecture can also be a powerful tool when implementing a data mesh. By using events to trigger changes to data or actions, services can be more loosely coupled, meaning they are less dependent on one another. This allows for more flexibility in how services are developed, deployed, and scaled.

Implementing decentralized governance

With multiple teams owning different data domains, the governance of that data needs to be decentralized as well. This means that each team is responsible for their own data domains and have the autonomy to make decisions about how that data is used, accessed, and protected.

Implement a governance framework

In order to support decentralized governance, a governance framework needs to be put in place. This can include guidelines, policies, and tools for managing data governance across the organization. This means ensuring that the data being used is accurate, complete, and compliant with any regulations. The cross-functional teams that have ownership of the data domains can also play an important role in coming up with a framework and maintaining it.

Best Practices

Implementing a data mesh architecture can bring a lot of benefits to your organization, but to truly succeed, it’s important to keep a few best practices in mind.

First, communication and collaboration across teams is key when implementing a data mesh. With decentralized data access and management, it’s important to make sure everyone is on the same page and understands how the different data services interact with one another. Encourage cross-functional teams to communicate and collaborate, and make sure everyone is aware of changes and updates to the data services.

Next, Domain-Driven Design and modeling can be a powerful tool when implementing a data mesh. DDD is a way of thinking about and designing software that helps to align the domain model with the business domain. This can be particularly useful in identifying data domains and creating self-sufficient services.

Establishing and enforcing data governance is also an important part of data mesh best practices. This means making sure the data being used is accurate, complete, and compliant with any regulations. This can include establishing data quality checks, creating processes for data maintenance, and developing guidelines for how data is used.

Another best practice is managing data quality. With data coming from various sources, it’s important to make sure it’s accurate and complete. This includes things like monitoring for errors, creating data validation checks, and implementing data lineage to track where data came from and how it’s being used.

Finally, it’s important to think about scalability when implementing a data mesh. With many smaller, self-sufficient services, it’s important to make sure that each service can handle the load it needs to support. This may include things like load balancing, horizontal scaling, and performance testing.

Keep in mind, implementing a data mesh architecture is a journey, not a destination. Continuously evaluate and iterate as you learn what works best for your organization and its data needs.

Challenges and Limitations

Implementing a data mesh architecture can be very beneficial to an organization’s infrastructure, but it’s not without its challenges. Here, I try to outline some of the challenges of this architecture. Also, it is worth mentioning that Data Mesh architecture is a relatively new concept and its practical implementation is still in its early stages. However, it’s important to keep these issues in mind and think about these potential problems.

One of the biggest challenges in adopting a data mesh architecture is the mental shift required to move away from a monolithic data system. This can be difficult for some organizations, as it requires rethinking the way data is organized and managed. It also might require additional coordination and communication between teams, as well as a change in the way data is accessed and used.

Another challenge with data mesh architecture is the complexity of event-driven architecture. The use of events to trigger changes to data or actions can add an additional layer of complexity, which can be difficult to manage. It also might require an update in the way the team is used to work and a change in the way communication and coordination happens internally.

Furthermore, there is currently a lack of specific tools and libraries for data mesh, meaning that it may require a higher level of manual development, testing, and deployment compared to more traditional architectures.

And lastly, managing the complexity and coordination with a data mesh architecture can be challenging. With many independently-managed services, it’s important to have clear communication and coordination in place to ensure that everything is running smoothly. This might require additional planning, documentation and testing.

It’s important to keep in mind that despite these challenges, data mesh architecture can offer many benefits, such as increased scalability and resilience, improved development velocity and more efficient use of data. It’s just a matter of finding the right balance and figuring out what works best for your organization and its data needs.

Resources for further learning

1. “Data Mesh: A New Architecture for a New Era of Data”, by Christian Posta and Michael Bryzek. This is the article where the term “Data Mesh” was first coined, and it provides an overview of the challenges faced by organizations in scaling their data infrastructure, and how data mesh architecture aims to address those challenges.

2. “Data Mesh: A new way to think about data infrastructure” by Emma Jane Westby. This article provides a brief introduction of data mesh and how it’s different from monolithic and data lake architecture.

3. “Moving from Data Lake to Data Mesh” by Zhamak Dehghani, this article discuss about the evolution of data architecture from database to data lake to data mesh, and the benefits of data mesh architecture over the other architectures.

4. “Building a Data Mesh: A Guide for Practitioners” by Tim Berglund and Sameer Wadnerkar, this article provides an overview of the key principles and concepts of data mesh, and how organizations can start implementing it. It includes guidance on how to identify data domains, assign ownership, and implement event-driven architecture. It also provides a high-level overview of the tools and technologies that can be used to implement a data mesh architecture and some best practices to keep in mind while designing it.

5. How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh (here)

6. Introduction to Data Mesh — Zhamak Dehghani (here)

7. Decentralized Data Engineering (here)

8. Data Mesh 101: What is Data Mesh? (here)

9. Data Mesh Implementation Patterns (here)

10. From Data Silos to Data Fabric: The Next Step in Data Management Evolution (link)

11. Five must-read books for data engineers (link)

12. How to Build a Data Platform: A Comprehensive Guide for Technical Teams (link)

13. Data Solution Architects: The Future of Data Management (link)

14. Implementing Data Governance: A Step by Step Guide for Achieving Compliance and Data-Driven Insights (link)

15. Designing a data warehouse from the ground up: Tips and Best Practices (link)

16. Query Optimization 101: Techniques and Best Practices (link)

17. Advanced Strategies for Partitioning and Clustering in BigQuery (link)

18. Data Processing Evolution: ETL vs ELT- Which One Suits Your Business Needs? (link)

19. Data Orchestration 101: Understanding the Different Types of Tools and Their Use Cases (link)

20. The Evolution of SQL: A Look at the Past, Present, and Future of SQL Standards (link)

I hope you enjoyed reading this 🙂. If you’d like to support me as a writer consider signing up to become a Medium member. It’s just $5 a month and you get unlimited access to Medium 🙏 .
Before leaving this page, I appreciate if you follow me to see my future articles in your home page 👉
Also, if you are a medium writer yourself, you can join my Linkedin group. In that group, I share curated articles about data and technology. You can find it: Linkedin Group