The Challenge
The customer wanted to develop a data-driven organization to support all important business decisions. In addition, the customer was implementing monitoring of key KPIs for data-driven goals to set the organization’s strategic direction.
Problems to solve
- Complex data platform: The customer deployed a modern, cloud-based data platform using Azure Stack, Data Lake (ADLS), SQL for data storage, Azure Data Factory, Databricks, and Azure SQL for data processing, and PowerBI for reporting
- Rapid deployment by multiple teams: Multiple data products were rapidly deployed on the platform by different cross-functional teams, resulting in a lack of unified visibility
- Lack of visibility: The data platform owner struggled with visibility into platform usage and operations to ensure proper tool utilization, optimal resource allocation, and increased platform adoption
The solution
- Data catalog integration: Implement a data catalog integrated with the observability dashboard, including:
- Describe all objects (tables, columns, pipelines, …)
- Set Tier 1
- Set owners for objects
- Add Tags
- Add Glossary terms
- Set Data Domains
- Set Data products
- Define KPI for OKRs
- Add DQ tests for tables and columns
- Extended observability: Extend observability to mission-critical data for fast and reliable decision making
Results
- Business risk mitigation: Identify and resolve data issues before they impact critical business processes, reducing the risk of decisions based on inaccurate or incomplete information
- Increased data trust: Ensure data reliability and integrity, building greater trust in the data
- Accelerated insight time: Reduce the time to generate insights by quickly detecting and resolving data issues. Optimizing data processing prioritizes delivering the most relevant data
- Improved customer experience: Reliable, high-quality data helps organizations understand customer behavior and preferences better
Observability – Phases I and II