The Challenge
Our esteemed client, a leader in human pharmaceuticals, was experiencing significant data management setbacks. Critical data pipelines were not being actively monitored, resulting in service level agreement violations and a lack of timely alerts for failed ETL jobs. A lack of record auditing, coupled with opaque prioritization of ETL tasks, was impacting their operational agility.
Problems to solve
- Lack of active monitoring of data pipelines
- Inadequate notification of failed ETL jobs
- Lack of auditing for frequently used data sets
- Challenges in assessing data freshness and volume
- Sub-optimal use of Databricks clusters resulting in increased compute costs
The solution
Our intervention was twofold. First, we implemented an observability – FinOps and monitoring platform that refined their data operations and management.
Implementation Highlights:
- Activating the monitoring platform: This critical step enabled close tracking of performance metrics and data integrity
- PowerBI Dashboard Integration: Enabled a single view of the truth, enabling the client to proactively address data anomalies
- Optimized resource allocation: Through strategic data analysis, we maximized resource efficiency and significantly reduced unnecessary spending
- Improved data prioritization: With improved data tables, we streamlined the prioritization process for ETL jobs
Results
Our solutions delivered remarkable results:
- A 10% reduction in compute costs for low-level ETL workloads
- Increased reliability and speed of data availability by proactively monitoring and reducing errors in ETL jobs
- PowerBI dashboards facilitated instant verification of data freshness and accelerated root cause analysis of problems
- Custom metrics provided by OpenTelemetry enabled sophisticated job tracking and fine-tuning of ETL processes, improving overall data management efficiency
Transforming Data Observability in Pharma