Transforming data observability in pharma

Our Insights
Client Success Stories
Transforming data observability in pharma
Posted on

The Challenge

Our esteemed client, a leader in human pharmaceuticals, was experiencing significant data management setbacks. Critical data pipelines were not being actively monitored, resulting in service level agreement violations and a lack of timely alerts for failed ETL jobs. A lack of record auditing, coupled with opaque prioritization of ETL tasks, was impacting their operational agility.

Problems to solve

  • Lack of active monitoring of data pipelines.
  • Inadequate notification of failed ETL jobs.
  • Lack of auditing for frequently used data sets.
  • Challenges in assessing data freshness and volume.
  • Sub-optimal use of Databricks clusters resulting in increased compute costs.

The solution

Our intervention was twofold. First, we implemented an observability – FinOps and monitoring platform that refined their data operations and management.

Implementation Highlights:

  • Activating the monitoring platform: This critical step enabled close tracking of performance metrics and data integrity.
  • PowerBI Dashboard Integration: Enabled a single view of the truth, enabling the client to proactively address data anomalies.
  • Optimized resource allocation: Through strategic data analysis, we maximized resource efficiency and significantly reduced unnecessary spending.
  • Improved data prioritization: With improved data tables, we streamlined the prioritization process for ETL jobs.


Our solutions delivered remarkable results:

A 10% reduction in compute costs for low-level ETL workloads.

Increased reliability and speed of data availability by proactively monitoring and reducing errors in ETL jobs.

PowerBI dashboards facilitated instant verification of data freshness and accelerated root cause analysis of problems.

Custom metrics provided by OpenTelemetry enabled sophisticated job tracking and fine-tuning of ETL processes, improving overall data management efficiency.