Optimizing data infrastructure for a healthcare provider with cloud and analytics solutions

To address the client's initial concerns around data security in an outsourced data engineering setup, Binariks proposed a tailored security plan with multiple options to ensure data protection. These options included using client-owned devices, implementing virtual machines, and other data-safe approaches. The client selected a strategy they felt confident in, leading us to integrate our data engineer with their internal team seamlessly.

Our data engineer began with a comprehensive assessment of the client's data infrastructure, focusing on Snowflake and Tableau and gathering some data (for example, regarding data sources and users) via API to ensure a complete inventory of assets.

Given the complex state of the existing data structure, our engineer also thoroughly reviewed performance and technology, identifying areas that needed optimization.

Throughout the project, we established a structured approach to communication and task alignment. Also, we worked closely with the client by continually presenting various third-party tools and solutions options and several POCs to align with their specific needs, preferences, and budget constraints.

Infrastructure setup:

• Leveraged Terraform to establish a reliable and standardized infrastructure on AWS, creating a foundation for data modeling and operational workflows.
• Integrated a robust data modeling platform for business analytics to enhance data organization and usability.

Data modeling and data architecture:

• Explored DBT Labs and alternative vendors, conducting multiple POCs to evaluate cost-effectiveness and performance.
• Deployed a custom AWS-based solution, offering scalable data management and user-friendly analytics.
• Built a dual-layer architecture: a data lake for standardized raw data storage, and a data warehouse, using the Kimball methodology to facilitate structured reporting.
• Migrated legacy dashboards and created critical new reports, optimizing analytic processes to provide actionable insights.

External reporting:

• Designed and migrated external reporting systems to support data sharing with the client’s partners, enabling streamlined access to provider-specific and application-derived data.

Data testing frameworks:

• Added Great Expectations framework for source data testing, automating batch jobs to monitor data integrity, unique values, and patterns.
• Integrated Elementary for real-time monitoring of model performance, build durations, and table ownership, with notification capabilities in Slack and email.

Future plans:

• To implement a self-service data platform, enabling business users to independently explore and manipulate data without needing dedicated business analysts.

This streamlined architecture and analytics framework enable scalable, efficient data management and empower the client to achieve data-driven insights independently. Enhancing drug safety and compliance with innovative pharmacovigilance tools, reducing costs, and improving risk management.