Automated & accelerated data reporting with Microsoft Fabric

About client 

Our client is a medical equipment manufacturing company who helps patients relieve from neurological disorders and chronic pain. Specialized in neuromodulation therapies, they design medical systems and controlling technology like spinal cord simulators, which adjust automatically to each person's needs, reducing the manual intervention from medical professionals. Backed by over a decade of research, they are improving patient lives and helping them achieve a holistic cure. 

Challenges 

Multiple systems and poor data integration: The client had data walking in from all directions, which led to data consolidation issues and prevented a centralized view. Having no single platform and inefficient data management led to delayed data availability and eventually affected their team’s decision-making process.  

Inability to get equipment performance data on time: As leading equipment manufacturers, they test the effectiveness of products and components. Likewise, they had many other fast reporting requirements related to inventory tracking, supply chain, quality control, and compliance. But batch processing and static pipelines rendered the ability to access to real-time data, causing decision-fatigue and delays in all the above-mentioned activities. 

An overburdened data team: Their current architecture only had ADF pipelines for data movement and a data Lakehouse for storage. This was inefficient and hard-to-manage due to many reasons. No parallel-processing capabilities, lack of or partial support for metadata approach, static notebook and pipelines, no monitoring dashboards, and manual quality checks. The data team had to work twice as hard to move data into the destination and produce business reports. 

Data quality issues: Data format differs from one system to another, putting too much strain on data teams to take care of non-analytical tasks. This inconsistency and incompleteness also affected the quality of insights produced. 

Lack of scalability: Current architecture cannot support their future goals, which involves handling high volumes of data and facilitating nuanced reporting. Besides, managing real-time streaming data from IoT-enabled systems in the future could be a challenge, leaving less room for predictive and AI systems. 

datakulture solution 

Reviewing their requirements, we selected the solution - building a centralized enterprise data layer using Microsoft Fabric and employing medallion architecture with a series of data layers.  

Here’s the high-level overview of our solution for the client: 

- Scalable data movement from sources to the bronze layer using data pipelines. 

- From the bronze zone to the silver zone for data cleansing and transformation. 

- Transformed and aggregated data is moved to the final gold layer. 

- Curated data is available for business users for further analysis. 

- Migration of old reports and dashboards from previously-used systems to Power BI, so the entire data ecosystem can be nested in one place. 

How did this solution help them? 

1. Meta-data driven approach used throughout the three layers, where ETL workloads are powered by metadata with marked characteristics. 

2. Automated check for data loss before and after processing. Meaning, no data loss, less intervention from data teams, and accurate data reporting. 

3. Leveraging Purview from Microsoft Fabric for data governance to ensure that the data is properly managed. 

4. Setting up workspace security which goes beyond regular RBAC. For example, enabled row-based and column-based security, which involves limiting access to users for specific rows and columns within tables. An admin can define which user should access which row or column or whatsoever. This granular control was very much required for them, given their stringent data security and governance needs. We helped them leverage Fabric to mask their sensitive information and strengthen their fence against breaches. 

5. Before Fabric, manual data movement took more than a day to retrieve past data and prepare ad-hoc reports. Now, it’s all automated, and holistic business insights are available in near-real-time within 15 minutes, saving time for business users. 

6. One of the major impacts observed after Microsoft Fabric implementation is the reduction in expenses due to equipment expiration. The real-time data availability helps the client align current and future demands with inventory, which brings down inventory overhead and overstocking. 

Conclusion 

Thanks to Microsoft Fabric, the client broke silos between data systems and work with more relevant, clean, and aggregated insights. The real-time data processing enabled their teams to detect anomalies and issues faster and meet their production goals and quality standards. Such a scalable and modern architecture was required in place for them to adapt to future challenges, innovate, and sustain its leadership in the medical equipment manufacturing sector.