Data governance challenges

Challenges of data governance

Data should be an asset; not a liability. But that’s what happens when growing organizations encounter data governance challenges like the following.  

1. Data silos 

Data silos happen when organizations have data trapped and locked within systems, departments, and functional areas. For example, marketing team functions on their own with their third-party tools, databases, and cloud applications. Sales, on the other hand, operates CRM, ERP, and other systems independently functioning. But none of these data sources are linked, meaning marketing has no input over what campaign is effective, and sales has no idea about what happened before the customer came into the pipeline. 

How does data silos affect data governance? 

• The data fragmentation situation impacts data governance in many ways, leading to compliance issues, format inconsistencies, and data-based decision-making. 

• Every team follows their own formats and standards, making it difficult to implement data governance. 

• Lack of centralized governance could mean each team takes up security measures on their own accord. This could lead to data breaches, non-compliance risks, and poor access control.  

 What is solution to fix this data governance challenge?  

1. Implement master data management systems that connect and unify customer data.  

2. Data integration tools implementation with the help of ETL/ELT pipelines and APIs to connect various data sources. 

3. Set up organization-wide policies for data formats, validation, and reporting activities. 

2. Misallocated resources 

Misallocated resources happens when companies spend more efforts, resources, and money on data storage and don’t align the data initiatives with business objectives. This could happen due to many reasons. 

• Companies are fixated on short term wins over long-term solutions. 

• Resources are spent primarily on non-priority tasks. 

• Unable to tap into data’s value, but data costs seem to balloon up. 

• Too much time spent on manual data processing and cleaning, and too less time available for analysis. 

• Not using the right tools, or employees avoiding using those tools due to complications. 

In either case, it’s the effort and money spent on data management go down the drain, when organizations don’t focus on data quality issues.  

How to fix data governance challenges due to poor resources? 

• Balance the investment in storage and governance, giving importance to data quality, usability, and compliance. 

• Setting up automation to reduce effort spent on manual data cleaning.  

• Taking care of old and outdated data that requires more resources and storage but serve less to zero value. 

• Looking into shadow IT and fragmented data sources that occupy more storage than intended.    

3. Data quality 

Poorly maintained data, inconsistent formatting, lack of standards, and integration challenges – all of this leads to data quality issues, which directly impact data governance. When you are left with poor quality data, you can’t rely on it for business decision-making. Not just that, you are left with mismatched records, inaccurate reporting, and compliance violations. 

How data quality issues affect data governance? 

• Can lead to miscommunication across teams, when each team has their own ways of storing and managing data. 

• Not knowing who to reach for data quality or accuracy issues, that is lack of data stewardship. 

• Lots of missing and incomplete data which isn’t validated on time, leads to incomplete analysis. 

• Data lineage can become tricky. No one could go back and trace what happened to data, who worked on it, and so on. 

• Lack of standardization across teams or locations. One team might be using formats like dd/mm/yyyy while other teams stick to mm/dd/yyyy. 

Fixing data governance challenges by fixing data quality issues 

Data governance and data quality are interdependent. Setting one right can fix the other too. Here's how you go about it. 

• Set up data quality standards, especially for accuracy, consistency, timeliness, and completeness. This will leave your records reliable and accurate, something that meets business analytics standards. 

• Data validation and cleansing tools like Informatica, Talend, or even Excel for small workloads. This could catch errors, fix incomplete records, and alert when there are major quality issues. 

• Start tracking data quality with data quality metrics using real-time dashboards and set threshold values. Normalize good quality data. 

• Start tracking journey of data. Take metadata management and data lineage tracking seriously, so the business user could really trust data and its origin.  

4. Not having clear roles and responsibilities 

Some organizations have proper resources and clear data governance framework to begin with. But they could still end up with chaotic data management, security risks, and compliance failures when roles and responsibilities are not clearly drawn out.  

Why this happens? 

• Firstly, it’s the lack of accountability. Modern data architecture is complex and multi-dimensional. So, data is bound to have errors, duplications, and inconsistencies. But if there is no one to review consistently, all of them reach till the end users. 

• Without data protection or security personnels, no one knows which data must be anonymized and encrypted, which could expose private data and open doors for compliance violations. 

• Data access management could get messed up in both ways. Business users get access to more assets than the required. Or critical business users don’t get access to reports on time. One is a vital risk while the other is a heavy inconvenience. 

• IT ends up taking the brunt of it all, spending too much time fixing issues and reviewing access control needs. Business users complain as they don’t know who they should reach out to, all credits to the unclear governance and management. 

How to fix this and make data governance work as it’s supposed to? 

• Typically, a data or IT team should have the following roles chalked out. Chief data officer, data owners, data stewards, data protection officers, and finally business users. 

• There should be data owners for each department or divisions – someone who owns, defines, modifies, and has access to their share of data assets. 

• Stress importance to role-based access control, so users only have access to what their roles need. 

• Data protection officers should be appointed. This person or team will oversee compliance and monitor any possible violations. 

• Assign data stewards, even though it feels like a redundant role (while it’s really not). They take care of everything from assigning data ownership to managing data quality to ensuring communication and collaboration across teams.  

5. Resistance from business teams 

The biggest barrier any data team would face while implementing data governance is resistance from internal teams. People often see data governance as restricting and time-consuming task. All the additional steps they are asked to take – from approvals to access limitations, create a rather far-fetched outcome.  

How does this affect data governance? 

• Business users might hesitate or skip basic due diligence they are requested to do and find shortcuts that might put everything in risk. For example, creating an unauthorized guest access for users while sharing, rather than the traditional approach. 

• Data quality takes a direct hit. Data becomes unreliable, even with all the right tools, processes, and people. 

How to fix this? 

• Governance enforcement is important. But it must be practical. Ensuring balance between user convenience and data protection could be done. 

• Spread awareness. Conduct training programs. Appoint data stewards within teams and assign them responsibilities.  

• Consult with business heads and representatives before making changes in governance procedures. 

• Go from IT-controlled to self-service analytics with built in governance controls. 

6. Managing hybrid environments 

Many organization run on hybrid systems these days – a mix of cloud and on-premises systems or multi-cloud systems. Different systems and vendors have different practices; their approach to security, compliance, and data storage varies. This doesn’t cause major problems until data governance arrives. 

How does hybrid environments affect data governance? 

• Multi-cloud or hybrid systems require more nuanced frameworks for data quality, security, and governance standards. With integration problems, the effect snowballs, making it impossible to handle for IT/data teams. 

• Redundant storage costs, yet no centralized view. Or, the centralized view is there, but the insights are clouded and contaminated. 

How to fix this? 

• For multi-environment, design a custom cloud data governance framework that considers the current systems and their specifications. 

• Use single source of truth using a data lakehouse or a warehouse for centralized insights. 

• Find out the best of all existing systems and work on your cloud bills to do away with redundant costs. 

• Set up data security and data monitoring tools in place. Use AI and automation to find out issues in real-time and trigger alerts and next-best-steps. 

Final thoughts 

Many companies see data governance as a business function. Or not even something that needs attention. However, this is a major business imperative, because without it, there are a lot to lose. Poor data quality issues, compliance fines, bad decisions, and security vulnerabilities. Remember bad data could cost you more than you can imagine. According to IBM, annual expenses due to bad data was $12.9 million during 2021. Hence, it’s high time, businesses look into their data governance and take the right steps, be it setting data ownership or defining data standards or tracking data quality metrics. That’s how, an organization could trespass all the above data governance challenges and maintain high-quality, secure, and well-managed data assets.