Business and technology continues to struggle to find a harmonious unity. With new buzzwords floating around like 'Big Data', ‘Data Science' and 'Machine Learning', it has now become even more difficult for the business and tech teams to work together coherently.
In the book, Analyzing the Analyzers: An Introspective Survey of Data Scientists and Their Work, the author points out that “Despite the excitement around ‘data science’, ‘big data’, and ‘analytics’, the ambiguity of these terms has led to poor communication between data scientists and those who seek their help”.There remains a chasm in between technology and business. In the end, how do we get these two estranged business entities to find a common language?
We recognize that technology has always been a source that creates buzzwords and inspires innovation. Data science has been the source of a lot of the excitement for about a decade. When people experience and read what companies like Google, Amazon and Facebook have been able to do with their data and the value they have gained, of course managers want to jump on the opportunity.
Wouldn’t you want to be the one that brought your company to the level of all these titans of the tech industry? That would be pretty sweet!
However, in the rush to get to their final destination, these management teams try to cut corners and end up skipping the journey. These giant corporations like Facebook and Google, have had years to perfect their craft. They didn’t jump on the band wagon, they built it. They realized that good data science and machine learning started with good data, and proper processes to get the end user (whether the data scientist or the customer) the data they really need.
Most of these companies have been doing it for over a decade. Now, we have lots of copycats trying to imitate other companies success. Managers are hiring data science teams, BI teams, and applied mathematicians left and right. Then, they try incorporating them into their companies current tangled eco-systems of specialized teams.
When this occurs, most managers run into several problems.
Your Data Is Locked up
A common problem most data scientist and BI analyst experience is getting to their companies data. This typically is caused by people with great intentions. DBAs are often the sole guardian between data that is operation critical and the slew of internal threats like devs, BI analysts, and fresh out of school grads. If allowed, these intelligent but unaware miscreants would drop an entire database, delete tables, and cause all sorts of mayhem without even knowing it.
Guess who gets blamed for all of this? And who has to fix it?
Yup, the DBA.
They have a lot of good reason for not wanting to freely let anyone have access to their data stashes. However, this causes data scientist to be bogged down with processes. Even if they are allowed data, they often have to construct their own pipelines/ETLs and structure new databases.
All of this eats into a data scientist’s ability to be productive at what they do best. Suddenly, a simple 4-week project of analyzing sales data, becomes a eight-month-long mission to create a new datawarehouse, build ETLs and QA the data before they can even start your first bit of analytics work.
When you are paying an employee upwards of 100k, this isn't practical. You need them to be fully functional, and fast!
Your Data is Not Reliable
For all the hubbub about the value of data, very few data evangelist warn companies about dirty data. Most Hadoop infrastructure salesmen, and tableau specialist are just trying to sell the next “it” product. They convince companies and managers that no matter what shape their data is in, they have the tools to fix it.
If they’re anything like that, it should tell you that they have never worked with data beyond the 3 months of intro their start up gave them. Of course, the less the salesman knows the better. If he doesn’t realize the limitations of his own product, it is easier for him to tell the truth without knowing he is lying.
In some cases, data is just wrong. Over the years, systems may have never been QAed and the gold mine you think you are siting on, may just be a garbage dump.
First thing is first, before you go off and buy yourself new toys to utilize your data, make sure your room is clean. I don’t care how this is done. Whether you hire someone internally, or look for consultants. Get someone to analyze your data sources every few years.
Some form of an audit can be very valuable. Something that guarantees your data is good and can trace when it might have gone bad. Otherwise, when you spend several hundred thousand dollars of capital budget on a data science tool or new machine learning algorithm, you might just end up with an expensive paper weight “so to speak”.
Your Data Team Is New To Business
Data scientist are a rare breed. While every business analyst who has used SQL or R for three months are suddenly throwing the title on their resume, there are truly very few data scientist that meet the true qualifications that businesses are looking for.
Larger companies have gone so far as to plucking professors out of colleges with three PhDs, twenty research papers and a Nobel Peace Prize because they prove to be the most qualified for the role of data scientist. Of course, once you have captured a few of these illustrious creatures, what are you going to do with them?
A lot of businesses just throw them at data, and expect value to be found. Don’t get me wrong, ambiguity is great and all but how do you expect them to know what is valuable to you, or the company? Especially if you just hired the team?
You can’t just expect them to know what you want.
There needs to be open dialogue that allows them to see what the business needs to reduce the tech gap. There needs to be conversations of what the businesses sees as strategic initiatives, expected data science team outputs and external threats that can be mitigated using the company's new possible competitive advantage. Even machine learning engineers are given 1.5 years to come up with each valuable new algorithm or concept. Data science teams won’t always be successful right away. It will take even longer if your company doesn’t include an executive to lead these teams who is also part of high level strategic meetings. That way, he can help provide insight to the team in where the company is going.
This gap between business and technology isn’t new. We have dealt with it since before this generation and only a few companies that were tech centric really seemed to find the solution. Others remained in the dark, and fumbled when it came to finding value from their tech and data teams.
This gap is caused by the fact that management can’t qualify what they want properly and the data scientist don’t ask the right clarifying questions. Yes, requirements should be somewhat vague to avoid boxing in your results. However, managers are still responsible for defining what is valuable. Without doing that, your team might learn some cool things and create awesome dashboards and websites. They might do all of this, and provide your company a utility of 0, Nilch, Nada!
Your department might have just spent hundreds of thousands, maybe even millions of dollars to start this team and they might provide some cool deliverables. All of which, don’t benefit you, or your strategy team can’t implement. Then what was the point. If you just spent so much time, and money on a project that yielded no form of benefit besides a cool dashboard...why?
Data scientists have a lot of good energy. They are brilliant in multiple fields. Only some are brilliant in making value for the business without being told what that means (in all fairness, this is a trait that is not specific to data scientist, accountants, analysts, and even managers sometimes fail here). When you are an individual contributor, it can sometimes be hard to see the big picture. This is what I believe business managers are for.
Overall, data science and analytics provide companies the opportunity for a competitive advantage. It allows managers to make better decisions, and connect better with the customer. However, it is important to take a close look at what resources you have to work with, before attempting to bring projects in front of your data science teams.
We are a team of data scientists and network engineers who want to help your functional teams reach their full potential!