Loading files into databases are not exactly exciting work. It can be tedious and boring work. Often times it is constrained by tools like SSIS that freak out when a small change is made.
This is one of the core roles of BI and data engineering teams. They need to load files from operational databases into data warehouses. They do this by developing pipelines that process and transform data.
​
One problem we see is teams have multiple file types, data spec versions, and data providers. Each of which requires a different pipeline just to load into their raw database.

This forces teams to manage 10, 20 maybe even 50 smaller pipelines that are all subject to the change of the data providers. In addition, each new data provider requires an entirely new pipeline to be developed. It also requires managing infrastructure around each pipeline in order to log and track data pipelines statuses. This can quickly become very difficult to manage and tedious to build.

Another solution instead of attempting to manage multiple pipelines is to develop a central system that manages the various insert statements required to insert data into each different raw table. Kind of how we explain in the video below.

What the system does is essentially act as a lookup system that matches flat files to their insert queries. This reduces the need to redundantly have separate pipelines that would also all need an insert statement.

Instead, now the system uses one main system to load all the various tables. Theoretically, you can also put version numbers on each of the files that come in. This could be set up by tracking each version in the metadata table that will then allow you to tie a raw file to an insert file version.

Loading multiple file types by individual pipelines requires a lot of tedious development. Whether it be through SSIS or another system, the constant development quickly weighs heavily on data engineers. Instead of spending time developing analytical pipelines that drive clear value for your company. They will be stuck maintaining and developing operational pipelines.

This reduces its impact and reduces its overall value.

Thank you so much for reading! We hope you enjoyed the video. Please feel free to send us any questions. We would love to help you design your next automation system.
Also, if you liked this, feel free to check out our video on using bulk insert!

For further reading and videos on data science, SQL and Python:
How To Develop Robust Algorithms
Dynamically Bulk Inserting CSV Data Into A SQL Server
 SQL Best Practices - Designing An ETL - Part 1
How Algorithms Can Become Unethical and Biased
4 Must Have Skills For Data Scientists
What is A Decision Tree

By Ben Rogojan
We recently did an AMA on Reddit. The most common question that came up was what is the difference between a data scientist and a data engineer. So we wanted to make a more in-depth post on the subject.

There are a lot of data specialist positions that sound similar and use similar tools so it can be difficult to know what the role of each role should do. In addition, smaller companies might be limited on how many data engineers or data scientists they can hire. This means that many times the specific tasks and goals might start to intermingle.
​

This can make it much more difficult to clearly differentiate the two roles. So we wanted to go over how the two positions were different by discussing the different goals, mindsets, tools and backgrounds data engineers and data scientists have.


Before we go into the difference we would like to make a quick preface. The truth is, many data scientists and data engineers will perform the tasks of other technical roles. A data scientists may need to develop an ETL and a data engineer might need to develop an API and front-end. So the distinctions we are pointing out below are just to make clear where the technical differences are.

GoalsThe goals of a data engineer are much more task and development focused. Data engineers build automated systems and model data structures to allow data to be efficiently processed. This means the goal of a data engineer is to create and develop tables and data pipelines to support analytical dashboards and other data customers (like data scientists, analysts, and other engineers). It’s similar to most engineers. There is a lot of designing, assumptions, limitations, and development that occurs in order to be able to perform a final task. Each design and solution having its own set of limitations, even if it all can perform the end task.
​
In comparison, data scientists tend to be question focused. In the sense that they are looking for ways to reduce costs/increase profits, improve customer experience or business efficiencies. This means they need to ask and then answer questions ( ask a question, hypothesize and then conclude). So they need to ask questions like, what impacts patient readmission, would a customer spend more if shown an add like A vs. B, is there a faster route to deliver packages? Skipping over the rest of the process. The goal from here is to find an answer to whatever question is posed. It might be a final conclusion or to more questions. Throughout the process, data scientists analyze, gather support and can develop a conclusion to the question.

Tools

This is where things can get confusing. Data scientist and data engineers both often rely on python and SQL. However, how the two tech roles use these skills varies. Again, this ties back to the mindset differences. Python is a very robust language that has libraries that help manage operational tasks as well as analytical ones.

Data scientists will use languages like Pandas and Scikit Learn whereas data engineers will use python to manage pipelines. This is where libraries like Airflow and Luigi come in handy.
Similarly, data scientist queries will be ad-hoc focused (e.g. questions focused). Whereas data engineers queries will be focused on cleaning up and transforming data.

Now there are other tools both data specialists might use. This includes Tableau, Jupyter notebooks and a handful of others. At the end of the day, it is how they use them that makes the distinction.
BackgroundsNow, one other common question when it comes to the differences of data engineers and data scientists is what background is required.

Data engineering and data science both require some understanding of data and programming. Even if it is a limited scope. However, there are some distinctions that go beyond programming. Specifically for data scientists. Due to the fact that a data scientist is more like a researcher, having a background that is research-based is a benefit.

This might be in economics, psychology, epidemiology, etc. Combine a research background with SQL, Python and a good sense of business and you have a data scientist. Now, these are not set in stone degrees. In fact, we have run into a data scientist with various degrees. Most employers will prefer to hire a data scientist with at least a masters degree that has some sort of technical or mathematical focus.

Data engineering positions usually won’t require a masters degree. Data engineering is more about being a developer. This requires much more practical experience rather than theoretical knowledge. So gaining a Masters does not supply the same value.

An Example
Let’s say a director of a healthcare company decide they would like to figure out how to reduce the number of patients readmitted prior to 30-days from their original visit. From a data point of view, there are a couple of things that need to occur.

Data scientists will need to figure out what drives patient readmission. That is the question they will be trying to answer. Based on the conclusions they reach, they will work with the business to develop metrics and policies to help improve patient readmission rates.

Data engineers will be developing tables to help support the data scientists answer the question while at the same time developing analytical tables to help track past and future patient readmission metrics. How these metrics are created will be driven by the answers the data scientist get.

Data scientists and data engineers have plenty of differences. They have different goals, and backgrounds, but this is where the value of utilizing both together comes from. The fact that data engineers focus more on engineering robust systems allows data scientists to query data easily and analyze it efficiently. Their partnership is what brings companies value from data.

We hope this post was helpful! Please feel free to reach out with any questions you may have.

Read more about data science, healthcare analytics and automation below:
​How To Develop Robust Algorithms
Dynamically Bulk Inserting CSV Data Into A SQL Server SQL Best Practices - Designing An ETL - Part 1How Algorithms Can Become Unethical and Biased
4 Must Have Skills For Data Scientists
What is A Decision Tree

Data engineering has many facets. One of the most common projects a data engineer takes on is developing an ETL pipeline from an operational DB to a data warehouse. Our team wanted to cover the overarching design of an ETL.

What are the typical principal components, stages, considerations, etc?

We started this by first writing Creating An ETL Part 1(more to come) and we and now have worked on a video that is below that walks through the process. We wanted to discuss why each stage is important and what occurs when data goes from raw to stage, why do we need a raw database and so on.

Data engineering is a complex discipline that partners automation, programming, system design, databases, and analytics in order to ensure that analysts, data scientists and end-users have access to clean data.
​
This all starts with the basic ETL design.

We are practicing up for a webinar that we will be hosting on ETL development with Python and SQL. The webinar itself will be much more technical and dive much deeper into each component described in the video. However, we wanted to see how using the whiteboard was like in case we need it.
If you would like to sign up for the free webinar we will be hosting it on February 23rd at 10 AM PT. Feel free to sign up here! If you have other questions please do contact us.

Read more about data science, healthcare analytics and automation below:
​How Algorithms Can Become Unethical and Biased
How To Develop Robust Algorithms
4 Must Have Skills For Data Scientists
Time Series Model Development: Basic Models
How To Measure The Accuracy Of Predictive Models
What is A Decision Tree

An ETL (and it’s not so far off cousin ELT) is a concept that is not usually taught in college, at least not in undergrad courses. Yet, ELTs play an important piece of almost every company’s day to day operations.

ETLs are the pipelines that populate data into business dashboards and algorithms that provide vital insights and metrics to managers. These pipelines go from very simple processes that extract data from operational databases into a data warehouse to complex analytical layers that support more complex transformations and applications.

The problem is, many of our IT graduates are not taught this very important skill that they will more than likely use in their everyday life if they decide to pursue a career in data science or data engineering. Although data science work doesn’t really include ETLs technically, a lot of work they do can end up crossing over with data engineers.
​
We wanted to cover the basics of an ETL and good ETL design.
What does a good ETL look like, what are the key components, what makes a pipeline maintainable and healthy? These are all important questions data engineers and database managers need to ask before taking on new projects.

This will be the first of several posts. In this post, we will be covering some basic concepts such as extracting/loading raw data and logging. These are important topics that have a lot of nuances that we would like to start to cover

We will also be offering a free webinar February 23 10 AM PT to cover this topic. We would love to have you attended!

Operational To Raw

Operational Database
All of the data your ETL will be ingesting will have some source that represents real life transactions of one sort of another. The data could represent patients visiting hospitals, people posting on social media or customers purchasing products. Some might ask why you can’t just analyze data from the operational database instead of going through the extra steps of processing the data into another system.

The biggest issue is that trying to run analytical processes that aggregate and analyze large data set from the operational database will slow down the operational database for customers. In addition, operational databases are heavily normalized which make them inefficient to query from an analytical standpoint. This is why a data warehouse of one form or another is devel

Here is an example of a method for bulk insert data into SQL Server

The first step in the processes is to extract the data into some form of raw format.

Raw File Extracts
How the data is pulled from the operational database can vary. However, one of the most consistent ways is to extract the data into a raw file like a CSV or XML file. Extracting the data into a raw file has several benefits. One, a direct link that pulls data limits the interactions outside systems have with the operational database. This will avoid blocking important operational tasks that need to occur. In addition, it creates a snapshot of what the data looked like at the time of pulling. This can be beneficial if you ever need to reload data because of an issue that might occur in the pipeline.

Having this back-up of data is beneficial just in case someone accidentally deletes crucial data. Then you will be very happy to have raw extracts that are easy to reload. These raw extracts should be tracked in some table that keeps track of what files have been loaded and where they exist. That way you can easily automate any reloading. Also, the files should have a naming standard.

An easy example would be DateExtracted_DateRangeInTheFile_BusinessObject (e.g. 20170101_20160901_20161231_Customers.csv)

Metadatabase And Logging
Tracking what is occurring inside your ETL process is a crucial step in developing an automated, maintainable and robust system. How extensive your tracking and logging depends on how large your company is and how many ETLs/ELTs and other data transformations need to occur. Larger companies might have a standardized tool like Airflow to help manage DAGs and logging.

However, if you are a start-up or a non-tech company, it will probably be ok to have a simplified logging system. The purpose of this system is to manage the files that need to be loaded/have been loaded, tracking of what stored procedures have run, errors, dependencies, etc.

Having good insight into these aspects of your system will help maintain it, improve it and backfill any data that either has been missed or loaded incorrectly. This means you need to actually create 3–4 tables (at least) that track what is running, how long do they take, computation, etc. Especially as we push for more and more automated systems, this will be the key to making sure the systems are maintainable.

Loading Raw Data Into Raw tables
Once you have extracted the raw data from the operational databases and updated your metatable with the new file information, then you can start focusing on loading the data.

It can be tempting to try to add a lot of complex business logic to your initial loads into raw. It can seem to simplify the process and create fewer steps in order to load the data. So why add more steps. The truth is adding too much logic to early can make it difficult to track where errors occur. Did the error occur in the extract, in the business logic or somewhere else? In addition, it becomes more difficult to update new logic because you have to analyze where the best place to update the logic will be.

All of these extra considerations make the overall system more difficult to maintain. That is why it is important to just load the raw data as is. If there is de-duplication logic or mapping that needs to happen then it can happen in the staging portion of the pipeline.

The next steps after loading the data to the raw database are QA and loading data into the staging database. We will continue that discussion in our next post. If you want to learn more about ETL development/automation then sign up for our ETL webinar where we will be discussing the process of creating an ETL from end to end.

Read more about data science, healthcare analytics and automation below:
Calculating A Running Total SQL
How To Develop Robust Algorithms
4 Must Have Skills For Data Scientists
Time Series Model Development: Basic Models
How To Measure The Accuracy Of Predictive Models
What is A Decision Tree
Automation Best Practices

It might seem strange to have data stored this way, but there are cases where condensing the data in this fashion might make sense. Let’s say you’re team is storing billions of transactions per day and you want to run a median calculation over several years.

In this case, it makes more sense to keep the data aggregated because it will improve the performance. You might not even be able to properly store the transactions if it isn’t aggregated.
It could be tempting to “unroll” the data so to speak. To take the data and create a new table that has a row for each of the instances of a person at a specific age. This could cause your rows to grow significantly, to terabytes upon terabytes. Also is just another step that might not need to be done.
Instead, let’s look this query.

SELECT TOP(1) Age
   FROM  
       ( SELECT Age
               , SUM(Total) OVER (ORDER BY Age) AS runsum
            , SUM(Total) OVER () AS total
        FROM Age_Exp ) AS d  
​WHERE runsum >= total / 2 ORDER BY Age;

​This query is short and sweet and you will also notice it brings in the running total we discussed recently(video). The goal of the medium is to find the number halfway through all the data. So that means you need to know the total and find the number halfway through.

The way this works is by using the rolling sum you can figure out where the half-way point will end up because the running total will sum of the total value.

Then using the other analytic function that gets the sum you can divide it by half and only select the top value. The median will be the top value.
​
We would love to know your thoughts on this problem. Also, if you have any problems you would like to discuss then please feel free to contact us! Thanks for reading.

by Ben Rogojan

Automation isn’t always about automated cars and drones that will deliver our purchases to our doorsteps. The goal of automation is to make people’s lives easier and not have them come in on work on Saturdays. There is a lot of tasks in the workplace that still can be automated to avoid having late nights and long weekends stuck at work. This doesn’t mean that automation in itself is easy to implement. Developing automated systems is a discipline, it requires patience, robust design and clearly thinking through the process.
​
There are good practices that can be overlooked when implementing an automated systems that can create a lot of tech debt. This will weigh down your system and possibly just waste the time for future developers who either have maintain or recreate the system. In order to avoid this an automated system should have solid error tracking, QA and be easy to maintain. Here are a few great tips for developing an automated system.

Error And Logging Tracking
Logging and error tracking does not come across as exciting work. Even developing basic tracking functionality can seem tedious. However, it is a necessary step for developing an automated system. Logging and error tracking can take place in several layers.

The simplest form is to track in raw files everything that occurs. This has the least overhead because it doesn’t need to be stored in a specific data system. This allows for low-level maintenance and future data parsing if required. Creating standardized logs allows for large amounts of unstructured data to be stored outside of a database and with modern technology and data storage systems searching through these files is much faster.
Having higher level logs with less information stored in standard data warehouse allows for tracking of issues and unhealthy systems. This can be used to detect issues such as slow load times, persistent bugs, etc. Unlike unstructured data, storing high-level issues in a data warehouse allows for easy views into trends and allows future analysis.

Especially as more complex automated systems come to fruition, being able to track and analyze errors and trends in usage will be a game changer. Whether this is for automated cars, e-commerce, and the world of IOT devices. The only way to manage the sheer volume of transactions and communication protocols between the various systems will be with automation. Error tracking and logging can’t be an afterthought in this automated systems. Otherwise, we will never be able to manage the holes in the systems we create.

QA And Alerting
Error logging is for issues that occur after release. The hope is we squash as many bugs before they occur. The problem is that automation leaves a lot of trust in the programmers and developers that set up the systems. Even trying to think through all the possible ways automated systems can go wrong, there are always failures and making sure to develop a QA suite that sweeps through the data and logs to make sure whatever automated processes are occurring, are occurring correctly.
​
There are basic QA tests that just make sure systems are running and then there are more complex systems that can take more of a machine learning approach. Looking for patterns and data profiles that could speak to process issues. For instance, sometimes null values are normal but sometimes they occur because of a specific issue. This could be because of an outage in a particular region, an update that works well on one OS and not another, and various other issues that might not be detected by standard integration and unit tests.

Thus, going more in depth and developing a QA system that can do basic tests just to ensure systems are functioning as well as test for more complex issues. Again, this is somewhere we would agree in implementing a form of a machine learning/data science system that can profile data to detect non-specific issues.

Simple Vs. Complex

One of the reasons for this comic above is the fact that automation, when poorly designed quickly, becomes a nightmare. Anti-patterns, over customizability, and poor design in general makes code difficult to maintain. This will take more time to manage instead of saving time. Instead of dealing with a hands off system you are managing a tech debt-ridden monstrosity that sucks up developer time.
​
If anyone recalls the Rube Goldberg machines we developed in middle school or elementary school you will recall the fact that in order to do simple tasks you over complicated and designed some system. This is a great example of some automated systems. In some cases, the task you might have been attempting to automate could have been done simpler or the system you developed is just too complex. Over-engineering can be a serious problem that makes it very difficult for future engineers to maintain. Over-abstracting code, developing too many functions, too many plausible paths can make code unmaintainable and untestable.

This often won’t be apparent until the developer that wrote the code leaves and the other programmers need to pick up where they left off.

This means good design, that doesn’t try to be too smart and over-engineered. Most professions suffer from this issue. Chefs try to put too much on one dish or an architects try to put too many features into a building. At the end of the day, the best Chefs, Architects and engineers understand that sometimes simple exceeds elegant. This doesn’t mean that concepts like abstraction and object-oriented are bad. It’s just that some developers place these concepts over the maintainable code. There is always a balance.

Dependency Management
Automated systems and processes have a lot of moving parts. This means at some point some components will need to occur in sequence and some will need to wait until multiple dependencies have completed in order to ensure the safety and health of the system. As more and more automated systems get developed to interact with people, beyond just robot arms behind safety walls and yellow tape, their systems will have to have multiple dependencies to ensure that the tasks they are performing are both correct and safe.

No matter if the system is interacting with people or if it is operating in the confines of a computer, having simplistic dependency management that allows for the developers to easily draw the paths makes it much easier to ensure that the system won’t run when it shouldn’t.

Automation does make life simpler when it is developed purposefully and fully thought through. This means planning how the error will be tracked and how the system will manage and alert larger problems. Designing a robust and maintainable system is not easy. However, when correctly implemented it can improve your team’s productivity and simplify your life.

If instead, your automation project goes a bit off the rails, then please feel free to contact our team!

Interested In Reading More About Being A Better Data Scientist?
How To Grow As A Data Scientist
Boosting Bagging And Building Better Algorithms
How To Survive Corporate Politics As A Data Scientist
8 Top Python Libraries For Machine Learning
What Is A Decision Tree
4 Ways To Calculate A Rolling Sum With SQL

The ubiquity of data makes SQL one of the most utilized technical skills whether you are a data engineer, analyst, or data scientist.

Even as companies that rely on larger data sets that are in data systems and tools like Hadoop, Hive, and Presto. SQL remains an important. It is one of the few “languages” you can learn once and only have to make minor syntax changes as you switch data storage systems. SQL can seem limited when you first start out because you haven’t approached a diverse set of questions. Once you get past JOINs, Loops and some other basic concepts in SQL like subqueries analytic functions, and aggregation, it can seem like you understand everything you will need to know.

In an odd way, you are right. You know all the basic building blocks to do a lot of complex work. But just because you know the building blocks doesn’t mean you know all the applications. It is like saying you understand addition, multiplication, division, and subtraction thus you must also understand how algebra works.

Algebra uses all the basic concepts but applies them in a new way. In the same way, basic SQL concepts can be applied to complex problems, but it doesn’t mean you know how.
How these problems are solved isn’t always consistent. How you may answer a question in Oracle and Mysql could be different in Redshift or SQL Server. These problems can be analytical, performance and optimization or even design.

Just because you are using many of the same commands does not mean you are using the same approach.

Let’s talk about some of these problems.

Analytic Functions
Before we start, we wanted to talk about a useful tool to help solve a lot of problems.

These are called analytic functions. These might not be the first things you learn, but once you do, you will probably start using them everywhere. If you don’t already have a decent grasp of them then we recommend you get a quick intro.

One quick note: Not all SQL engines have analytic functions built in so you will have to use work arounds. We will discuss a few below.

Calculating a rolling sumLet’s say you want to calculate a rolling sum for the profits of a food truck on a daily basis. For now we won’t complicate it by adding in month or year. Simply a rolling sum for all time.

How would you do that with SQL? Any ideas?

Well, there is actually a really simple way to do this with analytic functions.


Select
    SUM(Profit) over (partition by food_truck_id order by day)  
   ,food_truck_id ,day  
FROM f_daily_sales

That is it…you’re done!
Wow! That was really simple. Using this quick script you were able to calculate the rolling sum for all the food trucks. That was almost too easy!

Now, in case you don’t know, using this style of “function() over (partition by x order by y) “ clause is what is known as either an analytic or window function. It allows the end user to do something similar to a group by, but instead of grouping under a specific set of fields, it partitions by that set of fields and then applies a function using the order by. This function can be a row_number, rank, sum, avg, etc. All of these functions essentially work in a rolling fashion based on the order by statement. Row_number will add a row number for the group of fields and restart at one once the group of fields is different.

As we said, analytic functions are very useful. So useful that sometimes interviewers don’t like you using them. So let’s say you’re in an interview and the interviewer says that they want you to resolve this problem without the analytic function…

Did your stomach just drop? Maybe this was the only way you knew how to solve this problem. Take a moment and think about how you could solve this problem with the basic SQL tools you have. Think about what the SQL statement is actually asking.

SELECT
   day ,SUM(Profit)
FROM  
      (
      Select  
           t1.day
         ,t2.Profit
     FROM f_daily_sales t1
     JOIN f_daily_sales t2
   On t1.day >=t2.day
   ) t3
Group by
   day
​
In this case you can use the self join with a greater than or equal to statement creates a table like the one below

As you can see you will be able to group on the t1.day field and get a rolling sum effect. One of the major downsides depends on the amount of data. Because depending on how the data is structured and how far back you would like to go, this method could cause a system to run out of memory.
If by any chance this is a problem. Here is another thought. You could create a summary table that basically aggregates the total per day.

This depends on you are partitioning the rolling sum. For instance, this will work well as long as you know you will constantly be reporting the rolling sum on a monthly and yearly basis. This would lock your design in, but add both speed and a reduction of the computer when you actually query. Again this is more of a design question. What is the right answer depending on your overall goals?

Joining consecutive events
Another question you might be asked is to provide insight into consecutive events for patients, customers, or other entities. For instance, you might want to calculate the number of days between the time a customer visits a store, or your website. Maybe you want a simple flag that dictates whether or not a patient is readmitted in x days after each their initial visit. Well at a basic level, these have a similar goal.

You need to know how many days are between two different transactions.

Most transactions are lined up row by row, so you can’t simply go to the next row to figure out this information.
This is where an analytic function can work again. Using the row number function you are able to partition the data by each customer and patient and add a row number to each event. What would like below?

Using this as your base data you can self join this table to the next row using a +1 on the row number row to join on the following row.

Select  
    Customer_ID
   ,row_number() over (partition by Customer_ID order by date) r_num  
   ,date ,total_paid
Into #Customer_Transactions  
From Customer_Transactions


Select
     t1.Customer_ID
    ,datediff(dd,t1.date ,t2.date) --this will calculate  the difference in days
From #Customer_Transactions t1
Join #Customer_Transactions t2
on t1.Customer_ID = t2.Customer_ID
​And t1.r_num = t2.r_num+1

With this self-joined the table and each row can be looked at side by side you can tell more information. You can answer questions like, how many days were in between each event on average, or how many times did customers return in less than 15 days.

This can be a great starting point into some analysis focused on retention or policy improvement. You can start to see where your problem areas are, and possibly notice trends between stores or doctors based on they perform.

Another approach that some people like to take is looping through the data. This can get a little more complex and is usually slower, but it is another solution we won’t go over in this post.

These were two examples of solving SQL problems. We will continue to post various problems and different ways you can solve them. We might even re-approach the same problems above. That is what is great about SQL, depending on how your data is laid out and your limitations force you to use different solutions. You could have worked somewhere with SQL Server and get accustomed to temporary tables that allow you to put indexes on them. Then you go to the database that no longer allows temp tables and your performance goes way down.

We would love to come and help increase your teams analytical and SQL abilities. Please feel free to reach out! Also, please let us know if you have any questions or problems you are working through. We would love to poke around on new problems!

Interested In Reading More About Being A Better Data Scientist?
How To Grow As A Data Scientist
Boosting Bagging And Building Better Algorithms
How To Survive Corporate Politics As A Data Scientist
8 Top Python Libraries For Machine Learning
What Is A Decision Tree

During our years working for hospitals and insurance providers, there were many similar issues that caused unnecessarily increased costs. Healthcare providers, like many other industries, are facing drastically increased costsand decreased margins. Unlike tech companies that had the ability to develop technology to help their basic operations automate and scale, hospitals and healthcare providers have not. This has caused multiple administrative and analytical costs to grow year over year because of the lack of automation and process improvement. Some of the biggest costs that we have seen heavily weigh on healthcare providers are billing and financial processes, fraud detection and third-party contract management. Automation and process improvements are needed in these three areas if healthcare providers want to start reducing some of their biggest costs.


Billing And Financial Analysis
Managing billing and financial analysis can be very tedious tasks that require a combination of accounting discipline and entrepreneurial spirit. They are necessary practices that need to happen to manage expenses and revenue.

The problem is that even in billion dollar healthcare organizations lack automated billing and financial processes. Instead, hundreds of thousands to millions of dollars are often manually managed in excel spreadsheets. The processes to get the data and then slice and dice wastes analysts time. There are lots of ways that these financial tasks can start to weigh on the financial teams as they are not scalable. As hospitals and insurance providers merge and grow the problem amplifies and becomes even more difficult to manually manage. Typically, hospitals will approach this problem by increasing their staff rather than create a process or system that can continue to scale basic financial analytics and billing. This leads to increased operational costs that are difficult to manage without laying off employees(this isn’t even considering the costs of hiring, firing, rehiring…).

The solution is automation(mic drop….).

Okay, automation is easier said than done.

It requires a combination of buy-in from multiple stakeholders to financially back the projects and trust from directors and manager who read the reports and output of these automated systems. However, when executed well, these systems can reduce hundreds of hours of manual work.
​
It seems complicated, but often times automation, when done well really isn’t. Many times it actually simplifies the overall workload on your analysts. There might need to be some upskilling in basic SQL. However, we are living in a data driven world and SQL is the data language(even NOSQL databases have SQL layers because that is what we humans understand).

If your team doesn’t have that the skill set to build the tools themselves, then find it. Automation consultants and engineers can be found both internally and externally. Once the main systems are built it can be maintained by people with a lesser knowledge in programming and automation. Automation will help provide consistency, reduce waste and allow your analysts to focus on more important work.

Fraud Detection
Fraud detection and adjudication are necessary practices of insurance providers. This is because healthcare fraud costs billions of a dollars a year. it comes from patients and also healthcare providers that charge millions of dollars of upcoded and wasteful procedure purely to bolster their bottom line. In fact, there are even consultants and websites that specialize in helping healthcare providers bill creatively and max out their claims. These practices don’t increase costs for the insurance provider, they increase cost for the patients at the end of the day.

Insurance providers have to pay analysts to spend the time to manually go through claims to look for possible patterns of abuse. If you notice, the same problem in billing is occurring here. The problem is the manual step. Healthcare billing and claims processing have gotten too big to effectively handle manually. The solution of manually processing claims just don’t scale. This is where the tools of automation, big data, data warehousing, and analytics work well. They allow a specialist to create systems to munge the data effectively and scale even as the data grows.

Currently, most insurance providers have teams for fraud detection but they often can only go through a small percentage of claims( even billion dollar companies usually only go through about 5–10% of claims manually).

Sometimes they will even hire outside firms to again manually go through and look for low hanging easy fruit. For instance, billing consultants found that 78% of 99215 codes in Wisconsin (highest level established patient office visit) were incorrectly used. This is a very easy issue to spot and it becomes very expensive very fast.
​
All of this is usually limited to manual processes. Getting data pulls from databases into excel and then slicing and dicing the output. The beauty of automated methods is that they can quickly reduce the number of claims required to manually process.

​

What is even better, once you have created a system, you can replicate the results at a regular cadence. Even as analysts leave and new hires take their place, it is much easier to train them to understand the results of a system that works rather than have them have to relearn what to look for in the raw data. This increases the number of claims an insurance provider adjudicates while at the same time improving the efficiency of the process.

Many think you need complex machine learning algorithms to detect fraud when you first start developing models and your data isn’t even classified yet. However, when starting out, it is important to focus on cutting the number of targets claims your analysts will look at by 60,70 or 80%. This doesn’t take a complex algorithm. It requires developing basic business rules that can help sort through the false positives.

After properly managing claims and tracking which ones are fraud and which aren’t then it is easier to develop a machine learning model because your data is classified(fraud, not fraud).

Fraud detection and adjudication is a slow and costly process for many healthcare providers. Automation and well-executed analytics offer thousands to hundreds of thousands of dollars of saving.

Third Parties And Contract Management
One of the ways to get some systems automated and integrated is to use third-parties. Hospitals, insurance providers and other healthcare institutions are not tech companies. They don’t focus on developing technical tools to help in their day to day operations and this is ok! What is not ok is constantly signing contracts with new third parties for redundant features. For instance, paying for multiple data visualization tools is pointless like Tableau, Quilk, and OBIEE all provide report features and yet some companies use all three and more. Similar things can be said about relying on multiple financial systems.

Having redundant third-party contracts causes several issues. One, and most obvious is uncontrolled operational costs. Besides the upfront costs of signing multiple contracts it also requires more employee to manage the systems, manage the contracts and deal with the billing. These contracts are also difficult to break without paying a hefty fee for breaking the contract early. Thus, it is very important to reduce redundancy because the costs will be more than the cost of the contract.

The second issues are that these systems don’t have easy ways to access the data behind them. Lacking access to your own data is a problem. It is difficult to make good financial decisions if you don’t know what is happening in your institution. When you rely on third-party software, you need to be aware of the terms of your contracts and the features of the software. Otherwise, your expenses will grow from nowhere and you won’t be able to make good decisions.
​
Administrative costs like billing, fraud detection, and contract management are driving up health care costs for insurance providers, healthcare systems and patients. With a little bit of automation and process management, many of these costs can start to be mitigated. Automating processes like billing and financial analysis can be done with a combination of SQL and python. Contract management requires a combination of process improvement and analysis of the previous contracts. In the end, it can all lead to scalable cost savings that can occur systematically and don’t constantly need manual intervention.

Does your team need help automating your data systems and analytics? Then please contact us today!

Read more about data science, healthcare analytics and automation below:
How To Use R To Develop Predictive Models
Web scraping With Google Sheets
What is A Decision Tree
​How Algorithms Can Become Unethical and Biased
How To Develop Robust Algorithms
4 Must Have Skills For Data Scientists

​In 2018 it is estimated the US spent 3.5 trillion dollars on healthcare. A cost that is growing at nearly 5% a year. Rapidly growing pharmaceutical and administrative costs are just some of the driving factors behind this growth.

​This leaves the question, how are we going to reduce healthcare costs? We can’t ignore the problem. Otherwise, we will be spending way more than $15 for a Tylenol in 20 years.

The first part starts with recognizing some of the largest problems in healthcare. We pointed these out in an infographic here(Fraud, waste, and administrative costs). There are nearly a trillion dollars of spending that could be improved if the correct processes and policies are put into place. However, simply knowing what the problems are is not enough. We need to know where and why they are occurring. This brings us to our first step in starting a healthcare analytics project.
​

​Develop Accessible and Accurate Data Systems
In order to find out where the problems are, you need data that accurately represents what is occurring in your hospital/healthcare system. Relying purely on anecdotal evidence is no longer sufficient. It’s great to start and can guide data scientists and engineers to develop to correct data warehouses and dashboards. All of which requires quality data that is accessible to the correct users.

Developing a data system (ETLs, data warehouses, etc) that is both accurate and is accessible to the right people is a necessary step in confirming what the biggest cost-saving opportunities are.
This means thinking about what data you have, what data you need, who needs to access it and what questions are you hoping to answer. Not all of these can be answered right away. So it would be unwise to start by developing data warehouses that heavily adulterated from their original formats. That is why most data teams develop systems with the same basic pattern(example below).

You start with raw flat files that get ingested into a raw database. That data is run through a QA process just to double check that the data is clean and makes sense. One the data is QAed it is processed into a set of staging tables with minimal business logic added. Typically, this is more of a mapping phase. Anything that needs to be standardized is, data that is duplicated is cleaned up and data normalization may occur. The goal here is to limit the amount of business logic and focus more on creating an accurate picture of what the data represents. Whether this is procedures done on patients, supplies used by a hospital or computers and devices used by employees. This data is just an abstract depiction of what is occurring at the healthcare provider.

Once the data is loaded successfully and again QAed (yes, QA should happen any time data has an opportunity to be adulterated) then it can be loaded into the tables that everyone that should have access to them can have access to them.

This base data layer will provide your data scientist the ability to create analytical layers that sit on top and populate various aggregate and metric tables.
​
This basic outline is tried and true. More than likely, your data warehouse team is already supporting a system similar to the one we described. Typically the problem lies in accessibility and silos. The data might be produced for various teams like finance, or IT. However, they might all exist in separate systems with different keys and structures. This makes it difficult to find value.

At this point, there needs to be a much larger initiative to provide the correct accessibility as well as clear connections between the data systems. This is why using different keys for the same data across systems is a bad idea. Going into this would require a whole other set of blog posts and discussions so we will leave this with the basic data system.

Define Problems Clearly
As data engineers and data scientists we often learn about the subject matter we are working on through osmosis. We have to understand what the data represents to a point. However, it is difficult for us to know all the nuances of a specific subject and to understand all the context around the problems we are trying to solve.

That is why it is important to try to clearly state the questions you are trying to answer as well as provide as much context as possible. It gives data professionals a better picture of the problem they are trying to solve. The better they understand the problem, the clearer the solution becomes.

Otherwise, they could spend hours going in circles or answering the wrong questions. This is actually pretty common because sometimes what a stakeholder says is understood completely differently by a data professional. Especially if they don’t know why. Then they have to come up with their own why that drives their analysis. That means they could provide the wrong support, or answers at the wrong granularity.

Create Concise Metrics
With all the modern verbiage floating around it can be tempting to attempt to create algorithms and metrics that require too much adulteration of the original data and the value it could offer.

Metrics that are concisely stated are also more easily understood by everyone. This allows managers to make better decisions because they understand what the metrics abstractly represent. Rather than having to struggle to get how some random ratio or calculated value means they should target customer x or y.

This starts with a well-defined population. Whether that population is procedures, people, transactions, etc. It represents a specific group of entities. It is not always wise to look at an entire population first. Larger populations are more likely to have confounding factors that are harder to eliminate with simpler metrics. Developing metrics focused on specific populations to start also makes it easier to notice patterns in larger groups.

Review The Outcome
Analyzing the outcome and trends in the metrics that are developed can help drive new insights and policies. This requires that the outcomes are actually studied. Not just when the results are initially released but on a regular cadence. Typically, the first few times the metrics are examined there can be immediate benefits as policies are changed, entities are reached out to (like in the case of fraud) and hopefully cost savings are found.

After the initial release of the dashboards, there needs to be a plan on how often the results will be reviewed. To often will cause unnecessary actions to be made before the previous ones have impacted the outcomes and not often enough (or none at all) will waste both the dashboard and the time spent developing it.

Make a plan with the necessary team members (usually a director or manager, some subject matter experts for context and one of the data team members). Having this mix will provide the correct amount of perspective while informing the director and data team member of any needs required from the subject matter team. The data team might need to update the metrics based on the findings
Present The Results SimplyData, programming, and algorithms can all get very complicated. As data scientists and engineers we focus on the problem for so long we can begin to think everyone understands the problem as well as we do. Yet, this is not usually the case.

That is why we need to take a step back from our work and attempt to view our research, models, and analytics from the perspective of a teammate who hasn’t stared at the same problem for the past three months. It can be tempting to put all the graphs and logic that were analyzed over the past three months into a presentation or report. But this is likely to confuse a stakeholder who hasn’t been involved in your healthcare analytics project.
 
Usually, all that is required is a few key numbers to drive decisions. When you present too many numbers you risk burying the lead behind. It is important to focus on the key facts and figures first and if support is needed to provide it in a follow-up. This can be hard to do because it feels like we as engineers didn’t do a lot of work when we only show such a small amount of work. However, the key here is an impact, not showing off.
​
Quality data and analytics helps target and track savings opportunities in healthcare. When you start with accurate data and then develop concise metrics your team has the chance to make informed decisions on policies that can positively impact patients lives and at the end of the day that should theoretically be the goal.

We hope this helps you in your healthcare analytics project. Please feel free to reach out to our team if we can go in depth on any point! Our consulting team would be happy to help.

To read more data science and analytic articles see the below!
How To Use R To Develop Predictive Models
Web scraping With Google Sheets
What is A Decision Tree
​How Algorithms Can Become Unethical and Biased
How To Develop Robust Algorithms
4 Must Have Skills For Data Scientists

One of the few things democrats and Donald trump might agree on is reducing pharmaceutical costs.

Why? Healthcare costs continue to rise every year. Increasing pharmaceutical costs, technology improvements and wasteful procedures are costing the US more every year.

This increase in healthcare costs is starting to be felt by everyone. Some can’t afford it, while others struggle to keep up with their medical bills. Those of you who are younger might not understand now, but in 30 years when you need to go to the hospital in 2048 and you’re charged $200 dollars for a single Tylenol pill it might make sense. You might understand how ridiculous healthcare costs have become if we don’t start fixing problems now.
​
Why are healthcare costs continually going up?

There are really many reasons driving up medical costs. With many causes comes many solutions to improve the situation. However, let’s start by defining some of the biggest problems from a financial perspective. Below is an infographic of areas where there are massive opportunities to reduce healthcare costs. There are billions of dollars spent that provide opportunities to reduce the overall US healthcare bill.

How Do You Reduce Healthcare Costs?
Reducing healthcare costs starts with recognizing the biggest problems that will have the biggest impact when solved. Problems like fraud, waste and inefficient systems are some of the largest costs on the healthcare systems.

How do you recognize the biggest costs? Well, that is where data comes in! When we examine the successes of some of the large corporations today like Amazon, one of the differences they have between less data driven companies is they avoid terms like “I think” without having the data to back it up.
​
Our team will be posting on how you can reduce your healthcare costs using data in the next few days. If you have any questions on the subject matter before then, please reach out!

​Read More On Data Analytics Below!

How To Develop A Successful Healthcare Analytics Product
How To Use R To Develop Predictive Models
Web scraping With Google Sheets
What is A Decision Tree
​How Algorithms Can Become Unethical and Biased
How To Develop Robust Algorithms
4 Must Have Skills For Data Scientists