Have You Ever Wanted To Read About What Goes On In The Backend of Pinterest, Google, Uber and your other favorite tech companies?

A great thing about the modern engineering world is that many companies are much more open about their solutions they have developed. Companies like Google, Chef, Facebook, Lyft, Uber and more, all post about some of the designs and tools they have recently developed

This can act as a lure to attract talent, advertisement for B2B customers and another avenue to build brand awareness. It also provides insights into creative solutions for big data, machine learning, data science and programming problems.

If you are lacking inspiration or just need an interesting read, then consider checking out these blogs!

Quora Engineering Blog
​
Quora has been a great place to go for very diverse engineering blog posts. They discuss NLP, big data, machine learning and almost every other buzzword. That is because Quora’s site needs to deal with a very diverse set of NLP problems. There are so many plausible subjects and questions that finding good questions to show to the write people requires a lot of interesting solutions.
Engineering at Quora

Read More Here!

Uber Engineering Blog

​Despite all the business and ethical scandals the Uber has gone through in the past few years. Uber’s engineering blog is a personal favorite. Uber, unlike many other tech companies operates in the real world. That is to say, they have to deal with real world logistical problems. They deal with real-time issues, GIS, pricing, safety of both drivers and passengers and so many other complexities that they are forced to be clever. This leads to some interesting posts!

Read More Here!

Databricks Blog

This blog does border on the side of somewhat salesy. But, if you are a Databrix fan and use it on a daily basis, then you might as well get some ideas for how you could be using Databrix better!

Read More Here!

Pinterest Engineering Blog

One of our other favorite blogs is Pinterest. Pinterest post’s a lot of great infrastructure blogs that we really enjoy reading. For instance, they recently posted Optimizing Kafka for the cloud. In this post they do a great job of describing the problem and their solution. This can help inspire you when you come up to a similar problem!

Read More Here!

Netflix Engineering Blog

Have you ever wondered how one of your favorite media distribution companies manages to stream so much data to your TV, Laptop and other surfaces? Think about it, Netflix and Youtube make up a bulk of the data flowing through the internet. Imagine how well of a system needs to be designed to make sure you get your Stranger Things uninterrupted at the right time!

Read More Here!

​Google, Facebook and Amazon’s Research Blogs

Companies like Amazon, Google and Facebook spend a good amount on research every year. Imagine if you could get a glimpse of what these times were looking into. Well with the blogs below you can read into what Google, Facebook and Amazon are researching.

Google Research 

Facebook Research

Amazon Research 

The BlackRock Engineering Blog

Have you ever heard of BlackRock? Some of you have, others probably haven’t Wherever you stand, BlackRock is the world’s largest Asset’s Manager. It has over $6.5 Trillion under management. A company with this much capital has to spend a lot of time developing better products. Especially with Fin-tech companies banging the down the doors!

Read More Here!

Lyft Engineering Blog

We already shared Uber’s engineering blog, now it is time to mention the other recent IPO ride sharing services blog. It is similar to Uber. However, we find it a little more focused on soft-skills compared to Uber. There is nothing wrong with that, we just tend to really enjoy reading the engineering focused blogs.

Read More Here!

Salesforce Developer Blog

If you work for any large company, then chances are you use salesforce. This means your company probably needs to pulls from Salesforce’s API. That means it is important to keep up to date with what new tools Salesforce is releasing as well as any changes you might need to keep up to date. Similar to Databricks, this blog is more for people who use or are thinking of purchasing Salesforce.

Read More Here!


Other Great Data Science And Engineering Resources!
The Interview Study Guide For Software Engineers
Learning Data Science: Our Top 25 Data Science Courses
The Best And Only Python Tutorial You Will Ever Need To Watch
Dynamically Bulk Inserting CSV Data Into A SQL Server
4 Must Have Skills For Data Scientists
Engineering Dashboards, Metrics And Algorithms Part 2
Read Last Weeks Top Ten Article For Python Libraries
How Algorithms Can Become Unethical and Biased

In our last post, we discussed getting a goal from high-level stakeholders and making sure it comes to you as a clear and concise statement or one pager.

This makes it easier for data engineers and product developers to see why they are creating what they are creating.
​
So in this step, we will be discussing the data source as well as starting our discussion on the value of designing prototypes. These topics are somewhat hinged on each other. The prototype can only represent metrics based on your data source.
​

​Prototyping

Design a prototype is a key step in developing any product. Now, in this case, I am referring to purely a design and not an MVP. The prototype doesn’t need to be at all functional. Instead, it is more of a tool used to sell to executives. This good is a fake report with somewhat accurate numbers, a set of slides that demonstrate how an application could work, etc. The point is to have something your audience can see and can visualize using. This will help you sell your final product, even before you have it. Oddly enough…this can be more valuable than your actual technical work because if you can’t sell an idea to your stakeholders, it won’t be picked up.

How much time you spend on developing your prototype can also depend on your company culture as well as what the companies focus is. If the company’s focus is to create analytics to sell to other companies, then they will spend a lot of time on prototyping (or at least, they should).

This is because they have to convince the company to continue paying for the product for years to come. Also, if your company is very rigorous when it comes to developing internal tools, then you again, might spend a lot of time designing and prototyping.

However, there are many companies that will draw up a dashboard on a whiteboard and call that the prototype. There is nothing wrong with that, but the final product might not have the same buy-in. This eventually leads to the final product being forgotten quickly. When the stakeholders are forced to be brought into parts of the process, and also forced to dedicate time to it(as long as the product has a clear impact) they will continue to use the tool after it is finished because they will feel more attached as well as understand the product better.

This isn’t a fact that I could quote, or provide research on, but I have at least seen it anecdotally to be true. When the end-users weren’t forced to put time into thinking about the product, they can easily dismiss it in the future. This, in turn, wastes the time of the engineers and the resources of the company.

Prototyping is a key step in the case. It helps crystallize what exactly the data engineers are trying to create and the impact it will have. In addition, it forces the stakeholders to own more than just the concept of the product.

The limitation of what the prototype can be when referring to data products and algorithms is the data source, which will talk about next.

Read The Rest Here

Are You Interested In Learning About Data Science Or Tech?
​
The Advantages Healthcare Providers Have In Healthcare Analytics
25 Of The Best Data Science Courses Online
Learning Data Science: Our Favorite Data Science Books
What Is Data Science Really As Told By An Ex-FAANG Data Scientist
How Algorithms Can Become Unethical and Biased
How To Load Multiple Files With SQL
How To Develop Robust Algorithms
Dynamically Bulk Inserting CSV Data Into A SQL Server
4 Must Have Skills For Data Scientists
SQL Best Practices — Designing An ETL Video

Our team has collected several posts on learning data science. These posts will cover courses, books, and youtube videos. We hope they can help you on your journey.

​25 OF THE BEST DATA SCIENCE COURSES ONLINE

Bootcamps and Specializations

1. Introduction to Probability and Data

This course introduces you to sampling and exploring data, as well as basic probability theory and Bayes’ rule. You will examine various types of sampling methods, and discuss how such methods can impact the scope of inference. A variety of exploratory data analysis techniques will be covered, including numeric summary statistics and basic data visualization. You will be guided through installing and using R and RStudio (free statistical software), and will use this software for lab exercises and a final project. The concepts and techniques in this course will serve as building blocks for the inference and modeling courses in the Specialization.

Take The Course

2. Full Statistics Courses

In this Specialization, you will learn to analyze and visualize data in R and create reproducible data analysis reports, demonstrate a conceptual understanding of the unified nature of statistical inference, perform frequentist and Bayesian statistical inference and modeling to understand natural phenomena and make data-based decisions, communicate statistical results correctly, effectively, and in context without relying on statistical jargon, critique data-based claims and evaluated data-based decisions, and wrangle and visualize data with R packages for data analysis.

Take The Courses

3. The Data Scientist’s Toolbox

In this course you will learn how to program in R and how to use R for effective data analysis. You will learn how to install and configure software necessary for a statistical programming environment and describe generic programming language concepts as they are implemented in a high-level statistical language. The course covers practical issues in statistical computing which includes programming in R, reading data into R, accessing R packages, writing R functions, debugging, profiling R code, and organizing and commenting R code. Topics in statistical data analysis will provide working examples.

Take The Courses

Read More Here

LEARNING DATA SCIENCE: OUR FAVORITE DATA SCIENCE BOOKS

In data science, there are many topics to cover, so we wanted to focused on several specific topics. This post will cover books on python, R programming, big data, SQL and just some generally good reads for data scientists.

Data Science Books

As a data scientist, you have a very important role. Your goal is to provide your company insights into improving the companies bottom or top line. The problem is, we can make data say anything we want. It can be very easy to manipulate data to prove that our feature was effective and it can be tempting if the company incentivizes that type of behavior.

Thus, a great general read for data scientists (and really anyone in our modern world) is Naked Statistics.

This is kind of like the much older book How To Lie With Statistics which you can read for free.

We do prefer Naked Statistics because it is a little more modern and covers much more complex statistical debauchery than its much older counterpart. It just goes to show you that numbers are at your whim and you have a lot of responsibility to make sure your numbers are right. If something seems amiss with your data…it probably is. Rather than reporting it out right away, think about how you might unknowingly be miss representing the facts.

Another similar book is Everybody Lies: Big Data, New Data, and What the Internet Can Tell Us About Who We Really Are.

LEARNING DATA SCIENCE: OUR FAVORITE RESOURCES FROM FREE TO NOT

Data science has many facets. Statistics, data cleansing, programming, system design and really…almost anything else data related depending on how large the company is.

This post will discuss our favorite resources for these topics. Now, most of these courses and books are primers for topics like statistics, Python and data science in general. They really will only provide the base knowledge. At the end of the day, real practical experience is one for the few things that will really train your data science knowledge. You should learn as much as you can from these resources and then apply for as many internships and entry-level positions as possible and study for interviews.

You will learn much more and gain more than just technical knowledge. You will also gain a lot of business experience.

Free Statistics Courses

Let’s start with, learning/reviewing basic statistical concepts. Many of you have probably taken a statistics course or two in college. But you might not remember everything clearly so it’s a good idea to review from the beginning.

It can be tempting to try to start taking on complex statistical concepts and models. But most algorithms and models require some sort of accuracy and hypothesis testing. This means you actually need to be able to understand concepts like p-test vs at-value, z-statistics vs t-statistics, ROC vs AUC., random variables, etc.

These all seem like basic concepts and maybe you kind of remember these words. However, we find they often get forgotten as many of us focus more on learning how to implement models in python and R vs basic statistics. Although both of these concepts do not necessarily rely on each other. You can start to assume you understand what the p-value means when you run models in either language without fully grasping the importance of it.

Statistics Courses

1. Khan Academy
This is why we recommend at least going back and walking through the Khan Academy statistics section. They cover concepts like Hypothesis testing, T-statistics vs Z-Statistics, Confidence Intervals, etc.

Khan is always a good place to start because the videos are a great combination of visual and audio examples.

Personally, there aren’t too many books we like when it comes to pure statistics. In the R programming section of this resource list, we will reference our favorite R + Statistics book.

2. Duke University On Coursera
For a full course that are free you can try Duke Universities Statistics Course. This is actually several courses that cover multiple types of statistics like classical vs Bayesian. These are two different methods that are worth looking into.
​
Python Videos, Books And Courses

Python is an interesting topic. The thing about python is there are so many plausible sub-sections of the programming language. For instance, when we prepare for interviews we always like to clarify which type of python questions we will be dealing with. Will we be asked questions that focus on concepts that are operational, analytical, optimization based, algorithms and data structures or possibly data science algorithms. All of these are different topics that have different styles of interview questions. Getting a question on how to traverse a binary tree is very different from having to implement a decision tree algorithm.

As a data scientist, typically you will benefit from the analytical and operational aspects of python. The operational portion will provide you with the ability to automate the boring stuff (<-3. as the cliche book is titled).

This book is great for really…any data focused person. Data scientists, business analysts, Business Intelligence Engineers, and Database developers all can benefit from automation. Now, you don’t need to use python, if you’re in a windows environment there is PowerShell Linux has to bash. Learning some form of scripting language helps improve your workflow and design thinking.​
​

​OUR FAVORITE PYTHON BOOKS, COURSES AND YOUTUBE VIDEOS

Python is a common language that is used by both data engineers and data scientists. This is because it can automate the operational work that data engineers need to do and has the algorithms, analytics, and data visualization libraries required by data scientist.

In both rolls, the need to manage, automate and analyze data is made easier by only a few lines of code. So much so that one of the books we have read and seen in many data focused practitioners libraries in the book Automate The Boring Stuff With Python.
​
The book covers python basics and some simple automation tips. This is especially good for business analysts who work heavily in Excel.
​
There are also books by O’Reilly that are also a great overview of the basics.

Read More Here

​Are You Interested In Learning About Data Science Or Tech?
Learning Data Science: Our Favorite Data Science Books
What Is Data Science Really As Told By An Ex-FAANG Data Scientist
How Algorithms Can Become Unethical and Biased
How To Load Multiple Files With SQL
How To Develop Robust Algorithms
Dynamically Bulk Inserting CSV Data Into A SQL Server
4 Must Have Skills For Data Scientists
SQL Best Practices — Designing An ETL Video

​

As a data engineer, data scientist or just a general data team, how do you go from high level goal to a final data product? What is the process taken to successfully create a product that provides insights and action?

Just to clarify a data product can be a model, a dashboard, a web app/API or even a simple excel output. But it all has to start somewhere.

Note: In this series we will be walking through creating a data product. In this case we will be basing it on a company that has both online and in person sales. We will be creating videos, posts, designs, like an actual project. Going through the SQL, python, and other possible steps.
If you are interested in keeping up with all our posts, then please sign up here.

Intro:

Let’s say we are a company like Ikea, Bernhardt or Dorel. A furniture company in the modern era would probably have an e-commerce website and in-person locations that sells furniture.

A top level director or VP at this company will decide they want to increase sales of product category X or Y, or they want to improve sales in a specific region, or perhaps improve profit margins, reduce costs etc. This is where the ball starts rolling for metric and KPI development.

High Level Goals

The problem with metrics is you can track everything. You can track down to very specific granularities. But this doesn’t always provide value.

In order to provide value you need to first define what is important to track. In order to know what to track you need to have a high level goal of what you are trying to improve. That is why the first step in the process is deciding what metrics will help track and support the business goals.

In order to track and create these metrics the data team will need a general understanding of what the business team is trying to do.

Nothing needs to be solidified, but it is good for the stakeholders to know what is important and what strategies they are looking to put into place. In addition, having some general goals like “we want to increase the average sale per person by 5%” is good because it makes it easier to foresee what possible strategies might work (but it also makes it easier to point out when the strategy failed).
So step one is making sure these goals are written out somewhere as a proposal. It doesn’t have to be more than a page. It just clearly states the background of the goal, the why, etc. This will help the data team decipher what the business team is looking for.
Why:

We pointed this out earlier, but the reason it is important for the business team to have a clear idea of what they are trying to do is because there are so many metrics that could be tracked.

This could cause a few issues. For instance, a good data team won’t start work until you have provided more context. However, a less experienced team might try to do the work and struggle to create anything of value. Instead, they might spend 6 months creating a product that has too many metrics, the wrong metrics or maybe they just never finish because it is really hard to make forward progress when you don’t know where you are going.

This is why it is important in step one for the business team to have a clear ask. Just to show some of the metrics that could be tracked we have listed them out below. They all could provide value, but it depends on what the overall goal is.

Metrics:
There are a lot of possible metrics out there. All strange abbreviations, with different rules and all that somehow track very similar things from different angles.

Let’s go over a few that the furniture store might be interested in using.

Average revenue per user (ARPU) — This would be a pretty standard metric. However, if in this case we are adding in the idea of some form of ad-campaign then there is another angle we might want to consider. That is when the person was exposed to the ad. We will discuss how to analyze this in a later post
​
Average Purchases Per Week,Month,Year Per Store — This metric can be tricky. This can be very dependent on how much your average product costs, and the types of up-sell opportunities there are. For instance, a car is very expensive, very few people are going to buy two at once. However, people may be interested in spending $1000 dollars for a dvd player to be included or a specific set of rims, etc. For instance on Amazon you are buying products that are usually within that 20 -200 dollar range. That is easy to bundle an extra book or two, or a pan when you buy a spatula, or something like that.

Customer Acquisition Cost (CAC) — This metric is used to calculate the cost to acquire new customers. It is important because one goal a company might have is to reduce the overall CAC. Depending on how this approached it can tell if ads are getting more effective or possibly if products are getting better. It would depend on what the company did prior to shift in CAC.

Number Of New Visitors e.e. Acquisition — This would be a whole number and not a difficult metric to calculate.

Percent Of New Users Compared To Base — Compared to the number of new visitor metric, viewing things as a percentage can tell you more fairly how your growth is going.

These are just a few examples of metrics and we will continue to walk through this process.

Up Next:

In our next article we will start to discuss taking data from an operational database and moving it into a data warehouse. This is a key step into analyzing data because it makes the data available for data engineers and analysts. We will discuss ETLs, data models, etc.

Please scroll to the bottom if you would like to sign up for our future articles and videos where we will continue the process of developing a data product.

Are You Interested In Learning About Data Science Or Tech?
25 Of The Best Data Science Courses Online
Learning Data Science: Our Favorite Data Science Books
What Is Data Science Really As Told By An Ex-FAANG Data Scientist
How Algorithms Can Become Unethical and Biased
How To Load Multiple Files With SQL
How To Develop Robust Algorithms
Dynamically Bulk Inserting CSV Data Into A SQL Server
4 Must Have Skills For Data Scientists
SQL Best Practices — Designing An ETL Video

DeepMind and Google: the battle to control artificial intelligence
By Hal Hodson
 
One afternoon in August 2010, in a conference hall perched on the edge of San Francisco Bay, a 34-year-old Londoner called Demis Hassabis took to the stage. Walking to the podium with the deliberate gait of a man trying to control his nerves, he pursed his lips into a brief smile and began to speak: “So today I’m going to be talking about different approaches to building…” He stalled, as though just realizing  that he was stating his momentous ambition out loud. And then he said it: “AGI”.
 
Read More Here

Learning Data Science: Our Favorite Resources From Free To Not

Today we wanted to cover some of our favorite resources for data science. As the title suggests, these resources will be from free to not. Some people like buying books and other people prefer online courses. So we have created this list of data resources that range from books to courses, from free to not.

Data science has many facets. Statistics, data cleansing, programming, system design and really…almost anything else data related depending on how large the company is.
This post will discuss our favorite resources for these topics. Now, most of these courses and books are primers for topics like statistics, Python and data science in general. They really will only provide the base knowledge. At the end of the day, real practical experience is one for the few things that will really train your data science knowledge. You should learn as much as you can from these resources and then apply for as many internships and entry-level positions as possible and study for interviews.
Read More Here

Object Detection with 10 lines of code
By Moses Olafenwa

One of the important fields of Artificial Intelligence is Computer Vision. Computer Vision is the science of computers and software systems that can recognize and understand images and scenes. Computer Vision is also composed of various aspects such as image recognition, object detection, image generation, image super-resolution and more. Object detection is probably the most profound aspect of computer vision due the number practical use cases. In this tutorial, I will briefly introduce the concept of modern object detection, challenges faced by software developers, the solution my team has provided as well as code tutorials to perform high performance object detection.

Read More Here

Loading files into databases are not exactly exciting work. It can be tedious and boring work. Often times it is constrained by tools like SSIS that freak out when a small change is made.
This is one of the core roles of BI and data engineering teams. They need to load files from operational databases into data warehouses. They do this by developing pipelines that process and transform data.
​
One problem we see is teams have multiple file types, data spec versions, and data providers. Each of which requires a different pipeline just to load into their raw database.

This forces teams to manage 10, 20 maybe even 50 smaller pipelines that are all subject to the change of the data providers. In addition, each new data provider requires an entirely new pipeline to be developed. It also requires managing infrastructure around each pipeline in order to log and track data pipelines statuses. This can quickly become very difficult to manage and tedious to build.

Another solution instead of attempting to manage multiple pipelines is to develop a central system that manages the various insert statements required to insert data into each different raw table. Kind of how we explain in the video below.

What the system does is essentially act as a lookup system that matches flat files to their insert queries. This reduces the need to redundantly have separate pipelines that would also all need an insert statement.

Instead, now the system uses one main system to load all the various tables. Theoretically, you can also put version numbers on each of the files that come in. This could be set up by tracking each version in the metadata table that will then allow you to tie a raw file to an insert file version.

Loading multiple file types by individual pipelines requires a lot of tedious development. Whether it be through SSIS or another system, the constant development quickly weighs heavily on data engineers. Instead of spending time developing analytical pipelines that drive clear value for your company. They will be stuck maintaining and developing operational pipelines.

This reduces its impact and reduces its overall value.

Thank you so much for reading! We hope you enjoyed the video. Please feel free to send us any questions. We would love to help you design your next automation system.
Also, if you liked this, feel free to check out our video on using bulk insert!

For further reading and videos on data science, SQL and Python:
How To Develop Robust Algorithms
Dynamically Bulk Inserting CSV Data Into A SQL Server
 SQL Best Practices - Designing An ETL - Part 1
How Algorithms Can Become Unethical and Biased
4 Must Have Skills For Data Scientists
What is A Decision Tree

By Ben Rogojan
We recently did an AMA on Reddit. The most common question that came up was what is the difference between a data scientist and a data engineer. So we wanted to make a more in-depth post on the subject.

There are a lot of data specialist positions that sound similar and use similar tools so it can be difficult to know what the role of each role should do. In addition, smaller companies might be limited on how many data engineers or data scientists they can hire. This means that many times the specific tasks and goals might start to intermingle.
​

This can make it much more difficult to clearly differentiate the two roles. So we wanted to go over how the two positions were different by discussing the different goals, mindsets, tools and backgrounds data engineers and data scientists have.


Before we go into the difference we would like to make a quick preface. The truth is, many data scientists and data engineers will perform the tasks of other technical roles. A data scientists may need to develop an ETL and a data engineer might need to develop an API and front-end. So the distinctions we are pointing out below are just to make clear where the technical differences are.

GoalsThe goals of a data engineer are much more task and development focused. Data engineers build automated systems and model data structures to allow data to be efficiently processed. This means the goal of a data engineer is to create and develop tables and data pipelines to support analytical dashboards and other data customers (like data scientists, analysts, and other engineers). It’s similar to most engineers. There is a lot of designing, assumptions, limitations, and development that occurs in order to be able to perform a final task. Each design and solution having its own set of limitations, even if it all can perform the end task.
​
In comparison, data scientists tend to be question focused. In the sense that they are looking for ways to reduce costs/increase profits, improve customer experience or business efficiencies. This means they need to ask and then answer questions ( ask a question, hypothesize and then conclude). So they need to ask questions like, what impacts patient readmission, would a customer spend more if shown an add like A vs. B, is there a faster route to deliver packages? Skipping over the rest of the process. The goal from here is to find an answer to whatever question is posed. It might be a final conclusion or to more questions. Throughout the process, data scientists analyze, gather support and can develop a conclusion to the question.

Tools

This is where things can get confusing. Data scientist and data engineers both often rely on python and SQL. However, how the two tech roles use these skills varies. Again, this ties back to the mindset differences. Python is a very robust language that has libraries that help manage operational tasks as well as analytical ones.

Data scientists will use languages like Pandas and Scikit Learn whereas data engineers will use python to manage pipelines. This is where libraries like Airflow and Luigi come in handy.
Similarly, data scientist queries will be ad-hoc focused (e.g. questions focused). Whereas data engineers queries will be focused on cleaning up and transforming data.

Now there are other tools both data specialists might use. This includes Tableau, Jupyter notebooks and a handful of others. At the end of the day, it is how they use them that makes the distinction.
BackgroundsNow, one other common question when it comes to the differences of data engineers and data scientists is what background is required.

Data engineering and data science both require some understanding of data and programming. Even if it is a limited scope. However, there are some distinctions that go beyond programming. Specifically for data scientists. Due to the fact that a data scientist is more like a researcher, having a background that is research-based is a benefit.

This might be in economics, psychology, epidemiology, etc. Combine a research background with SQL, Python and a good sense of business and you have a data scientist. Now, these are not set in stone degrees. In fact, we have run into a data scientist with various degrees. Most employers will prefer to hire a data scientist with at least a masters degree that has some sort of technical or mathematical focus.

Data engineering positions usually won’t require a masters degree. Data engineering is more about being a developer. This requires much more practical experience rather than theoretical knowledge. So gaining a Masters does not supply the same value.

An Example
Let’s say a director of a healthcare company decide they would like to figure out how to reduce the number of patients readmitted prior to 30-days from their original visit. From a data point of view, there are a couple of things that need to occur.

Data scientists will need to figure out what drives patient readmission. That is the question they will be trying to answer. Based on the conclusions they reach, they will work with the business to develop metrics and policies to help improve patient readmission rates.

Data engineers will be developing tables to help support the data scientists answer the question while at the same time developing analytical tables to help track past and future patient readmission metrics. How these metrics are created will be driven by the answers the data scientist get.

Data scientists and data engineers have plenty of differences. They have different goals, and backgrounds, but this is where the value of utilizing both together comes from. The fact that data engineers focus more on engineering robust systems allows data scientists to query data easily and analyze it efficiently. Their partnership is what brings companies value from data.

We hope this post was helpful! Please feel free to reach out with any questions you may have.

Read more about data science, healthcare analytics and automation below:
​How To Develop Robust Algorithms
Dynamically Bulk Inserting CSV Data Into A SQL Server SQL Best Practices - Designing An ETL - Part 1How Algorithms Can Become Unethical and Biased
4 Must Have Skills For Data Scientists
What is A Decision Tree

Data engineering has many facets. One of the most common projects a data engineer takes on is developing an ETL pipeline from an operational DB to a data warehouse. Our team wanted to cover the overarching design of an ETL.

What are the typical principal components, stages, considerations, etc?

We started this by first writing Creating An ETL Part 1(more to come) and we and now have worked on a video that is below that walks through the process. We wanted to discuss why each stage is important and what occurs when data goes from raw to stage, why do we need a raw database and so on.

Data engineering is a complex discipline that partners automation, programming, system design, databases, and analytics in order to ensure that analysts, data scientists and end-users have access to clean data.
​
This all starts with the basic ETL design.

We are practicing up for a webinar that we will be hosting on ETL development with Python and SQL. The webinar itself will be much more technical and dive much deeper into each component described in the video. However, we wanted to see how using the whiteboard was like in case we need it.
If you would like to sign up for the free webinar we will be hosting it on February 23rd at 10 AM PT. Feel free to sign up here! If you have other questions please do contact us.

Read more about data science, healthcare analytics and automation below:
​How Algorithms Can Become Unethical and Biased
How To Develop Robust Algorithms
4 Must Have Skills For Data Scientists
Time Series Model Development: Basic Models
How To Measure The Accuracy Of Predictive Models
What is A Decision Tree

An ETL (and it’s not so far off cousin ELT) is a concept that is not usually taught in college, at least not in undergrad courses. Yet, ELTs play an important piece of almost every company’s day to day operations.

ETLs are the pipelines that populate data into business dashboards and algorithms that provide vital insights and metrics to managers. These pipelines go from very simple processes that extract data from operational databases into a data warehouse to complex analytical layers that support more complex transformations and applications.

The problem is, many of our IT graduates are not taught this very important skill that they will more than likely use in their everyday life if they decide to pursue a career in data science or data engineering. Although data science work doesn’t really include ETLs technically, a lot of work they do can end up crossing over with data engineers.
​
We wanted to cover the basics of an ETL and good ETL design.
What does a good ETL look like, what are the key components, what makes a pipeline maintainable and healthy? These are all important questions data engineers and database managers need to ask before taking on new projects.

This will be the first of several posts. In this post, we will be covering some basic concepts such as extracting/loading raw data and logging. These are important topics that have a lot of nuances that we would like to start to cover

We will also be offering a free webinar February 23 10 AM PT to cover this topic. We would love to have you attended!

Operational To Raw

Operational Database
All of the data your ETL will be ingesting will have some source that represents real life transactions of one sort of another. The data could represent patients visiting hospitals, people posting on social media or customers purchasing products. Some might ask why you can’t just analyze data from the operational database instead of going through the extra steps of processing the data into another system.

The biggest issue is that trying to run analytical processes that aggregate and analyze large data set from the operational database will slow down the operational database for customers. In addition, operational databases are heavily normalized which make them inefficient to query from an analytical standpoint. This is why a data warehouse of one form or another is devel

Here is an example of a method for bulk insert data into SQL Server

The first step in the processes is to extract the data into some form of raw format.

Raw File Extracts
How the data is pulled from the operational database can vary. However, one of the most consistent ways is to extract the data into a raw file like a CSV or XML file. Extracting the data into a raw file has several benefits. One, a direct link that pulls data limits the interactions outside systems have with the operational database. This will avoid blocking important operational tasks that need to occur. In addition, it creates a snapshot of what the data looked like at the time of pulling. This can be beneficial if you ever need to reload data because of an issue that might occur in the pipeline.

Having this back-up of data is beneficial just in case someone accidentally deletes crucial data. Then you will be very happy to have raw extracts that are easy to reload. These raw extracts should be tracked in some table that keeps track of what files have been loaded and where they exist. That way you can easily automate any reloading. Also, the files should have a naming standard.

An easy example would be DateExtracted_DateRangeInTheFile_BusinessObject (e.g. 20170101_20160901_20161231_Customers.csv)

Metadatabase And Logging
Tracking what is occurring inside your ETL process is a crucial step in developing an automated, maintainable and robust system. How extensive your tracking and logging depends on how large your company is and how many ETLs/ELTs and other data transformations need to occur. Larger companies might have a standardized tool like Airflow to help manage DAGs and logging.

However, if you are a start-up or a non-tech company, it will probably be ok to have a simplified logging system. The purpose of this system is to manage the files that need to be loaded/have been loaded, tracking of what stored procedures have run, errors, dependencies, etc.

Having good insight into these aspects of your system will help maintain it, improve it and backfill any data that either has been missed or loaded incorrectly. This means you need to actually create 3–4 tables (at least) that track what is running, how long do they take, computation, etc. Especially as we push for more and more automated systems, this will be the key to making sure the systems are maintainable.

Loading Raw Data Into Raw tables
Once you have extracted the raw data from the operational databases and updated your metatable with the new file information, then you can start focusing on loading the data.

It can be tempting to try to add a lot of complex business logic to your initial loads into raw. It can seem to simplify the process and create fewer steps in order to load the data. So why add more steps. The truth is adding too much logic to early can make it difficult to track where errors occur. Did the error occur in the extract, in the business logic or somewhere else? In addition, it becomes more difficult to update new logic because you have to analyze where the best place to update the logic will be.

All of these extra considerations make the overall system more difficult to maintain. That is why it is important to just load the raw data as is. If there is de-duplication logic or mapping that needs to happen then it can happen in the staging portion of the pipeline.

The next steps after loading the data to the raw database are QA and loading data into the staging database. We will continue that discussion in our next post. If you want to learn more about ETL development/automation then sign up for our ETL webinar where we will be discussing the process of creating an ETL from end to end.

Read more about data science, healthcare analytics and automation below:
Calculating A Running Total SQL
How To Develop Robust Algorithms
4 Must Have Skills For Data Scientists
Time Series Model Development: Basic Models
How To Measure The Accuracy Of Predictive Models
What is A Decision Tree
Automation Best Practices

It might seem strange to have data stored this way, but there are cases where condensing the data in this fashion might make sense. Let’s say you’re team is storing billions of transactions per day and you want to run a median calculation over several years.

In this case, it makes more sense to keep the data aggregated because it will improve the performance. You might not even be able to properly store the transactions if it isn’t aggregated.
It could be tempting to “unroll” the data so to speak. To take the data and create a new table that has a row for each of the instances of a person at a specific age. This could cause your rows to grow significantly, to terabytes upon terabytes. Also is just another step that might not need to be done.
Instead, let’s look this query.

SELECT TOP(1) Age
   FROM  
       ( SELECT Age
               , SUM(Total) OVER (ORDER BY Age) AS runsum
            , SUM(Total) OVER () AS total
        FROM Age_Exp ) AS d  
​WHERE runsum >= total / 2 ORDER BY Age;

​This query is short and sweet and you will also notice it brings in the running total we discussed recently(video). The goal of the medium is to find the number halfway through all the data. So that means you need to know the total and find the number halfway through.

The way this works is by using the rolling sum you can figure out where the half-way point will end up because the running total will sum of the total value.

Then using the other analytic function that gets the sum you can divide it by half and only select the top value. The median will be the top value.
​
We would love to know your thoughts on this problem. Also, if you have any problems you would like to discuss then please feel free to contact us! Thanks for reading.