One of the common questions we get when it comes to learning how to program, is: “What are some good ideas for projects to build?”

Now, we hear the common cliche answers often, like “build a chess game” or “command line interface”. There is nothing wrong with these answers.

However, we think these examples don’t match modern programming needs. A big portion of modern software is SaaS and web apps. This means you need to know how to program online.
There are a lot more complexities that go into programming a website, or app, that has users, requires servers, authentication, and databases. This forces you to interact with technologies you’ll never need when developing a command line tic-tac-toe game.

Some of this can also be managed by AWS and other third parties that are commonly used by large corporations. Again, exposing you to technologies that are useful and heavily used in industry. This will be far more practical on a resume, as well as help you learn how to use new technologies.

Tip: If you pick a complex project, then focus on building one feature at a time. Building an entire website or app all at once is difficult. Start by building a login page, or maybe the main landing page after the user logs in.If you try to take on the whole project at once then you will likely fail.

Entertainment

1. A web scraper that posts top 10 blogs without human intervention

One of the issues we find with some project recommendation posts, is thatthey recommend projects that aren’t implemented in a way that excites the programmer to continue development. For instance, I see that a lot of people recommend building a web scraper.
Once you’ve built that web scraper and scraped the data, what are you going to do with it?

Instead of just scraping the data, why not build a website with that data. It doesn’t have to be fancy or get a lot of views. This scraper could pull the data into a database and then select the most popular posts. From there it could copy the title, along with a few sentences, and then create a post that it shares online. This would be an impressive and simple project that you can actually show off.
You’ve now shown that you can do more than just code a small segment of a system. Instead, you can think through an entire system. You need to consider how you are going to automate the process, manage the database, create the website and select the posts. This also allows you to actually have a tangible end product.

Without a tangible end product, it’s really easy to become unmotivated and simply stop at only a web scraper.

In addition, you never know, maybe your site will become popular!

Skills: Database, web scraper, automation, web development (for the blog), and general programming.

2. An event-alert system using Meetup and Eventbrite APIs

Have you ever wanted to go to a band or comedian show, but realized it was last week? Maybe there was a free conference in your area on data science or big data and you missed out because you forgot to check.

Why not make your own aggregator using the Meetup and Eventbrite APIs, that will warn you when keywords are in event descriptions or titles? Now, I assume both Meetup and Eventbrite have similar options. But it is always fun to try to build your own system.

You can customize the system to work the way you want, and maybe even allow other people to make their own alerts by making this a website. What we enjoy about this project is that you can practice working with two different APIs. This will allow you to compare and contract what you like and dislike about them. That way, if you’re ever in charge of building an API, you’ll have a better picture of what works and what doesn’t.

Skills: APIs, database, automation, web development, and general programming.

3. A 9GAG copy cat

You don’t always need to try to reinvent the wheel when creating your own projects. Simple projects like a site that lets you login, post photos, GIFs, and lets you scroll through a feed, provides an opportunity to create a solid base site first. Then you can add lots of interesting features like following, liking, and search. Search in particular would be a great chance to learn how recommendation systems and machine learning work!

It’s always fun to try and replicate popular sites. In fact, it is actually a great way to learn because you have to reverse-engineer each feature. Reverse engineering is a great skill, because as a software engineer you will constantly be maintaining other people’s code and you will need to get in their heads.

Skills: Machine learning (for recommendation system), database, automation, web development, and general programming.

Retail Type Sites

4. A gift recommendation app

Have you ever struggled to find the right gift for your friend? What if you could create a website that helps to predict what to buy a friend for a gift. It could allow the end user to either create an account or just get a gift recommendation.

Again, this allows for the opportunity to create an account which requires authentication, database development, etc.

Also, another great part about this project is you can use Amazon’s API for affiliate links. This will allow you to do a few things. One, learn about how to use APIs and get you comfortable with reading API documentation. Two, if you do it well, you can get a commission for each product someone buys.

This project also has an opportunity to try to create a basic machine learning model. You can create a quiz of sorts that tries to figure out what the best gift is and then, based on if people click the gift or not, can drive the model to learn based on the response rate.

Skills: APIs, database, general programming, and app development.

5. A site for bartering and trading

Think OfferUp, but instead of money, why not create a website that only allows trades. This concept will force you to develop several features that need some thought. You won’t be able to just attack this project without a plan.

How will people post, where will people find recently posted items and how will people search. All of these are separate features you can build. In addition, you need to think how users will interact, and maybe even how they actually make the trade.

The idea doesn’t have to be 100% practical for real life — it needs to be practical in the sense of improving your skill set as a programmer.

Skills: Database, web development, general programming, and app development (if you choose to make it an app).

B2B

6. Invoice and contract management system

Contract and invoice management are very complex processes. Contracts can have a lot of nuanced clauses and stipulations that can be difficult to track.

This makes this a very good project, even if you simplify it to some of its core components. Having to translate a complex business process into software is not easy. But it is what makes this project a good challenge.

Again, we wouldn’t overcomplicate this. Take a basic feature, like inputting the terms of a contract, and develop this part first. Then you can add other features like invoice tracking, contract analytics and forecasting.

Skills: Process management, database, web development, and general programming.

7. Task management system

Task boards like KanbanFlow are built with several modular features that make it a great project to play around with. It will take a little work to get started, as you will need to set up a UI that is robust and dynamic as well. In fact, this project would be more of a two person job. One person to work on the front end and another person to work on the back end.

Don’t let that discourage you! This is actually a chance for you to work on your communication and team work skills. You will need to talk through designs to make sure you both fully understand it, and you know where your modules will be connecting.
This is always more challenging than it seems.

Skills: Communication, front end, database, web development, and general programming.

8. A job board

Any project that forces you to allow users to input as various types of users adds an interesting design aspect. How will you ensure that the way employers experience the site meets their needs vs. prospecting job searchers? Like most of the other projects, you don’t need to focus on all of it at once. Start out by trying to create the ability to create a job posting first. Then you can go and focus on the job searchers and how they respond.

Skills: Database, web development, and general programming.

9. A website that forecasts profits based on standardized data sets

There are a lot of data sets that are very standardized for most companies. This includes accounting data which is usually based on cost centers, accounts, line descriptions, and finally the actual transaction cost.
What is great about the standardization of any data set, is that it makes it easy to create analytics on top of said data sets. Why not create a standardized dashboard that can help companies predict spend, see monthly outgoings, and possibly help them improve their spending.
For this project you will probably have to spend a lot of time learning about how to make sure you keep your data secure. Of course, we recommend first trying to build the modules that focus on uptaking and standardizing the data and displaying it, before you go too deep into security. That’s a rabbit hole you may never escape!
Skills: Forecasting, business logic, database, web development, and general programming

Game Ideas

10. Snake

If you had a cellphone in the early 2000s, you’ve probably played Snake. It’s a simple game but you can always try to make things more complex! First, start by just trying to make the game.
This will require you to figure out how to develop a game online. This neon Snake by Sebastian Opperman is a great place to start. But after that, maybe you can add some cool new features like special items or special powers.

This would be a chance to play around and have fun. This project won’t be as technical from the stand point of having lots of users that sign up and use your site. However, it is a good challenge to figure out how to make a game run online.

Skills: Web development, general programming, and UI

We do hope this list inspires you to create an awesome new project that you can add to your resume and talk about in interviews. Maybe we’ll see you as the next CEO of a billion dollar startup!

Are You Interested In Learning About Data Science Or Tech?
10 Amazing Articles On Python Programming And Machine Learning
Learning Data Science: Our Top 25 Data Science Courses
The Best And Only Python Tutorial You Will Ever Need To Watch
Engineering Dashboards, Metrics And Algorithms Part 2
Read Last Weeks Top Ten Article For Python Libraries
The Best And Last Python Tutorial You Will Ever Watch
How Algorithms Can Become Unethical and Biased
Dynamically Bulk Inserting CSV Data Into A SQL Server
4 Must Have Skills For Data Scientists
SQL Best Practices — Designing An ETL Video

In the past few weeks, we have watched Tableau be purchased by Salesforce for 15.7 billion dollars and Looker for 2.7 billion dollars by Google.
​
Why are these companies so valuable?

Why are so many companies willing to pay for the services that Tableau and Looker provide?
The truth is dashboards can be expensive wastes of time and resources or they can provide insights that are worth exponentially more than the original investment. In fact, creating dashboards is a very lucrative business.

I worked for a company whose business model revolved around selling dashboards and insights to external customers. We only had four dashboard products. That was it, and yet, this company has existed for over a decade.

This is because they were really good at creating dashboards that provided impact continuously. These weren’t dashboards that you use for a month or two and forgot about. Instead, the dashboards they created were concise, clear and helped directors make decisions quickly.
They were able to do this by creating dashboards that followed a few basic principles. We wanted to share some of these basic tips below. Who knows, maybe your team will build the next million dollar dashboard.

Don’t Bury the Lede

When developing a dashboard, you need to first entice your audience with a high-level metric. This could be the total amount of money lost in insurance fraud for the year, a ranking that highlights how well a company is doing, or perhaps a trio of specific metrics.
The point of these high-level metrics is to pique interest. It needs to capture the audience and force them to ask questions and keep digging deeper into the dashboard.
Specifically, you want them to ask questions like why is the number so high or low and what are the factors driving this number?
Let’s walk through an example (see the dashboard below):

​

This dashboard is actually pretty well done. It is clean and provides very succinct information.

However, there is no initial draw. There is no distinct sign of whether the company’s income is bad or good.

So why should I be interested? I don’t really have any questions off the top of my head because I wasn’t primed to do so by any driving metric or dollar figure.

Having the dollar amount that depicts how far off target the net-income is in the top-left corner along with the percentage would have been a good design decision. This set of numbers would drive me to ask the right questions. Specifically, why is the net profit above or below what is expected?
This would then drive the end-reader to look into the charts below. Net profit exists on this dashboard, but it is hidden in the corner. Essentially burying the lead.
​
The most important information should be prioritized to the top of the dashboard.

​​Focus on Substance and Style

Creating a great dashboard is like working as a chef in fine dining.
If you focus too much on the aesthetics and not on flavor, your food will taste terrible. If you put no focus on aesthetics, your customers won’t see the value in your $50 salmon and carrots dish.
It can be tempting to create very flashy dashboards with all the bells and whistles. Similar to the way it was tempting in the early 2000s to throw foams, spheres, and gels on every dish in the fine dining world.

But bells and whistles don’t tell a story.

They can accentuate it, but they can also distract the end-user and make it more difficult to make a clear decision.

So focusing too much on style will hinder rather than help.

​On the flip side, presentation is important too. Presentation can entice an end-user to pay attention, even if you have bad data. For example, let’s look at this beautiful dish below.

Created by @chefjacqueslamerde on Instagram…wait…thats just celery, peanut butter, and raisins…
But, somehow, it looks like so much more. Even bad data can sometimes be forgiven by a beautiful dashboard (this is not encouraging bad data!).

We are trying to accentuate the fact that design plays a role in how we perceive information.

As data professionals, we can sometimes focus solely on the numbers and metrics. Numbers alone don’t capture peoples interest the same way it might capture ours. Capturing peoples attention is where design principals come into play.

Again, if we look at the net income dashboard, it is clean and has a very simple style. It is pretty clear to get what is going on in each chart. Often people assume that having lots of bells and whistles make a beautiful dashboard.

It is actually usually the opposite. Keeping a dashboard simple with only the data required makes the message clearer. If you have too much going on you muddy the message and could confuse the end-user.

It’s almost like a dashboard’s UX should be treated like a website’s UX.

Don’t Hide Too Much Behind Interactions

We have mentioned a lot about avoiding bells and whistles because they often deteriorate the message you are trying to tell with a dashboard.

One more specific feature most dashboards allow for are filters and tool-tips. These are great in some ways until you put too much information behind them.

Requiring too many interactions from an end-user to get to data could dilute the message you are trying to depict.

Rule of thumb: If you can’t print your dashboard and tell your same story, then you have too much information hidden behind tooltips and filters.

One example of this is providing one too many filters that might interact with varying widgets on your dashboard. This forces the end-user to need to flip through multiple states of each widget to get the message you are trying to tell them.

Sometimes our desire to over-inform only further hinders the message.Dashboards are about prioritization of information. You should only be showing the most important information that can help an executive make a clear decision.

Most directors will want lots of options to slice and dice the data, but that doesn’t mean that you should do it. Too many interactions may push them to continually analyze new configurations of filters vs. making a decision. This means you will have to push back for a better design.

Provide Context

When you create dashboards you can often create them in vacuums. Teams often create dashboards or metrics and don’t realize that other companies or teams might not know what is going on. Especially when you start to create custom metrics.
This is why we recommend providing context for the metrics, the dashboard purposes, and the axes.

The metrics
Regardless if you assume your metrics are well known or esoteric. Defining what the metric represents in a clear and concise manner benefits everyone. This also forces teams to use metrics that are easy enough to explain to an end-user. Some teams might create super complex models to calculate metrics, but this often abstracts the value a metric can have. Instead, you should focus on simple metrics that you can explain to anyone.

If you can’t understand a metric, you can’t make decisions.

The dashboard’s purpose
The first time an end-user experiences a dashboard, they might not know why they are looking at it. Putting together a few sentences of what a user is looking at can help better frame their understanding. In many ways, you should strive to not require a summary, but it’s always good to have. Similar to documentation with code.

Good code should require little documentation, but that doesn’t mean you shouldn’t document your code! Always document everything.

The axes
How you define the axes isn’t always straight-forward. You want just enough tick marks that end-users can understand what the line charts and bar charts mean, but you don’t want so many ticks that it is confusing. This comes down to good judgment. For instance, when you are using a sparkline, you will often limit the tick marks, but if it is a main line chart, you will probably provide more context along each axis.

The Devil is in the Details

High-level metrics usually provide the “what” but not the “why”.

The high-level metrics push users to look for answers.

What is driving the high-level numbers like net profits?

Thus, the next level of your dashboard should help answer these questions.

This means you need to look at the top metric and think about what questions an end-user might have looking at it. At this step in your dashboard development process you should try listing out what questions you would have as an end-user. It is helpful to have a member or two of your team who hasn’t seen the dashboard yet write out their questions. That way you have a fresh pair of eyes.
Based on these questions you can start to create secondary dashboards that provide the required insights.

Tableau has a great feature called actions. This can allow you to create easy drill-downs that go to more detail specific dashboards.

The goal here is to provide high-level metrics and low-level details that can help directors make concrete decisions.

Conclusion

These are a few great tips we have learned from working for a company whose sole purpose was creating dashboards to provide value to external customers. We hope it helps you create amazing dashboards that can create value for your company. Remember to focus on creating dashboards that don’t bury the lede and allow directors to make good decisions easily. That is how you make a million dollar dashboard.

To contrast, too many bells and whistles and poorly designed dashboards can lead to bad decisions or no decisions.

If you have any questions feel free to send them our way!


If you enjoyed this video about software engineering then consider these videos as well!
142 Resources for Mastering Coding Interviews
Learning Data Science: Our Top 25 Data Science Courses
The Best And Only Python Tutorial You Will Ever Need To Watch
Dynamically Bulk Inserting CSV Data Into A SQL Server
4 Must Have Skills For Data Scientists
What Is A Data Scientist

Software engineers spend a lot of time gaining skills for interviews by practicing leet code problems and perfecting resumes.
Once they finally get that job at a start-up, Google, Amazon or another corporation you might find the skills they used to get the job don’t tend to match the ones you need in your everyday work.
Our team was inspired by the 7 skills of highly effective programmers created by the TechLead and wanted to provide our own take on the topic.
Here are our 7 skills of effective programmers.

Read the rest here!

​

Have You Ever Wanted To Read About What Goes On In The Backend of Pinterest, Google, Uber and your other favorite tech companies?

A great thing about the modern engineering world is that many companies are much more open about their solutions they have developed. Companies like Google, Chef, Facebook, Lyft, Uber and more, all post about some of the designs and tools they have recently developed

This can act as a lure to attract talent, advertisement for B2B customers and another avenue to build brand awareness. It also provides insights into creative solutions for big data, machine learning, data science and programming problems.

If you are lacking inspiration or just need an interesting read, then consider checking out these blogs!

Quora Engineering Blog
​
Quora has been a great place to go for very diverse engineering blog posts. They discuss NLP, big data, machine learning and almost every other buzzword. That is because Quora’s site needs to deal with a very diverse set of NLP problems. There are so many plausible subjects and questions that finding good questions to show to the write people requires a lot of interesting solutions.
Engineering at Quora

Read More Here!

Uber Engineering Blog

​Despite all the business and ethical scandals the Uber has gone through in the past few years. Uber’s engineering blog is a personal favorite. Uber, unlike many other tech companies operates in the real world. That is to say, they have to deal with real world logistical problems. They deal with real-time issues, GIS, pricing, safety of both drivers and passengers and so many other complexities that they are forced to be clever. This leads to some interesting posts!

Read More Here!

Databricks Blog

This blog does border on the side of somewhat salesy. But, if you are a Databrix fan and use it on a daily basis, then you might as well get some ideas for how you could be using Databrix better!

Read More Here!

Pinterest Engineering Blog

One of our other favorite blogs is Pinterest. Pinterest post’s a lot of great infrastructure blogs that we really enjoy reading. For instance, they recently posted Optimizing Kafka for the cloud. In this post they do a great job of describing the problem and their solution. This can help inspire you when you come up to a similar problem!

Read More Here!

Netflix Engineering Blog

Have you ever wondered how one of your favorite media distribution companies manages to stream so much data to your TV, Laptop and other surfaces? Think about it, Netflix and Youtube make up a bulk of the data flowing through the internet. Imagine how well of a system needs to be designed to make sure you get your Stranger Things uninterrupted at the right time!

Read More Here!

​Google, Facebook and Amazon’s Research Blogs

Companies like Amazon, Google and Facebook spend a good amount on research every year. Imagine if you could get a glimpse of what these times were looking into. Well with the blogs below you can read into what Google, Facebook and Amazon are researching.

Google Research 

Facebook Research

Amazon Research 

The BlackRock Engineering Blog

Have you ever heard of BlackRock? Some of you have, others probably haven’t Wherever you stand, BlackRock is the world’s largest Asset’s Manager. It has over $6.5 Trillion under management. A company with this much capital has to spend a lot of time developing better products. Especially with Fin-tech companies banging the down the doors!

Read More Here!

Lyft Engineering Blog

We already shared Uber’s engineering blog, now it is time to mention the other recent IPO ride sharing services blog. It is similar to Uber. However, we find it a little more focused on soft-skills compared to Uber. There is nothing wrong with that, we just tend to really enjoy reading the engineering focused blogs.

Read More Here!

Salesforce Developer Blog

If you work for any large company, then chances are you use salesforce. This means your company probably needs to pulls from Salesforce’s API. That means it is important to keep up to date with what new tools Salesforce is releasing as well as any changes you might need to keep up to date. Similar to Databricks, this blog is more for people who use or are thinking of purchasing Salesforce.

Read More Here!


Other Great Data Science And Engineering Resources!
The Interview Study Guide For Software Engineers
Learning Data Science: Our Top 25 Data Science Courses
The Best And Only Python Tutorial You Will Ever Need To Watch
Dynamically Bulk Inserting CSV Data Into A SQL Server
4 Must Have Skills For Data Scientists
Engineering Dashboards, Metrics And Algorithms Part 2
Read Last Weeks Top Ten Article For Python Libraries
How Algorithms Can Become Unethical and Biased

In our last post, we discussed getting a goal from high-level stakeholders and making sure it comes to you as a clear and concise statement or one pager.

This makes it easier for data engineers and product developers to see why they are creating what they are creating.
​
So in this step, we will be discussing the data source as well as starting our discussion on the value of designing prototypes. These topics are somewhat hinged on each other. The prototype can only represent metrics based on your data source.
​

​Prototyping

Design a prototype is a key step in developing any product. Now, in this case, I am referring to purely a design and not an MVP. The prototype doesn’t need to be at all functional. Instead, it is more of a tool used to sell to executives. This good is a fake report with somewhat accurate numbers, a set of slides that demonstrate how an application could work, etc. The point is to have something your audience can see and can visualize using. This will help you sell your final product, even before you have it. Oddly enough…this can be more valuable than your actual technical work because if you can’t sell an idea to your stakeholders, it won’t be picked up.

How much time you spend on developing your prototype can also depend on your company culture as well as what the companies focus is. If the company’s focus is to create analytics to sell to other companies, then they will spend a lot of time on prototyping (or at least, they should).

This is because they have to convince the company to continue paying for the product for years to come. Also, if your company is very rigorous when it comes to developing internal tools, then you again, might spend a lot of time designing and prototyping.

However, there are many companies that will draw up a dashboard on a whiteboard and call that the prototype. There is nothing wrong with that, but the final product might not have the same buy-in. This eventually leads to the final product being forgotten quickly. When the stakeholders are forced to be brought into parts of the process, and also forced to dedicate time to it(as long as the product has a clear impact) they will continue to use the tool after it is finished because they will feel more attached as well as understand the product better.

This isn’t a fact that I could quote, or provide research on, but I have at least seen it anecdotally to be true. When the end-users weren’t forced to put time into thinking about the product, they can easily dismiss it in the future. This, in turn, wastes the time of the engineers and the resources of the company.

Prototyping is a key step in the case. It helps crystallize what exactly the data engineers are trying to create and the impact it will have. In addition, it forces the stakeholders to own more than just the concept of the product.

The limitation of what the prototype can be when referring to data products and algorithms is the data source, which will talk about next.

Read The Rest Here

Are You Interested In Learning About Data Science Or Tech?
​
The Advantages Healthcare Providers Have In Healthcare Analytics
25 Of The Best Data Science Courses Online
Learning Data Science: Our Favorite Data Science Books
What Is Data Science Really As Told By An Ex-FAANG Data Scientist
How Algorithms Can Become Unethical and Biased
How To Load Multiple Files With SQL
How To Develop Robust Algorithms
Dynamically Bulk Inserting CSV Data Into A SQL Server
4 Must Have Skills For Data Scientists
SQL Best Practices — Designing An ETL Video

Our team has collected several posts on learning data science. These posts will cover courses, books, and youtube videos. We hope they can help you on your journey.

​25 OF THE BEST DATA SCIENCE COURSES ONLINE

Bootcamps and Specializations

1. Introduction to Probability and Data

This course introduces you to sampling and exploring data, as well as basic probability theory and Bayes’ rule. You will examine various types of sampling methods, and discuss how such methods can impact the scope of inference. A variety of exploratory data analysis techniques will be covered, including numeric summary statistics and basic data visualization. You will be guided through installing and using R and RStudio (free statistical software), and will use this software for lab exercises and a final project. The concepts and techniques in this course will serve as building blocks for the inference and modeling courses in the Specialization.

Take The Course

2. Full Statistics Courses

In this Specialization, you will learn to analyze and visualize data in R and create reproducible data analysis reports, demonstrate a conceptual understanding of the unified nature of statistical inference, perform frequentist and Bayesian statistical inference and modeling to understand natural phenomena and make data-based decisions, communicate statistical results correctly, effectively, and in context without relying on statistical jargon, critique data-based claims and evaluated data-based decisions, and wrangle and visualize data with R packages for data analysis.

Take The Courses

3. The Data Scientist’s Toolbox

In this course you will learn how to program in R and how to use R for effective data analysis. You will learn how to install and configure software necessary for a statistical programming environment and describe generic programming language concepts as they are implemented in a high-level statistical language. The course covers practical issues in statistical computing which includes programming in R, reading data into R, accessing R packages, writing R functions, debugging, profiling R code, and organizing and commenting R code. Topics in statistical data analysis will provide working examples.

Take The Courses

Read More Here

LEARNING DATA SCIENCE: OUR FAVORITE DATA SCIENCE BOOKS

In data science, there are many topics to cover, so we wanted to focused on several specific topics. This post will cover books on python, R programming, big data, SQL and just some generally good reads for data scientists.

Data Science Books

As a data scientist, you have a very important role. Your goal is to provide your company insights into improving the companies bottom or top line. The problem is, we can make data say anything we want. It can be very easy to manipulate data to prove that our feature was effective and it can be tempting if the company incentivizes that type of behavior.

Thus, a great general read for data scientists (and really anyone in our modern world) is Naked Statistics.

This is kind of like the much older book How To Lie With Statistics which you can read for free.

We do prefer Naked Statistics because it is a little more modern and covers much more complex statistical debauchery than its much older counterpart. It just goes to show you that numbers are at your whim and you have a lot of responsibility to make sure your numbers are right. If something seems amiss with your data…it probably is. Rather than reporting it out right away, think about how you might unknowingly be miss representing the facts.

Another similar book is Everybody Lies: Big Data, New Data, and What the Internet Can Tell Us About Who We Really Are.

LEARNING DATA SCIENCE: OUR FAVORITE RESOURCES FROM FREE TO NOT

Data science has many facets. Statistics, data cleansing, programming, system design and really…almost anything else data related depending on how large the company is.

This post will discuss our favorite resources for these topics. Now, most of these courses and books are primers for topics like statistics, Python and data science in general. They really will only provide the base knowledge. At the end of the day, real practical experience is one for the few things that will really train your data science knowledge. You should learn as much as you can from these resources and then apply for as many internships and entry-level positions as possible and study for interviews.

You will learn much more and gain more than just technical knowledge. You will also gain a lot of business experience.

Free Statistics Courses

Let’s start with, learning/reviewing basic statistical concepts. Many of you have probably taken a statistics course or two in college. But you might not remember everything clearly so it’s a good idea to review from the beginning.

It can be tempting to try to start taking on complex statistical concepts and models. But most algorithms and models require some sort of accuracy and hypothesis testing. This means you actually need to be able to understand concepts like p-test vs at-value, z-statistics vs t-statistics, ROC vs AUC., random variables, etc.

These all seem like basic concepts and maybe you kind of remember these words. However, we find they often get forgotten as many of us focus more on learning how to implement models in python and R vs basic statistics. Although both of these concepts do not necessarily rely on each other. You can start to assume you understand what the p-value means when you run models in either language without fully grasping the importance of it.

Statistics Courses

1. Khan Academy
This is why we recommend at least going back and walking through the Khan Academy statistics section. They cover concepts like Hypothesis testing, T-statistics vs Z-Statistics, Confidence Intervals, etc.

Khan is always a good place to start because the videos are a great combination of visual and audio examples.

Personally, there aren’t too many books we like when it comes to pure statistics. In the R programming section of this resource list, we will reference our favorite R + Statistics book.

2. Duke University On Coursera
For a full course that are free you can try Duke Universities Statistics Course. This is actually several courses that cover multiple types of statistics like classical vs Bayesian. These are two different methods that are worth looking into.
​
Python Videos, Books And Courses

Python is an interesting topic. The thing about python is there are so many plausible sub-sections of the programming language. For instance, when we prepare for interviews we always like to clarify which type of python questions we will be dealing with. Will we be asked questions that focus on concepts that are operational, analytical, optimization based, algorithms and data structures or possibly data science algorithms. All of these are different topics that have different styles of interview questions. Getting a question on how to traverse a binary tree is very different from having to implement a decision tree algorithm.

As a data scientist, typically you will benefit from the analytical and operational aspects of python. The operational portion will provide you with the ability to automate the boring stuff (<-3. as the cliche book is titled).

This book is great for really…any data focused person. Data scientists, business analysts, Business Intelligence Engineers, and Database developers all can benefit from automation. Now, you don’t need to use python, if you’re in a windows environment there is PowerShell Linux has to bash. Learning some form of scripting language helps improve your workflow and design thinking.​
​

​OUR FAVORITE PYTHON BOOKS, COURSES AND YOUTUBE VIDEOS

Python is a common language that is used by both data engineers and data scientists. This is because it can automate the operational work that data engineers need to do and has the algorithms, analytics, and data visualization libraries required by data scientist.

In both rolls, the need to manage, automate and analyze data is made easier by only a few lines of code. So much so that one of the books we have read and seen in many data focused practitioners libraries in the book Automate The Boring Stuff With Python.
​
The book covers python basics and some simple automation tips. This is especially good for business analysts who work heavily in Excel.
​
There are also books by O’Reilly that are also a great overview of the basics.

Read More Here

​Are You Interested In Learning About Data Science Or Tech?
Learning Data Science: Our Favorite Data Science Books
What Is Data Science Really As Told By An Ex-FAANG Data Scientist
How Algorithms Can Become Unethical and Biased
How To Load Multiple Files With SQL
How To Develop Robust Algorithms
Dynamically Bulk Inserting CSV Data Into A SQL Server
4 Must Have Skills For Data Scientists
SQL Best Practices — Designing An ETL Video

​

As a data engineer, data scientist or just a general data team, how do you go from high level goal to a final data product? What is the process taken to successfully create a product that provides insights and action?

Just to clarify a data product can be a model, a dashboard, a web app/API or even a simple excel output. But it all has to start somewhere.

Note: In this series we will be walking through creating a data product. In this case we will be basing it on a company that has both online and in person sales. We will be creating videos, posts, designs, like an actual project. Going through the SQL, python, and other possible steps.
If you are interested in keeping up with all our posts, then please sign up here.

Intro:

Let’s say we are a company like Ikea, Bernhardt or Dorel. A furniture company in the modern era would probably have an e-commerce website and in-person locations that sells furniture.

A top level director or VP at this company will decide they want to increase sales of product category X or Y, or they want to improve sales in a specific region, or perhaps improve profit margins, reduce costs etc. This is where the ball starts rolling for metric and KPI development.

High Level Goals

The problem with metrics is you can track everything. You can track down to very specific granularities. But this doesn’t always provide value.

In order to provide value you need to first define what is important to track. In order to know what to track you need to have a high level goal of what you are trying to improve. That is why the first step in the process is deciding what metrics will help track and support the business goals.

In order to track and create these metrics the data team will need a general understanding of what the business team is trying to do.

Nothing needs to be solidified, but it is good for the stakeholders to know what is important and what strategies they are looking to put into place. In addition, having some general goals like “we want to increase the average sale per person by 5%” is good because it makes it easier to foresee what possible strategies might work (but it also makes it easier to point out when the strategy failed).
So step one is making sure these goals are written out somewhere as a proposal. It doesn’t have to be more than a page. It just clearly states the background of the goal, the why, etc. This will help the data team decipher what the business team is looking for.
Why:

We pointed this out earlier, but the reason it is important for the business team to have a clear idea of what they are trying to do is because there are so many metrics that could be tracked.

This could cause a few issues. For instance, a good data team won’t start work until you have provided more context. However, a less experienced team might try to do the work and struggle to create anything of value. Instead, they might spend 6 months creating a product that has too many metrics, the wrong metrics or maybe they just never finish because it is really hard to make forward progress when you don’t know where you are going.

This is why it is important in step one for the business team to have a clear ask. Just to show some of the metrics that could be tracked we have listed them out below. They all could provide value, but it depends on what the overall goal is.

Metrics:
There are a lot of possible metrics out there. All strange abbreviations, with different rules and all that somehow track very similar things from different angles.

Let’s go over a few that the furniture store might be interested in using.

Average revenue per user (ARPU) — This would be a pretty standard metric. However, if in this case we are adding in the idea of some form of ad-campaign then there is another angle we might want to consider. That is when the person was exposed to the ad. We will discuss how to analyze this in a later post
​
Average Purchases Per Week,Month,Year Per Store — This metric can be tricky. This can be very dependent on how much your average product costs, and the types of up-sell opportunities there are. For instance, a car is very expensive, very few people are going to buy two at once. However, people may be interested in spending $1000 dollars for a dvd player to be included or a specific set of rims, etc. For instance on Amazon you are buying products that are usually within that 20 -200 dollar range. That is easy to bundle an extra book or two, or a pan when you buy a spatula, or something like that.

Customer Acquisition Cost (CAC) — This metric is used to calculate the cost to acquire new customers. It is important because one goal a company might have is to reduce the overall CAC. Depending on how this approached it can tell if ads are getting more effective or possibly if products are getting better. It would depend on what the company did prior to shift in CAC.

Number Of New Visitors e.e. Acquisition — This would be a whole number and not a difficult metric to calculate.

Percent Of New Users Compared To Base — Compared to the number of new visitor metric, viewing things as a percentage can tell you more fairly how your growth is going.

These are just a few examples of metrics and we will continue to walk through this process.

Up Next:

In our next article we will start to discuss taking data from an operational database and moving it into a data warehouse. This is a key step into analyzing data because it makes the data available for data engineers and analysts. We will discuss ETLs, data models, etc.

Please scroll to the bottom if you would like to sign up for our future articles and videos where we will continue the process of developing a data product.

Are You Interested In Learning About Data Science Or Tech?
25 Of The Best Data Science Courses Online
Learning Data Science: Our Favorite Data Science Books
What Is Data Science Really As Told By An Ex-FAANG Data Scientist
How Algorithms Can Become Unethical and Biased
How To Load Multiple Files With SQL
How To Develop Robust Algorithms
Dynamically Bulk Inserting CSV Data Into A SQL Server
4 Must Have Skills For Data Scientists
SQL Best Practices — Designing An ETL Video

DeepMind and Google: the battle to control artificial intelligence
By Hal Hodson
 
One afternoon in August 2010, in a conference hall perched on the edge of San Francisco Bay, a 34-year-old Londoner called Demis Hassabis took to the stage. Walking to the podium with the deliberate gait of a man trying to control his nerves, he pursed his lips into a brief smile and began to speak: “So today I’m going to be talking about different approaches to building…” He stalled, as though just realizing  that he was stating his momentous ambition out loud. And then he said it: “AGI”.
 
Read More Here

Learning Data Science: Our Favorite Resources From Free To Not

Today we wanted to cover some of our favorite resources for data science. As the title suggests, these resources will be from free to not. Some people like buying books and other people prefer online courses. So we have created this list of data resources that range from books to courses, from free to not.

Data science has many facets. Statistics, data cleansing, programming, system design and really…almost anything else data related depending on how large the company is.
This post will discuss our favorite resources for these topics. Now, most of these courses and books are primers for topics like statistics, Python and data science in general. They really will only provide the base knowledge. At the end of the day, real practical experience is one for the few things that will really train your data science knowledge. You should learn as much as you can from these resources and then apply for as many internships and entry-level positions as possible and study for interviews.
Read More Here

Object Detection with 10 lines of code
By Moses Olafenwa

One of the important fields of Artificial Intelligence is Computer Vision. Computer Vision is the science of computers and software systems that can recognize and understand images and scenes. Computer Vision is also composed of various aspects such as image recognition, object detection, image generation, image super-resolution and more. Object detection is probably the most profound aspect of computer vision due the number practical use cases. In this tutorial, I will briefly introduce the concept of modern object detection, challenges faced by software developers, the solution my team has provided as well as code tutorials to perform high performance object detection.

Read More Here

Loading files into databases are not exactly exciting work. It can be tedious and boring work. Often times it is constrained by tools like SSIS that freak out when a small change is made.
This is one of the core roles of BI and data engineering teams. They need to load files from operational databases into data warehouses. They do this by developing pipelines that process and transform data.
​
One problem we see is teams have multiple file types, data spec versions, and data providers. Each of which requires a different pipeline just to load into their raw database.

This forces teams to manage 10, 20 maybe even 50 smaller pipelines that are all subject to the change of the data providers. In addition, each new data provider requires an entirely new pipeline to be developed. It also requires managing infrastructure around each pipeline in order to log and track data pipelines statuses. This can quickly become very difficult to manage and tedious to build.

Another solution instead of attempting to manage multiple pipelines is to develop a central system that manages the various insert statements required to insert data into each different raw table. Kind of how we explain in the video below.

What the system does is essentially act as a lookup system that matches flat files to their insert queries. This reduces the need to redundantly have separate pipelines that would also all need an insert statement.

Instead, now the system uses one main system to load all the various tables. Theoretically, you can also put version numbers on each of the files that come in. This could be set up by tracking each version in the metadata table that will then allow you to tie a raw file to an insert file version.

Loading multiple file types by individual pipelines requires a lot of tedious development. Whether it be through SSIS or another system, the constant development quickly weighs heavily on data engineers. Instead of spending time developing analytical pipelines that drive clear value for your company. They will be stuck maintaining and developing operational pipelines.

This reduces its impact and reduces its overall value.

Thank you so much for reading! We hope you enjoyed the video. Please feel free to send us any questions. We would love to help you design your next automation system.
Also, if you liked this, feel free to check out our video on using bulk insert!

For further reading and videos on data science, SQL and Python:
How To Develop Robust Algorithms
Dynamically Bulk Inserting CSV Data Into A SQL Server
 SQL Best Practices - Designing An ETL - Part 1
How Algorithms Can Become Unethical and Biased
4 Must Have Skills For Data Scientists
What is A Decision Tree

By Ben Rogojan
We recently did an AMA on Reddit. The most common question that came up was what is the difference between a data scientist and a data engineer. So we wanted to make a more in-depth post on the subject.

There are a lot of data specialist positions that sound similar and use similar tools so it can be difficult to know what the role of each role should do. In addition, smaller companies might be limited on how many data engineers or data scientists they can hire. This means that many times the specific tasks and goals might start to intermingle.
​

This can make it much more difficult to clearly differentiate the two roles. So we wanted to go over how the two positions were different by discussing the different goals, mindsets, tools and backgrounds data engineers and data scientists have.


Before we go into the difference we would like to make a quick preface. The truth is, many data scientists and data engineers will perform the tasks of other technical roles. A data scientists may need to develop an ETL and a data engineer might need to develop an API and front-end. So the distinctions we are pointing out below are just to make clear where the technical differences are.

GoalsThe goals of a data engineer are much more task and development focused. Data engineers build automated systems and model data structures to allow data to be efficiently processed. This means the goal of a data engineer is to create and develop tables and data pipelines to support analytical dashboards and other data customers (like data scientists, analysts, and other engineers). It’s similar to most engineers. There is a lot of designing, assumptions, limitations, and development that occurs in order to be able to perform a final task. Each design and solution having its own set of limitations, even if it all can perform the end task.
​
In comparison, data scientists tend to be question focused. In the sense that they are looking for ways to reduce costs/increase profits, improve customer experience or business efficiencies. This means they need to ask and then answer questions ( ask a question, hypothesize and then conclude). So they need to ask questions like, what impacts patient readmission, would a customer spend more if shown an add like A vs. B, is there a faster route to deliver packages? Skipping over the rest of the process. The goal from here is to find an answer to whatever question is posed. It might be a final conclusion or to more questions. Throughout the process, data scientists analyze, gather support and can develop a conclusion to the question.

Tools

This is where things can get confusing. Data scientist and data engineers both often rely on python and SQL. However, how the two tech roles use these skills varies. Again, this ties back to the mindset differences. Python is a very robust language that has libraries that help manage operational tasks as well as analytical ones.

Data scientists will use languages like Pandas and Scikit Learn whereas data engineers will use python to manage pipelines. This is where libraries like Airflow and Luigi come in handy.
Similarly, data scientist queries will be ad-hoc focused (e.g. questions focused). Whereas data engineers queries will be focused on cleaning up and transforming data.

Now there are other tools both data specialists might use. This includes Tableau, Jupyter notebooks and a handful of others. At the end of the day, it is how they use them that makes the distinction.
BackgroundsNow, one other common question when it comes to the differences of data engineers and data scientists is what background is required.

Data engineering and data science both require some understanding of data and programming. Even if it is a limited scope. However, there are some distinctions that go beyond programming. Specifically for data scientists. Due to the fact that a data scientist is more like a researcher, having a background that is research-based is a benefit.

This might be in economics, psychology, epidemiology, etc. Combine a research background with SQL, Python and a good sense of business and you have a data scientist. Now, these are not set in stone degrees. In fact, we have run into a data scientist with various degrees. Most employers will prefer to hire a data scientist with at least a masters degree that has some sort of technical or mathematical focus.

Data engineering positions usually won’t require a masters degree. Data engineering is more about being a developer. This requires much more practical experience rather than theoretical knowledge. So gaining a Masters does not supply the same value.

An Example
Let’s say a director of a healthcare company decide they would like to figure out how to reduce the number of patients readmitted prior to 30-days from their original visit. From a data point of view, there are a couple of things that need to occur.

Data scientists will need to figure out what drives patient readmission. That is the question they will be trying to answer. Based on the conclusions they reach, they will work with the business to develop metrics and policies to help improve patient readmission rates.

Data engineers will be developing tables to help support the data scientists answer the question while at the same time developing analytical tables to help track past and future patient readmission metrics. How these metrics are created will be driven by the answers the data scientist get.

Data scientists and data engineers have plenty of differences. They have different goals, and backgrounds, but this is where the value of utilizing both together comes from. The fact that data engineers focus more on engineering robust systems allows data scientists to query data easily and analyze it efficiently. Their partnership is what brings companies value from data.

We hope this post was helpful! Please feel free to reach out with any questions you may have.

Read more about data science, healthcare analytics and automation below:
​How To Develop Robust Algorithms
Dynamically Bulk Inserting CSV Data Into A SQL Server SQL Best Practices - Designing An ETL - Part 1How Algorithms Can Become Unethical and Biased
4 Must Have Skills For Data Scientists
What is A Decision Tree