Photo by Davi Mendes on Unsplash
HADOOP 
Apache Hadoop is an open-source software that offers various utilities that facilitate the usage of a network on multiple computers to solve the problems on big data.

Hadoop also provides a software framework for distributed computing as well as distributed storage. For doing so it divides a document onto several stores and blocks across a cluster of machines. To achieve fault tolerance, Hadoop replicates these stores on the cluster. Nextly, it performs distributed processing by dividing a job into several smaller independent tasks.

​This task and then run in parallel over the cluster of computers as discussed earlier Hadoop works with distributed processing on large data sets across a cluster service to work on multiple machines simultaneously. to process any data on Hadoop, the client submits the program and data to this utility. The HDFS stores data while the MapReduce function processes the data and then YARN divides these tasks.

Let's discuss the working of Hadoop in detail:

1. HDFS: HDFS or Hadoop distributed file system is a master-slave topology that has two daemons running; DataNode and NameNode.
2. MapReduce: it is an algorithm that processes your big data in parallel on the distributed cluster. MapReduce can subsequently combine this data into results.
3. YARN: the function of yarn is to divide a task on resource management as well as job monitoring/scheduling into separate daemons. Yarn can skill beyond just a few thousand nodes. This is because the yarn federation allows a user to via multiple sab clusters into one big cluster. We can use many independent clusters together in one larger job which can be achieved with a larger-scale system.

HDFS 

As mentioned earlier, HDFS is a master-slave topology running on two daemons; DataNode and NameNode. Datanode runs on the slave nodes it stores the data in Hadoop. On the very starting Datanode connects to NameNode and keeps checking for requests to access data. NameNode, on the other hand, stores directory of files into the file system. It is also responsible for cracking down where the cluster of file data resides however it does not store the data contained in those.

With the Hadoop distributed file system you can write data once on the server and then subsequently read over there and also use it many times. HDFS is a great choice to deal with high volumes of data required in the current day. The reason is that HDFS works with the main node and multiple nodes on a commodity of hardware cluster. All the nodes are usually organized within the same physical rack on the data center so when the data is broken down into different blocks, they are distributed among different nodes for storage. The blocks are also replicated across nodes to reduce the chances of failure. 

HDFS uses validations and transaction logs to ensure data integrity across the cluster. Usually, there is only one NameNode and a possible data node running on the physical server while all other servers only run data nodes.

HBase 

Although Hadoop is great for working with big data sets it only performs batch processing and therefore data can only be accessed sequentially. The good thing is that Hadoop has some other applications such as Hbase which can randomly access huge amounts of data files according to the user’s requirement. This is extremely useful for gaining concrete insights which are exactly why 97.2% of organizations are investing in big data-related tools and software.

Hbase is an open-source column-oriented database that is built on top of the Hadoop file system. It is horizontally scalable. The data model of HBase is very similar to that of Google's big table design. it not only provides quick random access to great amounts of unstructured data but also leverage is equal fault tolerance as provided by HDFS. 

HBase is part of the Hadoop ecosystem that provides read and write access in real-time for data in the Hadoop file system. Many big companies use HBase for their day-to-day functions for the very same reason. Pinterest, for instance, works with 38 clusters of HBase to perform around 5 million operations every second!

what's even greater is the fact that HBase provides lower latency access to single rows from A million number of records. to work, HBase internally uses hash tables and then provides random access to indexed HDFS files.

HIVENow while Hadoop is very scalable reliable and the great for extracting data its learning curve is very steep to be cost-efficient and time-effective. Another great alternative to it is the Apache hive. Hive is a data warehouse software that allows users to quickly and easily write SQL-like queries to extract data from Hadoop.

The main purpose of this open-source framework is to process and store huge amounts of data. In the case of Hadoop, you can implement SQL queries using MapReduce Java API while in the case of Apache Hive you can easily bypass the Java and simply access data using the SQL like queries. 

The working of Apache Hive is very simple it translates the input program written in HiveQL into one or more Java a MapReduce, Spark, or Tez jobs. It then organizes the data into HDFS tables and runs the jobs on a cluster to produce results. Hive is a simple way of applying structure to large amounts of unstructured data and then performing SQL based queries on them. since it uses interface very familiar with JDBC (Java Database Connectivity), it can easily integrate with traditional data center technologies.

Some of the most important components of the Hive are: 

1. Metascore: This is the schema in which Hive tables are stored. The Hive Metastore is mainly used to hold all information regarding partitions and tables in the warehouse. It runs the same process as Hive service by default. 
2. SerDe: SerDe or Serializer/Deserializer is a function that gives instructions to the hive regarding how a record is to be processed

​

CONCLUSIONIn the above article, we discussed Hadoop, Hive, HBase, and HDFS. All these open-source tools and software are designed to process and store big data and derive useful insights.

Hadoop, on one hand, works with file storage and grid compute processing with sequential operations. Hive, on the other hand, provides an SQL-like interface based on Hadoop to bypass JAVA coding. HBase is a column-based distributed database system built like Google's Big Table - which is great for randomly accessing Hadoop files. Lastly, HDFS is a master-slave topology built on top of Hadoop to store files in a Hadoop environment. 
​
All these frameworks are based on big data technology since their main purpose is to store and process massive amounts of data. 

​If you would like to read more about data science, cloud computing and technology, check out the articles below!
Data Engineering 101: Writing Your First Pipeline
5 Great Libraries To Manage Big Data
What Are The Different Kinds Of Cloud Computing
4 Simple Python Ideas To Automate Your Workflow
4 Must Have Skills For Data Scientists
SQL Best Practices --- Designing An ETL Video
5 Great Libraries To Manage Big Data With Python
Joining Data in DynamoDB and S3 for Live Ad Hoc Analysis

Photo by Clint Adair on Unsplash

Top 6 Tips to Start Career as a Data Scientist

Now, if you are interested in solving new, challenging problems in routine, data science is definitely the field you can make a career in. There is a huge demand for data science market - as research suggests - which the United States leads, requiring over 190,000 data scientists in the year 2020! Here are some tips that you can follow:

1. Choosing a Role: 
Choosing the right role is very important in this field. You can be a machine learning expert, a data engineer, a data visualizer, or even a data engineer, etc. Depending on your work experience and study background, you can choose the role that suits you. For instance, if you have studied, software engineering, getting into data engineering wouldn't be difficult for you. In order to make the best choice, it is suggested that you talk to people who are already in the field.

2. Take Up a Course: 
Now since you have chosen a role, the next thing is to put dedicated efforts into learning the work you will be required to do in the certain role. The awesome thing about data science is, you can learn a great deal about it online. In fact, According to a recent study by IBM, the demand for data scientists is expected to increase by 28% this year! 
You can go for Udacity, IBM, Coursera, EdX, etc. and take certifications or courses to polish your skills. All these online universities bring coursework, assignments, quizzes, case studies, and comprehensively prepared study flow. So taking up two or three courses can bring a lot of value to your work.

3. Build your Portfolio:
Once you have completed your courses, the next logical step is to build your data science portfolio. It is important to bring a well-thought-out portfolio when you are working in this super-competitive field. So now, since you don't hold relevant industry experience, you can share personal data science projects which demonstrate your skillsets. You can also share the course projects created while completing your online certifications. and lastly, you can create some volunteer projects and showcase them in your portfolio.

4. Network with Relevant People: 
Now that you are ready to jump in the race, it is essential to network with industry peers and even get support/ advice from them. In order to do so, the best option is to attend events and meetups in the field. It is important because not only will your peers keep you motivated but also help you overcome hurdles. We do have some meetups in Dublin, under the name of Dublin Data Science. So you can search for the ones in your area, and jump right in.
5. Master Soft Skills: 
Now some of the skills that are required to establish a career in data science, are obvious like you need to be expert at coding in a certain language or have a sound understanding of how the technology works. But there are some lesser-known skills that you need to master in order to stand out of the crowd. Soft Skills such as creative thinking, time management, and innovation are important for working in this sector. 

Photo by Shahadat Rahman on Unsplash

You see, data scientists have to approach daily routine problems by fusing their creative thinking with logical concepts to build best-fit solutions. In fact, a recent research by RS News suggests that now employers use scales from 0-10 for each skill when you go for a job interview. Therefore, mastering soft skills like this are also very important to make you stand out.

6. Follow the Field Experts:
No matter at which point of learning/ expertise you are, it is important to engulf the right sources of knowledge. One of the most useful sources can be the informative blogs being run by data scientists. Interestingly, experts in this field are quite active online and therefore, they continue to update their followers with findings, and advancements in Data Science frequently. Some of the best data scientists to follow include:

1. Yoshua Bengio - He is the found of ApSTAT Technologies and also holds an experience of 22 years as Professor at University. He has also worked with MIT and AT&T as a machine learning researcher in the past.
2. Geoffrey Hinton - He is a P.hD holder in AI from Edinburg and the co-founder of the ‘Back Propagation’ concept used in deep learning simulations as well as algorithms employed for training neural nets. He also worked on artificial neural networks with Google AI team in 2013 
3. Peter Norvig - He is a data scientist and is currently employed as a research director at Google. In the past, he has also worked at NASA as head of Computational Science Divison. So you know, the level of authenticity is actually great.
4. Jurgen Schmidhuber - He is an AI specialist currently working on RNN and Machine Learning (currently being used by Baidu, Google, IBM, and Microsoft). So far, he has published about 333 peer-reviewed papers and earned Hemholtz Award (2013) as well as IEEE Neural Networks Pioneer Award (2016).

So Are You Ready To Start Your Data Science Journey?
In this era, the demand for data science is huge and employers are actually investing both time and money for data scientists. Therefore, it is important for you to take the right steps in order to enjoy exponential growth. Before you step in, have a look at the tips we have shared above, and follow them in order to start off without making any costly mistakes.

If you would like to learn more about data science or data in general, please feel free to read some of the posts below!

What Is Dask, And How Can It Help Data Scientists?
​Data Engineering 101: How To Develop Your First Data Pipeline
How Do Machine Learning Algorithms Learn Bias?
Personalization With Contextual Bandits
How To Survive Corporate Politics As A Data Scientist
What Is A Decision Tree

​​

Over the years, the applications of data science methods are producing limitless innovations for various industries. For the finance world this is nothing new. The usage of quants and algorithmic trading has been goin on for decades. However, now with increased computation and cheap storage the ability to automate and improve these models have drastically changed.
The finance industry has been benefiting from the integrating data science and statistics into their processes for decades.
 
Several use cases are presently helpful for enhancing the customer experience, improving models and helping fight the data imbalance problem. These use cases are set to shape the future, and give rise to new processes in time to come.

The success witnessed in several use cases comes from the ability to seek the right algorithms, assembling suitable datasets, and building well-organized infrastructure. Data science is creating significant inroads within the financial services industry. And the solutions offered are seeing more implementation with Artificial Intelligence, Machine Learning, and Reinforcement Learning.

With the rapid increase in computing power and ability to store larger amounts of data, financial companies have a lot of opportunities to both improve the customer experience as well as increase profits. Here are a few great examples of how companies are using data science in the finance industry today.

1. Process Automation — Improving Scale Of Information (Robo Advisors And Customer Chatbots)

Process automation might not seem like a data science topic.

However, for almost every predictive model, there is an automated process required to ensure that the data being ingested is constantly up to date. 

The new technologies developed are seeking to automate repetitive processes, change manual tasks, and enhance productivity. Consequently, data science is also enabling several brands to create better scale-up solutions, cost optimization processes, and customer experience. Below are some of the major process automation use cases of data science process in the finance world:

• Robo-Advisors
• Chatbots
• Call-Center Automation.
• Predictive model selection
• Paperwork automation, and more.

A few great examples of large corporations automating processes include JPMorgan Chase which unveiled its Contract Intelligence (COiN) platform. This platform makes use of natural language processing(NLP). 

The platform offers a solution to processing legal documents and extraction of vital data from them. A typical manual review would approximately take 360,000 labor hours for 12,000 annual commercial credit agreements. But, with the new Contract Intelligence (COiN) platform, the time to review a similar number of contracts would be only a few hours.

BNY Mello is also integrating process automation into its banking network. This system is responsible for several operational improvements and also responsible for $300,000 in annual savings.

Another example is Wells Fargo. It is using an AI-driven chatbot. The chatbot is set to communicate with users via the Facebook Messenger platform and offer assistance as regards to accounts and passwords. 

All of these systems will inevitably have some form of models that are amplified by process automation.

2. Improving Back And Forward Testing

In a live market, traders with a keen interest in trying new trading ideas often will develop models by backtesting outcomes to dictate if a system would be profitable. Backtesting denotes the application of a trading process to past data in verifying how it would perform during a specific time frame. Nowadays, backtesting in trading platforms offers:

• The ability to test ideas using few keystrokes to gaining insight into how effective a system can be without risking funds.
• The evaluation of simple ideas, for instance, the performance of moving average crossover on systems with varying inputs.

Backtesting remains a valuable option available in many third-party tools. The division of data into multiple sets (out-of-sample and in-sample testing) can allow traders with a practical and useful approach to evaluate a trading idea and system. 

As a result, most traders apply optimization techniques in backtesting in evaluating systems and integrate variables from user-defined input that permit system “tweaks.”

While backtesting offers valuable information, it may be misleading as it is just a part of the process of evaluation. 

Forward performance and out-of-sample testing offer more confirmation as regards to the effectiveness of a system and can give a real response to a system’s performance before the involvement of real money.

Forward performance offers traders a new range of out-of-sample information for system evaluation. It simulates actual trading and entails how such systems would act in an active marketplace. 

A key aspect to forward performance testing includes the exact following for the system’s logic, which helps to precisely evaluate the process.

A good correlation amongst backtesting, forward performance testing, and out-of-sample testing results is key to defining the practicability of a trading system. The continual use of forward performance testing and out-of-sample testing offers an extra safety layer before placing the system for market use. Positive outcomes alongside good correlation amongst out-of-sample and in-sample backtesting with forward performance testing enhance the prospect of a system performing well in actual trading.

3. Fighting The Imbalance Problem For Fraud Detection

The detection of financial fraud using data science is another major use case that is both currently being used and constantly improved.
 
Even with new methods of security such as adding a chip into cards and current predictive models that are attempting to reduce the fraudulent claims are far from perfect.

It is still predicted that online card fraud will rise to a tune of $32 billion in 2020. 

With data science, issues like fraud detection are typically considered as classification problems, which is the prediction of the class label with discrete output given a data observation. 

To solve such problems with data science, it requires the creation of models with enough intelligence to accurately classify transactions as either fraudulent or legit based on transaction details.

This unique solution using the data science approach also faces a significant challenge termed imbalanced data. 

In real-world finance, it usually arises from having data with a large part of its transaction classified as legit while only a small portion of the transaction are fraudulent. This is a similar problem with most fraud including insurance and healthcare.

For imbalanced data, the bottom-line is the prediction might be 99% accurate but the 1% of inaccuracy would be the fraud claims being labeled non-fraud. 

Investment in technology for tackling fraud has been on the rise over the years. Nevertheless, since imbalanced data offers a unique instance with most variables adding no context, it is always starting with some EDA (Exploratory Data Analysis) before using any prediction models. This allows the data scientists to see trends in the fraudulent claims themselves. Often times you can take a large data set of valid and invalid transactions and find traits that are unique to a subset of said transactions.

From there you can build a model on that subset of data to provide improved balance.

In addition, after EDA, there are several approaches to dealing with imbalanced data. The three popular approaches that stand out include combined class methods, oversampling, and undersampling.
​
In Conclusion

The benefits of data science in the finance world are seeking an enhanced customer experience, as well as applications for the future to offer more optimized systems and processes. By 2020,

Gartner forecasts a massive 84% of customer dealings with any enterprise would happen without another human interaction. With data science set to drive this revolution, the finance world can leverage more its applications to deliver practical solutions to proper use of data, security threats, and more. The future of the finance world is set to witness even more significant growth.

If you enjoyed this post, consider reading some of these posts!
Healthcare Fraud Detection With Python 
5 Uses Cases For DynamoDB
Learning Data Science And Machine Learning With Youtube And Coursera
Hadoop Vs Relational Database
How Algorithms Can Become Unethical and Biased
Top 10 Business Intelligence (BI) Implementation Tips​
5 Great Big Data Tools For The Future – From Hadoop To Cassandra

Photo by Tabea Damm on Unsplash

Creating an effective data strategy is not as simple as hiring a few data scientists and data engineers and purchasing a tableau license. Nor is it just about using data to make decisions.

Creating an effective data strategy is about creating an ecosystem where getting to the right data, metrics and resources is easy. It’s about developing a culture that learns to question data, and look at a business problem from multiple angles before making the final conclusion.

Our data consulting team has worked with companies from billion dollar tech companies, to healthcare and just about every type of company in between. We have seen the good, the bad and the ugly of data being utilized for strategy. We wanted to share some of the simple changes that can help improve your companies approach to data.

Find A Balance Between Centralized And Decentralized Practices

Standards and over-centralization inevitably slow teams down. Making small changes to tables, databases and schemas might be forced to go through some overly complex process that keep teams from being productive.
​
On the other hand, centralization can make it easier to implement new changes in strategy without having to go to each team and then force them to take on a new process.

In our opinion, one of the largest advantages companies can gain is developing tools and strategies that help find a happy medium between centralized and decentralized. This usually involves creating standards to simplify development decisions while improving the ability to manage common tasks that every data team needs to perform like documentation and data visualization. While at the same time decentralizing decisions that are often department and domain specific.

Here are some examples where there are opportunities to provide standardized tools and processes for unstandardized topics.

​Creating UDFs and Libraries For Similar Metrics
​After working in several industries including healthcare, banking and marketing one thing you realize is that many teams are using the same metrics.

This could be across industries or at the very least across internal teams. The problem is every team will inevitably create different methods for calculating the exact same number.

This can lead to duplicate work, code and executives making conflicting decisions because of top-line metrics that vary.

Instead of relying on each team to be responsible for creating a process to calculate the various metrics you could create centralized libraries that uses the same fields to calculate the correct metrics. This standardizes the process while still providing enough flexibility for end-users to develop their reports based off their specific needs.

This only works if the metrics are used consistently. For example in the healthcare industry metrics such as per patient per month costs (PMPM), readmission rates, or bed turn over rates are used consistently. These sometimes are calculated by EMR like EPIC, but might still be calculated by analysts again for more specific cases. It also might be calculated by external consultants.

Creating functions or libraries that do this work easily can help improve consistency and save time. Instead having each team develop their own method you can simply provide a framework that makes it easy to implement the same metrics.

Automate Mundane But Necessary Tasks

Creating an effective data strategy is about making the usage and management of data easy.
A part of this process requires taking mundane tasks that all data teams need to do and automating them.

An example of this is creating documentation. Documentation is an important factor in helping analysts understand the tables and processes they are working with. Having good documentation allows for analysts to perform better analysis. However, documentation is often put off until the last minute or never done at all.

Instead of forcing engineers to document every new table, a great idea is creating a system that automatically scrapes the available databases on a regular interval and keeps track of what tables exist, who created them, what columns they have, and if they have relationships to other tables.
​
This would be a project for the devops team to take on, or you could also look into a third party system such as dbForge documentation for SQL Server. Now this doesn’t cover everything, and this tool in particular only works for SQL Server. But a similar tool can help simplify a lot of peoples lives.
Teams will still need to describe what the table and columns are. But, the initial work of actually going through and setting up the basic information can all be automatically tracked. 
​
This can help reduce necessary but repetitive work that can help make everyones life a little easier.

Provide Easier Methods To Share And Track Analysis

This is very specifically geared towards data scientist.

Data scientists will often do their work in Jupyter notebooks and Excel that they only have access to. In addition, many companies don’t enforce the need to use some form of repository like git so that data scientists can version control their work.

This limits the ability to share files as well as keep track of changes that can occur in one’s analysis over time. 

In this situation, collaboration becomes difficult because co-workers are often stuck passing files back and forth and self version controlling. Typically that looks like files with suffixes like _20190101_final, _20190101_finalfile…

For those of you who don’t get it, you hopefully never will have to.

On top of this, since many of these python scripts utilize multiple libraries it can be a pain to ensure that as you pip install all the correct versions onto your environment.
All of these small difficulties can honestly can cause the loss of a day or two due to trouble shoot depending on how complex the analysis is that you are trying to run.
However, there are plenty of solutions!

There are actually a lot of great tools out there that can help your data science teams collaborate. This includes companies like Domino Data Lab. 

Now, you can always use git and virtual environments as well, but this also demands that your data scientist be very proficient with said technologies. This is not always the case.

Again, this allows your teams to work independently but also share their work easily. 

Data Cultural Shift

Adding in new libraries and tools is not the only change that needs to happen when you are trying to create a company that is more data driven. A more important and much more difficult shift is cultural. 

Changing how people look and treat data is a key aspect that is very challenging. Here are a couple of reasons why.

Data Lies

For those who haven’t read the book, How To Lie With Statistics, spoiler alert, it is really easy to make numbers to tell the story you want.

There are a lot of ways you can do this.

A team can cherry pick the statistics they want to help their agenda triumph. Or perhaps a research team ignores confounding factors and reports on some statistic that seems to be shocking if you don’t consider all the other variables.

Being data driven as a company means that you need to develop a culture that attempts to look at statistics and metrics and ensures there isn’t anything interfering with the number. This is far from easy. When it comes to data science and analytics.
Most metrics and statistics often have some stipulations that could negate whatever message they are trying to say. That is why creating a culture that looks at a metric and asks why is part of the process. If it were as simple as just getting outputs and p-values. Then data scientists would be out of a job because there are plenty of third-party companies that have products that find the best algorithm and do feature selection for you.

But that is not the only job of a data scientist. They are there to question every p-value and really dig into the why of the number they are seeing.

Data Is Still Messy

Truth be told, data is still very messy. Even with todays modern ERPs and applications, data is messy and sometimes bad data gets through that can mislead managers an analysts.

This can be due to a lot of reasons. How the applications manage data, how system admins of those applications modified said system, etc. Even changes that seem insignificant from a business process side can majorly impact how data is stored.

In turn, when data engineers are pulling data they might not accurately be representing data because of bad assumptions and limited knowledge. 

This is why just having numbers is not good enough. Teams also need to have a good sense of the business and the process that create said data to ensure they don’t allow data that is messy into the tables which analysts use directly. 

Our perspective is that data analysts need confidence that the data they are looking at correctly represents their corresponding businesses processes. If analysts have to remove any data, or consistently perform joins and where clauses to accurately represent the business, then the data is not “self-service”. This is why, whenever data engineers create new data models, they need to work closely with the business to make sure the correct business logic is collected and represented in the base layer of tables.
​
That way, analysts can have near 100% trust in their data.

Conclusion

At the end of the day, creating an effective data culture requires a both top down and bottom up shift in thinking. From the executive level, decisions need to be made in what key areas they can help make access to data easier. Then teams can start working at becoming more proficient at actually using data to make decisions. We often find most teams spend too much time working on data tasks that need to get done but could be automated. Improving your companies approach to data can provide a large competitive advantage and allow your analysts and data scientists the ability to work on projects they both enjoy and help your bottom line!

If you team needs data consulting help feel free to contact us! If you would like to read more posts about data science and data engineering, Check out the links below!

Using Python to Scrape the Meet-Up API
The Advantages Healthcare Providers Have In Healthcare Analytics
142 Resources for Mastering Coding Interviews
Learning Data Science: Our Top 25 Data Science Courses
The Best And Only Python Tutorial You Will Ever Need To Watch
Dynamically Bulk Inserting CSV Data Into A SQL Server
4 Must Have Skills For Data Scientists
What Is A Data Scientist

We recently posted some ideas for projects you could take on, to add to your resume and help you learn more about programming.
​
One of those projects involved scraping the Meet-up and Eventbrite APIs to create an aggregate site of events.

This is a great project and it opens up the opportunity to take on several concepts. You could use this idea to make an alerting system — the user inputs their API keys to track local events they have an interest in. You could develop a site to predict which live acts will become highly popular, before they get there, by tracking metrics over time.

Honestly, the APIs give a decent amount of data, even to the point of giving you member names (and supposedly emails too, if the member is authenticated). It’s a lot of fun — you can use this data for the basis of your own site!

To StartTo start this project, break down the basic pieces you will need to build the backend. More than likely you will need:

• An API “Scraper”
• Database interface
• Operational database
• Data-warehouse (optional)
• ORM

To start out you will need to develop a scraper class. This class should be agnostic of the specific API call you’re making. That way, you can avoid having to make a specific class or script for each call. In addition, when the API changes, you won’t have to spend as much time going through every script to update every variable.

Instead, you’ll only need to go through and update the configurations.
That being said, we don’t recommend trying to develop a perfectly abstract class right away. Trying to build a perfectly abstracted class that has no hard-coded variables, from the beginning, can be difficult. If anything goes wrong or doesn’t work then it is harder to debug because of the layers of abstraction.

We’ll start by trying to develop pieces that work.

The first decision you need to make is where the scraper will be putting the data. We’re creating a folder structure in which each day has its own folder.

You could use a general folder on a server, S3, or a similar raw file structure. These offer the ability to easily store the raw data that we’re storing in a JSON file. Other data storage methods, like csv and tsv, are thrown off by the way the description data is formatted.

Let’s took a look at the basic script. Think about how you could start better configuring and refactoring the codebase to be better developed.

One place right off the bat is the API key. While you’re testing it’s easy to hard-code your own API key. But if your eventual goal is to allow multiple users to gain access to this data then you will want their API keys set up.

The next portion you will want to update is the hardcoded references to data you are pulling. This hard-coding limits the code to only work with one API call. One example of this is how we pull the different endpoints and reference what fields you would like to pull from what is returned.

For this example, we are just dumping everything in JSON. Perhaps you want to be very choosy — in that case, you might want to configure what columns are attached to each field.
​
For example:

This allows you to create a scraper that is agnostic of which API event you will be using. It puts the settings outside the code, which can be easier to maintain.

For example, what happens if Meet-up changes the API endpoints or column names? Well, instead of having to go into 10 different code files you can just change the config file.

The next stage is creating a database and ETL, to load and store all the data, and a system that automatically parses the data from the JSON files into an operational style database. This database can be used to help track events that you might be interested in. In addition, creating a data warehouse could help track metrics.

Perhaps you’re interested in the rate at which events have people RSVP, or how quickly events get sold out.

Based on that you could analyze what types of descriptions or groups that quickly run out of slots.
Personally, there is a lot of fun analysis you could take on.

Over the next few weeks and months, we’ll be working to continue developing this project. This includes building a database, maybe doing some analysis, and more!

We hope you enjoyed this piece!

If you enjoyed this video about software engineering then consider these videos as well!
The Advantages Healthcare Providers Have In Healthcare Analytics
142 Resources for Mastering Coding Interviews
Learning Data Science: Our Top 25 Data Science Courses
The Best And Only Python Tutorial You Will Ever Need To Watch
Dynamically Bulk Inserting CSV Data Into A SQL Server
4 Must Have Skills For Data Scientists
What Is A Data Scientist
​

Big data has moved from just being a buzzword to a necessity that executives need to figure out how to wrangle. 

Today the adoption of big data technologies and tools have witnessed significant growth with over 40% of organizations implementing big data as forecasted by Forrester, while IDC predicts that the big data and business analytics market is set to hit an all-time high of $274.3 billion in 2022 from $189.1 billion it’s expected to reach this year.

With this push for big data and big data analytics, finding the write system, best practices and data models that allow analysts and engineers access to the treasure troves of data can be difficult. Do you use traditional databases, columnar databases, or some other data storage system?
​
Let’s start with the this discussion by comparing a traditional relational database to Hadoop(specifically Hadoop partnered with a layer like Presto or Hive). 

​

​What is Apache Hadoop?
Hadoop is a distributed file system with an open-source infrastructure that allows for the distributing and processing of Big data sets.

Hadoop is designed to scale up from single servers to lots of machines, offering local storage and computation to each server. Apache Hadoop comes with a distributed file system and other components like Mapreduce (framework for parallel computation using a key-value pair), Yarn and Hadoop common (Java Libraries).

Presto 
Presto is a distributed SQL query engine that can be used to sit on top of data systems like HDFS, Hadoop, Cassandra, and even traditional relational databases. It allows analysts the ability to use the benefits of Hadoop with out having to understand the complexities and intricacies of what is going on underneath the hood. This allows engineers the ability to use abstractions such as tables to organize data in a more traditional data warehouse format.

What is Relational Database (DB)?
Relational DB is formed from a set of described tables from which data can be reassembled or assessed in various ways without needing to reorganize the entire database tables. I know this kind of sounds weird, but in its simplest form, RDB is the basics for all SQL as well as all database management systems like Microsoft SQL Server, Oracle and MySQL.
RDB can also be called RDBMS, which stands for Relational database management system. RDB is a database management system that works with a relational model. RDBMS is the evolution of all databases; it’s more like any typical database rather than a significant ban.

The Differences..

Data architecture and volume

Unlike RDBMS, Hadoop is not a database, but rather a distributed file system that can store and process a massive amount of data clusters across computers. However, RDBMS is a structured database approach in which data is stored in rows and columns which can be updated with SQL and presented in different tables. This structured approach of RDB limits its capability to store and process a large amount of data. So Hadoop, with Mapreduce or Spark can handle large volumes of data.

Data Variety
Data variety is typically referred to as the type of data processed. For now, we have three main types of data types; Structured, unstructured, and semi-structured. Relational DB can only manage and process structured and semi-structured data in a limited volume. RDB is limited in managing unstructured data. However, Hadoop leverages its ability to manage and process all of the above data types; structured, unstructured, and semi-structured data. As a matter of fact, Hadoop is now the fastest known method for managing and processing huge volumes of unstructured data

Datawarehouses And Hadoop
As stated earlier. Hadoop on it’s own isn’t a database. However, thanks to open source projects like Hive and Presto you can abstract the file system into a table like format that is accesible with SQL. 
This has allowed many companies to start switching over parts or all of their datawarehouses to Hadoop. 

Why?

It is for accessibility, and hopes for performance on cheaper machines. Whether or not this is actually working depends from company to company and data management team to data management team. 

Because although systems like hadoop promise better performance. There are a lot of downsides that don’t always get discussed.

​Weakness in RDBMS and Hadoop
“Before we get started, it’s going to seem as if I dislike Hadoop. This is not the case, I am just going to be pointing out some of the largest pitfalls and weaknesses.

Technical Abilities
We will get into the technical difficulties shortly. But before we cover the technical cons, we wanted to discuss the talent issue.

Both Hadoop and traditional relational databases require technical know how. Now, this might be up for debate, but generally speaking, most relational databases are arguably easier to use.
This is because there are so few moving pieces in comparison. With Hadoop you need to think about managing cluster, the Hadoop nodes, security, Presto or whatever interface you are using and really several other technically administrative tasks that take up lots of time and skill.

In comparison, most relation database systems like SQL Server or Oracle are “some what” more straightforward. Security is built in, performance tuning is built in and in most importantly, there is a larger talent pool of people who understand how to manage and use standard DBs.

Why do you think interfaces like Presto and Hive that are both very SQL like exist? It’s because data professionals needed a way to interface with Hadoop that was much more familiar.

So the biggest issue most companies face is not the complexity of Hadoop but the lack/cost of talent that can operate Hadoop correctly. 

Security issues
Unlike RDBMS, Hadoop faces a lot of security problems which can be challenging when managing complex applications. In fact, the original Hadoop releases had no authentication system set up under the assumption that the system would be running in a safe environment. 

More recent releases do have access and permissions, authentication and encryption modules. They aren’t that straight forward to use and usually require a decent amount of ramp up. This can make it difficult to support and scale if you are just using the Hadoop out of the box without any form of third-party like Hortonworks($$$). 

Functional Issues
Hadoop is designed with the concept of write once read many. Hadoop is not designed for write once and update many. So for data specialists who are used to have the ability to update, forget about it.

For those who aren’t into data modeling, the issue this causes might not be apparent right away. Nor is it all that exhilarating to understand…
But not being able to run an update statement limits a lot of modeling that can be beneficial from the perspective of data volume.

For example (we are about to go granular). Let’s say you want to track someone’s promotions in a company. Traditionally, in a RDMS you can simply track the employee_id, position and start and end date of said position. You don’t need to track all the days in between. When a position is switched you can update the end date, add a new row for that employee with a new position and the start date, leaving the end date null. It would look like the below.

This took two rows of data and we now have all the information required.

In comparison, there are a couple of ways you could store this data using Presto to save similar information

One method is to store a person’s position every day in a date partition. The downside here is you will essentially need a row for everyday the person was in said position. The problem here is you will be storing a massive amount of data. If you have 10,000 employees, that is 10,000 rows daily.

Another method would be to use a similar data model as the RDBMS. That is only have one row for every employee_id and position combination with a start and end date. However, using this method will only work if you have access to the previous days information. This is a limiting factor. 

In the end, you are probably storing much more data than is really necessary and or performing a lot of transactions that are unneeded. The point here is that although Hadoop can provide some advantages, it’s not always the best tool.

If you enjoyed this video about software engineering then consider these videos as well!

142 Resources for Mastering Coding Interviews
Learning Data Science: Our Top 25 Data Science Courses
The Best And Only Python Tutorial You Will Ever Need To Watch
Dynamically Bulk Inserting CSV Data Into A SQL Server
4 Must Have Skills For Data Scientists
What Is A Data Scientist
​Solving The Balanced Bracket Problem

One of the common questions we get when it comes to learning how to program, is: “What are some good ideas for projects to build?”

Now, we hear the common cliche answers often, like “build a chess game” or “command line interface”. There is nothing wrong with these answers.

However, we think these examples don’t match modern programming needs. A big portion of modern software is SaaS and web apps. This means you need to know how to program online.
There are a lot more complexities that go into programming a website, or app, that has users, requires servers, authentication, and databases. This forces you to interact with technologies you’ll never need when developing a command line tic-tac-toe game.

Some of this can also be managed by AWS and other third parties that are commonly used by large corporations. Again, exposing you to technologies that are useful and heavily used in industry. This will be far more practical on a resume, as well as help you learn how to use new technologies.

Tip: If you pick a complex project, then focus on building one feature at a time. Building an entire website or app all at once is difficult. Start by building a login page, or maybe the main landing page after the user logs in.If you try to take on the whole project at once then you will likely fail.

Entertainment

1. A web scraper that posts top 10 blogs without human intervention

One of the issues we find with some project recommendation posts, is thatthey recommend projects that aren’t implemented in a way that excites the programmer to continue development. For instance, I see that a lot of people recommend building a web scraper.
Once you’ve built that web scraper and scraped the data, what are you going to do with it?

Instead of just scraping the data, why not build a website with that data. It doesn’t have to be fancy or get a lot of views. This scraper could pull the data into a database and then select the most popular posts. From there it could copy the title, along with a few sentences, and then create a post that it shares online. This would be an impressive and simple project that you can actually show off.
You’ve now shown that you can do more than just code a small segment of a system. Instead, you can think through an entire system. You need to consider how you are going to automate the process, manage the database, create the website and select the posts. This also allows you to actually have a tangible end product.

Without a tangible end product, it’s really easy to become unmotivated and simply stop at only a web scraper.

In addition, you never know, maybe your site will become popular!

Skills: Database, web scraper, automation, web development (for the blog), and general programming.

2. An event-alert system using Meetup and Eventbrite APIs

Have you ever wanted to go to a band or comedian show, but realized it was last week? Maybe there was a free conference in your area on data science or big data and you missed out because you forgot to check.

Why not make your own aggregator using the Meetup and Eventbrite APIs, that will warn you when keywords are in event descriptions or titles? Now, I assume both Meetup and Eventbrite have similar options. But it is always fun to try to build your own system.

You can customize the system to work the way you want, and maybe even allow other people to make their own alerts by making this a website. What we enjoy about this project is that you can practice working with two different APIs. This will allow you to compare and contract what you like and dislike about them. That way, if you’re ever in charge of building an API, you’ll have a better picture of what works and what doesn’t.

Skills: APIs, database, automation, web development, and general programming.

3. A 9GAG copy cat

You don’t always need to try to reinvent the wheel when creating your own projects. Simple projects like a site that lets you login, post photos, GIFs, and lets you scroll through a feed, provides an opportunity to create a solid base site first. Then you can add lots of interesting features like following, liking, and search. Search in particular would be a great chance to learn how recommendation systems and machine learning work!

It’s always fun to try and replicate popular sites. In fact, it is actually a great way to learn because you have to reverse-engineer each feature. Reverse engineering is a great skill, because as a software engineer you will constantly be maintaining other people’s code and you will need to get in their heads.

Skills: Machine learning (for recommendation system), database, automation, web development, and general programming.

Retail Type Sites

4. A gift recommendation app

Have you ever struggled to find the right gift for your friend? What if you could create a website that helps to predict what to buy a friend for a gift. It could allow the end user to either create an account or just get a gift recommendation.

Again, this allows for the opportunity to create an account which requires authentication, database development, etc.

Also, another great part about this project is you can use Amazon’s API for affiliate links. This will allow you to do a few things. One, learn about how to use APIs and get you comfortable with reading API documentation. Two, if you do it well, you can get a commission for each product someone buys.

This project also has an opportunity to try to create a basic machine learning model. You can create a quiz of sorts that tries to figure out what the best gift is and then, based on if people click the gift or not, can drive the model to learn based on the response rate.

Skills: APIs, database, general programming, and app development.

5. A site for bartering and trading

Think OfferUp, but instead of money, why not create a website that only allows trades. This concept will force you to develop several features that need some thought. You won’t be able to just attack this project without a plan.

How will people post, where will people find recently posted items and how will people search. All of these are separate features you can build. In addition, you need to think how users will interact, and maybe even how they actually make the trade.

The idea doesn’t have to be 100% practical for real life — it needs to be practical in the sense of improving your skill set as a programmer.

Skills: Database, web development, general programming, and app development (if you choose to make it an app).

B2B

6. Invoice and contract management system

Contract and invoice management are very complex processes. Contracts can have a lot of nuanced clauses and stipulations that can be difficult to track.

This makes this a very good project, even if you simplify it to some of its core components. Having to translate a complex business process into software is not easy. But it is what makes this project a good challenge.

Again, we wouldn’t overcomplicate this. Take a basic feature, like inputting the terms of a contract, and develop this part first. Then you can add other features like invoice tracking, contract analytics and forecasting.

Skills: Process management, database, web development, and general programming.

7. Task management system

Task boards like KanbanFlow are built with several modular features that make it a great project to play around with. It will take a little work to get started, as you will need to set up a UI that is robust and dynamic as well. In fact, this project would be more of a two person job. One person to work on the front end and another person to work on the back end.

Don’t let that discourage you! This is actually a chance for you to work on your communication and team work skills. You will need to talk through designs to make sure you both fully understand it, and you know where your modules will be connecting.
This is always more challenging than it seems.

Skills: Communication, front end, database, web development, and general programming.

8. A job board

Any project that forces you to allow users to input as various types of users adds an interesting design aspect. How will you ensure that the way employers experience the site meets their needs vs. prospecting job searchers? Like most of the other projects, you don’t need to focus on all of it at once. Start out by trying to create the ability to create a job posting first. Then you can go and focus on the job searchers and how they respond.

Skills: Database, web development, and general programming.

9. A website that forecasts profits based on standardized data sets

There are a lot of data sets that are very standardized for most companies. This includes accounting data which is usually based on cost centers, accounts, line descriptions, and finally the actual transaction cost.
What is great about the standardization of any data set, is that it makes it easy to create analytics on top of said data sets. Why not create a standardized dashboard that can help companies predict spend, see monthly outgoings, and possibly help them improve their spending.
For this project you will probably have to spend a lot of time learning about how to make sure you keep your data secure. Of course, we recommend first trying to build the modules that focus on uptaking and standardizing the data and displaying it, before you go too deep into security. That’s a rabbit hole you may never escape!
Skills: Forecasting, business logic, database, web development, and general programming

Game Ideas

10. Snake

If you had a cellphone in the early 2000s, you’ve probably played Snake. It’s a simple game but you can always try to make things more complex! First, start by just trying to make the game.
This will require you to figure out how to develop a game online. This neon Snake by Sebastian Opperman is a great place to start. But after that, maybe you can add some cool new features like special items or special powers.

This would be a chance to play around and have fun. This project won’t be as technical from the stand point of having lots of users that sign up and use your site. However, it is a good challenge to figure out how to make a game run online.

Skills: Web development, general programming, and UI

We do hope this list inspires you to create an awesome new project that you can add to your resume and talk about in interviews. Maybe we’ll see you as the next CEO of a billion dollar startup!

Are You Interested In Learning About Data Science Or Tech?
10 Amazing Articles On Python Programming And Machine Learning
Learning Data Science: Our Top 25 Data Science Courses
The Best And Only Python Tutorial You Will Ever Need To Watch
Engineering Dashboards, Metrics And Algorithms Part 2
Read Last Weeks Top Ten Article For Python Libraries
The Best And Last Python Tutorial You Will Ever Watch
How Algorithms Can Become Unethical and Biased
Dynamically Bulk Inserting CSV Data Into A SQL Server
4 Must Have Skills For Data Scientists
SQL Best Practices — Designing An ETL Video

In the past few weeks, we have watched Tableau be purchased by Salesforce for 15.7 billion dollars and Looker for 2.7 billion dollars by Google.
​
Why are these companies so valuable?

Why are so many companies willing to pay for the services that Tableau and Looker provide?
The truth is dashboards can be expensive wastes of time and resources or they can provide insights that are worth exponentially more than the original investment. In fact, creating dashboards is a very lucrative business.

I worked for a company whose business model revolved around selling dashboards and insights to external customers. We only had four dashboard products. That was it, and yet, this company has existed for over a decade.

This is because they were really good at creating dashboards that provided impact continuously. These weren’t dashboards that you use for a month or two and forgot about. Instead, the dashboards they created were concise, clear and helped directors make decisions quickly.
They were able to do this by creating dashboards that followed a few basic principles. We wanted to share some of these basic tips below. Who knows, maybe your team will build the next million dollar dashboard.

Don’t Bury the Lede

When developing a dashboard, you need to first entice your audience with a high-level metric. This could be the total amount of money lost in insurance fraud for the year, a ranking that highlights how well a company is doing, or perhaps a trio of specific metrics.
The point of these high-level metrics is to pique interest. It needs to capture the audience and force them to ask questions and keep digging deeper into the dashboard.
Specifically, you want them to ask questions like why is the number so high or low and what are the factors driving this number?
Let’s walk through an example (see the dashboard below):

​

This dashboard is actually pretty well done. It is clean and provides very succinct information.

However, there is no initial draw. There is no distinct sign of whether the company’s income is bad or good.

So why should I be interested? I don’t really have any questions off the top of my head because I wasn’t primed to do so by any driving metric or dollar figure.

Having the dollar amount that depicts how far off target the net-income is in the top-left corner along with the percentage would have been a good design decision. This set of numbers would drive me to ask the right questions. Specifically, why is the net profit above or below what is expected?
This would then drive the end-reader to look into the charts below. Net profit exists on this dashboard, but it is hidden in the corner. Essentially burying the lead.
​
The most important information should be prioritized to the top of the dashboard.

​​Focus on Substance and Style

Creating a great dashboard is like working as a chef in fine dining.
If you focus too much on the aesthetics and not on flavor, your food will taste terrible. If you put no focus on aesthetics, your customers won’t see the value in your $50 salmon and carrots dish.
It can be tempting to create very flashy dashboards with all the bells and whistles. Similar to the way it was tempting in the early 2000s to throw foams, spheres, and gels on every dish in the fine dining world.

But bells and whistles don’t tell a story.

They can accentuate it, but they can also distract the end-user and make it more difficult to make a clear decision.

So focusing too much on style will hinder rather than help.

​On the flip side, presentation is important too. Presentation can entice an end-user to pay attention, even if you have bad data. For example, let’s look at this beautiful dish below.

Created by @chefjacqueslamerde on Instagram…wait…thats just celery, peanut butter, and raisins…
But, somehow, it looks like so much more. Even bad data can sometimes be forgiven by a beautiful dashboard (this is not encouraging bad data!).

We are trying to accentuate the fact that design plays a role in how we perceive information.

As data professionals, we can sometimes focus solely on the numbers and metrics. Numbers alone don’t capture peoples interest the same way it might capture ours. Capturing peoples attention is where design principals come into play.

Again, if we look at the net income dashboard, it is clean and has a very simple style. It is pretty clear to get what is going on in each chart. Often people assume that having lots of bells and whistles make a beautiful dashboard.

It is actually usually the opposite. Keeping a dashboard simple with only the data required makes the message clearer. If you have too much going on you muddy the message and could confuse the end-user.

It’s almost like a dashboard’s UX should be treated like a website’s UX.

Don’t Hide Too Much Behind Interactions

We have mentioned a lot about avoiding bells and whistles because they often deteriorate the message you are trying to tell with a dashboard.

One more specific feature most dashboards allow for are filters and tool-tips. These are great in some ways until you put too much information behind them.

Requiring too many interactions from an end-user to get to data could dilute the message you are trying to depict.

Rule of thumb: If you can’t print your dashboard and tell your same story, then you have too much information hidden behind tooltips and filters.

One example of this is providing one too many filters that might interact with varying widgets on your dashboard. This forces the end-user to need to flip through multiple states of each widget to get the message you are trying to tell them.

Sometimes our desire to over-inform only further hinders the message.Dashboards are about prioritization of information. You should only be showing the most important information that can help an executive make a clear decision.

Most directors will want lots of options to slice and dice the data, but that doesn’t mean that you should do it. Too many interactions may push them to continually analyze new configurations of filters vs. making a decision. This means you will have to push back for a better design.

Provide Context

When you create dashboards you can often create them in vacuums. Teams often create dashboards or metrics and don’t realize that other companies or teams might not know what is going on. Especially when you start to create custom metrics.
This is why we recommend providing context for the metrics, the dashboard purposes, and the axes.

The metrics
Regardless if you assume your metrics are well known or esoteric. Defining what the metric represents in a clear and concise manner benefits everyone. This also forces teams to use metrics that are easy enough to explain to an end-user. Some teams might create super complex models to calculate metrics, but this often abstracts the value a metric can have. Instead, you should focus on simple metrics that you can explain to anyone.

If you can’t understand a metric, you can’t make decisions.

The dashboard’s purpose
The first time an end-user experiences a dashboard, they might not know why they are looking at it. Putting together a few sentences of what a user is looking at can help better frame their understanding. In many ways, you should strive to not require a summary, but it’s always good to have. Similar to documentation with code.

Good code should require little documentation, but that doesn’t mean you shouldn’t document your code! Always document everything.

The axes
How you define the axes isn’t always straight-forward. You want just enough tick marks that end-users can understand what the line charts and bar charts mean, but you don’t want so many ticks that it is confusing. This comes down to good judgment. For instance, when you are using a sparkline, you will often limit the tick marks, but if it is a main line chart, you will probably provide more context along each axis.

The Devil is in the Details

High-level metrics usually provide the “what” but not the “why”.

The high-level metrics push users to look for answers.

What is driving the high-level numbers like net profits?

Thus, the next level of your dashboard should help answer these questions.

This means you need to look at the top metric and think about what questions an end-user might have looking at it. At this step in your dashboard development process you should try listing out what questions you would have as an end-user. It is helpful to have a member or two of your team who hasn’t seen the dashboard yet write out their questions. That way you have a fresh pair of eyes.
Based on these questions you can start to create secondary dashboards that provide the required insights.

Tableau has a great feature called actions. This can allow you to create easy drill-downs that go to more detail specific dashboards.

The goal here is to provide high-level metrics and low-level details that can help directors make concrete decisions.

Conclusion

These are a few great tips we have learned from working for a company whose sole purpose was creating dashboards to provide value to external customers. We hope it helps you create amazing dashboards that can create value for your company. Remember to focus on creating dashboards that don’t bury the lede and allow directors to make good decisions easily. That is how you make a million dollar dashboard.

To contrast, too many bells and whistles and poorly designed dashboards can lead to bad decisions or no decisions.

If you have any questions feel free to send them our way!


If you enjoyed this video about software engineering then consider these videos as well!
142 Resources for Mastering Coding Interviews
Learning Data Science: Our Top 25 Data Science Courses
The Best And Only Python Tutorial You Will Ever Need To Watch
Dynamically Bulk Inserting CSV Data Into A SQL Server
4 Must Have Skills For Data Scientists
What Is A Data Scientist

Software engineers spend a lot of time gaining skills for interviews by practicing leet code problems and perfecting resumes.
Once they finally get that job at a start-up, Google, Amazon or another corporation you might find the skills they used to get the job don’t tend to match the ones you need in your everyday work.
Our team was inspired by the 7 skills of highly effective programmers created by the TechLead and wanted to provide our own take on the topic.
Here are our 7 skills of effective programmers.

Read the rest here!

​