Learn. Do. Earn.

Month: May 2016

How Hadoop is Used in Organizations

Hadoop-in-organizations
“There were 5 Exabytes of information created between the dawn of civilization through 2003, but that much information is now created every 2 days.” These words were quoted by Ex Google CEO Eric Schmidt in 2010. We must understand the background of his statements. In the past decades the...
Read More
Leave a Comment

MapReduce Custom Partitioner

custom-mapreduce-parttition
In this post, we will be looking at how the custom partitioner in MapReduce Hadoop works. This post will give you a good idea of how a user can split reducer into multiple parts (sub-reducers) and store the particular group results in the split reducers via custom partitioner. Before...
Read More
Leave a Comment

Querying Hive Using Apache Drill

Querying-Hive-With-Apache-Drill
Apache Drill is an open source software framework which has been derived from Google’s Dremel System available as an infrastructure service called Google BigQuery. The specilaity of the Drill is to scale up to 10,000 servers or more and to be able to process petabytes of data and trillions...
Read More
2 Comments

Spark Use Case – Uber Data Analysis

Uber data analysis
In this post, we will be performing analysis on the Uber dataset  in Apache spark using Scala. The Uber dataset consists of 4 columns. They are dispatching_base_number, date, active_vehicles and trips. You can download the dataset from the below link: https://drive.google.com/open?id=0ByJLBTmJojjzS2c2UktqLW5uRG8 Problem Statement: Find the days on which each basement...
Read More
1 Comment

Why Learning MongoDB Will Boost Your Career

mongo db career
There is an increasing demand for NoSQL database experts, especially those highly trained on MongoDB. This is mainly due to the fact that NoSQL databases are replacing traditional RDBMs as NoSQL databases like MongoDB offers the flexibility to organize data in a way that makes the data usable. This along...
Read More
2 Comments

Job Responsibilities of Hadoop Professionals

Job Responsibilities of Hadoop Professionals
It is the age of Big Data and Hadoop and countless professionals have made it their dream career. After all, who doesn’t want to be a part of the most happening thing in the IT sector. We have been receiving so many queries regarding the job opportunities in Hadoop...
Read More
Leave a Comment

Graphical Exploratory Data Analysis-II

exploratory data analysis
In the previous blog, we discussed about using histograms to check the central tendency measures of a continuous variable. We also plotted the frequencies of different ordinal variables to see the distribution of observations in each category. In this blog, we will discuss about other types of data visualizations,...
Read More
Leave a Comment

Integrating SparkSQL with MySQL

Integrating Mysql with Spark SQL
In this post, we will be learning how to connect to a JDBC data-source using SparkSQL data frames. In case you are not familiar with SparkSQL, you can refer to this post for a comprehensive Introduction to SparkSQL and the post on Analyzing Crime Data using SparkSQL. We know...
Read More
Leave a Comment

Skewed Join in Pig

Skewed Join in Pig
In our previous blogs we discussed about Replicated Join and Merge Join in Pig. In this post we will be continuing our discussion by implementing skewed joins. Skewed join can be implemented if user’s underlying data is sufficiently skewed and the control needs to be given to user over...
Read More
Leave a Comment

Spark Use Case – Travel Data Analysis

Spark-Use-Case
In this blog, we will discuss on the analysis of travel dataset and gain insights from the dataset using Apache Spark. The travel dataset is publically available and the contents are detailed under the heading, ‘Travel Sector Dataset Description’. Based on the data, we will find the top 20...
Read More
Leave a Comment
© Copyright 2016.  ACADGILD.