Big Data With Hadoop

About Course

More and more organizations are taking on the challenge of analyzing big data. This course teaches you how to use the Hadoop technologies.
In this course, you’ll learn how to use technologies like Hive, Pig, Oozie, and Sqoop with Hadoop . you will learn how big data is driving organisational change and the key challenges organizations face when trying to analyse massive data sets. You will learn fundamental techniques, such as data mining and stream processing.

You will also learn how to design and implement PageRank algorithms using MapReduce, a programming paradigm that allows for massive scalability across hundreds or thousands of servers in a Hadoop cluster. You will learn how big data has improved web search and how online advertising systems work. By the end of this course, you will have a better understanding of the various applications of big data methods in industry and research.

Introduction To Big Data

What is Big data
Big Data opportunities
Big Data Challenges
Characteristics of Big data
Real Time Big Data Use cases

Introduction To Hadoop

Hadoop Distributed File System
Comparing Hadoop & SQL
Industries using Hadoop
Data Locality
Hadoop Architecture
Map Reduce & HDFS
Using the Hadoop single node image (Clone)

The Hadoop Distributed File System (HDFS): Storage.

HDFS Design & Concepts
Blocks, Name nodes and Data nodes
Anatomy of File Write and Read
Hadoop DFS the Command-Line Interface
Basic File System Operations
Multi Node Cluster Setup and its operations
More detailed explanation about Configuration files
Metadata, FS image, Edit log, Secondary Name Node and Safe Mode
FSCK Utility. (Block report)
How to add New Data Node dynamically
HDFS High-Availability and HDFS Federation
How to decommission a Data Node dynamically (Without stopping cluster)
How to override default configuration at system level and Programming level
HDFS Federation
ZOOKEEPER Leader Election Algorithm
Java API for HDFS Commands
Exercise and small use case on HDFS

Core Hadoop (Map-Reduce)

Functional Programming Basics
Map and Reduce Basics and its Architecture
Anatomy of a Map Reduce Job Run
Legacy Architecture ->Job Submission, Job Initialization, Task Assignment, Task
Execution, Progress and Status Updates
Job Completion, Failures, Shuffling and Sorting
Splits, Record reader, Partition, Types of partitions & Combiner
Hands on “Word Count” in Map/Reduce in standalone and Pseudo distribution Mode
Types of Schedulers and Counters
Getting the data from RDBMS into HDFS using Custom data types
Distributed Cache and Hadoop Streaming (Python, Ruby and R)
Sequential Files and Map Files
Optimization Techniques -> Speculative Execution, JVM Reuse and No. Slots
Enabling Compressions and Compression Codec’s
Map side Join with distributed Cache
Secondary Sorting. Creating custom data types, comparators
Types of I/O Formats: Multiple outputs, NLINE input format
Handling small files using Combine File Input Format
YARN and Hands on Practical session
Hadoop 1.x VS Hadoop 2.x VS Hadoop 3.x

NOSQL Basics And HBSASE Database

NOSQL
ACID in RDBMS and BASE in NOSQL
CAP Theorem and Types of Consistency
Types of No SQL Databases in detail, Columnar Databases in Detail (HBASE and
CASSANDRA).TTL, Bloom Filters and Compensation
HBASE
HBase concepts
HBase Data Model and Comparison between RDBMS and NOSQL
HBase Architecture, Master and Region Servers
Block Cache and sharing
HBase Operations (DDL and DML) through Shell and JAVA API Programming.
Splits, Catalog Tables
Data Modelling (Sequential, Salted, Promoted and Random Keys)
Client Side Buffering and Process 1 million records using Client side Buffering
HBASE Counters, Enabling Replication and HBASE RAW Scans
HBASE Filters, Bulk Loading and Coprocessors (Endpoints and Observers)
Real world use case consisting of HDFS, MR and HBASE
Hadoop In Cloud
Introduction to AWS (Amazon Web Service)
Launching 4 Node Cluster in AWS and EMR

HIVE With MYSQL

Introduction and Architecture
Hive Services, Hive Shell, Hive Server and Hive Web Interface (HWI)
Meta store, Types of Meta Stores
Configuring External Meta store
HIVE QL
OLTP vs OLAP
Working with Tables and different File Formats in HIVE
Primitive data types and complex data types
Working with Partitions, Multiple Inserts and dynamic Partitioning
User Defined Functions
Hive Bucketed Tables and Sampling
External partitioned tables, Map the data to the partition in the table, writing the output of one query to another table, multiple inserts
Differences between ORDER BY, DISTRIBUTE BY and SORT BY
RC File. INDEXES and VIEWS.MAPSIDE JOINS
Compression on hive tables and Migrating Hive tables
Dynamic substation of Hive and Different ways of running Hive
How to enable Update in HIVE
Log Analysis on Hive
Access HBASE tables using Hive

PIG, HCATALOG And INTEGRATIONS.

PIG
Execution Types
Grunt Shell, Pig Latin, Data Processing, and Schema on read
Primitive data types and complex data types
Tuple schema, BAG Schema and MAP Schema
Loading and Storing, Filtering, Grouping & Joining
Debugging commands (Illustrate and Explain)
Validations in PIG, Type casting in PIG
Working with Functions, User Defined Functions
Types of JOINS in pig and Replicated Join in detail
SPLITS and Multiquery execution
Error Handling, FLATTEN and ORDER BY
Parameter Substitution, Nested For Each
User Defined Functions, Dynamic Invokers and Macros
How to access HBASE using PIG
How to Load and Write JSON DATA using PIG
Piggy Bank
HCATALOG
Installation
Introduction to HCATALOG
About Hcatalog with PIG, HIVE and MR

SQOOP, FLUME, OOZIE And ZOOKEEPER

SQOOP
Installation
Import Data: (Full table, Only Subset, Target Directory, protecting Password, file format other than CSV, Compressing, Control Parallelism, All tables Import)
Modelling sequences
Incremental Import: (Import only New data, Last Imported data, storing Password in
Metastore, Sharing Metastore between Sqoop Clients)
Free Form Query Import
Export data to RDBMS, HIVE and HBASE
Hands on Exercises
FLUME
Installation
Introduction to Flume
Flume Agents: Sources, Channels and Sinks
Log User information using Java program in to HDFS using LOG4J and Avro
Source
Log User information using Java program in to HBASE using LOG4J and Avro
Source
Log User information using Java program in to HDFS using Tail Source
Use case of Flume: Flume the data from twitter in to HDFS and HBASE
Do some analysis using HIVE and PIG
OOZIE
Workflow (Action, Start, Action, End, Kill, Join and Fork), Schedulers, Coordinators and Bundles
Workflow to show how to schedule Sqoop Job, Hive, MR and PIG
Real world Use case which will find the top websites used by users of certain ages and will be scheduled to run for every one hour

SCALA (Object Oriented And Functional Programming)

Getting started With Scala
Interactive Scala – REPL, data types, variables, expressions, simple functions
Running the program with Scala Compiler
Explore the type lattice and use type inference
Define Methods and Pattern Matching
Scala Environment Set up
Scala set up on Windows
Scala set up on UNIX
What is Functional Programming
Differences between OOPS and FPP
Collections (Very Important for Spark)
Iterating, mapping, filtering and counting
Regular expressions and matching with them
Maps, Sets, group By, Options, flatten, flat Map
Word count, IO operations, file access, flatMap
Object Oriented Programming
Classes and Properties
Objects, Packaging and Imports
Traits
Objects, classes, inheritance, Lists with multiple related types, apply Integrations
What is SBT?
Integration of Scala in Eclipse IDE
Integration of SBT with Eclipse

Download Complete Course Details

Download Now

Pre-Requisites

Graduate /PG with computer science or a related numerical discipline.
Basic Knowledge of Java , Map Reduce, Data structure, SQL, to learn hadoop.
Knowledge of statistics and mathematics is beneficial.

Job Opportunities

Big Data Analytics Business Consultant
Big Data Analytics Architect
Big Data Engineer
Data Administrator
Big Data Solution Architect
Big Data Analyst
Analytics Associate
Data Analyst Business
Analyst Data/Analytics Manager
Business Intelligence Manager
Business Intelligence Analytics Consultant
Metrics and Analytics Specialist

0 %

Big Data Business Consultant

0 %

Data Analyst

0 %

Data Scientist

Satisfied Clients

Sanjana

I attended Big data hadoop course, training went on very well and I was able to explore in and out concepts in working with big data eco system. Trainer who taught me had a vast knowledge about the big data solutions and the exercise which the institute provided really helped me to understand the in depth idea of Big data. Trainer was very friendly and ready to provide help and support all the time. Never hesitated to clarify our questions. I would strong recommend this institute, if someone is looking for Big data hadoop training centre in Pune.

Tushar K

I am Tushar and completed my hadoop admin training in Pune at A.I. Analytics. My trainer is very professional and his way of approach toward the class was very interactive and interesting. He always used to ensure that everyone in the class is clear about the days hadoop training topics. I like to thank A.I. Analytics and my trainer for providing this big data training and placement in Pune

Yashesha

I completed my Big Data Training in A.I. analytics. The explanations given by trainer was nice and easy to learn. He will move to the next topic only after we completely understood the current session. He will clear the doubts whenever we call him, He is very friendly, The whole training session are very interactive and useful.