Apache Flume, Kafka, Sqoop

In Big Data project, the main challenge is to collect an enormous volume of data. We need a distributed high throughput messaging systems to overcome it.
- Apache Kafka a Messaging System is typically
responsible for transferring data from one
application to another.
- Flume is a distributed, reliable, and available
service for efficiently collecting, aggregating,
and moving large amounts of streaming event
data.
- Apache Sqoop has been designed for efficiently
transferring bulk data between Apache Hadoop
and structured data stores such as relational
databases and vice versa.

Hadoop Distributed File Systems

HDFS is a filesystem designed for storing very large files running on clusters of commodity hardware. Very large means files that are hundreds of megabytes, gigabytes or terabytes in size.
- It is designed on the principle of storage less
number of large files.
- HDFS stores data reliably even in the case of
hardware failure.
- It provides fault-tolerance storage layer for
Hadoop and its other components (Eco System).

Map-Reduce, Spark, Flink

Big Data processing engines are responsible for computing over data in a data system. While there is no authoritative definition setting apart "engines" from "frameworks", it is sometimes useful to define the former as the actual component responsible for operating on data and the latter as a set of components designed to do the same.
- Batch-only Engine:- Map-Reduce
- Stream processing Engine:- Spark, Flink

Hive, Tableau

Big Data analytics is the process of collecting, organizing and analyzing large sets of data (called Big Data) to discover patterns and other useful information. The apache Hive is a data warehouse system built on top of the Apache Hadoop. Hive can be utilized for easy data summarization, ad-hoc queries, analysis of large datasets stores in various databases or file systems integrated with Hadoop. Ideally, we use Hive to apply structure (tables) on persisted a large amount of unstructured data in HDFS and subsequently query those data for analysis.

Apache Ranger

Apache Ranger is a framework to enable, monitor and manage comprehensive data security across the Hadoop platform.

The vision with Ranger is to provide comprehensive security across the Apache Hadoop ecosystem. With the advent of Apache YARN, the Hadoop platform can now support a true data lake architecture. Enterprises can potentially run multiple workloads, in a multi tenant environment. Data security within Hadoop needs to evolve to support multiple use cases for data access, while also providing a framework for the central administration of security policies and monitoring of user access.