Hadoop

Hadoop has become the benchmark platform for writing applications to store and process decentralised data in batch mode.

Hadoop is presented as a framework that provides a number of key bricks to developers and administrators:

•     The infrastructure components for the set-up of data storage and processing clusters. Data storage relies on HDFS, a distributed file system that houses data for processing;
•     MapReduce: a distributed processing Java framework used to develop processing tasks to be performed on cluster data;
•     Hadoop also supplies a set of tools that can launch and monitor tasks at the cluster level.

The main idea behind Hadoop is that HDFS automatically distributes the data across the cluster, and processing is performed as close as possible to the data level (a function made possible by MapReduce). In this way, data transfers are reduced to the strict minimum necessary.
Hadoop is written in Java and is supported by multiple US start-ups. Further, it has become a sort of de facto standard for writing data processing applications and has won over all of the major players in the sector.