MPI Parallel Computing + Hadoop

MPI – Message Passing Interface

------------------

http://mpich.org/

https://www.open-mpi.org/

http://www.mpi-forum.org/docs/docs.html

------------------

https://computing.llnl.gov/tutorials/mpi/ ; 국가연구소, 자세한 설명

http://mpitutorial.com

------------------

*API list and guide

http://www.mcs.anl.gov/research/projects/mpi/www/www3/

------------------

*mpi4py python API

https://groups.google.com/forum/#!forum/mpi4py

https://mpi4py.scipy.org/docs/usrman/install.html

pip install mpi4py

------------------

A brief introduction to Apache Hadoop

The technology known as Apache Hadoop is an open-source framework for developing distributed applications hosted by the Apache Software Foundation. The framework contains a number of subprojects. The one we are interested in is the Hadoop Core, also known as Hadoop Common.The Hadoop Common project is located within the overall Hadoop framework. It allows the development of cloud computing environments via off-the-shelf hardware such as the Raspberry Pi. The developer interacts with it by using its Java based API.Within Hadoop Common there are several significant areas that help us achieve our goal of developing parallel computing applications. Two of the most important areas are as follows:

• Hadoop MapReduce environment
• Hadoop Distributed File System (HDFS)

------------------

MapReduce = Map + Reduce

------------------

MPI, Hadoop, and parallel computing

Further resources on MPI are as follows:

• Java examples—Cardiff University
• http://users.cs.cf.ac.uk/David.W.Walker/CM0323/code.html
• MPI tutorials—Lawrence Livermore National Laboratory
• https://computing.llnl.gov/tutorials/mpi/
• Hadoop MapReduce tutorial—Apache Foundation
• http://hadoop.apache.org/docs/stable/mapred_tutorial.html
• Raspberry Pi Cloud blog
• http://raspberrypicloud.wordpress.com/2013/04/25/getting-hadoop-to-run-on-the-raspberry-pi/
• Beowulf clusters—Duke University
• http://www.phy.duke.edu/~rgb/brahma/Resources/beowulf/papers/ICPP95/icpp95.html
• Parallel Computing—Intel
• http://www.intel.com/pressroom/kits/upcrc/ParallelComputing_backgrounder.pdf
• Parallel Programming—Gribble Lab
• http://gribblelab.org/CBootcamp/A2_Parallel_Programming_in_C.html
• Virtualization—Red Hat
• http://www.redhat.com/products/cloud-computing/virtualization/
• Virtual Machines—Virtual Box
• https://www.virtualbox.org/