314 / 1971-01-01 00:00:00

Research On Multiple Files Input Programming Method Based On Mapreduce

mapreduce

Abstract: Hadoop is a software framework that supports distributed processing for the analysis and transformation of very large data sets. Hadoop can also handle a large number of small files, and many articles have reported the study on the performance of handling small files. This article first carries depth research and detailed introduction on HDFS and MapReduce, and then proposing a programming method that can allows the order of output contents in accordance with the order of input files when processing large numbers of multiple files. It has been proved in the experiment that the method is feasible and effective, meanwhile solving the problem of multiple files input. Through such output, it will bring great convenience in data processing on the later work.