Hadoop ch5

  • View
    190

  • Download
    2

Embed Size (px)

Transcript

  • 1. DEVELOPING A MAPREDUCE APPLICATION CHAPTER 5 13 8 11

2. 13 8 11 3. 13 8 11 4. 13 8 11 5. 13 8 11 6. API API . Conguration . String , . 13 8 11 7. 13 8 11 8. . 13 8 11 9. - POM Maven POM . hadoop-core hamcrest-all mrunit 13 8 11 10. (pseudodistributed), . , . 13 8 11 11. hadoop-local.xml - Filesystem Jobtracker . hadoop-localhost.xml - . hadoop-cluster.xml - . 13 8 11 12. $HADOOP_INSTALL . Standalone or pseudodistributed 13 8 11 13. HELPER GenericOptionsParser - Conguration . - Tool, ToolRunner . Tool, ToolRunner - 13 8 11 14. MRUNIT 13 8 11 15. 13 8 11 16. REDUCER 13 8 11 17. 13 8 11 18. 13 8 11 19. 13 8 11 20. JOB 13 8 11 21. 13 8 11 22. 13 8 11 23. JVM Job , , . , Job JAR . setJarByClass() JAR job JAR . 13 8 11 24. Ant Maven JAR . % mvn package -DskipTests JAR job , JAR manifest main . main manifest , . 13 8 11 25. JAR job JAR lib . classes . 13 8 11 26. JOB Mapper Mapper Reducers Reduce Slot Reducers Combiners Combiner Custom serialization Writable comparator , RawComparator . Shue tweaks MapReduce Shue . 13 8 11 27. Job . . HPROF Proler 13 8 11