Hadoop專欄（二）配置曆史伺服器、日志的聚集、配置檔案說明

一、配置曆史伺服器

置史伺服器可以檢視MapReduce任務的詳情資訊，檢視日志，定位錯誤

步驟如下：

1）配置mapred-site.xml

[root@nodetwo logs]# cd $HADOOP_HOME/etc/hadoop
[root@nodetwo hadoop]# vi mapred-site.xml      

<property>
<name>mapreduce.jobhistory.address</name>
<value>nodetwo:10020</value>
</property>
<property>
    <name>mapreduce.jobhistory.webapp.address</name>
    <value>nodetwo:19888</value>
</property>      

2）啟動曆史伺服器

[root@nodetwo hadoop]# cd /opt/install/hadoop
[root@nodetwo hadoop]# sbin/mr-jobhistory-daemon.sh start historyserver
starting historyserver, logging to /opt/install/hadoop-2.6.0-cdh5.14.2/logs/mapred-root-historyserver-nodetwo.out      

3）檢視曆史伺服器是否啟動

[root@nodetwo hadoop]# jps
5392 Jps
1857 DataNode
4068 ResourceManager
1767 NameNode
5352 JobHistoryServer
4154 NodeManager
2031 SecondaryNameNode      

4）檢視jobhistory

http://nodetwo:19888/jobhistory      

二、配置日志的聚集

日志聚集概念：應用運作完成以後，将日志資訊上傳到HDFS系統上。友善檢視

開啟日志聚集功能步驟：

（1）配置yarn-site.xml

[root@nodetwo hadoop]# cd $HADOOP_HOME/etc/hadoop
[root@nodetwohadoop]$ vi yarn-site.xml      

<!-- 日志聚集功能使能 -->
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<!-- 日志保留時間設定7天 -->
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value>604800</value>
</property>      

（2）關閉nodemanager 、resourcemanager和historymanager

[root@nodetwo hadoop]# sbin/yarn-daemon.sh stop resourcemanager
stopping resourcemanager
[root@nodetwo hadoop]# sbin/yarn-daemon.sh stop nodemanager
stopping nodemanager
[root@nodetwo hadoop]# sbin/mr-jobhistory-daemon.sh stop historyserver
stopping historyserver      

（3）啟動nodemanager 、resourcemanager和historymanager

[root@nodetwo hadoop]# sbin/yarn-daemon.sh start resourcemanager
starting resourcemanager, logging to /opt/install/hadoop-2.6.0-cdh5.14.2/logs/yarn-root-resourcemanager-nodetwo.out
[root@nodetwo hadoop]# sbin/yarn-daemon.sh start nodemanager
starting nodemanager, logging to /opt/install/hadoop-2.6.0-cdh5.14.2/logs/yarn-root-nodemanager-nodetwo.out
[root@nodetwo hadoop]# sbin/mr-jobhistory-daemon.sh start historyserver
starting historyserver, logging to /opt/install/hadoop-2.6.0-cdh5.14.2/logs/mapred-root-historyserver-nodetwo.out      

（4）測試：删除hdfs上已經存在的hdfs檔案

[root@nodetwo hadoop]$ bin/hdfs dfs -rm -R /user/hadoop/output      

（5）執行wordcount程式

[root@hadoop101 hadoop]$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-cdh5.14.2.jar wordcount /user/hadoop/input /user/hadoop/output      

（6）檢視日志

http://192.168.202.201:19888/jobhistory      

三、配置檔案說明

Hadoop配置檔案分兩類：預設配置檔案和自定義配置檔案，隻有使用者想修改某一預設配置值時，才需要修改自定義配置檔案，更改相應屬性值。

（1）預設配置檔案：存放在hadoop相應的jar包中

[core-default.xml]
      hadoop-common-*.jar/ core-default.xml
    [hdfs-default.xml]
hadoop-hdfs-*.jar/ hdfs-default.xml
    [yarn-default.xml]
hadoop-yarn-common-*.jar/ yarn-default.xml
    [mapred-default.xml]
hadoop-mapreduce-client-core-*.jar/ mapred-default.xml      

（2）自定義配置檔案：存放在$HADOOP_HOME/etc/hadoop

三、完全分布式運作模式（可自行嘗試）