1.1、按照CentOS中Hadoop之HDFS的安裝和配置搭建環境之後，發現啟動的時候報:
2020-08-03 22:09:39,841 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Encountered exception loading fsimageorg.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /home/hadoop/tmp/dfs/name. Reported: -63. Expecting = -60. at org.apache.hadoop.hdfs.server.common.StorageInfo.setLayoutVersion(StorageInfo.java:178) at org.apache.hadoop.hdfs.server.common.StorageInfo.setFieldsFromProperties(StorageInfo.java:131) at org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:635) at org.apache.hadoop.hdfs.server.namenode.NNStorage.readProperties(NNStorage.java:664) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:389) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:228) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1152) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:799) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:614) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:676) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:844) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:823) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1547) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1615)
解決方法：
hdfs-site.xml
<configuration> <property> <name>dfs.replicationname> <value>2value> property> <property> <name>dfs.namenode.http-addressname> <value>192.168.225.133:50070value> property> <property> <name>dfs.name.dirname> <value>/home/hadoop/data/hadoopfs/namevalue> property> <property> <name>dfs.data.dirname> <value>/home/hadoop/data/hadoopfs/datavalue> property>configuration>
core-site.xml
<configuration> <property> <name>fs.defaultFSname> <value>hdfs://hadoop-master:9000value> property> <property> <name>hadoop.tmp.dirname> <value>/home/hadoop/tmpvalue> property> <property> <name>io.file.buffer.sizename> <value>131072value> property>configuration>
啟動
# 格式化hadoop namenode –format/usr/local/hadoop/bin/hdfs namenode -format# 啟動 -- 這裡直接執行就可以start-dfs.sh# 驗證jps
界面通路：
http://192.168.225.133:50070/
1.2、異常：
2020-08-04 02:00:37,011 WARN org.apache.hadoop.hdfs.server.common.Storage: Failed to add storage directory [DISK]file:/home/hadoop/data/hadoopfs/data/java.io.IOException: Incompatible clusterIDs in /home/hadoop/data/hadoopfs/data: namenode clusterID = CID-cc346a2f-1d0b-49d3-9899-f8ed3e8fdd84; datanode clusterID = CID-9f0a79f5-7478-440a-aa71-e44436d3bed2 at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:779) at org.apache.hadoop.hdfs.server.datanode.DataStorage.loadStorageDirectory(DataStorage.java:302) at org.apache.hadoop.hdfs.server.datanode.DataStorage.loadDataStorage(DataStorage.java:418) at org.apache.hadoop.hdfs.server.datanode.DataStorage.addStorageLocations(DataStorage.java:397) at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:575) at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1570) at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1530) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:354) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:219) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:674) at java.lang.Thread.run(Thread.java:745)2020-08-04 02:00:37,018 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool ID needed, but service not yet registered with NN, trace:
解決方法：
清空叢集每個節點目錄的檔案：/home/hadoop/data/hadoopfs/data/并執行：hadoop namenode -format
2、叢集配置
192.168.225.133 hadoop-master192.168.225.134 hadoop-slave
免密登入
# --- hadoop-master# 生成密鑰(Overwrite (y/n)? y )ssh-keygen -t rsa -P ""# (> 為替換 >> 為追加)cat /root/.ssh/id_rsa.pub > /root/.ssh/authorized_keys# 按照提示執行，需要輸入一次密碼ssh-copy-id -i /root/.ssh/id_rsa.pub -p22 [email protected]# 測試ssh @hadoop-slave# ----hadoop-slave# 生成密鑰(Overwrite (y/n)? y )ssh-keygen -t rsa -P ""# (> 為替換 >> 為追加)cat /root/.ssh/id_rsa.pub > /root/.ssh/authorized_keys# 按照提示執行，需要輸入一次密碼ssh-copy-id -i /root/.ssh/id_rsa.pub -p22 [email protected]# 測試ssh @hadoop-master
同時修改hdfs-site.xml和slaves檔案
# 修改hdfs-site.xml --- 可以不用配置SecondaryNameNode# 增加 dfs.namenode.secondary.http-address 192.168.225.134:50090
# slaves檔案 --- 統一配置hadoop-masterhadoop-slave
yarn-site.xml
<configuration> <property> <name>yarn.nodemanager.aux-servicesname> <value>mapreduce_shufflevalue> property> <property> <name>yarn.resourcemanager.hostnamename> <value>hadoop-mastervalue> property>configuration>
3.最終
# 通路下面連接配接http://192.168.225.133:50070/dfshealth.html#tab-datanodehttp://192.168.225.133:8088/cluster/nodes
# 如果達不到下面的效果，就需要執行：# 清空叢集每個節點目錄的檔案：/home/hadoop/data/hadoopfs/data/# 并執行：hadoop namenode -format
附帶：yarn-site.xml
NodeManager相關配置參數(1) yarn.nodemanager.resource.memory-mb參數解釋：NodeManager總的可用實體記憶體。注意，該參數是不可修改的，一旦設定，整個運作過程中不可動态修改。另外，該參數的預設值是8192MB，即使你的機器記憶體不夠8192MB，YARN也會按照這些記憶體來使用(傻不傻？)，是以，這個值通過一定要配置。不過，Apache已經正在嘗試将該參數做成可動态修改的。預設值：8192(2) yarn.nodemanager.vmem-pmem-ratio參數解釋：每使用1MB實體記憶體，最多可用的虛拟記憶體數。預設值：2.1(3) yarn.nodemanager.resource.cpu-vcores參數解釋：NodeManager總的可用虛拟CPU個數。預設值：8(4) yarn.nodemanager.local-dirs參數解釋：中間結果存放位置，類似于1.0中的mapred.local.dir。注意，這個參數通常會配置多個目錄，已分攤磁盤IO負載。預設值：${hadoop.tmp.dir}/nm-local-dir(5) yarn.nodemanager.log-dirs參數解釋：日志存放位址(可配置多個目錄)。預設值：${yarn.log.dir}/userlogs(6) yarn.nodemanager.log.retain-seconds參數解釋：NodeManager上日志最多存放時間(不啟用日志聚集功能時有效)。預設值：10800(3小時)(7) yarn.nodemanager.aux-services參數解釋：NodeManager上運作的附屬服務。需配置成mapreduce_shuffle，才可運作MapReduce程式預設值：""