service network restart重新開機失敗_NodeManager 啟動失敗問題處理

歡迎關注公衆号

NodeManager 啟動失敗一例：

報錯如下：

20/09/16 15:47:20 ERROR nodemanager.NodeManager: Error starting NodeManagerjava.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2670)        at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:270)        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)        at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)        at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:323)        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)        at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:516)        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:974)        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:1054)Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2638)        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2662)        ... 10 moreCaused by: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2542)        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2636)        ... 11 more
           

檢視配置

<property>    <name>yarn.nodemanager.aux-services.spark_shuffle.classname>    <value>org.apache.spark.network.yarn.YarnShuffleServicevalue>  property>
           

由于配置的 Spark 的 shuffle 配置導緻，由于環境沒有安裝 Spark，是以導緻找不到包。

解決辦法

将Spark 安裝包下面的 ${SPARK_HOME}/yarn/spark-3.0.1-yarn-shuffle.jar 拷貝到 ${HADOOP_HOME}/share/hadoop/yarn/lib/
           

⚠️  注意是 share/hadoop/yarn/lib 不是 share/hadoop/yarn 

歡迎關注公衆号