service network restart重启失败_NodeManager 启动失败问题处理

欢迎关注公众号

NodeManager 启动失败一例：

报错如下：

20/09/16 15:47:20 ERROR nodemanager.NodeManager: Error starting NodeManagerjava.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2670)        at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:270)        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)        at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)        at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:323)        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)        at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:516)        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:974)        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:1054)Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2638)        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2662)        ... 10 moreCaused by: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2542)        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2636)        ... 11 more
           

查看配置

<property>    <name>yarn.nodemanager.aux-services.spark_shuffle.classname>    <value>org.apache.spark.network.yarn.YarnShuffleServicevalue>  property>
           

由于配置的 Spark 的 shuffle 配置导致，由于环境没有安装 Spark，所以导致找不到包。

解决办法

将Spark 安装包下面的 ${SPARK_HOME}/yarn/spark-3.0.1-yarn-shuffle.jar 拷贝到 ${HADOOP_HOME}/share/hadoop/yarn/lib/
           

⚠️  注意是 share/hadoop/yarn/lib 不是 share/hadoop/yarn 

欢迎关注公众号