序言
海量的数据无论是存储还是计算，总是要保证其架构的高可用，数据仓库的构建是一个合的过程，而微服务又是一个分的过程，天下大势，分分合合。
不同的场景适合于不同的技术，不要在一个里面觉得这个技术就是银弹，可能也只是昙花一现。
hadoop相关问题
序：namenode高可用问题
namenode的高可用是由QJM和zkfc加zk集群来实现的，当宕机再启动的时候，会切换很快，但是如果直接宕机或者是hang机，当ssh无法登录上去的时候，就会导致切换不成功，一直为standby状态。。。需要登录上去，进行探测namenode进程是否存在，如果存在则杀死，这就是sshfence的实现，但是要在宕机的时候，也可以切换，必须修改为如下配置：
<property> <name>dfs.ha.fencing.methods</name> <value> sshfence shell(/bin/true) </value></property> 
namenode不可用导致集群不可用：
#只读操作也不可以 hdfs dfs -ls hdfs://ns/user 21/03/02 04:21:21 INFO retry.RetryInvocationHandler: Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB over KEL1/192.168.1.99:9000 after 1 fail over attempts. Trying to fail over after sleeping for 659ms. org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87) at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1774) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1313) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3850) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1011) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:843) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043) ##namenode log 2021-03-02 05:30:27,301 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Get corrupt file blocks returned error: Operation category READ is not supported in state standby 2021-03-02 05:30:28,171 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Get corrupt file blocks returned error: Operation category READ is not supported in state standby ##zkfc log 2021-03-02 06:14:13,573 WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at KEL/192.168.1.10:9000: java.net.SocketTimeoutException: 45000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/192.168.1.10:54563 remote=KEL/192.168.1.10:9000] Call From KEL/192.168.1.10 to KEL:9000 failed on socket timeout exception: java.net.SocketTimeoutException: 45000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/192.168.1.10:54563 remote=KEL/192.168.1.10:9000]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout ##两次切换的日志 2021-03-02 08:11:18,790 WARN org.apache.hadoop.ha.FailoverController: Unable to gracefully make NameNode at KEL/192.168.1.10:9000 standby (unable to connect) org.apache.hadoop.net.ConnectTimeoutException: Call From KEL1/192.168.1.99 to KEL:9000 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 20000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=KEL/192.168.1.10:9000]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:751) at org.apache.hadoop.ipc.Client.call(Client.java:1479) at org.apache.hadoop.ipc.Client.call(Client.java:1412) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy9.transitionToStandby(Unknown Source) at org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.transitionToStandby(HAServiceProtocolClientSideTranslatorPB.java:112) at org.apache.hadoop.ha.FailoverController.tryGracefulFence(FailoverController.java:172) at org.apache.hadoop.ha.ZKFailoverController.doFence(ZKFailoverController.java:514) at org.apache.hadoop.ha.ZKFailoverController.fenceOldActive(ZKFailoverController.java:505) at org.apache.hadoop.ha.ZKFailoverController.access$1100(ZKFailoverController.java:61) at org.apache.hadoop.ha.ZKFailoverController$ElectorCallbacks.fenceOldActive(ZKFailoverController.java:892) at org.apache.hadoop.ha.ActiveStandbyElector.fenceOldActive(ActiveStandbyElector.java:910) at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:809) at org.apache.hadoop.ha.ActiveStandbyElector.proce***esult(ActiveStandbyElector.java:418) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:599) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Caused by: org.apache.hadoop.net.ConnectTimeoutException: 20000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=KEL/192.168.1.10:9000] at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:534) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:614) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:712) at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1528) at org.apache.hadoop.ipc.Client.call(Client.java:1451) ... 14 more 2021-03-02 08:11:18,791 INFO org.apache.hadoop.ha.NodeFencer: ====== Beginning Service Fencing Process... ====== 2021-03-02 08:11:18,791 INFO org.apache.hadoop.ha.NodeFencer: Trying method 1/2: org.apache.hadoop.ha.SshFenceByTcpPort(null) 2021-03-02 08:11:18,793 INFO org.apache.hadoop.ha.SshFenceByTcpPort: Connecting to KEL... 2021-03-02 08:11:18,793 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: Connecting to KEL port 22 2021-03-02 08:11:48,818 WARN org.apache.hadoop.ha.SshFenceByTcpPort: Unable to connect to KEL as user root com.jcraft.jsch.JSchException: timeout: socket is not established at com.jcraft.jsch.Util.createSocket(Util.java:386) at com.jcraft.jsch.Session.connect(Session.java:182) at org.apache.hadoop.ha.SshFenceByTcpPort.tryFence(SshFenceByTcpPort.java:100) at org.apache.hadoop.ha.NodeFencer.fence(NodeFencer.java:97) at org.apache.hadoop.ha.ZKFailoverController.doFence(ZKFailoverController.java:532) at org.apache.hadoop.ha.ZKFailoverController.fenceOldActive(ZKFailoverController.java:505) at org.apache.hadoop.ha.ZKFailoverController.access$1100(ZKFailoverController.java:61) at org.apache.hadoop.ha.ZKFailoverController$ElectorCallbacks.fenceOldActive(ZKFailoverController.java:892) at org.apache.hadoop.ha.ActiveStandbyElector.fenceOldActive(ActiveStandbyElector.java:910) at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:809) at org.apache.hadoop.ha.ActiveStandbyElector.proce***esult(ActiveStandbyElector.java:418) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:599) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) 2021-03-02 08:11:48,820 WARN org.apache.hadoop.ha.NodeFencer: Fencing method org.apache.hadoop.ha.SshFenceByTcpPort(null) was unsuccessful. 2021-03-02 08:11:48,820 INFO org.apache.hadoop.ha.NodeFencer: Trying method 2/2: org.apache.hadoop.ha.ShellCommandFencer(/bin/true) 2021-03-02 08:11:48,852 INFO org.apache.hadoop.ha.ShellCommandFencer: Launched fencing command '/bin/true' with pid 5931 2021-03-02 08:11:48,856 INFO org.apache.hadoop.ha.NodeFencer: ====== Fencing successful by method org.apache.hadoop.ha.ShellCommandFencer(/bin/true) ====== 2021-03-02 08:11:48,856 INFO org.apache.hadoop.ha.ActiveStandbyElector: Writing znode /hadoop-ha/ns/ActiveBreadCrumb to indicate that the local node is the most recent active... 2021-03-02 08:11:48,863 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at KEL1/192.168.1.99:9000 active... 2021-03-02 08:11:49,387 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at KEL1/192.168.1.99:9000 to active state 1 数据副本问题
在查看namenode的日志时，出现数据块管理报错，没有足够的副本来存放数据，不停的WARN：
2021-03-02 04:35:38,652 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to place enough replicas, still in need of 1 to reach 3 (unavailableStorages=[], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=false) For more information, please enable DEBUG log level on org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy2021-03-02 04:35:38,653 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to place enough replicas, still in need of 1 to reach 3 (unavailableStorages=[DISK], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=false) For more information, please enable DEBUG log level on org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy 看报错的信息，大致的意思就是本来设置了一个block为三个副本，但是还差一个才能到达三个副本。
这种情况出现的原因有很多，毕竟hadoop是运行在廉价的物理机之上，那么无论是服务器节点还是硬盘都随时可能出现故障，出现坏盘或者坏机器，从而导致数据副本不足。
hdfs为了保证数据的可靠性，会将一个文件切分成一个个的block，一个block默认大小为124M，然后将block分布在不同的datanode节点之上，当有节点出现问题的时候，会自动的进行复制迁移到正常的节点上。
偶尔也需要进行手动的迁移，也就是本来是三副本现在不足，那么就先设置为2副本，然后再设置为3副本，从而手动触发数据迁移复制。
[root@KEL logs]# hdfs fsck /Connecting to namenode via http://KEL1:50070/fsck?ugi=root&path=%2FFSCK started by root (auth:SIMPLE) from /192.168.1.10 for path / at Wed Mar 03 13:03:26 CST 2021............................................................................................................................................................................................................................................./tmp/hadoop-yarn/staging/root/.staging/job_1614266885310_0001/job.jar: Under replicated BP-184102405-192.168.1.10-1612873956948:blk_1073742395_1571. Target Replicas is 10 but found 2 replica(s)../tmp/hadoop-yarn/staging/root/.staging/job_1614266885310_0001/job.split: Under replicated BP-184102405-192.168.1.10-1612873956948:blk_1073742396_1572. Target Replicas is 10 but found 3 replica(s)....../tmp/hadoop-yarn/staging/root/.staging/job_1614529282649_0001/job.jar: Under replicated BP-184102405-192.168.1.10-1612873956948:blk_1073747250_6426. Target Replicas is 10 but found 3 replica(s)../tmp/hadoop-yarn/staging/root/.staging/job_1614529282649_0001/job.split: Under replicated BP-184102405-192.168.1.10-1612873956948:blk_1073747251_6427. Target Replicas is 10 but found 3 replica(s)..../tmp/hadoop-yarn/staging/root/.staging/job_1614648088274_0001/job.jar: Under replicated BP-184102405-192.168.1.10-1612873956948:blk_1073747449_6625. Target Replicas is 10 but found 3 replica(s)../tmp/hadoop-yarn/staging/root/.staging/job_1614648088274_0001/job.split: Under replicated BP-184102405-192.168.1.10-1612873956948:blk_1073747450_6626. Target Replicas is 10 but found 3 replica(s)...........................................................................................................................................................................................................................................................................................................................................................................................Status: HEALTHY Total size: 577584452 B Total dirs: 300 Total files: 626 Total symlinks: 0 Total blocks (validated): 521 (avg. block size 1108607 B) Minimally replicated blocks: 521 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 6 (1.1516315 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 2.9980807 Corrupt blocks: 0 Missing replicas: 43 (2.6791277 %) Number of data-nodes: 3 Number of racks: 1FSCK ended at Wed Mar 03 13:03:26 CST 2021 in 70 millisecondsThe filesystem under path '/' is HEALTHY 可以使用hdfs fsck来进行文件的检查，会检查所有的副本数是否满足，如上所示，有几个文件的副本数不足，产生的原因是运行任务的过程中出现了报错，而没有将相关的数据进行清理。
#手动设置所有文件的副本数为3[root@KEL module]hdfs dfs -setrep -w 3 -R / 发现用了很久时间都没有进行设置。Waiting for /tmp/hadoop-yarn/staging/history/done/2021/03/02/000000/job_1614665577456_0002_conf.xml ... doneWaiting for /tmp/hadoop-yarn/staging/root/.staging/job_1614266885310_0001/job.jar 在tmp目录很久也未设置成功，从而直接将此文件删除：
[root@KEL module]# hdfs dfs -rm -r /tmp/hadoop-yarn/staging/21/03/03 21:51:42 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.Deleted /tmp/hadoop-yarn/staging#删除临时文件之后，再次进行检查[root@KEL module]# hdfs fsck /Connecting to namenode via http://KEL1:50070/fsck?ugi=root&path=%2FFSCK started by root (auth:SIMPLE) from /192.168.1.10 for path / at Wed Mar 03 13:41:53 CST 2021..........................................................................................................................................................................................................................................................................................................................................................................................Status: HEALTHY Total size: 533172477 B Total dirs: 268 Total files: 378 Total symlinks: 0 Total blocks (validated): 274 (avg. block size 1945884 B) Minimally replicated blocks: 274 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 3.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 3 Number of racks: 1FSCK ended at Wed Mar 03 13:41:53 CST 2021 in 76 millisecondsThe filesystem under path '/' is HEALTHY 在进行查看job history文件的时候，出现错误（historyserver日志）：
2021-03-03 21:56:50,962 ERROR org.apache.hadoop.mapreduce.v2.hs.JobHistory: Error while scanning intermediate done dirjava.io.FileNotFoundException: File /tmp/hadoop-yarn/staging/history/done_intermediate does not exist. at org.apache.hadoop.fs.Hdfs$DirListingIterator.<init>(Hdfs.java:211) at org.apache.hadoop.fs.Hdfs$DirListingIterator.<init>(Hdfs.java:195) at org.apache.hadoop.fs.Hdfs$2.<init>(Hdfs.java:177) at org.apache.hadoop.fs.Hdfs.listStatusIterator(Hdfs.java:177) at org.apache.hadoop.fs.FileContext$22.next(FileContext.java:1494) at org.apache.hadoop.fs.FileContext$22.next(FileContext.java:1489) at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90) at org.apache.hadoop.fs.FileContext.listStatus(FileContext.java:1489) at org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils.listFilteredStatus(JobHistoryUtils.java:505) at org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils.localGlobber(JobHistoryUtils.java:452) at org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils.localGlobber(JobHistoryUtils.java:444) at org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils.localGlobber(JobHistoryUtils.java:439) at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.scanIntermediateDirectory(HistoryFileManager.java:795) at org.apache.hadoop.mapreduce.v2.hs.JobHistory$MoveIntermediateToDoneRunnable.run(JobHistory.java:189) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 页面出现500报错。。。开启了mapreduce的日志聚合之后，默认会将日志存放在这个tmp路径之中。（当不能触发自动迁移的时候，就应该主动去手动触发迁移。）
hdfs-site.xml配置文件中，参数dfs.replication设置副本数量，默认为3.
2 hdfs多目录挂载
这种情况一般发生在刚上线的时候，或者是服务器的盘坏了需要重新加一块盘的时候。
#对磁盘进行分区 [root@KEL ~]# fdisk /dev/sdb Welcome to fdisk (util-linux 2.23.2). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Device does not contain a recognized partition table Building a new DOS disklabel with disk identifier 0x4670e170. Command (m for help): n Partition type: p primary (0 primary, 0 extended, 4 free) e extended Select (default p): p Partition number (1-4, default 1): First sector (2048-20971519, default 2048): Using default value 2048 Last sector, +sectors or +size{K,M,G} (2048-20971519, default 20971519): Using default value 20971519 Partition 1 of type Linux and of size 10 GiB is set Command (m for help): w The partition table has been altered! #进行检查分区结果 [root@KEL ~]# fdisk -l Disk /dev/sdb: 10.7 GB, 10737418240 bytes, 20971520 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: dos Disk identifier: 0x4670e170 Device Boot Start End Blocks Id System /dev/sdb1 2048 20971519 10484736 83 Linux 再进行格式化磁盘：
[root@KEL data2]# mkfs.ext3 /dev/sdb1mke2fs 1.42.9 (28-Dec-2013)Filesystem label=OS type: LinuxBlock size=4096 (log=2)Fragment size=4096 (log=2)Stride=0 blocks, Stripe width=0 blocks655360 inodes, 2621184 blocks131059 blocks (5.00%) reserved for the super userFirst data block=0Maximum filesystem blocks=268435456080 block groups32768 blocks per group, 32768 fragments per group8192 inodes per groupSuperblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632Allocating group tables: done Writing inode tables: done Creating journal (32768 blocks): doneWriting superblocks and filesystem accounting information: done#创建挂载目录[root@KEL hadoop-2.7.2]# mkdir data2#添加开启启动文件[root@KEL data2]# cat /etc/fstab ## /etc/fstab# Created by anaconda on Sat Feb 6 18:18:57 2021## Accessible filesystems, by reference, are maintained under '/dev/disk'# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info#/dev/mapper/rhel-root / xfs defaults 0 0UUID=a6f53869-d27b-4a99-94b5-4ec61040d008 /boot xfs defaults 0 0/dev/mapper/rhel-swap swap swap defaults 0 0/dev/sdb1 /opt/module/hadoop-2.7.2/data2 ext3 defaults 0 0#测试挂载[root@KEL data2]# mount -a#修改hdfs配置文件[root@KEL hadoop]# vim hdfs-site.xml<property> <name>dfs.datanode.data.dir</name> <value>file://${hadoop.tmp.dir}/dfs/data,file:///opt/module/hadoop-2.7.2/data2</value></property>#启动本机上的相关进程[root@KEL hadoop]# start-dfs.sh #检查文件结果（主要查看版本号是否一致）[root@KEL current]# pwd/opt/module/hadoop-2.7.2/data2/current[root@KEL current]# cat VERSION #Wed Mar 03 17:07:06 CST 2021storageID=DS-72c239ca-daff-4655-b95a-4e5a50bd28a8clusterID=CID-073cc3af-78f6-4716-993b-92a0b398291acTime=0datanodeUuid=1c4ffb79-0fae-42a1-af32-65fecc962cdastorageType=DATA_NODElayoutVersion=-56[root@KEL current]# cd -/opt/module/hadoop-2.7.2/data/tmp/dfs/data/current[root@KEL current]# cat VERSION #Wed Mar 03 17:07:06 CST 2021storageID=DS-ff23d5a7-fe98-404b-a5b9-5d449e3b31d2clusterID=CID-073cc3af-78f6-4716-993b-92a0b398291acTime=0datanodeUuid=1c4ffb79-0fae-42a1-af32-65fecc962cdastorageType=DATA_NODElayoutVersion=-56[root@KEL current]# pwd/opt/module/hadoop-2.7.2/data/tmp/dfs/data/current#查看空间是否增大[root@KEL hadoop]# hdfs dfsadmin -report-------------------------------------------------Live datanodes (3):Name: 192.168.1.10:50010 (KEL)Hostname: KELDecommission Status : NormalConfigured Capacity: 29180092416 (27.18 GB)DFS Used: 788815872 (752.27 MB)Non DFS Used: 3924152320 (3.65 GB)DFS Remaining: 24467124224 (22.79 GB)DFS Used%: 2.70%DFS Remaining%: 83.85%Configured Cache Capacity: 0 (0 B)Cache Used: 0 (0 B)Cache Remaining: 0 (0 B)Cache Used%: 100.00%Cache Remaining%: 0.00%Xceivers: 1Last contact: Wed Mar 03 09:12:20 CST 2021 WEB页面查看增大的空间：
3 数据不均衡
在集群中新加入一台机器之后，那么就会造成数据不均衡，新上线的机器的负载低，存储的数据量也少，从而要对数据进行一个balance操作。
#启动一个balancer操作，threshold表示每个datanode上数据量差别为1%[root@KEL sbin]# ./start-balancer.sh -threshold 1starting balancer, logging to /opt/module/hadoop-2.7.2/logs/hadoop-root-balancer-KEL.out[root@KEL sbin]# vim ../logs/hadoop-root-balancer-KEL.log [root@KEL sbin]# jps10121 Balancer2021-03-03 17:47:38,546 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.1.199:500102021-03-03 17:47:38,546 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.1.99:500102021-03-03 17:47:38,546 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.1.10:500102021-03-03 17:47:38,547 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: 1 over-utilized: [192.168.1.199:50010:DISK]2021-03-03 17:47:38,547 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: 0 underutilized: []2021-03-03 17:47:38,547 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Need to move 7.59 MB to make the cluster balanced.2021-03-03 17:47:38,547 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Decided to move 27.38 MB bytes from 192.168.1.199:50010:DISK to 192.168.1.99:50010:DISK2021-03-03 17:47:38,547 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Decided to move 151.40 MB bytes from 192.168.1.199:50010:DISK to 192.168.1.10:50010:DISK2021-03-03 17:47:38,547 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Will move 178.78 MB in this iteration2021-03-03 17:47:47,568 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.1.10:500102021-03-03 17:47:47,569 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.1.99:500102021-03-03 17:47:47,569 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.1.199:50010 在页面上查看均衡结果：
在大致平衡完成之后，需要手动进行关闭，否则大量的数据迁移会导致运行中的任务性能的降低。
[root@KEL sbin]# ./stop-balancer.sh 4 强制关机导致zookeeper无法启动
强制关机的时候，导致启动脚本中还能检测pid，从而误认为进程还存在，从而不启动。
#status说未启动[root@KEL2 zkdata]# /opt/module/zookeeper-3.4.12/bin/zkServer.sh status /opt/module/zookeeper-3.4.12/zkdata/zoo.cfg ZooKeeper JMX enabled by defaultUsing config: /opt/module/zookeeper-3.4.12/zkdata/zoo.cfgError contacting service. It is probably not running.#start的时候，检测到pid文件，认为已经在运行[root@KEL2 zkdata]# /opt/module/zookeeper-3.4.12/bin/zkServer.sh start /opt/module/zookeeper-3.4.12/zkdata/zoo.cfg ZooKeeper JMX enabled by defaultUsing config: /opt/module/zookeeper-3.4.12/zkdata/zoo.cfgStarting zookeeper ... already running as process 2260. 两种方式解决：
#找到pid文件，然后将文件删除，再次启动即可。[root@KEL2 zkdata]# grep data zoo.cfg dataDir=/opt/module/zookeeper-3.4.12/zkdata[root@KEL2 zkdata]# ls -ltotal 16-rw-r--r-- 1 root root 2 Feb 9 18:31 myiddrwxr-xr-x 2 root root 4096 Feb 17 19:38 version-2-rw-r--r-- 1 root root 170 Feb 9 18:38 zoo.cfg-rw-r--r-- 1 root root 4 Feb 16 18:36 zookeeper_server.pid[root@KEL2 zkdata]# cat zookeeper_server.pid 2260[root@KEL2 zkdata]# rm -rf zookeeper_server.pid [root@KEL2 zkdata]# /opt/module/zookeeper-3.4.12/bin/zkServer.sh start /opt/module/zookeeper-3.4.12/zkdata/zoo.cfg ZooKeeper JMX enabled by defaultUsing config: /opt/module/zookeeper-3.4.12/zkdata/zoo.cfgStarting zookeeper ... STARTED 或者是先停止，然后再启动也行。
除了zk启动会碰到这种问题，对于haddoop其他进程也可能碰到此类问题，都可以先停止，然后再启动。