今天在HBase群裡碰到一個用戶端死鎖問題，堆棧如下
Java stack information for the threads listed above:
===================================================
"KVQueueService-Handler-2-EventThread":
at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.stop(ZooKeeperNodeTracker.java:98)
- waiting to lock <0x0000000760ca2680> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:605)
- locked <0x0000000760c88e00> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1698)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:374)
at org.apache.hadoop.hbase.zookee
per.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
"TableDataQueueService-Handler-4":
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:600)
- waiting to lock <0x0000000760c88e00> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1698)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:132)
- locked <0x0000000760ca2680> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
at org.
apache.hadoop.hbase.zookeeper.RootRegionTracker.waitRootRegionLocation(RootRegionTracker.java:83)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:830)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:942)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:841)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.j
ava:942)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:845)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1520)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1376)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:936)
at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:792)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:775)
at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.put(HTablePool.java:402)
at com.alibaba.alimonitor.store.hbase.TableDataDaoService.writeData(TableDataDaoService.java:111)
at com.alibaba.alimonitor.store.service.Tab
leDataQueueService.handle(TableDataQueueService.java:28)
at com.alibaba.alimonitor.store.service.TableDataQueueService.handle(TableDataQueueService.java:14)
at com.alibaba.alimonitor.store.service.AbstractQueueService$HandleThread.run(AbstractQueueService.java:115)
Found 1 deadlock.
可見發生死鎖的業務線程和zookeeper的eventThread線程。EventThread是zookeeper用來通知用戶端watcher的線程，它會回調使用者注冊的watcher。
在這裡，首先是eventThread處理connectionEvent的Expired事件
case Expired:
if (ZKUtil.isSecureZooKeeper(this.conf)) {
// We consider Expired equivalent to AuthFailed for this
// connection. Authentication is never going to complete. The
// client should proceed to do cleanup.
saslLatch.countDown();
}
String msg = prefix(this.identifier + " received expired from " +
"ZooKeeper, aborting");
// TODO: One thought is to add call to ZooKeeperListener so say,
// ZooKeeperNodeTracker can zero out its data values.
if (this.abortable != null) this.abortable.abort(msg,
new KeeperException.SessionExpiredException());
break;
}
這裡會abort調用，其實是調用的HConnectionImplementation的abort方法
public void abort(final String msg, Throwable t) {
if (t instanceof KeeperException) {
LOG.info("This client just lost it's session with ZooKeeper, will"
+ " automatically reconnect when needed.");
if (t instanceof KeeperException.SessionExpiredException) {
LOG.info("ZK session expired. This disconnect could have been" +
" caused by a network partition or a long-running GC pause," +
" either way it's recommended that you verify your environment.");
resetZooKeeperTrackers();
}
return;
}
if (t != null) LOG.fatal(msg, t);
else LOG.fatal(msg);
this.aborted = true;
close();
}
再看resetZooKeeperTrackers方法，是一個同步方法，EventThread在這裡拿到HConnectionImplementation對象鎖
private synchronized void resetZooKeeperTrackers() {
if (masterAddressTracker != null) {
masterAddressTracker.stop();
masterAddressTracker = null;
}
if (rootRegionTracker != null) {
rootRegionTracker.stop();
rootRegionTracker = null;
}
clusterId = null;
if (zooKeeper != null) {
zooKeeper.close();
zooKeeper = null;
}
}
這個時候業務線程開始region location，進入RootRegionTrack的waitRootRegionLocation
public ServerName waitRootRegionLocation(long timeout)
throws InterruptedException {
if (false == checkIfBaseNodeAvailable()) {
String errorMsg = "Check the value configured in 'zookeeper.znode.parent'. "
+ "There could be a mismatch with the one configured in the master.";
LOG.error(errorMsg);
throw new IllegalArgumentException(errorMsg);
}
return dataToServerName(super.blockUntilAvailable(timeout, true));
}
業務線程進入blockUntilAvailable，拿到父類ZookeeperNodeTracker的對象鎖
public synchronized byte [] blockUntilAvailable(long timeout, boolean refresh)
throws InterruptedException {
if (timeout < 0) throw new IllegalArgumentException();
boolean notimeout = timeout == 0;
long startTime = System.currentTimeMillis();
long remaining = timeout;
if (refresh) {
try {
//這裡正好抛出異常
this.data = ZKUtil.getDataAndWatch(watcher, node);
} catch(KeeperException e) {
abortable.abort("Unexpected exception handling blockUntilAvailable", e);
}
}
while (!this.stopped && (notimeout || remaining > 0) && this.data == null) {
// We expect a notification; but we wait with a
// a timeout to lower the impact of a race condition if any
wait(100);
remaining = timeout - (System.currentTimeMillis() - startTime);
}
return this.data;
}
由于這裡zookeeper調用正好抛出異常導緻業務線程去abort，這個時候因為EventThread已經先abort了，是以業務線程需要等待HConnectionImplementation對象鎖的釋放。而這個時候EventThread繼續執行，resetZooKeeperTrackers中繼續調用 rootRegionTracker.stop()方法
public synchronized void stop() {
this.stopped = true;
notifyAll();
}
這個方法需要拿到父類ZookeeperNodeTracker的對象鎖，而此時這個鎖已經被業務線程拿到了，這就發生了死鎖。根本原因abort操作沒有做并發控制。最新版本的hbase已經修複了這個bug，修複後的abort方法，增加了resetting同步狀态，後續abort請求直接return了。
public void abort(final String msg, Throwable t) {
if (t instanceof KeeperException) {
LOG.info("This client just lost it's session with ZooKeeper, will"
+ " automatically reconnect when needed.");
if (t instanceof KeeperException.SessionExpiredException) {
LOG.info("ZK session expired. This disconnect could have been" +
" caused by a network partition or a long-running GC pause," +
" either way it's recommended that you verify your environment.");
//同步狀态
synchronized (resetLock) {
if (resetting) return;
this.resetting = true;
}
resetZooKeeperTrackers();
this.resetting = false;
}
return;
}
if (t != null) LOG.fatal(msg, t);
else LOG.fatal(msg);
this.aborted = true;
close();
}
官方issue位址 https://issues.apache.org/jira/browse/HBASE-7259
![hbase shell出現ERROR: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException[圖]](data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=)