HiveServer2支援多用戶端的并發通路,使用ZooKeeper來管理Hive表的讀寫鎖。實際環境中,遇到了HiveServer2連接配接ZooKeeper出現Too many connections的問題,這裡是對這一問題的排查和解決過程。
問題描述
HiveServer2服務無法執行hive指令,日志中提示如下錯誤:
2013-03-2212:54:43,946 WARN zookeeper.ClientCnxn(ClientCnxn.java:run(1089)) - Session 0x0 for serverhostname/***.***.***.***:2181, unexpected error, closing socket connection andattempting reconnect
java.io.IOException: Connection reset by peer
atsun.nio.ch.FileDispatcher.read0(Native Method)
atsun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
atsun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
atsun.nio.ch.IOUtil.read(IOUtil.java:200)
atsun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
atorg.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68)
atorg.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355)
atorg.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
問題排查
1. 首先,根據HiveServer2的錯誤日志,提示是由于Connection reset by peer,即連接配接被ZooKeeper拒絕。
2. 進一步檢視HiveServer2上所配置的ZooKeeper叢集日志(使用者Hive表的讀寫鎖管理),發現如下錯誤資訊:
2013-03-2212:52:48,938 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:[email protected]] -Too many connections from /***.***.***.*** - max is 50
3. 結合HiveServer2的日志,可見是由于HiveServer2所在機器對ZooKeeper的連接配接數超過了ZooKeeper設定允許的單個client最大連接配接數(這裡是50)。
4. 我們進一步确認了是不是完全都是HiveServer2占用了這50個連接配接,顯示确實是HiveServer2程序内部占用了這50個連接配接(程序号26871即為HiveServer2程序):
[[email protected]~]$ sudo netstat -nap | grep2181
tcp 0 0 ***.***.***.***:58089 ***.***.***.***:2181 ESTABLISHED 26871/java
tcp 0 0 ***.***.***.***:57837 ***.***.***.***:2181 ESTABLISHED 26871/java
tcp 0 0 ***.***.***.***:57853 ***.***.***.***:2181 ESTABLISHED 26871/java
……
(共計50個)
5. 為什麼HiveServer2會占用這麼多連接配接?而實際并發請求量并沒有這麼多。隻能從HiveServer2的實作原理找找線索,由于HiveServer2是通過Thrift實作的,懷疑是不是其内部維護連接配接池導緻的?經過檢視hive-default.xml中發現,其中預設配置了工作線程數(這裡猜測每個工作線程會維護一個與ZooKeeper的連接配接,有待從代碼級别進行驗證):
<property>
<name>hive.server2.thrift.min.worker.threads</name>
<value>5</value>
<description>Minimum number ofThrift workerthreads</description>
</property>
<property>
<name>hive.server2.thrift.max.worker.threads</name>
<value>100</value>
<description>Maximum number ofThrift worker threads</description>
</property>
問題解決
方法一:
通過在hive-site.xml中修改HiveServer2的Thrift工作線程數,減少與ZooKeeper的連接配接請求數。這樣可能降低HiveServer2的并發處理能力。
方法二:
通過修改ZooKeeper的zoo.cfg檔案中的maxClientCnxns選項,調大對于單個Client的連接配接數限制。
以上兩個方法,需要根據自己的實際生産情況進行合理設定。
相關的配置選項:
1)hive-site.xml中:
<property>
<name>hive.server2.thrift.min.worker.threads</name>
<value>10</value>
<description>Minimum number ofThrift workerthreads</description>
</property>
<property>
<name>hive.server2.thrift.max.worker.threads</name>
<value>200</value>
<description>Maximum number ofThrift workerthreads</description>
</property>
<property>
<name>hive.zookeeper.session.timeout</name>
<value>60000</value>
<description>Zookeeper client'ssession timeout. The client is disconnected, and as a result, all locksreleased, if a heartbeat is not sent in thetimeout.</description>
</property>
2)zoo.cfg中:
#Limits the number of concurrent connections (at the socket level) that a singleclient, identified by IP address
maxClientCnxns=200
# The minimum session timeout in milliseconds that the server will allow theclient to negotiate
minSessionTimeout=1000
# The maximum session timeout in milliseconds that the server will allow theclient to negotiate
maxSessionTimeout=60000
來自 <http://www.cnblogs.com/panfeng412/archive/2013/03/23/hiveserver2-too-many-zookeeper-connections-issues.html>