Hbase重啟之後就無法啟動的現象
當我們使用Hbase
的時候
遇到了
個奇怪的問題
我們使用了數台機器構建了一個集群並且按照Hadoop/Hbase的"Getting Started"安裝配置了Hadoop和Hbase之後能夠正常啟動Hadoop和Hbase並且創建table和插入數據
不過當我們訪問Master的頁面時 我們發現了第一個問題在regionserver區域出現了個的regionserver但是我們並沒有在conf/regionservers將master設置為regionserver
代碼如下 復制代碼
Region Servers
Address Start Code Load
: requests= regions= usedHeap= maxHeap=
: requests= regions= usedHeap= maxHeap=
WAMDM: requests= regions= usedHeap= maxHeap=
WAMDM: requests= regions= usedHeap= maxHeap=
WAMDM: requests= regions= usedHeap= maxHeap=
WAMDM: requests= regions= usedHeap= maxHeap=
……
但是雖然出現了以上的怪現象但是hbase似乎仍然能夠正常工作只是當我們打算重啟hbase的時候我們發現了第二個問題我們嘗試運行bin/stophbasesh之後又運行啟動hbase的腳本bin/ starthbasesh這一次當我們訪問master頁面的時候出現了如下的錯誤
代碼如下 復制代碼
HTTP ERROR:
Trying to contact region server null for region row " but failed after attempts
Exceptions:
orgapachehadoophbaseclientRetriesExhaustedException: Failed setting up proxy to /: after attempts=
orgapachehadoophbaseclientRetriesExhaustedException: Failed setting up proxy to /: after attempts=
orgapachehadoophbaseclientRetriesExhaustedException: Failed setting up proxy to /: after attempts=
RequestURI=/masterjsp
Caused by:
orgapachehadoophbaseclientRetriesExhaustedException: Trying to contact region server null for region row " but failed after attempts
Exceptions:
orgapachehadoophbaseclientRetriesExhaustedException: Failed setting up proxy to /: after attempts=
orgapachehadoophbaseclientRetriesExhaustedException: Failed setting up proxy to /: after attempts=
orgapachehadoophbaseclientRetriesExhaustedException: Failed setting up proxy to /: after attempts=
at orgapachehadoophbaseclientHConnectionManager$TableServersgetRegionServerWithRetries(HConnectionManagerjava:)
at orgapachehadoophbaseclientaScan(MetaScannerjava:)
at orgapachehadoophbaseclientaScan(MetaScannerjava:)
at orgapachehadoophbaseclientHConnectionManager$TableServerslistTables(HConnectionManagerjava:)
at orgapachehadoophbaseclientHBaseAdminlistTables(HBaseAdminjava:)
at orgapachehadoophbasegeneratedmastermaster_jsp_jspService(master_jspjava:)
at orgapachejasperruntimeHttpJspBaseservice(HttpJspBasejava:)
at javaxservlet(HttpServletjava:)
at orgmortbayjettyservletServletHolderhandle(ServletHolderjava:)
at orgmortbayjettyservletServletHandlerhandle(ServletHandlerjava:)
at orgmortbayjettysecuritySecurityHandlerhandle(SecurityHandlerjava:)
at orgmortbayjettyservletSessionHandlerhandle(SessionHandlerjava:)
at orgmortbayjettyhandlerContextHandlerhandle(ContextHandlerjava:)
at orgmortbayjettywebappWebAppContexthandle(WebAppContextjava:)
at orgmortbayjettyhandlerContextHandlerCollectionhandle(ContextHandlerCollectionjava:)
at orgmortbayjettyhandlerHandlerWrapperhandle(HandlerWrapperjava:)
at orgmortbayjettyServerhandle(Serverjava:)
at orgmortbayjettyHttpConnectionhandleRequest(HttpConnectionjava:)
at orgmortbayjettyHttpConnection$RequestHandlerheaderComplete(HttpConnectionjava:)
at orgmortbayjettyHttpParserparseNext(HttpParserjava:)
at orgmortbayjettyHttpParserparseAvailable(HttpParserjava:)
at orgmortbayjettyHttpConnectionhandle(HttpConnectionjava:)
at orgmortbayionioSelectChannelEndPointrun(SelectChannelEndPointjava:)
at orgmortbaythreadQueuedThreadPool$PoolThreadrun(QueuedThreadPooljava:)
Powered by Jetty://
同時在master的log中出現了如下的錯誤
:: DEBUG orgapachehadoophbasezookeeperZooKeeperWrapper: Read ZNode /hbase/rootregionserver got :
:: INFO orgapachehadoopipcHbaseRPC: Server at /: could not be reached after tries giving up
:: DEBUG orgapachehadoophbaseclientHConnectionManager$TableServers: Root region location changed Sleeping
:: DEBUG orgapachehadoophbaseclientHConnectionManager$TableServers: Wake Retry finding root region
:: DEBUG orgapachehadoophbasezookeeperZooKeeperWrapper: Read ZNode /hbase/rootregionserver got :
:: INFO orgapachehadoopipcHbaseRPC: Server at /: could not be reached after tries giving up
:: DEBUG orgapachehadoophbaseclientHConnectionManager$TableServers: Root region location changed Sleeping
:: DEBUG orgapachehadoophbaseclientHConnectionManager$TableServers: Wake Retry finding root region
:: DEBUG orgapachehadoophbasezookeeperZooKeeperWrapper: Read ZNode /hbase/rootregionserver got :
:: INFO orgapachehadoopipcHbaseRPC: Server at /: could not be reached after tries giving up
:: DEBUG orgapachehadoophbaseclientHConnectionManager$TableServers: locateRegionInMeta attempt of failed; retrying after sleep of
orgapachehadoophbaseclientRetriesExhaustedException: Failed setting up proxy to /: after attempts=
at orgapachehadoophbaseipcHBaseRPCwaitForProxy(HBaseRPCjava:)
at orgapachehadoophbaseclientHConnectionManager$TableServersgetHRegionConnection(HConnectionManagerjava:)
at orgapachehadoophbaseclientHConnectionManager$TableServersgetHRegionConnection(HConnectionManagerjava:)
at orgapachehadoophbaseclientHConnectionManager$TableServerslocateRootRegion(HConnectionManagerjava:)
at orgapachehadoophbaseclientHConnectionManager$TableServerslocateRegion(HConnectionManagerjava:)
at orgapachehadoophbaseclientHConnectionManager$TableServerslocateRegion(HConnectionManagerjava:)
at orgapachehadoophbaseclientHConnectionManager$TableServerslocateRegionInMeta(HConnectionManagerjava:)
at orgapachehadoophbaseclientHConnectionManager$TableServerslocateRegion(HConnectionManagerjava:)
at orgapachehadoophbaseclientHConnectionManager$TableServerslocateRegion(HConnectionManagerjava:)
at orgapachehadoophbaseclientHConnectionManager$TableServersgetRegionLocation(HConnectionManagerjava:)
at orgapachehadoophbaseclientServerCallableinstantiateServer(ServerCallablejava:)
at orgapachehadoophbaseclientScannerCallableinstantiateServer(ScannerCallablejava:)
at orgapachehadoophbaseclientHConnectionManager$TableServersgetRegionServerWithRetries(HConnectionManagerjava:)
at orgapachehadoophbaseclientaScan(MetaScannerjava:)
at orgapachehadoophbaseclientaScan(MetaScannerjava:)
at orgapachehadoophbaseclientHConnectionManager$TableServerslistTables(HConnectionManagerjava:)
at orgapachehadoophbaseclientHBaseAdminlistTables(HBaseAdminjava:)
at orgapachehadoophbasegeneratedmastermaster_jsp_jspService(master_jspjava:)
at orgapachejasperruntimeHttpJspBaseservice(HttpJspBasejava:)
at javaxservlet(HttpServletjava:)
at orgmortbayjettyservletServletHolderhandle(ServletHolderjava:)
at orgmortbayjettyservletServletHandlerhandle(ServletHandlerjava:)
at orgmortbayjettysecuritySecurityHandlerhandle(SecurityHandlerjava:)
at orgmortbayjettyservletSessionHandlerhandle(SessionHandlerjava:)
at orgmortbayjettyhandlerContextHandlerhandle(ContextHandlerjava:)
at orgmortbayjettywebappWebAppContexthandle(WebAppContextjava:)
at orgmortbayjettyhandlerContextHandlerCollectionhandle(ContextHandlerCollectionjava:)
at orgmortbayjettyhandlerHandlerWrapperhandle(HandlerWrapperjava:)
at orgmortbayjettyServerhandle(Serverjava:)
at orgmortbayjettyHttpConnectionhandleRequest(HttpConnectionjava:)
at orgmortbayjettyHttpConnection$RequestHandlerheaderComplete(HttpConnectionjava:)
at orgmortbayjettyHttpParserparseNext(HttpParserjava:)
at orgmortbayjettyHttpParserparseAvailable(HttpParserjava:)
at orgmortbayjettyHttpConnectionhandle(HttpConnectionjava:)
at orgmortbayionioSelectChannelEndPointrun(SelectChannelEndPointjava:)
at orgmortbaythreadQueuedThreadPool$PoolThreadrun(QueuedThreadPooljava:)
:: DEBUG orgapachehadoophbasezookeeperZooKeeperWrapper: Read ZNode /hbase/rootregionserver got :
:: INFO orgapachehadoopipcHbaseRPC: Server at /: could not be reached after tries giving up
:: DEBUG orgapachehadoophbaseclientHConnectionManager$TableServers: Root region location changed Sleeping
:: DEBUG orgapachehadoophbaseclientHConnectionManager$TableServers: Wake Retry finding root region
:: DEBUG orgapachehadoophbasezookeeperZooKeeperWrapper: Read ZNode /hbase/rootregionserver got :
:: INFO orgapachehadoopipcHbaseRPC: Server at /: could not be reached after tries giving up
:: DEBUG orgapachehadoophbaseclientHConnectionManager$TableServers: Root region location changed Sleeping
:: DEBUG orgapachehadoophbaseclientHConnectionManager$TableServers: Wake Retry finding root region
:: DEBUG orgapachehadoophbasezookeeperZooKeeperWrapper: Read ZNode /hbase/rootregionserver got :
:: INFO orgapachehadoopipcHbaseRPC: Server at /: could not be reached after tries giving up
:: DEBUG orgapachehadoophbasezookeeperZooKeeperWrapper: Read ZNode /hbase/rootregionserver got :
:: INFO orgapachehadoopipcHbaseRPC: Server at /: could not be reached after tries giving up
:: DEBUG orgapachehadoophbaseclientHConnectionManager$TableServers: Root region location changed Sleeping
:: DEBUG orgapachehadoophbaseclientHConnectionManager$TableServers: Wake Retry finding root region
:: DEBUG orgapachehadoophbasezookeeperZooKeeperWrapper: Read ZNode /hbase/rootregionserver got :
:: INFO orgapachehadoopipcHbaseRPC: Server at /: could not be reached after tries giving up
:: DEBUG orgapachehadoophbaseclientHConnectionManager$TableServers: Root region location changed Sleeping
:: DEBUG orgapachehadoophbaseclientHConnectionManager$TableServers: Wake Retry finding root region
:: DEBUG orgapachehadoophbasezookeeperZooKeeperWrapper: Read ZNode /hbase/rootregionserver got :
:: INFO orgapachehadoopipcHbaseRPC: Server at /: could not be reached after tries giving up
:: DEBUG orgapachehadoophbaseclientHConnectionManager$TableServers: locateRegionInMeta attempt of failed; retrying after sleep of
orgapachehadoophbaseclientRetriesExhaustedException: Failed setting up proxy to /: after attempts=
at orgapachehadoophbaseipcHBaseRPCwaitForProxy(HBaseRPCjava:)
at orgapachehadoophbaseclientHConnectionManager$TableServersgetHRegionConnection(HConnectionManagerjava:)
at orgapachehadoophbaseclientHConnectionManager$TableServersgetHRegionConnection(HConnectionManagerjava:)
at orgapachehadoophbaseclientHConnectionManager$TableServerslocateRootRegion(HConnectionManagerjava:)
at orgapachehadoophbaseclientHConnectionManager$TableServerslocateRegion(HConnectionManagerjava:)
at orgapachehadoophbaseclientHConnectionManager$TableServerslocateRegion(HConnectionManagerjava:)
at orgapachehadoophbaseclientHConnectionManager$TableServerslocateRegionInMeta(HConnectionManagerjava:)
at orgapachehadoophbaseclientHConnectionManager$TableServerslocateRegion(HConnectionManagerjava:)
at orgapachehadoophbaseclientHConnectionManager$TableServersrelocateRegion(HConnectionManagerjava:)
at orgapachehadoophbaseclientHConnectionManager$TableServersgetRegionLocation(HConnectionManagerjava:)
at orgapachehadoophbaseclientServerCallableinstantiateServer(ServerCallablejava:)
at orgapachehadoophbaseclientScannerCallableinstantiateServer(ScannerCallablejava:)
at orgapachehadoophbaseclientHConnectionManager$TableServersgetRegionServerWithRetries(HConnectionManagerjava:)
at orgapachehadoophbaseclientaScan(MetaScannerjava:)
at orgapachehadoophbaseclientaScan(MetaScannerjava:)
at orgapachehadoophbaseclientHConnectionManager$TableServerslistTables(HConnectionManagerjava:)
at orgapachehadoophbaseclientHBaseAdminlistTables(HBaseAdminjava:)
at orgapachehadoophbasegeneratedmastermaster_jsp_jspService(master_jspjava:)
at orgapachejasperruntimeHttpJspBaseservice(HttpJspBasejava:)
at javaxservlet(HttpServletjava:)
at orgmortbayjettyservletServletHolderhandle(ServletHolderjava:)
at orgmortbayjettyservletServletHandlerhandle(ServletHandlerjava:)
at orgmortbayjettysecuritySecurityHandlerhandle(SecurityHandlerjava:)
at orgmortbayjettyservletSessionHandlerhandle(SessionHandlerjava:)
at orgmortbayjettyhandlerContextHandlerhandle(ContextHandlerjava:)
at orgmortbayjettywebappWebAppContexthandle(WebAppContextjava:)
at orgmortbayjettyhandlerContextHandlerCollectionhandle(ContextHandlerCollectionjava:)
at orgmortbayjettyhandlerHandlerWrapperhandle(HandlerWrapperjava:)
at orgmortbayjettyServerhandle(Serverjava:)
at orgmortbayjettyHttpConnectionhandleRequest(HttpConnectionjava:)
at orgmortbayjettyHttpConnection$RequestHandlerheaderComplete(HttpConnectionjava:)
at orgmortbayjettyHttpParserparseNext(HttpParserjava:)
at orgmortbayjettyHttpParserparseAvailable(HttpParserjava:)
at orgmortbayjettyHttpConnectionhandle(HttpConnectionjava:)
at orgmortbayionioSelectChannelEndPointrun(SelectChannelEndPointjava:)
at orgmortbaythreadQueuedThreadPool$PoolThreadrun(QueuedThreadPooljava:)
:: DEBUG orgapachehadoophbasezookeeperZooKeeperWrapper: Read ZNode /hbase/rootregionserver got :
:: INFO orgapachehadoopipcHbaseRPC: Server at /: could not be reached after tries giving up
:: DEBUG orgapachehadoophbaseclientHConnectionManager$TableServers: Root region location changed Sleeping
:: DEBUG orgapachehadoophbaseclientHConnectionManager$TableServers: Wake Retry finding root region
:: DEBUG orgapachehadoophbasezookeeperZooKeeperWrapper: Read ZNode /hbase/rootregionserver got :
:: INFO orgapachehadoopipcHbaseRPC: Server at /: could not be reached after tries giving up
:: DEBUG orgapachehadoophbaseclientHConnectionManager$TableServers: Root region location changed Sleeping
:: DEBUG orgapachehadoophbaseclientHConnectionManager$TableServers: Wake Retry finding root region
:: DEBUG orgapachehadoophbasezookeeperZooKeeperWrapper: Read ZNode /hbase/rootregionserver got :
:: INFO orgapachehadoopipcHbaseRPC: Server at /: could not be reached after tries giving up
:: DEBUG orgapachehadoophbasezookeeperZooKeeperWrapper: Read ZNode /hbase/rootregionserver got :
:: INFO orgapachehadoopipcHbaseRPC: Server at /: could not be reached after tries giving up
:: DEBUG orgapachehadoophbaseclientHConnectionManager$TableServers: Root region location changed Sleeping
:: DEBUG orgapachehadoophbaseclientHConnectionManager$TableServers: Wake Retry finding root region
:: DEBUG orgapachehadoophbasezookeeperZooKeeperWrapper: Read ZNode /hbase/rootregionserver got :
:: INFO orgapachehadoopipcHbaseRPC: Server at /: could not be reached after tries giving up
:: DEBUG orgapachehadoophbaseclientHConnectionManager$TableServers: Root region location changed Sleeping
:: DEBUG orgapachehadoophbaseclientHConnectionManager$TableServers: Wake Retry finding root region
:: DEBUG orgapachehadoophbasezookeeperZooKeeperWrapper: Read ZNode /hbase/rootregionserver got :
:: INFO orgapachehadoopipcHbaseRPC: Server at /: could not be reached after tries giving up
:: DEBUG orgapachehadoophbaseclientHConnectionManager$TableServers: locateRegionInMeta attempt of failed; retrying after sleep of
orgapachehadoophbaseclientRetriesExhaustedException: Failed setting up proxy to /: after attempts=
at orgapachehadoophbaseipcHBaseRPCwaitForProxy(HBaseRPCjava:)
at orgapachehadoophbaseclientHConnectionManager$TableServersgetHRegionConnection(HConnectionManagerjava:)
at orgapachehadoophbaseclientHConnectionManager$TableServersgetHRegionConnection(HConnectionManagerjava:)
at orgapachehadoophbaseclientHConnectionManager$TableServerslocateRootRegion(HConnectionManagerjava:)
at orgapachehadoophbaseclientHConnectionManager$TableServerslocateRegion(HConnectionManagerjava:)
at orgapachehadoophbaseclientHConnectionManager$TableServerslocateRegion(HConnectionManagerjava:)
at orgapachehadoophbaseclientHConnectionManager$TableServerslocateRegionInMeta(HConnectionManagerjava:)
at orgapachehadoophbaseclientHConnectionManager$TableServerslocateRegion(HConnectionManagerjava:)
at orgapachehadoophbaseclientHConnectionManager$TableServersrelocateRegion(HConnectionManagerjava:)
at orgapachehadoophbaseclientHConnectionManager$TableServersgetRegionLocation(HConnectionManagerjava:)
at orgapachehadoophbaseclientServerCallableinstantiateServer(ServerCallablejava:)
at orgapachehadoophbaseclientScannerCallableinstantiateServer(ScannerCallablejava:)
at orgapachehadoophbaseclientHConnectionManager$TableServersgetRegionServerWithRetries(HConnectionManagerjava:)
at orgapachehadoophbaseclientaScan(MetaScannerjava:)
at orgapachehadoophbaseclientaScan(MetaScannerjava:)
at orgapachehadoophbaseclientHConnectionManager$TableServerslistTables(HConnectionManagerjava:)
at orgapachehadoophbaseclientHBaseAdminlistTables(HBaseAdminjava:)
at orgapachehadoophbasegeneratedmastermaster_jsp_jspService(master_jspjava:)
at orgapachejasperruntimeHttpJspBaseservice(HttpJspBasejava:)
at javaxservlet(HttpServletjava:)
at orgmortbayjettyservletServletHolderhandle(ServletHolderjava:)
at orgmortbayjettyservletServletHandlerhandle(ServletHandlerjava:)
at orgmortbayjettysecuritySecurityHandlerhandle(SecurityHandlerjava:)
at orgmortbayjettyservletSessionHandlerhandle(SessionHandlerjava:)
at orgmortbayjettyhandlerContextHandlerhandle(ContextHandlerjava:)
at orgmortbayjettywebappWebAppContexthandle(WebAppContextjava:)
at orgmortbayjettyhandlerContextHandlerCollectionhandle(ContextHandlerCollectionjava:)
at orgmortbayjettyhandlerHandlerWrapperhandle(HandlerWrapperjava:)
at orgmortbayjettyServerhandle(Serverjava:)
at orgmortbayjettyHttpConnectionhandleRequest(HttpConnectionjava:)
at orgmortbayjettyHttpConnection$RequestHandlerheaderComplete(HttpConnectionjava:)
at orgmortbayjettyHttpParserparseNext(HttpParserjava:)
at orgmortbayjettyHttpParserparseAvailable(HttpParserjava:)
at orgmortbayjettyHttpConnectionhandle(HttpConnectionjava:)
at orgmortbayionioSelectChannelEndPointrun(SelectChannelEndPointjava:)
at orgmortbaythreadQueuedThreadPool$PoolThreadrun(QueuedThreadPooljava:)
:: DEBUG orgapachehadoophbasezookeeperZooKeeperWrapper: Read ZNode /hbase/rootregionserver got :
:: INFO orgapachehadoopipcHbaseRPC: Server at /: could not be reached after tries giving up
:: DEBUG orgapachehadoophbaseclientHConnectionManager$TableServers: Root region location changed Sleeping
:: DEBUG orgapachehadoophbaseclientHConnectionManager$TableServers: Wake Retry finding root region
:: DEBUG orgapachehadoophbasezookeeperZooKeeperWrapper: Read ZNode /hbase/rootregionserver got :
:: INFO orgapachehadoopipcHbaseRPC: Server at /: could not be reached after tries giving up
:: DEBUG orgapachehadoophbaseclientHConnectionManager$TableServers: Root region location changed Sleeping
:: INFO orgapachehadoophbasemasterServerManager: region servers dead average load
:: DEBUG orgapachehadoophbaseclientHConnectionManager$TableServers: Wake Retry finding root region
:: DEBUG orgapachehadoophbasezookeeperZooKeeperWrapper: Read ZNode /hbase/rootregionserver got :
:: INFO orgapachehadoopipcHbaseRPC: Server at /: could not be reached after tries giving up
:: INFO orgapachehadoophbasemasterBaseScanner: RegionManagerrootScanner scanning meta region {server: : regionname: ROOT startKey: <>}
:: INFO orgapachehadoophbasemasterBaseScanner: All META region(s) scanned
:: INFO orgapachehadoopipcHbaseRPC: Server at /: could not be reached after tries giving up
:: WARN orgapachehadoophbasemasterBaseScanner: Scan ROOT region
orgapachehadoophbaseclientRetriesExhaustedException: Failed setting up proxy to /: after attempts=
at orgapachehadoophbaseipcHBaseRPCwaitForProxy(HBaseRPCjava:)
at orgapachehadoophbaseclientHConnectionManager$TableServersgetHRegionConnection(HConnectionManagerjava:)
at orgapachehadoophbaseclientHConnectionManager$TableServersgetHRegionConnection(HConnectionManagerjava:)
at orgapachehadoophbasemasterBaseScannerscanRegion(BaseScannerjava:)
at orgapachehadoophbasemasterRootScannerscanRoot(RootScannerjava:)
at orgapachehadoophbasemasterRootScannermaintenanceScan(RootScannerjava:)
at orgapachehadoophbasemasterBaseScannerchore(BaseScannerjava:)
at orgapachehadoophbaseChorerun(Chorejava:)
:: INFO orgapachehadoophbasemasterServerManager: region servers dead average load
:: INFO orgapachehadoophbasemasterBaseScanner: RegionManagerrootScanner scanning meta region {server: : regionname: ROOT startKey: <>}
:: INFO orgapachehadoopipcHbaseRPC: Server at /: could not be reached after tries giving up
:: WARN orgapachehadoophbasemasterBaseScanner: Scan ROOT region
orgapachehadoophbaseclientRetriesExhaustedException: Failed setting up proxy to /: after attempts=
at orgapachehadoophbaseipcHBaseRPCwaitForProxy(HBaseRPCjava:)
at orgapachehadoophbaseclientHConnectionManager$TableServersgetHRegionConnection(HConnectionManagerjava:)
at orgapachehadoophbaseclientHConnectionManager$TableServersgetHRegionConnection(HConnectionManagerjava:)
at orgapachehadoophbasemasterBaseScannerscanRegion(BaseScannerjava:)
at orgapachehadoophbasemasterRootScannerscanRoot(RootScannerjava:)
at orgapachehadoophbasemasterRootScannermaintenanceScan(RootScannerjava:)
at orgapachehadoophbasemasterBaseScannerchore(BaseScannerjava:)
at orgapachehadoophbaseChorerun(Chorejava:)
:: INFO orgapachehadoophbasemasterBaseScanner: All META region(s) scanned
:: INFO orgapachehadoophbasemasterServerManager: region servers dead average load
:: INFO orgapachehadoophbasemasterBaseScanner: RegionManagerrootScanner scanning meta region {server: : regionname: ROOT startKey: <>}
:: INFO orgapachehadoopipcHbaseRPC: Server at /: could not be reached after tries giving up
:: WARN orgapachehadoophbasemasterBaseScanner: Scan ROOT region
orgapachehadoophbaseclientRetriesExhaustedException: Failed setting up proxy to /: after attempts=
at orgapachehadoophbaseipcHBaseRPCwaitForProxy(HBaseRPCjava:)
at orgapachehadoophbaseclientHConnectionManager$TableServersgetHRegionConnection(HConnectionManagerjava:)
at orgapachehadoophbaseclientHConnectionManager$TableServersgetHRegionConnection(HConnectionManagerjava:)
at orgapachehadoophbasemasterBaseScannerscanRegion(BaseScannerjava:)
at orgapachehadoophbasemasterRootScannerscanRoot(RootScannerjava:)
at orgapachehadoophbasemasterRootScannermaintenanceScan(RootScannerjava:)
at orgapachehadoophbasemasterBaseScannerchore(BaseScannerjava:)
at orgapachehadoophbaseChorerun(Chorejava:)
此時Hbase能夠進入shell但是無法執行任何操作於是再次試圖關閉hbase的時候卻發現無法停止master那個"stop master"後面的“”出現了許許多多仍然無法停止master節點於是我們不得不強制kill掉masterHbase就這樣掛掉了
Hbase重啟之後就無法啟動的原因分析
經過多方排查最後我在嘗試使用netstat an查看端口占用情況的時候發現
在WAMDM節點上regionserver占用的端口占用為:
而在WAMDM節點上 regionserver占用的端口占用為:
我感覺頗為蹊跷之後便檢查/etc/hosts文件果然發現在WAMDM和WAMDM下的hosts文件不同在WAMDM的hosts文件中的內容為
代碼如下 復制代碼
WAMDM localhostlocaldomain localhost
WAMDM WAMDM
WAMDM WAMDM
WAMDM WAMDM
WAMDM WAMDM
WAMDM WAMDM
大家注意第一行我們在使用配置Hadoop/Hbase的時候常常使用主機名來代替IP使用但是在WAMDM的機器上WAMDM被映射為於是master和regionserver之間的通信就出錯這也就是為什麼我們經常在日志中以及錯誤提示中看到如下信息
代碼如下 復制代碼
Server at /: could not be reached after tries giving up
orgapachehadoophbaseclientRetriesExhaustedException: Failed setting up proxy to /: after attempts=
at orgapachehadoophbaseipcHBaseRPCwaitForProxy(HBaseRPCjava:)
at orgapachehadoophbaseclientHConnectionManager$TableServersgetHRegionConnection(HConnectionManagerjava:)
at orgapachehadoophbaseclientHConnectionManager$TableServersgetHRegionConnection(HConnectionManagerjava:)
at orgapachehadoophbasemasterBaseScannerscanRegion(BaseScannerjava:)
at orgapachehadoophbasemasterRootScannerscanRoot(RootScannerjava:)
at orgapachehadoophbasemasterRootScannermaintenanceScan(RootScannerjava:)
at orgapachehadoophbasemasterBaseScannerchore(BaseScannerjava:)
at orgapachehadoophbaseChorerun(Chorejava:)
Hbase重啟之後就無法啟動的解決辦法
於是我將所有機器的/etc/hosts文件都改為了如下配置
代碼如下 復制代碼
localhost
WAMDM WAMDM
WAMDM WAMDM
WAMDM WAMDM
WAMDM WAMDM
WAMDM WAMDM
# The following lines are desirable for IPv capable hosts
:: localhost iplocalhost iploopback
fe:: iplocalnet
ff:: ipmcastprefix
ff:: ipallnodes
ff:: ipallrouters
ff:: ipallhosts
同時為了保險起見我在stophbasesh中也加入停止regionservers的命令（雖然在網上查不到停止regionservers的必要性以及這個腳本存在bug的證據但是這樣改經過測試是沒有問題的）
代碼如下 復制代碼
"$bin"/hbasedaemonssh config "${HBASE_CONF_DIR}" hosts "${HBASE_REGIONSERVERS}" stop regionserver
這個命令需要加在stop master之前至於不加入這個停止regionservers的命令是否可行在未來我會進一步測試
通過以上修改出現兩個的Regionserver的問題以及Hbase重啟就掛的問題得到徹底解決！
From:http://tw.wingwit.com/Article/program/Java/gj/201405/30822.html
