大数据|HBase Balancer失败日志报错 ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler 大数据|HBase

现象 HBase shell下面打开Balance并手动执行Balancer很快结束，显示为false（此时balance_switch为true）

hbase(main):007:0* balancer false

【大数据|HBase Balancer失败日志报错 ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler】查看RegionServer日志有如下报错信息，

2020-07-08 16:49:01,821 INFO org.apache.hadoop.hbase.coordination.ZkOpenRegionCoordination: Opening of region {ENCODED => 830a8f32d019fa5122cd533fc68ef1a1, NAME => 'TRAFODION.V7PE RF.KAPB_CPXXHZ,,1579251646163.830a8f32d019fa5122cd533fc68ef1a1.', STARTKEY => '', ENDKEY => ''} failed, transitioning from OPENING to FAILED_OPEN in ZK, expecting version 7 2020-07-08 16:49:01,965 INFO org.apache.hadoop.hbase.regionserver.RSRpcServices: Open TRAFODION.V7PERF.KAPB_CPXXHZ,,1579251646163.830a8f32d019fa5122cd533fc68ef1a1. 2020-07-08 16:49:01,971 ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open of region=TRAFODION.V7PERF.KAPB_CPXXHZ,,1579251646163.830a8f32d019fa5122c d533fc68ef1a1., starting to roll back the global memstore size. java.lang.IllegalStateException: Could not instantiate a region instance. at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:6351) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6652) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6624) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6580) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6531) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:362) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:129) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:6348) ... 10 more Caused by: java.lang.IllegalArgumentException: Need table descriptor at org.apache.hadoop.hbase.regionserver.HRegion.(HRegion.java:657) at org.apache.hadoop.hbase.regionserver.HRegion.(HRegion.java:634) at org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion.(TransactionalRegion.java:70) ... 15 more

解决

hbase shell使用list 'TRAFODION.V7PERF.KAPB_CPXXHZ,'检查表是否存在 --不存在
hadoop fs -ls /hbase/data/default/TRAFODION.V7PERF.KAPB_CPXXHZ检查目录是否存在 --不存在
scan hbase:meta表检查是否有残留数据 --是

scan 'hbase:meta', {STARTROW=>'TRAFODION.V7PERF.KAPB_CPXXHZ', LIMIT=>1} hbase(main):002:0> scan 'hbase:meta', {STARTROW=>'TRAFODION.V7PERF.KAPB_CPXXHZ', LIMIT=>1} ROWCOLUMN+CELL TRAFODION.V7PERF.KAPB_CPXXHZ,,1579251646163.830a8f column=info:regioninfo, timestamp=1579251646633, value=https://www.it610.com/article/{ENCODED => 830a8f32d019fa5122cd533fc68ef1a1, NAME => 'TRAFODION.V7PERF.KAPB_CPXXHZ,,1579251646 32d019fa5122cd533fc68ef1a1.163.830a8f32d019fa5122cd533fc68ef1a1.', STARTKEY => '', ENDKEY => ''} TRAFODION.V7PERF.KAPB_CPXXHZ,,1579251646163.830a8f column=info:seqnumDuringOpen, timestamp=1579251646633, value=https://www.it610.com/x00/x00/x00/x00/x00/x00/x00/x02 32d019fa5122cd533fc68ef1a1. TRAFODION.V7PERF.KAPB_CPXXHZ,,1579251646163.830a8f column=info:server, timestamp=1579251646633, value=hxpernode08.esgyn.cn:60020 32d019fa5122cd533fc68ef1a1. TRAFODION.V7PERF.KAPB_CPXXHZ,,1579251646163.830a8f column=info:serverstartcode, timestamp=1579251646633, value=1579054953133 32d019fa5122cd533fc68ef1a1. 1 row(s) in 0.0150 seconds

delete根据table/rowkey/column删除对应信息

delete 'hbase:meta','TRAFODION.V7PERF.KAPB_CPXXHZ,,1579251646163.830a8f32d019fa5122cd533fc68ef1a1.','info:regioninfo' delete 'hbase:meta','TRAFODION.V7PERF.KAPB_CPXXHZ,,1579251646163.830a8f32d019fa5122cd533fc68ef1a1.','info:seqnumDuringOpen' delete 'hbase:meta','TRAFODION.V7PERF.KAPB_CPXXHZ,,1579251646163.830a8f32d019fa5122cd533fc68ef1a1.','info:server' delete 'hbase:meta','TRAFODION.V7PERF.KAPB_CPXXHZ,,1579251646163.830a8f32d019fa5122cd533fc68ef1a1.','info:serverstartcode'