HBase 错误之 NoClassDefFoundError：ProtobufUtil

发表于 | 分类于
字数统计 3,200字 | 阅读时长 ≈ 18分钟

背景说明：通过 dubbo 部署一个服务，服务中的业务逻辑会查询 HBase 表的数据，但是 dubbo 服务在初始化注册时，HBase 初始化的过程中会报错：

1
java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hbase.protobuf.ProtobufUtil

本文涉及的开发环境，基于 HBase v1.1.2、Zookeeper v3.4.6、dubbo v2.8.4、Hadoop v2.7.1。

问题出现

通过 k8s 多节点发布服务，但是只有在某一台机器上面出现错误【其它节点日志显示正常，也可以提供正常的服务】，发布后 dubbo 服务注册初始化时出现的错误如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
2019-09-19_18:03:49 [http-nio-28956-exec-2-SendThread (192.168.20.101:2181)] INFO zookeeper.ClientCnxn:852: Socket connection established to 192.168.20.101/192.168.20.101:2181, initiating session
2019-09-19_18:03:49 [http-nio-28956-exec-2-SendThread (192.168.20.101:2181)] INFO zookeeper.ClientCnxn:1235: Session establishment complete on server 192.168.20.101/192.168.20.101:2181, sessionid = 0x36af032f505e830, negotiated timeout = 90000
2019-09-19_18:03:50 [http-nio-28956-exec-2] WARN hdfs.DFSUtil:689: Namenode for hdfs-cluster remains unresolved for ID nn1.  Check your hdfs-site.xml file to ensure namenodes are configured properly.
2019-09-19_18:03:50 [http-nio-28956-exec-9] ERROR filter.ExceptionFilter:87:  [DUBBO] Got unchecked and undeclared exception which called by 10.200.0.2. service: com.yyy.zzz.service.es.weibo.IXxxService, method: search, exception: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hbase.protobuf.ProtobufUtil, dubbo version: 2.8.4, current host: 127.0.0.1
java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hbase.protobuf.ProtobufUtil
	at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.getMetaRegionState (MetaTableLocator.java:482)
	at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.getMetaRegionLocation (MetaTableLocator.java:167)
	at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable (MetaTableLocator.java:598)
	at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable (MetaTableLocator.java:579)
	at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable (MetaTableLocator.java:558)
	at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation (ZooKeeperRegistry.java:61)
	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta (ConnectionManager.java:1192)
	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion (ConnectionManager.java:1159)
	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion (ConnectionManager.java:1133)
	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta (ConnectionManager.java:1338)
	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion (ConnectionManager.java:1162)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.findAllLocationsOrFail (AsyncProcess.java:940)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction (AsyncProcess.java:857)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$100 (AsyncProcess.java:575)
	at org.apache.hadoop.hbase.client.AsyncProcess.submitAll (AsyncProcess.java:557)
	at org.apache.hadoop.hbase.client.HTable.batch (HTable.java:933)
	at org.apache.hadoop.hbase.client.HTable.batch (HTable.java:950)
	at org.apache.hadoop.hbase.client.HTable.get (HTable.java:911)
	at com.yyy.zzz.commons.search.reader.hbase.BaseHBaseReader.batchGet (BaseHBaseReader.java:94)
	at com.yyy.zzz.commons.search.reader.hbase.weibo.WeiboContentHbaseReader.batchGet (WeiboContentHbaseReader.java:98)
	at com.yyy.zzz.commons.search.searcher.AbstractBaseSearcher.getContent (AbstractBaseSearcher.java:269)
	at com.yyy.zzz.commons.search.searcher.AbstractBaseSearcher.getInfo (AbstractBaseSearcher.java:188)
	at com.yyy.zzz.runner.search.BaseSearchRunner.search (BaseSearchRunner.java:89)
	at com.yyy.zzz.api.weibo.WeiboContentServiceImpl.search (WeiboContentServiceImpl.java:33)
	at com.alibaba.dubbo.common.bytecode.Wrapper3.invokeMethod (Wrapper3.java)
	at com.alibaba.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke (JavassistProxyFactory.java:46)
	at com.alibaba.dubbo.rpc.proxy.AbstractProxyInvoker.invoke (AbstractProxyInvoker.java:72)
	at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke (InvokerWrapper.java:53)
	at com.alibaba.dubbo.rpc.filter.ExceptionFilter.invoke (ExceptionFilter.java:64)
	at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke (ProtocolFilterWrapper.java:91)
	at com.alibaba.dubbo.monitor.support.MonitorFilter.invoke (MonitorFilter.java:75)
	at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke (ProtocolFilterWrapper.java:91)
	at com.alibaba.dubbo.rpc.filter.TimeoutFilter.invoke (TimeoutFilter.java:42)
	at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke (ProtocolFilterWrapper.java:91)
	at com.alibaba.dubbo.rpc.protocol.dubbo.filter.TraceFilter.invoke (TraceFilter.java:78)
	at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke (ProtocolFilterWrapper.java:91)
	at com.alibaba.dubbo.rpc.filter.ContextFilter.invoke (ContextFilter.java:70)
	at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke (ProtocolFilterWrapper.java:91)
	at com.alibaba.dubbo.rpc.filter.GenericFilter.invoke (GenericFilter.java:132)
	at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke (ProtocolFilterWrapper.java:91)
	at com.alibaba.dubbo.rpc.filter.ClassLoaderFilter.invoke (ClassLoaderFilter.java:38)
	at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke (ProtocolFilterWrapper.java:91)
	at com.alibaba.dubbo.rpc.filter.EchoFilter.invoke (EchoFilter.java:38)
	at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke (ProtocolFilterWrapper.java:91)
	at com.alibaba.dubbo.rpc.proxy.InvokerInvocationHandler.invoke (InvokerInvocationHandler.java:52)
	at com.alibaba.dubbo.common.bytecode.proxy1.search (proxy1.java)
	at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke (Method.java:498)
	at org.jboss.resteasy.core.MethodInjectorImpl.invoke (MethodInjectorImpl.java:137)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget (ResourceMethodInvoker.java:288)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invoke (ResourceMethodInvoker.java:242)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invoke (ResourceMethodInvoker.java:229)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke (SynchronousDispatcher.java:356)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke (SynchronousDispatcher.java:179)
	at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service (ServletContainerDispatcher.java:220)
	at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service (HttpServletDispatcher.java:56)
	at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service (HttpServletDispatcher.java:51)
	at javax.servlet.http.HttpServlet.service (HttpServlet.java:790)
	at com.alibaba.dubbo.rpc.protocol.rest.DubboHttpServer$RestHandler.handle (DubboHttpServer.java:86)
	at com.alibaba.dubbo.remoting.http.servlet.DispatcherServlet.service (DispatcherServlet.java:64)
	at javax.servlet.http.HttpServlet.service (HttpServlet.java:790)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter (ApplicationFilterChain.java:291)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter (ApplicationFilterChain.java:206)
	at org.apache.catalina.core.StandardWrapperValve.invoke (StandardWrapperValve.java:219)
	at org.apache.catalina.core.StandardContextValve.invoke (StandardContextValve.java:106)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke (AuthenticatorBase.java:504)
	at org.apache.catalina.core.StandardHostValve.invoke (StandardHostValve.java:142)
	at org.apache.catalina.valves.ErrorReportValve.invoke (ErrorReportValve.java:79)
	at org.apache.catalina.core.StandardEngineValve.invoke (StandardEngineValve.java:88)
	at org.apache.catalina.connector.CoyoteAdapter.service (CoyoteAdapter.java:534)
	at org.apache.coyote.http11.AbstractHttp11Processor.process (AbstractHttp11Processor.java:1081)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process (AbstractProtocol.java:658)
	at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process (Http11NioProtocol.java:222)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun (NioEndpoint.java:1566)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run (NioEndpoint.java:1523)
	at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run (TaskThread.java:61)
	at java.lang.Thread.run (Thread.java:748)

注意查看重点的内容：

1
2
2019-09-19_18:03:50 [http-nio-28956-exec-2] WARN hdfs.DFSUtil:689: Namenode for hdfs-cluster remains unresolved for ID nn1.  Check your hdfs-site.xml file to ensure namenodes are configured properly.
java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hbase.protobuf.ProtobufUtil

第一行是 hdfs 无法解析 HA 的域名，应该是系统环境问题；第二行是 HBase 初始化环境失败，看起来像是缺失依赖包或者依赖包冲突导致的 NoClassDefFoundError。

同时还出现了未知主机名异常：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
com.yyy.zzz.exception.es.EsConnException: java.net.UnknownHostException: host40: Temporary failure in name resolution
	at com.yyy.zzz.commons.infrastructure.client.EsClient.<init>(EsClient.java:46)
	at com.yyy.zzz.commons.infrastructure.client.EsClient.getInstance (EsClient.java:57)
	at com.yyy.zzz.commons.search.searcher.AbstractBaseSearcher.<init>(AbstractBaseSearcher.java:69)
	at com.yyy.zzz.commons.search.searcher.weibo.WeiboContentSearcher.<init>(WeiboContentSearcher.java:14)
	at com.yyy.zzz.commons.search.searcher.weibo.WeiboContentSearcher.getInstance (WeiboContentSearcher.java:22)
	at com.yyy.zzz.runner.search.weibo.WeiboContentSearchRunner.<init>(WeiboContentSearchRunner.java:26)
	at com.yyy.zzz.runner.search.weibo.WeiboContentSearchRunner.<init>(WeiboContentSearchRunner.java:20)
	at com.yyy.zzz.api.weibo.WeiboContentServiceImpl.search (WeiboContentServiceImpl.java:32)
	at com.alibaba.dubbo.common.bytecode.Wrapper3.invokeMethod (Wrapper3.java)
	at com.alibaba.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke (JavassistProxyFactory.java:46)
	at com.alibaba.dubbo.rpc.proxy.AbstractProxyInvoker.invoke (AbstractProxyInvoker.java:72)
	at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke (InvokerWrapper.java:53)
	at com.alibaba.dubbo.rpc.filter.ExceptionFilter.invoke (ExceptionFilter.java:64)
	at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke (ProtocolFilterWrapper.java:91)
	at com.alibaba.dubbo.monitor.support.MonitorFilter.invoke (MonitorFilter.java:75)
	at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke (ProtocolFilterWrapper.java:91)
	at com.alibaba.dubbo.rpc.filter.TimeoutFilter.invoke (TimeoutFilter.java:42)
	at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke (ProtocolFilterWrapper.java:91)
	at com.alibaba.dubbo.rpc.protocol.dubbo.filter.TraceFilter.invoke (TraceFilter.java:78)
	at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke (ProtocolFilterWrapper.java:91)
	at com.alibaba.dubbo.rpc.filter.ContextFilter.invoke (ContextFilter.java:70)
	at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke (ProtocolFilterWrapper.java:91)
	at com.alibaba.dubbo.rpc.filter.GenericFilter.invoke (GenericFilter.java:132)
	at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke (ProtocolFilterWrapper.java:91)
	at com.alibaba.dubbo.rpc.filter.ClassLoaderFilter.invoke (ClassLoaderFilter.java:38)
	at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke (ProtocolFilterWrapper.java:91)
	at com.alibaba.dubbo.rpc.filter.EchoFilter.invoke (EchoFilter.java:38)
	at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke (ProtocolFilterWrapper.java:91)
	at com.alibaba.dubbo.rpc.proxy.InvokerInvocationHandler.invoke (InvokerInvocationHandler.java:52)
	at com.alibaba.dubbo.common.bytecode.proxy1.search (proxy1.java)
	at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke (Method.java:498)
	at org.jboss.resteasy.core.MethodInjectorImpl.invoke (MethodInjectorImpl.java:137)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget (ResourceMethodInvoker.java:288)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invoke (ResourceMethodInvoker.java:242)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invoke (ResourceMethodInvoker.java:229)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke (SynchronousDispatcher.java:356)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke (SynchronousDispatcher.java:179)
	at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service (ServletContainerDispatcher.java:220)
	at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service (HttpServletDispatcher.java:56)
	at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service (HttpServletDispatcher.java:51)
	at javax.servlet.http.HttpServlet.service (HttpServlet.java:790)
	at com.alibaba.dubbo.rpc.protocol.rest.DubboHttpServer$RestHandler.handle (DubboHttpServer.java:86)
	at com.alibaba.dubbo.remoting.http.servlet.DispatcherServlet.service (DispatcherServlet.java:64)
	at javax.servlet.http.HttpServlet.service (HttpServlet.java:790)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter (ApplicationFilterChain.java:291)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter (ApplicationFilterChain.java:206)
	at org.apache.catalina.core.StandardWrapperValve.invoke (StandardWrapperValve.java:219)
	at org.apache.catalina.core.StandardContextValve.invoke (StandardContextValve.java:106)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke (AuthenticatorBase.java:504)
	at org.apache.catalina.core.StandardHostValve.invoke (StandardHostValve.java:142)
	at org.apache.catalina.valves.ErrorReportValve.invoke (ErrorReportValve.java:79)
	at org.apache.catalina.core.StandardEngineValve.invoke (StandardEngineValve.java:88)
	at org.apache.catalina.connector.CoyoteAdapter.service (CoyoteAdapter.java:534)
	at org.apache.coyote.http11.AbstractHttp11Processor.process (AbstractHttp11Processor.java:1081)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process (AbstractProtocol.java:658)
	at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process (Http11NioProtocol.java:222)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun (NioEndpoint.java:1566)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run (NioEndpoint.java:1523)
	at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run (TaskThread.java:61)
	at java.lang.Thread.run (Thread.java:748)
Caused by: java.net.UnknownHostException: host40: Temporary failure in name resolution
	at java.net.Inet6AddressImpl.lookupAllHostAddr (Native Method)
	at java.net.InetAddress$2.lookupAllHostAddr (InetAddress.java:928)
	at java.net.InetAddress.getAddressesFromNameService (InetAddress.java:1323)
	at java.net.InetAddress.getAllByName0 (InetAddress.java:1276)
	at java.net.InetAddress.getAllByName (InetAddress.java:1192)
	at java.net.InetAddress.getAllByName (InetAddress.java:1126)
	at java.net.InetAddress.getByName (InetAddress.java:1076)
	at com.yyy.zzz.commons.infrastructure.client.EsClient.<init>(EsClient.java:43)
	... 64 more

同时，在之后的请求中，只要是转发到这个服务节点的请求，就会出现如下错误：

1
2
3
4
5
6
7
8
9
com.yyy.zzz.exception.hbase.HBaseException: java.lang.reflect.InvocationTargetException
Caused by: java.io.IOException: java.lang.reflect.InvocationTargetException
    at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection (ConnectionFactory.java:240)
    at org.apache.hadoop.hbase.client.ConnectionManager.createConnection (ConnectionManager.java:433)
    at org.apache.hadoop.hbase.client.ConnectionManager.createConnection (ConnectionManager.java:426)
    at org.apache.hadoop.hbase.client.ConnectionManager.getConnectionInternal (ConnectionManager.java:304)
    at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:185)
    at org.apache.hadoop.hbase.client.HTableFactory.createHTableInterface (HTableFactory.java:41)
    ... 18 more

通过排查代码，这个异常是在业务逻辑代码连接 HBase 表取数时出现的：

1
hTableInterface.get (List<Get>)

每一次连接 HBase 取数，都会有这个异常出现。

问题排查

首先怀疑的是 protobuf 版本冲突问题，但是通过对比，发现只有一个确定版本的 jar 包，而且对比其它节点，并没有这个问题出现，最终否定了这个猜测。

接着尝试发送多次请求，查看日志，以下错误不再出现【也很合理，这些异常是在服务注册初始化时只出现一次】：

1
2
java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hbase.protobuf.ProtobufUtil
com.yyy.zzz.exception.es.EsConnException: java.net.UnknownHostException: host40: Temporary failure in name resolution

反而出现的全部是 HBase 取数异常：

1
2
3
4
5
6
7
8
9
com.yyy.zzz.exception.hbase.HBaseException: java.lang.reflect.InvocationTargetException
Caused by: java.io.IOException: java.lang.reflect.InvocationTargetException
    at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection (ConnectionFactory.java:240)
    at org.apache.hadoop.hbase.client.ConnectionManager.createConnection (ConnectionManager.java:433)
    at org.apache.hadoop.hbase.client.ConnectionManager.createConnection (ConnectionManager.java:426)
    at org.apache.hadoop.hbase.client.ConnectionManager.getConnectionInternal (ConnectionManager.java:304)
    at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:185)
    at org.apache.hadoop.hbase.client.HTableFactory.createHTableInterface (HTableFactory.java:41)
    ... 18 more

更神奇的是，只在一台节点上面有问题，其它相同功能的节点没问题。

最终，通过运维排查，从 NoClassDefFoundError 以及 UnknownHostException 发现了异常原因：在某个时间点发布服务时，恰好此时机器负载过高，导致 DNS 解析异常，于是 dubbo 服务在注册时无法获取 hdfs 信息。而 HBase 在初始化时需要依赖 hdfs 上面的某个 hbase.version 文件【用来确定 HBase 的版本】，导致 HBase 在初始化时无法找到这个文件，也就无法确定版本，最终没有加载 ProtobufUtil 类文件。

hdfs-site.xml 配置文件中的重要内容如下，nn1 节点无法被识别：

1
2
3
4
<property>
    <name>dfs.ha.namenodes.hdfs-cluster</name>
    <value>nn1,nn2</value>
</property>

hbase-site.xml 配置文件中的重要内容如下，对于 HBase 来说，这个 hdfs 路径里面存放着重要的信息，如果无法读取它也就无法成功初始化 HBase 环境：

1
2
3
4
<property>
    <name>hbase.rootdir</name>
    <value>hdfs://hdfs-cluster/apps/hbase/data</value>
</property>

所以此后所有的请求需要连接 HBase 取数时，都会出现 java.lang.reflect.InvocationTargetException 异常。

这里会进一步引发一个严重的问题，由于 dubbo 服务在注册时出现问题没有退出，仍旧提供服务，但是这个服务是有问题的，每次需要连接 HBase 取数时都会出现异常，由于没有处理好异常，导致大量的 Zookeeper 连接没有关闭。

进一步导致当前机器的 Zookeeper 连接数接近 1000 个，严重影响了其它业务连接 Zookeeper，一律是等待、超时重试。

问题解决

找到问题原因，就很容易解决了，重启对应的服务，观察初始化日志，一切正常。

虾丸派 wechat
扫一扫添加博主，进技术交流群，共同学习进步
永不止步
0%