文章目录
- Pre
- jstack
- Thread dump
- Dead Lock 分析
- 分析代码
- 解决
JVM-11虚拟机性能监控与故障处理工具之【JDK的可视化工具-JConsole】
jstack主要用来查看某个Java进程内的线程堆栈信息 ，建议多杀几次线程dump ,每次都有某个事件，基本没跑了~
用法也很简单
Found one Java-level deadlock:
=============================
"T-ShortConn-4":
waiting to lock monitor 0x00007f87e1d95718 (object 0x00000006cc86c090, a java.lang.Class),
which is held by "T-ShortConn-0"
"T-ShortConn-0":
waiting to lock monitor 0x00007f875c00f188 (object 0x00000006c6f34238, a java.util.concurrent.ConcurrentHashMap),
which is held by "localhost-startStop-1"
"localhost-startStop-1":
waiting to lock monitor 0x00007f87e1d95718 (object 0x00000006cc86c090, a java.lang.Class),
which is held by "T-ShortConn-0"
Java stack information for the threads listed above:
===================================================
"T-ShortConn-4":
at com.artisan.bus.tool.redis.RedisClusterHelper.getClient(RedisClusterHelper.java:38)
- waiting to lock <0x00000006cc86c090> (a java.lang.Class for com.artisan.bus.tool.redis.RedisClusterHelper)
at com.artisan.bus.tool.redis.RedisClusterHelper.listOs(RedisClusterHelper.java:328)
at com.artisan.bus.tool.redis.RedisClusterHelper.lSize(RedisClusterHelper.java:193)
at com.artisan.bus.service.comunication.CommunicationService.send(CommunicationService.java:1922)
at com.artisan.bus.service.comunication.CommunicationService$$FastClassBySpringCGLIB$$cacacc60.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
at com.artisan.bus.service.comunication.CommunicationService$$EnhancerBySpringCGLIB$$8902c097.send(<generated>)
at net.artisan.manager.DataProcessorManager.send(DataProcessorManager.java:850)
at net.artisan.manager.DataProcessorManager.processShortLinkData(DataProcessorManager.java:833)
at net.artisan.common.ShortLinkProcessorThread.run(ShortLinkProcessorThread.java:22)
at java.lang.Thread.run(Thread.java:748)
"T-ShortConn-0":
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:187)
- waiting to lock <0x00000006c6f34238> (a java.util.concurrent.ConcurrentHashMap)
at org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:486)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:432)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:403)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:389)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1002)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:345)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:340)
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1093)
at com.artisan.bus.tool.utils.SpringApplicationContext.getBean(SpringApplicationContext.java:31)
at com.artisan.bus.tool.redis.RedisClusterHelper.getClient(RedisClusterHelper.java:40)
- locked <0x00000006cc86c090> (a java.lang.Class for com.artisan.bus.tool.redis.RedisClusterHelper)
at com.artisan.bus.tool.redis.RedisClusterHelper.listOs(RedisClusterHelper.java:328)
at com.artisan.bus.tool.redis.RedisClusterHelper.lSize(RedisClusterHelper.java:193)
at com.artisan.bus.service.comunication.CommunicationService.send(CommunicationService.java:1922)
at com.artisan.bus.service.comunication.CommunicationService$$FastClassBySpringCGLIB$$cacacc60.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
at com.artisan.bus.service.comunication.CommunicationService$$EnhancerBySpringCGLIB$$8902c097.send(<generated>)
at net.artisan.manager.DataProcessorManager.send(DataProcessorManager.java:850)
at net.artisan.manager.DataProcessorManager.processShortLinkData(DataProcessorManager.java:833)
at net.artisan.common.ShortLinkProcessorThread.run(ShortLinkProcessorThread.java:22)
at java.lang.Thread.run(Thread.java:748)
"localhost-startStop-1":
at com.artisan.bus.tool.redis.RedisClusterHelper.getClient(RedisClusterHelper.java:38)
- waiting to lock <0x00000006cc86c090> (a java.lang.Class for com.artisan.bus.tool.redis.RedisClusterHelper)
at com.artisan.bus.tool.redis.RedisClusterHelper.del(RedisClusterHelper.java:63)
at com.artisan.bus.dao.impl.settings.template.strategy.StrategyTemplateDao.cacheStrategyTemplate(StrategyTemplateDao.java:173)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:136)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
at com.sun.proxy.$Proxy82.cacheStrategyTemplate(Unknown Source)
at com.artisan.bus.service.impl.task.CheckTaskService.cacheStrategyTemplate(CheckTaskService.java:3658)
at com.artisan.bus.service.impl.task.CheckTaskService$$FastClassBySpringCGLIB$$b794226f.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
at com.artisan.bus.service.impl.task.CheckTaskService$$EnhancerBySpringCGLIB$$7b018424.cacheStrategyTemplate(<generated>)
at com.artisan.bus.web.filter.ApplicationConfigue.setApplicationContext(ApplicationConfigue.java:74)
at org.springframework.context.support.ApplicationContextAwareProcessor.invokeAwareInterfaces(ApplicationContextAwareProcessor.java:121)
at org.springframework.context.support.ApplicationContextAwareProcessor.postProcessBeforeInitialization(ApplicationContextAwareProcessor.java:97)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:409)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1620)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
- locked <0x00000006c6f34238> (a java.util.concurrent.ConcurrentHashMap)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:866)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:542)
- locked <0x00000006c6f655a8> (a java.lang.Object)
at org.springframework.web.servlet.FrameworkServlet.configureAndRefreshWebApplicationContext(FrameworkServlet.java:668)
at org.springframework.web.servlet.FrameworkServlet.createWebApplicationContext(FrameworkServlet.java:634)
at org.springframework.web.servlet.FrameworkServlet.createWebApplicationContext(FrameworkServlet.java:682)
at org.springframework.web.servlet.FrameworkServlet.initWebApplicationContext(FrameworkServlet.java:553)
at org.springframework.web.servlet.FrameworkServlet.initServletBean(FrameworkServlet.java:494)
at org.springframework.web.servlet.HttpServletBean.init(HttpServletBean.java:138)
at javax.servlet.GenericServlet.init(GenericServlet.java:158)
at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1269)
- locked <0x00000006c6d54a78> (a org.apache.catalina.core.StandardWrapper)
at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182)
- locked <0x00000006c6d54a78> (a org.apache.catalina.core.StandardWrapper)
at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072)
- locked <0x00000006c6d54a78> (a org.apache.catalina.core.StandardWrapper)
at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660)
- locked <0x00000006c54f5f78> (a org.apache.catalina.core.StandardContext)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
- locked <0x00000006c54f5f78> (a org.apache.catalina.core.StandardContext)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:875)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1260)
at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:2002)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Found 1 deadlock.
jstack 杀出来的线程堆栈， 划重点~~~
Found one Java-level deadlock:
=============================
"T-ShortConn-4":
waiting to lock monitor 0x00007f87e1d95718 (object 0x00000006cc86c090, a java.lang.Class),
which is held by "T-ShortConn-0"
"T-ShortConn-0":
waiting to lock monitor 0x00007f875c00f188 (object 0x00000006c6f34238, a java.util.concurrent.ConcurrentHashMap),
which is held by "localhost-startStop-1"
"localhost-startStop-1":
waiting to lock monitor 0x00007f87e1d95718 (object 0x00000006cc86c090, a java.lang.Class),
which is held by "T-ShortConn-0"
Java stack information for the threads listed above:
===================================================
三个线程 T-ShortConn-4 、 T-ShortConn-0 和 localhost-startStop-1
看括号里面的 object xxxxxxx
- T-ShortConn-4 想去获取lock 【object 0x00000006cc86c090】 , 但这个lock被 T-ShortConn-0持有
- T-ShortConn-0 想去获取lock 【object 0x00000006c6f34238】 , 但这个lock被 localhost-startStop-1持有
- localhost-startStop-1 想去获取lock 【object 0x00000006cc86c090】 , 但这个lock被 T-ShortConn-0持有
====》得出结论
- T-ShortConn-0 持有的lock 【0x00000006cc86c090】 被 T-ShortConn-4 和 localhost-startStop-1 需要
- localhost-startStop-1 持有的lock 【0x00000006c6f34238】 被 T-ShortConn-0 需要
这不就产生deal lock了么？
T-ShortConn-0 持有的lock 【0x00000006cc86c090】 那就去dump中查找这个关键字吧
at com.artisan.bus.tool.utils.SpringApplicationContext.getBean(SpringApplicationContext.java:31)
at com.artisan.bus.tool.redis.RedisClusterHelper.getClient(RedisClusterHelper.java:40)
- locked <0x00000006cc86c090> (a java.lang.Class for com.artisan.bus.tool.redis.RedisClusterHelper)
localhost-startStop-1 持有的lock 【0x00000006c6f34238】
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
- locked <0x00000006c6f34238> (a java.util.concurrent.ConcurrentHashMap)