主页 > 知识库 > dbcp 连接池不合理的锁导致连接耗尽解决方案

dbcp 连接池不合理的锁导致连接耗尽解决方案

热门标签:Linux服务器 银行业务 科大讯飞语音识别系统 服务器配置 阿里云 团购网站 电子围栏 Mysql连接数设置

dbcp 连接池不合理的锁导致连接耗尽解决方案

应用报错,表象来看是连接池爆满了。

org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool exhausted
    at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:241) ~[spring-jdbc-3.2.2.RELEASE.jar:3.2.2.RELEASE]
    at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:82) [dubbo-2.5.3.jar:2.5.3]
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [na:1.6.0_33]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [na:1.6.0_33]
    at java.lang.Thread.run(Thread.java:662) [na:1.6.0_33]
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool exhausted
    at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:103) ~[commons-dbcp.jar:1.2.1]
    at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540) ~[commons-dbcp.jar:1.2.1]
    at com.eshore.crmpub.jdbc.datasource.MultiDataSource.getConnection(MultiDataSource.java:74) ~[crmpub-jdbc-1.0.jar:1.0]
    at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:203) ~[spring-jdbc-3.2.2.RELEASE.jar:3.2.2.RELEASE]
    ... 32 common frames omitted
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
    at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1174) ~[commons-pool-1.6.jar:1.6]
    at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:74) ~[commons-dbcp.jar:1.2.1]
    at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95) ~[commons-dbcp.jar:1.2.1]
    ... 35 common frames omitted

应用堆栈,几十条线程都阻塞到dbcp池的PoolableConnectionFactory.makeObject方法了,此方法在等待0x0000000709a638a0对象锁,但0x0000000709a638a0对象一直被其中一条线程执行PoolableConnectionFactory.makeObject时锁了,而且这条线程获取到锁后还阻塞住了,所以导致后面几十条线程都BLOCKED了。

"DubboServerHandler-132.121.95.136:28101-thread-300" daemon prio=10 tid=0x00007f02b4136800 nid=0x4954 waiting for monitor entry [0x00007f0286018000]
  java.lang.Thread.State: BLOCKED (on object monitor)
  at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:290)
  - waiting to lock 0x0000000709a638a0> (a org.apache.commons.dbcp.PoolableConnectionFactory)
  at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1188)
  at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:74)
  at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95)
  at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540)
  at com.eshore.crmpub.jdbc.datasource.MultiDataSource.getConnection(MultiDataSource.java:74)
  at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:203)
  at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:372)
  at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:417)
  at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:255)
  at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
  at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
  at com.eshore.crm.service.common.pub.aspect.DataSourceRouteAspect.around(DataSourceRouteAspect.java:74)
  at sun.reflect.GeneratedMethodAccessor253.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  at java.lang.reflect.Method.invoke(Method.java:597)
  at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
  at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
  at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
  at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
  at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:631)
  at com.eshore.crm.service.sysmgr.sca.impl.SysMgrServiceImpl$$EnhancerByCGLIB$$bfd71326.qryBulletinList(generated>)
  at com.alibaba.dubbo.common.bytecode.Wrapper101.invokeMethod(Wrapper101.java)
  at com.alibaba.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:46)
  at com.alibaba.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:72)
  at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:53)
  at com.eshore.crmpub.sca.dubbo.filter.BaseFilter.invoke(BaseFilter.java:90)
  at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
  at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:108)
  at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:84)
  at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:170)
  at com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:52)
  at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:82)
  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  at java.lang.Thread.run(Thread.java:662)

  Locked ownable synchronizers:
  - 0x000000070a120f00> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"DubboServerHandler-132.121.95.136:28101-thread-290" daemon prio=10 tid=0x00007f03203da800 nid=0x4948 runnable [0x00007f0286a22000]
  java.lang.Thread.State: RUNNABLE
  at java.net.SocketInputStream.socketRead0(Native Method)
  at java.net.SocketInputStream.read(SocketInputStream.java:129)
  at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:112)
  at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:159)
  at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:187)
  - locked 0x0000000715a9ce70> (a com.mysql.jdbc.util.ReadAheadInputStream)
  at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3140)
  at com.mysql.jdbc.MysqlIO.readPacket(MysqlIO.java:597)
  at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1085)
  at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2494)
  at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2527)
  at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2309)
  - locked 0x00000007159d1898> (a com.mysql.jdbc.JDBC4Connection)
  at com.mysql.jdbc.ConnectionImpl.init>(ConnectionImpl.java:834)
  at com.mysql.jdbc.JDBC4Connection.init>(JDBC4Connection.java:46)
  at sun.reflect.GeneratedConstructorAccessor78.newInstance(Unknown Source)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
  at com.mysql.jdbc.Util.handleNewInstance(Util.java:408)
  at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:419)
  at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:344)
  at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:37)
  at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:290)
  - locked 0x0000000709a638a0> (a org.apache.commons.dbcp.PoolableConnectionFactory)
  at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1188)
  at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:74)
  at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95)
  at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540)
  at com.eshore.crmpub.jdbc.datasource.MultiDataSource.getConnection(MultiDataSource.java:74)
  at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:203)
  at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:372)
  at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:417)
  at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:255)
  at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
  at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
  at com.eshore.crm.service.common.pub.aspect.DataSourceRouteAspect.around(DataSourceRouteAspect.java:74)
  at sun.reflect.GeneratedMethodAccessor253.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  at java.lang.reflect.Method.invoke(Method.java:597)
  at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
  at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
  at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
  at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
  at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:631)
  at com.eshore.crm.service.cachemgr.sca.common.DictCacheServiceImpl$$EnhancerByCGLIB$$e27e8e25.getDict(generated>)
  at com.alibaba.dubbo.common.bytecode.Wrapper93.invokeMethod(Wrapper93.java)
  at com.alibaba.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:46)
  at com.alibaba.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:72)
  at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:53)
  at com.eshore.crmpub.sca.dubbo.filter.BaseFilter.invoke(BaseFilter.java:90)
  at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
  at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:108)
  at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:84)
  at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:170)
  at com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:52)
  at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:82)
  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  at java.lang.Thread.run(Thread.java:662)

  Locked ownable synchronizers:
  - 0x000000070a0f9218> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

可以看到其实获取锁的线程是阻塞在这里了,做mysql握手包的时候一直在等待读,阻塞在网络IO了。

 java.lang.Thread.State: RUNNABLE
  at java.net.SocketInputStream.socketRead0(Native Method)
  at java.net.SocketInputStream.read(SocketInputStream.java:129)
  at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:112)
  at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:159)
  at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:187)
  - locked 0x0000000715a9ce70> (a com.mysql.jdbc.util.ReadAheadInputStream)
  at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3140)
  at com.mysql.jdbc.MysqlIO.readPacket(MysqlIO.java:597)
  at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1085)
  at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2494)
  at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2527)
  at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2309)

这个是dbcp1.x版本在创建新的连接对象时会把整个连接工厂类对象锁了,一旦创建过程中发生阻塞就会导致整个池都死掉。这个问题很大。

解决方案,升级dbcp2.x版本或tomcat jdbc或druid连接池,推荐druid连接池,功能强大自带监控。

感谢阅读,希望能帮助到大家,谢谢大家对本站的支持!

您可能感兴趣的文章:
  • 浅谈在Spring中如何使用数据源(DBCP、C3P0、JNDI)
  • Java数据库连接池之DBCP浅析_动力节点Java学院整理
  • Tomcat 7-dbcp配置数据库连接池详解
  • java使用dbcp2数据库连接池
  • java配置dbcp连接池(数据库连接池)示例分享
  • Java dbcp连接池基本使用方法详解

标签:江苏 大理 衡水 广元 萍乡 衢州 蚌埠 枣庄

巨人网络通讯声明:本文标题《dbcp 连接池不合理的锁导致连接耗尽解决方案》,本文关键词  ;如发现本文内容存在版权问题,烦请提供相关信息告之我们,我们将及时沟通与处理。本站内容系统采集于网络,涉及言论、版权与本站无关。
  • 相关文章
  • 收缩
    • 微信客服
    • 微信二维码
    • 电话咨询

    • 400-1100-266