生产环境中，MySQL数据库有时候底层连不上。

Blade 未结

关注举报

 1  258

社会你方哥剑圣 2024-03-25 09:43

生产环境中，后端程序运行着一段时间之后，程序就挂了，看了一下生产服务器环境：

CPU都已经超负荷运行了，再看看错误日志如下：

2024-03-25 01:18:32.076 [pool-3-thread-7] [] [o.s.s.support.TaskUtils$LoggingErrorHandler]

ERROR: Unexpected error occurred in scheduled task.

org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is com.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 61996, active 2, maxActive 20, creating 0, createErrorCount 3

at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:304)

at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:378)

at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:476)

at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:289)

at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)

at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691)

at com.dmf.job.dynamicabstract.impl.IpMissionCreate$$EnhancerBySpringCGLIB$$6a386788.createMission(<generated>)

at com.dmf.job.MissionJob.autoCreateMission(MissionJob.java:52)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84)

at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)

at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:93)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Caused by: com.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 61996, active 2, maxActive 20, creating 0, createErrorCount 3

at com.alibaba.druid.pool.DruidDataSource.getConnectionInternal(DruidDataSource.java:1737)

at com.alibaba.druid.pool.DruidDataSource.getConnectionDirect(DruidDataSource.java:1409)

at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1389)

at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1379)

at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:109)

at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:261)

... 22 common frames omitted

Caused by: java.lang.OutOfMemoryError: Java heap space

2024-03-25 09:17:56.892 [pool-3-thread-5] [] [o.s.s.support.TaskUtils$LoggingErrorHandler]

ERROR: Unexpected error occurred in scheduled task.

org.mybatis.spring.MyBatisSystemException: nested exception is org.apache.ibatis.exceptions.PersistenceException:

### Error querying database. Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is com.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 62653, active 2, maxActive 20, creating 0, createErrorCount 3

### The error may exist in com/dmf/biz/dynamicmanage/mission/mapper/MissionListMapper.java (best guess)

### The error may involve com.dmf.biz.dynamicmanage.mission.mapper.MissionListMapper.selectList

### The error occurred while executing a query

### Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is com.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 62653, active 2, maxActive 20, creating 0, createErrorCount 3

at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:92)

at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:440)

at com.sun.proxy.$Proxy137.selectList(Unknown Source)

at org.mybatis.spring.SqlSessionTemplate.selectList(SqlSessionTemplate.java:223)

at com.baomidou.mybatisplus.core.override.MybatisMapperMethod.executeForMany(MybatisMapperMethod.java:177)

at com.baomidou.mybatisplus.core.override.MybatisMapperMethod.execute(MybatisMapperMethod.java:78)

at com.baomidou.mybatisplus.core.override.MybatisMapperProxy.invoke(MybatisMapperProxy.java:96)

at com.sun.proxy.$Proxy160.selectList(Unknown Source)

at com.baomidou.mybatisplus.extension.conditions.query.ChainQuery.list(ChainQuery.java:39)

at com.dmf.job.dynamicabstract.impl.IpMissionCreate.createTask(IpMissionCreate.java:63)

at com.dmf.job.dynamicabstract.impl.IpMissionCreate$$FastClassBySpringCGLIB$$a78ac6ef.invoke(<generated>)

at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)

at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:687)

at com.dmf.job.dynamicabstract.impl.IpMissionCreate$$EnhancerBySpringCGLIB$$6a386788.createTask(<generated>)

at com.dmf.job.MissionJob.autoCreateMtTask(MissionJob.java:92)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84)

at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)

at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:93)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Caused by: org.apache.ibatis.exceptions.PersistenceException:

### The error may exist in com/dmf/biz/dynamicmanage/mission/mapper/MissionListMapper.java (best guess)

### The error may involve com.dmf.biz.dynamicmanage.mission.mapper.MissionListMapper.selectList

### The error occurred while executing a query

at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)

at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:149)

at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:140)

at sun.reflect.GeneratedMethodAccessor608.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:426)

... 27 common frames omitted

Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is com.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 62653, active 2, maxActive 20, creating 0, createErrorCount 3

at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:82)

at org.mybatis.spring.transaction.SpringManagedTransaction.openConnection(SpringManagedTransaction.java:80)

at org.mybatis.spring.transaction.SpringManagedTransaction.getConnection(SpringManagedTransaction.java:67)

at org.apache.ibatis.executor.BaseExecutor.getConnection(BaseExecutor.java:336)

at com.baomidou.mybatisplus.core.executor.MybatisSimpleExecutor.prepareStatement(MybatisSimpleExecutor.java:91)

at com.baomidou.mybatisplus.core.executor.MybatisSimpleExecutor.doQuery(MybatisSimpleExecutor.java:66)

at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:324)

at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156)

at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:136)

at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:147)

... 32 common frames omitted

Caused by: com.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 62653, active 2, maxActive 20, creating 0, createErrorCount 3

at com.alibaba.druid.pool.DruidDataSource.getConnectionInternal(DruidDataSource.java:1737)

at com.alibaba.druid.pool.DruidDataSource.getConnectionDirect(DruidDataSource.java:1409)

at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1389)

at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1379)

at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:109)

at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:158)

at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:116)

at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:79)

... 41 common frames omitted

Caused by: java.lang.OutOfMemoryError: Java heap space

看报错原因好像是数据库连接不上，最后不知道什么原因就导致内存溢出了。我用客户端工具连接数据库是正常能连接的。