blade-log服务出现oom

Blade 未结 1 117
2930417211
2930417211 剑者 2025-04-10 18:28
悬赏:10

一、该问题的重现步骤是什么?

1. 线上运行了大半年没出现什么问题,突然grafana提示oom

2. 查看了下log服务的日志,发下全表查询了blade_log_error表,花了200S。代码里面找了一圈没找到全表查询的地方。

3.


二、你期待的结果是什么?实际看到的又是什么?


三、你正在使用的是什么产品,什么版本?在什么操作系统上?

Bladex-Avue联合版,3.1.0.RELEASE,龙蜥8.6

四、请提供详细的错误堆栈信息,这很重要。

image.png

==============  Sql Start  ==============
Execute SQL : select id, stack_trace, exception_name, message, file_name , line_number, tenant_id, 
service_id, server_ip, server_host , env, remote_ip, user_agent, request_uri, method , method_class, 
method_name, params, create_by, create_time from blade_log_error
Execute Time200.669s
==============  Sql  End   ==============

image.png

五、若有更多详细信息,请在下面提供。

2025-04-10 16:57:31.440 [XNIO-1 task-19] [44aca2adab8e8f0d63726b6be76863fc] [org.springblade.core.mp.plugins.SqlLogInterceptor]
INFO :
==============  Sql Start  ==============
Execute SQL : insert into blade_log_error (id, stack_trace, exception_name, message, line_number , tenant_id, service_id, server_ip, server_host, env , remote_ip, user_agent, request_uri, method, method_class , method_name, params, create_by, create_time) values (1910255813519273986'java.io.IOException: Broken pipe
        at sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
        at sun.nio.ch.SocketDispatcher.writev(Unknown Source)
        at sun.nio.ch.IOUtil.write(Unknown Source)
        at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
        at org.xnio.nio.NioSocketConduit.write(NioSocketConduit.java:162)
        at io.undertow.conduits.BytesSentStreamSinkConduit.write(BytesSentStreamSinkConduit.java:76)
        at io.undertow.server.protocol.http.HttpResponseConduit.write(HttpResponseConduit.java:652)
        at io.undertow.conduits.AbstractFixedLengthStreamSinkConduit.write(AbstractFixedLengthStreamSinkConduit.java:148)
        at org.xnio.conduits.ConduitStreamSinkChannel.write(ConduitStreamSinkChannel.java:158)
        at io.undertow.channels.DetachableStreamSinkChannel.write(DetachableStreamSinkChannel.java:179)
        at io.undertow.server.HttpServerExchange$WriteDispatchChannel.write(HttpServerExchange.java:2163)
        at org.xnio.channels.Channels.writeBlocking(Channels.java:202)
        at io.undertow.servlet.spec.ServletOutputStreamImpl.writeTooLargeForBuffer(ServletOutputStreamImpl.java:200)
        at io.undertow.servlet.spec.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:148)
        at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source)
        at sun.nio.cs.StreamEncoder.implWrite(Unknown Source)
        at sun.nio.cs.StreamEncoder.write(Unknown Source)
        at sun.nio.cs.StreamEncoder.write(Unknown Source)
        at java.io.OutputStreamWriter.write(Unknown Source)
        at java.io.Writer.write(Unknown Source)
        at org.springframework.util.StreamUtils.copy(StreamUtils.java:147)
        at org.springframework.http.converter.StringHttpMessageConverter.writeInternal(StringHttpMessageConverter.java:126)
        at org.springframework.http.converter.StringHttpMessageConverter.writeInternal(StringHttpMessageConverter.java:44)
        at org.springframework.http.converter.AbstractHttpMessageConverter.write(AbstractHttpMessageConverter.java:227)
        at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.writeWithMessageConverters(AbstractMessageConverterMethodProcessor.java:293)
        at org.springframework.web.servlet.mvc.method.annotation.HttpEntityMethodProcessor.handleReturnValue(HttpEntityMethodProcessor.java:219)
        at org.springframework.web.method.support.HandlerMethodReturnValueHandlerComposite.handleReturnValue(HandlerMethodReturnValueHandlerComposite.java:78)
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:135)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1072)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:965)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:497)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:584)
        at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
        at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
        at org.springblade.core.log.filter.LogTraceFilter.doFilter(LogTraceFilter.java:39)
        at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67)
        at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
                at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67)
        at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
        at org.springblade.core.boot.request.BladeRequestFilter.doFilter(BladeRequestFilter.java:58)
        at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67)
        at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
        at com.alibaba.druid.support.http.WebStatFilter.doFilter(WebStatFilter.java:114)
        at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67)
        at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
        at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67)
        at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
        at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
        at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67)
        at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
        at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
        at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67)
        at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
        at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67)
        at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
        at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
        at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
        at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
        at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
        at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68)
        at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:117)
        at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
        at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
        at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
        at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
        at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
        at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
        at io.undertow.servlet.handlers.SendErrorPageHandler.handleRequest(SendErrorPageHandler.java:52)
        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
        at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:275)
        at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:79)
        at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:134)
        at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:131)
        at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
        at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
        at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:255)
        at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:79)
        at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:100)
        at io.undertow.server.Connectors.executeRootHandler(Connectors.java:393)
        at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:852)
        at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
        at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2019)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1558)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1449)
        at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282)
        at java.lang.Thread.run(Unknown Source)
''java.io.IOException''Broken pipe'-2 , '000000''blade-log''172.17.0.1:8103''iZon201hq73bvaq79zt3scZ''prod' , '10.222.142.18''Prometheus/2.45.3''/actuator/prometheus''GET''sun.nio.ch.FileDispatcherImpl' , 'writev0'''''TIMESTAMP '2025-04-10 16:57:31.000')
Execute Time2.136ms
==============  Sql  End   ==============
2025-04-10 16:57:31.440 [XNIO-1 task-19] [44aca2adab8e8f0d63726b6be76863fc] [org.springblade.core.log.aspect.RequestLogAspect]
INFO :
===============  Response Start  ================
===Result===  {"code":200,"success":true,"data":true,"msg":"操作成功"}
<=== POST: /log/saveErrorLog (3 ms)
===============   Response End   ================
2025-04-10 16:58:46.761 [XNIO-1 task-23] [c8389c22023241a4fb04dfda7053938f] [org.springblade.core.mp.plugins.SqlLogInterceptor]
INFO :
==============  Sql Start  ==============
Execute SQL : select id, stack_trace, exception_name, message, file_name , line_number, tenant_id, service_id, server_ip, server_host , env, remote_ip, user_agent, request_uri, method , method_class, method_name, params, create_by, create_time from blade_log_error
Execute Time200.669s
==============  Sql  End   ==============
2025-04-10 16:58:46.761 [XNIO-1 task-23] [c8389c22023241a4fb04dfda7053938f] [org.springblade.core.log.aspect.RequestLogAspect]
INFO :
===============  Response Start  ================
<=== GET: /error/detail (201384 ms)
===============   Response End   ================


1条回答
  • 2025-04-10 19:15

    看下这个200s的前后完整日志是什么,看看是什么请求触发的。按理说应该都是分页接口,系统默认也就提供了两个接口,一个是详情,一个是分页

    CleanShot20250410191449@2x.png

    0 讨论(0)
代码语言
提交回复