后端微服务健康检查异常问题咨询

Blade 未结

关注举报

 1  228

627588637 剑圣 2024-09-27 10:36

一、该问题的重现步骤是什么？

1.后端微服务接入prometheus健康检查，容器启用健康检查后，日志会间隔一段时间打印异常，但未影响业务功能使用

2.容器健康检查启用后，应用系统大概间隔60秒左右出现一次Connection reset by peer异常问题，业务功能使用正常，不影响业务使用

3.容器健康检查关闭后，应用系统不出现Connection reset by peer异常问题，业务功能使用正常

pic(3).png

pic(1).png

二、你期待的结果是什么？实际看到的又是什么？

期望后端微服务发布到容器平台，容器健康检查启用后，不会出现异常错误

三、你正在使用的是什么产品，什么版本？在什么操作系统上？

BladeX企业版，4.0.1.RELEASE，Windows开发环境

四、请提供详细的错误堆栈信息，这很重要。

从代码逻辑看，是Blade框架全局Request包装拦截HttpServletRequest，缓存报文读取流，获取请求体的时候报错

pic(2).png

pic(4).png

pic(5).png

五、若有更多详细信息，请在下面提供。

2024-09-27 at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)

2024-09-27 at org.xnio.XnioWorker$Work2024-09-27T10:30:32.371525193+08:00 erThreadFactory$1$1.run(XnioWorker.java:1282)

2024-09-27 at java.base/java.lang.Thread.run(Thread.java:840)

2024-09-27

2024-09-27 2024-09-27 10:30:32.371 WARN 1 --- [ XNIO-1 task-5] .w.s.m.s.DefaultHandlerExceptionResolver : Resolved [org.springframework.web.context.request.async.AsyncRequestNotUsableException: ServletOutputStream failed to write: Connection reset by peer]

2024-09-27 2024-09-27 10:30:32.375 WARN 1 --- [c-scheduler4461] RetryableFeignBlockingLoadBalancerClient : Service instance was not resolved, executing the original request

2024-09-27 2024-09-27 10:32:32.457 ERROR 1 --- [ XNIO-1 task-5] o.s.c.l.e.BladeRestExceptionTranslator : 服务器异常

2024-09-27

2024-09-27 org.springframework.web.context.request.async.AsyncRequestNotUsableException: ServletOutputStream failed to write: Connection reset by peer

2024-09-27 at org.springframework.web.context.request.async.StandardServletAsyncWebRequest$LifecycleHttpServletResponse.handleIOException(StandardServletAsyncWebRequest.java:320)

2024-09-27 at org.springframework.web.context.request.async.StandardServletAsyncWebRequest$LifecycleServletOutputStream.write(StandardServletAsyncWebRequest.java:378)

2024-09-27 at com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2203)

2024-09-27 at com.fasterxml.jackson.core.json.UTF8JsonGenerator.writeString(UTF8JsonGenerator.java:521)

2024-09-27 at com.fasterxml.jackson.databind.ser.std.StringSerializer.serialize(StringSerializer.java:41)

2024-09-27 at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119)

2024-09-27 at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79)

2024-09-27 at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18)

2024-09-27 at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:808)

2024-09-27 at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeWithoutTypeInfo(MapSerializer.java:764)

2024-09-27 at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:720)

2024-09-27 at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:35)

2024-09-27 at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:808)

2024-09-27 at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeWithoutTypeInfo(MapSerializer.java:764)

2024-09-27 at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:720)

2024-09-27 at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:35)

2024-09-27 at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:732)

2024-09-27 at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:772)

2024-09-27 at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:178)

2024-09-27 at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:808)

2024-09-27 at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeWithoutTypeInfo(MapSerializer.java:764)

2024-09-27 at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:720)

2024-09-27 at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:35)

2024-09-27 at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:732)

2024-09-27 at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:772)

2024-09-27 at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:178)

2024-09-27 at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:479)

2024-09-27 at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:318)

2024-09-27 at com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1572)

2024-09-27 at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:1061)

2024-09-27 at org.springblade.core.tool.jackson.AbstractReadWriteJackson2HttpMessageConverter.writeInternal(AbstractReadWriteJackson2HttpMessageConverter.java:141)

2024-09-27 at org.springblade.core.tool.jackson.MappingApiJackson2HttpMessageConverter.writeInternal(MappingApiJackson2HttpMessageConverter.java:98)

2024-09-27 at org.springframework.http.converter.AbstractGenericHttpMessageConverter.write(AbstractGenericHttpMessageConverter.java:114)

2024-09-27 at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.writeWithMessageConverters(AbstractMessageConverterMethodProcessor.java:297)

2024-09-27 at org.springframework.web.servlet.mvc.method.annotation.HttpEntityMethodProcessor.handleReturnValue(HttpEntityMethodProcessor.java:245)

2024-09-27 at org.springframework.web.method.support.HandlerMethodReturnValueHandlerComposite.handleReturnValue(HandlerMethodReturnValueHandlerComposite.java:78)

2024-09-27 at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:136)

2024-09-27 at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:925)

2024-09-27 at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:830)

2024-09-27 at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)

2024-09-27 at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1089)

2024-09-27 at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:979)

2024-09-27 at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014)

2024-09-27 at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:903)

2024-09-27 at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:527)

2024-09-27 at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885)

2024-09-27 at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:614)

2024-09-27 at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)

2024-09-27 at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)

2024-09-27 at org.springblade.core.log.filter.LogTraceFilter.doFilter(LogTraceFilter.java:48)

2024-09-27 at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67)

2024-09-27 at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

2024-09-27 at org.springblade.core.boot.request.BladeRequestFilter.doFilter(BladeRequestFilter.java:67)

2024-09-27 at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67)

2024-09-27 at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

2024-09-27 at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)

2024-09-27 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)

2024-09-27 at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67)

2024-09-27 at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

2024-09-27 at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)

2024-09-27 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)

2024-09-27 at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67)

2024-09-27 at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

2024-09-27 at org.springframework.web.filter.ServerHttpObservationFilter.doFilterInternal(ServerHttpObservationFilter.java:109)

2024-09-27 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)

2024-09-27 at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67)

2024-09-27 at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

2024-09-27 at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)

2024-09-27 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)

2024-09-27 at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67)

2024-09-27 at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

2024-09-27 at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)

2024-09-27 at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)

2024-09-27 at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)

2024-09-27 at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)

2024-09-27 at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68)

2024-09-27 at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:2024-09-27T10:32:32.458380662+08:00 117)

2024-09-27 at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)

2024-09-27 at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

2024-09-27 at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)

2024-09-27 at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)

2024-09-27 at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)

2024-09-27 at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)

2024-09-27 at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)

2024-09-27 at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

2024-09-27 at io.undertow.servlet.handlers.SendErrorPageHandler.handleRequest(SendErrorPageHandler.java:52)

2024-09-27 at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

2024-09-27 at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:276)

2024-09-27 at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)

2024-09-27 at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:132)

2024-09-27 at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)

2024-09-27 at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)

2024-09-27 at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:256)

2024-09-27 at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:101)

2024-09-27 at io.undertow.server.Connectors.executeRootHandler(Connectors.java:393)

2024-09-27 at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:859)

2024-09-27 at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)

2024-09-27 at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)

2024-09-27 at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)

2024-09-27 at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282)

2024-09-27 at java.base/java.lang.Thread.run(Thread.java:840)

2024-09-27 Caused by: java.io.IOException: Connection reset by peer

2024-09-27 at java.base/sun.nio.ch.FileDispatcherImpl.writev0(Native Method)

2024-09-27 at java.base/sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:66)

2024-09-27 at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:217)

2024-09-27 at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:153)

2024-09-27 at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:563)

2024-09-27 at org.xnio.nio.NioSocketConduit.write(NioSocketConduit.java:162)

2024-09-27 at io.undertow.conduits.BytesSentStreamSinkConduit.write(BytesSentStreamSinkConduit.java:76)

2024-09-27 at io.undertow.server.protocol.http.HttpResponseConduit.write(HttpResponseConduit.java:667)

2024-09-27 at org.xnio.conduits.AbstractStreamSinkConduit.write(AbstractStreamSinkConduit.java:55)

2024-09-27 at org.xnio.conduits.ConduitStreamSinkChannel.write(ConduitStreamSinkChannel.java:158)

2024-09-27 at io.undertow.channels.DetachableStreamSinkChannel.write(DetachableStreamSinkChannel.java:179)

2024-09-27 at io.undertow.server.HttpServerExchange$WriteDispatchChannel.write(HttpServerExchange.java:2172)

2024-09-27 at org.xnio.channels.Channels.writeBlocking(Channels.java:202)

2024-09-27 at io.undertow.servlet.spec.ServletOutputStreamImpl.writeTooLargeForBuffer(ServletOutputStreamImpl.java:201)

2024-09-27 at io.undertow.servlet.spec.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:149)

2024-09-27 at org.springframework.web.context.request.async.StandardServletAsyncWebRequest$LifecycleServletOutputStream.write(StandardServletAsyncWebRequest.java:375)

2024-09-27 ... 99 common frames omitted

2024-09-27

2024-09-27 java.io.IOException: UT010029: Stream is closed

2024-09-27 at io.undertow.servlet.spec.ServletInputStreamImpl.read(ServletInputStreamImpl.java:168)

2024-09-27 at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:281)

2024-09-27 at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:324)

2024-09-27 at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:189)

2024-09-27 at java.base/java.io.InputStreamReader.read(InputStreamReader.java:177)

2024-09-27 at java.base/java.io.BufferedReader.fill(BufferedReader.java:162)

2024-09-27 at java.base/java.io.BufferedReader.readLine(BufferedReader.java:329)

2024-09-27 at java.base/java.io.BufferedReader.readLine(BufferedReader.java:396)

2024-09-27 at org.springblade.core.tool.utils.WebUtil.getRequestBody(WebUtil.java:262)

2024-09-27 at org.springblade.core.tool.utils.WebUtil.getRequestContent(WebUtil.java:302)

2024-09-27 at org.springblade.core.log.utils.LogAbstractUtil.addRequestInfoToLog(LogAbstractUtil.java:58)

2024-09-27 at org.springblade.core.log.publisher.ErrorLogPublisher.publishEvent(ErrorLogPublisher.java:66)

2024-09-27 at org.springblade.core.log.error.BladeRestExceptionTranslator.handleError(BladeRestExceptionTranslator.java:91)

2024-09-27 at jdk.internal.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)

2024-09-27 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

2024-09-27 at java.base/java.lang.reflect.Method.invoke(Method.java:568)

2024-09-27 at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:255)

2024-09-27 at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:188)

2024-09-27 at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118)

2024-09-27 at org.springframework.web.servlet.mvc.method.annotation.ExceptionHandlerExceptionResolver.doResolveHandlerMethodException(ExceptionHandlerExceptionResolver.java:432)

2024-09-27 at org.springframework.web.servlet.handler.AbstractHandlerMethodExceptionResolver.doResolveException(AbstractHandlerMethodExceptionResolver.java:74)

2024-09-27 at org.springframework.web.servlet.handler.AbstractHandlerExceptionResolver.resolveException(AbstractHandlerExceptionResolver.java:175)

2024-09-27 at org.springframework.web.servlet.handler.HandlerExceptionResolverComposite.resolveException(HandlerExceptionResolverComposite.java:80)

2024-09-27 at org.springframework.web.servlet.DispatcherServlet.processHandlerException(DispatcherServlet.java:1357)

2024-09-27 at org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1160)

2024-09-27 at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1106)