cloud版本集成elk后运行两三天后elk报错

Blade 未结 2 1349
hnmas
hnmas 剑者 2020-11-02 09:17
悬赏:20

一、该问题的重现步骤是什么?

1. 按文档集成elk,  docker方式启动,正常;

2. 项目中开启elk,指定elk监听ip和端口9000; 

3. 启动项目后,elk中正常显示日志

4. 运行约二到三天左右,频繁报elk警告的异常;

5. 出下面二个错误时,elk能收到部分日志,丢失部分日志,原因不明,重启后一切正常,再过二三天问题仍旧。


三、你正在使用的是什么产品,什么版本?在什么操作系统上?

2.6版本


四、请提供详细的错误堆栈信息,这很重要。

08:58:43,538 |-WARN in net.logstash.logback.appender.LogstashTcpSocketAppender[INFO_LOGSTASH] - Log destination 192.168.239.11:9000: unable to send event: Broken pipe (Write failed) Reconnecting. java.net.SocketException: Broken pipe (Write failed)

at java.net.SocketException: Broken pipe (Write failed)

at at java.net.SocketOutputStream.socketWrite0(Native Method)

at at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)

at at java.net.SocketOutputStream.write(SocketOutputStream.java:155)

at at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)

at at java.io.BufferedOutputStream.write(BufferedOutputStream.java:121)

at at java.io.FilterOutputStream.write(FilterOutputStream.java:97)

at at net.logstash.logback.appender.AbstractLogstashTcpSocketAppender$TcpSendingEventHandler.writeEvent(AbstractLogstashTcpSocketAppender.java:602)

at at net.logstash.logback.appender.AbstractLogstashTcpSocketAppender$TcpSendingEventHandler.onEvent(AbstractLogstashTcpSocketAppender.java:564)

at at net.logstash.logback.appender.AbstractLogstashTcpSocketAppender$TcpSendingEventHandler.onEvent(AbstractLogstashTcpSocketAppender.java:289)

at at net.logstash.logback.appender.AsyncDisruptorAppender$EventClearingEventHandler.onEvent(AsyncDisruptorAppender.java:339)

at at net.logstash.logback.appender.AsyncDisruptorAppender$EventClearingEventHandler.onEvent(AsyncDisruptorAppender.java:327)

at at net.logstash.logback.encoder.com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168)

at at net.logstash.logback.encoder.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)

at at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

at at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

at at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at at java.lang.Thread.run(Thread.java:748)

08:58:43,539 |-WARN in net.logstash.logback.appender.LogstashTcpSocketAppender[INFO_LOGSTASH] - Log destination 192.168.239.11:9000: Waiting 29990ms before attempting reconnection.


2条回答
  • 2020-11-02 09:20

    补充说明 :

    1. elk频繁出错时,测试到elk服务的网络正常;

    2. 重启后恢复正常;

    1 讨论(0)
  • 2020-11-03 11:11

    elk对服务器内存要求较高,最好是16g-32g只跑elk,你们服务器的资源是如何?如果内存资源不够的话,尝试用一台高内存的测试下

    0 讨论(0)
提交回复