docker部署微服务发送sentinel心跳失败

Blade 未结

关注举报

 1  3565

Tiger 剑童 2020-03-24 22:14

悬赏：10

一、该问题的重现步骤是什么？

1. nacos, sentinel, redis, mysql, blade-admin, blade-log以及业务微服务均使用docker部署，参考script/docker-compose.yml方式

2. 所有服务均正常启动，各自功能均能正常访问

3. spring boot admin所有服务的sentinel状态为DOWN

4. 查询各微服务的健康信息："sentinel":{"status":"DOWN","details":{"dataSource":{},"enabled":true,"dashboard":{"description":"172.30.0.58:8858 can't be connected","status":"DOWN"}}}, IP及端口(172.30.0.58:8858)均正确，但提示：can't be connected

5. 进入容器查看sentinel-record.log.2020-03-24.0提示：WARNING [SimpleHttpHeartbeatSender] Failed to send heartbeat to sentinel.docker_blade_net/172.30.0.58:8858 : java.net.ConnectException: Connection refused (Connection refused)

6. 同一容器内检查连通性：

curl 172.30.0.58:8858
<!doctype html>
<html class="no-js">
<head>
  <meta charset="utf-8">
  <title>Sentinel Dashboard</title>
  <meta name="description" content="Sentinel 控制台,Sentinel Dashboard,Admin">
  <meta name="viewport" content="width=device-width">
  <link rel="stylesheet" href="./lib/css/bootstrap.min.css" />
  <link rel="stylesheet" href="./lib/css/font-awesome.min.css">
  <link rel="stylesheet" href="./dist/css/app.css">
</head>
<body>
  <div ng-app="sentinelDashboardApp">
    <div ui-view></div>
  </div>
  <script src="./lib/js/jquery.min.js"></script>
  <script src="./lib/js/bootstrap.min.js"></script>
  <script src="./lib/js/angular.min.js"></script>
  <script src="./lib/js/g2.min.js"></script>
  <script src="./dist/js/app.vendor.js"></script>
  <script src="./dist/js/app.js"></script>
</body>

bash-4.4# curl -i "172.30.0.58:8858/registry/machine?ip=172.30.0.9&port=8719"
HTTP/1.1 200Content-Type: application/json;charset=UTF-8Transfer-Encoding: chunkedDate: Tue, 24 Mar 2020 13:52:03 GMT
{"success":true,"code":0,"msg":"success","data":null}

bash-4.4# ip a
1: lo:
mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
713: eth0@if714:
mtu 1500 qdisc noqueue state UP
    link/ether 02:42:ac:1e:00:09 brd ff:ff:ff:ff:ff:ff
    inet 172.30.0.9/16 brd 172.30.255.255 scope global eth0
       valid_lft forever preferred_lft forever

二、你期待的结果是什么？实际看到的又是什么？

容器内的各微服务网络连通性正常，但sentinel心跳提示拒绝连接，微服务的sentinel健康状态为DOWN

三、你正在使用的是什么产品，什么版本？在什么操作系统上？

Bladex 2.1.0.RELEASE企业版本。

操作系统：CentOS Linux release 7.7.1908 (Core)

Docker：

Client: Docker Engine - Community Version: 19.03.7 API version: 1.40 Go version: go1.12.17 Git commit: 7141c199a2 Built: Wed Mar 4 01:24:10 2020 OS/Arch: linux/amd64 Experimental: false

Server: Docker Engine - Community Engine: Version: 19.03.7 API version: 1.40 (minimum version 1.12) Go version: go1.12.17 Git commit: 7141c199a2 Built: Wed Mar 4 01:22:45 2020 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.2.13 GitCommit: 7ad184331fa3e55e52b890ea95e65ba581ae3429 runc: Version: 1.0.0-rc10 GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd docker-init: Version: 0.18.0 GitCommit: fec3683

docker-compose version 1.18.0, build 8dd22a9 docker-py version: 2.6.1 CPython version: 3.6.8 OpenSSL version: OpenSSL 1.0.2k-fips 26 Jan 2017

四、请提供详细的错误堆栈信息，这很重要。

微服务健康信息：

curl 172.30.0.9:13001/actuator/health
{"status":"DOWN","details":{"nacosConfig":{"status":"UP"},"sentinel":{"status":"DOWN","details":{"dataSource":{},"enabled":true,"dashboard":{"description":"172.30.0.58:8858 can't be connected","status":"DOWN"}}},"diskSpace":{"status":"UP","details":{"total":105552769024,"free":41895972864,"threshold":10485760}},"db":{"status":"UP","details":{"database":"MySQL","hello":1}},"refreshScope":{"status":"UP"},"discoveryComposite":{"status":"UP","details":{"discoveryClient":{"status":"UP","details":{"services":["blade-admin","fire-biz-sys","fire-biz-om","blade-log","fire-msgservice"]}}}},"hystrix":{"status":"UP"},"redis":{"status":"UP","details":{"version":"5.0.2"}}}}

curl 172.30.0.9:13001/actuator/sentinel
{"blockPage":"","appName":"fire-biz-sys","consoleServer":"172.30.0.58:8858","coldFactor":"3","rules":{"systemRules":[],"authorityRule":[],"paramFlowRule":[],"flowRules":[],"degradeRules":[]},"metricsFileCharset":"UTF-8","filter":{"order":-2147483648,"urlPatterns":["/*"],"enabled":true},"totalMetricsFileCount":6,"datasource":{},"clientIp":"172.30.0.9","clientPort":"8719","logUsePid":false,"metricsFileSize":52428800,"logDir":"/root/logs/csp/","heartbeatIntervalMs":10000}

微服务sentinel日志：

sentinel-record.log.2020-03-24.zip

WARNING [SimpleHttpHeartbeatSender] Failed to send heartbeat to sentinel.docker_blade_net/172.30.0.58:8858 : java.net.SocketTimeoutException: connect timed out at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at com.alibaba.csp.sentinel.transport.heartbeat.client.SimpleHttpClient.request(SimpleHttpClient.java:96) at com.alibaba.csp.sentinel.transport.heartbeat.client.SimpleHttpClient.post(SimpleHttpClient.java:81) at com.alibaba.csp.sentinel.transport.heartbeat.SimpleHttpHeartbeatSender.sendHeartbeat(SimpleHttpHeartbeatSender.java:72) at com.alibaba.csp.sentinel.transport.init.HeartbeatSenderInitFunc$1.run(HeartbeatSenderInitFunc.java:91) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

容器内连通信检查：

ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
713: eth0@if714: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
    link/ether 02:42:ac:1e:00:09 brd ff:ff:ff:ff:ff:ff
    inet 172.30.0.9/16 brd 172.30.255.255 scope global eth0
       valid_lft forever preferred_lft forever

curl 172.30.0.58:8858
<!doctype html>
<html class="no-js">
<head>
  <meta charset="utf-8">
  <title>Sentinel Dashboard</title>
  <meta name="description" content="Sentinel 控制台,Sentinel Dashboard,Admin">
  <meta name="viewport" content="width=device-width">
  <link rel="stylesheet" href="./lib/css/bootstrap.min.css" />
  <link rel="stylesheet" href="./lib/css/font-awesome.min.css">
  <link rel="stylesheet" href="./dist/css/app.css">

 curl "172.30.0.58:8858/registry/machine?ip=172.30.0.9&port=8719"
{"success":true,"code":0,"msg":"success","data":null}

netstat -pl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:8719            0.0.0.0:*               LISTEN      1/java
tcp        0      0 127.0.0.11:44868        0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:13001           0.0.0.0:*               LISTEN      1/java
udp        0      0 0.0.0.0:52464           0.0.0.0:*                           1/java
udp        0      0 127.0.0.11:45776        0.0.0.0:*                           -
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node PID/Program name    Path

五、若有更多详细信息，请在下面提供。