Winse Blog

走走停停, 熙熙攘攘, 忙忙碌碌, 不知何畏.

两台主机的docker通过route互联互通

前面一直用k8s的flannel来建立主机间docker容器的互联,但是当仅有两台机器用来做测试的时刻,安装一个flannel也是挺纠结的:麻烦、还有未知的问题,起一个服务在那里总会有那么些担忧。

其实可以直接通过建立路由来实现两台机器间容器的互联互通:Docker多台宿主机间的容器互联-centos7(直接路由)

两台主机(centos7/docker-1.12.6):

  • 192.168.191.140 kube-master
  • 192.168.191.141 kube-worker1

安装/配置docker

这里不多讲了,参考 Kubeadm部署kubernetes 进行docker的安装。

建立新网卡,修改docker配置使用新网卡

  • 安装/更新依赖
1
yum install net-tools bridge-utils -y
  • 关防火墙、关selinux
1
2
3
4
5
6
7
setenforce 0

vi /etc/selinux/config
SELINUX=disabled

systemctl stop firewalld
systemctl disable firewalld
  • 设置ip转发
1
2
3
4
echo 1 > /proc/sys/net/ipv4/ip_forward

vi /etc/sysctl.conf
net.ipv4.ip_forward = 1  
  • 删docker0,建kbr0

先停docker!先停docker!先停docker!(好像docker会缓冲bridge的ip)

1
2
3
4
service docker stop
brctl addbr kbr0
ip link set dev docker0 down
ip link del dev docker0

下面的配置,两台机不同,如下:

192.168.191.140 kube-master 192.168.191.141 kube-worker1
vi /etc/sysconfig/network-scripts/ifcfg-kbr0
DEVICE=kbr0 DEVICE=kbr0
ONBOOT=yes ONBOOT=yes
BOOTPROTO=static BOOTPROTO=static
IPADDR=172.17.3.1 IPADDR=172.17.4.1
NETMASK=255.255.255.0 NETMASK=255.255.255.0
GATEWAY=172.17.3.0 GATEWAY=172.17.4.0
USERCTL=no USERCTL=no
TYPE=Bridge TYPE=Bridge
IPV6INIT=no IPV6INIT=no
———————————————– ——————————————————–
vi /etc/sysconfig/network-scripts/route-ens33 (ip对应的网卡名称)
172.17.4.0/24 via 192.168.191.141 dev ens33 172.17.3.0/24 via 192.168.191.140 dev ens33
———————————————– ——————————————————–

参考: Configuring Static Routes

  • 修改docker配置
1
2
3
4
vi /usr/lib/systemd/system/docker.service     
ExecStart=/usr/bin/dockerd --bridge=kbr0 

systemctl daemon-reload 
  • 重新启动

先起网卡!先起网卡!先起网卡!

1
2
3
service network restart

systemctl start docker

最终效果

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
| 192.168.191.140 kube-master                                                   | 192.168.191.141 kube-worker1                            
|:------------------------------------------------------------------------------|:-------------------------------------------------------
| [root@kube-master ~]# ifconfig                                                | [root@kube-worker1 ~]# ifconfig
| ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500                   | ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
|         inet 192.168.191.140  netmask 255.255.255.0  broadcast 192.168.191.255|         inet 192.168.191.141  netmask 255.255.255.0  broadcast 192.168.191.255
|         inet6 fe80::1186:2fe5:9ee5:8790  prefixlen 64  scopeid 0x20<link>     |         inet6 fe80::3995:4490:e2e7:1d0f  prefixlen 64  scopeid 0x20<link>
|         ether 00:0c:29:40:2d:15  txqueuelen 1000  (Ethernet)                  |         ether 00:0c:29:2e:67:4d  txqueuelen 1000  (Ethernet)
|         RX packets 18010  bytes 10754845 (10.2 MiB)                           |         RX packets 19871  bytes 12247126 (11.6 MiB)
|         RX errors 0  dropped 0  overruns 0  frame 0                           |         RX errors 0  dropped 0  overruns 0  frame 0
|         TX packets 4797  bytes 475332 (464.1 KiB)                             |         TX packets 5647  bytes 561624 (548.4 KiB)
|         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0            |         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
|                                                                               | 
| kbr1: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500                            | kbr0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
|         inet 172.17.3.1  netmask 255.255.255.0  broadcast 172.17.3.255        |         inet 172.17.4.1  netmask 255.255.255.0  broadcast 172.17.4.255
|         ether 00:00:00:00:00:00  txqueuelen 1000  (Ethernet)                  |         ether 00:00:00:00:00:00  txqueuelen 1000  (Ethernet)
|         RX packets 179  bytes 13932 (13.6 KiB)                                |         RX packets 139  bytes 10492 (10.2 KiB)
|         RX errors 0  dropped 0  overruns 0  frame 0                           |         RX errors 0  dropped 0  overruns 0  frame 0
|         TX packets 43  bytes 3894 (3.8 KiB)                                   |         TX packets 36  bytes 3004 (2.9 KiB)
|         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0            |         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
|                                                                               | 
| lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536                                  | lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
|         inet 127.0.0.1  netmask 255.0.0.0                                     |         inet 127.0.0.1  netmask 255.0.0.0
|         inet6 ::1  prefixlen 128  scopeid 0x10<host>                          |         inet6 ::1  prefixlen 128  scopeid 0x10<host>
|         loop  txqueuelen 1  (Local Loopback)                                  |         loop  txqueuelen 1  (Local Loopback)
|         RX packets 140  bytes 11644 (11.3 KiB)                                |         RX packets 215  bytes 18260 (17.8 KiB)
|         RX errors 0  dropped 0  overruns 0  frame 0                           |         RX errors 0  dropped 0  overruns 0  frame 0
|         TX packets 140  bytes 11644 (11.3 KiB)                                |         TX packets 215  bytes 18260 (17.8 KiB)
|         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0            |         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
|                                                                               | 
| [root@kube-master ~]# route -n                                                | [root@kube-worker1 ~]# route -n 
| Kernel IP routing table                                                       | Kernel IP routing table
| Destination     Gateway         Genmask         Flags Metric Ref    Use Iface | Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
| 0.0.0.0         192.168.191.2   0.0.0.0         UG    100    0        0 ens33 | 0.0.0.0         192.168.191.2   0.0.0.0         UG    100    0        0 ens33
| 172.17.3.0      0.0.0.0         255.255.255.0   U     427    0        0 kbr1  | 172.17.3.0      192.168.191.140 255.255.255.0   UG    100    0        0 ens33
| 172.17.4.0      192.168.191.141 255.255.255.0   UG    100    0        0 ens33 | 172.17.4.0      0.0.0.0         255.255.255.0   U     425    0        0 kbr0
| 192.168.191.0   0.0.0.0         255.255.255.0   U     100    0        0 ens33 | 192.168.191.0   0.0.0.0         255.255.255.0   U     100    0        0 ens33
| [root@kube-master ~]#                                                         | [root@kube-worker1 ~]# 
| [root@kube-master ~]# docker run -ti --rm busybox sh                          | [root@kube-worker1 ~]# docker run -ti --rm busybox sh                  
| / # ifconfig                                                                  | / # ifconfig
| eth0      Link encap:Ethernet  HWaddr 02:42:AC:11:03:02                       | eth0      Link encap:Ethernet  HWaddr 02:42:AC:11:04:02  
|           inet addr:172.17.3.2  Bcast:0.0.0.0  Mask:255.255.255.0             |           inet addr:172.17.4.2  Bcast:0.0.0.0  Mask:255.255.255.0
|           inet6 addr: fe80::42:acff:fe11:302/64 Scope:Link                    |           inet6 addr: fe80::42:acff:fe11:402/64 Scope:Link
|           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1                  |           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
|           RX packets:23 errors:0 dropped:0 overruns:0 frame:0                 |           RX packets:16 errors:0 dropped:0 overruns:0 frame:0
|           TX packets:15 errors:0 dropped:0 overruns:0 carrier:0               |           TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
|           collisions:0 txqueuelen:0                                           |           collisions:0 txqueuelen:0 
|           RX bytes:1870 (1.8 KiB)  TX bytes:1222 (1.1 KiB)                    |           RX bytes:1296 (1.2 KiB)  TX bytes:648 (648.0 B)
|                                                                               | 
| lo        Link encap:Local Loopback                                           | lo        Link encap:Local Loopback  
|           inet addr:127.0.0.1  Mask:255.0.0.0                                 |           inet addr:127.0.0.1  Mask:255.0.0.0
|           inet6 addr: ::1/128 Scope:Host                                      |           inet6 addr: ::1/128 Scope:Host
|           UP LOOPBACK RUNNING  MTU:65536  Metric:1                            |           UP LOOPBACK RUNNING  MTU:65536  Metric:1
|           RX packets:0 errors:0 dropped:0 overruns:0 frame:0                  |           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
|           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0                |           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
|           collisions:0 txqueuelen:1                                           |           collisions:0 txqueuelen:1 
|           RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)                              |           RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
| / # ping 172.17.4.2                                                           | 
| PING 172.17.4.2 (172.17.4.2): 56 data bytes                                   | / # ping 172.17.3.2
| 64 bytes from 172.17.4.2: seq=0 ttl=62 time=2.598 ms                          | PING 172.17.3.2 (172.17.3.2): 56 data bytes
| 64 bytes from 172.17.4.2: seq=1 ttl=62 time=1.569 ms                          | 64 bytes from 172.17.3.2: seq=0 ttl=62 time=1.421 ms
| 64 bytes from 172.17.4.2: seq=2 ttl=62 time=1.194 ms                          | 64 bytes from 172.17.3.2: seq=1 ttl=62 time=1.446 ms
| ^C                                                                            | ^C
| --- 172.17.4.2 ping statistics ---                                            | --- 172.17.3.2 ping statistics ---
| 3 packets transmitted, 3 packets received, 0% packet loss                     | 2 packets transmitted, 2 packets received, 0% packet loss
| round-trip min/avg/max = 1.194/1.787/2.598 ms                                 | round-trip min/avg/max = 1.421/1.433/1.446 ms
| 
|-------------------------------------------------------------------------------|--------------------------------------------------------

效果还不错,什么都没有安装route两台机器的docker就互联互通了。二三台机器使用这种方式最省事的,并且理论上效率也是最高的。

其他参考

–END

Comments