]>
Commit | Line | Data |
---|---|---|
1e59de90 | 1 | # quay.io/ceph/keepalived |
2a845540 TL |
2 | |
3 | A small [ubi8-minimal](https://catalog.redhat.com/software/containers/registry/registry.access.redhat.com/repository/ubi8/ubi-minimal) based Docker container that provides a method of IP high availability via [keepalived](http://www.keepalived.org/) (VRRP failover), and optional Kubernetes API Server monitoring. If allowed to auto configure (default behaviour) it will automatically generate a unicast based failover configuration with a minimal amount of user supplied information. | |
4 | ||
5 | For specific information on Keepalived, please see the man page on [keepalived.conf](http://linux.die.net/man/5/keepalived.conf) or the [Keepalived User Guide](http://www.keepalived.org/pdf/UserGuide.pdf). | |
6 | ||
7 | ||
8 | ## Index | |
1e59de90 | 9 | - [quay.io/ceph/keepalived](#cephkeepalived) |
2a845540 TL |
10 | - [Index](#index) |
11 | - [Prerequisites](#prerequisites) | |
12 | - [Configuration](#configuration) | |
13 | - [Execution Control](#execution-control) | |
14 | - [Autoconfiguration Options](#autoconfiguration-options) | |
15 | - [Kubernetes Options](#kubernetes-options) | |
16 | - [Suggested Kubernetes Settings](#suggested-kubernetes-settings) | |
17 | - [Example Keepalived Configs](#example-keepalived-configs) | |
18 | - [Example Autogenerated Keepalived Master Config](#example-autogenerated-keepalived-master-config) | |
19 | - [Example Autogenerated Keepalived Backup Config](#example-autogenerated-keepalived-backup-config) | |
20 | - [Example Run Commands](#example-run-commands) | |
21 | - [Example Master Run Command](#example-master-run-command) | |
22 | - [Example Backup Run Command](#example-backup-run-command) | |
23 | ||
24 | ||
25 | ## Prerequisites | |
26 | ||
27 | Before attempting to deploy the keepalived container, the host must allow non local binding of ipv4 addresses. To do this, configure the sysctl tunable `net.ipv4.ip_nonlocal_bind=1`. | |
28 | ||
29 | In addition to enabling the nonlocal binds, the container must be run with both host networking (`--net=host`) and security setting CAP_NET_ADMIN (`--cap-add NET_ADMIN`) capability. These allow the container to manage the host's networking configuration, and this is essential to the function of keepalived. | |
30 | ||
31 | ||
32 | ## Configuration | |
33 | ### Execution Control | |
34 | ||
35 | | Variable | Default | | |
36 | |:---------------------:|:------------------------------------------------:| | |
37 | | `KEEPALIVED_AUTOCONF` | `true` | | |
38 | | `KEEPALIVED_CONF` | `/etc/keepalived/keepalived.conf` | | |
39 | | `KEEPALIVED_CMD` | `/usr/sbin/keepalived -n -l -f $KEEPALIVED_CONF` | | |
40 | | `KEEPALIVED_DEBUG` | `false` | | |
41 | ||
42 | * `KEEPALIVED_AUTOCONF` - Enables or disables the auto-configuration of keepalived. | |
43 | ||
44 | * `KEEPALIVED_CONF` - The path to the keepalived configuration file. | |
45 | ||
46 | * `KEEPALIVED_CMD` - The command called to execute keepalived. | |
47 | ||
48 | * `KEEPALIVED_DEBUG` - Enables or disables debug level logging for keepalived (adds `-D` to `KEEPALIVED_CMD`. | |
49 | ||
50 | ||
51 | ### Autoconfiguration Options | |
52 | ||
53 | | Variable | Default | | |
54 | |:-------------------------------------------:|:----------------------------------:| | |
55 | | `KEEPALIVED_ADVERT_INT` | `1` | | |
56 | | `KEEPALIVED_AUTH_PASS` | `pwd$KEEPALIVED_VIRTUAL_ROUTER_ID` | | |
57 | | `KEEPALIVED_INTERFACE` | `eth0` | | |
58 | | `KEEPALIVED_PRIORITY` | `200` | | |
59 | | `KEEPALIVED_STATE` | `MASTER` | | |
60 | | `KEEPALIVED_TRACK_INTERFACE_###` | | | |
61 | | `KEEPALIVED_UNICAST_SRC_IP` | | | |
62 | | `KEEPALIVED_UNICAST_PEER_###` | | | |
63 | | `KEEPALIVED_VIRTUAL_IPADDRESS_###` | | | |
64 | | `KEEPALIVED_VIRTUAL_IPADDRESS_EXCLUDED_###` | | | |
65 | | `KEEPALIVED_VIRTUAL_ROUTER_ID` | `1` | | |
66 | | `KEEPALIVED_KUBE_APISERVER_CHECK` | `false` | | |
67 | ||
68 | * `KEEPALIVED_ADVERT_INT` - The VRRP advertisement interval (in seconds). | |
69 | ||
70 | * `KEEPALIVED_AUTH_PASS` - A shared password used to authenticate each node in a VRRP group (**Note:** If password is longer than 8 characters, only the first 8 characters are used). | |
71 | ||
72 | * `KEEPALIVED_INTERFACE` - The host interface that keepalived will monitor and use for VRRP traffic. | |
73 | ||
74 | * `KEEPALIVED_PRIORITY` - Election value, the server configured with the highest priority will become the Master. | |
75 | ||
76 | * `KEEPALIVED_STATE` - Defines the server role as Master or Backup. (**Options:** `MASTER` or `BACKUP`). | |
77 | ||
78 | * `KEEPALIVED_TRACK_INTERFACE_###` - An interface that's state should be monitored (e.g. eth0). More than one can be supplied as long as the variable name ends in a number from 0-999. | |
79 | ||
80 | * `KEEPALIVED_UNICAST_SRC_IP` - The IP on the host that the keepalived daemon should bind to. **Note:** If not specified, it will be the first IP bound to the interface specified in `KEEPALIVED_INTERFACE`. | |
81 | ||
82 | * `KEEPALIVED_UNICAST_PEER_###` - An IP of a peer participating in the VRRP group. More tha one can be supplied as long as the variable name ends in a number from 0-999. | |
83 | ||
84 | * `KEEPALIVED_VIRTUAL_IPADDRESS_###` - An instance of an address that will be monitored and failed over from one host to another. These should be a quoted string in the form of: `<IPADDRESS>/<MASK> brd <BROADCAST_IP> dev <DEVICE> scope <SCOPE> label <LABEL>` At a minimum the ip address, mask and device should be specified e.g. `KEEPALIVED_VIRTUAL_IPADDRESS_1="10.10.0.2/24 dev eth0"`. More than one can be supplied as long as the variable name ends in a number from 0-999. **Note:** Keepalived has a hard limit of **20** addresses that can be monitored. More can be failed over with the monitored addresses via `KEEPALIVED_VIRTUAL_IPADDRESS_EXCLUDED_###`. | |
85 | ||
86 | ||
87 | * `KEEPALIVED_VIRTUAL_IPADDRESS_EXCLUDED_###` - An instance of an address that will be failed over with the monitored addresses supplied via `KEEPALIVED_VIRTUAL_IPADDRESS_###`. These should be a quoted string in the form of: `<IPADDRESS>/<MASK> brd <BROADCAST_IP> dev <DEVICE> scope <SCOPE> label <LABEL>` At a minimum the ip address, mask and device should be specified e.g. `KEEPALIVED_VIRTUAL_IPADDRESS_EXCLUDED_1="172.16.1.20/24 dev eth1"`. More than one can be supplied as long as the variable name ends in a number from 0-999. | |
88 | ||
89 | * `KEEPALIVED_VIRTUAL_ROUTER_ID` - A unique number from 0 to 255 that should identify the VRRP group. Master and Backup should have the same value. Multiple instances of keepalived can be run on the same host, but each pair **MUST** have a unique virtual router id. | |
90 | ||
91 | * `KEEPALIVED_KUBE_APISERVER_CHECK` - If enabled it configures a simple check script for the Kubernetes API-Server. For more information on this feature, please see the [Kubernetes Options](#kubernetes-options) section. | |
92 | ||
93 | ||
94 | ### Kubernetes Options | |
95 | ||
96 | ||
97 | | **Variable** | **Default** | | |
98 | |:-----------------------------:|:----------------------------------------------:| | |
99 | | `KUBE_APISERVER_ADDRESS` | parsed from `KEEPALIVED_VIRTUAL_IPADDRESS_###` | | |
100 | | `KUBE_APISERVER_PORT` | `6443` | | |
101 | | `KUBE_APISERVER_CHK_INTERVAL` | `3` | | |
102 | | `KUBE_APISERVER_CHK_FALL` | `10` | | |
103 | | `KUBE_APISERVER_CHK_RISE` | `2` | | |
104 | | `KUBE_APISERVER_CHK_WEIGHT` | `-50` | | |
105 | ||
106 | ||
107 | ||
108 | * `KUBE_APISERVER_ADDRESS` - The Virtual IP being used for the Kube API Server. If none is supplied, it is assumed to be the lowest numbered entry in the `KEEPALIVED_VIRTUAL_IPADDRESS_###` variables. | |
109 | ||
110 | * `KUBE_APISERVER_PORT` - The port to use in conjunction with the `KUBE_APISERVER_ADDRESS`. | |
111 | ||
112 | * `KUBE_APISERVER_CHK_INTERVAL` - The interval in seconds between calling the script. | |
113 | ||
114 | * `KUBE_APISERVER_CHK_FALL` - The number of consecutive non-zero script exits before setting the state to `FAULT`. | |
115 | ||
116 | * `KUBE_APISERVER_CHK_RISE` - The number of consecutive zero script exits before exiting the `FAULT` state. | |
117 | ||
118 | * `KUBE_APISERVER_CHK_WEIGHT` - The weight to apply to the priority when the service enters the `FAULT` state. | |
119 | ||
120 | ||
121 | ||
122 | --- | |
123 | ||
124 | ### Suggested Kubernetes Settings | |
125 | ||
126 | Assuming there are three nodes running the kube-apiserver, you cannot rely on setting just the`KEEPALIVED_STATE` parameter to manage failover across the nodes. | |
127 | ||
128 | To manage kube-apiserver failover, enable the healthcheck option with `KEEPALIVED_KUBE_APISERVER_CHECK`, and set the `KEEPALIVED_PRIORITY` manually for the three instances. | |
129 | ||
130 | | **Node** | **Priority** | | |
131 | |:--------:|:------------:| | |
132 | | node-01 | 200 | | |
133 | | node-02 | 190 | | |
134 | | node-03 | 180 | | |
135 | ||
136 | With the default weight of `-50`, if `node-01` has an issue, it's priority will drop to `150` and allow `node-02` to take over, the same is repeated if `node-02` has a failure dropping it's weight to `140` and `node-03` takes over. | |
137 | ||
138 | Recovery occurs in the same order with the system with the highest priority being promoted to master. | |
139 | ||
140 | ### Example Keepalived Configs | |
141 | ||
142 | ##### Example Autogenerated Keepalived Master Config | |
143 | ``` | |
144 | vrrp_instance MAIN { | |
145 | state MASTER | |
146 | interface eth0 | |
147 | virtual_router_id 2 | |
148 | priority 200 | |
149 | advert_int 1 | |
150 | unicast_src_ip 10.10.0.21 | |
151 | unicast_peer { | |
152 | 10.10.0.22 | |
153 | } | |
154 | authentication { | |
155 | auth_type PASS | |
156 | auth_pass pwd1 | |
157 | } | |
158 | virtual_ipaddress { | |
159 | 10.10.0.2/24 dev eth0 | |
160 | } | |
161 | virtual_ipaddress_excluded { | |
162 | 172.16.1.20/24 dev eth1 | |
163 | } | |
164 | track_interface { | |
165 | eth0 | |
166 | eth1 | |
167 | } | |
168 | } | |
169 | ``` | |
170 | ||
171 | ##### Example Autogenerated Keepalived Backup Config | |
172 | ``` | |
173 | vrrp_instance MAIN { | |
174 | state BACKUP | |
175 | interface eth0 | |
176 | virtual_router_id 2 | |
177 | priority 100 | |
178 | advert_int 1 | |
179 | unicast_src_ip 10.10.0.22 | |
180 | unicast_peer { | |
181 | 10.10.0.21 | |
182 | } | |
183 | authentication { | |
184 | auth_type PASS | |
185 | auth_pass pwd1 | |
186 | } | |
187 | virtual_ipaddress { | |
188 | 10.10.0.2/24 dev eth0 | |
189 | } | |
190 | virtual_ipaddress_excluded { | |
191 | 172.16.1.20/24 dev eth1 | |
192 | } | |
193 | track_interface { | |
194 | eth0 | |
195 | eth1 | |
196 | } | |
197 | } | |
198 | ||
199 | ``` | |
200 | ||
201 | ||
202 | ## Example Run Commands | |
203 | ##### Example Master Run Command | |
204 | ```bash | |
205 | docker run -d --net=host --cap-add NET_ADMIN \ | |
206 | -e KEEPALIVED_AUTOCONF=true \ | |
207 | -e KEEPALIVED_STATE=MASTER \ | |
208 | -e KEEPALIVED_INTERFACE=eth0 \ | |
209 | -e KEEPALIVED_VIRTUAL_ROUTER_ID=2 \ | |
210 | -e KEEPALIVED_UNICAST_SRC_IP=10.10.0.21 \ | |
211 | -e KEEPALIVED_UNICAST_PEER_0=10.10.0.22 \ | |
212 | -e KEEPALIVED_TRACK_INTERFACE_1=eth0 \ | |
1e59de90 | 213 | -e KEEPALIVED_TRACK_INTERFACE_2=eth1 \ |
2a845540 TL |
214 | -e KEEPALIVED_VIRTUAL_IPADDRESS_1="10.10.0.3/24 dev eth0" \ |
215 | -e KEEPALIVED_VIRTUAL_IPADDRESS_EXCLUDED_1="172.16.1.20/24 dev eth1" \ | |
1e59de90 | 216 | quay.io/ceph/keepalived |
2a845540 TL |
217 | ``` |
218 | ||
219 | ##### Example Backup Run Command | |
220 | ```bash | |
221 | docker run -d --net=host --cap-add NET_ADMIN \ | |
222 | -e KEEPALIVED_AUTOCONF=true \ | |
223 | -e KEEPALIVED_STATE=BACKUP \ | |
224 | -e KEEPALIVED_INTERFACE=eth0 \ | |
225 | -e KEEPALIVED_VIRTUAL_ROUTER_ID=2 \ | |
226 | -e KEEPALIVED_UNICAST_SRC_IP=10.10.0.22 \ | |
227 | -e KEEPALIVED_UNICAST_PEER_0=10.10.0.21 \ | |
228 | -e KEEPALIVED_TRACK_INTERFACE_1=eth0 \ | |
1e59de90 | 229 | -e KEEPALIVED_TRACK_INTERFACE_2=eth1 \ |
2a845540 TL |
230 | -e KEEPALIVED_VIRTUAL_IPADDRESS_1="10.10.0.3/24 dev eth0" \ |
231 | -e KEEPALIVED_VIRTUAL_IPADDRESS_EXCLUDED_1="172.16.1.20/24 dev eth1" \ | |
1e59de90 | 232 | quay.io/ceph/keepalived |
2a845540 | 233 | ``` |