Documentation/networking/vrf.txt

   1 Virtual Routing and Forwarding (VRF)
   2 ====================================
   3 The VRF device combined with ip rules provides the ability to create virtual
   4 routing and forwarding domains (aka VRFs, VRF-lite to be specific) in the
   5 Linux network stack. One use case is the multi-tenancy problem where each
   6 tenant has their own unique routing tables and in the very least need
   7 different default gateways.
   8
   9 Processes can be "VRF aware" by binding a socket to the VRF device. Packets
  10 through the socket then use the routing table associated with the VRF
  11 device. An important feature of the VRF device implementation is that it
  12 impacts only Layer 3 and above so L2 tools (e.g., LLDP) are not affected
  13 (ie., they do not need to be run in each VRF). The design also allows
  14 the use of higher priority ip rules (Policy Based Routing, PBR) to take
  15 precedence over the VRF device rules directing specific traffic as desired.
  16
  17 In addition, VRF devices allow VRFs to be nested within namespaces. For
  18 example network namespaces provide separation of network interfaces at the
  19 device layer, VLANs on the interfaces within a namespace provide L2 separation
  20 and then VRF devices provide L3 separation.
  21
  22 Design
  23 ------
  24 A VRF device is created with an associated route table. Network interfaces
  25 are then enslaved to a VRF device:
  26
  27          +-----------------------------+
  28          |           vrf-blue          |  ===> route table 10
  29          +-----------------------------+
  30             |        |            |
  31          +------+ +------+     +-------------+
  32          | eth1 | | eth2 | ... |    bond1    |
  33          +------+ +------+     +-------------+
  34                                   |       |
  35                               +------+ +------+
  36                               | eth8 | | eth9 |
  37                               +------+ +------+
  38
  39 Packets received on an enslaved device and are switched to the VRF device
  40 in the IPv4 and IPv6 processing stacks giving the impression that packets
  41 flow through the VRF device. Similarly on egress routing rules are used to
  42 send packets to the VRF device driver before getting sent out the actual
  43 interface. This allows tcpdump on a VRF device to capture all packets into
  44 and out of the VRF as a whole.[1] Similarly, netfilter[2] and tc rules can be
  45 applied using the VRF device to specify rules that apply to the VRF domain
  46 as a whole.
  47
  48 [1] Packets in the forwarded state do not flow through the device, so those
  49     packets are not seen by tcpdump. Will revisit this limitation in a
  50     future release.
  51
  52 [2] Iptables on ingress supports PREROUTING with skb->dev set to the real
  53     ingress device and both INPUT and PREROUTING rules with skb->dev set to
  54     the VRF device. For egress POSTROUTING and OUTPUT rules can be written
  55     using either the VRF device or real egress device.
  56
  57 Setup
  58 -----
  59 1. VRF device is created with an association to a FIB table.
  60    e.g, ip link add vrf-blue type vrf table 10
  61         ip link set dev vrf-blue up
  62
  63 2. An l3mdev FIB rule directs lookups to the table associated with the device.
  64    A single l3mdev rule is sufficient for all VRFs. The VRF device adds the
  65    l3mdev rule for IPv4 and IPv6 when the first device is created with a
  66    default preference of 1000. Users may delete the rule if desired and add
  67    with a different priority or install per-VRF rules.
  68
  69    Prior to the v4.8 kernel iif and oif rules are needed for each VRF device:
  70        ip ru add oif vrf-blue table 10
  71        ip ru add iif vrf-blue table 10
  72
  73 3. Set the default route for the table (and hence default route for the VRF).
  74        ip route add table 10 unreachable default
  75
  76 4. Enslave L3 interfaces to a VRF device.
  77        ip link set dev eth1 master vrf-blue
  78
  79    Local and connected routes for enslaved devices are automatically moved to
  80    the table associated with VRF device. Any additional routes depending on
  81    the enslaved device are dropped and will need to be reinserted to the VRF
  82    FIB table following the enslavement.
  83
  84    The IPv6 sysctl option keep_addr_on_down can be enabled to keep IPv6 global
  85    addresses as VRF enslavement changes.
  86        sysctl -w net.ipv6.conf.all.keep_addr_on_down=1
  87
  88 5. Additional VRF routes are added to associated table.
  89        ip route add table 10 ...
  90
  91
  92 Applications
  93 ------------
  94 Applications that are to work within a VRF need to bind their socket to the
  95 VRF device:
  96
  97     setsockopt(sd, SOL_SOCKET, SO_BINDTODEVICE, dev, strlen(dev)+1);
  98
  99 or to specify the output device using cmsg and IP_PKTINFO.
 100
 101 TCP & UDP services running in the default VRF context (ie., not bound
 102 to any VRF device) can work across all VRF domains by enabling the
 103 tcp_l3mdev_accept and udp_l3mdev_accept sysctl options:
 104     sysctl -w net.ipv4.tcp_l3mdev_accept=1
 105     sysctl -w net.ipv4.udp_l3mdev_accept=1
 106
 107 netfilter rules on the VRF device can be used to limit access to services
 108 running in the default VRF context as well.
 109
 110 The default VRF does not have limited scope with respect to port bindings.
 111 That is, if a process does a wildcard bind to a port in the default VRF it
 112 owns the port across all VRF domains within the network namespace.
 113
 114 ################################################################################
 115
 116 Using iproute2 for VRFs
 117 =======================
 118 iproute2 supports the vrf keyword as of v4.7. For backwards compatibility this
 119 section lists both commands where appropriate -- with the vrf keyword and the
 120 older form without it.
 121
 122 1. Create a VRF
 123
 124    To instantiate a VRF device and associate it with a table:
 125        $ ip link add dev NAME type vrf table ID
 126
 127    As of v4.8 the kernel supports the l3mdev FIB rule where a single rule
 128    covers all VRFs. The l3mdev rule is created for IPv4 and IPv6 on first
 129    device create.
 130
 131 2. List VRFs
 132
 133    To list VRFs that have been created:
 134        $ ip [-d] link show type vrf
 135          NOTE: The -d option is needed to show the table id
 136
 137    For example:
 138    $ ip -d link show type vrf
 139    11: mgmt: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
 140        link/ether 72:b3:ba:91:e2:24 brd ff:ff:ff:ff:ff:ff promiscuity 0
 141        vrf table 1 addrgenmode eui64
 142    12: red: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
 143        link/ether b6:6f:6e:f6:da:73 brd ff:ff:ff:ff:ff:ff promiscuity 0
 144        vrf table 10 addrgenmode eui64
 145    13: blue: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
 146        link/ether 36:62:e8:7d:bb:8c brd ff:ff:ff:ff:ff:ff promiscuity 0
 147        vrf table 66 addrgenmode eui64
 148    14: green: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
 149        link/ether e6:28:b8:63:70:bb brd ff:ff:ff:ff:ff:ff promiscuity 0
 150        vrf table 81 addrgenmode eui64
 151
 152
 153    Or in brief output:
 154
 155    $ ip -br link show type vrf
 156    mgmt         UP             72:b3:ba:91:e2:24 <NOARP,MASTER,UP,LOWER_UP>
 157    red          UP             b6:6f:6e:f6:da:73 <NOARP,MASTER,UP,LOWER_UP>
 158    blue         UP             36:62:e8:7d:bb:8c <NOARP,MASTER,UP,LOWER_UP>
 159    green        UP             e6:28:b8:63:70:bb <NOARP,MASTER,UP,LOWER_UP>
 160
 161
 162 3. Assign a Network Interface to a VRF
 163
 164    Network interfaces are assigned to a VRF by enslaving the netdevice to a
 165    VRF device:
 166        $ ip link set dev NAME master NAME
 167
 168    On enslavement connected and local routes are automatically moved to the
 169    table associated with the VRF device.
 170
 171    For example:
 172    $ ip link set dev eth0 master mgmt
 173
 174
 175 4. Show Devices Assigned to a VRF
 176
 177    To show devices that have been assigned to a specific VRF add the master
 178    option to the ip command:
 179        $ ip link show vrf NAME
 180        $ ip link show master NAME
 181
 182    For example:
 183    $ ip link show vrf red
 184    3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP mode DEFAULT group default qlen 1000
 185        link/ether 02:00:00:00:02:02 brd ff:ff:ff:ff:ff:ff
 186    4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP mode DEFAULT group default qlen 1000
 187        link/ether 02:00:00:00:02:03 brd ff:ff:ff:ff:ff:ff
 188    7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master red state DOWN mode DEFAULT group default qlen 1000
 189        link/ether 02:00:00:00:02:06 brd ff:ff:ff:ff:ff:ff
 190
 191
 192    Or using the brief output:
 193    $ ip -br link show vrf red
 194    eth1             UP             02:00:00:00:02:02 <BROADCAST,MULTICAST,UP,LOWER_UP>
 195    eth2             UP             02:00:00:00:02:03 <BROADCAST,MULTICAST,UP,LOWER_UP>
 196    eth5             DOWN           02:00:00:00:02:06 <BROADCAST,MULTICAST>
 197
 198
 199 5. Show Neighbor Entries for a VRF
 200
 201    To list neighbor entries associated with devices enslaved to a VRF device
 202    add the master option to the ip command:
 203        $ ip [-6] neigh show vrf NAME
 204        $ ip [-6] neigh show master NAME
 205
 206    For example:
 207    $  ip neigh show vrf red
 208    10.2.1.254 dev eth1 lladdr a6:d9:c7:4f:06:23 REACHABLE
 209    10.2.2.254 dev eth2 lladdr 5e:54:01:6a:ee:80 REACHABLE
 210
 211    $ ip -6 neigh show vrf red
 212    2002:1::64 dev eth1 lladdr a6:d9:c7:4f:06:23 REACHABLE
 213
 214
 215 6. Show Addresses for a VRF
 216
 217    To show addresses for interfaces associated with a VRF add the master
 218    option to the ip command:
 219        $ ip addr show vrf NAME
 220        $ ip addr show master NAME
 221
 222    For example:
 223    $ ip addr show vrf red
 224    3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP group default qlen 1000
 225        link/ether 02:00:00:00:02:02 brd ff:ff:ff:ff:ff:ff
 226        inet 10.2.1.2/24 brd 10.2.1.255 scope global eth1
 227           valid_lft forever preferred_lft forever
 228        inet6 2002:1::2/120 scope global
 229           valid_lft forever preferred_lft forever
 230        inet6 fe80::ff:fe00:202/64 scope link
 231           valid_lft forever preferred_lft forever
 232    4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP group default qlen 1000
 233        link/ether 02:00:00:00:02:03 brd ff:ff:ff:ff:ff:ff
 234        inet 10.2.2.2/24 brd 10.2.2.255 scope global eth2
 235           valid_lft forever preferred_lft forever
 236        inet6 2002:2::2/120 scope global
 237           valid_lft forever preferred_lft forever
 238        inet6 fe80::ff:fe00:203/64 scope link
 239           valid_lft forever preferred_lft forever
 240    7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master red state DOWN group default qlen 1000
 241        link/ether 02:00:00:00:02:06 brd ff:ff:ff:ff:ff:ff
 242
 243    Or in brief format:
 244    $ ip -br addr show vrf red
 245    eth1             UP             10.2.1.2/24 2002:1::2/120 fe80::ff:fe00:202/64
 246    eth2             UP             10.2.2.2/24 2002:2::2/120 fe80::ff:fe00:203/64
 247    eth5             DOWN
 248
 249
 250 7. Show Routes for a VRF
 251
 252    To show routes for a VRF use the ip command to display the table associated
 253    with the VRF device:
 254        $ ip [-6] route show vrf NAME
 255        $ ip [-6] route show table ID
 256
 257    For example:
 258    $ ip route show vrf red
 259    prohibit default
 260    broadcast 10.2.1.0 dev eth1  proto kernel  scope link  src 10.2.1.2
 261    10.2.1.0/24 dev eth1  proto kernel  scope link  src 10.2.1.2
 262    local 10.2.1.2 dev eth1  proto kernel  scope host  src 10.2.1.2
 263    broadcast 10.2.1.255 dev eth1  proto kernel  scope link  src 10.2.1.2
 264    broadcast 10.2.2.0 dev eth2  proto kernel  scope link  src 10.2.2.2
 265    10.2.2.0/24 dev eth2  proto kernel  scope link  src 10.2.2.2
 266    local 10.2.2.2 dev eth2  proto kernel  scope host  src 10.2.2.2
 267    broadcast 10.2.2.255 dev eth2  proto kernel  scope link  src 10.2.2.2
 268
 269    $ ip -6 route show vrf red
 270    local 2002:1:: dev lo  proto none  metric 0  pref medium
 271    local 2002:1::2 dev lo  proto none  metric 0  pref medium
 272    2002:1::/120 dev eth1  proto kernel  metric 256  pref medium
 273    local 2002:2:: dev lo  proto none  metric 0  pref medium
 274    local 2002:2::2 dev lo  proto none  metric 0  pref medium
 275    2002:2::/120 dev eth2  proto kernel  metric 256  pref medium
 276    local fe80:: dev lo  proto none  metric 0  pref medium
 277    local fe80:: dev lo  proto none  metric 0  pref medium
 278    local fe80::ff:fe00:202 dev lo  proto none  metric 0  pref medium
 279    local fe80::ff:fe00:203 dev lo  proto none  metric 0  pref medium
 280    fe80::/64 dev eth1  proto kernel  metric 256  pref medium
 281    fe80::/64 dev eth2  proto kernel  metric 256  pref medium
 282    ff00::/8 dev red  metric 256  pref medium
 283    ff00::/8 dev eth1  metric 256  pref medium
 284    ff00::/8 dev eth2  metric 256  pref medium
 285
 286
 287 8. Route Lookup for a VRF
 288
 289    A test route lookup can be done for a VRF:
 290        $ ip [-6] route get vrf NAME ADDRESS
 291        $ ip [-6] route get oif NAME ADDRESS
 292
 293    For example:
 294    $ ip route get 10.2.1.40 vrf red
 295    10.2.1.40 dev eth1  table red  src 10.2.1.2
 296        cache
 297
 298    $ ip -6 route get 2002:1::32 vrf red
 299    2002:1::32 from :: dev eth1  table red  proto kernel  src 2002:1::2  metric 256  pref medium
 300
 301
 302 9. Removing Network Interface from a VRF
 303
 304    Network interfaces are removed from a VRF by breaking the enslavement to
 305    the VRF device:
 306        $ ip link set dev NAME nomaster
 307
 308    Connected routes are moved back to the default table and local entries are
 309    moved to the local table.
 310
 311    For example:
 312    $ ip link set dev eth0 nomaster
 313
 314 --------------------------------------------------------------------------------
 315
 316 Commands used in this example:
 317
 318 cat >> /etc/iproute2/rt_tables.d/vrf.conf <<EOF
 319 1  mgmt
 320 10 red
 321 66 blue
 322 81 green
 323 EOF
 324
 325 function vrf_create
 326 {
 327     VRF=$1
 328     TBID=$2
 329
 330     # create VRF device
 331     ip link add ${VRF} type vrf table ${TBID}
 332
 333     if [ "${VRF}" != "mgmt" ]; then
 334         ip route add table ${TBID} unreachable default
 335     fi
 336     ip link set dev ${VRF} up
 337 }
 338
 339 vrf_create mgmt 1
 340 ip link set dev eth0 master mgmt
 341
 342 vrf_create red 10
 343 ip link set dev eth1 master red
 344 ip link set dev eth2 master red
 345 ip link set dev eth5 master red
 346
 347 vrf_create blue 66
 348 ip link set dev eth3 master blue
 349
 350 vrf_create green 81
 351 ip link set dev eth4 master green
 352
 353
 354 Interface addresses from /etc/network/interfaces:
 355 auto eth0
 356 iface eth0 inet static
 357       address 10.0.0.2
 358       netmask 255.255.255.0
 359       gateway 10.0.0.254
 360
 361 iface eth0 inet6 static
 362       address 2000:1::2
 363       netmask 120
 364
 365 auto eth1
 366 iface eth1 inet static
 367       address 10.2.1.2
 368       netmask 255.255.255.0
 369
 370 iface eth1 inet6 static
 371       address 2002:1::2
 372       netmask 120
 373
 374 auto eth2
 375 iface eth2 inet static
 376       address 10.2.2.2
 377       netmask 255.255.255.0
 378
 379 iface eth2 inet6 static
 380       address 2002:2::2
 381       netmask 120
 382
 383 auto eth3
 384 iface eth3 inet static
 385       address 10.2.3.2
 386       netmask 255.255.255.0
 387
 388 iface eth3 inet6 static
 389       address 2002:3::2
 390       netmask 120
 391
 392 auto eth4
 393 iface eth4 inet static
 394       address 10.2.4.2
 395       netmask 255.255.255.0
 396
 397 iface eth4 inet6 static
 398       address 2002:4::2
 399       netmask 120