Overview of the Zebra Protocol
==============================
-The Zebra protocol is used by protocol daemons to communicate with the
-**zebra** daemon.
-
-Each protocol daemon may request and send information to and from the **zebra**
-daemon such as interface states, routing state, nexthop-validation, and so on.
-Protocol daemons may also install routes with **zebra**. The **zebra** daemon
-manages which routes are installed into the forwarding table with the kernel.
+The Zebra protocol (or ``ZAPI``) is used by protocol daemons to
+communicate with the **zebra** daemon.
+
+Each protocol daemon may request and send information to and from the
+**zebra** daemon such as interface states, routing state,
+nexthop-validation, and so on. Protocol daemons may also install
+routes with **zebra**. The **zebra** daemon manages which routes are
+installed into the forwarding table with the kernel. Some daemons use
+more than one ZAPI connection. This is supported: each ZAPI session is
+identified by a tuple of: ``{protocol, instance, session_id}``. LDPD
+is an example: it uses a second, synchronous ZAPI session to manage
+label blocks. The default value for ``session_id`` is zero; daemons
+who use multiple ZAPI sessions must assign unique values to the
+sessions' ids.
The Zebra protocol is a streaming protocol, with a common header. Version 0
lacks a version field and is implicitly versioned. Version 1 and all subsequent
+------------------------------------+-------+
| ZEBRA_INTERFACE_SET_MASTER | 6 |
+------------------------------------+-------+
-| ZEBRA_ROUTE_ADD | 7 |
+| ZEBRA_INTERFACE_SET_PROTODOWN | 7 |
++------------------------------------+-------+
+| ZEBRA_ROUTE_ADD | 8 |
++------------------------------------+-------+
+| ZEBRA_ROUTE_DELETE | 9 |
++------------------------------------+-------+
+| ZEBRA_ROUTE_NOTIFY_OWNER | 10 |
++------------------------------------+-------+
+| ZEBRA_REDISTRIBUTE_ADD | 11 |
++------------------------------------+-------+
+| ZEBRA_REDISTRIBUTE_DELETE | 12 |
++------------------------------------+-------+
+| ZEBRA_REDISTRIBUTE_DEFAULT_ADD | 13 |
++------------------------------------+-------+
+| ZEBRA_REDISTRIBUTE_DEFAULT_DELETE | 14 |
++------------------------------------+-------+
+| ZEBRA_ROUTER_ID_ADD | 15 |
++------------------------------------+-------+
+| ZEBRA_ROUTER_ID_DELETE | 16 |
++------------------------------------+-------+
+| ZEBRA_ROUTER_ID_UPDATE | 17 |
++------------------------------------+-------+
+| ZEBRA_HELLO | 18 |
++------------------------------------+-------+
+| ZEBRA_CAPABILITIES | 19 |
++------------------------------------+-------+
+| ZEBRA_NEXTHOP_REGISTER | 20 |
++------------------------------------+-------+
+| ZEBRA_NEXTHOP_UNREGISTER | 21 |
++------------------------------------+-------+
+| ZEBRA_NEXTHOP_UPDATE | 22 |
++------------------------------------+-------+
+| ZEBRA_INTERFACE_NBR_ADDRESS_ADD | 23 |
++------------------------------------+-------+
+| ZEBRA_INTERFACE_NBR_ADDRESS_DELETE | 24 |
+------------------------------------+-------+
-| ZEBRA_ROUTE_DELETE | 8 |
+| ZEBRA_INTERFACE_BFD_DEST_UPDATE | 25 |
+------------------------------------+-------+
-| ZEBRA_ROUTE_NOTIFY_OWNER | 9 |
+| ZEBRA_IMPORT_ROUTE_REGISTER | 26 |
+------------------------------------+-------+
-| ZEBRA_REDISTRIBUTE_ADD | 10 |
+| ZEBRA_IMPORT_ROUTE_UNREGISTER | 27 |
+------------------------------------+-------+
-| ZEBRA_REDISTRIBUTE_DELETE | 11 |
+| ZEBRA_IMPORT_CHECK_UPDATE | 28 |
+------------------------------------+-------+
-| ZEBRA_REDISTRIBUTE_DEFAULT_ADD | 12 |
+| ZEBRA_BFD_DEST_REGISTER | 29 |
+------------------------------------+-------+
-| ZEBRA_REDISTRIBUTE_DEFAULT_DELETE | 13 |
+| ZEBRA_BFD_DEST_DEREGISTER | 30 |
+------------------------------------+-------+
-| ZEBRA_ROUTER_ID_ADD | 14 |
+| ZEBRA_BFD_DEST_UPDATE | 31 |
+------------------------------------+-------+
-| ZEBRA_ROUTER_ID_DELETE | 15 |
+| ZEBRA_BFD_DEST_REPLAY | 32 |
+------------------------------------+-------+
-| ZEBRA_ROUTER_ID_UPDATE | 16 |
+| ZEBRA_REDISTRIBUTE_ROUTE_ADD | 33 |
+------------------------------------+-------+
-| ZEBRA_HELLO | 17 |
+| ZEBRA_REDISTRIBUTE_ROUTE_DEL | 34 |
+------------------------------------+-------+
-| ZEBRA_CAPABILITIES | 18 |
+| ZEBRA_VRF_UNREGISTER | 35 |
+------------------------------------+-------+
-| ZEBRA_NEXTHOP_REGISTER | 19 |
+| ZEBRA_VRF_ADD | 36 |
+------------------------------------+-------+
-| ZEBRA_NEXTHOP_UNREGISTER | 20 |
+| ZEBRA_VRF_DELETE | 37 |
+------------------------------------+-------+
-| ZEBRA_NEXTHOP_UPDATE | 21 |
+| ZEBRA_VRF_LABEL | 38 |
+------------------------------------+-------+
-| ZEBRA_INTERFACE_NBR_ADDRESS_ADD | 22 |
+| ZEBRA_INTERFACE_VRF_UPDATE | 39 |
+------------------------------------+-------+
-| ZEBRA_INTERFACE_NBR_ADDRESS_DELETE | 23 |
+| ZEBRA_BFD_CLIENT_REGISTER | 40 |
+------------------------------------+-------+
-| ZEBRA_INTERFACE_BFD_DEST_UPDATE | 24 |
+| ZEBRA_BFD_CLIENT_DEREGISTER | 41 |
+------------------------------------+-------+
-| ZEBRA_IMPORT_ROUTE_REGISTER | 25 |
+| ZEBRA_INTERFACE_ENABLE_RADV | 42 |
+------------------------------------+-------+
-| ZEBRA_IMPORT_ROUTE_UNREGISTER | 26 |
+| ZEBRA_INTERFACE_DISABLE_RADV | 43 |
+------------------------------------+-------+
-| ZEBRA_IMPORT_CHECK_UPDATE | 27 |
+| ZEBRA_IPV4_NEXTHOP_LOOKUP_MRIB | 44 |
+------------------------------------+-------+
-| ZEBRA_BFD_DEST_REGISTER | 28 |
+| ZEBRA_INTERFACE_LINK_PARAMS | 45 |
+------------------------------------+-------+
-| ZEBRA_BFD_DEST_DEREGISTER | 29 |
+| ZEBRA_MPLS_LABELS_ADD | 46 |
+------------------------------------+-------+
-| ZEBRA_BFD_DEST_UPDATE | 30 |
+| ZEBRA_MPLS_LABELS_DELETE | 47 |
+------------------------------------+-------+
-| ZEBRA_BFD_DEST_REPLAY | 31 |
+| ZEBRA_MPLS_LABELS_REPLACE | 48 |
+------------------------------------+-------+
-| ZEBRA_REDISTRIBUTE_ROUTE_ADD | 32 |
+| ZEBRA_IPMR_ROUTE_STATS | 49 |
+------------------------------------+-------+
-| ZEBRA_REDISTRIBUTE_ROUTE_DEL | 33 |
+| ZEBRA_LABEL_MANAGER_CONNECT | 50 |
+------------------------------------+-------+
-| ZEBRA_VRF_UNREGISTER | 34 |
+| ZEBRA_LABEL_MANAGER_CONNECT_ASYNC | 51 |
+------------------------------------+-------+
-| ZEBRA_VRF_ADD | 35 |
+| ZEBRA_GET_LABEL_CHUNK | 52 |
+------------------------------------+-------+
-| ZEBRA_VRF_DELETE | 36 |
+| ZEBRA_RELEASE_LABEL_CHUNK | 53 |
+------------------------------------+-------+
-| ZEBRA_VRF_LABEL | 37 |
+| ZEBRA_FEC_REGISTER | 54 |
+------------------------------------+-------+
-| ZEBRA_INTERFACE_VRF_UPDATE | 38 |
+| ZEBRA_FEC_UNREGISTER | 55 |
+------------------------------------+-------+
-| ZEBRA_BFD_CLIENT_REGISTER | 39 |
+| ZEBRA_FEC_UPDATE | 56 |
+------------------------------------+-------+
-| ZEBRA_BFD_CLIENT_DEREGISTER | 40 |
+| ZEBRA_ADVERTISE_DEFAULT_GW | 57 |
+------------------------------------+-------+
-| ZEBRA_INTERFACE_ENABLE_RADV | 41 |
+| ZEBRA_ADVERTISE_SVI_MACIP | 58 |
+------------------------------------+-------+
-| ZEBRA_INTERFACE_DISABLE_RADV | 42 |
+| ZEBRA_ADVERTISE_SUBNET | 59 |
+------------------------------------+-------+
-| ZEBRA_IPV3_NEXTHOP_LOOKUP_MRIB | 44 |
+| ZEBRA_ADVERTISE_ALL_VNI | 60 |
+------------------------------------+-------+
-| ZEBRA_INTERFACE_LINK_PARAMS | 44 |
+| ZEBRA_LOCAL_ES_ADD | 61 |
+------------------------------------+-------+
-| ZEBRA_MPLS_LABELS_ADD | 45 |
+| ZEBRA_LOCAL_ES_DEL | 62 |
+------------------------------------+-------+
-| ZEBRA_MPLS_LABELS_DELETE | 46 |
+| ZEBRA_VNI_ADD | 63 |
+------------------------------------+-------+
-| ZEBRA_IPMR_ROUTE_STATS | 47 |
+| ZEBRA_VNI_DEL | 64 |
+------------------------------------+-------+
-| ZEBRA_LABEL_MANAGER_CONNECT | 48 |
+| ZEBRA_L3VNI_ADD | 65 |
+------------------------------------+-------+
-| ZEBRA_LABEL_MANAGER_CONNECT_ASYNC | 49 |
+| ZEBRA_L3VNI_DEL | 66 |
+------------------------------------+-------+
-| ZEBRA_GET_LABEL_CHUNK | 50 |
+| ZEBRA_REMOTE_VTEP_ADD | 67 |
+------------------------------------+-------+
-| ZEBRA_RELEASE_LABEL_CHUNK | 51 |
+| ZEBRA_REMOTE_VTEP_DEL | 68 |
+------------------------------------+-------+
-| ZEBRA_FEC_REGISTER | 52 |
+| ZEBRA_MACIP_ADD | 69 |
+------------------------------------+-------+
-| ZEBRA_FEC_UNREGISTER | 53 |
+| ZEBRA_MACIP_DEL | 70 |
+------------------------------------+-------+
-| ZEBRA_FEC_UPDATE | 54 |
+| ZEBRA_IP_PREFIX_ROUTE_ADD | 71 |
+------------------------------------+-------+
-| ZEBRA_ADVERTISE_DEFAULT_GW | 55 |
+| ZEBRA_IP_PREFIX_ROUTE_DEL | 72 |
+------------------------------------+-------+
-| ZEBRA_ADVERTISE_SUBNET | 56 |
+| ZEBRA_REMOTE_MACIP_ADD | 73 |
+------------------------------------+-------+
-| ZEBRA_ADVERTISE_ALL_VNI | 57 |
+| ZEBRA_REMOTE_MACIP_DEL | 74 |
+------------------------------------+-------+
-| ZEBRA_LOCAL_ES_ADD | 58 |
+| ZEBRA_DUPLICATE_ADDR_DETECTION | 75 |
+------------------------------------+-------+
-| ZEBRA_LOCAL_ES_DEL | 59 |
+| ZEBRA_PW_ADD | 76 |
+------------------------------------+-------+
-| ZEBRA_VNI_ADD | 60 |
+| ZEBRA_PW_DELETE | 77 |
+------------------------------------+-------+
-| ZEBRA_VNI_DEL | 61 |
+| ZEBRA_PW_SET | 78 |
+------------------------------------+-------+
-| ZEBRA_L2VNI_ADD | 63 |
+| ZEBRA_PW_UNSET | 79 |
+------------------------------------+-------+
-| ZEBRA_L2VNI_DEL | 64 |
+| ZEBRA_PW_STATUS_UPDATE | 80 |
+------------------------------------+-------+
-| ZEBRA_REMOTE_VTEP_ADD | 64 |
+| ZEBRA_RULE_ADD | 81 |
+------------------------------------+-------+
-| ZEBRA_REMOTE_VTEP_DEL | 65 |
+| ZEBRA_RULE_DELETE | 82 |
+------------------------------------+-------+
-| ZEBRA_MACIP_ADD | 66 |
+| ZEBRA_RULE_NOTIFY_OWNER | 83 |
+------------------------------------+-------+
-| ZEBRA_MACIP_DEL | 67 |
+| ZEBRA_TABLE_MANAGER_CONNECT | 84 |
+------------------------------------+-------+
-| ZEBRA_IP_PREFIX_ROUTE_ADD | 68 |
+| ZEBRA_GET_TABLE_CHUNK | 85 |
+------------------------------------+-------+
-| ZEBRA_IP_PREFIX_ROUTE_DEL | 69 |
+| ZEBRA_RELEASE_TABLE_CHUNK | 86 |
+------------------------------------+-------+
-| ZEBRA_REMOTE_MACIP_ADD | 70 |
+| ZEBRA_IPSET_CREATE | 87 |
+------------------------------------+-------+
-| ZEBRA_REMOTE_MACIP_DEL | 71 |
+| ZEBRA_IPSET_DESTROY | 88 |
+------------------------------------+-------+
-| ZEBRA_PW_ADD | 72 |
+| ZEBRA_IPSET_ENTRY_ADD | 89 |
+------------------------------------+-------+
-| ZEBRA_PW_DELETE | 73 |
+| ZEBRA_IPSET_ENTRY_DELETE | 90 |
+------------------------------------+-------+
-| ZEBRA_PW_SET | 74 |
+| ZEBRA_IPSET_NOTIFY_OWNER | 91 |
+------------------------------------+-------+
-| ZEBRA_PW_UNSET | 75 |
+| ZEBRA_IPSET_ENTRY_NOTIFY_OWNER | 92 |
+------------------------------------+-------+
-| ZEBRA_PW_STATUS_UPDATE | 76 |
+| ZEBRA_IPTABLE_ADD | 93 |
+------------------------------------+-------+
-| ZEBRA_RULE_ADD | 77 |
+| ZEBRA_IPTABLE_DELETE | 94 |
+------------------------------------+-------+
-| ZEBRA_RULE_DELETE | 78 |
+| ZEBRA_IPTABLE_NOTIFY_OWNER | 95 |
+------------------------------------+-------+
-| ZEBRA_RULE_NOTIFY_OWNER | 79 |
+| ZEBRA_VXLAN_FLOOD_CONTROL | 96 |
+------------------------------------+-------+
-| ZEBRA_TABLE_MANAGER_CONNECT | 80 |
+| ZEBRA_VXLAN_SG_ADD | 97 |
+------------------------------------+-------+
-| ZEBRA_GET_TABLE_CHUNK | 81 |
+| ZEBRA_VXLAN_SG_DEL | 98 |
+------------------------------------+-------+
-| ZEBRA_RELEASE_TABLE_CHUNK | 82 |
+| ZEBRA_VXLAN_SG_REPLAY | 99 |
+------------------------------------+-------+
-| ZEBRA_IPSET_CREATE | 83 |
+| ZEBRA_MLAG_PROCESS_UP | 100 |
+------------------------------------+-------+
-| ZEBRA_IPSET_DESTROY | 84 |
+| ZEBRA_MLAG_PROCESS_DOWN | 101 |
+------------------------------------+-------+
-| ZEBRA_IPSET_ENTRY_ADD | 85 |
+| ZEBRA_MLAG_CLIENT_REGISTER | 102 |
+------------------------------------+-------+
-| ZEBRA_IPSET_ENTRY_DELETE | 86 |
+| ZEBRA_MLAG_CLIENT_UNREGISTER | 103 |
+------------------------------------+-------+
-| ZEBRA_IPSET_NOTIFY_OWNER | 87 |
+| ZEBRA_MLAG_FORWARD_MSG | 104 |
+------------------------------------+-------+
-| ZEBRA_IPSET_ENTRY_NOTIFY_OWNER | 88 |
+| ZEBRA_ERROR | 105 |
+------------------------------------+-------+
-| ZEBRA_IPTABLE_ADD | 89 |
+| ZEBRA_CLIENT_CAPABILITIES | 106 |
+------------------------------------+-------+
-| ZEBRA_IPTABLE_DELETE | 90 |
+| ZEBRA_OPAQUE_MESSAGE | 107 |
+------------------------------------+-------+
-| ZEBRA_IPTABLE_NOTIFY_OWNER | 91 |
+| ZEBRA_OPAQUE_REGISTER | 108 |
+------------------------------------+-------+
-| ZEBRA_VXLAN_FLOOD_CONTROL | 92 |
+| ZEBRA_OPAQUE_UNREGISTER | 109 |
+------------------------------------+-------+
+| ZEBRA_NEIGH_DISCOVER | 110 |
++------------------------------------+-------+
+
+Dataplane batching
+==================
+
+Dataplane batching is an optimization feature that reduces the processing
+time involved in the user space to kernel space transition for every message we
+want to send.
+
+Design
+-----------
+
+With our dataplane abstraction, we create a queue of dataplane context objects
+for the messages we want to send to the kernel. In a separate pthread, we
+loop over this queue and send the context objects to the appropriate
+dataplane. A batching enhancement tightly integrates with the dataplane
+context objects so they are able to be batch sent to dataplanes that support
+it.
+
+There is one main change in the dataplane code. It does not call
+kernel-dependent functions one-by-one, but instead it hands a list of work down
+to the kernel level for processing.
+
+Netlink
+^^^^^^^
+
+At the moment, this is the only dataplane that allows for batch sending
+messages to it.
+
+When messages must be sent to the kernel, they are consecutively added
+to the batch represented by the `struct nl_batch`. Context objects are firstly
+encoded to their binary representation. All the encoding functions use the same
+interface: take a context object, a buffer and a size of the buffer as an
+argument. It is important that they should handle a situation in which a message
+wouldn't fit in the buffer and return a proper error. To achieve a zero-copy
+(in the user space only) messages are encoded to the same buffer which will
+be passed to the kernel. Hence, we can theoretically hit the boundary of the
+buffer.
+
+Messages stored in the batch are sent if one of the conditions occurs:
+
+- When an encoding function returns the buffer overflow error. The context
+ object that caused this error is re-added to the new, empty batch.
+
+- When the size of the batch hits certain limit.
+
+- When the namespace of a currently being processed context object is
+ different from all the previous ones. They have to be sent through
+ distinct sockets, so the messages cannot share the same buffer.
+
+- After the last message from the list is processed.
+
+As mentioned earlier, there is a special threshold which is smaller than
+the size of the underlying buffer. It prevents the overflow error and thus
+eliminates the case, in which a message is encoded twice.
+
+The buffer used in the batching is global, since allocating that big amount of
+memory every time wouldn't be most effective. However, its size can be changed
+dynamically, using hidden vtysh command:
+``zebra kernel netlink batch-tx-buf (1-1048576) (1-1048576)``. This feature is
+only used in tests and shouldn't be utilized in any other place.
+
+For every failed message in the batch, the kernel responds with an error
+message. Error messages are kept in the same order as they were sent, so parsing the
+response is straightforward. We use the two pointer technique to match
+requests with responses and then set appropriate status of dataplane context
+objects. There is also a global receive buffer and it is assumed that whatever
+the kernel sends it will fit in this buffer. The payload of netlink error messages
+consists of a error code and the original netlink message of the request, so
+the batch response won't be bigger than the batch request increased by
+some space for the headers.
projects / mirror_frr.git / blobdiff