8 One of the most frequent decisions to make while writing code for FRR is what
9 to log, what level to log it at, and when to log it. Here is a list of
10 recommendations for these decisions.
16 ``printfrr()`` is FRR's modified version of ``printf()``, designed to make
17 life easier when printing nontrivial datastructures. The following variants
20 .. c:function:: ssize_t snprintfrr(char *buf, size_t len, const char *fmt, ...)
21 .. c:function:: ssize_t vsnprintfrr(char *buf, size_t len, const char *fmt, va_list)
23 These correspond to ``snprintf``/``vsnprintf``. If you pass NULL for buf
24 or 0 for len, no output is written but the return value is still calculated.
26 The return value is always the full length of the output, unconstrained by
27 `len`. It does **not** include the terminating ``\0`` character. A
28 malformed format string can result in a ``-1`` return value.
30 .. c:function:: ssize_t csnprintfrr(char *buf, size_t len, const char *fmt, ...)
31 .. c:function:: ssize_t vcsnprintfrr(char *buf, size_t len, const char *fmt, va_list)
33 Same as above, but the ``c`` stands for "continue" or "concatenate". The
34 output is appended to the string instead of overwriting it.
36 .. c:function:: char *asprintfrr(struct memtype *mt, const char *fmt, ...)
37 .. c:function:: char *vasprintfrr(struct memtype *mt, const char *fmt, va_list)
39 These functions allocate a dynamic buffer (using MTYPE `mt`) and print to
40 that. If the format string is malformed, they return a copy of the format
41 string, so the return value is always non-NULL and always dynamically
44 .. c:function:: char *asnprintfrr(struct memtype *mt, char *buf, size_t len, const char *fmt, ...)
45 .. c:function:: char *vasnprintfrr(struct memtype *mt, char *buf, size_t len, const char *fmt, va_list)
47 This variant tries to use the static buffer provided, but falls back to
48 dynamic allocation if it is insufficient.
50 The return value can be either `buf` or a newly allocated string using
51 `mt`. You MUST free it like this::
53 char *ret = asnprintfrr(MTYPE_FOO, buf, sizeof(buf), ...);
55 XFREE(MTYPE_FOO, ret);
57 .. c:function:: ssize_t bprintfrr(struct fbuf *fb, const char *fmt, ...)
58 .. c:function:: ssize_t vbprintfrr(struct fbuf *fb, const char *fmt, va_list)
60 These are the "lowest level" functions, which the other variants listed
61 above use to implement their functionality on top. Mainly useful for
62 implementing printfrr extensions since those get a ``struct fbuf *`` to
63 write their output to.
65 .. c:macro:: FMT_NSTD(expr)
67 This macro turns off/on format warnings as needed when non-ISO-C
68 compatible printfrr extensions are used (e.g. ``%.*p`` or ``%Ld``.)::
70 vty_out(vty, "standard compatible %pI4\n", &addr);
71 FMT_NSTD(vty_out(vty, "non-standard %-47.*pHX\n", (int)len, buf));
73 When the frr-format plugin is in use, this macro is a no-op since the
74 frr-format plugin supports all printfrr extensions. Since the FRR CI
75 includes a system with the plugin enabled, this means format errors will
76 not slip by undetected even with FMT_NSTD.
80 ``printfrr()`` does not support the ``%n`` format.
85 ``printfrr()`` are AS-Safe under the following conditions:
87 * the ``[v]as[n]printfrr`` variants are not AS-Safe (allocating memory)
88 * floating point specifiers are not AS-Safe (system printf is used for these)
89 * the positional ``%1$d`` syntax should not be used (8 arguments are supported
91 * extensions are only AS-Safe if their printer is AS-Safe
96 ``printfrr()`` format strings can be extended with suffixes after `%p` or `%d`.
97 Printf features like field lengths can be used normally with these extensions,
98 e.g. ``%-15pI4`` works correctly, **except if the extension consumes the
99 width or precision**. Extensions that do so are listed below as ``%*pXX``
100 rather than ``%pXX``.
102 The extension specifier after ``%p`` or ``%d`` is always an uppercase letter;
103 by means of established pattern uppercase letters and numbers form the type
104 identifier which may be followed by lowercase flags.
106 You can grep the FRR source for ``printfrr_ext_autoreg`` to see all extended
107 printers and what exactly they do. More printers are likely to be added as
108 needed/useful, so the list here may be outdated.
112 The ``zlog_*``/``flog_*`` and ``vty_out`` functions all use printfrr
113 internally, so these extensions are available there. However, they are
114 **not** available when calling ``snprintf`` directly. You need to call
115 ``snprintfrr`` instead.
117 Networking data types
118 ^^^^^^^^^^^^^^^^^^^^^
120 .. role:: frrfmtout(code)
122 .. frrfmt:: %pI4 (struct in_addr *, in_addr_t *)
126 ``%pI4s``: :frrfmtout:`*` — print star instead of ``0.0.0.0`` (for multicast)
128 .. frrfmt:: %pI6 (struct in6_addr *)
130 :frrfmtout:`fe80::1234`
132 ``%pI6s``: :frrfmtout:`*` — print star instead of ``::`` (for multicast)
134 .. frrfmt:: %pEA (struct ethaddr *)
136 :frrfmtout:`01:23:45:67:89:ab`
138 .. frrfmt:: %pIA (struct ipaddr *)
140 :frrfmtout:`1.2.3.4` / :frrfmtout:`fe80::1234`
142 ``%pIAs``: — print star instead of zero address (for multicast)
144 .. frrfmt:: %pFX (struct prefix *)
146 :frrfmtout:`1.2.3.0/24` / :frrfmtout:`fe80::1234/64`
148 This accepts the following types:
151 - :c:struct:`prefix_ipv4`
152 - :c:struct:`prefix_ipv6`
153 - :c:struct:`prefix_eth`
154 - :c:struct:`prefix_evpn`
155 - :c:struct:`prefix_fs`
157 It does **not** accept the following types:
159 - :c:struct:`prefix_ls`
160 - :c:struct:`prefix_rd`
161 - :c:struct:`prefix_ptr`
162 - :c:struct:`prefix_sg` (use :frrfmt:`%pPSG4`)
163 - :c:union:`prefixptr` (dereference to get :c:struct:`prefix`)
164 - :c:union:`prefixconstptr` (dereference to get :c:struct:`prefix`)
166 .. frrfmt:: %pPSG4 (struct prefix_sg *)
168 :frrfmtout:`(*,1.2.3.4)`
170 This is *(S,G)* output for use in pimd. (Note prefix_sg is not a prefix
171 "subclass" like the other prefix_* structs.)
173 .. frrfmt:: %pSU (union sockunion *)
175 ``%pSU``: :frrfmtout:`1.2.3.4` / :frrfmtout:`fe80::1234`
177 ``%pSUs``: :frrfmtout:`1.2.3.4` / :frrfmtout:`fe80::1234%89`
178 (adds IPv6 scope ID as integer)
180 ``%pSUp``: :frrfmtout:`1.2.3.4:567` / :frrfmtout:`[fe80::1234]:567`
183 ``%pSUps``: :frrfmtout:`1.2.3.4:567` / :frrfmtout:`[fe80::1234%89]:567`
184 (adds port and scope ID)
186 .. frrfmt:: %pRN (struct route_node *, struct bgp_node *, struct agg_node *)
188 :frrfmtout:`192.168.1.0/24` (dst-only node)
190 :frrfmtout:`2001:db8::/32 from fe80::/64` (SADR node)
192 .. frrfmt:: %pNH (struct nexthop *)
194 ``%pNHvv``: :frrfmtout:`via 1.2.3.4, eth0` — verbose zebra format
196 ``%pNHv``: :frrfmtout:`1.2.3.4, via eth0` — slightly less verbose zebra format
198 ``%pNHs``: :frrfmtout:`1.2.3.4 if 15` — same as :c:func:`nexthop2str()`
200 ``%pNHcg``: :frrfmtout:`1.2.3.4` — compact gateway only
202 ``%pNHci``: :frrfmtout:`eth0` — compact interface only
204 .. frrfmt:: %pBD (struct bgp_dest *)
206 :frrfmtout:`fe80::1234/64`
208 (only available in bgpd.)
210 .. frrfmt:: %dPF (int)
214 Prints an `AF_*` / `PF_*` constant. ``PF`` is used here to avoid confusion
215 with `AFI` constants, even though the FRR codebase prefers `AF_INET` over
218 .. frrfmt:: %dSO (int)
220 :frrfmtout:`SOCK_STREAM`
222 General utility formats
223 ^^^^^^^^^^^^^^^^^^^^^^^
225 .. frrfmt:: %m (no argument)
227 :frrfmtout:`Permission denied`
229 Prints ``strerror(errno)``. Does **not** consume any input argument, don't
232 (This is a GNU extension not specific to FRR. FRR guarantees it is
233 available on all systems in printfrr, though BSDs support it in printf too.)
235 .. frrfmt:: %pSQ (char *)
237 ([S]tring [Q]uote.) Like ``%s``, but produce a quoted string. Options:
239 ``n`` - treat ``NULL`` as empty string instead.
241 ``q`` - include ``""`` quotation marks. Note: ``NULL`` is printed as
242 ``(null)``, not ``"(null)"`` unless ``n`` is used too. This is
245 ``s`` - use escaping suitable for RFC5424 syslog. This means ``]`` is
248 If a length is specified (``%*pSQ`` or ``%.*pSQ``), null bytes in the input
249 string do not end the string and are just printed as ``\x00``.
251 .. frrfmt:: %pSE (char *)
253 ([S]tring [E]scape.) Like ``%s``, but escape special characters.
256 ``n`` - treat ``NULL`` as empty string instead.
258 Unlike :frrfmt:`%pSQ`, this escapes many more characters that are fine for
259 a quoted string but not on their own.
261 If a length is specified (``%*pSE`` or ``%.*pSE``), null bytes in the input
262 string do not end the string and are just printed as ``\x00``.
264 .. frrfmt:: %pVA (struct va_format *)
266 Recursively invoke printfrr, with arguments passed in through:
268 .. c:struct:: va_format
270 .. c:member:: const char *fmt
272 Format string to use for the recursive printfrr call.
274 .. c:member:: va_list *va
276 Formatting arguments. Note this is passed as a pointer, not - as in
277 most other places - a direct struct reference. Internally uses
278 ``va_copy()`` so repeated calls can be made (e.g. for determining
281 .. frrfmt:: %pFB (struct fbuf *)
283 Insert text from a ``struct fbuf *``, i.e. the output of a call to
284 :c:func:`bprintfrr()`.
286 .. frrfmt:: %*pHX (void *, char *, unsigned char *)
288 ``%pHX``: :frrfmtout:`12 34 56 78`
290 ``%pHXc``: :frrfmtout:`12:34:56:78` (separate with [c]olon)
292 ``%pHXn``: :frrfmtout:`12345678` (separate with [n]othing)
294 Insert hexdump. This specifier requires a precision or width to be
295 specified. A precision (``%.*pHX``) takes precedence, but generates a
296 compiler warning since precisions are undefined for ``%p`` in ISO C. If
297 no precision is given, the width is used instead (and normal handling of
298 the width is suppressed).
300 Note that width and precision are ``int`` arguments, not ``size_t``. Use
306 snprintfrr(out, sizeof(out), "... %*pHX ...", (int)len, buf);
308 /* with padding to width - would generate a warning due to %.*p */
309 FMT_NSTD(snprintfrr(out, sizeof(out), "... %-47.*pHX ...", (int)len, buf));
311 .. frrfmt:: %*pHS (void *, char *, unsigned char *)
313 ``%pHS``: :frrfmtout:`hex.dump`
315 This is a complementary format for :frrfmt:`%*pHX` to print the text
316 representation for a hexdump. Non-printable characters are replaced with
324 These formats currently only exist for advanced type checking with the
325 ``frr-format`` GCC plugin. They should not be used directly since they will
326 cause compiler warnings when used without the plugin. Use with
327 :c:macro:`FMT_NSTD` if necessary.
329 It is possible ISO C23 may introduce another format for these, possibly
330 ``%w64d`` discussed in `JTC 1/SC 22/WG 14/N2680 <http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2680.pdf>`_.
332 .. frrfmt:: %Lu (uint64_t)
336 .. frrfmt:: %Ld (int64_t)
346 If it is something that the user will want to look at and maybe do
347 something, it is either an **error** or a **warning**.
349 We're expecting that warnings and errors are in some way visible to the
350 user (in the worst case by looking at the log after the network broke, but
351 maybe by a syslog collector from all routers.) Therefore, anything that
352 needs to get the user in the loop—and only these things—are warnings or
355 Note that this doesn't necessarily mean the user needs to fix something in
356 the FRR instance. It also includes when we detect something else needs
357 fixing, for example another router, the system we're running on, or the
358 configuration. The common point is that the user should probably do
361 Deciding between a warning and an error is slightly less obvious; the rule
362 of thumb here is that an error will cause considerable fallout beyond its
363 direct effect. Closing a BGP session due to a malformed update is an error
364 since all routes from the peer are dropped; discarding one route because
365 its attributes don't make sense is a warning.
367 This also loosely corresponds to the kind of reaction we're expecting from
368 the user. An error is likely to need immediate response while a warning
369 might be snoozed for a bit and addressed as part of general maintenance.
370 If a problem will self-repair (e.g. by retransmits), it should be a
371 warning—unless the impact until that self-repair is very harsh.
373 Examples for warnings:
375 * a BGP update, LSA or LSP could not be processed, but operation is
376 proceeding and the broken pieces are likely to self-fix later
377 * some kind of controller cannot be reached, but we can work without it
378 * another router is using some unknown or unsupported capability
382 * dropping a BGP session due to malformed data
383 * a socket for routing protocol operation cannot be opened
384 * desynchronization from network state because something went wrong
385 * *everything that we as developers would really like to be notified about,
386 i.e. some assumption in the code isn't holding up*
389 Informational messages
390 ^^^^^^^^^^^^^^^^^^^^^^
392 Anything that provides introspection to the user during normal operation
393 is an **info** message.
395 This includes all kinds of operational state transitions and events,
396 especially if they might be interesting to the user during the course of
397 figuring out a warning or an error.
399 By itself, these messages should mostly be statements of fact. They might
400 indicate the order and relationship in which things happened. Also covered
401 are conditions that might be "operational issues" like a link failure due
402 to an unplugged cable. If it's pretty much the point of running a routing
403 daemon for, it's not a warning or an error, just business as usual.
405 The user should be able to see the state of these bits from operational
406 state output, i.e. `show interface` or `show foobar neighbors`. The log
407 message indicating the change may have been printed weeks ago, but the
408 state can always be viewed. (If some state change has an info message but
409 no "show" command, maybe that command needs to be added.)
413 * all kinds of up/down state changes
415 * interface coming up or going down
416 * addresses being added or deleted
417 * peers and neighbors coming up or going down
419 * rejection of some routes due to user-configured route maps
420 * backwards compatibility handling because another system on the network
421 has a different or smaller feature set
424 The previously used **notify** priority is replaced with *info* in all
425 cases. We don't currently have a well-defined use case for it.
428 Debug messages and asserts
429 ^^^^^^^^^^^^^^^^^^^^^^^^^^
431 Everything that is only interesting on-demand, or only while developing,
432 is a **debug** message. It might be interesting to the user for a
433 particularly evasive issue, but in general these are details that an
434 average user might not even be able to make sense of.
436 Most (or all?) debug messages should be behind a `debug foobar` category
437 switch that controls which subset of these messages is currently
438 interesting and thus printed. If a debug message doesn't have such a
439 guard, there should be a good explanation as to why.
441 Conversely, debug messages are the only thing that should be guarded by
442 these switches. Neither info nor warning or error messages should be
445 **Asserts** should only be used as pretty crashes. We are expecting that
446 asserts remain enabled in production builds, but please try to not use
447 asserts in a way that would cause a security problem if the assert wasn't
448 there (i.e. don't use them for length checks.)
450 The purpose of asserts is mainly to help development and bug hunting. If
451 the daemon crashes, then having some more information is nice, and the
452 assert can provide crucial hints that cut down on the time needed to track
453 an issue. That said, if the issue can be reasonably handled and/or isn't
454 going to crash the daemon, it shouldn't be an assert.
456 For anything else where internal constraints are violated but we're not
457 breaking due to it, it's an error instead (not a debug.) These require
458 "user action" of notifying the developers.
462 * mismatched :code:`prev`/:code:`next` pointers in lists
463 * some field that is absolutely needed is :code:`NULL`
464 * any other kind of data structure corruption that will cause the daemon
465 to crash sooner or later, one way or another
467 Thread-local buffering
468 ----------------------
470 The core logging code in :file:`lib/zlog.c` allows setting up per-thread log
471 message buffers in order to improve logging performance. The following rules
472 apply for this buffering:
474 * Only messages of priority *DEBUG* or *INFO* are buffered.
475 * Any higher-priority message causes the thread's entire buffer to be flushed,
476 thus message ordering is preserved on a per-thread level.
477 * There is no guarantee on ordering between different threads; in most cases
478 this is arbitrary to begin with since the threads essentially race each
479 other in printing log messages. If an order is established with some
480 synchronization primitive, add calls to :c:func:`zlog_tls_buffer_flush()`.
481 * The buffers are only ever accessed by the thread they are created by. This
482 means no locking is necessary.
484 Both the main/default thread and additional threads created by
485 :c:func:`frr_pthread_new()` with the default :c:func:`frr_run()` handler will
486 initialize thread-local buffering and call :c:func:`zlog_tls_buffer_flush()`
489 If some piece of code runs for an extended period, it may be useful to insert
490 calls to :c:func:`zlog_tls_buffer_flush()` in appropriate places:
492 .. c:function:: void zlog_tls_buffer_flush(void)
494 Write out any pending log messages that the calling thread may have in its
495 buffer. This function is safe to call regardless of the per-thread log
496 buffer being set up / in use or not.
498 When working with threads that do not use the :c:struct:`thread_master`
499 event loop, per-thread buffers can be managed with:
501 .. c:function:: void zlog_tls_buffer_init(void)
503 Set up thread-local buffering for log messages. This function may be
504 called repeatedly without adverse effects, but remember to call
505 :c:func:`zlog_tls_buffer_fini()` at thread exit.
509 If this function is called, but :c:func:`zlog_tls_buffer_flush()` is
510 not used, log message output will lag behind since messages will only be
511 written out when the buffer is full.
513 Exiting the thread without calling :c:func:`zlog_tls_buffer_fini()`
514 will cause buffered log messages to be lost.
516 .. c:function:: void zlog_tls_buffer_fini(void)
518 Flush pending messages and tear down thread-local log message buffering.
519 This function may be called repeatedly regardless of whether
520 :c:func:`zlog_tls_buffer_init()` was ever called.
525 The actual logging subsystem (in :file:`lib/zlog.c`) is heavily separated
526 from the actual log writers. It uses an atomic linked-list (`zlog_targets`)
527 with RCU to maintain the log targets to be called. This list is intended to
528 function as "backend" only, it **is not used for configuration**.
530 Logging targets provide their configuration layer on top of this and maintain
531 their own capability to enumerate and store their configuration. Some targets
532 (e.g. syslog) are inherently single instance and just stuff their config in
533 global variables. Others (e.g. file/fd output) are multi-instance capable.
534 There is another layer boundary here between these and the VTY configuration
540 .. c:struct:: zlog_target
542 This struct needs to be filled in by any log target and then passed to
543 :c:func:`zlog_target_replace()`. After it has been registered,
544 **RCU semantics apply**. Most changes to associated data should make a
545 copy, change that, and then replace the entire struct.
547 Additional per-target data should be "appended" by embedding this struct
548 into a larger one, for use with `containerof()`, and
549 :c:func:`zlog_target_clone()` and :c:func:`zlog_target_free()` should be
550 used to allocate/free the entire container struct.
552 Do not use this structure to maintain configuration. It should only
553 contain (a copy of) the data needed to perform the actual logging. For
554 example, the syslog target uses this:
559 struct zlog_target zt;
563 static void zlog_syslog(struct zlog_target *zt, struct zlog_msg *msgs[], size_t nmsgs)
565 struct zlt_syslog *zte = container_of(zt, struct zlt_syslog, zt);
568 for (i = 0; i < nmsgs; i++)
569 if (zlog_msg_prio(msgs[i]) <= zt->prio_min)
570 syslog(zlog_msg_prio(msgs[i]) | zte->syslog_facility, "%s",
571 zlog_msg_text(msgs[i], NULL));
575 .. c:function:: struct zlog_target *zlog_target_clone(struct memtype *mt, struct zlog_target *oldzt, size_t size)
577 Allocates a logging target struct. Note that the ``oldzt`` argument may be
578 ``NULL`` to allocate a "from scratch". If ``oldzt`` is not ``NULL``, the
579 generic bits in :c:struct:`zlog_target` are copied. **Target specific
580 bits are not copied.**
582 .. c:function:: struct zlog_target *zlog_target_replace(struct zlog_target *oldzt, struct zlog_target *newzt)
584 Adds, replaces or deletes a logging target (either ``oldzt`` or ``newzt`` may be ``NULL``.)
586 Returns ``oldzt`` for freeing. The target remains possibly in use by
587 other threads until the RCU cycle ends. This implies you cannot release
588 resources (e.g. memory, file descriptors) immediately.
590 The replace operation is not atomic; for a brief period it is possible that
591 messages are delivered on both ``oldzt`` and ``newzt``.
595 ``oldzt`` must remain **functional** until the RCU cycle ends.
597 .. c:function:: void zlog_target_free(struct memtype *mt, struct zlog_target *zt)
599 Counterpart to :c:func:`zlog_target_clone()`, frees a target (using RCU.)
601 .. c:member:: void (*zlog_target.logfn)(struct zlog_target *zt, struct zlog_msg *msgs[], size_t nmsg)
603 Called on a target to deliver "normal" logging messages. ``msgs`` is an
604 array of opaque structs containing the actual message. Use ``zlog_msg_*``
605 functions to access message data (this is done to allow some optimizations,
606 e.g. lazy formatting the message text and timestamp as needed.)
610 ``logfn()`` must check each individual message's priority value against
611 the configured ``prio_min``. While the ``prio_min`` field is common to
612 all targets and used by the core logging code to early-drop unneeded log
613 messages, the array is **not** filtered for each ``logfn()`` call.
615 .. c:member:: void (*zlog_target.logfn_sigsafe)(struct zlog_target *zt, const char *text, size_t len)
617 Called to deliver "exception" logging messages (i.e. SEGV messages.)
618 Must be Async-Signal-Safe (may not allocate memory or call "complicated"
619 libc functions.) May be ``NULL`` if the log target cannot handle this.
624 :file:`lib/zlog_targets.c` provides the standard file / fd / syslog targets.
625 The syslog target is single-instance while file / fd targets can be
626 instantiated as needed. There are 3 built-in targets that are fully
627 autonomous without any config:
629 - startup logging to `stderr`, until either :c:func:`zlog_startup_end()` or
630 :c:func:`zlog_aux_init()` is called.
631 - stdout logging for non-daemon programs using :c:func:`zlog_aux_init()`
632 - crashlogs written to :file:`/var/tmp/frr.daemon.crashlog`
634 The regular CLI/command-line logging setup is handled by :file:`lib/log_vty.c`
635 which makes the appropriate instantiations of syslog / file / fd targets.
639 :c:func:`zlog_startup_end()` should do an explicit switchover from
640 startup stderr logging to configured logging. Currently, configured logging
641 starts in parallel as soon as the respective setup is executed. This results
642 in some duplicate logging.