8 One of the most frequent decisions to make while writing code for FRR is what
9 to log, what level to log it at, and when to log it. Here is a list of
10 recommendations for these decisions.
16 ``printfrr()`` is FRR's modified version of ``printf()``, designed to make
17 life easier when printing nontrivial datastructures. The following variants
20 .. c:function:: ssize_t snprintfrr(char *buf, size_t len, const char *fmt, ...)
21 .. c:function:: ssize_t vsnprintfrr(char *buf, size_t len, const char *fmt, va_list)
23 These correspond to ``snprintf``/``vsnprintf``. If you pass NULL for buf
24 or 0 for len, no output is written but the return value is still calculated.
26 The return value is always the full length of the output, unconstrained by
27 `len`. It does **not** include the terminating ``\0`` character. A
28 malformed format string can result in a ``-1`` return value.
30 .. c:function:: ssize_t csnprintfrr(char *buf, size_t len, const char *fmt, ...)
31 .. c:function:: ssize_t vcsnprintfrr(char *buf, size_t len, const char *fmt, va_list)
33 Same as above, but the ``c`` stands for "continue" or "concatenate". The
34 output is appended to the string instead of overwriting it.
36 .. c:function:: char *asprintfrr(struct memtype *mt, const char *fmt, ...)
37 .. c:function:: char *vasprintfrr(struct memtype *mt, const char *fmt, va_list)
39 These functions allocate a dynamic buffer (using MTYPE `mt`) and print to
40 that. If the format string is malformed, they return a copy of the format
41 string, so the return value is always non-NULL and always dynamically
44 .. c:function:: char *asnprintfrr(struct memtype *mt, char *buf, size_t len, const char *fmt, ...)
45 .. c:function:: char *vasnprintfrr(struct memtype *mt, char *buf, size_t len, const char *fmt, va_list)
47 This variant tries to use the static buffer provided, but falls back to
48 dynamic allocation if it is insufficient.
50 The return value can be either `buf` or a newly allocated string using
51 `mt`. You MUST free it like this::
53 char *ret = asnprintfrr(MTYPE_FOO, buf, sizeof(buf), ...);
55 XFREE(MTYPE_FOO, ret);
57 .. c:function:: ssize_t bprintfrr(struct fbuf *fb, const char *fmt, ...)
58 .. c:function:: ssize_t vbprintfrr(struct fbuf *fb, const char *fmt, va_list)
60 These are the "lowest level" functions, which the other variants listed
61 above use to implement their functionality on top. Mainly useful for
62 implementing printfrr extensions since those get a ``struct fbuf *`` to
63 write their output to.
65 .. c:macro:: FMT_NSTD(expr)
67 This macro turns off/on format warnings as needed when non-ISO-C
68 compatible printfrr extensions are used (e.g. ``%.*p`` or ``%Ld``.)::
70 vty_out(vty, "standard compatible %pI4\n", &addr);
71 FMT_NSTD(vty_out(vty, "non-standard %-47.*pHX\n", (int)len, buf));
73 When the frr-format plugin is in use, this macro is a no-op since the
74 frr-format plugin supports all printfrr extensions. Since the FRR CI
75 includes a system with the plugin enabled, this means format errors will
76 not slip by undetected even with FMT_NSTD.
80 ``printfrr()`` does not support the ``%n`` format.
85 ``printfrr()`` are AS-Safe under the following conditions:
87 * the ``[v]as[n]printfrr`` variants are not AS-Safe (allocating memory)
88 * floating point specifiers are not AS-Safe (system printf is used for these)
89 * the positional ``%1$d`` syntax should not be used (8 arguments are supported
91 * extensions are only AS-Safe if their printer is AS-Safe
96 ``printfrr()`` format strings can be extended with suffixes after `%p` or `%d`.
97 Printf features like field lengths can be used normally with these extensions,
98 e.g. ``%-15pI4`` works correctly, **except if the extension consumes the
99 width or precision**. Extensions that do so are listed below as ``%*pXX``
100 rather than ``%pXX``.
102 The extension specifier after ``%p`` or ``%d`` is always an uppercase letter;
103 by means of established pattern uppercase letters and numbers form the type
104 identifier which may be followed by lowercase flags.
106 You can grep the FRR source for ``printfrr_ext_autoreg`` to see all extended
107 printers and what exactly they do. More printers are likely to be added as
108 needed/useful, so the list here may be outdated.
112 The ``zlog_*``/``flog_*`` and ``vty_out`` functions all use printfrr
113 internally, so these extensions are available there. However, they are
114 **not** available when calling ``snprintf`` directly. You need to call
115 ``snprintfrr`` instead.
117 Networking data types
118 ^^^^^^^^^^^^^^^^^^^^^
120 .. role:: frrfmtout(code)
122 .. frrfmt:: %pI4 (struct in_addr *, in_addr_t *)
126 .. frrfmt:: %pI6 (struct in6_addr *)
128 :frrfmtout:`fe80::1234`
130 .. frrfmt:: %pEA (struct ethaddr *)
132 :frrfmtout:`01:23:45:67:89:ab`
134 .. frrfmt:: %pIA (struct ipaddr *)
136 :frrfmtout:`1.2.3.4` / :frrfmtout:`fe80::1234`
138 .. frrfmt:: %pFX (struct prefix *)
140 :frrfmtout:`1.2.3.0/24` / :frrfmtout:`fe80::1234/64`
142 This accepts the following types:
145 - :c:struct:`prefix_ipv4`
146 - :c:struct:`prefix_ipv6`
147 - :c:struct:`prefix_eth`
148 - :c:struct:`prefix_evpn`
149 - :c:struct:`prefix_fs`
151 It does **not** accept the following types:
153 - :c:struct:`prefix_ls`
154 - :c:struct:`prefix_rd`
155 - :c:struct:`prefix_ptr`
156 - :c:struct:`prefix_sg` (use :frrfmt:`%pPSG4`)
157 - :c:union:`prefixptr` (dereference to get :c:struct:`prefix`)
158 - :c:union:`prefixconstptr` (dereference to get :c:struct:`prefix`)
160 .. frrfmt:: %pPSG4 (struct prefix_sg *)
162 :frrfmtout:`(*,1.2.3.4)`
164 This is *(S,G)* output for use in pimd. (Note prefix_sg is not a prefix
165 "subclass" like the other prefix_* structs.)
167 .. frrfmt:: %pSU (union sockunion *)
169 ``%pSU``: :frrfmtout:`1.2.3.4` / :frrfmtout:`fe80::1234`
171 ``%pSUs``: :frrfmtout:`1.2.3.4` / :frrfmtout:`fe80::1234%89`
172 (adds IPv6 scope ID as integer)
174 ``%pSUp``: :frrfmtout:`1.2.3.4:567` / :frrfmtout:`[fe80::1234]:567`
177 ``%pSUps``: :frrfmtout:`1.2.3.4:567` / :frrfmtout:`[fe80::1234%89]:567`
178 (adds port and scope ID)
180 .. frrfmt:: %pRN (struct route_node *, struct bgp_node *, struct agg_node *)
182 :frrfmtout:`192.168.1.0/24` (dst-only node)
184 :frrfmtout:`2001:db8::/32 from fe80::/64` (SADR node)
186 .. frrfmt:: %pNH (struct nexthop *)
188 ``%pNHvv``: :frrfmtout:`via 1.2.3.4, eth0` — verbose zebra format
190 ``%pNHv``: :frrfmtout:`1.2.3.4, via eth0` — slightly less verbose zebra format
192 ``%pNHs``: :frrfmtout:`1.2.3.4 if 15` — same as :c:func:`nexthop2str()`
194 ``%pNHcg``: :frrfmtout:`1.2.3.4` — compact gateway only
196 ``%pNHci``: :frrfmtout:`eth0` — compact interface only
198 .. frrfmt:: %pBD (struct bgp_dest *)
200 :frrfmtout:`fe80::1234/64`
202 (only available in bgpd.)
204 .. frrfmt:: %dPF (int)
208 Prints an `AF_*` / `PF_*` constant. ``PF`` is used here to avoid confusion
209 with `AFI` constants, even though the FRR codebase prefers `AF_INET` over
212 .. frrfmt:: %dSO (int)
214 :frrfmtout:`SOCK_STREAM`
216 General utility formats
217 ^^^^^^^^^^^^^^^^^^^^^^^
219 .. frrfmt:: %m (no argument)
221 :frrfmtout:`Permission denied`
223 Prints ``strerror(errno)``. Does **not** consume any input argument, don't
226 (This is a GNU extension not specific to FRR. FRR guarantees it is
227 available on all systems in printfrr, though BSDs support it in printf too.)
229 .. frrfmt:: %pSQ (char *)
231 ([S]tring [Q]uote.) Like ``%s``, but produce a quoted string. Options:
233 ``n`` - treat ``NULL`` as empty string instead.
235 ``q`` - include ``""`` quotation marks. Note: ``NULL`` is printed as
236 ``(null)``, not ``"(null)"`` unless ``n`` is used too. This is
239 ``s`` - use escaping suitable for RFC5424 syslog. This means ``]`` is
242 If a length is specified (``%*pSQ`` or ``%.*pSQ``), null bytes in the input
243 string do not end the string and are just printed as ``\x00``.
245 .. frrfmt:: %pSE (char *)
247 ([S]tring [E]scape.) Like ``%s``, but escape special characters.
250 ``n`` - treat ``NULL`` as empty string instead.
252 Unlike :frrfmt:`%pSQ`, this escapes many more characters that are fine for
253 a quoted string but not on their own.
255 If a length is specified (``%*pSE`` or ``%.*pSE``), null bytes in the input
256 string do not end the string and are just printed as ``\x00``.
258 .. frrfmt:: %pVA (struct va_format *)
260 Recursively invoke printfrr, with arguments passed in through:
262 .. c:struct:: va_format
264 .. c:member:: const char *fmt
266 Format string to use for the recursive printfrr call.
268 .. c:member:: va_list *va
270 Formatting arguments. Note this is passed as a pointer, not - as in
271 most other places - a direct struct reference. Internally uses
272 ``va_copy()`` so repeated calls can be made (e.g. for determining
275 .. frrfmt:: %pFB (struct fbuf *)
277 Insert text from a ``struct fbuf *``, i.e. the output of a call to
278 :c:func:`bprintfrr()`.
280 .. frrfmt:: %*pHX (void *, char *, unsigned char *)
282 ``%pHX``: :frrfmtout:`12 34 56 78`
284 ``%pHXc``: :frrfmtout:`12:34:56:78` (separate with [c]olon)
286 ``%pHXn``: :frrfmtout:`12345678` (separate with [n]othing)
288 Insert hexdump. This specifier requires a precision or width to be
289 specified. A precision (``%.*pHX``) takes precedence, but generates a
290 compiler warning since precisions are undefined for ``%p`` in ISO C. If
291 no precision is given, the width is used instead (and normal handling of
292 the width is suppressed).
294 Note that width and precision are ``int`` arguments, not ``size_t``. Use
300 snprintfrr(out, sizeof(out), "... %*pHX ...", (int)len, buf);
302 /* with padding to width - would generate a warning due to %.*p */
303 FMT_NSTD(snprintfrr(out, sizeof(out), "... %-47.*pHX ...", (int)len, buf));
305 .. frrfmt:: %*pHS (void *, char *, unsigned char *)
307 ``%pHS``: :frrfmtout:`hex.dump`
309 This is a complementary format for :frrfmt:`%*pHX` to print the text
310 representation for a hexdump. Non-printable characters are replaced with
318 These formats currently only exist for advanced type checking with the
319 ``frr-format`` GCC plugin. They should not be used directly since they will
320 cause compiler warnings when used without the plugin. Use with
321 :c:macro:`FMT_NSTD` if necessary.
323 It is possible ISO C23 may introduce another format for these, possibly
324 ``%w64d`` discussed in `JTC 1/SC 22/WG 14/N2680 <http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2680.pdf>`_.
326 .. frrfmt:: %Lu (uint64_t)
330 .. frrfmt:: %Ld (int64_t)
340 If it is something that the user will want to look at and maybe do
341 something, it is either an **error** or a **warning**.
343 We're expecting that warnings and errors are in some way visible to the
344 user (in the worst case by looking at the log after the network broke, but
345 maybe by a syslog collector from all routers.) Therefore, anything that
346 needs to get the user in the loop—and only these things—are warnings or
349 Note that this doesn't necessarily mean the user needs to fix something in
350 the FRR instance. It also includes when we detect something else needs
351 fixing, for example another router, the system we're running on, or the
352 configuration. The common point is that the user should probably do
355 Deciding between a warning and an error is slightly less obvious; the rule
356 of thumb here is that an error will cause considerable fallout beyond its
357 direct effect. Closing a BGP session due to a malformed update is an error
358 since all routes from the peer are dropped; discarding one route because
359 its attributes don't make sense is a warning.
361 This also loosely corresponds to the kind of reaction we're expecting from
362 the user. An error is likely to need immediate response while a warning
363 might be snoozed for a bit and addressed as part of general maintenance.
364 If a problem will self-repair (e.g. by retransmits), it should be a
365 warning—unless the impact until that self-repair is very harsh.
367 Examples for warnings:
369 * a BGP update, LSA or LSP could not be processed, but operation is
370 proceeding and the broken pieces are likely to self-fix later
371 * some kind of controller cannot be reached, but we can work without it
372 * another router is using some unknown or unsupported capability
376 * dropping a BGP session due to malformed data
377 * a socket for routing protocol operation cannot be opened
378 * desynchronization from network state because something went wrong
379 * *everything that we as developers would really like to be notified about,
380 i.e. some assumption in the code isn't holding up*
383 Informational messages
384 ^^^^^^^^^^^^^^^^^^^^^^
386 Anything that provides introspection to the user during normal operation
387 is an **info** message.
389 This includes all kinds of operational state transitions and events,
390 especially if they might be interesting to the user during the course of
391 figuring out a warning or an error.
393 By itself, these messages should mostly be statements of fact. They might
394 indicate the order and relationship in which things happened. Also covered
395 are conditions that might be "operational issues" like a link failure due
396 to an unplugged cable. If it's pretty much the point of running a routing
397 daemon for, it's not a warning or an error, just business as usual.
399 The user should be able to see the state of these bits from operational
400 state output, i.e. `show interface` or `show foobar neighbors`. The log
401 message indicating the change may have been printed weeks ago, but the
402 state can always be viewed. (If some state change has an info message but
403 no "show" command, maybe that command needs to be added.)
407 * all kinds of up/down state changes
409 * interface coming up or going down
410 * addresses being added or deleted
411 * peers and neighbors coming up or going down
413 * rejection of some routes due to user-configured route maps
414 * backwards compatibility handling because another system on the network
415 has a different or smaller feature set
418 The previously used **notify** priority is replaced with *info* in all
419 cases. We don't currently have a well-defined use case for it.
422 Debug messages and asserts
423 ^^^^^^^^^^^^^^^^^^^^^^^^^^
425 Everything that is only interesting on-demand, or only while developing,
426 is a **debug** message. It might be interesting to the user for a
427 particularly evasive issue, but in general these are details that an
428 average user might not even be able to make sense of.
430 Most (or all?) debug messages should be behind a `debug foobar` category
431 switch that controls which subset of these messages is currently
432 interesting and thus printed. If a debug message doesn't have such a
433 guard, there should be a good explanation as to why.
435 Conversely, debug messages are the only thing that should be guarded by
436 these switches. Neither info nor warning or error messages should be
439 **Asserts** should only be used as pretty crashes. We are expecting that
440 asserts remain enabled in production builds, but please try to not use
441 asserts in a way that would cause a security problem if the assert wasn't
442 there (i.e. don't use them for length checks.)
444 The purpose of asserts is mainly to help development and bug hunting. If
445 the daemon crashes, then having some more information is nice, and the
446 assert can provide crucial hints that cut down on the time needed to track
447 an issue. That said, if the issue can be reasonably handled and/or isn't
448 going to crash the daemon, it shouldn't be an assert.
450 For anything else where internal constraints are violated but we're not
451 breaking due to it, it's an error instead (not a debug.) These require
452 "user action" of notifying the developers.
456 * mismatched :code:`prev`/:code:`next` pointers in lists
457 * some field that is absolutely needed is :code:`NULL`
458 * any other kind of data structure corruption that will cause the daemon
459 to crash sooner or later, one way or another
461 Thread-local buffering
462 ----------------------
464 The core logging code in :file:`lib/zlog.c` allows setting up per-thread log
465 message buffers in order to improve logging performance. The following rules
466 apply for this buffering:
468 * Only messages of priority *DEBUG* or *INFO* are buffered.
469 * Any higher-priority message causes the thread's entire buffer to be flushed,
470 thus message ordering is preserved on a per-thread level.
471 * There is no guarantee on ordering between different threads; in most cases
472 this is arbitrary to begin with since the threads essentially race each
473 other in printing log messages. If an order is established with some
474 synchronization primitive, add calls to :c:func:`zlog_tls_buffer_flush()`.
475 * The buffers are only ever accessed by the thread they are created by. This
476 means no locking is necessary.
478 Both the main/default thread and additional threads created by
479 :c:func:`frr_pthread_new()` with the default :c:func:`frr_run()` handler will
480 initialize thread-local buffering and call :c:func:`zlog_tls_buffer_flush()`
483 If some piece of code runs for an extended period, it may be useful to insert
484 calls to :c:func:`zlog_tls_buffer_flush()` in appropriate places:
486 .. c:function:: void zlog_tls_buffer_flush(void)
488 Write out any pending log messages that the calling thread may have in its
489 buffer. This function is safe to call regardless of the per-thread log
490 buffer being set up / in use or not.
492 When working with threads that do not use the :c:struct:`thread_master`
493 event loop, per-thread buffers can be managed with:
495 .. c:function:: void zlog_tls_buffer_init(void)
497 Set up thread-local buffering for log messages. This function may be
498 called repeatedly without adverse effects, but remember to call
499 :c:func:`zlog_tls_buffer_fini()` at thread exit.
503 If this function is called, but :c:func:`zlog_tls_buffer_flush()` is
504 not used, log message output will lag behind since messages will only be
505 written out when the buffer is full.
507 Exiting the thread without calling :c:func:`zlog_tls_buffer_fini()`
508 will cause buffered log messages to be lost.
510 .. c:function:: void zlog_tls_buffer_fini(void)
512 Flush pending messages and tear down thread-local log message buffering.
513 This function may be called repeatedly regardless of whether
514 :c:func:`zlog_tls_buffer_init()` was ever called.
519 The actual logging subsystem (in :file:`lib/zlog.c`) is heavily separated
520 from the actual log writers. It uses an atomic linked-list (`zlog_targets`)
521 with RCU to maintain the log targets to be called. This list is intended to
522 function as "backend" only, it **is not used for configuration**.
524 Logging targets provide their configuration layer on top of this and maintain
525 their own capability to enumerate and store their configuration. Some targets
526 (e.g. syslog) are inherently single instance and just stuff their config in
527 global variables. Others (e.g. file/fd output) are multi-instance capable.
528 There is another layer boundary here between these and the VTY configuration
534 .. c:struct:: zlog_target
536 This struct needs to be filled in by any log target and then passed to
537 :c:func:`zlog_target_replace()`. After it has been registered,
538 **RCU semantics apply**. Most changes to associated data should make a
539 copy, change that, and then replace the entire struct.
541 Additional per-target data should be "appended" by embedding this struct
542 into a larger one, for use with `containerof()`, and
543 :c:func:`zlog_target_clone()` and :c:func:`zlog_target_free()` should be
544 used to allocate/free the entire container struct.
546 Do not use this structure to maintain configuration. It should only
547 contain (a copy of) the data needed to perform the actual logging. For
548 example, the syslog target uses this:
553 struct zlog_target zt;
557 static void zlog_syslog(struct zlog_target *zt, struct zlog_msg *msgs[], size_t nmsgs)
559 struct zlt_syslog *zte = container_of(zt, struct zlt_syslog, zt);
562 for (i = 0; i < nmsgs; i++)
563 if (zlog_msg_prio(msgs[i]) <= zt->prio_min)
564 syslog(zlog_msg_prio(msgs[i]) | zte->syslog_facility, "%s",
565 zlog_msg_text(msgs[i], NULL));
569 .. c:function:: struct zlog_target *zlog_target_clone(struct memtype *mt, struct zlog_target *oldzt, size_t size)
571 Allocates a logging target struct. Note that the ``oldzt`` argument may be
572 ``NULL`` to allocate a "from scratch". If ``oldzt`` is not ``NULL``, the
573 generic bits in :c:struct:`zlog_target` are copied. **Target specific
574 bits are not copied.**
576 .. c:function:: struct zlog_target *zlog_target_replace(struct zlog_target *oldzt, struct zlog_target *newzt)
578 Adds, replaces or deletes a logging target (either ``oldzt`` or ``newzt`` may be ``NULL``.)
580 Returns ``oldzt`` for freeing. The target remains possibly in use by
581 other threads until the RCU cycle ends. This implies you cannot release
582 resources (e.g. memory, file descriptors) immediately.
584 The replace operation is not atomic; for a brief period it is possible that
585 messages are delivered on both ``oldzt`` and ``newzt``.
589 ``oldzt`` must remain **functional** until the RCU cycle ends.
591 .. c:function:: void zlog_target_free(struct memtype *mt, struct zlog_target *zt)
593 Counterpart to :c:func:`zlog_target_clone()`, frees a target (using RCU.)
595 .. c:member:: void (*zlog_target.logfn)(struct zlog_target *zt, struct zlog_msg *msgs[], size_t nmsg)
597 Called on a target to deliver "normal" logging messages. ``msgs`` is an
598 array of opaque structs containing the actual message. Use ``zlog_msg_*``
599 functions to access message data (this is done to allow some optimizations,
600 e.g. lazy formatting the message text and timestamp as needed.)
604 ``logfn()`` must check each individual message's priority value against
605 the configured ``prio_min``. While the ``prio_min`` field is common to
606 all targets and used by the core logging code to early-drop unneeded log
607 messages, the array is **not** filtered for each ``logfn()`` call.
609 .. c:member:: void (*zlog_target.logfn_sigsafe)(struct zlog_target *zt, const char *text, size_t len)
611 Called to deliver "exception" logging messages (i.e. SEGV messages.)
612 Must be Async-Signal-Safe (may not allocate memory or call "complicated"
613 libc functions.) May be ``NULL`` if the log target cannot handle this.
618 :file:`lib/zlog_targets.c` provides the standard file / fd / syslog targets.
619 The syslog target is single-instance while file / fd targets can be
620 instantiated as needed. There are 3 built-in targets that are fully
621 autonomous without any config:
623 - startup logging to `stderr`, until either :c:func:`zlog_startup_end()` or
624 :c:func:`zlog_aux_init()` is called.
625 - stdout logging for non-daemon programs using :c:func:`zlog_aux_init()`
626 - crashlogs written to :file:`/var/tmp/frr.daemon.crashlog`
628 The regular CLI/command-line logging setup is handled by :file:`lib/log_vty.c`
629 which makes the appropriate instantiations of syslog / file / fd targets.
633 :c:func:`zlog_startup_end()` should do an explicit switchover from
634 startup stderr logging to configured logging. Currently, configured logging
635 starts in parallel as soon as the respective setup is executed. This results
636 in some duplicate logging.