5 Sometimes there is a need to investigate why a cephadm command failed or why
6 a specific service no longer runs properly.
8 As cephadm deploys daemons as containers, troubleshooting daemons is slightly
9 different. Here are a few tools and commands to help investigating issues.
11 Pausing or disabling cephadm
12 ----------------------------
14 If something goes wrong and cephadm is doing behaving in a way you do
15 not like, you can pause most background activity with::
19 This will stop any changes, but cephadm will still periodically check hosts to
20 refresh its inventory of daemons and devices. You can disable cephadm
23 ceph orch set backend ''
24 ceph mgr module disable cephadm
26 This will disable all of the ``ceph orch ...`` CLI commands but the previously
27 deployed daemon containers will still continue to exist and start as they
33 You can monitor the cephadm log in real time with::
37 You can see the last few messages with::
41 If you have enabled logging to files, you can see a cephadm log file called
42 ``ceph.cephadm.log`` on monitor hosts (see :ref:`cephadm-logs`).
47 Use journalctl to gather the log files of all daemons:
49 .. note:: By default cephadm now stores logs in journald. This means
50 that you will no longer find daemon logs in ``/var/log/ceph/``.
52 To read the log file of one specific daemon, run::
54 cephadm logs --name <name-of-daemon>
56 Note: this only works when run on the same host where the daemon is running. To
57 get logs of a daemon running on a different host, give the ``--fsid`` option::
59 cephadm logs --fsid <fsid> --name <name-of-daemon>
61 where the ``<fsid>`` corresponds to the cluster ID printed by ``ceph status``.
63 To fetch all log files of all daemons on a given host, run::
65 for name in $(cephadm ls | jq -r '.[].name') ; do
66 cephadm logs --fsid <fsid> --name "$name" > $name;
69 Collecting systemd status
70 -------------------------
72 To print the state of a systemd unit, run::
74 systemctl status "ceph-$(cephadm shell ceph fsid)@<service name>.service";
77 To fetch all state of all daemons of a given host, run::
79 fsid="$(cephadm shell ceph fsid)"
80 for name in $(cephadm ls | jq -r '.[].name') ; do
81 systemctl status "ceph-$fsid@$name.service" > $name;
85 List all downloaded container images
86 ------------------------------------
88 To list all container images that are downloaded on a host:
90 .. note:: ``Image`` might also be called `ImageID`
94 podman ps -a --format json | jq '.[].Image'
95 "docker.io/library/centos:8"
96 "registry.opensuse.org/opensuse/leap:15.2"
99 Manually running containers
100 ---------------------------
102 Cephadm writes small wrappers that run a containers. Refer to
103 ``/var/lib/ceph/<cluster-fsid>/<service-name>/unit.run`` for the
104 container execution command.