]>
Commit | Line | Data |
---|---|---|
9f95a23c TL |
1 | ================== |
2 | Cephadm Operations | |
3 | ================== | |
4 | ||
5 | Watching cephadm log messages | |
6 | ============================= | |
7 | ||
8 | Cephadm logs to the ``cephadm`` cluster log channel, meaning you can | |
9 | monitor progress in realtime with:: | |
10 | ||
11 | # ceph -W cephadm | |
12 | ||
13 | By default it will show info-level events and above. To see | |
14 | debug-level messages too:: | |
15 | ||
16 | # ceph config set mgr mgr/cephadm/log_to_cluster_level debug | |
17 | # ceph -W cephadm --watch-debug | |
18 | ||
19 | Be careful: the debug messages are very verbose! | |
20 | ||
21 | You can see recent events with:: | |
22 | ||
23 | # ceph log last cephadm | |
24 | ||
25 | These events are also logged to the ``ceph.cephadm.log`` file on | |
26 | monitor hosts and to the monitor daemons' stderr. | |
27 | ||
28 | ||
801d1391 TL |
29 | .. _cephadm-logs: |
30 | ||
9f95a23c TL |
31 | Ceph daemon logs |
32 | ================ | |
33 | ||
34 | Logging to stdout | |
35 | ----------------- | |
36 | ||
37 | Traditionally, Ceph daemons have logged to ``/var/log/ceph``. By | |
38 | default, cephadm daemons log to stderr and the logs are | |
39 | captured by the container runtime environment. For most systems, by | |
40 | default, these logs are sent to journald and accessible via | |
41 | ``journalctl``. | |
42 | ||
43 | For example, to view the logs for the daemon ``mon.foo`` for a cluster | |
44 | with ID ``5c5a50ae-272a-455d-99e9-32c6a013e694``, the command would be | |
45 | something like:: | |
46 | ||
47 | journalctl -u ceph-5c5a50ae-272a-455d-99e9-32c6a013e694@mon.foo | |
48 | ||
49 | This works well for normal operations when logging levels are low. | |
50 | ||
51 | To disable logging to stderr:: | |
52 | ||
53 | ceph config set global log_to_stderr false | |
54 | ceph config set global mon_cluster_log_to_stderr false | |
55 | ||
56 | Logging to files | |
57 | ---------------- | |
58 | ||
59 | You can also configure Ceph daemons to log to files instead of stderr, | |
60 | just like they have in the past. When logging to files, Ceph logs appear | |
61 | in ``/var/log/ceph/<cluster-fsid>``. | |
62 | ||
63 | To enable logging to files:: | |
64 | ||
65 | ceph config set global log_to_file true | |
66 | ceph config set global mon_cluster_log_to_file true | |
67 | ||
68 | We recommend disabling logging to stderr (see above) or else everything | |
69 | will be logged twice:: | |
70 | ||
71 | ceph config set global log_to_stderr false | |
72 | ceph config set global mon_cluster_log_to_stderr false | |
73 | ||
74 | By default, cephadm sets up log rotation on each host to rotate these | |
75 | files. You can configure the logging retention schedule by modifying | |
76 | ``/etc/logrotate.d/ceph.<cluster-fsid>``. | |
77 | ||
78 | ||
79 | Data location | |
80 | ============= | |
81 | ||
82 | Cephadm daemon data and logs in slightly different locations than older | |
83 | versions of ceph: | |
84 | ||
85 | * ``/var/log/ceph/<cluster-fsid>`` contains all cluster logs. Note | |
86 | that by default cephadm logs via stderr and the container runtime, | |
87 | so these logs are normally not present. | |
88 | * ``/var/lib/ceph/<cluster-fsid>`` contains all cluster daemon data | |
89 | (besides logs). | |
90 | * ``/var/lib/ceph/<cluster-fsid>/<daemon-name>`` contains all data for | |
91 | an individual daemon. | |
92 | * ``/var/lib/ceph/<cluster-fsid>/crash`` contains crash reports for | |
93 | the cluster. | |
94 | * ``/var/lib/ceph/<cluster-fsid>/removed`` contains old daemon | |
95 | data directories for stateful daemons (e.g., monitor, prometheus) | |
96 | that have been removed by cephadm. | |
97 | ||
98 | Disk usage | |
99 | ---------- | |
100 | ||
101 | Because a few Ceph daemons may store a significant amount of data in | |
102 | ``/var/lib/ceph`` (notably, the monitors and prometheus), we recommend | |
103 | moving this directory to its own disk, partition, or logical volume so | |
104 | that it does not fill up the root file system. | |
105 | ||
106 | ||
107 | ||
108 | SSH Configuration | |
109 | ================= | |
110 | ||
111 | Cephadm uses SSH to connect to remote hosts. SSH uses a key to authenticate | |
112 | with those hosts in a secure way. | |
113 | ||
114 | ||
115 | Default behavior | |
116 | ---------------- | |
117 | ||
118 | Cephadm stores an SSH key in the monitor that is used to | |
119 | connect to remote hosts. When the cluster is bootstrapped, this SSH | |
120 | key is generated automatically and no additional configuration | |
121 | is necessary. | |
122 | ||
123 | A *new* SSH key can be generated with:: | |
124 | ||
125 | ceph cephadm generate-key | |
126 | ||
127 | The public portion of the SSH key can be retrieved with:: | |
128 | ||
129 | ceph cephadm get-pub-key | |
130 | ||
131 | The currently stored SSH key can be deleted with:: | |
132 | ||
133 | ceph cephadm clear-key | |
134 | ||
135 | You can make use of an existing key by directly importing it with:: | |
136 | ||
137 | ceph config-key set mgr/cephadm/ssh_identity_key -i <key> | |
138 | ceph config-key set mgr/cephadm/ssh_identity_pub -i <pub> | |
139 | ||
140 | You will then need to restart the mgr daemon to reload the configuration with:: | |
141 | ||
142 | ceph mgr fail | |
143 | ||
144 | ||
145 | Customizing the SSH configuration | |
146 | --------------------------------- | |
147 | ||
148 | Cephadm generates an appropriate ``ssh_config`` file that is | |
149 | used for connecting to remote hosts. This configuration looks | |
150 | something like this:: | |
151 | ||
152 | Host * | |
153 | User root | |
154 | StrictHostKeyChecking no | |
155 | UserKnownHostsFile /dev/null | |
156 | ||
157 | There are two ways to customize this configuration for your environment: | |
158 | ||
159 | #. Import a customized configuration file that will be stored | |
160 | by the monitor with:: | |
161 | ||
162 | ceph cephadm set-ssh-config -i <ssh_config_file> | |
163 | ||
164 | To remove a customized SSH config and revert back to the default behavior:: | |
165 | ||
166 | ceph cephadm clear-ssh-config | |
167 | ||
168 | #. You can configure a file location for the SSH configuration file with:: | |
169 | ||
170 | ceph config set mgr mgr/cephadm/ssh_config_file <path> | |
171 | ||
172 | We do *not recommend* this approach. The path name must be | |
173 | visible to *any* mgr daemon, and cephadm runs all daemons as | |
174 | containers. That means that the file either need to be placed | |
175 | inside a customized container image for your deployment, or | |
176 | manually distributed to the mgr data directory | |
177 | (``/var/lib/ceph/<cluster-fsid>/mgr.<id>`` on the host, visible at | |
178 | ``/var/lib/ceph/mgr/ceph-<id>`` from inside the container). | |
179 | ||
180 | ||
181 | Health checks | |
182 | ============= | |
183 | ||
184 | CEPHADM_PAUSED | |
185 | -------------- | |
186 | ||
187 | Cephadm background work has been paused with ``ceph orch pause``. Cephadm | |
188 | continues to perform passive monitoring activities (like checking | |
189 | host and daemon status), but it will not make any changes (like deploying | |
190 | or removing daemons). | |
191 | ||
192 | Resume cephadm work with:: | |
193 | ||
194 | ceph orch resume | |
195 | ||
196 | CEPHADM_STRAY_HOST | |
197 | ------------------ | |
198 | ||
199 | One or more hosts have running Ceph daemons but are not registered as | |
200 | hosts managed by *cephadm*. This means that those services cannot | |
201 | currently be managed by cephadm (e.g., restarted, upgraded, included | |
202 | in `ceph orch ps`). | |
203 | ||
204 | You can manage the host(s) with:: | |
205 | ||
206 | ceph orch host add *<hostname>* | |
207 | ||
208 | Note that you may need to configure SSH access to the remote host | |
209 | before this will work. | |
210 | ||
211 | Alternatively, you can manually connect to the host and ensure that | |
212 | services on that host are removed or migrated to a host that is | |
213 | managed by *cephadm*. | |
214 | ||
215 | You can also disable this warning entirely with:: | |
216 | ||
217 | ceph config set mgr mgr/cephadm/warn_on_stray_hosts false | |
218 | ||
219 | CEPHADM_STRAY_DAEMON | |
220 | -------------------- | |
221 | ||
222 | One or more Ceph daemons are running but not are not managed by | |
223 | *cephadm*. This may be because they were deployed using a different | |
224 | tool, or because they were started manually. Those | |
225 | services cannot currently be managed by cephadm (e.g., restarted, | |
226 | upgraded, or included in `ceph orch ps`). | |
227 | ||
228 | If the daemon is a stateful one (monitor or OSD), it should be adopted | |
229 | by cephadm; see :ref:`cephadm-adoption`. For stateless daemons, it is | |
230 | usually easiest to provision a new daemon with the ``ceph orch apply`` | |
231 | command and then stop the unmanaged daemon. | |
232 | ||
233 | This warning can be disabled entirely with:: | |
234 | ||
235 | ceph config set mgr mgr/cephadm/warn_on_stray_daemons false | |
236 | ||
237 | CEPHADM_HOST_CHECK_FAILED | |
238 | ------------------------- | |
239 | ||
240 | One or more hosts have failed the basic cephadm host check, which verifies | |
241 | that (1) the host is reachable and cephadm can be executed there, and (2) | |
242 | that the host satisfies basic prerequisites, like a working container | |
243 | runtime (podman or docker) and working time synchronization. | |
244 | If this test fails, cephadm will no be able to manage services on that host. | |
245 | ||
246 | You can manually run this check with:: | |
247 | ||
248 | ceph cephadm check-host *<hostname>* | |
249 | ||
250 | You can remove a broken host from management with:: | |
251 | ||
252 | ceph orch host rm *<hostname>* | |
253 | ||
254 | You can disable this health warning with:: | |
255 | ||
256 | ceph config set mgr mgr/cephadm/warn_on_failed_host_check false |