1 ===============================================================================
2 A Detailed Documentation on How to Set up Ceph Kerberos Authentication
3 ===============================================================================
5 This document provides details on the Kerberos authorization protocol. This is
6 the 1st draft and we will try to keep it updated along with code changes that
9 Several free implementations of this protocol are available (MIT, Heimdal,
10 MS...), covering a wide range of operating systems. The Massachusetts
11 Institute of Technology (MIT), where Kerberos was originally developed,
12 continues to develop their Kerberos package and it is the implementation we
13 chose to work with. `MIT Kerberos <http://web.mit.edu/Kerberos/>`_.
15 Please, provide feedback to Daniel Oliveira (doliveira@suse.com)
17 *Last update: Dec 3, 2018*
24 Before we get into *Kerberos details*, let us define a few terms so we can
25 understand what to expect from it, *what it can and can't do*:
28 A directory service is a customizable information store that functions as
29 a single point from which users can locate resources and services
30 distributed throughout the network. This customizable information store
31 also gives administrators a single point for managing its objects and their
32 attributes. Although this information store appears as a single point to
33 the users of the network, it is actually most often stored in a distributed
34 form. A directory service consists of at least one *Directory Server and a
35 Directory Client* and are implemented based on *X.500 standards*.
37 *OpenLDAP, 389 Directory Server, MS Active Directory, NetIQ eDirectory* are
40 A directory service is often characterized as a *write-once-read-many-times
41 service*, meaning the data that would normally be stored in an directory
42 service would not be expected to change on every access.
44 The database that forms a directory service *is not designed for
49 LDAP (Lightweight Directory Access Protocol v3)
50 LDAP is a set of LDAP Protocol Exchanges *(not an implementation of a
51 server)* that defines the method by which data is accessed. LDAPv3 is a
52 standard defined by the IETF in RFC 2251 and describes how data is
53 represented in the Directory Service (the Data Model or DIT).
55 Finally, it defines how data is loaded into (imported) and saved from
56 (exported) a directory service (using LDIF). LDAP does not define how data
57 is stored or manipulated. Data Store is an 'automagic' process as far as
58 the standard is concerned and is generally handled by back-end modules.
60 No Directory Service implementation has all the features of LDAP v3
61 protocol implemented. All Directory Server implementations have their
62 different problems and/or anomalies, and features that may not return
63 results as another Directory Server implementation would.
68 Authentication is about validating credentials (like User Name/ID and
69 password) to verify the identity. The system determines whether one is what
70 they say they are using their credentials.
72 Usually, authentication is done by a username and password, and sometimes
73 in conjunction with *(single, two, or multi) factors of authentication*,
74 which refers to the various ways to be authenticated.
79 Authorization occurs after the identity is successfully authenticated by
80 the system, which ultimately gives one full permission to access the
81 resources such as information, files, databases, and so forth, almost
82 anything. It determines the ability to access the system and up to what
83 extent (what kind of permissions/rights are given and to where/what).
88 Auditing takes the results from both *authentication and authorization* and
89 records them into an audit log. The audit log records records all actions
90 taking by/during the authentication and authorization for later review by
91 the administrators. While authentication and authorization are preventive
92 systems (in which unauthorized access is prevented), auditing is a reactive
93 system (in which it gives detailed log of how/when/where someone accessed
99 Kerberos is a network *authentication protocol*. It is designed to provide
100 strong authentication for client/server applications by using secret-key
101 cryptography (symmetric key). A free implementation of this protocol is
102 available from the MIT. However, Kerberos is available in many commercial
105 It was designed to provide secure authentication to services over an
106 insecure network. Kerberos uses tickets to authenticate a user, or service
107 application and never transmits passwords over the network in the clear.
108 So both client and server can prove their identity without sending any
109 unencrypted secrets over the network.
111 Kerberos can be used for single sign-on (SSO). The idea behind SSO is
112 simple, we want to login just once and be able to use any service that we
113 are entitled to, without having to login on each of those services.
117 Simple Authentication and Security Layer (SASL)
118 SASL **(RFC 4422)** is a framework that helps developers to implement
119 different authentication mechanisms (implementing a series of challenges
120 and responses), allowing both clients and servers to negotiate a mutually
121 acceptable mechanism for each connection, instead of hard-coding them.
123 Examples of SASL mechanisms:
125 * ANONYMOUS **(RFC 4505)**
127 - For guest access, meaning *unauthenticated*
129 * CRAM-MD5 **(RFC 2195)**
131 - Simple challenge-response scheme based on *HMAC-MD5*.
132 It does not establish any security layer. *Less secure than
133 DIGEST-MD5 and GSSAPI.*
135 * DIGEST-MD5 **(RFC 2831)**
137 - HTTP Digest compatible *(partially)* challenge-response scheme
138 based upon MD5, offering a *data security layer*. It is preferred
139 over PLAIN text passwords, protecting against plain text attacks.
140 It is a mandatory authentication method for LDAPv3 servers.
142 * EXTERNAL **(RFCs 4422, 5246, 4301, 2119)**
144 - Where *authentication is implicit* in the context (i.e; for
145 protocols using IPsec or TLS [TLS/SSL to performing certificate-
146 based authentication] already). This method uses public keys for
147 strong authentication.
151 - Family of mechanisms supports arbitrary GSS-API mechanisms in
154 * NTLM (MS Proprietary)
156 - MS Windows NT LAN Manager authentication mechanism
158 * OAuth 1.0/2.0 **(RFCs 5849, 6749, 7628)**
160 - Authentication protocol for delegated resource access
164 - One-time password mechanism *(obsoletes the SKEY mechanism)*
166 * PLAIN **(RFC 4616)**
168 - Simple Cleartext password mechanism **(RFC 4616)**. This is not a
169 preferred mechanism for most applications because of its relative
172 * SCRAM **(RFCs 5802, 7677)**
174 - Modern challenge-response scheme based mechanism with channel
179 Generic Security Services Application Program Interface (GSSAPI)
180 GSSAPI **(RFCs 2078, 2743, 2744, 4121, 4752)** is widely used by protocol
181 implementers as a way to implement Kerberos v5 support in their
182 applications. It provides a generic interface and message format that can
183 encapsulate authentication exchanges from any authentication method that
184 has a GSSAPI-compliant library.
186 It does not define a protocol, authentication, or security mechanism
187 itself; it instead makes it easier for application programmers to support
188 multiple authentication mechanisms by providing a uniform, generic API for
189 security services. It is a set of functions that include both an API and a
190 methodology for approaching authentication, aiming to insulate application
191 protocols from the specifics of security protocols as much as possible.
193 *Microsoft Windows Kerberos* implementation does not include GSSAPI support
194 but instead includes a *Microsoft-specific API*, the *Security Support
195 Provider Interface (SSPI)*. In Windows, an SSPI client can communicate with
198 *Most applications that support GSSAPI also support Kerberos v5.*
202 Simple and Protected GSSAPI Negotiation Mechanism (SPNEGO)
203 As we can see, GSSAPI solves the problem of providing a single API to
204 different authentication mechanisms. However, it does not solve the problem
205 of negotiating which mechanism to use. In fact for GSSAPI to work, the two
206 applications communicating with each other must know in advance what
207 authentication mechanism they plan to use, which usually is not a problem
208 if only one mechanism is supported (meaning Kerberos v5).
210 However, if there are multiple mechanisms to choose from, a method is
211 needed to securely negotiate an authentication mechanism that is mutually
212 supported between both client and server; which is where
213 *SPNEGO (RFC 2478, 4178)* makes a difference.
215 *SPNEGO* provides a framework for two parties that are engaged in
216 authentication to select from a set of possible authentication mechanisms,
217 in a manner that preserves the opaque nature of the security protocols to
218 the application protocol that uses it.
220 It is a security protocol that uses a *GSSAPI authentication mechanism* and
221 negotiates among several available authentication mechanisms in an
222 implementation, selecting one for use to satisfy the authentication needs
223 of the application protocol.
225 It is a *meta protocol* that travels entirely in other application
226 protocols; it is never used directly without an application protocol.
230 *Why is this important and why do we care? Like, at all?*
232 Having this background information in mind, we can easily describe things
235 1. *Ceph Kerberos authentication* is based totally on MIT *Kerberos*
236 implementation using *GSSAPI*.
238 2. At the moment we are still using *Kerberos default backend
239 database*, however we plan on adding LDAP as a backend which would
240 provide us with *authentication with GSSAPI (KRB5)* and *authorization
241 with LDAP (LDAPv3)*, via *SASL mechanism*.
248 We assume the environment already has some external services up and running
251 * Kerberos needs to be properly configured, which also means (for both
252 every server and KDC):
254 - Time Synchronization (either using `NTP <http://www.ntp.org/>`_ or `chrony <https://chrony.tuxfamily.org/>`_).
256 + Not only Kerberos, but also Ceph depends and relies on time
261 + Both *(forward and reverse)* zones, with *fully qualified domain
262 name (fqdn)* ``(hostname + domain.name)``
264 + KDC discover can be set up to use DNS ``(srv resources)`` as
265 service location protocol *(RFCs 2052, 2782)*, as well as *host
266 or domain* to the *appropriate realm* ``(txt record)``.
268 + Even though these DNS entries/settings are not required to run a
269 ``Kerberos realm``, they certainly help to eliminate the need for
270 manual configuration on all clients.
272 + This is extremely important, once most of the Kerberos issues are
273 usually related to name resolution. Kerberos is very picky when
274 checking on systems names and host lookups.
276 * Whenever possible, in order to avoid a *single point of failure*, set up
277 a *backup, secondary, or slave*, for every piece/part in the
278 infrastructure ``(ntp, dns, and kdc servers)``.
281 Also, the following *Kerberos terminology* is important:
285 - Tickets or Credentials, are a set of information that can be used to
286 verify the client's identity. Kerberos tickets may be stored in a
287 file, or they may exist only in memory.
289 - The first ticket obtained is a ticket-granting ticket (TGT), which
290 allows the clients to obtain additional tickets. These additional
291 tickets give the client permission for specific services. The
292 requesting and granting of these additional tickets happens
295 + The TGT, which expires at a specified time, permits the client to
296 obtain additional tickets, which give permission for specific
297 services. The requesting and granting of these additional tickets
300 * Key Distribution Center (KDC).
302 - The KDC creates a ticket-granting ticket (TGT) for the client,
303 encrypts it using the client's password as the key, and sends the
304 encrypted TGT back to the client. The client then attempts to decrypt
305 the TGT, using its password. If the client successfully decrypts the
306 TGT (i.e., if the client gave the correct password), it keeps the
307 decrypted TGT, which indicates proof of the client's identity.
309 - The KDC is comprised of three components:
311 + Kerberos database, which stores all the information about the
312 principals and the realm they belong to, among other things.
313 + Authentication service (AS)
314 + Ticket-granting service (TGS)
318 - Either a *user, host or a service* who sends a request for a ticket.
322 - It is a unique identity to which Kerberos can assign tickets.
323 Principals can have an arbitrary number of components. Each component
324 is separated by a component separator, generally ``/``. The last
325 component is the *realm*, separated from the rest of the principal by
326 the realm separator, generally ``@``.
328 - If there is no realm component in the principal, then it will be
329 assumed that the principal is in the default realm for the context in
330 which it is being used.
332 - Usually, a principal is divided into three parts:
334 + The ``primary``, the ``instance``, and the ``realm``
336 + The format of a typical Kerberos V5 principal is
337 ``primary/instance@REALM``.
339 + The ``primary`` is the first part of the principal. In the case
340 of a user, it's the same as the ``username``. For a host, the
341 primary is the word ``host``. For Ceph, will use ``ceph`` as a
342 primary name which makes it easier to organize and identify Ceph
345 + The ``instance`` is an optional string that qualifies the
346 primary. The instance is separated from the primary by a slash
347 ``/``. In the case of a user, the instance is usually ``null``,
348 but a user might also have an additional principal, with an
349 instance called ``admin``, which one uses to administrate a
352 The principal ``johndoe@MYDOMAIN.COM`` is completely separate
353 from the principal ``johndoe/admin@MYDOMAIN.COM``, with a
354 separate password, and separate permissions. In the case of a
355 host, the instance is the fully qualified hostname,
356 i.e., ``osd1.MYDOMAIN.COM``.
358 + The ``realm`` is the Kerberos realm. Usually, the Kerberos realm
359 is the domain name, in *upper-case letters*. For example, the
360 machine ``osd1.MYDOMAIN.COM`` would be in the realm
365 - A keytab file stores the actual encryption key that can be used in
366 lieu of a password challenge for a given principal. Creating keytab
367 files are useful for noninteractive principals, such as *Service
368 Principal Names*, which are often associated with long-running
369 processes like Ceph daemons. A keytab file does not have to be a
370 "1:1 mapping" to a single principal. Multiple different principal
371 keys can be stored in a single keytab file:
373 + The keytab file allows a user/service to authenticate without
374 knowledge of the password. Due to this, *keytabs should be
375 protected* with appropriate controls to prevent unauthorized
376 users from authenticating with it.
378 + The default client keytab file is ``/etc/krb5.keytab``
382 The 'Ceph side' of the things
383 ------------------------------
385 In order to configure connections (from Ceph nodes) to the KDC:
387 1. Login to the Kerberos client (Ceph server nodes) and confirm it is properly
388 configured, by checking and editing ``/etc/krb5.conf`` file properly: ::
392 dns_canonicalize_hostname = false
395 dns_lookup_realm = true
396 dns_lookup_kdc = true
397 allow_weak_crypto = false
398 default_realm = MYDOMAIN.COM
399 default_ccache_name = KEYRING:persistent:%{uid}
402 kdc = kerberos.mydomain.com
403 admin_server = kerberos.mydomain.com
409 2. Login to the *KDC Server* and confirm it is properly configured to
410 authenticate to the Kerberos realm in question:
412 a. Kerberos related DNS RRs: ::
414 /var/lib/named/master/mydomain.com
415 kerberos IN A 192.168.10.21
416 kerberos-slave IN A 192.168.10.22
417 _kerberos IN TXT "MYDOMAIN.COM"
418 _kerberos._udp IN SRV 1 0 88 kerberos
419 _kerberos._tcp IN SRV 1 0 88 kerberos
420 _kerberos._udp IN SRV 20 0 88 kerberos-slave
421 _kerberos-master._udp IN SRV 0 0 88 kerberos
422 _kerberos-adm._tcp IN SRV 0 0 749 kerberos
423 _kpasswd._udp IN SRV 0 0 464 kerberos
427 b. KDC configuration file: ::
429 /var/lib/kerberos/krb5kdc/kdc.conf
434 acl_file = /var/lib/kerberos/krb5kdc/kadm5.acl
435 admin_keytab = FILE:/var/lib/kerberos/krb5kdc/kadm5.keytab
436 default_principal_flags = +postdateable +forwardable +renewable +proxiable
437 +dup-skey -preauth -hwauth +service
438 +tgt-based +allow-tickets -pwchange
440 dict_file = /var/lib/kerberos/krb5kdc/kadm5.dict
441 key_stash_file = /var/lib/kerberos/krb5kdc/.k5.MYDOMAIN.COM
443 max_life = 0d 10h 0m 0s
444 max_renewable_life = 7d 0h 0m 0s
449 3. Still on the KDC Server, run the Kerberos administration utility;
450 ``kadmin.local`` so we can list all the principals already created. ::
452 kadmin.local: listprincs
454 krbtgt/MYDOMAIN.COM@MYDOMAIN.COM
455 kadmin/admin@MYDOMAIN.COM
456 kadmin/changepw@MYDOMAIN.COM
457 kadmin/history@MYDOMAIN.COM
458 kadmin/kerberos.mydomain.com@MYDOMAIN.COM
459 root/admin@MYDOMAIN.COM
463 4. Add a *principal for each Ceph cluster node* we want to be authenticated by
466 a. Adding principals: ::
468 kadmin.local: addprinc -randkey ceph/ceph-mon1
469 Principal "ceph/ceph-mon1@MYDOMAIN.COM" created.
470 kadmin.local: addprinc -randkey ceph/ceph-osd1
471 Principal "ceph/ceph-osd1@MYDOMAIN.COM" created.
472 kadmin.local: addprinc -randkey ceph/ceph-osd2
473 Principal "ceph/ceph-osd2@MYDOMAIN.COM" created.
474 kadmin.local: addprinc -randkey ceph/ceph-osd3
475 Principal "ceph/ceph-osd3@MYDOMAIN.COM" created.
476 kadmin.local: addprinc -randkey ceph/ceph-osd4
477 Principal "ceph/ceph-osd4@MYDOMAIN.COM" created.
478 kadmin.local: listprincs
480 krbtgt/MYDOMAIN.COM@MYDOMAIN.COM
481 kadmin/admin@MYDOMAIN.COM
482 kadmin/changepw@MYDOMAIN.COM
483 kadmin/history@MYDOMAIN.COM
484 kadmin/kerberos.mydomain.com@MYDOMAIN.COM
485 root/admin@MYDOMAIN.COM
486 ceph/ceph-mon1@MYDOMAIN.COM
487 ceph/ceph-osd1@MYDOMAIN.COM
488 ceph/ceph-osd2@MYDOMAIN.COM
489 ceph/ceph-osd3@MYDOMAIN.COM
490 ceph/ceph-osd4@MYDOMAIN.COM
494 b. This follows the same idea if we are creating a *user principal* ::
496 kadmin.local: addprinc johndoe
497 WARNING: no policy specified for johndoe@MYDOMAIN.COM; defaulting to no policy
498 Enter password for principal "johndoe@MYDOMAIN.COM":
499 Re-enter password for principal "johndoe@MYDOMAIN.COM":
500 Principal "johndoe@MYDOMAIN.COM" created.
504 5. Create a *keytab file* for each Ceph cluster node:
506 As the default client keytab file is ``/etc/krb5.keytab``, we will want to
507 use a different file name, so we especify which *keytab file to create* and
508 which *principal to export keys* from: ::
510 kadmin.local: ktadd -k /etc/gss_client_mon1.ktab ceph/ceph-mon1
511 Entry for principal ceph/ceph-mon1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/etc/gss_client_mon1.ktab.
512 Entry for principal ceph/ceph-mon1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/etc/gss_client_mon1.ktab.
513 Entry for principal ceph/ceph-mon1 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/etc/gss_client_mon1.ktab.
514 Entry for principal ceph/ceph-mon1 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/etc/gss_client_mon1.ktab.
515 kadmin.local: ktadd -k /etc/gss_client_osd1.ktab ceph/ceph-osd1
516 Entry for principal ceph/ceph-osd1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/etc/gss_client_osd1.ktab.
517 Entry for principal ceph/ceph-osd1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/etc/gss_client_osd1.ktab.
518 Entry for principal ceph/ceph-osd1 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/etc/gss_client_osd1.ktab.
519 Entry for principal ceph/ceph-osd1 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/etc/gss_client_osd1.ktab.
520 kadmin.local: ktadd -k /etc/gss_client_osd2.ktab ceph/ceph-osd2
521 Entry for principal ceph/ceph-osd2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/etc/gss_client_osd2.ktab.
522 Entry for principal ceph/ceph-osd2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/etc/gss_client_osd2.ktab.
523 Entry for principal ceph/ceph-osd2 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/etc/gss_client_osd2.ktab.
524 Entry for principal ceph/ceph-osd2 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/etc/gss_client_osd2.ktab.
525 kadmin.local: ktadd -k /etc/gss_client_osd3.ktab ceph/ceph-osd3
526 Entry for principal ceph/ceph-osd3 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/etc/gss_client_osd3.ktab.
527 Entry for principal ceph/ceph-osd3 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/etc/gss_client_osd3.ktab.
528 Entry for principal ceph/ceph-osd3 with kvno 3, encryption type des3-cbc-sha1 added to keytab WRFILE:/etc/gss_client_osd3.ktab.
529 Entry for principal ceph/ceph-osd3 with kvno 3, encryption type arcfour-hmac added to keytab WRFILE:/etc/gss_client_osd3.ktab.
530 kadmin.local: ktadd -k /etc/gss_client_osd4.ktab ceph/ceph-osd4
531 Entry for principal ceph/ceph-osd4 with kvno 4, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/etc/gss_client_osd4.ktab.
532 Entry for principal ceph/ceph-osd4 with kvno 4, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/etc/gss_client_osd4.ktab.
533 Entry for principal ceph/ceph-osd4 with kvno 4, encryption type des3-cbc-sha1 added to keytab WRFILE:/etc/gss_client_osd4.ktab.
534 Entry for principal ceph/ceph-osd4 with kvno 4, encryption type arcfour-hmac added to keytab WRFILE:/etc/gss_client_osd4.ktab.
536 # ls -1 /etc/gss_client_*
537 /etc/gss_client_mon1.ktab
538 /etc/gss_client_osd1.ktab
539 /etc/gss_client_osd2.ktab
540 /etc/gss_client_osd3.ktab
541 /etc/gss_client_osd4.ktab
544 We can also check these newly created keytab client files by: ::
546 # klist -kte /etc/gss_client_mon1.ktab
547 Keytab name: FILE:/etc/gss_client_mon1.ktab
548 KVNO Timestamp Principal
549 ---- ------------------- ------------------------------------------------------
550 2 10/8/2018 14:35:30 ceph/ceph-mon1@MYDOMAIN.COM (aes256-cts-hmac-sha1-96)
551 2 10/8/2018 14:35:31 ceph/ceph-mon1@MYDOMAIN.COM (aes128-cts-hmac-sha1-96)
552 2 10/8/2018 14:35:31 ceph/ceph-mon1@MYDOMAIN.COM (des3-cbc-sha1)
553 2 10/8/2018 14:35:31 ceph/ceph-mon1@MYDOMAIN.COM (arcfour-hmac)
557 6. A new *set parameter* was added in Ceph, ``gss ktab client file`` which
558 points to the keytab file related to the Ceph node *(or principal)* in
561 By default it points to ``/var/lib/ceph/$name/gss_client_$name.ktab``. So,
562 in the case of a Ceph server ``osd1.mydomain.com``, the location and name
563 of the keytab file should be: ``/var/lib/ceph/osd1/gss_client_osd1.ktab``
565 Therefore, we need to ``scp`` each of these newly created keytab files from
566 the KDC to their respective Ceph cluster nodes (i.e):
567 ``# for node in mon1 osd1 osd2 osd3 osd4; do scp /etc/gss_client_$node*.ktab root@ceph-$node:/var/lib/ceph/$node/; done``
569 Or whatever other way one feels comfortable with, as long as each keytab
570 client file gets copied over to the proper location.
572 At this point, even *without using any keytab client file* we should be
573 already able to authenticate a *user principal*: ::
575 # kdestroy -A && kinit -f johndoe && klist -f
576 Password for johndoe@MYDOMAIN.COM:
577 Ticket cache: KEYRING:persistent:0:0
578 Default principal: johndoe@MYDOMAIN.COM
580 Valid starting Expires Service principal
581 10/10/2018 15:32:01 10/11/2018 07:32:01 krbtgt/MYDOMAIN.COM@MYDOMAIN.COM
582 renew until 10/11/2018 15:32:01, Flags: FRI
586 Given that the *keytab client file* is/should already be copied and available at the
587 Kerberos client (Ceph cluster node), we should be able to athenticate using it before
590 # kdestroy -A && kinit -k -t /etc/gss_client_mon1.ktab -f 'ceph/ceph-mon1@MYDOMAIN.COM' && klist -f
591 Ticket cache: KEYRING:persistent:0:0
592 Default principal: ceph/ceph-mon1@MYDOMAIN.COM
594 Valid starting Expires Service principal
595 10/10/2018 15:54:25 10/11/2018 07:54:25 krbtgt/MYDOMAIN.COM@MYDOMAIN.COM
596 renew until 10/11/2018 15:54:25, Flags: FRI
600 7. The default client keytab is used, if it is present and readable, to
601 automatically obtain initial credentials for GSSAPI client applications. The
602 principal name of the first entry in the client keytab is used by default
603 when obtaining initial credentials:
605 a. The ``KRB5_CLIENT_KTNAME environment`` variable.
606 b. The ``default_client_keytab_name`` profile variable in ``[libdefaults]``.
607 c. The hardcoded default, ``DEFCKTNAME``.
609 So, what we do is to internally, set the environment variable
610 ``KRB5_CLIENT_KTNAME`` to the same location as ``gss_ktab_client_file``,
611 so ``/var/lib/ceph/osd1/gss_client_osd1.ktab``, and change the ``ceph.conf``
612 file to add the new authentication method. ::
617 auth cluster required = gss
618 auth service required = gss
619 auth client required = gss
620 gss ktab client file = /{$my_new_location}/{$my_new_ktab_client_file.keytab}
624 8. With that the GSSAPIs will then be able to read the keytab file and using
625 the process of name and service resolution *(provided by the DNS)*, able to
626 request a *TGT* as follows:
628 a. User/Client sends principal identity and credentials to the KDC Server
630 b. KDC checks its internal database for the principal in question.
631 c. a TGT is created and wrapped by the KDC, using the principal's key
633 d. The newly created TGT, is decrypted and stored in the credentials
635 e. At this point, Kerberos/GSSAPI aware applications (and/or services) are
636 able to check the list of active TGT in the keytab file.
641 ** *For Ceph Developers Only* **
642 =================================
644 We certainly could have used straight native ``KRB5 APIs`` (instead of
645 ``GSSAPIs``), but we wanted a more portable option as regards network security,
646 which is the hallmark of the ``GSS`` *(Generic Security Standard)* ``-API``.
647 It does not actually provide security services itself.
649 Rather, it is a framework that provides security services to callers in a
652 +---------------------------------+
654 +---------------------------------+
655 | Protocol (RPC, Etc. [Optional]) |
656 +---------------------------------+
658 +---------------------------------+
659 | Security Mechs (Krb v5, Etc) |
660 +---------------------------------+
663 The GSS-API does two main things:
665 1. It creates a security context in which data can be passed between
666 applications. A context can be thought of as a sort of *"state of trust"*
667 between two applications.
669 Applications that share a context know who each other are and thus can
670 permit data transfers between them as long as the context lasts.
672 2. It applies one or more types of protection, known as *"security services"*,
673 to the data to be transmitted.
676 GSS-API provides several types of portability for applications:
678 a. **Mechanism independence.** GSS-API provides a generic interface to the
679 mechanisms for which it has been implemented. By specifying a default
680 security mechanism, an application does not need to know which mechanism
681 it is using (for example, Kerberos v5), or even what type of mechanism
682 it uses. As an example, when an application forwards a user's credential
683 to a server, it does not need to know if that credential has a Kerberos
684 format or the format used by some other mechanism, nor how the
685 credentials are stored by the mechanism and accessed by the application.
686 (If necessary, an application can specify a particular mechanism to use)
688 b. **Protocol independence.** The GSS-API is independent of any
689 communications protocol or protocol suite. It can be used with
690 applications that use, for example, sockets, RCP, or TCP/IP.
691 RPCSEC_GSS "RPCSEC_GSS Layer" is an additional layer that smoothly
692 integrates GSS-API with RPC.
694 c. **Platform independence.** The GSS-API is completely oblivious to the
695 type of operating system on which an application is running.
697 d. **Quality of Protection independence.** Quality of Protection (QOP) is
698 the name given to the type of algorithm used in encrypting data or
699 generating cryptographic tags; the GSS-API allows a programmer to ignore
700 QOP, using a default provided by the GSS-API.
701 (On the other hand, an application can specify the QOP if necessary.)
703 The basic security offered by the GSS-API is authentication. Authentication
704 is the verification of an identity: *if you are authenticated, it means
705 that you are recognized to be who you say you are.*
707 The GSS-API provides for two additional security services, if supported by the
708 underlying mechanisms:
710 1. **Integrity:** It's not always sufficient to know that an application
711 sending you data is who it claims to be. The data itself could have
712 become corrupted or compromised.
714 The GSS-API provides for data to be accompanied by a cryptographic tag,
715 known as an ``Message Integrity Code (MIC)``, to prove that the data
716 that arrives at your doorstep is the same as the data that the sender
717 transmitted. This verification of the data's validity is known as
720 2. **Confidentiality:** Both authentication and integrity, however, leave
721 the data itself alone, so if it's somehow intercepted, others can read
724 The GSS-API therefore allows data to be encrypted, if underlying
725 mechanisms support it. This encryption of data is known as *"confidentiality"*.
729 Mechanisms Available With GSS-API:
731 The current implementation of the GSS-API works only with the Kerberos v5 security
734 Mechanism Name Object Identifier Shared Library Kernel Module
735 ---------------------- ---------------------- -------------- --------------
736 diffie_hellman_640_0 1.3.6.4.1.42.2.26.2.4 dh640-0.so.1
737 diffie_hellman_1024_0 1.3.6.4.1.42.2.26.2.5 dh1024-0.so.1
740 SCRAM-SHA-1 1.3.6.1.5.5.14
741 SCRAM-SHA-256 1.3.6.1.5.5.18
742 GSS-EAP (arc) 1.3.6.1.5.5.15.1.1.*
743 kerberos_v5 1.2.840.113554.1.2.2 gl/mech_krb5.so gl_kmech_krb5
746 Kerberos Version 5 GSS-API Mechanism
747 OID {1.2.840.113554.1.2.2}
749 Kerberos Version 5 GSS-API Mechanism
750 Simple and Protected GSS-API Negotiation Mechanism
754 There are two different formats:
756 1. The first, ``{ 1 2 3 4 }``, is officially mandated by the GSS-API
757 specs. ``gss_str_to_oid()`` expects this first format.
759 2. The second, ``1.2.3.4``, is more widely used but is not an official
762 Although the GSS-API makes protecting data simple, it does not do certain
763 things, in order to maximize its generic nature. These include:
765 a. Provide security credentials for a user or application. These must
766 be provided by the underlying security mechanism(s). The GSS-API
767 does allow applications to acquire credentials, either automatically
770 b. Transfer data between applications. It is the application's
771 responsibility to handle the transfer of all data between peers,
772 whether it is security-related or "plain" data.
774 c. Distinguish between different types of transmitted data (for
775 example, to know or determine that a data packet is plain data and
776 not GSS-API related).
778 d. Indicate status due to remote (asynchronous) errors.
780 e. Automatically protect information sent between processes of a
781 multiprocess program.
783 f. Allocate string buffers ("Strings and Similar Data") to be passed to
786 g. Deallocate GSS-API data spaces. These must be explicitly deallocated
787 with functions such as ``gss_release_buffer()`` and
788 ``gss_delete_name()``.
792 These are the basic steps in using the GSS-API:
794 1. Each application, sender and recipient, acquires credentials explicitly,
795 if credentials have not been acquired automatically.
797 2. The sender initiates a security context and the recipient accepts it.
799 3. The sender applies security protection to the message (data) it wants to
800 transmit. This means that it either encrypts the message or stamps it
801 with an identification tag. The sender transmits the protected message.
802 (The sender can choose not to apply either security protection, in which
803 case the message has only the default GSS-API security service
804 associated with it. That is authentication, in which the recipient knows
805 that the sender is who it claims to be.)
807 4. The recipient decrypts the message (if needed) and verifies it
810 5. (Optional) The recipient returns an identification tag to the sender for
813 6. Both applications destroy the shared security context. If necessary,
814 they can also deallocate any *"leftover"* GSS-API data.
816 Applications that use the GSS-API should include the file ``gssapi.h``.
819 - `rfc1964 <https://tools.ietf.org/html/rfc1964>`_.
820 - `rfc2743 <https://tools.ietf.org/html/rfc2743>`_.
821 - `rfc2744 <https://tools.ietf.org/html/rfc2744>`_.
822 - `rfc4178 <https://tools.ietf.org/html/rfc4178>`_.
823 - `rfc6649 <https://tools.ietf.org/html/rfc6649>`_.
824 - `MIT Kerberos Documentation <https://web.mit.edu/kerberos/krb5-latest/doc/appdev/gssapi.html>`_.
828 ** *Kerberos Server Setup* **
829 ------------------------------
831 First and foremost, ``this is not a recommendation for a production
832 environment``. We are not covering ``Master/Slave replication cluster`` or
833 anything production environment related (*ntp/chrony, dns, pam/nss, sssd, etc*).
835 Also, on the server side there might be different dependencies and/or
836 configuration steps needed, depending on which backend database will be used.
837 ``LDAP as a backend database`` is a good example of that.
839 On the client side there are different steps depending on which client backend
840 configuration will be used. For example ``PAM/NSS`` or ``SSSD`` (along with
841 LDAP for identity service, [and Kerberos for authentication service]) which is
842 the best suited option for joining ``MS Active Directory domains``, and doing
843 ``User Logon Management``.
845 By no means we intend to cover every possible scenario/combination here. These
846 steps are for a simple *get a (MIT) Kerberos Server up and running*.
848 Please, note that *rpm packages might have slightly different names*, as well
849 as the locations for the binaries and/or configuration files, depending on
850 which Linux distro we are referring to.
852 Finally, keep in mind that some Linux distros will have their own ``wizards``,
853 which can perform the basic needed configuration: ::
860 pam/nss: yast2 ldapkrb
861 sssd: yast2 auth-client
864 However, we are going through the ``manual configuration``.
867 In order to get a new MIT KDC Server running:
869 1. Install the KDC server by:
871 a. Install the needed packages: ::
873 SUSE: zypper install krb5 krb5-server krb5-client
875 for development: krb5-devel
876 if using 'sssd': sssd-krb5 sssd-krb5-common
878 REDHAT: yum install krb5-server krb5-libs krb5-workstation
879 Additionally: 'Needs to be checked'
882 b. Edit the KDC Server configuration file: ::
884 /var/lib/kerberos/krb5kdc/kdc.conf
889 acl_file = /var/lib/kerberos/krb5kdc/kadm5.acl
890 admin_keytab = FILE:/var/lib/kerberos/krb5kdc/kadm5.keytab
891 default_principal_flags = +postdateable +forwardable +renewable +proxiable
892 +dup-skey -preauth -hwauth +service
893 +tgt-based +allow-tickets -pwchange
895 dict_file = /var/lib/kerberos/krb5kdc/kadm5.dict
896 key_stash_file = /var/lib/kerberos/krb5kdc/.k5.MYDOMAIN.COM
898 max_life = 0d 10h 0m 0s
899 max_renewable_life = 7d 0h 0m 0s
904 c. Edit the Kerberos Client configuration file: ::
908 dns_canonicalize_hostname = false
911 dns_lookup_realm = true //--> if using DNS/DNSMasq
912 dns_lookup_kdc = true //--> if using DNS/DNSMasq
913 allow_weak_crypto = false
914 default_realm = MYDOMAIN.COM
915 default_ccache_name = KEYRING:persistent:%{uid}
919 kdc = kerberos.mydomain.com
920 admin_server = kerberos.mydomain.com
926 2. Create the Kerberos database: ::
928 SUSE: kdb5_util create -s
930 REDHAT: kdb5_util create -s
933 3. Enable and Start both 'KDC and KDC admin' servers: ::
935 SUSE: systemctl enable/start krb5kdc
936 systemctl enable/start kadmind
938 REDHAT: systemctl enable/start krb5kdc
939 systemctl enable/start kadmin
942 4. Create a Kerberos Administrator
943 Kerberos principals can be created either locally on the KDC server itself
944 or through the network, using an 'admin principal'. On the KDC server,
945 using ``kadmin.local``:
947 a. List the existing principals: ::
949 kadmin.local: listprincs
951 krbtgt/MYDOMAIN.COM@MYDOMAIN.COM
952 kadmin/admin@MYDOMAIN.COM
953 kadmin/changepw@MYDOMAIN.COM
954 kadmin/history@MYDOMAIN.COM
955 kadmin/kerberos.mydomain.com@MYDOMAIN.COM
956 root/admin@MYDOMAIN.COM
960 b. In case we don't have a built-in 'admin principal', we then create one
961 (whatever ``principal name``, we are using ``root``, once by default
962 ``kinit`` tries to authenticate using the same system login user name,
963 unless a ``principal`` is passed as an argument ``kinit principal``): ::
965 # kadmin.local -q "addprinc root/admin"
966 Authenticating as principal root/admin@MYDOMAIN.COM with password.
967 WARNING: no policy specified for root/admin@MYDOMAIN.COM; defaulting to no policy
968 Enter password for principal "root/admin@MYDOMAIN.COM":
971 c. Confirm the newly created 'admin principal' has the needed permissions
972 in the KDC ACL (if ACLs are changed, ``kadmind`` needs to be restarted): ::
974 SUSE: /var/lib/kerberos/krb5kdc/kadm5.acl
975 REDHAT: /var/kerberos/krb5kdc/kadm5.acl
977 ###############################################################################
978 #Kerberos_principal permissions [target_principal] [restrictions]
979 ###############################################################################
981 */admin@MYDOMAIN.COM *
984 d. Create a simple 'user principal' (same steps as by *The 'Ceph side' of
987 kadmin.local: addprinc johndoe
988 WARNING: no policy specified for johndoe@MYDOMAIN.COM; defaulting to no policy
989 Enter password for principal "johndoe@MYDOMAIN.COM":
990 Re-enter password for principal "johndoe@MYDOMAIN.COM":
991 Principal "johndoe@MYDOMAIN.COM" created.
994 e. Confirm the newly created 'user principal' is able to authenticate (same
995 steps as by *The 'Ceph side' of the things*; 6): ::
997 # kdestroy -A && kinit -f johndoe && klist -f
998 Password for johndoe@MYDOMAIN.COM:
999 Ticket cache: KEYRING:persistent:0:0
1000 Default principal: johndoe@MYDOMAIN.COM
1002 Valid starting Expires Service principal
1003 11/16/2018 13:11:16 11/16/2018 23:11:16 krbtgt/MYDOMAIN.COM@MYDOMAIN.COM
1004 renew until 11/17/2018 13:11:16, Flags: FRI
1008 5. At this point, we should have a *simple (MIT) Kerberos Server up and running*:
1010 a. Considering we will want to work with keytab files, for both 'user and
1011 service' principals, refer to The *'Ceph side' of the things* starting
1014 b. Make sure you are comfortable with following and their ``manpages``: ::
1016 krb5.conf -> Krb client config file
1017 kdc.conf -> KDC server config file
1019 krb5kdc -> KDC server daemon
1020 kadmind -> KDC administration daemon
1022 kadmin -> Krb administration tool
1023 kdb5_util -> Krb low-level database administration tool
1025 kinit -> Obtain and cache Kerberos ticket-granting ticket tool
1026 klist -> List cached Kerberos tickets tool
1027 kdestroy -> Destroy Kerberos tickets tool
1031 As mentioned earlier, Kerberos *relies heavly on name resolution*. Most of
1032 the Kerberos issues are usually related to name resolution, since Kerberos
1033 is *very picky* on both *systems names* and *host lookups*.
1035 a. As described in *The 'Ceph side' of the things*; step 2a, DNS RRs
1036 greatly improves service location and host/domain resolution, by using
1037 ``(srv resources)`` and ``(txt record)`` respectively (as per
1038 *Before We Start*; *DNS resolution*). ::
1040 /var/lib/named/master/mydomain.com
1041 kerberos IN A 192.168.10.21
1042 kerberos-slave IN A 192.168.10.22
1043 _kerberos IN TXT "MYDOMAIN.COM"
1044 _kerberos._udp IN SRV 1 0 88 kerberos
1045 _kerberos._tcp IN SRV 1 0 88 kerberos
1046 _kerberos._udp IN SRV 20 0 88 kerberos-slave
1047 _kerberos-master._udp IN SRV 0 0 88 kerberos
1048 _kerberos-adm._tcp IN SRV 0 0 749 kerberos
1049 _kpasswd._udp IN SRV 0 0 464 kerberos
1053 b. For a small network or development environment, where a *DNS server is
1054 not available*, we have the option to use ``DNSMasq``, an
1055 ease-to-configure lightweight DNS server (along with some other
1058 These records can be added to ``/etc/dnsmasq.conf`` (in addition to the
1059 needed 'host records'): ::
1063 txt-record=_kerberos.mydomain.com,"MYDOMAIN.COM"
1064 srv-host=_kerberos._udp.mydomain.com,kerberos.mydomain.com,88,1
1065 srv-host=_kerberos._udp.mydomain.com,kerberos-2.mydomain.com,88,20
1066 srv-host=_kerberos-master._udp.mydomain.com,kerberos.mydomain.com,88,0
1067 srv-host=_kerberos-adm._tcp.mydomain.com,kerberos.mydomain.com,749,0
1068 srv-host=_kpasswd._udp.mydomain.com,kerberos.mydomain.com,464,0
1069 srv-host=_kerberos._tcp.mydomain.com,kerberos.mydomain.com,88,1
1073 c. After 'b)' is all set, and ``dnsmasq`` service up and running, we can
1077 Server: 192.168.10.1
1078 Address: 192.168.10.1#53
1080 Name: kerberos.mydomain.com
1081 Address: 192.168.10.21
1083 # host -t SRV _kerberos._tcp.mydomain.com
1084 _kerberos._tcp.mydomain.com has SRV record 1 0 88 kerberos.mydomain.com.
1086 # host -t SRV {each srv-host record}
1087 # host -t TXT _kerberos.mydomain.com
1088 _kerberos.mydomain.com descriptive text "MYDOMAIN.COM"
1092 f. As long as ``name resolution`` is working properly, either ``dnsmasq``
1093 or ``named``, Kerberos should be able to find the needed service