ceph/src/pybind/mgr/k8sevents/README.md

   1 # Testing
   2
   3 ## To test the k8sevents module
   4 enable the module with `ceph mgr module enable k8sevents`
   5 check that it's working `ceph k8sevents status`, you should see something like this;
   6 ```
   7 [root@ceph-mgr ~]# ceph k8sevents status
   8 Kubernetes
   9 - Hostname : https://localhost:30443
  10 - Namespace: ceph
  11 Tracker Health
  12 - EventProcessor : OK
  13 - CephConfigWatcher : OK
  14 - NamespaceWatcher : OK
  15 Tracked Events
  16 - namespace  :   5
  17 - ceph events:   0
  18
  19 ```
  20 Now run some commands to generate healthchecks and admin level events;
  21 - ```ceph osd set noout```
  22 - ```ceph osd unset noout```
  23 - ```ceph osd pool create mypool 4 4 replicated```
  24 - ```ceph osd pool delete mypool mypool --yes-i-really-really-mean-it```
  25
  26 In addition to tracking audit, healthchecks and configuration changes if you have the environment up for >1 hr you should also see and event that shows the clusters health and configuration overview.
  27
  28 As well as status, you can use k8sevents to see event activity in the target kubernetes namespace
  29 ```
  30 [root@rhcs4-3 kube]# ceph k8sevents ls
  31 Last Seen (UTC)       Type      Count  Message                                              Event Object Name
  32 2019/09/20 04:33:00   Normal        1  Pool 'mypool' has been removed from the cluster      mgr.ConfigurationChangeql2hj
  33 2019/09/20 04:32:55   Normal        1  Client 'client.admin' issued: ceph osd pool delete   mgr.audit.osd_pool_delete_
  34 2019/09/20 04:13:23   Normal        2  Client 'mds.rhcs4-2' issued: ceph osd blacklist      mgr.audit.osd_blacklist_
  35 2019/09/20 04:08:28   Normal        1  Ceph log -> event tracking started                   mgr.k8sevents-moduleq74k7
  36 Total :   4
  37 ```
  38 or, focus on the ceph specific events(audit & healthcheck) that are being tracked by the k8sevents module.
  39 ```
  40 [root@rhcs4-3 kube]# ceph k8sevents ceph
  41 Last Seen (UTC)       Type      Count  Message                                              Event Object Name
  42 2019/09/20 04:32:55   Normal        1  Client 'client.admin' issued: ceph osd pool delete   mgr.audit.osd_pool_delete_
  43 2019/09/20 04:13:23   Normal        2  Client 'mds.rhcs4-2' issued: ceph osd blacklist      mgr.audit.osd_blacklist_
  44 Total :   2
  45 ```
  46
  47 ## Sending events from a standalone Ceph cluster to remote Kubernetes cluster
  48 To test interaction from a standalone ceph cluster to a kubernetes environment, you need to make changes on the kubernetes cluster **and** on one of the mgr hosts.
  49 ### kubernetes (minikube)
  50 We need some basic RBAC in place to define a serviceaccount(and token) that we can use to push events into kubernetes. The `rbac_sample.yaml` file provides a quick means to create the required resources. Create them with `kubectl create -f rbac_sample.yaml`
  51
  52 Once the resources are defined inside kubernetes, we need a couple of things copied over to the Ceph mgr's filesystem.
  53 ### ceph admin host
  54 We need to run some commands against the cluster, so you'll needs access to a ceph admin host. If you don't have a dedicated admin host, you can use a mon or mgr machine. We'll need the root ca.crt of the kubernetes API, and the token associated with the service account we're using to access the kubernetes API.
  55
  56 1. Download/fetch the root ca.crt for the kubernetes cluster (on minikube this can be found at ~/minikube/ca.crt)
  57 2. Copy the ca.crt to your ceph admin host
  58 3. Extract the token from the service account we're going to use
  59 ```
  60 kubectl -n ceph get secrets -o jsonpath="{.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='ceph-mgr')].data.token}"|base64 -d > mytoken
  61 ```
  62 4. Copy the token to your ceph admin host
  63 5. On the ceph admin host, enable the module with `ceph mgr module enable k8sevents`
  64 6. Set up the configuration
  65 ```
  66 ceph k8sevents set-access cacrt -i <path to ca.crt file>
  67 ceph k8sevents set-access token -i <path to mytoken>
  68 ceph k8sevents set-config server https://<kubernetes api host>:<api_port>
  69 ceph k8sevents set-config namespace ceph
  70 ```
  71 7. Restart the module with `ceph mgr module disable k8sevents && ceph mgr module enable k8sevents`
  72 8. Check state with the `ceph k8sevents status` command
  73 9. Remove the ca.crt and mytoken files from your admin host
  74
  75 To remove the configuration keys used for external kubernetes access, run the following command
  76 ```
  77 ceph k8sevents clear-config
  78 ```
  79
  80 ## Networking
  81 You can use the above approach with a minikube based target from a standalone ceph cluster, but you'll need to have a tunnel/routing defined from the mgr host(s) to the minikube machine to make the kubernetes API accessible to the mgr/k8sevents module. This can just be a simple ssh tunnel.