[ceph.git] / ceph / doc / jaegertracing / index.rst

JAEGER- DISTRIBUTED TRACING
===========================

Jaeger provides ready-to-use tracing services for distributed systems. 

BASIC ARCHITECTURE AND TERMINOLOGY
----------------------------------

* TRACE: A trace shows the data/execution path through a system.
* SPAN: A single unit of a trace. A data structure that stores information such
  as the operation name, timestamps, and the ordering within a trace.
* JAEGER CLIENT: Language-specific implementations of the OpenTracing API.
* JAEGER AGENT: A daemon that listens for spans sent over User Datagram
  Protocol. The agent is meant to be placed on the same host as the
  instrumented application. (The Jaeger agent acts like a sidecar listener.)
* JAEGER COLLECTOR: A daemon that receives spans sent by the Jaeger agent. The
  Jaeger collector then stitches the spans together to form a trace. (A databse
  can be enabled to persist a database for these traces).
* JAEGER QUERY AND CONSOLE FRONTEND: The UI-based frontend that presents
  reports of the jaeger traces. Accessible at  http://<jaeger frontend host>:16686.

Read more about jaeger tracing:.

  https://www.jaegertracing.io/docs/

JAEGER DEPLOYMENT
-----------------

Jaeger can be deployed using cephadm, and Jaeger can be deployed manually.

Refer to one of the following:

`Cephadm Jaeger Services Deployment <../cephadm/services/tracing/>`_

`Jaeger Deployment <https://www.jaegertracing.io/docs/1.25/deployment/>`_

`Jaeger Performance Tuning <https://www.jaegertracing.io/docs/1.25/performance-tuning/>`_


Important Notes:
^^^^^^^^^^^^^^^^

- The Jaeger agent must be running on each host (and not running in all-in-one
  mode). This is because spans are sent to the local Jaeger agent. Spans of
  hosts that do not have active Jaeger agents will be lost.

- Ceph tracers are configured to send tracers to agents that listen to port
  6799. Use the option "--processor.jaeger-compact.server-host-port=6799" for
  manual Jaeger deployments. 


HOW TO ENABLE TRACING IN CEPH
-----------------------------

Tracing in Ceph is disabled by default.

Tracing can be enabled globally, and tracing can also be enabled separately for
each entity (for example, for rgw).

Enable tracing globally:

.. prompt:: bash $

   ceph config set global jaeger_tracing_enable true


Enable tracing for each entity:

.. prompt:: bash $

   ceph config set <entity> jaeger_tracing_enable true


TRACES IN RGW
-------------

Traces run on RGW can be found under the Service `rgw` in the Jaeger Frontend.

REQUESTS
^^^^^^^^
Every user request is traced. Each trace contains tags for `Operation name`,
`User id`, `Object name` and `Bucket name`.

There is also an `Upload id` tag for Multipart upload operations.

The names of request traces have the following format: `<command> <transaction
id>`.

MULTIPART UPLOAD
^^^^^^^^^^^^^^^^
There is a kind of trace that consists of a span for each request made by a
multipart upload, and it includes all `Put Object` requests.

The names of multipart traces have the following format: `multipart_upload
<upload id>`.


rgw service in Jaeger Frontend:

.. image:: ./rgw_jaeger.png
  :width: 400


osd service in Jaeger Frontend:

.. image:: ./osd_jaeger.png
  :width: 400
Commit	Line	Data
20effc67 TL	1	JAEGER- DISTRIBUTED TRACING
	2	===========================
	3
1e59de90	4	Jaeger provides ready-to-use tracing services for distributed systems.
20effc67 TL	5
	6	BASIC ARCHITECTURE AND TERMINOLOGY
	7	----------------------------------
	8
	9	* TRACE: A trace shows the data/execution path through a system.
1e59de90 TL	10	* SPAN: A single unit of a trace. A data structure that stores information such
	11	as the operation name, timestamps, and the ordering within a trace.
	12	* JAEGER CLIENT: Language-specific implementations of the OpenTracing API.
	13	* JAEGER AGENT: A daemon that listens for spans sent over User Datagram
	14	Protocol. The agent is meant to be placed on the same host as the
	15	instrumented application. (The Jaeger agent acts like a sidecar listener.)
	16	* JAEGER COLLECTOR: A daemon that receives spans sent by the Jaeger agent. The
	17	Jaeger collector then stitches the spans together to form a trace. (A databse
	18	can be enabled to persist a database for these traces).
	19	* JAEGER QUERY AND CONSOLE FRONTEND: The UI-based frontend that presents
	20	reports of the jaeger traces. Accessible at http://<jaeger frontend host>:16686.
	21
	22	Read more about jaeger tracing:.
20effc67 TL	23
	24	https://www.jaegertracing.io/docs/
	25
20effc67 TL	26	JAEGER DEPLOYMENT
	27	-----------------
	28
1e59de90	29	Jaeger can be deployed using cephadm, and Jaeger can be deployed manually.
20effc67	30
1e59de90	31	Refer to one of the following:
20effc67	32
1e59de90	33	`Cephadm Jaeger Services Deployment <../cephadm/services/tracing/>`_
20effc67	34
1e59de90 TL	35	`Jaeger Deployment <https://www.jaegertracing.io/docs/1.25/deployment/>`_
	36
	37	`Jaeger Performance Tuning <https://www.jaegertracing.io/docs/1.25/performance-tuning/>`_
	38
	39
	40	Important Notes:
	41	^^^^^^^^^^^^^^^^
	42
	43	- The Jaeger agent must be running on each host (and not running in all-in-one
	44	mode). This is because spans are sent to the local Jaeger agent. Spans of
	45	hosts that do not have active Jaeger agents will be lost.
	46
	47	- Ceph tracers are configured to send tracers to agents that listen to port
	48	6799. Use the option "--processor.jaeger-compact.server-host-port=6799" for
	49	manual Jaeger deployments.
20effc67	50
20effc67 TL	51
	52	HOW TO ENABLE TRACING IN CEPH
	53	-----------------------------
	54
1e59de90 TL	55	Tracing in Ceph is disabled by default.
	56
	57	Tracing can be enabled globally, and tracing can also be enabled separately for
	58	each entity (for example, for rgw).
	59
	60	Enable tracing globally:
	61
	62	.. prompt:: bash $
20effc67	63
1e59de90	64	ceph config set global jaeger_tracing_enable true
20effc67	65
20effc67	66
1e59de90	67	Enable tracing for each entity:
20effc67	68
1e59de90	69	.. prompt:: bash $
20effc67	70
1e59de90	71	ceph config set <entity> jaeger_tracing_enable true
20effc67 TL	72
	73
	74	TRACES IN RGW
	75	-------------
	76
1e59de90	77	Traces run on RGW can be found under the Service `rgw` in the Jaeger Frontend.
20effc67 TL	78
	79	REQUESTS
	80	^^^^^^^^
1e59de90 TL	81	Every user request is traced. Each trace contains tags for `Operation name`,
1e59de90 TL	82	`User id`, `Object name` and `Bucket name`.
20effc67	83
1e59de90	84	There is also an `Upload id` tag for Multipart upload operations.
20effc67	85
1e59de90 TL	86	The names of request traces have the following format: `<command> <transaction
1e59de90 TL	87	id>`.
20effc67 TL	88
	89	MULTIPART UPLOAD
	90	^^^^^^^^^^^^^^^^
1e59de90 TL	91	There is a kind of trace that consists of a span for each request made by a
1e59de90 TL	92	multipart upload, and it includes all `Put Object` requests.
20effc67	93
1e59de90 TL	94	The names of multipart traces have the following format: `multipart_upload
1e59de90 TL	95	<upload id>`.
20effc67 TL	96
	97
	98	rgw service in Jaeger Frontend:
	99
	100	.. image:: ./rgw_jaeger.png
	101	:width: 400
	102
	103
	104	osd service in Jaeger Frontend:
	105
	106	.. image:: ./osd_jaeger.png
	107	:width: 400