From: Christoph Lameter <cl@linux.com>
Date: Mon, 7 Apr 2014 22:39:42 +0000 (-0700)
Subject: slub: use raw_cpu_inc for incrementing statistics
X-Git-Tag: Ubuntu-5.13.0-19.19~20290^2~14
X-Git-Url: https://git.proxmox.com/?a=commitdiff_plain;h=88da03a67674bcd6e9ecf18a0a182cf1303056ba;p=mirror_ubuntu-jammy-kernel.git

slub: use raw_cpu_inc for incrementing statistics

Statistics are not critical to the operation of the allocation but
should also not cause too much overhead.

When __this_cpu_inc is altered to check if preemption is disabled this
triggers.  Use raw_cpu_inc to avoid the checks.  Using this_cpu_ops may
cause interrupt disable/enable sequences on various arches which may
significantly impact allocator performance.

[akpm@linux-foundation.org: add comment]
Signed-off-by: Christoph Lameter <cl@linux.com>
Cc: Fengguang Wu <fengguang.wu@intel.com>
Cc: Pekka Enberg <penberg@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
---

diff --git a/mm/slub.c b/mm/slub.c
index e7451861f95d..f620bbf4054a 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -224,7 +224,11 @@ static inline void memcg_propagate_slab_attrs(struct kmem_cache *s) { }
 static inline void stat(const struct kmem_cache *s, enum stat_item si)
 {
 #ifdef CONFIG_SLUB_STATS
-	__this_cpu_inc(s->cpu_slab->stat[si]);
+	/*
+	 * The rmw is racy on a preemptible kernel but this is acceptable, so
+	 * avoid this_cpu_add()'s irq-disable overhead.
+	 */
+	raw_cpu_inc(s->cpu_slab->stat[si]);
 #endif
 }