Disable aggressive arc_p growth by default

For specific workloads consisting mainly of mfu data and new anon data
buffers, the aggressive growth of arc_p found in the arc_get_data_buf()
function can have detrimental effects on the mfu list size and ghost
list hit rate.

Running a workload consisting of two processes:

    * Process 1 is creating many small files
    * Process 2 is tar'ing a directory consisting of many small files

I've seen arc_p and the mru grow to their maximum size, while the mru
ghost list receives 100K times fewer hits than the mfu ghost list.

Ideally, as the mfu ghost list receives hits, arc_p should be driven
down and the size of the mfu should increase. Given the specific
workload I was testing with, the mfu list size should grow to a point
where almost no mfu ghost list hits would occur. Unfortunately, this
does not happen because the newly dirtied anon buffers constancy drive
arc_p to its maximum value and keep it there (effectively prioritizing
the mru list and starving the mfu list down to a negligible size).

The logic to increment arc_p from within the arc_get_data_buf() function
was introduced many years ago in this upstream commit:

    commit 641fbdae3a027d12b3c3dcd18927ccafae6d58bc
    Author: maybee <none@none>
    Date:   Wed Dec 20 15:46:12 2006 -0800

        6505658 target MRU size (arc.p) needs to be adjusted more aggressively

and since I don't fully understand the motivation for the change, I am
reluctant to completely remove it.

As a way to test out how it's removal might affect performance, I've
disabled that code by default, but left it tunable via a module option.
Thus, if its removal is found to be grossly detrimental for certain
workloads, it can be re-enabled on the fly, without a code change.

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue #2110

diff --git a/man/man5/zfs-module-parameters.5 b/man/man5/zfs-module-parameters.5
index d33da62f42cbcba6c460ec640fa1ec7b810f6c0b..7f010c6cfc5bb246e64ac288b09651d8dd2f06ee 100644 (file)
--- a/man/man5/zfs-module-parameters.5
+++ b/man/man5/zfs-module-parameters.5
@@ -304,6 +304,17 @@ arc_c shift to calc min/max arc_p
 Default value: \fB4\fR.
 .RE
 
+.sp
+.ne 2
+.na
+\fBzfs_arc_p_aggressive_disable\fR (int)
+.ad
+.RS 12n
+Disable aggressive arc_p growth
+.sp
+Use \fB1\fR for yes (default) and \fB0\fR to disable.
+.RE
+
 .sp
 .ne 2
 .na

diff --git a/module/zfs/arc.c b/module/zfs/arc.c
index cbc9b9feeb9738910d2b919fa3cf59230b290dce..eac6ea4488fc5569ba6dee5fa61268ce73da9095 100644 (file)
--- a/module/zfs/arc.c
+++ b/module/zfs/arc.c
@@ -175,6 +175,9 @@ int zfs_arc_grow_retry = 5;
 /* shift of arc_c for calculating both min and max arc_p */
 int zfs_arc_p_min_shift = 4;
 
+/* disable anon data aggressively growing arc_p */
+int zfs_arc_p_aggressive_disable = 1;
+
 /* log2(fraction of arc to reclaim) */
 int zfs_arc_shrink_shift = 5;
 
@@ -2798,7 +2801,8 @@ out:
                 * If we are growing the cache, and we are adding anonymous
                 * data, and we have outgrown arc_p, update arc_p
                 */
-               if (arc_size < arc_c && hdr->b_state == arc_anon &&
+               if (!zfs_arc_p_aggressive_disable &&
+                   arc_size < arc_c && hdr->b_state == arc_anon &&
                    arc_anon->arcs_size + arc_mru->arcs_size > arc_p)
                        arc_p = MIN(arc_c, arc_p + size);
        }
@@ -5553,6 +5557,9 @@ MODULE_PARM_DESC(zfs_arc_meta_prune, "Bytes of meta data to prune");
 module_param(zfs_arc_grow_retry, int, 0644);
 MODULE_PARM_DESC(zfs_arc_grow_retry, "Seconds before growing arc size");
 
+module_param(zfs_arc_p_aggressive_disable, int, 0644);
+MODULE_PARM_DESC(zfs_arc_p_aggressive_disable, "disable aggressive arc_p grow");
+
 module_param(zfs_arc_shrink_shift, int, 0644);
 MODULE_PARM_DESC(zfs_arc_shrink_shift, "log2(fraction of arc to reclaim)");