[Devel] [PATCH RHEL7 COMMIT] mm: only drain per-cpu pagevecs once per pagevec usage

[Vasily Averin]

The commit is pushed to "branch-rh7-3.10.0-1127.10.1.vz7.162.x-ovz" and will appear at https://src.openvz.org/scm/ovz/vzkernel.git
after rh7-3.10.0-1127.10.1.vz7.162.13
------>
commit 488038d0ae3a76c3f4d5f937306da28f8d93162e
Author: Mel Gorman <mgorman at techsingularity.net>
Date:   Tue Jul 21 17:58:21 2020 +0300

    mm: only drain per-cpu pagevecs once per pagevec usage
    
    When a pagevec is initialised on the stack, it is generally used
    multiple times over a range of pages, looking up entries and then
    releasing them.  On each pagevec_release, the per-cpu deferred LRU
    pagevecs are drained on the grounds the page being released may be on
    those queues and the pages may be cache hot.  In many cases only the
    first drain is necessary as it's unlikely that the range of pages being
    walked is racing against LRU addition.  Even if there is such a race,
    the impact is marginal where as constantly redraining the lru pagevecs
    costs.
    
    This patch ensures that pagevec is only drained once in a given
    lifecycle without increasing the cache footprint of the pagevec
    structure.  Only sparsetruncate tiny is shown here as large files have
    many exceptional entries and calls pagecache_release less frequently.
    
    sparsetruncate (tiny)
                                  4.14.0-rc4             4.14.0-rc4
                            batchshadow-v1r1          onedrain-v1r1
    Min          Time      141.00 (   0.00%)      141.00 (   0.00%)
    1st-qrtle    Time      142.00 (   0.00%)      142.00 (   0.00%)
    2nd-qrtle    Time      142.00 (   0.00%)      142.00 (   0.00%)
    3rd-qrtle    Time      143.00 (   0.00%)      143.00 (   0.00%)
    Max-90%      Time      144.00 (   0.00%)      144.00 (   0.00%)
    Max-95%      Time      146.00 (   0.00%)      145.00 (   0.68%)
    Max-99%      Time      198.00 (   0.00%)      194.00 (   2.02%)
    Max          Time      254.00 (   0.00%)      208.00 (  18.11%)
    Amean        Time      145.12 (   0.00%)      144.30 (   0.56%)
    Stddev       Time       12.74 (   0.00%)        9.62 (  24.49%)
    Coeff        Time        8.78 (   0.00%)        6.67 (  24.06%)
    Best99%Amean Time      144.29 (   0.00%)      143.82 (   0.32%)
    Best95%Amean Time      142.68 (   0.00%)      142.31 (   0.26%)
    Best90%Amean Time      142.52 (   0.00%)      142.19 (   0.24%)
    Best75%Amean Time      142.26 (   0.00%)      141.98 (   0.20%)
    Best50%Amean Time      141.90 (   0.00%)      141.71 (   0.13%)
    Best25%Amean Time      141.80 (   0.00%)      141.43 (   0.26%)
    
    The impact on bonnie is marginal and within the noise because a
    significant percentage of the file being truncated has been reclaimed
    and consists of shadow entries which reduce the hotness of the
    pagevec_release path.
    
    Link: http://lkml.kernel.org/r/20171018075952.10627-5-mgorman@techsingularity.net
    Signed-off-by: Mel Gorman <mgorman at techsingularity.net>
    Cc: Andi Kleen <ak at linux.intel.com>
    Cc: Dave Chinner <david at fromorbit.com>
    Cc: Dave Hansen <dave.hansen at intel.com>
    Cc: Jan Kara <jack at suse.cz>
    Cc: Johannes Weiner <hannes at cmpxchg.org>
    Cc: Vlastimil Babka <vbabka at suse.cz>
    Signed-off-by: Andrew Morton <akpm at linux-foundation.org>
    Signed-off-by: Linus Torvalds <torvalds at linux-foundation.org>
    
    https://jira.sw.ru/browse/PSBM-98148
    (cherry picked from commit d9ed0d08b6c6a882da1d8e75bb3162fc889fd199)
    Signed-off-by: Andrey Ryabinin <aryabinin at virtuozzo.com>
---
 include/linux/pagevec.h | 4 +++-
 mm/swap.c               | 5 ++++-
 2 files changed, 7 insertions(+), 2 deletions(-)

diff --git a/include/linux/pagevec.h b/include/linux/pagevec.h
index 3c6b8b1e945b9..e64e6f35c8e7a 100644
--- a/include/linux/pagevec.h
+++ b/include/linux/pagevec.h
@@ -16,7 +16,8 @@ struct address_space;
 
 struct pagevec {
 	unsigned long nr;
-	unsigned long cold;
+	bool cold;
+	bool drained;
 	struct page *pages[PAGEVEC_SIZE];
 };
 
@@ -35,6 +36,7 @@ static inline void pagevec_init(struct pagevec *pvec, int cold)
 {
 	pvec->nr = 0;
 	pvec->cold = cold;
+	pvec->drained = false;
 }
 
 static inline void pagevec_reinit(struct pagevec *pvec)
diff --git a/mm/swap.c b/mm/swap.c
index 762038da1d0cf..8ea93ce3f8577 100644
--- a/mm/swap.c
+++ b/mm/swap.c
@@ -1131,7 +1131,10 @@ EXPORT_SYMBOL(release_pages);
  */
 void __pagevec_release(struct pagevec *pvec)
 {
-	lru_add_drain();
+	if (!pvec->drained) {
+		lru_add_drain();
+		pvec->drained = true;
+	}
 	release_pages(pvec->pages, pagevec_count(pvec), pvec->cold);
 	pagevec_reinit(pvec);
 }