[Devel] [PATCH hci-8.1] ms/mm/shrinker: allocate shrinker_info_unit from a dedicated kmem_cache

[Konstantin Khorenko]

shrinker_info_unit holds BITS_PER_LONG atomic_long_t of deferred work plus
a one-word bitmap, 520 bytes on 64-bit. kzalloc_node() takes it from
kmalloc-1k, so almost half of every unit is padding.

Allocate the units from a cache of their own, sized to the unit. The
cache is not cache-line aligned: the counters of a unit already share
cache lines between shrinkers, and aligning the unit would only keep
neighbouring units off each other's boundary line, at the cost of part of
the saving.

Link: https://lore.kernel.org/20261019-vstor-144715-shrinker_info_unit-kmem_cache-v3-1-0fb061e09bf4@virtuozzo.com
Signed-off-by: Konstantin Khorenko <khorenko at virtuozzo.com>
Signed-off-by: Andrew Morton <akpm at linux-foundation.org>
Suggested-by: Andrew Morton <akpm at linux-foundation.org>
Reviewed-by: Harry Yoo (Meta) <harry at kernel.org>
Reviewed-by: Qi Zheng <qi.zheng at linux.dev>
Acked-by: Muchun Song <muchun.song at linux.dev>
Assisted-by: LLM
Cc: Dave Chinner <david at fromorbit.com>
Cc: Roman Gushchin <roman.gushchin at linux.dev>
Cc: Johannes Weiner <hannes at cmpxchg.org>
Cc: Michal Hocko <mhocko at kernel.org>
Cc: Shakeel Butt <shakeel.butt at linux.dev>
(cherry picked from commit b36b71cb22027228fcc0f3a64c0f36900142c637)

Adapted: in 6.12 mem_cgroup_init() is a subsys_initcall and runs after
cgroup_init() has brought the root memcg online, which already needs
the cache. Create it where the root memcg is allocated, in
mem_cgroup_css_alloc(), instead: the function is __ref, and its root
branch runs once, from __init cgroup_init_subsys().

https://virtuozzo.atlassian.net/browse/VSTOR-144714

Feature: mm/shrinker: sparse per-memcg shrinker units
Assisted-by: LLM
Signed-off-by: Konstantin Khorenko <khorenko at virtuozzo.com>
---
Port of the mainline patch that moves the per-memcg shrinker_info_unit
from kmalloc-1k, where almost half of every unit is padding, to a cache
sized to the unit. Merged into mm-unstable as b36b71cb22027.

One adaptation: in 6.12 mem_cgroup_init() is a subsys_initcall and runs
after cgroup_init() has brought the root memcg online, so the cache is
created in mem_cgroup_css_alloc() where the root memcg is allocated.

https://virtuozzo.atlassian.net/browse/VSTOR-144714
---
 include/linux/memcontrol.h |  1 +
 mm/memcontrol.c            |  1 +
 mm/shrinker.c              | 11 +++++++++--
 3 files changed, 11 insertions(+), 2 deletions(-)

diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index 2772a7750fa4e..e5b8ac79f6c7c 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -1740,6 +1740,7 @@ static inline bool mem_cgroup_under_socket_pressure(struct mem_cgroup *memcg)
 	return false;
 }
 
+void shrinker_unit_cache_init(void);
 int alloc_shrinker_info(struct mem_cgroup *memcg);
 void free_shrinker_info(struct mem_cgroup *memcg);
 void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id);
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 4e0c0c46689aa..0734c0fa21d05 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -4393,6 +4393,7 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css)
 #endif
 		page_counter_init(&memcg->cache, NULL, false);
 
+		shrinker_unit_cache_init();
 		root_mem_cgroup = memcg;
 		return &memcg->css;
 	}
diff --git a/mm/shrinker.c b/mm/shrinker.c
index a70074f7135a9..5e45b230a5ccf 100644
--- a/mm/shrinker.c
+++ b/mm/shrinker.c
@@ -12,6 +12,12 @@ DEFINE_MUTEX(shrinker_mutex);
 
 #ifdef CONFIG_MEMCG
 static int shrinker_nr_max;
+static struct kmem_cache *shrinker_unit_cachep;
+
+void __init shrinker_unit_cache_init(void)
+{
+	shrinker_unit_cachep = KMEM_CACHE(shrinker_info_unit, SLAB_PANIC);
+}
 
 static inline int shrinker_unit_size(int nr_items)
 {
@@ -33,7 +39,7 @@ static inline void shrinker_unit_free(struct shrinker_info *info, int start)
 		if (!unit[i])
 			break;
 
-		kfree(unit[i]);
+		kmem_cache_free(shrinker_unit_cachep, unit[i]);
 		unit[i] = NULL;
 	}
 }
@@ -47,7 +53,8 @@ static inline int shrinker_unit_alloc(struct shrinker_info *new,
 	int i;
 
 	for (i = start; i < nr; i++) {
-		unit = kzalloc_node(sizeof(*unit), GFP_KERNEL, nid);
+		unit = kmem_cache_alloc_node(shrinker_unit_cachep,
+					     GFP_KERNEL | __GFP_ZERO, nid);
 		if (!unit) {
 			shrinker_unit_free(new, start);
 			return -ENOMEM;

---
base-commit: ed6900fdc06c26a6b1dd37b4b4420157c9af0c95
change-id: 20261020-b4-prep-khorenko-vstor-144714-shrinkers-kmem_cache-d968e1813b70

Best regards,
--  
Konstantin Khorenko <khorenko at virtuozzo.com>