[Devel] [PATCH hci-8.1 2/4] sched: keep cfs_rq::nr_iowait balanced for throttled and class changing tasks

[Vasileios Almpanis]

Two more paths break the cfs_rq::nr_iowait pairing:

 - A throttled task can go to sleep in iowait before it returns to
   user space, e.g. in blkcg_maybe_throttle_current().
   dequeue_throttled_task() doesn't increment the counter but the
   wakeup decrements it, so it wraps around.
 - A sleeping iowait task can change its class through
   sched_setscheduler() or rt_mutex_setprio(). Fair to RT leaves the
   counter incremented forever, RT to fair decrements it without an
   increment.

A CT with both CPU and IO limits hits the first one.

Fix it by incrementing the counter in dequeue_throttled_task() and by
moving the task between the class counters in check_class_changed(),
like sched_move_task() does for a cgroup change.

https://virtuozzo.atlassian.net/browse/VSTOR-147823
Fixes: 415fe063c53e ("sched: Account cfs_rq::nr_iowait")
Signed-off-by: Vasileios Almpanis <vasileios.almpanis at virtuozzo.com>

Feature: sched: enhance task states granularity and accounting
---
 kernel/sched/core.c | 8 ++++++++
 kernel/sched/fair.c | 2 ++
 2 files changed, 10 insertions(+)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index dc016d264d19..1acb98c0e8fa 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -2271,6 +2271,14 @@ void check_class_changed(struct rq *rq, struct task_struct *p,
 			 int oldprio)
 {
 	if (prev_class != p->sched_class) {
+		/* Move a sleeping iowait task to the new class accounting */
+		if (task_in_nr_iowait(p)) {
+			if (prev_class->nr_iowait_dec)
+				prev_class->nr_iowait_dec(p);
+			if (p->sched_class->nr_iowait_inc)
+				p->sched_class->nr_iowait_inc(p);
+		}
+
 		if (prev_class->switched_from)
 			prev_class->switched_from(rq, p);
 
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 5dd350a0e08b..df296edb828d 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -5834,6 +5834,8 @@ static void dequeue_throttled_task(struct task_struct *p, int flags)
 	/* task blocked after throttled */
 	if (flags & DEQUEUE_SLEEP) {
 		p->throttled = false;
+		if (p->in_iowait)
+			task_cfs_rq(p)->nr_iowait++;
 		return;
 	}
 

-- 
2.43.0