[Devel] [PATCH RHEL7 COMMIT] ms/x86/ticketlock: Fix spin_unlock_wait() livelock

[Konstantin Khorenko]

The commit is pushed to "branch-rh7-3.10.0-514.26.1.vz7.33.x-ovz" and will appear at https://src.openvz.org/scm/ovz/vzkernel.git
after rh7-3.10.0-514.26.1.vz7.33.5
------>
commit 39f6e91881c62646b243a53a7173c152d2c460d0
Author: Oleg Nesterov <oleg at redhat.com>
Date:   Thu Jul 13 19:30:24 2017 +0400

    ms/x86/ticketlock: Fix spin_unlock_wait() livelock
    
    arch_spin_unlock_wait() looks very suboptimal, to the point I
    think this is just wrong and can lead to livelock: if the lock
    is heavily contended we can never see head == tail.
    
    But we do not need to wait for arch_spin_is_locked() == F. If it
    is locked we only need to wait until the current owner drops
    this lock. So we could simply spin until old_head !=
    lock->tickets.head in this case, but .head can overflow and thus
    we can't check "unlocked" only once before the main loop.
    
    Also, the "unlocked" check can ignore TICKET_SLOWPATH_FLAG bit.
    
    Signed-off-by: Oleg Nesterov <oleg at redhat.com>
    Acked-by: Linus Torvalds <torvalds at linux-foundation.org>
    Cc: Jeremy Fitzhardinge <jeremy at goop.org>
    Cc: Paul E.McKenney <paulmck at linux.vnet.ibm.com>
    Cc: Peter Zijlstra <peterz at infradead.org>
    Cc: Waiman Long <Waiman.Long at hp.com>
    Link: http://lkml.kernel.org/r/20141201213417.GA5842@redhat.com
    Signed-off-by: Ingo Molnar <mingo at kernel.org>
    
    https://jira.sw.ru/browse/PSBM-68212
    
    (cherry picked from commit 78bff1c8684fb94f1ae7283688f90188b53fc433)
    Signed-off-by: Denis Plotnikov <dplotnikov at virtuozzo.com>
    Acked-by: Andrey Ryabinin <aryabinin at virtuozzo.com>
---
 arch/x86/include/asm/spinlock.h | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/arch/x86/include/asm/spinlock.h b/arch/x86/include/asm/spinlock.h
index 4dc8606..63140dc 100644
--- a/arch/x86/include/asm/spinlock.h
+++ b/arch/x86/include/asm/spinlock.h
@@ -183,8 +183,20 @@ static __always_inline void arch_spin_lock_flags(arch_spinlock_t *lock,
 
 static inline void arch_spin_unlock_wait(arch_spinlock_t *lock)
 {
-	while (arch_spin_is_locked(lock))
+	__ticket_t head = ACCESS_ONCE(lock->tickets.head);
+
+	for (;;) {
+		struct __raw_tickets tmp = ACCESS_ONCE(lock->tickets);
+		/*
+		 * We need to check "unlocked" in a loop, tmp.head == head
+		 * can be false positive because of overflow.
+		 */
+		if (tmp.head == (tmp.tail & ~TICKET_SLOWPATH_FLAG) ||
+		    tmp.head != head)
+			break;
+
 		cpu_relax();
+	}
 }
 
 /*