From patchwork Thu May 23 20:07:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Andr=C3=A9_Almeida?= X-Patchwork-Id: 90756 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 4728D3845BD3 for ; Thu, 23 May 2024 20:08:42 +0000 (GMT) X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from fanzine2.igalia.com (fanzine.igalia.com [178.60.130.6]) by sourceware.org (Postfix) with ESMTPS id 1088B3858CD1 for ; Thu, 23 May 2024 20:08:13 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 1088B3858CD1 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=igalia.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=igalia.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 1088B3858CD1 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=178.60.130.6 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1716494896; cv=none; b=TwOkDsJFD7i5JnpkI/IZ+CQOXKDfOubg/AH1E1r4juPNvFIIQElLJC00hErH21ZK06SSFEOE0/ozC51OdiySSV56p5022hmXkUNLU/YjwrZYoHONXNnsGemnYzo/L4uqRb3OtGgnJov3L5VNj7jc4N7zluzlMLqVKEno/zk5udI= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1716494896; c=relaxed/simple; bh=Opd2bSlYcIyMBwk1KoGA72+CQf9BDRA2NiY+NSNklQw=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=d+2+Bqq97Qr5DchgsnK3vV+GC7YMvD2Ah6UY2no+8e3ccsoGK7evHZB3w9mZvP2/wwKwSsuZzLGZnmbd6sayHBfUfJjumps+dg4sJnf2cDLAUYQPpJpyKgPKZFhBK6jzD1loZ0t5BUwAOh1bvBxO7F/mt+gNFczUZePz3IIPbV4= ARC-Authentication-Results: i=1; server2.sourceware.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=igalia.com; s=20170329; h=Content-Transfer-Encoding:Content-Type:MIME-Version:References: In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=dGvuPgGsg0VxczO9LN+XhOWYu9n5IUtz1G620Nf/Nuk=; b=PWEQh5uThVxdrbuQH4ohp4U9R8 gnq4XROotBE3zo3BSuC2NAqC1tvDbOXX8B+8HgaoDEKshIodHRP9L/bWiHPhEF7j4d/c4p+9EMaqm Ed27V8n7UYu32NgVaoYbaiyZ/JQuxnJnXzSU5c/2oJNanXl4zZqjkBrVT31WHxJe4rb2v6SVZGgRX 4e8AIEdeySt/Ec28+DTslnjz6yWOYt0L7cY4uuxQHJc8nJ9VBbCknW72msAaYvWPMbKF3XXm1x1A1 j4Xv4yiEO2RdSwdZ+C0/wWxDfAZvzMzRL5Zw2LIH1vrNZcunF+K4/Tu4qJ7kiv6RIB6NF7P3mfNxM j3AQWYfQ==; Received: from [191.8.29.37] (helo=localhost.localdomain) by fanzine2.igalia.com with esmtpsa (Cipher TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim) id 1sAEig-00BhS0-Oi; Thu, 23 May 2024 22:07:23 +0200 From: =?utf-8?q?Andr=C3=A9_Almeida?= To: Mathieu Desnoyers , Peter Zijlstra Cc: linux-kernel@vger.kernel.org, "Thomas Gleixner" , "Paul E . McKenney" , "Boqun Feng" , "H . Peter Anvin" , "Paul Turner" , linux-api@vger.kernel.org, "Christian Brauner" , "Florian Weimer" , David.Laight@ACULAB.COM, carlos@redhat.com, "Peter Oskolkov" , "Alexander Mikhalitsyn" , "Chris Kennelly" , "Ingo Molnar" , "Darren Hart" , "Davidlohr Bueso" , =?utf-8?q?Andr=C3=A9_Almeida?= , libc-alpha@sourceware.org, "Steven Rostedt" , "Jonathan Corbet" , "Noah Goldstein" , "Daniel Colascione" , longman@redhat.com, kernel-dev@igalia.com Subject: [PATCH v2 1/1] futex: Add FUTEX_SPIN operation Date: Thu, 23 May 2024 17:07:04 -0300 Message-ID: <20240523200704.281514-2-andrealmeid@igalia.com> X-Mailer: git-send-email 2.45.1 In-Reply-To: <20240523200704.281514-1-andrealmeid@igalia.com> References: <20240523200704.281514-1-andrealmeid@igalia.com> MIME-Version: 1.0 X-Spam-Status: No, score=-10.4 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces+patchwork=sourceware.org@sourceware.org Add a new mode for futex wait, the futex spin. Given the FUTEX2_SPIN flag, parse the futex value as the TID of the lock owner. Then, before going to the normal wait path, spins while the lock owner is running in a different CPU, to avoid the whole context switch operation and to quickly return to userspace. If the lock owner is not running, just sleep as the normal futex wait path. The user value is masked with FUTEX_TID_MASK, to allow some bits for future use. The check for the owner to be running or not is important to avoid spinning for something that won't be released quickly. Userspace is responsible on providing the proper TID, the kernel does a basic check. Signed-off-by: André Almeida --- include/uapi/linux/futex.h | 2 +- kernel/futex/futex.h | 6 ++- kernel/futex/waitwake.c | 78 +++++++++++++++++++++++++++++++++++++- 3 files changed, 82 insertions(+), 4 deletions(-) diff --git a/include/uapi/linux/futex.h b/include/uapi/linux/futex.h index d2ee625ea189..d77d692ffac2 100644 --- a/include/uapi/linux/futex.h +++ b/include/uapi/linux/futex.h @@ -63,7 +63,7 @@ #define FUTEX2_SIZE_U32 0x02 #define FUTEX2_SIZE_U64 0x03 #define FUTEX2_NUMA 0x04 - /* 0x08 */ +#define FUTEX2_SPIN 0x08 /* 0x10 */ /* 0x20 */ /* 0x40 */ diff --git a/kernel/futex/futex.h b/kernel/futex/futex.h index 8b195d06f4e8..180c1c10dc81 100644 --- a/kernel/futex/futex.h +++ b/kernel/futex/futex.h @@ -37,6 +37,7 @@ #define FLAGS_HAS_TIMEOUT 0x0040 #define FLAGS_NUMA 0x0080 #define FLAGS_STRICT 0x0100 +#define FLAGS_SPIN 0x0200 /* FUTEX_ to FLAGS_ */ static inline unsigned int futex_to_flags(unsigned int op) @@ -52,7 +53,7 @@ static inline unsigned int futex_to_flags(unsigned int op) return flags; } -#define FUTEX2_VALID_MASK (FUTEX2_SIZE_MASK | FUTEX2_PRIVATE) +#define FUTEX2_VALID_MASK (FUTEX2_SIZE_MASK | FUTEX2_PRIVATE | FUTEX2_SPIN) /* FUTEX2_ to FLAGS_ */ static inline unsigned int futex2_to_flags(unsigned int flags2) @@ -65,6 +66,9 @@ static inline unsigned int futex2_to_flags(unsigned int flags2) if (flags2 & FUTEX2_NUMA) flags |= FLAGS_NUMA; + if (flags2 & FUTEX2_SPIN) + flags |= FLAGS_SPIN; + return flags; } diff --git a/kernel/futex/waitwake.c b/kernel/futex/waitwake.c index 3a10375d9521..276c03804b92 100644 --- a/kernel/futex/waitwake.c +++ b/kernel/futex/waitwake.c @@ -372,6 +372,73 @@ void futex_wait_queue(struct futex_hash_bucket *hb, struct futex_q *q, __set_current_state(TASK_RUNNING); } +static inline bool task_on_cpu(struct task_struct *p) +{ +#ifdef CONFIG_SMP + return !!(p->on_cpu); +#else + return false; +#endif +} + +static int futex_spin(struct futex_hash_bucket *hb, struct futex_q *q, + struct hrtimer_sleeper *timeout, u32 uval) +{ + struct task_struct *p; + pid_t pid = uval & FUTEX_TID_MASK; + + p = find_get_task_by_vpid(pid); + + /* no task found, maybe it already exited */ + if (!p) { + futex_q_unlock(hb); + return -EAGAIN; + } + + /* can't spin in a kernel task */ + if (unlikely(p->flags & PF_KTHREAD)) { + put_task_struct(p); + futex_q_unlock(hb); + return -EPERM; + } + + futex_queue(q, hb); + + if (timeout) + hrtimer_sleeper_start_expires(timeout, HRTIMER_MODE_ABS); + + while (1) { + if (likely(!plist_node_empty(&q->list))) { + if (timeout && !timeout->task) + goto exit; + + if (task_on_cpu(p)) { + /* spin */ + continue; + } else { + /* task is not running, sleep */ + break; + } + } else { + goto exit; + } + } + + /* spinning didn't work, go to the normal path */ + set_current_state(TASK_INTERRUPTIBLE|TASK_FREEZABLE); + + if (likely(!plist_node_empty(&q->list))) { + if (!timeout || timeout->task) + schedule(); + } + + __set_current_state(TASK_RUNNING); + +exit: + put_task_struct(p); + return 0; +} + /** * futex_unqueue_multiple - Remove various futexes from their hash bucket * @v: The list of futexes to unqueue @@ -665,8 +732,15 @@ int __futex_wait(u32 __user *uaddr, unsigned int flags, u32 val, if (ret) return ret; - /* futex_queue and wait for wakeup, timeout, or a signal. */ - futex_wait_queue(hb, &q, to); + if (flags & FLAGS_SPIN) { + ret = futex_spin(hb, &q, to, val); + + if (ret) + return ret; + } else { + /* futex_queue and wait for wakeup, timeout, or a signal. */ + futex_wait_queue(hb, &q, to); + } /* If we were woken (and unqueued), we succeeded, whatever. */ if (!futex_unqueue(&q))