From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.6 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 193ECC4727E for ; Wed, 23 Sep 2020 15:00:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C866B206E5 for ; Wed, 23 Sep 2020 15:00:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="rTmFAlor" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726629AbgIWPAy (ORCPT ); Wed, 23 Sep 2020 11:00:54 -0400 Received: from smtp-fw-4101.amazon.com ([72.21.198.25]:15590 "EHLO smtp-fw-4101.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726130AbgIWPAy (ORCPT ); Wed, 23 Sep 2020 11:00:54 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1600873253; x=1632409253; h=to:cc:from:subject:message-id:date:mime-version: content-transfer-encoding; bh=ncTiycxbLLLABb828Z5CFC0O0VQndKnHSZDh9Pb2FUA=; b=rTmFAlorAGGAk+ILKIy10Y1JeVgXeMpaEnaYjejtXbzJCtfUxxXkp2hR IeF+b4vChQD82MagQ0hvshtJ9I+nuXv/ltJUhXltE5V6SdBu0BZSYiSUi wosUIlLQ7vIjmm795DSWyY2r1TmCBiqj6lBD7o6REiPBpcTRrrPaZohlX Q=; X-IronPort-AV: E=Sophos;i="5.77,293,1596499200"; d="scan'208";a="55868833" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-2b-859fe132.us-west-2.amazon.com) ([10.43.8.6]) by smtp-border-fw-out-4101.iad4.amazon.com with ESMTP; 23 Sep 2020 15:00:49 +0000 Received: from EX13MTAUWC002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan2.pdx.amazon.com [10.170.41.162]) by email-inbound-relay-2b-859fe132.us-west-2.amazon.com (Postfix) with ESMTPS id 63C8F222BE2; Wed, 23 Sep 2020 15:00:45 +0000 (UTC) Received: from EX13D12UWC002.ant.amazon.com (10.43.162.253) by EX13MTAUWC002.ant.amazon.com (10.43.162.240) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 23 Sep 2020 15:00:45 +0000 Received: from [10.95.178.71] (10.43.161.71) by EX13D12UWC002.ant.amazon.com (10.43.162.253) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 23 Sep 2020 15:00:43 +0000 To: Peter Zijlstra , Ingo Molnar , Thomas Gleixner , Sebastian Andrzej Siewior , Peter Xu , Kaitao Cheng CC: From: George Prekas Subject: [PATCH] latency improvement in __smp_call_single_queue Message-ID: <281da382-4511-e1df-6917-154a5914dd43@amazon.com> Date: Wed, 23 Sep 2020 10:00:41 -0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.12.0 MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Originating-IP: [10.43.161.71] X-ClientProxiedBy: EX13D48UWB004.ant.amazon.com (10.43.163.74) To EX13D12UWC002.ant.amazon.com (10.43.162.253) Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org If an interrupt arrives between llist_add and send_call_function_single_ipi in the following code snippet, then the remote CPU will not receive the IPI in a timely manner and subsequent SMP calls even from other CPUs for other functions will be delayed:     if (llist_add(node, &per_cpu(call_single_queue, cpu)))         send_call_function_single_ipi(cpu); Note: llist_add returns 1 if it was empty before the operation. CPU 0                           | CPU 1                     | CPU 2 __smp_call_single_q(2,f1)       | __smp_call_single_q(2,f2) |   llist_add returns 1           |                           |   interrupted                   |   llist_add returns 0     |       ...                       |   branch not taken        |       ...                       |                           |   resumed                       |                           |   send_call_function_single_ipi |                           |                                 |                           | f1                                 |                           | f2 The call from CPU 1 for function f2 will be delayed because CPU 0 was interrupted. Signed-off-by: George Prekas ---  kernel/smp.c | 4 ++++  1 file changed, 4 insertions(+) diff --git a/kernel/smp.c b/kernel/smp.c index aa17eedff5be..9dc679466cf0 100644 --- a/kernel/smp.c +++ b/kernel/smp.c @@ -135,6 +135,8 @@ static DEFINE_PER_CPU_SHARED_ALIGNED(call_single_data_t, csd_data);  void __smp_call_single_queue(int cpu, struct llist_node *node)  { +    unsigned long flags; +      /*       * The list addition should be visible before sending the IPI       * handler locks the list to pull the entry off it because of @@ -146,8 +148,10 @@ void __smp_call_single_queue(int cpu, struct llist_node *node)       * locking and barrier primitives. Generic code isn't really       * equipped to do the right thing...       */ +    local_irq_save(flags);      if (llist_add(node, &per_cpu(call_single_queue, cpu)))          send_call_function_single_ipi(cpu); +    local_irq_restore(flags);  }  /* -- 2.16.6