From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.9 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 572C9C32771 for ; Mon, 27 Jan 2020 06:17:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 2AB6120842 for ; Mon, 27 Jan 2020 06:17:14 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="Y/YSmaBF" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726871AbgA0GRM (ORCPT ); Mon, 27 Jan 2020 01:17:12 -0500 Received: from aserp2120.oracle.com ([141.146.126.78]:56656 "EHLO aserp2120.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725775AbgA0GRM (ORCPT ); Mon, 27 Jan 2020 01:17:12 -0500 Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id 00R6DIcW119854; Mon, 27 Jan 2020 06:16:37 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=content-type : mime-version : subject : from : in-reply-to : date : cc : content-transfer-encoding : message-id : references : to; s=corp-2019-08-05; bh=gJ/jpXTHwRBSq22rXvdlhHavOGJ9Q+e1no1Mq6F4Bvw=; b=Y/YSmaBFS9lARC5G6233J/9lchWbfrA75ugGtpPugGClHYpgmtGMotz2gMoQF8X/ATIJ ygHRsmQtsrk6wFVvWBC0NS2fLFkbJv8RkTGJBmuHc3+C4TDgVf+CJAwmIsG7qKEm7osv 3/R+M0BOGKKElRC4TBXo9DP5tu0yp92Kf9zNe3FPZbTkekOzi8hDht1OnMngD0E09fhL aRdznbzsQkbytkP5N8hEx/Wac5dc82MyS4xVbntjXZrKdO3L1qOmdOh6Jaqq/SvF98i+ an5wMSQtn2I3E1NGzNmEh9Dk0g8euyQ3jy12mvLH1cY1JMn1piQQ8PMWX9XKLEabYKfB MQ== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by aserp2120.oracle.com with ESMTP id 2xrdmq5adk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 27 Jan 2020 06:16:37 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.27/8.16.0.27) with SMTP id 00R6ECZ9111490; Mon, 27 Jan 2020 06:16:36 GMT Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by aserp3020.oracle.com with ESMTP id 2xry6qr03t-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 27 Jan 2020 06:16:36 +0000 Received: from abhmp0002.oracle.com (abhmp0002.oracle.com [141.146.116.8]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id 00R6GYZC001241; Mon, 27 Jan 2020 06:16:34 GMT Received: from [10.39.241.133] (/10.39.241.133) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Sun, 26 Jan 2020 22:16:34 -0800 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\)) Subject: Re: [PATCH v9 0/5] Add NUMA-awareness to qspinlock From: Alex Kogan In-Reply-To: Date: Mon, 27 Jan 2020 01:16:36 -0500 Cc: linux@armlinux.org.uk, Peter Zijlstra , mingo@redhat.com, will.deacon@arm.com, arnd@arndb.de, longman@redhat.com, linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, tglx@linutronix.de, bp@alien8.de, hpa@zytor.com, x86@kernel.org, guohanjun@huawei.com, jglauber@marvell.com, dave.dice@oracle.com, steven.sistare@oracle.com, daniel.m.jordan@oracle.com, Will Deacon Content-Transfer-Encoding: quoted-printable Message-Id: <3CBCD208-0859-458E-8A89-96E8E9A98664@oracle.com> References: <20200115035920.54451-1-alex.kogan@oracle.com> <4F71A184-42C0-4865-9AAA-79A636743C25@oracle.com> To: Lihao Liang X-Mailer: Apple Mail (2.3445.104.11) X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9512 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1911140001 definitions=main-2001270054 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9512 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1911140001 definitions=main-2001270054 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org >>> This is particularly relevant >>> in high contention situations when new threads keep arriving on the = same >>> socket as the lock holder. >> In this case, the lock will stay on the same NUMA node/socket for >> 2^numa_spinlock_threshold times, which is the worst case scenario if = we >> consider the long-term fairness. And if we have multiple nodes, it = will take >> up to 2^numa_spinlock_threshold X (nr_nodes - 1) + nr_cpus_per_node >> lock transitions until any given thread will acquire the lock >> (assuming 2^numa_spinlock_threshold > nr_cpus_per_node). >>=20 >=20 > You're right that the latest version of the patch handles long-term = fairness > deterministically. >=20 > As I understand it, the n-th thread in the main queue is guaranteed to > acquire the lock after N lock handovers, where N is bounded by >=20 > n - 1 + 2^numa_spinlock_threshold * (nr_nodes - 1) >=20 > I'm not sure what role the variable nr_cpus_per_node plays in your = analysis. Yeah, that=E2=80=99s a minor point, but let me try to clarify. The "n-th thread in the main queue=E2=80=9D is (at most) the = nr_cpus_per_node-th thread=20 for some node k. So when the node k gets the preference, that thread = will get the lock after at most nr_cpus_per_node-1 lock transitions. As we = consider the upper bound, your analysis is also correct; mine is just a bit = tighter. Makes sense? Regards, =E2=80=94 Alex From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: * X-Spam-Status: No, score=1.8 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED,URIBL_DBL_ABUSE_MALW autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A1759C32771 for ; Mon, 27 Jan 2020 06:17:09 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 64F1820842 for ; Mon, 27 Jan 2020 06:17:09 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="T7J3XJJX"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="Y/YSmaBF" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 64F1820842 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:To:References:Message-Id:Date: In-Reply-To:From:Subject:Mime-Version:Reply-To:Content-ID:Content-Description :Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=UEpVimSOrXuY4zN7ry/noCWKlZCWlyZ1OfZWsEySjYI=; b=T7J3XJJXHe4XVN 55X69tvSX4DwAywzWMDxmgZVk46CYtUlEO1iJ74e61HB0bVtK6kcL+nLLLUHtcj66/QLKmuWanWCT RsePw8gePxqp6OBAdZuAC0kA7+2vkzp+90rp/vITb5eolwl9m/mj0S3goYt4xVvS4VS+EN4BuqxY8 1cXgjkMBkbKjNVGzEHgbyy2FOVkwUYBuGENkax/gl/kfkuldbaTkWoq7cHsei8R+/umjHGl21luPL W0GPoTOiZXg2kjAwI9olS5j0IvJ1MZMg57GWI0dDgxf+nQACBgEYZrULULwwB6sRbKnLa5tUR2vq+ wz8VmIFet86wfak1si4A==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1ivxhs-0005tx-Ry; Mon, 27 Jan 2020 06:17:08 +0000 Received: from aserp2120.oracle.com ([141.146.126.78]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1ivxhp-0005tS-Hb for linux-arm-kernel@lists.infradead.org; Mon, 27 Jan 2020 06:17:07 +0000 Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id 00R6DIcW119854; Mon, 27 Jan 2020 06:16:37 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=content-type : mime-version : subject : from : in-reply-to : date : cc : content-transfer-encoding : message-id : references : to; s=corp-2019-08-05; bh=gJ/jpXTHwRBSq22rXvdlhHavOGJ9Q+e1no1Mq6F4Bvw=; b=Y/YSmaBFS9lARC5G6233J/9lchWbfrA75ugGtpPugGClHYpgmtGMotz2gMoQF8X/ATIJ ygHRsmQtsrk6wFVvWBC0NS2fLFkbJv8RkTGJBmuHc3+C4TDgVf+CJAwmIsG7qKEm7osv 3/R+M0BOGKKElRC4TBXo9DP5tu0yp92Kf9zNe3FPZbTkekOzi8hDht1OnMngD0E09fhL aRdznbzsQkbytkP5N8hEx/Wac5dc82MyS4xVbntjXZrKdO3L1qOmdOh6Jaqq/SvF98i+ an5wMSQtn2I3E1NGzNmEh9Dk0g8euyQ3jy12mvLH1cY1JMn1piQQ8PMWX9XKLEabYKfB MQ== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by aserp2120.oracle.com with ESMTP id 2xrdmq5adk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 27 Jan 2020 06:16:37 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.27/8.16.0.27) with SMTP id 00R6ECZ9111490; Mon, 27 Jan 2020 06:16:36 GMT Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by aserp3020.oracle.com with ESMTP id 2xry6qr03t-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 27 Jan 2020 06:16:36 +0000 Received: from abhmp0002.oracle.com (abhmp0002.oracle.com [141.146.116.8]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id 00R6GYZC001241; Mon, 27 Jan 2020 06:16:34 GMT Received: from [10.39.241.133] (/10.39.241.133) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Sun, 26 Jan 2020 22:16:34 -0800 Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\)) Subject: Re: [PATCH v9 0/5] Add NUMA-awareness to qspinlock From: Alex Kogan In-Reply-To: Date: Mon, 27 Jan 2020 01:16:36 -0500 Message-Id: <3CBCD208-0859-458E-8A89-96E8E9A98664@oracle.com> References: <20200115035920.54451-1-alex.kogan@oracle.com> <4F71A184-42C0-4865-9AAA-79A636743C25@oracle.com> To: Lihao Liang X-Mailer: Apple Mail (2.3445.104.11) X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9512 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1911140001 definitions=main-2001270054 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9512 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1911140001 definitions=main-2001270054 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200126_221705_665891_182085D3 X-CRM114-Status: GOOD ( 15.40 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, guohanjun@huawei.com, arnd@arndb.de, Peter Zijlstra , dave.dice@oracle.com, jglauber@marvell.com, x86@kernel.org, will.deacon@arm.com, linux@armlinux.org.uk, steven.sistare@oracle.com, linux-kernel@vger.kernel.org, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, longman@redhat.com, tglx@linutronix.de, daniel.m.jordan@oracle.com, Will Deacon , linux-arm-kernel@lists.infradead.org Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org Cj4+PiBUaGlzIGlzIHBhcnRpY3VsYXJseSByZWxldmFudAo+Pj4gaW4gaGlnaCBjb250ZW50aW9u IHNpdHVhdGlvbnMgd2hlbiBuZXcgdGhyZWFkcyBrZWVwIGFycml2aW5nIG9uIHRoZSBzYW1lCj4+ PiBzb2NrZXQgYXMgdGhlIGxvY2sgaG9sZGVyLgo+PiBJbiB0aGlzIGNhc2UsIHRoZSBsb2NrIHdp bGwgc3RheSBvbiB0aGUgc2FtZSBOVU1BIG5vZGUvc29ja2V0IGZvcgo+PiAyXm51bWFfc3Bpbmxv Y2tfdGhyZXNob2xkIHRpbWVzLCB3aGljaCBpcyB0aGUgd29yc3QgY2FzZSBzY2VuYXJpbyBpZiB3 ZQo+PiBjb25zaWRlciB0aGUgbG9uZy10ZXJtIGZhaXJuZXNzLiBBbmQgaWYgd2UgaGF2ZSBtdWx0 aXBsZSBub2RlcywgaXQgd2lsbCB0YWtlCj4+IHVwIHRvIDJebnVtYV9zcGlubG9ja190aHJlc2hv bGQgWCAobnJfbm9kZXMgLSAxKSArIG5yX2NwdXNfcGVyX25vZGUKPj4gbG9jayB0cmFuc2l0aW9u cyB1bnRpbCBhbnkgZ2l2ZW4gdGhyZWFkIHdpbGwgYWNxdWlyZSB0aGUgbG9jawo+PiAoYXNzdW1p bmcgMl5udW1hX3NwaW5sb2NrX3RocmVzaG9sZCA+IG5yX2NwdXNfcGVyX25vZGUpLgo+PiAKPiAK PiBZb3UncmUgcmlnaHQgdGhhdCB0aGUgbGF0ZXN0IHZlcnNpb24gb2YgdGhlIHBhdGNoIGhhbmRs ZXMgbG9uZy10ZXJtIGZhaXJuZXNzCj4gZGV0ZXJtaW5pc3RpY2FsbHkuCj4gCj4gQXMgSSB1bmRl cnN0YW5kIGl0LCB0aGUgbi10aCB0aHJlYWQgaW4gdGhlIG1haW4gcXVldWUgaXMgZ3VhcmFudGVl ZCB0bwo+IGFjcXVpcmUgdGhlIGxvY2sgYWZ0ZXIgTiBsb2NrIGhhbmRvdmVycywgd2hlcmUgTiBp cyBib3VuZGVkIGJ5Cj4gCj4gbiAtIDEgKyAyXm51bWFfc3BpbmxvY2tfdGhyZXNob2xkICogKG5y X25vZGVzIC0gMSkKPiAKPiBJJ20gbm90IHN1cmUgd2hhdCByb2xlIHRoZSB2YXJpYWJsZSBucl9j cHVzX3Blcl9ub2RlIHBsYXlzIGluIHlvdXIgYW5hbHlzaXMuCgpZZWFoLCB0aGF04oCZcyBhIG1p bm9yIHBvaW50LCBidXQgbGV0IG1lIHRyeSB0byBjbGFyaWZ5LgoKVGhlICJuLXRoIHRocmVhZCBp biB0aGUgbWFpbiBxdWV1ZeKAnSBpcyAoYXQgbW9zdCkgdGhlIG5yX2NwdXNfcGVyX25vZGUtdGgg dGhyZWFkIApmb3Igc29tZSBub2RlIGsuIFNvIHdoZW4gdGhlIG5vZGUgayBnZXRzIHRoZSBwcmVm ZXJlbmNlLCB0aGF0IHRocmVhZCB3aWxsCmdldCB0aGUgbG9jayBhZnRlciBhdCBtb3N0IG5yX2Nw dXNfcGVyX25vZGUtMSBsb2NrIHRyYW5zaXRpb25zLiBBcyB3ZSBjb25zaWRlcgp0aGUgdXBwZXIg Ym91bmQsIHlvdXIgYW5hbHlzaXMgaXMgYWxzbyBjb3JyZWN0OyBtaW5lIGlzIGp1c3QgYSBiaXQg dGlnaHRlci4KCk1ha2VzIHNlbnNlPwoKUmVnYXJkcywK4oCUIEFsZXgKCgpfX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpsaW51eC1hcm0ta2VybmVsIG1haWxp bmcgbGlzdApsaW51eC1hcm0ta2VybmVsQGxpc3RzLmluZnJhZGVhZC5vcmcKaHR0cDovL2xpc3Rz LmluZnJhZGVhZC5vcmcvbWFpbG1hbi9saXN0aW5mby9saW51eC1hcm0ta2VybmVsCg==