From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 652C9C4332B for ; Tue, 24 Mar 2020 19:41:45 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1988A20714 for ; Tue, 24 Mar 2020 19:41:45 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=Mellanox.com header.i=@Mellanox.com header.b="IBY+EK2a" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1988A20714 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=mellanox.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A2E386B0003; Tue, 24 Mar 2020 15:41:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9DF3B6B0006; Tue, 24 Mar 2020 15:41:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8CED36B0007; Tue, 24 Mar 2020 15:41:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0143.hostedemail.com [216.40.44.143]) by kanga.kvack.org (Postfix) with ESMTP id 75F5F6B0003 for ; Tue, 24 Mar 2020 15:41:44 -0400 (EDT) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 1B547181AEEDE for ; Tue, 24 Mar 2020 19:41:44 +0000 (UTC) X-FDA: 76631275728.29.frog15_8372cc87d031c X-HE-Tag: frog15_8372cc87d031c X-Filterd-Recvd-Size: 9906 Received: from EUR05-DB8-obe.outbound.protection.outlook.com (mail-db8eur05on2073.outbound.protection.outlook.com [40.107.20.73]) by imf49.hostedemail.com (Postfix) with ESMTP for ; Tue, 24 Mar 2020 19:41:43 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=btV3IYS54aG0XRL+2bBxy5h/HZ+vhXMXITA7vrEWaVX8CcblyU940ytxsks//dW4C8jl9jVkY/RQ2I1BIT2ilWvghOEAh41M5OovYzVyZs3VbhaY1NpMJxffb0oZYz9XSYHPGkT9pqw4OBiKuMaj1DtsZs/0wg0j7altH0LM4cEQjE9IqurKFKWZPrIje5ckCA0+Md9fBngBDmB3QsJSokPCJyCVlSanwpJDzBfa4HGAg9811k1UsXgRK5gHwR4hF1i8tN4JCZ2DFBgzpz1dwTvCJHdG5JUsGU224R61NDja8A8lIijghd+jg8aQXaqu3Ixp40rGO57TyWMdkSzFFw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=d/uLPJjGfMGvvnBU2UEU/+wx6/Cauxyg9wowmu9tVbs=; b=UU3AnuK024Dx0IOyUBIYZA4CTplLA+0ewFHWNsm7THhVqkw1lIruMJAGv3YHSG9/UHC8iLzLQLyQb7koukVHEnP3MdzDWSiCI1w+hXvWEVBHha6v3Wvxlhxx69P3au8HSraAZggwFKwWh94eGs+BLnPdWUxURRerOs6YhTj5DEuH6AcpK+2+csE3ut+DVZeAbzg5wmn6rpGSctlVu6ZZaskhcgKA1PePyOlLpIScSYHrmqqb1zJ4QTiGRTiIrpTsIM63YWjMp32JcVgxogD4UNDLDu0MxKy/1kV7d6omBj2Ce7kipvdyLlRPSeS2w+E3wTxzoQaC29/V0jYs/qbZhQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=mellanox.com; dmarc=pass action=none header.from=mellanox.com; dkim=pass header.d=mellanox.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=d/uLPJjGfMGvvnBU2UEU/+wx6/Cauxyg9wowmu9tVbs=; b=IBY+EK2aWKI/D8FWFtHLLIJjlJLhagYCDKx36M7gqEgzYakDvV2NOf/SopkJ1v+YnBq5/erPPCl71EyH2DJmP3Wl97CtezzZg2LEYzpT6koAb+dEaVR3fr/aLqll26iJpFMDpZCoLDgLVMCck4Xj/NP1fW/iREm1+yaiE1CAstw= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=jgg@mellanox.com; Received: from VI1PR05MB4141.eurprd05.prod.outlook.com (52.133.14.15) by VI1PR05MB5470.eurprd05.prod.outlook.com (20.177.200.91) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2835.20; Tue, 24 Mar 2020 19:41:41 +0000 Received: from VI1PR05MB4141.eurprd05.prod.outlook.com ([fe80::18d2:a9ea:519:add3]) by VI1PR05MB4141.eurprd05.prod.outlook.com ([fe80::18d2:a9ea:519:add3%7]) with mapi id 15.20.2835.023; Tue, 24 Mar 2020 19:41:41 +0000 Date: Tue, 24 Mar 2020 16:41:37 -0300 From: Jason Gunthorpe To: linux-mm@kvack.org Cc: Michal Hocko , =?utf-8?B?SsOpcsO0bWU=?= Glisse , Christoph Hellwig Subject: Re: [PATCH v3] mm/mmu_notifier: prevent unpaired invalidate_start and invalidate_end Message-ID: <20200324194137.GQ13183@mellanox.com> References: <20200211205252.GA10003@ziepe.ca> <20200228135006.GA30885@ziepe.ca> Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20200228135006.GA30885@ziepe.ca> User-Agent: Mutt/1.9.4 (2018-02-28) X-ClientProxiedBy: BL0PR02CA0051.namprd02.prod.outlook.com (2603:10b6:207:3d::28) To VI1PR05MB4141.eurprd05.prod.outlook.com (2603:10a6:803:44::15) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from mlx.ziepe.ca (142.68.57.212) by BL0PR02CA0051.namprd02.prod.outlook.com (2603:10b6:207:3d::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2835.18 via Frontend Transport; Tue, 24 Mar 2020 19:41:41 +0000 Received: from jgg by mlx.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1jGpQf-0002y3-HD; Tue, 24 Mar 2020 16:41:37 -0300 X-Originating-IP: [142.68.57.212] X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 940f5a6e-7c9c-4d31-e857-08d7d02b6013 X-MS-TrafficTypeDiagnostic: VI1PR05MB5470: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-Forefront-PRVS: 03524FBD26 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(4636009)(376002)(346002)(39860400002)(366004)(396003)(136003)(9786002)(9746002)(81156014)(81166006)(8676002)(316002)(66574012)(2906002)(1076003)(5660300002)(6916009)(54906003)(966005)(36756003)(52116002)(186003)(8936002)(4326008)(478600001)(2616005)(66946007)(33656002)(66476007)(26005)(66556008)(86362001)(24400500001);DIR:OUT;SFP:1101;SCL:1;SRVR:VI1PR05MB5470;H:VI1PR05MB4141.eurprd05.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords; Received-SPF: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: l3qA0GSQWLow/lUW09PbBu44tvS5DMjQyhxODLNa9OjrV3dKUiWlsDDhV0EVFzKPkWpfj7jOUsymwPwpBdloEbhi29YkNF2cIl8Ly1EVfos11Kfk1l/e5P1jY+MX33D7fRvWPIhkcfnjxjGpiwZ+AJ4xiSL9U7SoTC8w7FVDNVqRjlfGLhbaYsaFcL9RzfCIMMG+skjm5FOYjCyDKVWLnfPRLoRrwkFHXmVwQjS1EQWqQRrUx+6vIVuGOf7fFgJGFq4q/te563VSMD5GTO/Zcp6L48UqG/7YFWRSxZRFyQyHGF4Dwd1INjpXY8GBnlcYri9NM9z/MN53a8pcFGE9ggSgb1tZAKPT3ec/v01YT7TMz3pUSgQL2Q0AuywUD1MFcG9EX5Grm4cKtDPnrUgDiU+xksVGKt4K758fBQ2wIbQtuNB43dKKs7qlICVmS8AABMi8EisWPEba/eWpmCLH1KVFIGps0bdVn9FRoDIRzNLyJ+mXVOegNPe0PblPMXx9a+5zMqe+a5vS59bfBGNGKUqwQV3r9ObO2wI6+DlDiCquw2o8EO6zqxhtDNSW9/Yf2PDqm22WPc8yy3+wASoDzg== X-MS-Exchange-AntiSpam-MessageData: HmDj5yq3qIBwNd1Qf0yzmTkT4Li5+ZHPtz7GdWj+/oHEAnDNXkNxdSTR9EZB6ZHamyz+VLexeNp0qxtIOR3pFivb9tgDdoHGD1h1mebnwPpmykirtA+PI3vkxn6WKKbU+CrdIxmTv6GHJjUTSKJxlg== X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 940f5a6e-7c9c-4d31-e857-08d7d02b6013 X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Mar 2020 19:41:41.3381 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: yjFJCAz8Gvs1SRZCLlnRLWx9QqCdAwejgvbjF7AUw8lxUwcCPpZ5ZQiK2mVW/rd6Vd717SSx2IrmphI39KMCjA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR05MB5470 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, Feb 28, 2020 at 09:50:06AM -0400, Jason Gunthorpe wrote: > On Tue, Feb 11, 2020 at 04:52:52PM -0400, Jason Gunthorpe wrote: > > Many users of the mmu_notifier invalidate_range callbacks maintain > > locking/counters/etc on a paired basis and have long expected that > > invalidate_range_start/end() are always paired. > >=20 > > For instance kvm_mmu_notifier_invalidate_range_end() undoes > > kvm->mmu_notifier_count which was incremented during start(). > >=20 > > The recent change to add non-blocking notifiers breaks this assumptio= n > > when multiple notifiers are present in the list. When EAGAIN is retur= ned > > from an invalidate_range_start() then no invalidate_range_ends() are > > called, even if the subscription's start had previously been called. > >=20 > > Unfortunately, due to the RCU list traversal we can't reliably genera= te a > > subset of the linked list representing the notifiers already called t= o > > generate an invalidate_range_end() pairing. > >=20 > > One case works correctly, if only one subscription requires > > invalidate_range_end() and it is the last entry in the hlist. In this > > case, when invalidate_range_start() returns -EAGAIN there will be not= hing > > to unwind. > >=20 > > Keep the notifier hlist sorted so that notifiers that require > > invalidate_range_end() are always last, and if two are added then dis= able > > non-blocking invalidation for the mm. > >=20 > > A warning is printed for this case, if in future we determine this ne= ver > > happens then we can simply fail during registration when there are > > unsupported combinations of notifiers. > >=20 > > Fixes: 93065ac753e4 ("mm, oom: distinguish blockable mode for mmu not= ifiers") > > Cc: Michal Hocko > > Cc: "J=C3=A9r=C3=B4me Glisse" > > Cc: Christoph Hellwig > > Signed-off-by: Jason Gunthorpe > > mm/mmu_notifier.c | 53 ++++++++++++++++++++++++++++++++++++++++++++-= -- > > 1 file changed, 50 insertions(+), 3 deletions(-) > >=20 > > v1: https://lore.kernel.org/linux-mm/20190724152858.GB28493@ziepe.ca/ > > v2: https://lore.kernel.org/linux-mm/20190807191627.GA3008@ziepe.ca/ > > * Abandon attempting to fix it by calling invalidate_range_end() duri= ng an > > EAGAIN start > > * Just trivially ban multiple subscriptions > > v3: > > * Be more sophisticated, ban only multiple subscriptions if the resul= t is > > a failure. Allows multiple subscriptions without invalidate_range_e= nd > > * Include a printk when this condition is hit (Michal) > >=20 > > At this point the rework Christoph requested during the first posting > > is completed and there are now only 3 drivers using > > invalidate_range_end(): > >=20 > > drivers/misc/mic/scif/scif_dma.c: .invalidate_range_end =3D sci= f_mmu_notifier_invalidate_range_end}; > > drivers/misc/sgi-gru/grutlbpurge.c: .invalidate_range_end =3D g= ru_invalidate_range_end, > > virt/kvm/kvm_main.c: .invalidate_range_end =3D kvm_mmu_notifier_= invalidate_range_end, > >=20 > > While I think it is unlikely that any of these drivers will be used i= n > > combination with each other, display a printk in hopes to check. > >=20 > > Someday I expect to just fail the registration on this condition. > >=20 > > I think this also addresses Michal's concern about a 'big hammer' as > > it probably won't ever trigger now. >=20 > I'm going to put this in linux-next to see if there are any reports of > the pr_warn failing. >=20 > Michal, are you happy with this solution now? It's been a month in linux-next now, with no complaints. If there are no comments I will go ahead to send it in the hmm PR. Thanks, Jason