From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 68DE0C433ED for ; Sun, 2 May 2021 17:58:55 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id CCB71610A1 for ; Sun, 2 May 2021 17:58:54 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CCB71610A1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=nvidia.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:In-Reply-To:References:Message-ID:Date: Subject:CC:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=B/aek4zB8jWUdfeRjFgxQ2VWGCNkj8QGmbzOeTbMYtU=; b=MSWNY4OCIuE2uBfe+xjco68Zu 8/O67Bh0KQdv5VhIkn7ysJHiH4WTmy3+0ZtiaNz1eUdP2+9oAL6jjpxBNJD82r6xW39bjlVTG1ycm 9aRPCUaAB9LQgpKc1fw1yWENGPnXmJO9lwRJ1DjuqqKu8AP+annpx6m49ui3qaY0FZ6A4MSwHDiN6 P/E9iPrerk4KcNOQN/u1tfREEzXzUiYX6uqtfTOhRORoFqpwl3RQDgErN1hXcDr0WixZrHOKbv+F7 Jt04flJHQ7t8XMFQgOGyJzGpI5rLqQN6YJrxa+LIzGpzzNfLJo1DbyGr/hKOGSYUPLT3dkTRThTTe lfForuA1g==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1ldGKh-00CBt8-C5; Sun, 02 May 2021 17:56:43 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1ldGKd-00CBsw-Es for linux-arm-kernel@desiato.infradead.org; Sun, 02 May 2021 17:56:39 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=MIME-Version: Content-Transfer-Encoding:Content-Type:In-Reply-To:References:Message-ID:Date :Subject:CC:To:From:Sender:Reply-To:Content-ID:Content-Description; bh=DZLRSiQmUiqLxIIkAUhYDnzWcGrHV8uxrm0nUiKanuI=; b=wDUUYrcBLM2B++RVDff+jom4bo JrGdN/K4KWEZ6UEFEQQqBRiT8RX04JGBBPovDdYFahvs10BCCJW+s8ntmkOzNvFY/vbIJn7uEyYYK gMjCVnP17vpviX5MGWfJOG5/GlqPbwWDgX9q1tRYq/ENpLsdxKhf8i2ikrDsUqrAnXIlN8jC9aT0F eKRHSmj/JA2njopoxP0+HWO46jZT1qkM2KKhMhHkjeXEmjU9BwPywkOkNW6rnbQVS43xDPYiWGHix 8PdIQfwiKM+8LGtTqRpLkzcbqe5ta/HHwEmzbNN7TpvJIlPELkDosMI8WB+4erzn5tIkFoC7j5fPk xCsqs3QA==; Received: from mail-dm6nam11on2056.outbound.protection.outlook.com ([40.107.223.56] helo=NAM11-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1ldGKa-002cjN-78 for linux-arm-kernel@lists.infradead.org; Sun, 02 May 2021 17:56:37 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mZwlxIoTos6UrMynqPemo/A1ODhN2KV9MBiOUZOlTDEquqgclmgEZfKOGzjodTN5pGzarOx0vgpiz0hLKAWokX1lSIyf0hjs+byLNoJ0YrwznU8jNXxv9VPU+KLRQNDsiMy6H3e7N6cjTb1FrfQXI6QkCkknoVpzwRFI5c0V9GHV7NlVNRIlAfTOhTrt+x/mrp4UOrluNZXIyCC+fR71P5C9JyecquPYH1opiJqWf3y5lVmJoB18bgg6mwWuKGM/Qb39JjQjkYUiNvqGfFnekIwxcpTHiCbG1trlBXt/6Fn3CZeRE3cp0VdQXFJWh1RU0J4LQ4uL54RKx+xvRr/N1A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DZLRSiQmUiqLxIIkAUhYDnzWcGrHV8uxrm0nUiKanuI=; b=gnDXtd94FCuIfxp0O9pBE1NPhCRYW3UaFQqo3eGlU44r+zQeznUhvZbR85V5M5gMBLM/sZhBwVsRZ9PsUVGKdpXXCEJqgkusyqJIBb6cKHn6WX7IK84levplWkfxK/+HbckzDSS7eey4sqaXl87BtKyL/g2ya+OYWRCXmGhyZGsjxH0lOpFdP0cAWM0qZqlfeOI4/TSkcUFgn8Eeu6pypdtOlAUFV2vH9PFFeV1/o5brSBjBwFan987JBCvb4zUVVvPgyZETZG/SzIqDPOFSeLQl7V4m45mXCOiA0erJx7WawLUX+NF/Gg6sx91My0MCCdWYLsKRQFJHgZEVOLFB3A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DZLRSiQmUiqLxIIkAUhYDnzWcGrHV8uxrm0nUiKanuI=; b=PE3K2n0xrgUQDRZe8WFRTJIDAd80eqLpe3Dv+ZdymOi7pKUGSwCDdvJquk9ctzSprMapfR8bdnCLIGbdT1aYCWY/tw1X5FbEqsqxXh+VhfqjYj+hMl05677fGlsROvgTpgY9hRuecyA+N0LYgrOnriOM8hdurYdjnTgvGIm6Ho75SBC0Q+zg34MhYkO44aePhmqcUJZeaaPQ6mQ1OAggVBb7P3LtdinalW34986CIjE0KPVkbaQjMru7FYGU+rALafLhJJ7Q5oXmipZpRljjfpUvo89xyivDyq8uAnL2SrJE2OHaaZywjqGBoJ5YKt3Fzn6cVeqf7/1E2LM7CU+VRA== Received: from BL0PR12MB2532.namprd12.prod.outlook.com (2603:10b6:207:4a::20) by MN2PR12MB4288.namprd12.prod.outlook.com (2603:10b6:208:1d2::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4087.39; Sun, 2 May 2021 17:56:31 +0000 Received: from BL0PR12MB2532.namprd12.prod.outlook.com ([fe80::5105:e8df:9631:bf0f]) by BL0PR12MB2532.namprd12.prod.outlook.com ([fe80::5105:e8df:9631:bf0f%5]) with mapi id 15.20.4087.040; Sun, 2 May 2021 17:56:31 +0000 From: Vikram Sethi To: Marc Zyngier CC: Shanker Donthineni , Alex Williamson , Will Deacon , Catalin Marinas , Christoffer Dall , "linux-arm-kernel@lists.infradead.org" , "kvmarm@lists.cs.columbia.edu" , "linux-kernel@vger.kernel.org" , "kvm@vger.kernel.org" , Jason Sequeira Subject: RE: [RFC 1/2] vfio/pci: keep the prefetchable attribute of a BAR region in VMA Thread-Topic: [RFC 1/2] vfio/pci: keep the prefetchable attribute of a BAR region in VMA Thread-Index: AQHXPRTigOHff2apwEeawQns3nBPOqrL0MoAgAAM5gCAAAj7gIABBh4AgAAGT4CAADU7AIAACSoAgAABt+CAASvZAIACHFDw Date: Sun, 2 May 2021 17:56:31 +0000 Message-ID: References: <20210429162906.32742-1-sdonthineni@nvidia.com> <20210429162906.32742-2-sdonthineni@nvidia.com> <20210429122840.4f98f78e@redhat.com> <470360a7-0242-9ae5-816f-13608f957bf6@nvidia.com> <20210429134659.321a5c3c@redhat.com> <87czucngdc.wl-maz@kernel.org> <1edb2c4e-23f0-5730-245b-fc6d289951e1@nvidia.com> <878s4zokll.wl-maz@kernel.org> <87eeeqvm1d.wl-maz@kernel.org> In-Reply-To: <87eeeqvm1d.wl-maz@kernel.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: kernel.org; dkim=none (message not signed) header.d=none;kernel.org; dmarc=none action=none header.from=nvidia.com; x-originating-ip: [12.97.180.36] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: aadfdfc5-46ef-49a9-f55e-08d90d939e43 x-ms-traffictypediagnostic: MN2PR12MB4288: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: IU5BE2BNKccQFJmdkqliJVg4SXbL/v0GOajjIW/11Ku/i5mtg95inWXmmFULVx4xcBOHxuqvYKb8GhDxcD5SI2XmLKzaQdEvKOiSfNm8kd0JUueMz19nbIfzhGhfliCoiYbuPYuhLF2kXD7YhAXm1mEKLizb8vFEd8iVco+gsMVT3Z1O9Dek8W+drANzUvRTKf6BQECZXlt/46EL5gDb5pt1yUv88wPLFxYNqwW2+tESPqfSFyQl8vupwoGx0XV6HOyZH2b6H6RzQHoH+V+aRF8pkY2bQWL6arT5W0QkjjW27JPt+9Zsn5BP3V2MlUp3nLtoh1M5m3uSFsm1sJuYJEF7Yh18/udkxfKPvrh+UWZ0eLjhRaX47oV/nAsYAdVtGUhSNG37McGy//oyJtaNLcNQKg5ocZwgwQf9Ev15Y69n9ndXhKXXrXKoQyXzhJWrqF5AFSf/iKGYLRduV0mx4h1IFfvMRvqXCW+/vCzuEKKuq2UNGebRQXpsxhqppi7GoghhjRr6okPoYozqxAIroZfQQ7UOYza51xxqdqJ2xhTkDmVrcN5fsWGjxPgnCiw4bSUzh6uRgHzfVLqhpLoNotWgaE/Tvejk34bdaeJ7MF0= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB2532.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(376002)(396003)(366004)(136003)(346002)(39860400002)(6506007)(5660300002)(4326008)(7696005)(33656002)(71200400001)(107886003)(122000001)(38100700002)(86362001)(478600001)(8936002)(52536014)(64756008)(76116006)(6916009)(186003)(54906003)(8676002)(316002)(55016002)(66446008)(66946007)(2906002)(66556008)(66476007)(26005)(9686003)(83380400001); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?utf-8?B?Vi9ZVDlXMnRMckc1MnM0M29iMjBrUlAwM2Z6cWpLZFdJaDljNFU4ZUt5Wngy?= =?utf-8?B?YXRMY0gxN3gyQlN6Nk5ZZk91ZTU4UEl1TnV5Y0szb2RaN2Z6aFIyYnB1WWVr?= =?utf-8?B?WVh2OTIvQ3IrRDhwaFdHdTBrd2NPODUvRUFYaDNqaGJqbHNVUmQvSFUwZ2lN?= =?utf-8?B?cUN1SldZQk50UmliZXFvaURGbk04MlU1bG5rLzdvUEZJUnBCb1U1UHJ1SHZZ?= =?utf-8?B?MGhRcDBuS1IyNFB5ME4way9FZTB6SWdHVUR4SjBrUmlzMHhjQjVXOTRMQy9E?= =?utf-8?B?cFltZG1ETzE2Sm56dVpzUGZXZTU4TVc3dllkMFZHbW1tMW1qbndFNDdJQ0Vk?= =?utf-8?B?Q2ptTjQ4clJSREFtakNadnp3YWZBcVlXdnZteHB6czhydjIyVncyUWR5VzRV?= =?utf-8?B?MThkNjRNRW1ua0VkU1ZteDJnSTBvTFhRU0hTUHBDWnhPZHhLbGppTDc5WmFw?= =?utf-8?B?MEZ1UU9VTkRyY0J5WW41cjVTcWU3ajBTZzJLdTRmcXJROVZhNTR6OFRpUjdX?= =?utf-8?B?NFZVWVk4U1hiNXFFTnRvQnpqUllUT3I2NlJldk5nTFFzR1VIQnlpUzFLZW9i?= =?utf-8?B?U0dmVU5MSDd6V0Zsa3hwNVk2WnVBNDJ1WVhIQjhobjNQSU5KdkdOb1pwa083?= =?utf-8?B?RnYya3AwTnVYaEx5S0ZZaXFTZ1NHZmlTRnpyQXlVblI1TmZ3cnRQWDhDY1Bj?= =?utf-8?B?T0ZnSThRMVRzZmNSeEE3cW9vYWwrUm41WWRMc29iSkFRME5ibXpZZkUybVN3?= =?utf-8?B?SElDT0ZkOFhqOEQzMG9TQXRmM3NweUxjc09FbjFZdVlScDB5b084cE51ei9U?= =?utf-8?B?ckVuK3JFOGloR0FObEc3Z0lQSG5BVlJ6ZXNjN0R2SlRmQ0plTkN5UzhzK0kx?= =?utf-8?B?TVVNY0FudmNJSmVOQkgzQVdyclNQVEk0Kyt3NExlTzJIdHRUcktxcmtOZ05i?= =?utf-8?B?MHJ3ZjA3djJmL3diT2NMUHFXWXRLeDl1amJzYWp2eDd3bG9DT09TbWg3cW9x?= =?utf-8?B?YzJPM3A5VlVYKzZsZlo5UytqOXJkMHB4Skw3ek1lZXJqb0JtN2IxWXg3WEEw?= =?utf-8?B?dHBrNzRpN3puUXRhNWJBM05nNVBtS0w5NGtweTZ2RCtXTTExMEwvNE1rNGFw?= =?utf-8?B?ckJjZnpEMjRMRnBTeXpYNWJrZ2Yra2JDVjhYZ0xZcEtVSTNYUFZWcVF0RGo2?= =?utf-8?B?QitIQlhnblpnNlB6aU0zU3hUQ3d3bVU1cmJoVFovY2dwU1hEQklNU1p2VVc5?= =?utf-8?B?MDBzQ0JUK1dnRUY0b1Z5SWVKVVovM2ZHZkpKenJwUDRsY1J6c1dscWh2bFFF?= =?utf-8?B?cStZcWY3MFQ0Q0FhZGp0NXV6QkVFbFY5aWRzT2tsOGtrejZnQzNFWXR0RHlD?= =?utf-8?B?OHFxUnZZMnZqM2kxRlJUTGtLV1VFbHFKdXcvMkl2dEtSSXBHc1puN0ZNS3VX?= =?utf-8?B?cGFCYnNzR2hrRU1aU0xoWGFCd2dzMDlIM1h5SEVlTDBreTUzOEc5N2ZiRTU4?= =?utf-8?B?ZVZHb2JBRm0wWnByUG5adEdiVi9UaHVMbnZidVFSNmgzTUx3UGtDUUsyejNt?= =?utf-8?B?MERvellwaHg0dGFIby9FWTVBdGJuZzVGell4REFtYVg1RnhlQkZwRnBnSTlP?= =?utf-8?B?a1M0OTZFMm1TaFlXcHZid1Jxd3dkaTNnMkJXejdjbWQ3UTJBMDJJRXFXNi8v?= =?utf-8?B?cEk0TXBkK2hsTVhBajBzVk1mUWl1dm0ySjJEdzNHbG5JT0ZNV1FIcERFczZn?= =?utf-8?Q?o4i/h7Ikfx+UWYFdAo=3D?= MIME-Version: 1.0 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB2532.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: aadfdfc5-46ef-49a9-f55e-08d90d939e43 X-MS-Exchange-CrossTenant-originalarrivaltime: 02 May 2021 17:56:31.4334 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: swcvaY59vNx5wYIgWV8XYFOz2SXTuxZYJBh7i0Q73dpDp8pAEsUgzwWlhEKpoC78SNxWzYAiEw9bfVVXJota4g== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4288 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210502_105636_433918_C34025A8 X-CRM114-Status: GOOD ( 24.53 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi Marc, > From: Marc Zyngier > Hi Vikram, > > The problem I see is that we have VM and userspace being written in terms > of Write-Combine, which is: > > - loosely defined even on x86 > > - subject to interpretations in the way it maps to PCI > > - has no direct equivalent in the ARMv8 collection of memory > attributes (and Normal_NC comes with speculation capabilities which > strikes me as extremely undesirable on arbitrary devices) If speculation with Normal NC to prefetchable BARs in devices was a problem, those devices would already be broken in baremetal with ioremap_wc on arm64, and we would need quirks there to not do Normal NC for them but Device GRE, and if such a quirk was needed on baremetal, it could be picked up by vfio/KVM as well. But we haven't seen any broken devices doing wc on baremetal on ARM64, have we? I know we have tested NICs write combining on arm64 in baremetal, as well as GPU and NVMe CMB without issues. Further, I don't see why speculation to non cacheble would be an issue if prefetch without side effects is allowed by the device, which is what a prefetchable BAR is. If it is an issue for a device I would consider that a bug already needing a quirk in Baremetal/host kernel already. >From PCI spec " A prefetchable address range may have write side effects, but it may not have read side effects." > > How do we translate this into something consistent? I'd like to see an actual > description of what we *really* expect from WC on prefetchable PCI regions, > turn that into a documented definition agreed across architectures, and then > we can look at implementing it with one memory type or another on arm64. > > Because once we expose that memory type at S2 for KVM guests, it > becomes ABI and there is no turning back. So I want to get it right once and > for all. > I agree that we need a precise definition for the Linux ioremap_wc API wrt what drivers (kernel and userspace) can expect and whether memset/memcpy is expected to work or not and whether aligned accesses are a requirement. To the extent ABI is set, I would think that the ABI is also already set in the host kernel for arm64 WC = Normal NC, so why should that not also be the ABI for same driver in VMs. > Thanks, > > M. > > -- > Without deviation from the norm, progress is not possible. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel