From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.5 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_2 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61805C43461 for ; Thu, 3 Sep 2020 22:26:52 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3CCD120722 for ; Thu, 3 Sep 2020 22:26:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727804AbgICW0v (ORCPT ); Thu, 3 Sep 2020 18:26:51 -0400 Received: from kernel.crashing.org ([76.164.61.194]:54290 "EHLO kernel.crashing.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728127AbgICW0v (ORCPT ); Thu, 3 Sep 2020 18:26:51 -0400 Received: from localhost (gate.crashing.org [63.228.1.57]) (authenticated bits=0) by kernel.crashing.org (8.14.7/8.14.7) with ESMTP id 083MQOD4014891 (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Thu, 3 Sep 2020 17:26:28 -0500 Message-ID: <56d2f113acef3055d3bf771fa6cd7c976fc6da65.camel@kernel.crashing.org> Subject: Re: [PATCH] arm64: Enable PCI write-combine resources under sysfs From: Benjamin Herrenschmidt To: Lorenzo Pieralisi Cc: Clint Sbisa , linux-pci@vger.kernel.org, Bjorn Helgaas , linux-arm-kernel@lists.infradead.org, will@kernel.org, catalin.marinas@arm.com Date: Fri, 04 Sep 2020 08:26:23 +1000 In-Reply-To: <20200903110844.GB11284@e121166-lin.cambridge.arm.com> References: <20200831151827.pumm2p54fyj7fz5s@amazon.com> <20200902113207.GA27676@e121166-lin.cambridge.arm.com> <20200902142922.xc4x6m33unkzewuh@amazon.com> <20200902164702.GA30611@e121166-lin.cambridge.arm.com> <20200903110844.GB11284@e121166-lin.cambridge.arm.com> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.28.5-0ubuntu0.18.04.2 Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org On Thu, 2020-09-03 at 12:08 +0100, Lorenzo Pieralisi wrote: > "Additional Guidance on the Prefetchable Bit in Memory Space BARs" > > I read it 100 times and I still have no idea how it can be > implemented, > it sorts of acknowledges that read side-effects memory can be marked > as a prefetchable BAR *if* the system meets some criteria. > > As if endpoint designers knew the system where their endpoint is > plugged into (+ bit (3) in a BAR is read-only). > > I think that that implementation note must be removed from the > specifications - if anyone dares to follow it this whole > WC resource mapping can trigger trouble. Ah that one ! Yes you are right its completely broken. This part of the spec aims at working around the fact that bridges only have 64-bit prefetchable windows, so anything non-pref has to go below a 32-bit bridge window (effectively making most 64-bit non-pref BARs a pointless waste of silicon). The right fix of course would have been to create a new type of bridge window. But PCI... If you're going to mess around with the SIG, I would suggest that a better approach short of the above would be to allow system software to put 64-bit non-pref BARs below bridge pref windows on PCIe (provided the various otehr restrictions in that note are honored such as bridges not prefetching) and leave it at that. (Unless they already do somewhere else, I forgot ...). This should be sufficient to address the space concern without killing the meaning of the prefetchable bit. As for enabling the _wc files in sysfs, well, you need some serious priviledge to be able to access them, so I don't see a big issue allowing them to exist when "prefetchable" is set regardless of that rule. The Mellanox case might be different. Cheers, Ben. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_2 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49B39C10DAA for ; Thu, 3 Sep 2020 22:27:58 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1161C20716 for ; Thu, 3 Sep 2020 22:27:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="b6M36ovu" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1161C20716 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.crashing.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Mime-Version:References:In-Reply-To:Date:To:From: Subject:Message-ID:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=EfBCe0QpIsz3Ua0xJXgVO0ezJ5nILzU9AmZ+G5TIQ1I=; b=b6M36ovuLMHSSLEdJDin6Q691 7IUcUL5/eAQmNiYWpzWWpVHIeUtw+62Y0zCR9qXsAKP4UeZwifDMsKV6cb58APApVBO0tK/Tv7GdQ c2uveS8u9beHqD3uP+cN7QJnsONdlU1WQVLQHf5gme6YpnhyH9glmJ24FQewZOSuHgB1culu2AwMG CY2+4QmDl6yw99FAsZKix5M8brOr9JZ/bTathBlGP78kN+nSeVripY5lCjjSi1kHBgTvPpcVfgPpg 9ZGSy10Apa9YZd1laUxP7U/E4WDze5Kz0GPyXwHNsL0hOMfEC1PM7VefK30REt++UnOgxXseMY53N U0ZIEPXeA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kDxgw-0000p2-SE; Thu, 03 Sep 2020 22:26:51 +0000 Received: from kernel.crashing.org ([76.164.61.194]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kDxgs-0000oH-Nz for linux-arm-kernel@lists.infradead.org; Thu, 03 Sep 2020 22:26:48 +0000 Received: from localhost (gate.crashing.org [63.228.1.57]) (authenticated bits=0) by kernel.crashing.org (8.14.7/8.14.7) with ESMTP id 083MQOD4014891 (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Thu, 3 Sep 2020 17:26:28 -0500 Message-ID: <56d2f113acef3055d3bf771fa6cd7c976fc6da65.camel@kernel.crashing.org> Subject: Re: [PATCH] arm64: Enable PCI write-combine resources under sysfs From: Benjamin Herrenschmidt To: Lorenzo Pieralisi Date: Fri, 04 Sep 2020 08:26:23 +1000 In-Reply-To: <20200903110844.GB11284@e121166-lin.cambridge.arm.com> References: <20200831151827.pumm2p54fyj7fz5s@amazon.com> <20200902113207.GA27676@e121166-lin.cambridge.arm.com> <20200902142922.xc4x6m33unkzewuh@amazon.com> <20200902164702.GA30611@e121166-lin.cambridge.arm.com> <20200903110844.GB11284@e121166-lin.cambridge.arm.com> X-Mailer: Evolution 3.28.5-0ubuntu0.18.04.2 Mime-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200903_182646_868611_D499AD8C X-CRM114-Status: GOOD ( 16.96 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-pci@vger.kernel.org, Bjorn Helgaas , catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, Clint Sbisa Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Thu, 2020-09-03 at 12:08 +0100, Lorenzo Pieralisi wrote: > "Additional Guidance on the Prefetchable Bit in Memory Space BARs" > > I read it 100 times and I still have no idea how it can be > implemented, > it sorts of acknowledges that read side-effects memory can be marked > as a prefetchable BAR *if* the system meets some criteria. > > As if endpoint designers knew the system where their endpoint is > plugged into (+ bit (3) in a BAR is read-only). > > I think that that implementation note must be removed from the > specifications - if anyone dares to follow it this whole > WC resource mapping can trigger trouble. Ah that one ! Yes you are right its completely broken. This part of the spec aims at working around the fact that bridges only have 64-bit prefetchable windows, so anything non-pref has to go below a 32-bit bridge window (effectively making most 64-bit non-pref BARs a pointless waste of silicon). The right fix of course would have been to create a new type of bridge window. But PCI... If you're going to mess around with the SIG, I would suggest that a better approach short of the above would be to allow system software to put 64-bit non-pref BARs below bridge pref windows on PCIe (provided the various otehr restrictions in that note are honored such as bridges not prefetching) and leave it at that. (Unless they already do somewhere else, I forgot ...). This should be sufficient to address the space concern without killing the meaning of the prefetchable bit. As for enabling the _wc files in sysfs, well, you need some serious priviledge to be able to access them, so I don't see a big issue allowing them to exist when "prefetchable" is set regardless of that rule. The Mellanox case might be different. Cheers, Ben. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel