From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.9 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0850C2B9F4 for ; Sat, 19 Jun 2021 16:36:59 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id ADED6611AC for ; Sat, 19 Jun 2021 16:36:59 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org ADED6611AC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:Subject:Message-ID:Date:From: In-Reply-To:References:MIME-Version:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=05zbON1z3DgSQYiYDD6k6Cb7PF3D6lbmjEjlBN1zFJE=; b=YakIfiZHuWbcSH aeiw4KvusYlIxtAswwduEeR/hUXC4InHpYYPxoRNuUB5ZZiSAPIaDGFB45F7lCoOta4OnzRToE2kN 2QQsawatcHCANrrX1KwhCQt6Ziv4cQHcvnvJlTzWvaPMio9S+EnsmpDacNWOKzSjIn7gbiZbk4tDM RLtnIxUdQTBiKha/uO3OE42CDWb9RLhupxhqKA/anQ6nrqt9iwwD3+//pGTE7P1F2UkWHG+Kq9EXD YX8LSrwbYjIbq/yrgLRBAC3IProMrrVsJNxWnQYD8jAxk8y3F5WtUNrI08TAGyQHYu+lmYmsaJYEs TaIn6k1M5RxVrAtM5f0w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1ludwG-00HIfe-5a; Sat, 19 Jun 2021 16:35:20 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1ludwB-00HIfA-KD for linux-arm-kernel@lists.infradead.org; Sat, 19 Jun 2021 16:35:17 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1624120512; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=tYyzt4cNti3kmXkoi7E1mYGxs62FMf8+WgvjuaIB/pc=; b=c0BtxCCEIxsYpTSdEkxRoLID0k0UF3PmiWTjwzQb9JQNDlQbL6wenz16DHOMMtoNhvApzy CitCRhWBvvrXKJGX2QBc/jf+qIVT06wKCvGuwPvSIpzxf7WWkKZTLjxQHQKCL9V5R83tdQ 2EkPzbExGx9CRr74sLKTB9oQ99oNFYw= Received: from mail-vs1-f72.google.com (mail-vs1-f72.google.com [209.85.217.72]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-590-iPodsgJeMHqWX-nFymO23A-1; Sat, 19 Jun 2021 12:35:08 -0400 X-MC-Unique: iPodsgJeMHqWX-nFymO23A-1 Received: by mail-vs1-f72.google.com with SMTP id v15-20020a67c00f0000b029023607a23f3dso2561996vsi.10 for ; Sat, 19 Jun 2021 09:35:08 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=tYyzt4cNti3kmXkoi7E1mYGxs62FMf8+WgvjuaIB/pc=; b=andYSyEA+4WRuWHrktmmMoQlA+GIpUxX8Rjc5ThBV1hFOiwEhVEwsL1QdQ0UmjvifG sbLiJd4M6omjrozIDK694fTQm1LjVMtR6hwMHg45ODmK3wR7ADanUzvynuQ9GToBew1r inxb8nit5smQJgs2OCkQ72qcQ4DuypcUHnWwz/xyo6NHpMlVU6zSdswHTknTN2KQ6gKg SmcnGsOMgVG/G4HKpY1OnGSk9jYO4JZZdGT1snuwqYgbJjqXwhIQUdWD2E8M6JBUcrWw bokmRxoU4YOrW2A3wArtdXrLknZcPqUZihIbFVUzMb0a72Uds7ZN241CtCl8oNcl6XHN +sKA== X-Gm-Message-State: AOAM532oX7gDjd8SBvii/tpXj7DpRCGcAy1czmMcn+fjIh90jpi2oARu ytJTxDopZW8c5xMN6UaUcfVhq7SOOC9yUERWHJLgcusX3jUpANFgMdGAg+Go+TRTf5UVmzKSwVU 3mTTieLzusXCnXKAmrlW5to5E+G5vsllCDbgxaCF2pW9vU5MxjVg= X-Received: by 2002:a9f:3743:: with SMTP id a3mr17073585uae.92.1624120508224; Sat, 19 Jun 2021 09:35:08 -0700 (PDT) X-Google-Smtp-Source: ABdhPJz7tsnZ19MELoFuTC7GgQpSNKhNWpByQDwABidkB3EP/eqY0EW9ZaDVMwk8IdppZOrEep7tC0lXUZAQFnyRFoE= X-Received: by 2002:a9f:3743:: with SMTP id a3mr17073574uae.92.1624120508072; Sat, 19 Jun 2021 09:35:08 -0700 (PDT) MIME-Version: 1.0 References: <20210105045735.1709825-1-jeremy.linton@arm.com> <20210107181416.GA3536@willie-the-truck> <56375cd8-8e11-aba6-9e11-1e0ec546e423@jonmasters.org> <20210108103216.GA17931@e121166-lin.cambridge.arm.com> <20210122194829.GE25471@willie-the-truck> <20210126225351.GA30941@willie-the-truck> <20210325131231.GA18590@e121166-lin.cambridge.arm.com> <20210616173646.GA1840163@nvidia.com> <20210618140554.GD1002214@nvidia.com> In-Reply-To: <20210618140554.GD1002214@nvidia.com> From: Jon Masters Date: Sat, 19 Jun 2021 12:34:57 -0400 Message-ID: Subject: Re: [PATCH] arm64: PCI: Enable SMC conduit To: Jason Gunthorpe Cc: Lorenzo Pieralisi , Will Deacon , Vikram Sethi , Vidya Sagar , Thierry Reding , Jon Masters , Jeremy Linton , Mark Rutland , linux-pci@vger.kernel.org, Sudeep Holla , Linux Kernel Mailing List , Catalin Marinas , Bjorn Helgaas , linux-arm-kernel@lists.infradead.org, Eric Brower , Grant.Likely@arm.com Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jcm@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210619_093515_895079_773624EC X-CRM114-Status: GOOD ( 43.25 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi Jason, On Fri, Jun 18, 2021 at 10:06 AM Jason Gunthorpe wrote: > > On Fri, Jun 18, 2021 at 09:21:54AM -0400, Jon Masters wrote: > > Hi Jason, > > On Wed, Jun 16, 2021 at 1:38 PM Jason Gunthorpe <[1]jgg@nvidia.com> > > wrote: > > > > On Thu, Mar 25, 2021 at 01:12:31PM +0000, Lorenzo Pieralisi wrote: > > However, in modern server type systems the PCI config space is often > > a > > software fiction being created by firmware throughout the PCI > > space. This has become necessary as the config space has exploded in > > size and complexity and PCI devices themselves have become very, > > very > > complicated. Not just the config space of single devices, but even > > bridges and topology are SW created in some cases. > > HW that is doing this is already trapping the config cycles somehow, > > presumably with some very ugly way like x86's SMM. Allowing a > > designed > > in way to inject software into the config space cycles does sound a > > lot cleaner and better to me. > > > > This is not required. SMM is terrible, indeed. But we don't have to > > relive it in Arm just because that's [EL3] the easy place to shove > > things :) > > "This is not required"? What does that mean? It's not required to implement platform hacks in SMM-like EL3. The correct place to do this kind of thing is behind the scenes in a platform microcontroller (note that I do not necessarily mean Arm's SCP approach, you can do much better than that). > > For instance it may solve other pain points if ARM systems had a > > cheap > > way to emulate up a "PCI device" to wrapper around some IP blob on > > chip. The x86 world has really driven this approach where everything > > on SOC is PCI discoverable, and it does seem to work well. > > IMHO SW emulation of config space is an important ingredient to do > > this. > > > > There are certainly ways to build PCI configuration space in a > > programmable way that does not require software trapping into > > MM. > > Can you elaborate on what you'd like to see here? Where do you want to > put the software then? There are places other than EL3 where this should live. It should not involve the AP at all in a correct configuration. It should (only) appear to be done in hardware, but where you do it is up to an implementation. Doing it correctly also accounts for others accessing configuration space simultaneously. You don't want to have to stop the world, or break PCI ordering semantics on access. There is a right way (hardware) to do this, and a wrong way (EL3 hacks). But I'll leave folks to figure out how to implement it. There are several possible approaches to do this. > > I strongly agree with the value of an industry standard approach > > to this in hardware, particularly if the PCIe vendors would offer > > this as IP. In a perfect world, ECAM would simply be an > > abstraction and never directly map to fixed hardware, thus one > > could correct defects in behavior in the field. I believe on the > > x86 side of the house, there is some interesting trapping support > > in the LPC/IOH already and this is absolutely what Arm should be > > doing. > > AFAIK x86 has HW that traps the read/writes to the ECAM and can > trigger a FW flow to emulate them, maybe in SMM, I don't know the > details. It ceratinly used to be like this when SMM could trap the > config space io read/write registers. They trap to something that isn't in SMM, but it is in firmware. That is the correct (in my opinion) approach to this. It's one time where I'm going to say that all the Arm vendors should be doing what Intel is doing in their implementation today. > Is that what you want to see for ARM? Is that better than a SMC? Yes, because you preserve perfect ECAM semantics and correct it behind the scenes. That's what people should be building. > That is alot of special magic hardware to avoid a SMC call... And it's the correct way to do it. Either that, or get ECAM perfect up front and do pre-si testing under emulation to confirm. Jon. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel