LKML Archive on lore.kernel.org
 help / color / Atom feed
From: Alex Williamson <alex.williamson@redhat.com>
To: Vikram Sethi <vsethi@nvidia.com>
Cc: Mark Kettenis <mark.kettenis@xs4all.nl>,
	Marc Zyngier <maz@kernel.org>,
	Shanker Donthineni <sdonthineni@nvidia.com>,
	"will@kernel.org" <will@kernel.org>,
	"catalin.marinas@arm.com" <catalin.marinas@arm.com>,
	"christoffer.dall@arm.com" <christoffer.dall@arm.com>,
	"linux-arm-kernel@lists.infradead.org" 
	<linux-arm-kernel@lists.infradead.org>,
	"kvmarm@lists.cs.columbia.edu" <kvmarm@lists.cs.columbia.edu>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	Jason Sequeira <jsequeira@nvidia.com>
Subject: Re: [RFC 1/2] vfio/pci: keep the prefetchable attribute of a BAR region in VMA
Date: Mon, 3 May 2021 08:44:32 -0600
Message-ID: <20210503084432.75e0126d@x1.home.shazbot.org> (raw)
In-Reply-To: <BL0PR12MB253296086906C4A850EC68E6BD5B9@BL0PR12MB2532.namprd12.prod.outlook.com>

On Mon, 3 May 2021 13:59:43 +0000
Vikram Sethi <vsethi@nvidia.com> wrote:

> > From: Mark Kettenis <mark.kettenis@xs4all.nl>  
> > > From: Marc Zyngier <maz@kernel.org>  
> 
> snip
> > > If, by enumerating the properties of Prefetchable, you can show that
> > > they are a strict superset of Normal_NC, I'm on board. I haven't seen
> > > such an enumeration so far.
> > >  
> snip
> > > Right, so we have made a small step in the direction of mapping
> > > "prefetchable" onto "Normal_NC", thanks for that. What about all the
> > > other properties (unaligned accesses, ordering, gathering)?  
> >   
> Regarding gathering/write combining, that is also allowed to prefetchable per PCI spec

As others have stated, gather/write combining itself is not well
defined.

> From 1.3.2.2 of 5/0 base spec:
> A PCI Express Endpoint requesting memory resources through a BAR must set the BAR's Prefetchable bit unless
> the range contains locations with read side-effects or locations in which the Function does not tolerate write
> merging.

"write merging"  This is a very specific thing, per PCI 3.0, 3.2.6:

  Byte Merging – occurs when a sequence of individual memory writes
  (bytes or words) are merged into a single DWORD.

The semantics suggest quadword support in addition to dword, but don't
require it.  Writes to bytes within a dword can be merged, but
duplicate writes cannot.

It seems like an extremely liberal application to suggest that this one
write semantic encompasses full write combining semantics, which itself
is not clearly defined.

> Further 7.5.1.2.1 says " A Function is permitted
> to mark a range as prefetchable if there are no side effects on reads, the Function returns all bytes on reads regardless of
> the byte enables, and host bridges can merge processor writes into this range139 without causing errors"
> 
> The "regardless of byte enables" suggests to me that unaligned is OK, as only 
> certain byte enables may be set, what do you think?
> 
> So to me prefetchable in PCIe spec allows for write combining, read without

Ironically here, the above PCI spec section defining write merging has
separate sections for "combining", "merging", and "collapsing".  Only
merging is indicated as a requirement for prefetchable resources.

> sideeffect (prefetch/speculative as long as uncached), and unaligned. Regarding
> ordering I didn't find a statement one way or other in PCIe prefetchable definition, but
> I think that goes beyond what PCIe says or doesn't say anyway since reordering can 
> also happen in the CPU, and since driver must be aware of correctness issues in its 
> producer/consumer models it will need the right barriers where they are required 
> for correctness anyway (required for the driver/userspace to work on host w/ ioremap_wc).

A lot of hand waving here, imo.  Thanks,

Alex


  reply index

Thread overview: 30+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-04-29 16:29 [RFC 0/2] [RFC] Honor PCI prefetchable attributes for a virtual machine on ARM64 Shanker Donthineni
2021-04-29 16:29 ` [RFC 1/2] vfio/pci: keep the prefetchable attribute of a BAR region in VMA Shanker Donthineni
2021-04-29 18:28   ` Alex Williamson
2021-04-29 19:14     ` Shanker R Donthineni
2021-04-29 19:46       ` Alex Williamson
2021-04-29 22:08         ` Vikram Sethi
2021-04-30 11:25         ` Shanker R Donthineni
     [not found]           ` <87czucngdc.wl-maz@kernel.org>
2021-04-30 13:07             ` Shanker R Donthineni
2021-04-30 14:58             ` Shanker R Donthineni
     [not found]               ` <878s4zokll.wl-maz@kernel.org>
2021-04-30 16:57                 ` Vikram Sethi
2021-05-01  9:30                   ` Marc Zyngier
2021-05-01 11:36                     ` Shanker R Donthineni
     [not found]                       ` <87czu8uowe.wl-maz@kernel.org>
2021-05-03 12:08                         ` Shanker R Donthineni
2021-05-02 17:56                     ` Vikram Sethi
2021-05-03 10:17                       ` Marc Zyngier
2021-05-03 13:35                         ` Mark Kettenis
2021-05-03 13:59                           ` Vikram Sethi
2021-05-03 14:44                             ` Alex Williamson [this message]
2021-05-03 22:03                               ` Vikram Sethi
2021-05-04  8:30                                 ` Will Deacon
2021-05-05 18:02                                   ` Catalin Marinas
2021-05-06  7:22                                     ` Christoph Hellwig
2021-05-08 16:33                                     ` Shanker R Donthineni
2021-06-02  9:37                                       ` Marc Zyngier
2021-05-04 18:03                                 ` Alex Williamson
2021-06-02  9:11                                   ` Marc Zyngier
2021-04-30  9:54   ` Lorenzo Pieralisi
2021-04-30 12:38     ` Jason Gunthorpe
2021-04-29 16:29 ` [RFC 2/2] KVM: arm64: Add write-combine support for stage-2 entries Shanker Donthineni
2021-05-03  7:01 ` [RFC 0/2] [RFC] Honor PCI prefetchable attributes for a virtual machine on ARM64 Christoph Hellwig

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210503084432.75e0126d@x1.home.shazbot.org \
    --to=alex.williamson@redhat.com \
    --cc=catalin.marinas@arm.com \
    --cc=christoffer.dall@arm.com \
    --cc=jsequeira@nvidia.com \
    --cc=kvm@vger.kernel.org \
    --cc=kvmarm@lists.cs.columbia.edu \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mark.kettenis@xs4all.nl \
    --cc=maz@kernel.org \
    --cc=sdonthineni@nvidia.com \
    --cc=vsethi@nvidia.com \
    --cc=will@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

LKML Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/lkml/0 lkml/git/0.git
	git clone --mirror https://lore.kernel.org/lkml/1 lkml/git/1.git
	git clone --mirror https://lore.kernel.org/lkml/2 lkml/git/2.git
	git clone --mirror https://lore.kernel.org/lkml/3 lkml/git/3.git
	git clone --mirror https://lore.kernel.org/lkml/4 lkml/git/4.git
	git clone --mirror https://lore.kernel.org/lkml/5 lkml/git/5.git
	git clone --mirror https://lore.kernel.org/lkml/6 lkml/git/6.git
	git clone --mirror https://lore.kernel.org/lkml/7 lkml/git/7.git
	git clone --mirror https://lore.kernel.org/lkml/8 lkml/git/8.git
	git clone --mirror https://lore.kernel.org/lkml/9 lkml/git/9.git
	git clone --mirror https://lore.kernel.org/lkml/10 lkml/git/10.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 lkml lkml/ https://lore.kernel.org/lkml \
		linux-kernel@vger.kernel.org
	public-inbox-index lkml

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-kernel


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git