Re: [RFC PATCH V2 0/5] vhost: accelerate metadata access through vmap()

From: James Bottomley <James.Bottomley@HansenPartnership.com>
To: Andrea Arcangeli <aarcange@redhat.com>
Cc: "Michael S. Tsirkin" <mst@redhat.com>,
	Jason Wang <jasowang@redhat.com>,
	David Miller <davem@davemloft.net>,
	hch@infradead.org, kvm@vger.kernel.org,
	virtualization@lists.linux-foundation.org,
	netdev@vger.kernel.org, linux-kernel@vger.kernel.org,
	peterx@redhat.com, linux-mm@kvack.org,
	linux-arm-kernel@lists.infradead.org,
	linux-parisc@vger.kernel.org
Subject: Re: [RFC PATCH V2 0/5] vhost: accelerate metadata access through vmap()
Date: Tue, 12 Mar 2019 13:53:37 -0700	[thread overview]
Message-ID: <1552424017.14432.11.camel@HansenPartnership.com> (raw)
In-Reply-To: <20190312200450.GA25147@redhat.com>

On Tue, 2019-03-12 at 16:04 -0400, Andrea Arcangeli wrote:
> On Tue, Mar 12, 2019 at 08:46:50AM -0700, James Bottomley wrote:
> > On Tue, 2019-03-12 at 07:54 -0400, Michael S. Tsirkin wrote:
> > > On Tue, Mar 12, 2019 at 03:17:00PM +0800, Jason Wang wrote:
> > > > 
> > > > On 2019/3/12 ä¸Šåˆ11:52, Michael S. Tsirkin wrote:
> > > > > On Tue, Mar 12, 2019 at 10:59:09AM +0800, Jason Wang wrote:
> > 
> > [...]
> > > > At least for -stable, we need the flush?
> > > > 
> > > > 
> > > > > Three atomic ops per bit is way to expensive.
> > > > 
> > > > 
> > > > Yes.
> > > > 
> > > > Thanks
> > > 
> > > See James's reply - I stand corrected we do kunmap so no need to
> > > flush.
> > 
> > Well, I said that's what we do on Parisc.  The cachetlb document
> > definitely says if you alter the data between kmap and kunmap you
> > are responsible for the flush.  It's just that flush_dcache_page()
> > is a no-op on x86 so they never remember to add it and since it
> > will crash parisc if you get it wrong we finally gave up trying to
> > make them.
> > 
> > But that's the point: it is a no-op on your favourite architecture
> > so it costs you nothing to add it.
> 
> Yes, the fact Parisc gave up and is doing it on kunmap is reasonable
> approach for Parisc, but it doesn't move the needle as far as vhost
> common code is concerned, because other archs don't flush any cache
> on kunmap.
> 
> So either all other archs give up trying to optimize, or vhost still
> has to call flush_dcache_page() after kunmap.

I've got to say: optimize what?  What code do we ever have in the
kernel that kmap's a page and then doesn't do anything with it? You can
guarantee that on kunmap the page is either referenced (needs
invalidating) or updated (needs flushing). The in-kernel use of kmap is
always

kmap
do something with the mapped page
kunmap

In a very short interval.  It seems just a simplification to make
kunmap do the flush if needed rather than try to have the users
remember.  The thing which makes this really simple is that on most
architectures flush and invalidate is the same operation.  If you
really want to optimize you can use the referenced and dirty bits on
the kmapped pte to tell you what operation to do, but if your flush is
your invalidate, you simply assume the data needs flushing on kunmap
without checking anything.

> Which means after we fix vhost to add the flush_dcache_page after
> kunmap, Parisc will get a double hit (but it also means Parisc was
> the only one of those archs needed explicit cache flushes, where
> vhost worked correctly so far.. so it kinds of proofs your point of
> giving up being the safe choice).

What double hit?  If there's no cache to flush then cache flush is a
no-op.  It's also a highly piplineable no-op because the CPU has the L1
cache within easy reach.  The only event when flush takes a large
amount time is if we actually have dirty data to write back to main
memory.

James