From: Rusty Russell <rusty@rustcorp.com.au> To: "Michael S. Tsirkin" <mst@redhat.com> Cc: netdev@vger.kernel.org, virtualization@lists.linux-foundation.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, mingo@elte.hu, linux-mm@kvack.org, akpm@linux-foundation.org, hpa@zytor.com, gregory.haskins@gmail.com, s.hetze@linux-ag.com, Daniel Walker <dwalker@fifo99.com>, Eric Dumazet <eric.dumazet@gmail.com> Subject: Re: virtio: put last_used and last_avail index into ring itself. Date: Mon, 10 May 2010 12:41:56 +0930 [thread overview] Message-ID: <201005101241.57237.rusty@rustcorp.com.au> (raw) In-Reply-To: <20100509085733.GD16775@redhat.com> On Sun, 9 May 2010 06:27:33 pm Michael S. Tsirkin wrote: > On Fri, May 07, 2010 at 12:35:39PM +0930, Rusty Russell wrote: > > Then there's padding to page boundary. That puts us on a cacheline again > > for the used ring; also 2 bytes per entry. > > > > Hmm, is used ring really 2 bytes per entry? Err, no, I am an idiot. > /* u32 is used here for ids for padding reasons. */ > struct vring_used_elem { > /* Index of start of used descriptor chain. */ > __u32 id; > /* Total length of the descriptor chain which was used (written to) */ > __u32 len; > }; > > struct vring_used { > __u16 flags; > __u16 idx; > struct vring_used_elem ring[]; > }; OK, now I get it. Sorry, I was focussed on the avail ring. > I thought that used ring has 8 bytes per entry, and that struct > vring_used is aligned at page boundary, this > would mean that ring element is at offset 4 bytes from page boundary. > Thus with cacheline size 128 bytes, each 4th element crosses > a cacheline boundary. If we had a 4 byte padding after idx, each > used element would always be completely within a single cacheline. I think the numbers are: every 16th entry hits two cachelines. So currently the first 15 entries are "free" (assuming we hit the idx cacheline anyway), then 1 in 16 cost 2 cachelines. That makes the aligned version win when N > 240. But, we access the array linearly. So the extra cacheline cost is in fact amortized. I doubt it could be measured, but maybe vring_get_buf() should prefetch? While you're there, we could use an & rather than a mod on the calculation, which may actually be measurable :) Cheers, Rusty.
WARNING: multiple messages have this Message-ID (diff)
From: Rusty Russell <rusty@rustcorp.com.au> To: "Michael S. Tsirkin" <mst@redhat.com> Cc: netdev@vger.kernel.org, virtualization@lists.linux-foundation.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, mingo@elte.hu, linux-mm@kvack.org, akpm@linux-foundation.org, hpa@zytor.com, gregory.haskins@gmail.com, s.hetze@linux-ag.com, Daniel Walker <dwalker@fifo99.com>, Eric Dumazet <eric.dumazet@gmail.com> Subject: Re: virtio: put last_used and last_avail index into ring itself. Date: Mon, 10 May 2010 12:41:56 +0930 [thread overview] Message-ID: <201005101241.57237.rusty@rustcorp.com.au> (raw) In-Reply-To: <20100509085733.GD16775@redhat.com> On Sun, 9 May 2010 06:27:33 pm Michael S. Tsirkin wrote: > On Fri, May 07, 2010 at 12:35:39PM +0930, Rusty Russell wrote: > > Then there's padding to page boundary. That puts us on a cacheline again > > for the used ring; also 2 bytes per entry. > > > > Hmm, is used ring really 2 bytes per entry? Err, no, I am an idiot. > /* u32 is used here for ids for padding reasons. */ > struct vring_used_elem { > /* Index of start of used descriptor chain. */ > __u32 id; > /* Total length of the descriptor chain which was used (written to) */ > __u32 len; > }; > > struct vring_used { > __u16 flags; > __u16 idx; > struct vring_used_elem ring[]; > }; OK, now I get it. Sorry, I was focussed on the avail ring. > I thought that used ring has 8 bytes per entry, and that struct > vring_used is aligned at page boundary, this > would mean that ring element is at offset 4 bytes from page boundary. > Thus with cacheline size 128 bytes, each 4th element crosses > a cacheline boundary. If we had a 4 byte padding after idx, each > used element would always be completely within a single cacheline. I think the numbers are: every 16th entry hits two cachelines. So currently the first 15 entries are "free" (assuming we hit the idx cacheline anyway), then 1 in 16 cost 2 cachelines. That makes the aligned version win when N > 240. But, we access the array linearly. So the extra cacheline cost is in fact amortized. I doubt it could be measured, but maybe vring_get_buf() should prefetch? While you're there, we could use an & rather than a mod on the calculation, which may actually be measurable :) Cheers, Rusty. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2010-05-10 8:54 UTC|newest] Thread overview: 51+ messages / expand[flat|nested] mbox.gz Atom feed top [not found] <cover.1257349249.git.mst@redhat.com> 2009-11-04 15:55 ` [PATCHv8 1/3] tun: export underlying socket Michael S. Tsirkin 2009-11-04 15:55 ` Michael S. Tsirkin 2009-11-04 15:55 ` Michael S. Tsirkin 2009-11-04 15:55 ` Michael S. Tsirkin 2009-11-04 15:55 ` [PATCHv8 2/3] mm: export use_mm/unuse_mm to modules Michael S. Tsirkin 2009-11-04 15:55 ` Michael S. Tsirkin 2009-11-04 15:55 ` Michael S. Tsirkin 2009-11-04 15:55 ` Michael S. Tsirkin 2009-11-04 15:57 ` [PATCHv8 3/3] vhost_net: a kernel-level virtio server Michael S. Tsirkin 2009-11-04 15:57 ` Michael S. Tsirkin 2009-11-04 15:57 ` Michael S. Tsirkin 2009-11-04 15:57 ` Michael S. Tsirkin 2009-11-06 4:59 ` Rusty Russell 2009-11-06 4:59 ` Rusty Russell 2009-11-06 4:59 ` Rusty Russell 2009-11-08 11:35 ` Michael S. Tsirkin 2009-11-08 11:35 ` Michael S. Tsirkin 2009-11-09 6:17 ` Rusty Russell 2009-11-09 6:17 ` Rusty Russell 2009-11-09 6:17 ` Rusty Russell 2009-11-09 7:10 ` Michael S. Tsirkin 2009-11-09 7:10 ` Michael S. Tsirkin 2009-11-10 1:08 ` Rusty Russell 2009-11-10 1:08 ` Rusty Russell 2009-11-10 1:08 ` Rusty Russell 2009-11-09 7:10 ` Michael S. Tsirkin 2009-11-09 7:20 ` Michael S. Tsirkin 2009-11-09 7:20 ` Michael S. Tsirkin 2009-11-09 7:20 ` Michael S. Tsirkin 2009-11-09 11:55 ` Michael S. Tsirkin 2009-11-09 11:55 ` Michael S. Tsirkin 2009-11-09 11:55 ` Michael S. Tsirkin 2010-05-04 18:22 ` virtio: put last_used and last_avail index into ring itself Michael S. Tsirkin 2010-05-04 18:22 ` Michael S. Tsirkin 2010-05-06 0:52 ` Rusty Russell 2010-05-06 0:52 ` Rusty Russell 2010-05-06 0:52 ` Rusty Russell 2010-05-06 6:27 ` Michael S. Tsirkin 2010-05-06 6:27 ` Michael S. Tsirkin 2010-05-06 6:27 ` Michael S. Tsirkin 2010-05-07 3:05 ` Rusty Russell 2010-05-07 3:05 ` Rusty Russell 2010-05-07 3:05 ` Rusty Russell 2010-05-09 8:57 ` Michael S. Tsirkin 2010-05-09 8:57 ` Michael S. Tsirkin 2010-05-09 8:57 ` Michael S. Tsirkin 2010-05-10 3:11 ` Rusty Russell [this message] 2010-05-10 3:11 ` Rusty Russell 2010-05-10 3:11 ` Rusty Russell 2010-05-04 18:22 ` Michael S. Tsirkin 2009-11-08 11:35 ` [PATCHv8 3/3] vhost_net: a kernel-level virtio server Michael S. Tsirkin
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=201005101241.57237.rusty@rustcorp.com.au \ --to=rusty@rustcorp.com.au \ --cc=akpm@linux-foundation.org \ --cc=dwalker@fifo99.com \ --cc=eric.dumazet@gmail.com \ --cc=gregory.haskins@gmail.com \ --cc=hpa@zytor.com \ --cc=kvm@vger.kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=mingo@elte.hu \ --cc=mst@redhat.com \ --cc=netdev@vger.kernel.org \ --cc=s.hetze@linux-ag.com \ --cc=virtualization@lists.linux-foundation.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.