From: Mike Snitzer <snitzer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> To: Jens Axboe <axboe-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org> Cc: hch-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org, linux-raid-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-xfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, darrick.wong-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org, linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, colyli-l3A5Bk7waGM@public.gmane.org, linux-block-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, clm-b10kYP2dOMg@public.gmane.org, neilb-IBi9RG/b67k@public.gmane.org, bacik-b10kYP2dOMg@public.gmane.org, Kent Overstreet <kent.overstreet-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>, linux-btrfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, drbd-dev-cunTk1MwBs8qoQakbn7OcQ@public.gmane.org Subject: Re: [PATCH 00/13] convert block layer to bioset_init()/mempool_init() Date: Mon, 21 May 2018 11:04:40 -0400 [thread overview] Message-ID: <20180521150439.GA19379@redhat.com> (raw) In-Reply-To: <4b343aef-e11c-73ba-1d88-7e73ca838cad-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org> On Mon, May 21 2018 at 10:52am -0400, Jens Axboe <axboe-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org> wrote: > On 5/21/18 8:47 AM, Mike Snitzer wrote: > > On Mon, May 21 2018 at 10:36am -0400, > > Jens Axboe <axboe-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org> wrote: > > > >> On 5/21/18 8:31 AM, Mike Snitzer wrote: > >>> On Mon, May 21 2018 at 10:19am -0400, > >>> Jens Axboe <axboe-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org> wrote: > >>> > >>>> On 5/21/18 8:03 AM, Mike Snitzer wrote: > >>>>> On Sun, May 20 2018 at 6:25pm -0400, > >>>>> Kent Overstreet <kent.overstreet-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > >>>>> > >>>>>> Jens - this series does the rest of the conversions that Christoph wanted, and > >>>>>> drops bioset_create(). > >>>>>> > >>>>>> Only lightly tested, but the changes are pretty mechanical. Based on your > >>>>>> for-next tree. > >>>>> > >>>>> By switching 'mempool_t *' to 'mempool_t' and 'bio_set *' to 'bio_set' > >>>>> you've altered the alignment of members in data structures. So I'll > >>>>> need to audit all the data structures you've modified in DM. > >>>>> > >>>>> Could we get the backstory on _why_ you're making this change? > >>>>> Would go a long way to helping me appreciate why this is a good use of > >>>>> anyone's time. > >>>> > >>>> Yeah, it's in the first series, it gets rid of a pointer indirection. > >>> > >>> "Allows mempools to be embedded in other structs, getting rid of a > >>> pointer indirection from allocation fastpaths." > >>> > >>> So this is about using contiguous memory or avoiding partial allocation > >>> failure? Or both? > >>> > >>> Or more to it? Just trying to fully appreciate the theory behind the > >>> perceived associated benefit. > >> > >> It's about avoiding a pointer indirection. Instead of having to > >> follow a pointer to get to that struct, it's simple offset math off > >> your main structure. > >> > >>> I do think the increased risk of these embedded bio_set and mempool_t > >>> themselves crossing cachelines, or struct members that follow them doing > >>> so, really detracts from these types of changes. > >> > >> Definitely something to look out for, though most of them should be > >> per-dev structures and not in-flight structures. That makes it a bit > >> less sensitive. But can't hurt to audit the layouts and adjust if > >> necessary. This is why it's posted for review :-) > > > > This isn't something that is easily caught upfront. Yes we can all be > > busy little beavers with pahole to audit alignment. But chances are > > most people won't do it. > > > > Reality is there is potential for a regression due to false sharing to > > creep in if a hot struct member suddenly starts straddling a cacheline. > > That type of NUMA performance killer is pretty insidious and somewhat > > tedious to hunt down even when looking for it with specialized tools: > > https://joemario.github.io/blog/2016/09/01/c2c-blog/ > > IMHO you're making a big deal out of something that should not be. I raised an issue that had seemingly not been considered at all. Not making a big deal. Raising it for others' benefit. > If the dm bits are that sensitive and cache line honed to perfection > already due to previous regressions in that area, then it might > not be a bad idea to have some compile checks for false cacheline > sharing between sensitive members, or spilling of a sub-struct > into multiple cachelines. > > It's not like this was pushed behind your back. It's posted for > review. It's quite possible the net change is a win for dm. Let's > focus on getting it reviewed, rather than pontificate on what > could potentially go all wrong with this. Why are you making this personal? Or purely about DM? I'm merely pointing out this change isn't something that can be given a quick blanket "looks good". Mike
WARNING: multiple messages have this Message-ID (diff)
From: Mike Snitzer <snitzer@redhat.com> To: Jens Axboe <axboe@kernel.dk> Cc: Kent Overstreet <kent.overstreet@gmail.com>, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, hch@infradead.org, colyli@suse.de, darrick.wong@oracle.com, clm@fb.com, bacik@fb.com, linux-xfs@vger.kernel.org, drbd-dev@lists.linbit.com, linux-btrfs@vger.kernel.org, linux-raid@vger.kernel.org, neilb@suse.com Subject: Re: [PATCH 00/13] convert block layer to bioset_init()/mempool_init() Date: Mon, 21 May 2018 11:04:40 -0400 [thread overview] Message-ID: <20180521150439.GA19379@redhat.com> (raw) In-Reply-To: <4b343aef-e11c-73ba-1d88-7e73ca838cad@kernel.dk> On Mon, May 21 2018 at 10:52am -0400, Jens Axboe <axboe@kernel.dk> wrote: > On 5/21/18 8:47 AM, Mike Snitzer wrote: > > On Mon, May 21 2018 at 10:36am -0400, > > Jens Axboe <axboe@kernel.dk> wrote: > > > >> On 5/21/18 8:31 AM, Mike Snitzer wrote: > >>> On Mon, May 21 2018 at 10:19am -0400, > >>> Jens Axboe <axboe@kernel.dk> wrote: > >>> > >>>> On 5/21/18 8:03 AM, Mike Snitzer wrote: > >>>>> On Sun, May 20 2018 at 6:25pm -0400, > >>>>> Kent Overstreet <kent.overstreet@gmail.com> wrote: > >>>>> > >>>>>> Jens - this series does the rest of the conversions that Christoph wanted, and > >>>>>> drops bioset_create(). > >>>>>> > >>>>>> Only lightly tested, but the changes are pretty mechanical. Based on your > >>>>>> for-next tree. > >>>>> > >>>>> By switching 'mempool_t *' to 'mempool_t' and 'bio_set *' to 'bio_set' > >>>>> you've altered the alignment of members in data structures. So I'll > >>>>> need to audit all the data structures you've modified in DM. > >>>>> > >>>>> Could we get the backstory on _why_ you're making this change? > >>>>> Would go a long way to helping me appreciate why this is a good use of > >>>>> anyone's time. > >>>> > >>>> Yeah, it's in the first series, it gets rid of a pointer indirection. > >>> > >>> "Allows mempools to be embedded in other structs, getting rid of a > >>> pointer indirection from allocation fastpaths." > >>> > >>> So this is about using contiguous memory or avoiding partial allocation > >>> failure? Or both? > >>> > >>> Or more to it? Just trying to fully appreciate the theory behind the > >>> perceived associated benefit. > >> > >> It's about avoiding a pointer indirection. Instead of having to > >> follow a pointer to get to that struct, it's simple offset math off > >> your main structure. > >> > >>> I do think the increased risk of these embedded bio_set and mempool_t > >>> themselves crossing cachelines, or struct members that follow them doing > >>> so, really detracts from these types of changes. > >> > >> Definitely something to look out for, though most of them should be > >> per-dev structures and not in-flight structures. That makes it a bit > >> less sensitive. But can't hurt to audit the layouts and adjust if > >> necessary. This is why it's posted for review :-) > > > > This isn't something that is easily caught upfront. Yes we can all be > > busy little beavers with pahole to audit alignment. But chances are > > most people won't do it. > > > > Reality is there is potential for a regression due to false sharing to > > creep in if a hot struct member suddenly starts straddling a cacheline. > > That type of NUMA performance killer is pretty insidious and somewhat > > tedious to hunt down even when looking for it with specialized tools: > > https://joemario.github.io/blog/2016/09/01/c2c-blog/ > > IMHO you're making a big deal out of something that should not be. I raised an issue that had seemingly not been considered at all. Not making a big deal. Raising it for others' benefit. > If the dm bits are that sensitive and cache line honed to perfection > already due to previous regressions in that area, then it might > not be a bad idea to have some compile checks for false cacheline > sharing between sensitive members, or spilling of a sub-struct > into multiple cachelines. > > It's not like this was pushed behind your back. It's posted for > review. It's quite possible the net change is a win for dm. Let's > focus on getting it reviewed, rather than pontificate on what > could potentially go all wrong with this. Why are you making this personal? Or purely about DM? I'm merely pointing out this change isn't something that can be given a quick blanket "looks good". Mike
next prev parent reply other threads:[~2018-05-21 15:04 UTC|newest] Thread overview: 73+ messages / expand[flat|nested] mbox.gz Atom feed top 2018-05-20 22:25 [PATCH 00/13] convert block layer to bioset_init()/mempool_init() Kent Overstreet 2018-05-20 22:25 ` [PATCH 01/12] block: convert bounce, q->bio_split " Kent Overstreet 2018-05-22 10:08 ` Christoph Hellwig 2018-05-20 22:25 ` [PATCH 02/12] drbd: convert " Kent Overstreet 2018-05-20 22:25 ` [PATCH 03/12] pktcdvd: " Kent Overstreet 2018-05-20 22:25 ` [PATCH 04/12] lightnvm: " Kent Overstreet 2018-05-22 10:10 ` Javier Gonzalez 2018-05-20 22:25 ` [PATCH 05/12] bcache: " Kent Overstreet 2018-05-21 3:58 ` Coly Li 2018-05-20 22:25 ` [PATCH 06/12] md: " Kent Overstreet 2018-06-01 10:51 ` Arnd Bergmann 2018-05-20 22:25 ` [PATCH 07/12] dm: " Kent Overstreet [not found] ` <20180520222558.7053-8-kent.overstreet-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> 2018-05-30 19:27 ` Mike Snitzer 2018-05-30 19:27 ` Mike Snitzer 2018-05-20 22:25 ` [PATCH 08/12] target: " Kent Overstreet 2018-05-22 10:09 ` Christoph Hellwig 2018-05-20 22:25 ` [PATCH 09/12] fs: convert block_dev.c to bioset_init() Kent Overstreet 2018-05-22 10:09 ` Christoph Hellwig 2018-05-20 22:25 ` [PATCH 10/12] btrfs: convert to bioset_init()/mempool_init() Kent Overstreet 2018-05-30 21:30 ` Chris Mason 2018-05-30 21:30 ` Chris Mason 2018-05-20 22:25 ` [PATCH 11/12] xfs: " Kent Overstreet 2018-05-21 18:39 ` Darrick J. Wong [not found] ` <20180520222558.7053-12-kent.overstreet-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> 2018-05-22 10:10 ` Christoph Hellwig 2018-05-22 10:10 ` Christoph Hellwig 2018-05-20 22:25 ` [PATCH 12/12] block: Drop bioset_create() Kent Overstreet 2018-05-22 10:10 ` Christoph Hellwig 2018-05-20 23:08 ` [PATCH 00/13] convert block layer to bioset_init()/mempool_init() NeilBrown 2018-05-20 23:08 ` NeilBrown 2018-05-20 23:11 ` Kent Overstreet [not found] ` <20180520222558.7053-1-kent.overstreet-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> 2018-05-21 14:03 ` Mike Snitzer 2018-05-21 14:03 ` Mike Snitzer 2018-05-21 14:19 ` Jens Axboe [not found] ` <686d7df6-c7d1-48a6-b7ff-48dc8aff6a62-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org> 2018-05-21 14:31 ` Mike Snitzer 2018-05-21 14:31 ` Mike Snitzer 2018-05-21 14:36 ` Jens Axboe [not found] ` <2bbeeb1a-8b99-b06a-eb9b-eb8523c16460-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org> 2018-05-21 14:47 ` Mike Snitzer 2018-05-21 14:47 ` Mike Snitzer [not found] ` <20180521144703.GA19303-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> 2018-05-21 14:52 ` Jens Axboe 2018-05-21 14:52 ` Jens Axboe [not found] ` <4b343aef-e11c-73ba-1d88-7e73ca838cad-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org> 2018-05-21 15:04 ` Mike Snitzer [this message] 2018-05-21 15:04 ` Mike Snitzer [not found] ` <20180521150439.GA19379-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> 2018-05-21 15:09 ` Jens Axboe 2018-05-21 15:09 ` Jens Axboe [not found] ` <61e30dcf-a01c-f47d-087a-12930caf9aef-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org> 2018-05-21 15:18 ` Mike Snitzer 2018-05-21 15:18 ` Mike Snitzer [not found] ` <20180521151817.GA19454-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> 2018-05-21 15:36 ` Jens Axboe 2018-05-21 15:36 ` Jens Axboe [not found] ` <d01a150a-7752-f6ce-78f2-17a65c1e6fa5-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org> 2018-05-21 16:09 ` Mike Snitzer 2018-05-21 16:09 ` Mike Snitzer [not found] ` <20180521160907.GA19553-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> 2018-05-21 16:20 ` Jens Axboe 2018-05-21 16:20 ` Jens Axboe [not found] ` <f9e3714c-b7c9-d5f6-4018-2a87dd5babb2-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org> 2018-05-30 13:36 ` Mike Snitzer 2018-05-30 13:36 ` Mike Snitzer [not found] ` <20180530133629.GC5157-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> 2018-05-30 18:55 ` Jens Axboe 2018-05-30 18:55 ` Jens Axboe 2018-05-30 19:34 ` Kent Overstreet 2018-05-30 19:36 ` Jens Axboe 2018-05-30 19:36 ` Jens Axboe 2018-05-30 19:37 ` Mike Snitzer [not found] ` <20180530193707.GB6568-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> 2018-05-30 19:38 ` Jens Axboe 2018-05-30 19:38 ` Jens Axboe 2018-05-21 17:37 ` Kent Overstreet 2018-05-21 18:24 ` Mike Snitzer 2018-05-21 18:24 ` Mike Snitzer 2018-05-21 23:38 ` Kent Overstreet 2018-05-22 6:41 ` Christoph Hellwig [not found] ` <20180522064118.GA18704-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> 2018-05-22 19:09 ` Mike Snitzer 2018-05-22 19:09 ` Mike Snitzer 2018-05-21 15:12 ` David Sterba 2018-05-21 15:18 ` Jens Axboe 2018-05-21 14:20 ` Jens Axboe 2018-05-30 22:24 ` Jens Axboe
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20180521150439.GA19379@redhat.com \ --to=snitzer-h+wxahxf7alqt0dzr+alfa@public.gmane.org \ --cc=axboe-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org \ --cc=bacik-b10kYP2dOMg@public.gmane.org \ --cc=clm-b10kYP2dOMg@public.gmane.org \ --cc=colyli-l3A5Bk7waGM@public.gmane.org \ --cc=darrick.wong-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org \ --cc=drbd-dev-cunTk1MwBs8qoQakbn7OcQ@public.gmane.org \ --cc=hch-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org \ --cc=kent.overstreet-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org \ --cc=linux-block-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \ --cc=linux-btrfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \ --cc=linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \ --cc=linux-raid-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \ --cc=linux-xfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \ --cc=neilb-IBi9RG/b67k@public.gmane.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.