RE: Bluestore different allocator performance Vs FileStore

[Sage Weil <sage@newdream.net>]

On Thu, 11 Aug 2016, Allen Samuels wrote:
> > -----Original Message-----
> > From: Sage Weil [mailto:sage@newdream.net]
> > Sent: Thursday, August 11, 2016 9:38 AM
> > To: Ramesh Chander <Ramesh.Chander@sandisk.com>
> > Cc: Allen Samuels <Allen.Samuels@sandisk.com>; Somnath Roy
> > <Somnath.Roy@sandisk.com>; ceph-devel <ceph-devel@vger.kernel.org>
> > Subject: RE: Bluestore different allocator performance Vs FileStore
> > 
> > On Thu, 11 Aug 2016, Ramesh Chander wrote:
> > > I think the free list does not initialize all keys at mkfs time, it
> > > does sets key that has some allocations.
> > >
> > > Rest keys are assumed to have 0's if key does not exist.
> > 
> > Right.. it's the region "allocated" to bluefs that is consuming the time.
> > 
> > > The bitmap allocator insert_free is done in group of free bits
> > > together(maybe more than bitmap freelist keys at a time).
> > 
> > I think Allen is asking whether we are doing lots of inserts within a single
> > rocksdb transaction, or lots of separate transactions.
> > 
> > FWIW, my guess is that increasing the size of the value (i.e., increasing
> > 
> > OPTION(bluestore_freelist_blocks_per_key, OPT_INT, 128)
> > 
> > ) will probably speed this up.
> 
> If your assumption (> Right.. it's the region "allocated" to bluefs that 
> is consuming the time) is correct, then I don't understand why this 
> parameter has any effect on the problem.
> 
> Aren't we reading BlueFS extents and setting them in the 
> BitMapAllocator? That doesn't care about the chunking of bitmap bits 
> into KV keys.

I think this is something different.  During mkfs we take ~2% (or 
somethign like that) of the block device, mark it 'allocated' (from the 
bluestore freelist's perspective) and give it to bluefs.  On a large 
device that's a lot of bits to set.  Larger keys should speed that up.

The amount of space we start with comes from _open_db():

      uint64_t initial =
	bdev->get_size() * (g_conf->bluestore_bluefs_min_ratio +
			    g_conf->bluestore_bluefs_gift_ratio);
      initial = MAX(initial, g_conf->bluestore_bluefs_min);

Simply lowering min_ratio might also be fine.  The current value of 2% is 
meant to be enough for most stores, and to avoid giving over lots of 
little extents later (and making the bluefs_extents list too big).  That 
can overflow the superblock, another annoying thing we need to fix (though 
not a big deal to fix).

Anyway, adjust bluestore_bluefs_min_ratio to .01 should ~halve the time 
spent on this.. that is probably another useful test to confirm this is 
what is going on.

sage

> I would be cautious about just changing this option to affect this 
> problem (though as an experiment, we can change the value and see if it 
> has ANY affect on this problem -- which I don't think it will). The 
> value of this option really needs to be dictated by its effect on the 
> more mainstream read/write operations not on the initialization problem.
> > 
> > sage
> > 
> > 
> > >
> > > -Ramesh
> > >
> > > > -----Original Message-----
> > > > From: Allen Samuels
> > > > Sent: Thursday, August 11, 2016 9:34 PM
> > > > To: Ramesh Chander
> > > > Cc: Sage Weil; Somnath Roy; ceph-devel
> > > > Subject: Re: Bluestore different allocator performance Vs FileStore
> > > >
> > > > Is the initial creation of the keys for the bitmap one by one or are
> > > > they batched?
> > > >
> > > > Sent from my iPhone. Please excuse all typos and autocorrects.
> > > >
> > > > > On Aug 10, 2016, at 11:07 PM, Ramesh Chander
> > > > <Ramesh.Chander@sandisk.com> wrote:
> > > > >
> > > > > Somnath,
> > > > >
> > > > > Basically mkfs time has increased from 7.5 seconds (2min / 16) to
> > > > > 2 minutes
> > > > ( 32 / 16).
> > > > >
> > > > > But is there a reason you should create osds in serial? I think
> > > > > for mmultiple
> > > > osds mkfs can happen in parallel?
> > > > >
> > > > > As a fix I am looking to batch multiple insert_free calls for now.
> > > > > If still that
> > > > does not help, thinking of doing insert_free on different part of
> > > > device in parallel.
> > > > >
> > > > > -Ramesh
> > > > >
> > > > >> -----Original Message-----
> > > > >> From: Ramesh Chander
> > > > >> Sent: Thursday, August 11, 2016 10:04 AM
> > > > >> To: Allen Samuels; Sage Weil; Somnath Roy
> > > > >> Cc: ceph-devel
> > > > >> Subject: RE: Bluestore different allocator performance Vs
> > > > >> FileStore
> > > > >>
> > > > >> I think insert_free is limited by speed of function clear_bits here.
> > > > >>
> > > > >> Though set_bits and clear_bits have same logic except one sets
> > > > >> and another clears. Both of these does 64 bits (bitmap size) at a time.
> > > > >>
> > > > >> I am not sure if doing memset will make it faster. But if we can
> > > > >> do it for group of bitmaps, then it might help.
> > > > >>
> > > > >> I am looking in to code if we can handle mkfs and osd mount in
> > > > >> special way to make it faster.
> > > > >>
> > > > >> If I don't find an easy fix, we can go to path of deferring init
> > > > >> to later stage as and when required.
> > > > >>
> > > > >> -Ramesh
> > > > >>
> > > > >>> -----Original Message-----
> > > > >>> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
> > > > >>> owner@vger.kernel.org] On Behalf Of Allen Samuels
> > > > >>> Sent: Thursday, August 11, 2016 4:28 AM
> > > > >>> To: Sage Weil; Somnath Roy
> > > > >>> Cc: ceph-devel
> > > > >>> Subject: RE: Bluestore different allocator performance Vs
> > > > >>> FileStore
> > > > >>>
> > > > >>> We always knew that startup time for bitmap stuff would be
> > > > >>> somewhat longer. Still, the existing implementation can be
> > > > >>> speeded up significantly. The code in
> > > > >>> BitMapZone::set_blocks_used isn't very optimized. Converting it
> > > > >>> to use memset for all but the first/last bytes
> > > > >> should significantly speed it up.
> > > > >>>
> > > > >>>
> > > > >>> Allen Samuels
> > > > >>> SanDisk |a Western Digital brand
> > > > >>> 2880 Junction Avenue, San Jose, CA 95134
> > > > >>> T: +1 408 801 7030| M: +1 408 780 6416 allen.samuels@SanDisk.com
> > > > >>>
> > > > >>>
> > > > >>>> -----Original Message-----
> > > > >>>> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
> > > > >>>> owner@vger.kernel.org] On Behalf Of Sage Weil
> > > > >>>> Sent: Wednesday, August 10, 2016 3:44 PM
> > > > >>>> To: Somnath Roy <Somnath.Roy@sandisk.com>
> > > > >>>> Cc: ceph-devel <ceph-devel@vger.kernel.org>
> > > > >>>> Subject: RE: Bluestore different allocator performance Vs
> > > > >>>> FileStore
> > > > >>>>
> > > > >>>>> On Wed, 10 Aug 2016, Somnath Roy wrote:
> > > > >>>>> << inline with [Somnath]
> > > > >>>>>
> > > > >>>>> -----Original Message-----
> > > > >>>>> From: Sage Weil [mailto:sage@newdream.net]
> > > > >>>>> Sent: Wednesday, August 10, 2016 2:31 PM
> > > > >>>>> To: Somnath Roy
> > > > >>>>> Cc: ceph-devel
> > > > >>>>> Subject: Re: Bluestore different allocator performance Vs
> > > > >>>>> FileStore
> > > > >>>>>
> > > > >>>>>> On Wed, 10 Aug 2016, Somnath Roy wrote:
> > > > >>>>>> Hi, I spent some time on evaluating different Bluestore
> > > > >>>>>> allocator and freelist performance. Also, tried to gaze the
> > > > >>>>>> performance difference of Bluestore and filestore on the
> > > > >>>>>> similar
> > > > >> setup.
> > > > >>>>>>
> > > > >>>>>> Setup:
> > > > >>>>>> --------
> > > > >>>>>>
> > > > >>>>>> 16 OSDs (8TB Flash) across 2 OSD nodes
> > > > >>>>>>
> > > > >>>>>> Single pool and single rbd image of 4TB. 2X replication.
> > > > >>>>>>
> > > > >>>>>> Disabled the exclusive lock feature so that I can run
> > > > >>>>>> multiple write  jobs in
> > > > >>>> parallel.
> > > > >>>>>> rbd_cache is disabled in the client side.
> > > > >>>>>> Each test ran for 15 mins.
> > > > >>>>>>
> > > > >>>>>> Result :
> > > > >>>>>> ---------
> > > > >>>>>>
> > > > >>>>>> Here is the detailed report on this.
> > > > >>
> > > >
> > https://github.com/somnathr/ceph/blob/6e03a5a41fe2c9b213a610200b2e8a
> > > > >>>>>> 25 0cb05986/Bluestore_allocator_comp.xlsx
> > > > >>>>>>
> > > > >>>>>> Each profile I named based on <allocator>-<freelist> , so in
> > > > >>>>>> the graph for
> > > > >>>> example "stupid-extent" meaning stupid allocator and extent
> > freelist.
> > > > >>>>>>
> > > > >>>>>> I ran the test for each of the profile in the following order
> > > > >>>>>> after creating a
> > > > >>>> fresh rbd image for all the Bluestore test.
> > > > >>>>>>
> > > > >>>>>> 1. 4K RW for 15 min with 16QD and 10 jobs.
> > > > >>>>>>
> > > > >>>>>> 2. 16K RW for 15 min with 16QD and 10 jobs.
> > > > >>>>>>
> > > > >>>>>> 3. 64K RW for 15 min with 16QD and 10 jobs.
> > > > >>>>>>
> > > > >>>>>> 4. 256K RW for 15 min with 16QD and 10 jobs.
> > > > >>>>>>
> > > > >>>>>> The above are non-preconditioned case i.e ran before filling
> > > > >>>>>> up the entire
> > > > >>>> image. The reason is I don't see any reason of filling up the
> > > > >>>> rbd image before like filestore case where it will give stable
> > > > >>>> performance if we fill up the rbd images first. Filling up rbd
> > > > >>>> images in case of filestore will create the files in the filesystem.
> > > > >>>>>>
> > > > >>>>>> 5. Next, I did precondition the 4TB image with 1M seq write.
> > > > >>>>>> This is
> > > > >>>> primarily because I want to load BlueStore with more data.
> > > > >>>>>>
> > > > >>>>>> 6. Ran 4K RW test again (this is called out preconditioned in
> > > > >>>>>> the
> > > > >>>>>> profile) for 15 min
> > > > >>>>>>
> > > > >>>>>> 7. Ran 4K Seq test for similar QD for 15 min
> > > > >>>>>>
> > > > >>>>>> 8. Ran 16K RW test again for 15min
> > > > >>>>>>
> > > > >>>>>> For filestore test, I ran tests after preconditioning the
> > > > >>>>>> entire image
> > > > >> first.
> > > > >>>>>>
> > > > >>>>>> Each sheet on the xls have different block size result , I
> > > > >>>>>> often miss to navigate through the xls sheets , so, thought
> > > > >>>>>> of mentioning here
> > > > >>>>>> :-)
> > > > >>>>>>
> > > > >>>>>> I have also captured the mkfs time , OSD startup time and the
> > > > >>>>>> memory
> > > > >>>> usage after the entire run.
> > > > >>>>>>
> > > > >>>>>> Observation:
> > > > >>>>>> ---------------
> > > > >>>>>>
> > > > >>>>>> 1. First of all, in case of bitmap allocator mkfs time (and
> > > > >>>>>> thus cluster
> > > > >>>> creation time for 16 OSDs) are ~16X slower than stupid
> > > > >>>> allocator and
> > > > >>> filestore.
> > > > >>>> Each OSD creation is taking ~2min or so sometimes and I nailed
> > > > >>>> down the
> > > > >>>> insert_free() function call (marked ****) in the Bitmap
> > > > >>>> allocator is causing that.
> > > > >>>>>>
> > > > >>>>>> 2016-08-05 16:12:40.587148 7f4024d258c0 10 freelist
> > > > >>>>>> enumerate_next start
> > > > >>>>>> 2016-08-05 16:12:40.975539 7f4024d258c0 10 freelist
> > > > >>>>>> enumerate_next
> > > > >>>>>> 0x4663d00000~69959451000
> > > > >>>>>> 2016-08-05 16:12:40.975555 7f4024d258c0 10
> > > > >>>>>> bitmapalloc:init_add_free instance 139913322803328 offset
> > > > >>>>>> 0x4663d00000 length 0x69959451000
> > > > >>>>>> ****2016-08-05 16:12:40.975557 7f4024d258c0 20
> > > > >>>>>> bitmapalloc:insert_free instance 139913322803328 off
> > > > >>>>>> 0x4663d00000 len 0x69959451000****
> > > > >>>>>> ****2016-08-05 16:13:20.748934 7f4024d258c0 10 freelist
> > > > >>>>>> enumerate_next
> > > > >>>>>> end****
> > > > >>>>>> 2016-08-05 16:13:20.748978 7f4024d258c0 10
> > > > >>>>>> bluestore(/var/lib/ceph/osd/ceph-0) _open_alloc loaded 6757 G
> > > > >>>>>> in
> > > > >>>>>> 1 extents
> > > > >>>>>>
> > > > >>>>>> 2016-08-05 16:13:23.438511 7f4024d258c0 20 bluefs
> > > > >>>>>> _read_random read buffered 0x4a14eb~265 of
> > ^A:5242880+5242880
> > > > >>>>>> 2016-08-05 16:13:23.438587 7f4024d258c0 20 bluefs
> > > > >>>>>> _read_random got
> > > > >>>>>> 613
> > > > >>>>>> 2016-08-05 16:13:23.438658 7f4024d258c0 10 freelist
> > > > >>>>>> enumerate_next
> > > > >>>>>> 0x4663d00000~69959451000
> > > > >>>>>> 2016-08-05 16:13:23.438664 7f4024d258c0 10
> > > > >>>>>> bitmapalloc:init_add_free instance 139913306273920 offset
> > > > >>>>>> 0x4663d00000 length 0x69959451000
> > > > >>>>>> *****2016-08-05 16:13:23.438666 7f4024d258c0 20
> > > > >>>>>> bitmapalloc:insert_free instance 139913306273920 off
> > > > >>>>>> 0x4663d00000 len
> > > > >>>>>> 0x69959451000*****
> > > > >>>>>> *****2016-08-05 16:14:03.132914 7f4024d258c0 10 freelist
> > > > >>>>>> enumerate_next end
> > > > >>>>>
> > > > >>>>> I'm not sure there's any easy fix for this. We can amortize it
> > > > >>>>> by feeding
> > > > >>>> space to bluefs slowly (so that we don't have to do all the
> > > > >>>> inserts at once), but I'm not sure that's really better.
> > > > >>>>>
> > > > >>>>> [Somnath] I don't know that part of the code, so, may be a
> > > > >>>>> dumb
> > > > >>> question.
> > > > >>>> This is during mkfs() time , so, can't we say to bluefs entire
> > > > >>>> space is free ? I can understand for osd mount and all other
> > > > >>>> cases we need to feed the free space every time.
> > > > >>>>> IMO this is critical to fix as cluster creation time will be
> > > > >>>>> number of OSDs * 2
> > > > >>>> min otherwise. For me creating 16 OSDs cluster is taking ~32min
> > > > >>>> compare to
> > > > >>>> ~2 min for stupid allocator/filestore.
> > > > >>>>> BTW, my drive data partition is ~6.9TB , db partition is ~100G
> > > > >>>>> and WAL is
> > > > >>>> ~1G. I guess the time taking is dependent on data partition
> > > > >>>> size as well
> > > > (?
> > > > >>>>
> > > > >>>> Well, we're fundamentally limited by the fact that it's a
> > > > >>>> bitmap, and a big chunk of space is "allocated" to bluefs and
> > > > >>>> needs to have 1's
> > > > set.
> > > > >>>>
> > > > >>>> sage
> > > > >>>> --
> > > > >>>> To unsubscribe from this list: send the line "unsubscribe ceph-
> > devel"
> > > > >>>> in the body of a message to majordomo@vger.kernel.org More
> > > > >>> majordomo
> > > > >>>> info at http://vger.kernel.org/majordomo-info.html
> > > > >>> --
> > > > >>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> > > > >>> in the body of a message to majordomo@vger.kernel.org More
> > > > >> majordomo
> > > > >>> info at http://vger.kernel.org/majordomo-info.html
> > > PLEASE NOTE: The information contained in this electronic mail message is
> > intended only for the use of the designated recipient(s) named above. If the
> > reader of this message is not the intended recipient, you are hereby notified
> > that you have received this message in error and that any review,
> > dissemination, distribution, or copying of this message is strictly prohibited. If
> > you have received this communication in error, please notify the sender by
> > telephone or e-mail (as shown above) immediately and destroy any and all
> > copies of this message in your possession (whether hard copies or
> > electronically stored copies).
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> > > in the body of a message to majordomo@vger.kernel.org More
> > majordomo
> > > info at  http://vger.kernel.org/majordomo-info.html
> > >
> > >
> 
>