BlueStore deep-dive over bluejeans

BlueStore deep-dive over bluejeans

[Sage Weil]

Hey all,

I'm planning on doing an hour-ish deep-dive into the inner workings of 
BlueStore next Monday, 2/29, 5pm-6pm ET / 2pm-3pm PT.  It'll be a 
mostly-informal brain dump and walk-through of the code over bluejeans, 
and we'll record it.  Anyone who is or is planning to be involved in 
bluestore development, please join and ask questions etc!  I'll send out 
the bluejeans link shortly before we start.

sage

Re: BlueStore deep-dive over bluejeans

[Erwan Velu]

I cannot attend but will watch the recording.
If you can paste it on this thread, would be nice.

Thx,

----- Original Message -----
From: "Sage Weil" <sweil@redhat.com>
To: ceph-devel@vger.kernel.org
Sent: Wednesday, February 24, 2016 8:51:39 PM
Subject: BlueStore deep-dive over bluejeans

Hey all,

I'm planning on doing an hour-ish deep-dive into the inner workings of 
BlueStore next Monday, 2/29, 5pm-6pm ET / 2pm-3pm PT.  It'll be a 
mostly-informal brain dump and walk-through of the code over bluejeans, 
and we'll record it.  Anyone who is or is planning to be involved in 
bluestore development, please join and ask questions etc!  I'll send out 
the bluejeans link shortly before we start.

sage
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: BlueStore deep-dive over bluejeans

[Sage Weil]

On Wed, 24 Feb 2016, Sage Weil wrote:
> Hey all,
> 
> I'm planning on doing an hour-ish deep-dive into the inner workings of 
> BlueStore next Monday, 2/29, 5pm-6pm ET / 2pm-3pm PT.  It'll be a 
> mostly-informal brain dump and walk-through of the code over bluejeans, 
> and we'll record it.  Anyone who is or is planning to be involved in 
> bluestore development, please join and ask questions etc!  I'll send out 
> the bluejeans link shortly before we start.

I'm going to defer this until later--I'm not feeling well.  I'll send out 
another email soon with a new time slot.

Sorry!
sage

Re: BlueStore deep-dive over bluejeans

[Sage Weil]

On Mon, 29 Feb 2016, Sage Weil wrote:
> On Wed, 24 Feb 2016, Sage Weil wrote:
> > Hey all,
> > 
> > I'm planning on doing an hour-ish deep-dive into the inner workings of 
> > BlueStore next Monday, 2/29, 5pm-6pm ET / 2pm-3pm PT.  It'll be a 
> > mostly-informal brain dump and walk-through of the code over bluejeans, 
> > and we'll record it.  Anyone who is or is planning to be involved in 
> > bluestore development, please join and ask questions etc!  I'll send out 
> > the bluejeans link shortly before we start.
> 
> I'm going to defer this until later--I'm not feeling well.  I'll send out 
> another email soon with a new time slot.

We try again with the same time slot today (5pm ET, 2pm PT).  The 
bluejeans URL will be

	https://bluejeans.com/SWeil09
	
sage

Re: BlueStore deep-dive over bluejeans

[Sage Weil]

On Tue, 1 Mar 2016, Sage Weil wrote:
> On Mon, 29 Feb 2016, Sage Weil wrote:
> > On Wed, 24 Feb 2016, Sage Weil wrote:
> > > Hey all,
> > > 
> > > I'm planning on doing an hour-ish deep-dive into the inner workings of 
> > > BlueStore next Monday, 2/29, 5pm-6pm ET / 2pm-3pm PT.  It'll be a 
> > > mostly-informal brain dump and walk-through of the code over bluejeans, 
> > > and we'll record it.  Anyone who is or is planning to be involved in 
> > > bluestore development, please join and ask questions etc!  I'll send out 
> > > the bluejeans link shortly before we start.
> > 
> > I'm going to defer this until later--I'm not feeling well.  I'll send out 
> > another email soon with a new time slot.
> 
> We try again with the same time slot today (5pm ET, 2pm PT).  The 
> bluejeans URL will be
> 
> 	https://bluejeans.com/SWeil09

Here's the recording:

	https://trello.com/b/Zc0QlFzs/rhcs-2-0

sage

Re: BlueStore deep-dive over bluejeans

[Sage Weil]

On Wed, 2 Mar 2016, Sage Weil wrote:
> On Tue, 1 Mar 2016, Sage Weil wrote:
> > On Mon, 29 Feb 2016, Sage Weil wrote:
> > > On Wed, 24 Feb 2016, Sage Weil wrote:
> > > > Hey all,
> > > > 
> > > > I'm planning on doing an hour-ish deep-dive into the inner workings of 
> > > > BlueStore next Monday, 2/29, 5pm-6pm ET / 2pm-3pm PT.  It'll be a 
> > > > mostly-informal brain dump and walk-through of the code over bluejeans, 
> > > > and we'll record it.  Anyone who is or is planning to be involved in 
> > > > bluestore development, please join and ask questions etc!  I'll send out 
> > > > the bluejeans link shortly before we start.
> > > 
> > > I'm going to defer this until later--I'm not feeling well.  I'll send out 
> > > another email soon with a new time slot.
> > 
> > We try again with the same time slot today (5pm ET, 2pm PT).  The 
> > bluejeans URL will be
> > 
> > 	https://bluejeans.com/SWeil09
> 
> Here's the recording:

Oops, here's the correct URL:

	https://bluejeans.com/s/9dck/

sage

Re: BlueStore deep-dive over bluejeans

[Dan Mick]

> 	https://bluejeans.com/s/9dck/

Can you set this to allow download?  I can't get it to play in the
bluejeans interface, but downloadable files usually work for me


-- 
Dan Mick
Red Hat, Inc.
Ceph docs: http://ceph.com/docs

Re: BlueStore deep-dive over bluejeans

[Haomai Wang]

On Thu, Mar 3, 2016 at 12:08 PM, Dan Mick <dmick@redhat.com> wrote:
>
>>       https://bluejeans.com/s/9dck/
>
> Can you set this to allow download?  I can't get it to play in the
> bluejeans interface, but downloadable files usually work for me

yes, I can't play it online too.

>
>
> --
> Dan Mick
> Red Hat, Inc.
> Ceph docs: http://ceph.com/docs
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: BlueStore deep-dive over bluejeans

[Sage Weil]

On Wed, 2 Mar 2016, Dan Mick wrote:
> 
> > 	https://bluejeans.com/s/9dck/
> 
> Can you set this to allow download?  I can't get it to play in the
> bluejeans interface, but downloadable files usually work for me

Fixed!

sage

Re: BlueStore deep-dive over bluejeans

[huang jun]

where to get the ppt in the video?

2016-03-03 21:19 GMT+08:00 Sage Weil <sage@newdream.net>:
> On Wed, 2 Mar 2016, Dan Mick wrote:
>>
>> >     https://bluejeans.com/s/9dck/
>>
>> Can you set this to allow download?  I can't get it to play in the
>> bluejeans interface, but downloadable files usually work for me
>
> Fixed!
>
> sage
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
thanks
huangjun

Re: BlueStore deep-dive over bluejeans

[Sage Weil]

On Sat, 5 Mar 2016, huang jun wrote:
> where to get the ppt in the video?

There's no PPT.. I was just sharing my screen and walking through some of 
the code.

sage


> 
> 2016-03-03 21:19 GMT+08:00 Sage Weil <sage@newdream.net>:
> > On Wed, 2 Mar 2016, Dan Mick wrote:
> >>
> >> >     https://bluejeans.com/s/9dck/
> >>
> >> Can you set this to allow download?  I can't get it to play in the
> >> bluejeans interface, but downloadable files usually work for me
> >
> > Fixed!
> >
> > sage
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 
> 
> -- 
> thanks
> huangjun
> 
> 

Re: Re: BlueStore deep-dive over bluejeans

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2516 bytes --]

Hi,

On Sun, 6 Mar 2016, 陈静 wrote:
> Hi Sage,
> 
> So BlueStore will rely on RocksDB a lot, which just remind me of some
> experience regarding to LevelDB recovery and compaction some time ago.
> 
> We tried Ceph RGW 0.80, which had no Bucket Index Sharding feature. SSD
> managed by FileStore is used for Bucket Index Pool.
> After we put tens of millions of files into a bucket, LevelDB of the OSD
> holding the corresponding bucket index object grew very big.
> The OSD once crashed unexpectly. When we try to start the OSD again, it took
> hours to fully get up.
> We used pstack to observed what it was doing and noticed the OSD is busy in
> recovering and compacting its LevelDB.
> The recovering and compaction consumed a lot CPU and memory as well.
> 
> BlueStore seems to rely more on RocksDB, which is a variant of LevelDB.
> Then when the number of objects in BlueStore is large, will it take even
> more time for the OSD to start up (for recoverying and compaction) if it was
> not shut down cleanly previously?

In general, rocksdb/leveldb shouldn't need to compact on startup.  I'm not 
sure what caused that in your situation.  We've seen a few leveldb bugs in 
the past that prevented compaction from happening when it should; perhaps 
it was one of those.

We did do some testing with rocksdb where we inflated the size to he db to 
be very large and it performed pretty well.  We didn't try clearing large 
swaths of the keyspace to test the compaction side of things, though.

sage

> 
> 
> Thanks,
> Jeegn
>  
>       From: huang jun
> Date: 2016-03-05 18:41
> To: Sage Weil
> CC: Dan Mick; ceph-devel
> Subject: Re: BlueStore deep-dive over bluejeans
> where to get the ppt in the video?
>  
> 2016-03-03 21:19 GMT+08:00 Sage Weil <sage@newdream.net>:
> > On Wed, 2 Mar 2016, Dan Mick wrote:
> >>
> >> >     https://bluejeans.com/s/9dck/
> >>
> >> Can you set this to allow download?  I can't get it to play in the
> >> bluejeans interface, but downloadable files usually work for me
> >
> > Fixed!
> >
> > sage
> > --
> > To unsubscribe from this list: send the line "unsubscribe
> ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>  
>  
>  
> --
> thanks
> huangjun
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 
> 

Re: Re: BlueStore deep-dive over bluejeans

[chen kael]

Hi,sage
I have a question, does small files smaller than min_alloc_size will
be store in RocksDB for ever? And  I am not quite clear about the
differences between overlay write which is set off by default and WAL
write.

2016-03-07 0:34 GMT+08:00 Sage Weil <sage@newdream.net>:
> Hi,
>
> On Sun, 6 Mar 2016, 陈静 wrote:
>> Hi Sage,
>>
>> So BlueStore will rely on RocksDB a lot, which just remind me of some
>> experience regarding to LevelDB recovery and compaction some time ago.
>>
>> We tried Ceph RGW 0.80, which had no Bucket Index Sharding feature. SSD
>> managed by FileStore is used for Bucket Index Pool.
>> After we put tens of millions of files into a bucket, LevelDB of the OSD
>> holding the corresponding bucket index object grew very big.
>> The OSD once crashed unexpectly. When we try to start the OSD again, it took
>> hours to fully get up.
>> We used pstack to observed what it was doing and noticed the OSD is busy in
>> recovering and compacting its LevelDB.
>> The recovering and compaction consumed a lot CPU and memory as well.
>>
>> BlueStore seems to rely more on RocksDB, which is a variant of LevelDB.
>> Then when the number of objects in BlueStore is large, will it take even
>> more time for the OSD to start up (for recoverying and compaction) if it was
>> not shut down cleanly previously?
>
> In general, rocksdb/leveldb shouldn't need to compact on startup.  I'm not
> sure what caused that in your situation.  We've seen a few leveldb bugs in
> the past that prevented compaction from happening when it should; perhaps
> it was one of those.
>
> We did do some testing with rocksdb where we inflated the size to he db to
> be very large and it performed pretty well.  We didn't try clearing large
> swaths of the keyspace to test the compaction side of things, though.
>
> sage
>
>>
>>
>> Thanks,
>> Jeegn
>>
>>       From: huang jun
>> Date: 2016-03-05 18:41
>> To: Sage Weil
>> CC: Dan Mick; ceph-devel
>> Subject: Re: BlueStore deep-dive over bluejeans
>> where to get the ppt in the video?
>>
>> 2016-03-03 21:19 GMT+08:00 Sage Weil <sage@newdream.net>:
>> > On Wed, 2 Mar 2016, Dan Mick wrote:
>> >>
>> >> >     https://bluejeans.com/s/9dck/
>> >>
>> >> Can you set this to allow download?  I can't get it to play in the
>> >> bluejeans interface, but downloadable files usually work for me
>> >
>> > Fixed!
>> >
>> > sage
>> > --
>> > To unsubscribe from this list: send the line "unsubscribe
>> ceph-devel" in
>> > the body of a message to majordomo@vger.kernel.org
>> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
>>
>> --
>> thanks
>> huangjun
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>> in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
>>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Re: BlueStore deep-dive over bluejeans

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 3895 bytes --]

On Tue, 12 Apr 2016, chen kael wrote:
> Hi,sage
> I have a question, does small files smaller than min_alloc_size will
> be store in RocksDB for ever? And  I am not quite clear about the
> differences between overlay write which is set off by default and WAL
> write.

A WAL write is included in the transaction written to rocksdb.  
Once that commits, the IO (usually an overwrite) is immediately queued, 
and the WAL record is removed from rocksdb during the next commit cycle.

An overlay write is intended to stick around in rocksdb until some 
threshold is reached (N overlay records), at which point all of the 
IOs are done at once.

The overlay path wasn't fully updated after the most recent rewrite, and 
prior to that didn't seem to make things any faster.  I'm not sure if it's 
worth keeping around or not.  In theory it should help with spinning 
disks, but in practice it doesn't seem to.

sage


> 
> 2016-03-07 0:34 GMT+08:00 Sage Weil <sage@newdream.net>:
> > Hi,
> >
> > On Sun, 6 Mar 2016, 陈静 wrote:
> >> Hi Sage,
> >>
> >> So BlueStore will rely on RocksDB a lot, which just remind me of some
> >> experience regarding to LevelDB recovery and compaction some time ago.
> >>
> >> We tried Ceph RGW 0.80, which had no Bucket Index Sharding feature. SSD
> >> managed by FileStore is used for Bucket Index Pool.
> >> After we put tens of millions of files into a bucket, LevelDB of the OSD
> >> holding the corresponding bucket index object grew very big.
> >> The OSD once crashed unexpectly. When we try to start the OSD again, it took
> >> hours to fully get up.
> >> We used pstack to observed what it was doing and noticed the OSD is busy in
> >> recovering and compacting its LevelDB.
> >> The recovering and compaction consumed a lot CPU and memory as well.
> >>
> >> BlueStore seems to rely more on RocksDB, which is a variant of LevelDB.
> >> Then when the number of objects in BlueStore is large, will it take even
> >> more time for the OSD to start up (for recoverying and compaction) if it was
> >> not shut down cleanly previously?
> >
> > In general, rocksdb/leveldb shouldn't need to compact on startup.  I'm not
> > sure what caused that in your situation.  We've seen a few leveldb bugs in
> > the past that prevented compaction from happening when it should; perhaps
> > it was one of those.
> >
> > We did do some testing with rocksdb where we inflated the size to he db to
> > be very large and it performed pretty well.  We didn't try clearing large
> > swaths of the keyspace to test the compaction side of things, though.
> >
> > sage
> >
> >>
> >>
> >> Thanks,
> >> Jeegn
> >>
> >>       From: huang jun
> >> Date: 2016-03-05 18:41
> >> To: Sage Weil
> >> CC: Dan Mick; ceph-devel
> >> Subject: Re: BlueStore deep-dive over bluejeans
> >> where to get the ppt in the video?
> >>
> >> 2016-03-03 21:19 GMT+08:00 Sage Weil <sage@newdream.net>:
> >> > On Wed, 2 Mar 2016, Dan Mick wrote:
> >> >>
> >> >> >     https://bluejeans.com/s/9dck/
> >> >>
> >> >> Can you set this to allow download?  I can't get it to play in the
> >> >> bluejeans interface, but downloadable files usually work for me
> >> >
> >> > Fixed!
> >> >
> >> > sage
> >> > --
> >> > To unsubscribe from this list: send the line "unsubscribe
> >> ceph-devel" in
> >> > the body of a message to majordomo@vger.kernel.org
> >> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>
> >>
> >>
> >> --
> >> thanks
> >> huangjun
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> >> in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>
> >>
> >>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
>