Reshard is not stabilizing in latest master

Reshard is not stabilizing in latest master

[Somnath Roy]

Sage,
I am seeing reshard is not stabilizing even after I precondition an image with 4K seq. It was happening with 2 weeks old master. Anything change on that part of the code base recently ? Otherwise I need to debug.

Thanks & Regards
Somnath
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

Re: Reshard is not stabilizing in latest master

[Sage Weil]

On Thu, 8 Dec 2016, Somnath Roy wrote:
> Sage,
> I am seeing reshard is not stabilizing even after I precondition an image with 4K seq. It was happening with 2 weeks old master. Anything change on that part of the code base recently ? Otherwise I need to debug.

Nope.  If you have a cluster where it isn't stabilizing just turn on debug 
bluestore = 20 for a minute and send me the log.

sage

RE: Reshard is not stabilizing in latest master

[Somnath Roy]

Sage,
Please find it in the following location , it's a zipped file.

https://drive.google.com/open?id=0B7W-S0z_ymMJd0V1XzVhQWR4ak0


What I found that the following shard length always comes below bluestore_extent_map_shard_min_size and triggering reshard.

2016-12-08 20:07:56.950527 7fe9140f7700 20 bluestore.extentmap(0x557183546d48) update shard 0x3ff000 is 25 bytes (was 25) from 1 extents

Thanks & Regards
Somnath

-----Original Message-----
From: Sage Weil [mailto:sweil@redhat.com]
Sent: Thursday, December 08, 2016 5:00 PM
To: Somnath Roy
Cc: ceph-devel
Subject: Re: Reshard is not stabilizing in latest master

On Thu, 8 Dec 2016, Somnath Roy wrote:
> Sage,
> I am seeing reshard is not stabilizing even after I precondition an image with 4K seq. It was happening with 2 weeks old master. Anything change on that part of the code base recently ? Otherwise I need to debug.

Nope.  If you have a cluster where it isn't stabilizing just turn on debug bluestore = 20 for a minute and send me the log.

sage
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

RE: Reshard is not stabilizing in latest master

[Sage Weil]

On Fri, 9 Dec 2016, Somnath Roy wrote:
> Sage,
> Please find it in the following location , it's a zipped file.
> 
> https://drive.google.com/open?id=0B7W-S0z_ymMJd0V1XzVhQWR4ak0
> 
> 
> What I found that the following shard length always comes below bluestore_extent_map_shard_min_size and triggering reshard.
> 
> 2016-12-08 20:07:56.950527 7fe9140f7700 20 bluestore.extentmap(0x557183546d48) update shard 0x3ff000 is 25 bytes (was 25) from 1 extents

All of the instances are at offset 0x3ff000, which is right at the end of 
the object.  I think we just need some special case code to handle when 
the last little bit ends up being small, and pull it into the previous 
shard (probably).  

sage


> 
> Thanks & Regards
> Somnath
> 
> -----Original Message-----
> From: Sage Weil [mailto:sweil@redhat.com]
> Sent: Thursday, December 08, 2016 5:00 PM
> To: Somnath Roy
> Cc: ceph-devel
> Subject: Re: Reshard is not stabilizing in latest master
> 
> On Thu, 8 Dec 2016, Somnath Roy wrote:
> > Sage,
> > I am seeing reshard is not stabilizing even after I precondition an image with 4K seq. It was happening with 2 weeks old master. Anything change on that part of the code base recently ? Otherwise I need to debug.
> 
> Nope.  If you have a cluster where it isn't stabilizing just turn on debug bluestore = 20 for a minute and send me the log.
> 
> sage
> PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
> 
> 

Re: Reshard is not stabilizing in latest master

[Igor Fedotov]

yeah, I've got the same explanation 10 min ago.
PR is ready too:
https://github.com/ceph/ceph/pull/12447

On 12.12.2016 20:39, Sage Weil wrote:
> On Fri, 9 Dec 2016, Somnath Roy wrote:
>> Sage,
>> Please find it in the following location , it's a zipped file.
>>
>> https://drive.google.com/open?id=0B7W-S0z_ymMJd0V1XzVhQWR4ak0
>>
>>
>> What I found that the following shard length always comes below bluestore_extent_map_shard_min_size and triggering reshard.
>>
>> 2016-12-08 20:07:56.950527 7fe9140f7700 20 bluestore.extentmap(0x557183546d48) update shard 0x3ff000 is 25 bytes (was 25) from 1 extents
> All of the instances are at offset 0x3ff000, which is right at the end of
> the object.  I think we just need some special case code to handle when
> the last little bit ends up being small, and pull it into the previous
> shard (probably).
>
> sage
>
>
>> Thanks & Regards
>> Somnath
>>
>> -----Original Message-----
>> From: Sage Weil [mailto:sweil@redhat.com]
>> Sent: Thursday, December 08, 2016 5:00 PM
>> To: Somnath Roy
>> Cc: ceph-devel
>> Subject: Re: Reshard is not stabilizing in latest master
>>
>> On Thu, 8 Dec 2016, Somnath Roy wrote:
>>> Sage,
>>> I am seeing reshard is not stabilizing even after I precondition an image with 4K seq. It was happening with 2 weeks old master. Anything change on that part of the code base recently ? Otherwise I need to debug.
>> Nope.  If you have a cluster where it isn't stabilizing just turn on debug bluestore = 20 for a minute and send me the log.
>>
>> sage
>> PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Reshard is not stabilizing in latest master

[Somnath Roy]

Make sense , but, any idea why it is showing up now as it is not happening with old master ? What possibly I did to reproduce it, just curious.

Thanks & Regards
Somnath

-----Original Message-----
From: Igor Fedotov [mailto:ifedotov@mirantis.com] 
Sent: Monday, December 12, 2016 9:46 AM
To: Sage Weil; Somnath Roy
Cc: ceph-devel
Subject: Re: Reshard is not stabilizing in latest master

yeah, I've got the same explanation 10 min ago.
PR is ready too:
https://github.com/ceph/ceph/pull/12447

On 12.12.2016 20:39, Sage Weil wrote:
> On Fri, 9 Dec 2016, Somnath Roy wrote:
>> Sage,
>> Please find it in the following location , it's a zipped file.
>>
>> https://drive.google.com/open?id=0B7W-S0z_ymMJd0V1XzVhQWR4ak0
>>
>>
>> What I found that the following shard length always comes below bluestore_extent_map_shard_min_size and triggering reshard.
>>
>> 2016-12-08 20:07:56.950527 7fe9140f7700 20 
>> bluestore.extentmap(0x557183546d48) update shard 0x3ff000 is 25 bytes 
>> (was 25) from 1 extents
> All of the instances are at offset 0x3ff000, which is right at the end 
> of the object.  I think we just need some special case code to handle 
> when the last little bit ends up being small, and pull it into the 
> previous shard (probably).
>
> sage
>
>
>> Thanks & Regards
>> Somnath
>>
>> -----Original Message-----
>> From: Sage Weil [mailto:sweil@redhat.com]
>> Sent: Thursday, December 08, 2016 5:00 PM
>> To: Somnath Roy
>> Cc: ceph-devel
>> Subject: Re: Reshard is not stabilizing in latest master
>>
>> On Thu, 8 Dec 2016, Somnath Roy wrote:
>>> Sage,
>>> I am seeing reshard is not stabilizing even after I precondition an image with 4K seq. It was happening with 2 weeks old master. Anything change on that part of the code base recently ? Otherwise I need to debug.
>> Nope.  If you have a cluster where it isn't stabilizing just turn on debug bluestore = 20 for a minute and send me the log.
>>
>> sage
>> PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
> in the body of a message to majordomo@vger.kernel.org More majordomo 
> info at  http://vger.kernel.org/majordomo-info.html

Re: Reshard is not stabilizing in latest master

[Igor Fedotov]

Any changes in shards serialization or general encoding infra since that 
time?

shard min/max size changes?


On 12.12.2016 21:03, Somnath Roy wrote:
> Make sense , but, any idea why it is showing up now as it is not happening with old master ? What possibly I did to reproduce it, just curious.
>
> Thanks & Regards
> Somnath
>
> -----Original Message-----
> From: Igor Fedotov [mailto:ifedotov@mirantis.com]
> Sent: Monday, December 12, 2016 9:46 AM
> To: Sage Weil; Somnath Roy
> Cc: ceph-devel
> Subject: Re: Reshard is not stabilizing in latest master
>
> yeah, I've got the same explanation 10 min ago.
> PR is ready too:
> https://github.com/ceph/ceph/pull/12447
>
> On 12.12.2016 20:39, Sage Weil wrote:
>> On Fri, 9 Dec 2016, Somnath Roy wrote:
>>> Sage,
>>> Please find it in the following location , it's a zipped file.
>>>
>>> https://drive.google.com/open?id=0B7W-S0z_ymMJd0V1XzVhQWR4ak0
>>>
>>>
>>> What I found that the following shard length always comes below bluestore_extent_map_shard_min_size and triggering reshard.
>>>
>>> 2016-12-08 20:07:56.950527 7fe9140f7700 20
>>> bluestore.extentmap(0x557183546d48) update shard 0x3ff000 is 25 bytes
>>> (was 25) from 1 extents
>> All of the instances are at offset 0x3ff000, which is right at the end
>> of the object.  I think we just need some special case code to handle
>> when the last little bit ends up being small, and pull it into the
>> previous shard (probably).
>>
>> sage
>>
>>
>>> Thanks & Regards
>>> Somnath
>>>
>>> -----Original Message-----
>>> From: Sage Weil [mailto:sweil@redhat.com]
>>> Sent: Thursday, December 08, 2016 5:00 PM
>>> To: Somnath Roy
>>> Cc: ceph-devel
>>> Subject: Re: Reshard is not stabilizing in latest master
>>>
>>> On Thu, 8 Dec 2016, Somnath Roy wrote:
>>>> Sage,
>>>> I am seeing reshard is not stabilizing even after I precondition an image with 4K seq. It was happening with 2 weeks old master. Anything change on that part of the code base recently ? Otherwise I need to debug.
>>> Nope.  If you have a cluster where it isn't stabilizing just turn on debug bluestore = 20 for a minute and send me the log.
>>>
>>> sage
>>> PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
>>>
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>> in the body of a message to majordomo@vger.kernel.org More majordomo
>> info at  http://vger.kernel.org/majordomo-info.html

Re: Reshard is not stabilizing in latest master

[Varada Kari]

Somnath might be using some of config params to expedite the sharding
like max_blob_size etc.. along with some more configs. He shared them
in the call yesterday.

Varada

On Tue, Dec 13, 2016 at 7:13 PM, Igor Fedotov <ifedotov@mirantis.com> wrote:
> Any changes in shards serialization or general encoding infra since that
> time?
>
> shard min/max size changes?
>
>
>
> On 12.12.2016 21:03, Somnath Roy wrote:
>>
>> Make sense , but, any idea why it is showing up now as it is not happening
>> with old master ? What possibly I did to reproduce it, just curious.
>>
>> Thanks & Regards
>> Somnath
>>
>> -----Original Message-----
>> From: Igor Fedotov [mailto:ifedotov@mirantis.com]
>> Sent: Monday, December 12, 2016 9:46 AM
>> To: Sage Weil; Somnath Roy
>> Cc: ceph-devel
>> Subject: Re: Reshard is not stabilizing in latest master
>>
>> yeah, I've got the same explanation 10 min ago.
>> PR is ready too:
>> https://github.com/ceph/ceph/pull/12447
>>
>> On 12.12.2016 20:39, Sage Weil wrote:
>>>
>>> On Fri, 9 Dec 2016, Somnath Roy wrote:
>>>>
>>>> Sage,
>>>> Please find it in the following location , it's a zipped file.
>>>>
>>>> https://drive.google.com/open?id=0B7W-S0z_ymMJd0V1XzVhQWR4ak0
>>>>
>>>>
>>>> What I found that the following shard length always comes below
>>>> bluestore_extent_map_shard_min_size and triggering reshard.
>>>>
>>>> 2016-12-08 20:07:56.950527 7fe9140f7700 20
>>>> bluestore.extentmap(0x557183546d48) update shard 0x3ff000 is 25 bytes
>>>> (was 25) from 1 extents
>>>
>>> All of the instances are at offset 0x3ff000, which is right at the end
>>> of the object.  I think we just need some special case code to handle
>>> when the last little bit ends up being small, and pull it into the
>>> previous shard (probably).
>>>
>>> sage
>>>
>>>
>>>> Thanks & Regards
>>>> Somnath
>>>>
>>>> -----Original Message-----
>>>> From: Sage Weil [mailto:sweil@redhat.com]
>>>> Sent: Thursday, December 08, 2016 5:00 PM
>>>> To: Somnath Roy
>>>> Cc: ceph-devel
>>>> Subject: Re: Reshard is not stabilizing in latest master
>>>>
>>>> On Thu, 8 Dec 2016, Somnath Roy wrote:
>>>>>
>>>>> Sage,
>>>>> I am seeing reshard is not stabilizing even after I precondition an
>>>>> image with 4K seq. It was happening with 2 weeks old master. Anything change
>>>>> on that part of the code base recently ? Otherwise I need to debug.
>>>>
>>>> Nope.  If you have a cluster where it isn't stabilizing just turn on
>>>> debug bluestore = 20 for a minute and send me the log.
>>>>
>>>> sage
>>>> PLEASE NOTE: The information contained in this electronic mail message
>>>> is intended only for the use of the designated recipient(s) named above. If
>>>> the reader of this message is not the intended recipient, you are hereby
>>>> notified that you have received this message in error and that any review,
>>>> dissemination, distribution, or copying of this message is strictly
>>>> prohibited. If you have received this communication in error, please notify
>>>> the sender by telephone or e-mail (as shown above) immediately and destroy
>>>> any and all copies of this message in your possession (whether hard copies
>>>> or electronically stored copies).
>>>>
>>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>>> in the body of a message to majordomo@vger.kernel.org More majordomo
>>> info at  http://vger.kernel.org/majordomo-info.html
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Reshard is not stabilizing in latest master

[Somnath Roy]

Nope, using all default for shard related stuff. Regarding max blob size/max alloc size we discussed in the standups shouldn't affect shards.

-----Original Message-----
From: Varada Kari [mailto:varada.kari@gmail.com]
Sent: Tuesday, December 13, 2016 7:07 AM
To: Igor Fedotov
Cc: Somnath Roy; Sage Weil; ceph-devel
Subject: Re: Reshard is not stabilizing in latest master

Somnath might be using some of config params to expedite the sharding like max_blob_size etc.. along with some more configs. He shared them in the call yesterday.

Varada

On Tue, Dec 13, 2016 at 7:13 PM, Igor Fedotov <ifedotov@mirantis.com> wrote:
> Any changes in shards serialization or general encoding infra since
> that time?
>
> shard min/max size changes?
>
>
>
> On 12.12.2016 21:03, Somnath Roy wrote:
>>
>> Make sense , but, any idea why it is showing up now as it is not
>> happening with old master ? What possibly I did to reproduce it, just curious.
>>
>> Thanks & Regards
>> Somnath
>>
>> -----Original Message-----
>> From: Igor Fedotov [mailto:ifedotov@mirantis.com]
>> Sent: Monday, December 12, 2016 9:46 AM
>> To: Sage Weil; Somnath Roy
>> Cc: ceph-devel
>> Subject: Re: Reshard is not stabilizing in latest master
>>
>> yeah, I've got the same explanation 10 min ago.
>> PR is ready too:
>> https://github.com/ceph/ceph/pull/12447
>>
>> On 12.12.2016 20:39, Sage Weil wrote:
>>>
>>> On Fri, 9 Dec 2016, Somnath Roy wrote:
>>>>
>>>> Sage,
>>>> Please find it in the following location , it's a zipped file.
>>>>
>>>> https://drive.google.com/open?id=0B7W-S0z_ymMJd0V1XzVhQWR4ak0
>>>>
>>>>
>>>> What I found that the following shard length always comes below
>>>> bluestore_extent_map_shard_min_size and triggering reshard.
>>>>
>>>> 2016-12-08 20:07:56.950527 7fe9140f7700 20
>>>> bluestore.extentmap(0x557183546d48) update shard 0x3ff000 is 25
>>>> bytes (was 25) from 1 extents
>>>
>>> All of the instances are at offset 0x3ff000, which is right at the
>>> end of the object.  I think we just need some special case code to
>>> handle when the last little bit ends up being small, and pull it
>>> into the previous shard (probably).
>>>
>>> sage
>>>
>>>
>>>> Thanks & Regards
>>>> Somnath
>>>>
>>>> -----Original Message-----
>>>> From: Sage Weil [mailto:sweil@redhat.com]
>>>> Sent: Thursday, December 08, 2016 5:00 PM
>>>> To: Somnath Roy
>>>> Cc: ceph-devel
>>>> Subject: Re: Reshard is not stabilizing in latest master
>>>>
>>>> On Thu, 8 Dec 2016, Somnath Roy wrote:
>>>>>
>>>>> Sage,
>>>>> I am seeing reshard is not stabilizing even after I precondition
>>>>> an image with 4K seq. It was happening with 2 weeks old master.
>>>>> Anything change on that part of the code base recently ? Otherwise I need to debug.
>>>>
>>>> Nope.  If you have a cluster where it isn't stabilizing just turn
>>>> on debug bluestore = 20 for a minute and send me the log.
>>>>
>>>> sage
>>>> PLEASE NOTE: The information contained in this electronic mail
>>>> message is intended only for the use of the designated recipient(s)
>>>> named above. If the reader of this message is not the intended
>>>> recipient, you are hereby notified that you have received this
>>>> message in error and that any review, dissemination, distribution,
>>>> or copying of this message is strictly prohibited. If you have
>>>> received this communication in error, please notify the sender by
>>>> telephone or e-mail (as shown above) immediately and destroy any
>>>> and all copies of this message in your possession (whether hard copies or electronically stored copies).
>>>>
>>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>>> in the body of a message to majordomo@vger.kernel.org More majordomo
>>> info at  http://vger.kernel.org/majordomo-info.html
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in the body of a message to majordomo@vger.kernel.org More majordomo
> info at  http://vger.kernel.org/majordomo-info.html
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).