Re: RBD behavior for reads to a volume with no data written

Re: RBD behavior for reads to a volume with no data written

[Prashant Murthy]

[-- Attachment #1.1: Type: text/plain, Size: 908 bytes --]

I wanted to add that I was particularly interested about the behavior with
filestore, but was also curious how this works on bluestore.

Prashant

On Mon, May 1, 2017 at 10:04 PM, Prashant Murthy <pmurthy-7yRcRoh4mDtoiYX5Tdu9fQ@public.gmane.org>
wrote:

> Hi all,
>
> I was wondering what happens when reads are issued to an RBD device with
> no previously written data. Can somebody explain how such requests flow
> from rbd (client) into OSDs and whether any of these reads would hit the
> disks at all or whether OSD metadata would recognize that there is no data
> at the offsets requested and returns a bunch of zeros back to the client?
>
> Thanks,
> Prashant
>
> --
> Prashant Murthy
> Sr Director, Software Engineering | Salesforce
> Mobile: 919-961-3041 <(919)%20961-3041>
>
>
> --
>



-- 
Prashant Murthy
Sr Director, Software Engineering | Salesforce
Mobile: 919-961-3041 <(919)%20961-3041>


--

[-- Attachment #1.2: Type: text/html, Size: 2392 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: RBD behavior for reads to a volume with no data written

[Jason Dillaman]

[-- Attachment #1.1: Type: text/plain, Size: 2675 bytes --]

If the RBD object map feature is enabled, the read request would never even
be sent to the OSD if the client knows the backing object doesn't exist.
However, if the object map feature is disabled, the read request will be
sent to the OSD.

The OSD isn't my area of expertise, but I can try to explain what occurs to
the best of my knowledge. There is a small in-memory cache for object
contexts with the OSD PG -- which includes a whiteout flag to indicate the
object is deleted. I believe that the whiteout flag is only really used on
a cache tier to avoid having to attempt to promote a known non-existent
object. Therefore, in the common case the OSD would query the non-existent
object from the object store (FileStore or BlueStore).

In FileStore, it will attempt to open the associated backing file for
object. If the necessary dentries are cached in the kernel, I'd expect that
the -ENOENT error would be bubbled back to RBD w/o a disk hit. Otherwise,
the kernel would need to read the associated dentries from disk to
determine that the object is missing.

In BlueStore, there is another in-memory cache for onodes that can quickly
detect a missing object. If the object isn't in the cache, the associated
onode will be looked up within the backing RocksDB. If the RocksDB metadata
scan for the object's onode fails since the object is missing, the -ENOENT
error would be bubbled back to the client.



On Tue, May 2, 2017 at 1:24 PM, Prashant Murthy <pmurthy-7yRcRoh4mDtoiYX5Tdu9fQ@public.gmane.org>
wrote:

> I wanted to add that I was particularly interested about the behavior with
> filestore, but was also curious how this works on bluestore.
>
> Prashant
>
> On Mon, May 1, 2017 at 10:04 PM, Prashant Murthy <pmurthy-7yRcRoh4mDtoiYX5Tdu9fQ@public.gmane.org>
> wrote:
>
>> Hi all,
>>
>> I was wondering what happens when reads are issued to an RBD device with
>> no previously written data. Can somebody explain how such requests flow
>> from rbd (client) into OSDs and whether any of these reads would hit the
>> disks at all or whether OSD metadata would recognize that there is no data
>> at the offsets requested and returns a bunch of zeros back to the client?
>>
>> Thanks,
>> Prashant
>>
>> --
>> Prashant Murthy
>> Sr Director, Software Engineering | Salesforce
>> Mobile: 919-961-3041 <(919)%20961-3041>
>>
>>
>> --
>>
>
>
>
> --
> Prashant Murthy
> Sr Director, Software Engineering | Salesforce
> Mobile: 919-961-3041 <(919)%20961-3041>
>
>
> --
>
> _______________________________________________
> ceph-users mailing list
> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>


-- 
Jason

[-- Attachment #1.2: Type: text/html, Size: 5088 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: RBD behavior for reads to a volume with no data written

[Prashant Murthy]

[-- Attachment #1.1: Type: text/plain, Size: 3204 bytes --]

Thanks for the detailed explanation, Jason. It makes sense that such
operations would end up being a few metadata lookups only (and the metadata
lookups will hit the disk only if they are not cached in-memory).

Prashant

On Tue, May 2, 2017 at 11:29 AM, Jason Dillaman <jdillama-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:

> If the RBD object map feature is enabled, the read request would never
> even be sent to the OSD if the client knows the backing object doesn't
> exist. However, if the object map feature is disabled, the read request
> will be sent to the OSD.
>
> The OSD isn't my area of expertise, but I can try to explain what occurs
> to the best of my knowledge. There is a small in-memory cache for object
> contexts with the OSD PG -- which includes a whiteout flag to indicate the
> object is deleted. I believe that the whiteout flag is only really used on
> a cache tier to avoid having to attempt to promote a known non-existent
> object. Therefore, in the common case the OSD would query the non-existent
> object from the object store (FileStore or BlueStore).
>
> In FileStore, it will attempt to open the associated backing file for
> object. If the necessary dentries are cached in the kernel, I'd expect that
> the -ENOENT error would be bubbled back to RBD w/o a disk hit. Otherwise,
> the kernel would need to read the associated dentries from disk to
> determine that the object is missing.
>
> In BlueStore, there is another in-memory cache for onodes that can quickly
> detect a missing object. If the object isn't in the cache, the associated
> onode will be looked up within the backing RocksDB. If the RocksDB metadata
> scan for the object's onode fails since the object is missing, the -ENOENT
> error would be bubbled back to the client.
>
>
>
> On Tue, May 2, 2017 at 1:24 PM, Prashant Murthy <pmurthy-7yRcRoh4mDtoiYX5Tdu9fQ@public.gmane.org>
> wrote:
>
>> I wanted to add that I was particularly interested about the behavior
>> with filestore, but was also curious how this works on bluestore.
>>
>> Prashant
>>
>> On Mon, May 1, 2017 at 10:04 PM, Prashant Murthy <pmurthy-7yRcRoh4mDtoiYX5Tdu9fQ@public.gmane.org>
>> wrote:
>>
>>> Hi all,
>>>
>>> I was wondering what happens when reads are issued to an RBD device with
>>> no previously written data. Can somebody explain how such requests flow
>>> from rbd (client) into OSDs and whether any of these reads would hit the
>>> disks at all or whether OSD metadata would recognize that there is no data
>>> at the offsets requested and returns a bunch of zeros back to the client?
>>>
>>> Thanks,
>>> Prashant
>>>
>>> --
>>> Prashant Murthy
>>> Sr Director, Software Engineering | Salesforce
>>> Mobile: 919-961-3041 <(919)%20961-3041>
>>>
>>>
>>> --
>>>
>>
>>
>>
>> --
>> Prashant Murthy
>> Sr Director, Software Engineering | Salesforce
>> Mobile: 919-961-3041 <(919)%20961-3041>
>>
>>
>> --
>>
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>
>
> --
> Jason
>



-- 
Prashant Murthy
Sr Director, Software Engineering | Salesforce
Mobile: 919-961-3041


--

[-- Attachment #1.2: Type: text/html, Size: 6516 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com