What is omap

What is omap

[Pete Zaitcev]

Dear All:

I am a complete beginner in Ceph and due to various circumstances I became
curious about omap. I made some quick web searches and read some random
writings about it, but I still have basic questions.

 - If I had to refer to omap in a doc/foo.rst, is there a canonical
   web-page, blog post, mailing list thread, or a (white)paper that
   serves to explain what it is?

   So far the best explanation I found was in ceph-devel archives
   by Gregory Farnum.

 - Every casual explanation I found presumes that "an omap" (a set
   of K/V) is associated with an object. But it is not physically in
   the object. So, is there a free-standing omap (set of keys)?
   Or an omap associated with something else, like a pool?

 - Greg says " The on-disk layout for these is more complicated
   (read about leveldb if you're interested)". Fair enough... But would
   someone be willing to shortcut this for me by explaining how this
   works in concert with OSD. In particular, does OSD replicate
   the files that LevelDB uses (like, for instance, Swift replicates
   SQLite files), or does it rely on LevelDB's own replication?
   Or perhaps OSD replicates upper level view of K/V provided by
   LevelDB in a node and ignores its actual on-disk layout?

Thanks in advance,
-- Pete

Re: What is omap

[Gregory Farnum]

[ Re-send to include the ceph-devel list; sorry for the double mail. ]

On Thu, Jun 25, 2015 at 6:47 AM, Pete Zaitcev <zaitcev@redhat.com> wrote:
> Dear All:
>
> I am a complete beginner in Ceph and due to various circumstances I became
> curious about omap. I made some quick web searches and read some random
> writings about it, but I still have basic questions.
>
>  - If I had to refer to omap in a doc/foo.rst, is there a canonical
>    web-page, blog post, mailing list thread, or a (white)paper that
>    serves to explain what it is?
>
>    So far the best explanation I found was in ceph-devel archives
>    by Gregory Farnum.
>
>  - Every casual explanation I found presumes that "an omap" (a set
>    of K/V) is associated with an object. But it is not physically in
>    the object. So, is there a free-standing omap (set of keys)?
>    Or an omap associated with something else, like a pool?
>
>  - Greg says " The on-disk layout for these is more complicated
>    (read about leveldb if you're interested)". Fair enough... But would
>    someone be willing to shortcut this for me by explaining how this
>    works in concert with OSD. In particular, does OSD replicate
>    the files that LevelDB uses (like, for instance, Swift replicates
>    SQLite files), or does it rely on LevelDB's own replication?
>    Or perhaps OSD replicates upper level view of K/V provided by
>    LevelDB in a node and ignores its actual on-disk layout?

Each object consists of three different data storage areas, all of
which are 100% optional: the "bundle of bits" object data, the object
xattrs, and the object omap key-value store. It might help you to
think of each of these as a different kind of resource fork or stream.

The object data is most commonly stored as a single file in a local
filesystem on the OSD; the xattrs are most commonly stored as xattrs
on that file, and the omap is most commonly stored within a leveldb
instance. Each OSD has its *own* local leveldb where all that data
goes; there's no cross-OSD LevelDB replication or communication. We
tell you that the omap data is stored in LevelDB to give you an idea
what kind of data is good to put in it, but we don't want you to get
too hung up on the fact that it's using LevelDB — it could change in
the future! omap is just an optimized region for storing key-value
data.
-Greg
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: What is omap

[Pete Zaitcev]

On Fri, 26 Jun 2015 14:48:15 +0100
Gregory Farnum <greg@gregs42.com> wrote:

> Each object consists of three different data storage areas, all of
> which are 100% optional: the "bundle of bits" object data, the object
> xattrs, and the object omap key-value store.

Thanks for the explanation. This was unexpected. I thought an object had
one K/V namespace associated with it, with xattrs and omap being two
implementations of it, switcheable depending on OSD configuration.

> Each OSD has its *own* local leveldb where all that data
> goes; there's no cross-OSD LevelDB replication or communication.

Okay, that makes sense.

So, do I understand right that we don't have a document that
explains the above? All I want to write is "the list of buckets
is kept in omap of object such and such; see URL FOO for explanation
of omap".

-- Pete

Re: What is omap

[Gregory Farnum]

On Fri, Jun 26, 2015 at 8:02 PM, Pete Zaitcev <zaitcev@redhat.com> wrote:
> On Fri, 26 Jun 2015 14:48:15 +0100
> Gregory Farnum <greg@gregs42.com> wrote:
>
>> Each object consists of three different data storage areas, all of
>> which are 100% optional: the "bundle of bits" object data, the object
>> xattrs, and the object omap key-value store.
>
> Thanks for the explanation. This was unexpected. I thought an object had
> one K/V namespace associated with it, with xattrs and omap being two
> implementations of it, switcheable depending on OSD configuration.

Nope. Internally the xattrs are *sometimes* stored in leveldb,
but they're not in the omap namespace. Being in leveldb is one of the
things that has changed you shouldn't worry about; it was in response
to local filesystem limitations and we handle it without any
visibility on the client side (except occasional changes in access
latency).

>
>> Each OSD has its *own* local leveldb where all that data
>> goes; there's no cross-OSD LevelDB replication or communication.
>
> Okay, that makes sense.
>
> So, do I understand right that we don't have a document that
> explains the above? All I want to write is "the list of buckets
> is kept in omap of object such and such; see URL FOO for explanation
> of omap".

...I have no idea if there's a document about that. Probably not; we
don't have a ton of client developer documentation right now. :(
-Greg