Documentation on device-mapper and friends

Documentation on device-mapper and friends

[Kumar Amit Mehta]

I'm looking for information on device-mapper, the kernel space utility for 
Logical Volume Management (LVM2). It seems that the relevant code resides under
drivers/md and a lot of other information is under Documentation/device-mapper/
That's fine, but is there any other document that gives more finer details about
the device-mapper. I've also recently subsribed to the device-mapper mailing
list.

-Amit

Documentation on device-mapper and friends

[Greg Freemyer]

Kumar Amit Mehta <gmate.amit@gmail.com> wrote:

>I'm looking for information on device-mapper, the kernel space utility
>for 
>Logical Volume Management (LVM2). It seems that the relevant code
>resides under
>drivers/md and a lot of other information is under
>Documentation/device-mapper/
>That's fine, but is there any other document that gives more finer
>details about
>the device-mapper. I've also recently subsribed to the device-mapper
>mailing
>list.
>
>-Amit

A nice diagram of the overall storage subsystem is at http://www.thomas-krenn.com/en/oss/linux-io-stack-diagram.html

Dm is just a single block in it, but it can help to see where it fits in overall.

Btw: that diagram doesn't show the legacy ata driver that creates /dev/hdx style devices.  Has that been dropped while I wasn't paying attention?  I haven't used it in years, but I thought it was still used on embedded systems.

Greg
-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

Documentation on device-mapper and friends

[amit mehta]

On Sun, Apr 28, 2013 at 5:24 PM, Greg Freemyer <greg.freemyer@gmail.com> wrote:
> A nice diagram of the overall storage subsystem is at http://www.thomas-krenn.com/en/oss/linux-io-stack-diagram.html
>
> Dm is just a single block in it, but it can help to see where it fits in overall.
>
> Btw: that diagram doesn't show the legacy ata driver that creates /dev/hdx style devices.  Has that been dropped while I wasn't paying attention?  I haven't used it in years, but I thought it was still used on embedded systems.
>

Thank you for sharing the link, but I'm looking for more
detailed information on I/O stack in Linux, dm-mapper and
multipath in particular.

Documentation on device-mapper and friends

[Greg Freemyer]

On Mon, Apr 29, 2013 at 12:51 PM, amit mehta <gmate.amit@gmail.com> wrote:
> On Sun, Apr 28, 2013 at 5:24 PM, Greg Freemyer <greg.freemyer@gmail.com> wrote:
>> A nice diagram of the overall storage subsystem is at http://www.thomas-krenn.com/en/oss/linux-io-stack-diagram.html
>>
>> Dm is just a single block in it, but it can help to see where it fits in overall.
>>
>> Btw: that diagram doesn't show the legacy ata driver that creates /dev/hdx style devices.  Has that been dropped while I wasn't paying attention?  I haven't used it in years, but I thought it was still used on embedded systems.
>>
>
> Thank you for sharing the link, but I'm looking for more
> detailed information on I/O stack in Linux, dm-mapper and
> multipath in particular.

I actually assumed others would step in.  I haven't seen any dm docs
outside the kernel tree, but I haven't done and dm work.

I guess you know dm-cache just went into the kernel:
http://lwn.net/Articles/540996/

The primary use case is to use SSD (or similar fast storage) as a
cache for a rotating disk.

I assume you found this very old (and simple) article, but I doubt it
is useful 8 years later: http://lwn.net/Articles/124703/

For more useful links, I often go to wikipedia and follow the links:

You should find:

multi-path faq: http://christophe.varoqui.free.fr/faq.html

multi-path policies: http://christophe.varoqui.free.fr/policies.html

multi-path diagrams: http://christophe.varoqui.free.fr/graphics/

multi-path reference: http://christophe.varoqui.free.fr/refbook.html

Greg

Documentation on device-mapper and friends

[Anatol Pomozov]

Hi

On Mon, Apr 29, 2013 at 9:51 AM, amit mehta <gmate.amit@gmail.com> wrote:
> On Sun, Apr 28, 2013 at 5:24 PM, Greg Freemyer <greg.freemyer@gmail.com> wrote:
>> A nice diagram of the overall storage subsystem is at http://www.thomas-krenn.com/en/oss/linux-io-stack-diagram.html
>>
>> Dm is just a single block in it, but it can help to see where it fits in overall.
>>
>> Btw: that diagram doesn't show the legacy ata driver that creates /dev/hdx style devices.  Has that been dropped while I wasn't paying attention?  I haven't used it in years, but I thought it was still used on embedded systems.
>>
>
> Thank you for sharing the link, but I'm looking for more
> detailed information on I/O stack in Linux, dm-mapper and
> multipath in particular.

Some docs about multipath can be found here

http://www.sourceware.org/lvm2/wiki/MultipathUsageGuide
http://christophe.varoqui.free.fr/refbook.html

The userspace part for tools is here
http://sourceware.org/lvm2/

Documentation on device-mapper and friends

[Gaurav Mahajan]

Hi Amit,

I had compiled some notes on my blog.
Here are some links on writing your own device mapper target.
http://techgmm.blogspot.in/p/writing-your-own-device-mapper-target.html

Concept of device mapper target.
http://techgmm.blogspot.in/p/device-mapper-layer-explored-every.html

Thanks,
Gaurav.


On Tue, Apr 30, 2013 at 5:05 AM, Anatol Pomozov <anatol.pomozov@gmail.com>wrote:

> Hi
>
> On Mon, Apr 29, 2013 at 9:51 AM, amit mehta <gmate.amit@gmail.com> wrote:
> > On Sun, Apr 28, 2013 at 5:24 PM, Greg Freemyer <greg.freemyer@gmail.com>
> wrote:
> >> A nice diagram of the overall storage subsystem is at
> http://www.thomas-krenn.com/en/oss/linux-io-stack-diagram.html
> >>
> >> Dm is just a single block in it, but it can help to see where it fits
> in overall.
> >>
> >> Btw: that diagram doesn't show the legacy ata driver that creates
> /dev/hdx style devices.  Has that been dropped while I wasn't paying
> attention?  I haven't used it in years, but I thought it was still used on
> embedded systems.
> >>
> >
> > Thank you for sharing the link, but I'm looking for more
> > detailed information on I/O stack in Linux, dm-mapper and
> > multipath in particular.
>
> Some docs about multipath can be found here
>
> http://www.sourceware.org/lvm2/wiki/MultipathUsageGuide
> http://christophe.varoqui.free.fr/refbook.html
>
> The userspace part for tools is here
> http://sourceware.org/lvm2/
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20130430/ff8fcb5f/attachment.html 

Documentation on device-mapper and friends

[neha naik]

Hi Gaurav,
 I went through your blog and it is really informative. But after reading
that i realized that i have a question:
  If I want to write a block device driver which is going to sit on lvm
(and do some functionality on top of it) then should i go for the block
device driver api
  or write it as a device mapper target. What are the
advantages/disadvantages of both the approaches.

Regards,
Neha

On Tue, Apr 30, 2013 at 4:24 AM, Gaurav Mahajan <gauravmahajan2007@gmail.com
> wrote:

> Hi Amit,
>
> I had compiled some notes on my blog.
> Here are some links on writing your own device mapper target.
> http://techgmm.blogspot.in/p/writing-your-own-device-mapper-target.html
>
> Concept of device mapper target.
> http://techgmm.blogspot.in/p/device-mapper-layer-explored-every.html
>
> Thanks,
> Gaurav.
>
>
> On Tue, Apr 30, 2013 at 5:05 AM, Anatol Pomozov <anatol.pomozov@gmail.com>wrote:
>
>> Hi
>>
>> On Mon, Apr 29, 2013 at 9:51 AM, amit mehta <gmate.amit@gmail.com> wrote:
>> > On Sun, Apr 28, 2013 at 5:24 PM, Greg Freemyer <greg.freemyer@gmail.com>
>> wrote:
>> >> A nice diagram of the overall storage subsystem is at
>> http://www.thomas-krenn.com/en/oss/linux-io-stack-diagram.html
>> >>
>> >> Dm is just a single block in it, but it can help to see where it fits
>> in overall.
>> >>
>> >> Btw: that diagram doesn't show the legacy ata driver that creates
>> /dev/hdx style devices.  Has that been dropped while I wasn't paying
>> attention?  I haven't used it in years, but I thought it was still used on
>> embedded systems.
>> >>
>> >
>> > Thank you for sharing the link, but I'm looking for more
>> > detailed information on I/O stack in Linux, dm-mapper and
>> > multipath in particular.
>>
>> Some docs about multipath can be found here
>>
>> http://www.sourceware.org/lvm2/wiki/MultipathUsageGuide
>> http://christophe.varoqui.free.fr/refbook.html
>>
>> The userspace part for tools is here
>> http://sourceware.org/lvm2/
>>
>> _______________________________________________
>> Kernelnewbies mailing list
>> Kernelnewbies at kernelnewbies.org
>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>>
>
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20130501/ca68038c/attachment.html 

Documentation on device-mapper and friends

[Gaurav Mahajan]

Hi  Neha,

LVM uses device mapper. Advantages of using device mapper is that you can
stack different dm-targets on each other.
I am really not aware of block device drivers.

May be Greg can help us understand the actual pros and cons.

Thanks,
Gaurav


On Wed, May 1, 2013 at 9:45 PM, neha naik <nehanaik27@gmail.com> wrote:

> Hi Gaurav,
>  I went through your blog and it is really informative. But after reading
> that i realized that i have a question:
>   If I want to write a block device driver which is going to sit on lvm
> (and do some functionality on top of it) then should i go for the block
> device driver api
>   or write it as a device mapper target. What are the
> advantages/disadvantages of both the approaches.
>
> Regards,
> Neha
>
>
> On Tue, Apr 30, 2013 at 4:24 AM, Gaurav Mahajan <
> gauravmahajan2007 at gmail.com> wrote:
>
>> Hi Amit,
>>
>> I had compiled some notes on my blog.
>> Here are some links on writing your own device mapper target.
>> http://techgmm.blogspot.in/p/writing-your-own-device-mapper-target.html
>>
>> Concept of device mapper target.
>> http://techgmm.blogspot.in/p/device-mapper-layer-explored-every.html
>>
>> Thanks,
>> Gaurav.
>>
>>
>> On Tue, Apr 30, 2013 at 5:05 AM, Anatol Pomozov <anatol.pomozov@gmail.com
>> > wrote:
>>
>>> Hi
>>>
>>> On Mon, Apr 29, 2013 at 9:51 AM, amit mehta <gmate.amit@gmail.com>
>>> wrote:
>>> > On Sun, Apr 28, 2013 at 5:24 PM, Greg Freemyer <
>>> greg.freemyer at gmail.com> wrote:
>>> >> A nice diagram of the overall storage subsystem is at
>>> http://www.thomas-krenn.com/en/oss/linux-io-stack-diagram.html
>>> >>
>>> >> Dm is just a single block in it, but it can help to see where it fits
>>> in overall.
>>> >>
>>> >> Btw: that diagram doesn't show the legacy ata driver that creates
>>> /dev/hdx style devices.  Has that been dropped while I wasn't paying
>>> attention?  I haven't used it in years, but I thought it was still used on
>>> embedded systems.
>>> >>
>>> >
>>> > Thank you for sharing the link, but I'm looking for more
>>> > detailed information on I/O stack in Linux, dm-mapper and
>>> > multipath in particular.
>>>
>>> Some docs about multipath can be found here
>>>
>>> http://www.sourceware.org/lvm2/wiki/MultipathUsageGuide
>>> http://christophe.varoqui.free.fr/refbook.html
>>>
>>> The userspace part for tools is here
>>> http://sourceware.org/lvm2/
>>>
>>> _______________________________________________
>>> Kernelnewbies mailing list
>>> Kernelnewbies at kernelnewbies.org
>>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>>>
>>
>>
>> _______________________________________________
>> Kernelnewbies mailing list
>> Kernelnewbies at kernelnewbies.org
>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20130507/101dbe24/attachment-0001.html 

Documentation on device-mapper and friends

[Greg Freemyer]

The block layers can be layered both ways.  DM is the newer
infrastructure and was created in the early days of 2.6

If what I was writing could fit into a dm-target, that is what I would do.

There are significant projects like drbd and mdraid that are not
dm-targets, but I think their is a long term goal to incorporate
mdraid's functionality at a minimum into dm.  I doubt drbd is ever
moved to dm.  It is just too big of a project and in use in lots of
production server environments.

Greg

On Tue, May 7, 2013 at 1:46 AM, Gaurav Mahajan
<gauravmahajan2007@gmail.com> wrote:
> Hi  Neha,
>
> LVM uses device mapper. Advantages of using device mapper is that you can
> stack different dm-targets on each other.
> I am really not aware of block device drivers.
>
> May be Greg can help us understand the actual pros and cons.
>
> Thanks,
> Gaurav
>
>
> On Wed, May 1, 2013 at 9:45 PM, neha naik <nehanaik27@gmail.com> wrote:
>>
>> Hi Gaurav,
>>  I went through your blog and it is really informative. But after reading
>> that i realized that i have a question:
>>   If I want to write a block device driver which is going to sit on lvm
>> (and do some functionality on top of it) then should i go for the block
>> device driver api
>>   or write it as a device mapper target. What are the
>> advantages/disadvantages of both the approaches.
>>
>> Regards,
>> Neha
>>
>>
>> On Tue, Apr 30, 2013 at 4:24 AM, Gaurav Mahajan
>> <gauravmahajan2007@gmail.com> wrote:
>>>
>>> Hi Amit,
>>>
>>> I had compiled some notes on my blog.
>>> Here are some links on writing your own device mapper target.
>>> http://techgmm.blogspot.in/p/writing-your-own-device-mapper-target.html
>>>
>>> Concept of device mapper target.
>>> http://techgmm.blogspot.in/p/device-mapper-layer-explored-every.html
>>>
>>> Thanks,
>>> Gaurav.
>>>
>>>
>>> On Tue, Apr 30, 2013 at 5:05 AM, Anatol Pomozov
>>> <anatol.pomozov@gmail.com> wrote:
>>>>
>>>> Hi
>>>>
>>>> On Mon, Apr 29, 2013 at 9:51 AM, amit mehta <gmate.amit@gmail.com>
>>>> wrote:
>>>> > On Sun, Apr 28, 2013 at 5:24 PM, Greg Freemyer
>>>> > <greg.freemyer@gmail.com> wrote:
>>>> >> A nice diagram of the overall storage subsystem is at
>>>> >> http://www.thomas-krenn.com/en/oss/linux-io-stack-diagram.html
>>>> >>
>>>> >> Dm is just a single block in it, but it can help to see where it fits
>>>> >> in overall.
>>>> >>
>>>> >> Btw: that diagram doesn't show the legacy ata driver that creates
>>>> >> /dev/hdx style devices.  Has that been dropped while I wasn't paying
>>>> >> attention?  I haven't used it in years, but I thought it was still used on
>>>> >> embedded systems.
>>>> >>
>>>> >
>>>> > Thank you for sharing the link, but I'm looking for more
>>>> > detailed information on I/O stack in Linux, dm-mapper and
>>>> > multipath in particular.
>>>>
>>>> Some docs about multipath can be found here
>>>>
>>>> http://www.sourceware.org/lvm2/wiki/MultipathUsageGuide
>>>> http://christophe.varoqui.free.fr/refbook.html
>>>>
>>>> The userspace part for tools is here
>>>> http://sourceware.org/lvm2/
>>>>
>>>> _______________________________________________
>>>> Kernelnewbies mailing list
>>>> Kernelnewbies at kernelnewbies.org
>>>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>>>
>>>
>>>
>>> _______________________________________________
>>> Kernelnewbies mailing list
>>> Kernelnewbies at kernelnewbies.org
>>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>>>
>>
>

Documentation on device-mapper and friends

[neha naik]

Hi Greg,
  Thanks for the information. I have another question :).
  Is there some less flexibility if we use device mapper target?
  For example in block device driver you can use the api such that it won't
use the OS io scheduler, so the io comes directly to the block device
driver through the   'make_request' call. With the device mapper i don't
think that happens(looking at the api calls). Does this mean that stuff
like io scheduling. barrier control etc is done
 by the device mapper itself and we can focus only on 'mapping' the io.

Regards,
Neha

On Wed, May 8, 2013 at 8:11 AM, Greg Freemyer <greg.freemyer@gmail.com>wrote:

> The block layers can be layered both ways.  DM is the newer
> infrastructure and was created in the early days of 2.6
>
> If what I was writing could fit into a dm-target, that is what I would do.
>
> There are significant projects like drbd and mdraid that are not
> dm-targets, but I think their is a long term goal to incorporate
> mdraid's functionality at a minimum into dm.  I doubt drbd is ever
> moved to dm.  It is just too big of a project and in use in lots of
> production server environments.
>
> Greg
>
> On Tue, May 7, 2013 at 1:46 AM, Gaurav Mahajan
> <gauravmahajan2007@gmail.com> wrote:
> > Hi  Neha,
> >
> > LVM uses device mapper. Advantages of using device mapper is that you can
> > stack different dm-targets on each other.
> > I am really not aware of block device drivers.
> >
> > May be Greg can help us understand the actual pros and cons.
> >
> > Thanks,
> > Gaurav
> >
> >
> > On Wed, May 1, 2013 at 9:45 PM, neha naik <nehanaik27@gmail.com> wrote:
> >>
> >> Hi Gaurav,
> >>  I went through your blog and it is really informative. But after
> reading
> >> that i realized that i have a question:
> >>   If I want to write a block device driver which is going to sit on lvm
> >> (and do some functionality on top of it) then should i go for the block
> >> device driver api
> >>   or write it as a device mapper target. What are the
> >> advantages/disadvantages of both the approaches.
> >>
> >> Regards,
> >> Neha
> >>
> >>
> >> On Tue, Apr 30, 2013 at 4:24 AM, Gaurav Mahajan
> >> <gauravmahajan2007@gmail.com> wrote:
> >>>
> >>> Hi Amit,
> >>>
> >>> I had compiled some notes on my blog.
> >>> Here are some links on writing your own device mapper target.
> >>>
> http://techgmm.blogspot.in/p/writing-your-own-device-mapper-target.html
> >>>
> >>> Concept of device mapper target.
> >>> http://techgmm.blogspot.in/p/device-mapper-layer-explored-every.html
> >>>
> >>> Thanks,
> >>> Gaurav.
> >>>
> >>>
> >>> On Tue, Apr 30, 2013 at 5:05 AM, Anatol Pomozov
> >>> <anatol.pomozov@gmail.com> wrote:
> >>>>
> >>>> Hi
> >>>>
> >>>> On Mon, Apr 29, 2013 at 9:51 AM, amit mehta <gmate.amit@gmail.com>
> >>>> wrote:
> >>>> > On Sun, Apr 28, 2013 at 5:24 PM, Greg Freemyer
> >>>> > <greg.freemyer@gmail.com> wrote:
> >>>> >> A nice diagram of the overall storage subsystem is at
> >>>> >> http://www.thomas-krenn.com/en/oss/linux-io-stack-diagram.html
> >>>> >>
> >>>> >> Dm is just a single block in it, but it can help to see where it
> fits
> >>>> >> in overall.
> >>>> >>
> >>>> >> Btw: that diagram doesn't show the legacy ata driver that creates
> >>>> >> /dev/hdx style devices.  Has that been dropped while I wasn't
> paying
> >>>> >> attention?  I haven't used it in years, but I thought it was still
> used on
> >>>> >> embedded systems.
> >>>> >>
> >>>> >
> >>>> > Thank you for sharing the link, but I'm looking for more
> >>>> > detailed information on I/O stack in Linux, dm-mapper and
> >>>> > multipath in particular.
> >>>>
> >>>> Some docs about multipath can be found here
> >>>>
> >>>> http://www.sourceware.org/lvm2/wiki/MultipathUsageGuide
> >>>> http://christophe.varoqui.free.fr/refbook.html
> >>>>
> >>>> The userspace part for tools is here
> >>>> http://sourceware.org/lvm2/
> >>>>
> >>>> _______________________________________________
> >>>> Kernelnewbies mailing list
> >>>> Kernelnewbies at kernelnewbies.org
> >>>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> >>>
> >>>
> >>>
> >>> _______________________________________________
> >>> Kernelnewbies mailing list
> >>> Kernelnewbies at kernelnewbies.org
> >>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> >>>
> >>
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20130508/a1bb35d8/attachment.html 

Documentation on device-mapper and friends

[Greg Freemyer]

You should take this to the device mapper list, but I'll try here.

For lurkers, this drawing may be helpful:
http://www.thomas-krenn.com/en/oss/linux-io-stack-diagram/linux-io-stack-diagram_v1.0.pdf

On Wed, May 8, 2013 at 11:15 AM, neha naik <nehanaik27@gmail.com> wrote:
> Hi Greg,
>   Thanks for the information. I have another question :).
>   Is there some less flexibility if we use device mapper target?
>   For example in block device driver you can use the api such that it won't
> use the OS io scheduler, so the io comes directly to the block device driver
> through the   'make_request' call. With the device mapper i don't think that
> happens(looking at the api calls).

I believe that is correct and one of the reasons DRBD is not part of
DM.  DRBD has various modes including those where it guarantees the
I/Os on both the main target and on the replicated target hit in the
exact same order.  I don't believe that DM can be used to totally
control disk I/O.  It is meant to be stackable, so I think you lose
some exact control.

> Does this mean that stuff like io
> scheduling. barrier control etc is done
>  by the device mapper itself and we can focus only on 'mapping' the io.

As shown in the diagram linked above, DM sits above the i/o schedulers
so you should not have to worry about it.  If you want to play with
schedulers, I think that should be done outside of DM.

Thus I believe you can ignore barriers/scheduling UNLESS you create a
target that needs special barrier/scheduling control.  The obvious
example of something needing that is raid5.  If you get a barrier that
forces data out to a single disk in a raid, you MUST ensure that the
raid checksum is calculated and written out prior to calling the
barrier complete.  That is going to take special handling no matter
what you do.

It's been a couple years since I dug into raid 5/6 as relates to
barriers, but it used to be that code simply didn't do the right thing
in mdraid and DM did not support raid 5/6, so yes the coders could
ignore it, but they created broken logic when they did.

Greg