[linux-lvm] Does LVM RAID1 have TRIM support?

[linux-lvm] Does LVM RAID1 have TRIM support?

[Jarkko Oranen]

Hello

Yesterday I experimented a bit with my RAID configuration on a pair of
SSDs, and it seems that LVM's native RAID does not have TRIM support...
At least, when I try to run fstrim manually, it complains even though
issue_discards is enabled. Plain LVs on top of an MD RAID PV do work, of
course.

Am I perhaps missing some configuration, or do RAID1 logical volumes
simply not have support for TRIM yet? I'm running a fairly recent kernel
(3.15.8) and lvm version says this:

  LVM version:     2.02.106(2) (2014-04-10)
  Library version: 1.02.85 (2014-04-10)
  Driver version:  4.27.0

As an aside, can anyone point me to documentation or other resources
about the pros and cons of LVM native RAID1 setup (which I understand
uses MD RAID internally?) vs. MD RAID PV + LVM. It seems I might be able
to save some SSD space and only mirror the LVs I actually need to keep
safe from crashes.

--
Jarkko Oranen

Re: [linux-lvm] Does LVM RAID1 have TRIM support?

[Brassow Jonathan]

On Aug 24, 2014, at 11:08 AM, Jarkko Oranen wrote:

> Hello
> 
> Yesterday I experimented a bit with my RAID configuration on a pair of
> SSDs, and it seems that LVM's native RAID does not have TRIM support...
> At least, when I try to run fstrim manually, it complains even though
> issue_discards is enabled. Plain LVs on top of an MD RAID PV do work, of
> course.
> 
> Am I perhaps missing some configuration, or do RAID1 logical volumes
> simply not have support for TRIM yet? I'm running a fairly recent kernel
> (3.15.8) and lvm version says this:
> 
>  LVM version:     2.02.106(2) (2014-04-10)
>  Library version: 1.02.85 (2014-04-10)
>  Driver version:  4.27.0

TRIM is not yet supported in LVM RAID.  However, if MD has a solid TRIM implementation, it should be simple to enable it for LVM.  (This is because the MD kernel modules are used to perform RAID for LVM.  There is only a thin wrapper layer (linux/drivers/md/dm-raid.c) in device-mapper used to set-up the device.)

> As an aside, can anyone point me to documentation or other resources
> about the pros and cons of LVM native RAID1 setup (which I understand
> uses MD RAID internally?) vs. MD RAID PV + LVM. It seems I might be able
> to save some SSD space and only mirror the LVs I actually need to keep
> safe from crashes.

I don't know if there is a specific list to point to out there, but I can give you a couple pros/cons.
PRO:
- use one volume manager instead of two
- LVM is better suited to creating devices of varying sizes - leaving spare capacity for snapshots, etc

CONS:
- no trim support with RAID through LVM (although, I'm not sure of the state in MD)
- no reshaping (changing from one RAID type to another) capability in LVM RAID.

 brassow

Re: [linux-lvm] Does LVM RAID1 have TRIM support?

[Jarkko Oranen]

On Tue, 2014-09-02 at 16:49 -0500, Brassow Jonathan wrote:
> On Aug 24, 2014, at 11:08 AM, Jarkko Oranen wrote:
> 
> > Hello
> > 
> > Yesterday I experimented a bit with my RAID configuration on a pair of
> > SSDs, and it seems that LVM's native RAID does not have TRIM support...
> > At least, when I try to run fstrim manually, it complains even though
> > issue_discards is enabled. Plain LVs on top of an MD RAID PV do work, of
> > course.
> > 
> > Am I perhaps missing some configuration, or do RAID1 logical volumes
> > simply not have support for TRIM yet? I'm running a fairly recent kernel
> > (3.15.8) and lvm version says this:
> > 
> >  LVM version:     2.02.106(2) (2014-04-10)
> >  Library version: 1.02.85 (2014-04-10)
> >  Driver version:  4.27.0
> 
> TRIM is not yet supported in LVM RAID.  However, if MD has a solid TRIM implementation, it should be simple to enable it for LVM.  (This is because the MD kernel modules are used to perform RAID for LVM.  There is only a thin wrapper layer (linux/drivers/md/dm-raid.c) in device-mapper used to set-up the device.)

Thanks for the reply! Unfortunately, "simple" is relative. I have no
experience with kernel hacking at all (or even much C programming,
though I can read it), but I took a look at the code anyway to try to
pinpoint the place that tells userspace the ioctl isn't supported. 

From my cursory browsing of the code I couldn't really tell how it
works, but for the most part it seemed like it *should* just work... It
does work just fine for linear volumes, and md/dm-linear.c doesn't seem
to do anything special to enable discard/FITRIM support, while discard
is specifically marked as supported in the md/linear.c and md/raid*.c
files if the underlying device supports it.

However, in dm.c there's the dm_blk_ioctl function which says the
following:
        /* We only support devices that have a single target */
        if (dm_table_get_num_targets(map) != 1)
                goto out;

Intuitively it looks like this could prevent a raid1 target from working
as it apparently consists of rmeta and rimage subcomponents, but I
couldn't determine for certain if my intuition is correct. I did some
testing, and discards work just fine on a linear volume spanning two
different partitions, while they fail on a "RAID" mirror of two similar
partitions. Clearly, two devices in a mapping are not a problem.

> 
> > As an aside, can anyone point me to documentation or other resources
> > about the pros and cons of LVM native RAID1 setup (which I understand
> > uses MD RAID internally?) vs. MD RAID PV + LVM. It seems I might be able
> > to save some SSD space and only mirror the LVs I actually need to keep
> > safe from crashes.
> 
> I don't know if there is a specific list to point to out there, but I can give you a couple pros/cons.
> PRO:
> - use one volume manager instead of two
> - LVM is better suited to creating devices of varying sizes - leaving spare capacity for snapshots, etc
> 
> CONS:
> - no trim support with RAID through LVM (although, I'm not sure of the state in MD)
> - no reshaping (changing from one RAID type to another) capability in LVM RAID.
> 
>  brassow
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Does LVM RAID1 have TRIM support?

[Brassow Jonathan]

On Sep 4, 2014, at 6:06 AM, Jarkko Oranen wrote:

>> TRIM is not yet supported in LVM RAID.  However, if MD has a solid TRIM implementation, it should be simple to enable it for LVM.  (This is because the MD kernel modules are used to perform RAID for LVM.  There is only a thin wrapper layer (linux/drivers/md/dm-raid.c) in device-mapper used to set-up the device.)
> 
> Thanks for the reply! Unfortunately, "simple" is relative.

By "simple", I mean that from our side all we should need to do is set 'num_discard_bios = 1' in dm-raid.c.  That is, if things are working fine in MD.  We could even make that conditional depending on the type of RAID (e.g. discards for RAID1, but not RAID5).  It could end up being that simple, but I need to know if MD has proper support for discards and all the kinks have been worked out.

 brassow

Re: [linux-lvm] Does LVM RAID1 have TRIM support?

[Jarkko Oranen]

On Mon, 2014-09-08 at 12:47 -0500, Brassow Jonathan wrote:
> On Sep 4, 2014, at 6:06 AM, Jarkko Oranen wrote:
> 
> >> TRIM is not yet supported in LVM RAID.  However, if MD has a solid TRIM implementation, it should be simple to enable it for LVM.  (This is because the MD kernel modules are used to perform RAID for LVM.  There is only a thin wrapper layer (linux/drivers/md/dm-raid.c) in device-mapper used to set-up the device.)
> > 
> > Thanks for the reply! Unfortunately, "simple" is relative.
> 
> By "simple", I mean that from our side all we should need to do is set 'num_discard_bios = 1' in dm-raid.c.  That is, if things are working fine in MD.  We could even make that conditional depending on the type of RAID (e.g. discards for RAID1, but not RAID5).  It could end up being that simple, but I need to know if MD has proper support for discards and all the kinks have been worked out.
> 

dm-raid.c already has         
 
  ti->num_discard_bios = 1;

in raid_ctr in the kernel source I'm looking at (3.15.6; not the most
recent, but it's what I happen to have locally available). As I said, to
me it looks like it should already work, but it doesn't, so I spent a
bit of time trying to understand why. 

Digging at it again, eg. lsblk -D seems to report discard information
identical to other discard-supporting block devices, suggesting that
discard passthrough should work fine, but the actual ioctl fails (when
testing via either fstrim or blkdiscard) with "not supported" for
whatever reason.

As far as I know, discard support in MD is stable, and has been so for a
long while; my current setup consists of linear logical volumes on top
of a software RAID PV created with mdadm, and I haven't had a single
issue with discards. It seems RAID1 discard support was merged in
January 2011 (git commit 5fc2ffeabb9ee0fc0e71ff16b49f34f0ed3d05b4), so
it's definitely not new.

--
Jarkko

Re: [linux-lvm] Does LVM RAID1 have TRIM support?

[Brassow Jonathan]

On Sep 8, 2014, at 1:47 PM, Jarkko Oranen wrote:

> On Mon, 2014-09-08 at 12:47 -0500, Brassow Jonathan wrote:
>> On Sep 4, 2014, at 6:06 AM, Jarkko Oranen wrote:
>> 
>>>> TRIM is not yet supported in LVM RAID.  However, if MD has a solid TRIM implementation, it should be simple to enable it for LVM.  (This is because the MD kernel modules are used to perform RAID for LVM.  There is only a thin wrapper layer (linux/drivers/md/dm-raid.c) in device-mapper used to set-up the device.)
>>> 
>>> Thanks for the reply! Unfortunately, "simple" is relative.
>> 
>> By "simple", I mean that from our side all we should need to do is set 'num_discard_bios = 1' in dm-raid.c.  That is, if things are working fine in MD.  We could even make that conditional depending on the type of RAID (e.g. discards for RAID1, but not RAID5).  It could end up being that simple, but I need to know if MD has proper support for discards and all the kinks have been worked out.
>> 
> 
> dm-raid.c already has         
> 
>  ti->num_discard_bios = 1;
> 
> in raid_ctr in the kernel source I'm looking at (3.15.6; not the most
> recent, but it's what I happen to have locally available). As I said, to
> me it looks like it should already work, but it doesn't, so I spent a
> bit of time trying to understand why. 
> 
> Digging at it again, eg. lsblk -D seems to report discard information
> identical to other discard-supporting block devices, suggesting that
> discard passthrough should work fine, but the actual ioctl fails (when
> testing via either fstrim or blkdiscard) with "not supported" for
> whatever reason.
> 
> As far as I know, discard support in MD is stable, and has been so for a
> long while; my current setup consists of linear logical volumes on top
> of a software RAID PV created with mdadm, and I haven't had a single
> issue with discards. It seems RAID1 discard support was merged in
> January 2011 (git commit 5fc2ffeabb9ee0fc0e71ff16b49f34f0ed3d05b4), so
> it's definitely not new.

dm-raid.c does not have that - dm-raid1.c does.  dm-raid1.c is a mirroring implementation specific to device-mapper.  It is used by LVM for pvmove functionality (and is otherwise being slowly phased out).  dm-raid.c is the device-mapper wrapper around the MD RAID personalities.  It does not have discard support.

I will ask Neil what the current state of discard support is in MD.  I see that "support" was added in Oct of 2012 (starting at commit 2ff8cc2), but I remember them having problems with discards + any resync operations.  If Neil says it is ok, I will turn it on.

 brassow

Re: [linux-lvm] Does LVM RAID1 have TRIM support?

[Jarkko Oranen]

On Thu, 2014-09-11 at 18:31 -0500, Brassow Jonathan wrote:

> dm-raid.c does not have that - dm-raid1.c does.  dm-raid1.c is a mirroring implementation specific to device-mapper.  It is used by LVM for pvmove functionality (and is otherwise being slowly phased out).  dm-raid.c is the device-mapper wrapper around the MD RAID personalities.  It does not have discard support.
> 
> I will ask Neil what the current state of discard support is in MD.  I see that "support" was added in Oct of 2012 (starting at commit 2ff8cc2), but I remember them having problems with discards + any resync operations.  If Neil says it is ok, I will turn it on.
> 

In the code that I'm looking at dm-raid.c definitely sets
num_discard_bios to 1 on something... As does dm-raid1.c. However,
Thanks for looking into it!


>  brassow
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Does LVM RAID1 have TRIM support?

[Brassow Jonathan]

On Sep 12, 2014, at 2:24 AM, Jarkko Oranen wrote:

> On Thu, 2014-09-11 at 18:31 -0500, Brassow Jonathan wrote:
> 
>> dm-raid.c does not have that - dm-raid1.c does.  dm-raid1.c is a mirroring implementation specific to device-mapper.  It is used by LVM for pvmove functionality (and is otherwise being slowly phased out).  dm-raid.c is the device-mapper wrapper around the MD RAID personalities.  It does not have discard support.
>> 
>> I will ask Neil what the current state of discard support is in MD.  I see that "support" was added in Oct of 2012 (starting at commit 2ff8cc2), but I remember them having problems with discards + any resync operations.  If Neil says it is ok, I will turn it on.
>> 
> 
> In the code that I'm looking at dm-raid.c definitely sets
> num_discard_bios to 1 on something... As does dm-raid1.c. However,
> Thanks for looking into it!

Heinz Mauelshagen is looking to get this included upstream now.  Thanks for the poke!  :)

Initial status query:
http://marc.info/?l=linux-raid&m=141047869821032&w=2

Patch posted for consideration:
https://www.redhat.com/archives/dm-devel/2014-September/msg00126.html

 brassow