All of lore.kernel.org
 help / color / mirror / Atom feed
* how to refresh LV to apply fstrim online
@ 2016-10-20  8:52 Gianluca Cecchi
  2016-10-20 11:40 ` Zdenek Kabelac
  0 siblings, 1 reply; 6+ messages in thread
From: Gianluca Cecchi @ 2016-10-20  8:52 UTC (permalink / raw)
  To: device-mapper development


[-- Attachment #1.1: Type: text/plain, Size: 2813 bytes --]

Hello,
I have a cluster in RH EL 6.5 (I also have a case open fwiw...) where I'm
using HA-LVM.
I made an upgrade of the storage array Netappp -> Netapp.
I was able to do it online without service disruption using pvmove.
As a side effect the destination storage array reports the target LUN as
100% used due to the pvmove operation itself and how it works.
The LUN is now thin provisioned but it was not when pvmove was executed,
and running fstrim on the fs gives error

[root@dbatest1 ~]# fstrim /ALM/rdoffline
fstrim: /ALM/rdoffline: FITRIM ioctl failed: Operation not supported
[root@dbatest1 ~]#

The fs is on an LV with its PV on a multipath device

[root@dbatest1 ~]# df -h /ALM/rdoffline
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VG_ALMTEST_RDOF-LV_ALMTEST_RDOF
                       50G  1.9G   45G   4% /ALM/rdoffline

[root@dbatest1 ~]# pvs|grep VG_ALMTEST_RDOF
  /dev/mapper/3600a098038303769752b495147377858 VG_ALMTEST_RDOF lvm2 a--
 50.00g    0
[root@dbatest1 ~]#

[root@dbatest1 ~]# multipath -l
/dev/mapper/3600a098038303769752b495147377858
3600a098038303769752b495147377858 dm-40 NETAPP,LUN C-Mode
size=50G features='4 queue_if_no_path pg_init_retries 50
retain_attached_hw_handle' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
  |- 7:0:5:14 sddt 71:176  active undef running
  `- 8:0:5:14 sdef 128:112 active undef running
[root@dbatest1 ~]#

At storage array level the LUN appears as 100% used even if it is almost
empty
I found a way to do the fstrim and reclaim space at storage array level,
but only giving downtime, with this sequence of steps:

1) rescan single paths of the multipath device
[root@dbatest1 ~]# for dev in sddt sdef; do echo "1" >
/sys/block/${dev}/device/rescan; done

2) restart multipathd daemon
[root@dbatest1 ~]# service multipathd restart
ok
Stopping multipathd daemon:                                [  OK  ]
Starting multipathd daemon:                                [  OK  ]
[root@dbatest1 ~]#

3) disable/enable the cluster service that contains the fs resource where I
operated 1) and 2)
[root@dbatest1 ~]# clusvcadm -d ALMTEST
Local machine disabling service:ALMTEST...Success

[root@dbatest1 ~]# clusvcadm -e ALMTEST
Local machine trying to enable service:ALMTEST...Success
service:ALMTEST is now running on icldbatest1
[root@dbatest1 ~]#

4) now fstrim works ok
 [root@dbatest1 ~]# fstrim /ALM/rdoffline
[root@dbatest1 ~]#
(it takes about 10-20 seconds, depending on the work it has to do..)

It seems each of the 3 steps is necessary. If I don't execute one of them,
I keep getting the error.

Is there a way to refresh the LVM part, without disabling the service, that
corresponds to without deactivating/activating the LV?

I tried the --refresh option but it didn't work.

Thanks in advance,
Gianluca

[-- Attachment #1.2: Type: text/html, Size: 3742 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]



^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: how to refresh LV to apply fstrim online
  2016-10-20  8:52 how to refresh LV to apply fstrim online Gianluca Cecchi
@ 2016-10-20 11:40 ` Zdenek Kabelac
  2016-10-20 23:32   ` Gianluca Cecchi
  0 siblings, 1 reply; 6+ messages in thread
From: Zdenek Kabelac @ 2016-10-20 11:40 UTC (permalink / raw)
  To: dm-devel, Gianluca Cecchi

Dne 20.10.2016 v 10:52 Gianluca Cecchi napsal(a):
> Hello,
> I have a cluster in RH EL 6.5 (I also have a case open fwiw...) where I'm
> using HA-LVM.
> I made an upgrade of the storage array Netappp -> Netapp.
> I was able to do it online without service disruption using pvmove.
> As a side effect the destination storage array reports the target LUN as 100%
> used due to the pvmove operation itself and how it works.
> The LUN is now thin provisioned but it was not when pvmove was executed, and
> running fstrim on the fs gives error
>
> [root@dbatest1 ~]# fstrim /ALM/rdoffline
> fstrim: /ALM/rdoffline: FITRIM ioctl failed: Operation not supported
> [root@dbatest1 ~]#
>
> The fs is on an LV with its PV on a multipath device
>
> [root@dbatest1 ~]# df -h /ALM/rdoffline
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/mapper/VG_ALMTEST_RDOF-LV_ALMTEST_RDOF
>                        50G  1.9G   45G   4% /ALM/rdoffline
>
> [root@dbatest1 ~]# pvs|grep VG_ALMTEST_RDOF
>   /dev/mapper/3600a098038303769752b495147377858 VG_ALMTEST_RDOF lvm2 a--
>  50.00g    0
> [root@dbatest1 ~]#
>
> [root@dbatest1 ~]# multipath -l /dev/mapper/3600a098038303769752b495147377858
> 3600a098038303769752b495147377858 dm-40 NETAPP,LUN C-Mode
> size=50G features='4 queue_if_no_path pg_init_retries 50
> retain_attached_hw_handle' hwhandler='1 alua' wp=rw
> `-+- policy='round-robin 0' prio=0 status=active
>   |- 7:0:5:14 sddt 71:176  active undef running
>   `- 8:0:5:14 sdef 128:112 active undef running
> [root@dbatest1 ~]#
>
> At storage array level the LUN appears as 100% used even if it is almost empty
> I found a way to do the fstrim and reclaim space at storage array level, but
> only giving downtime, with this sequence of steps:
>
> 1) rescan single paths of the multipath device
> [root@dbatest1 ~]# for dev in sddt sdef; do echo "1" >
> /sys/block/${dev}/device/rescan; done
>
> 2) restart multipathd daemon
> [root@dbatest1 ~]# service multipathd restart
> ok
> Stopping multipathd daemon:                                [  OK  ]
> Starting multipathd daemon:                                [  OK  ]
> [root@dbatest1 ~]#
>
> 3) disable/enable the cluster service that contains the fs resource where I
> operated 1) and 2)
> [root@dbatest1 ~]# clusvcadm -d ALMTEST
> Local machine disabling service:ALMTEST...Success
>
> [root@dbatest1 ~]# clusvcadm -e ALMTEST
> Local machine trying to enable service:ALMTEST...Success
> service:ALMTEST is now running on icldbatest1
> [root@dbatest1 ~]#
>
> 4) now fstrim works ok
>  [root@dbatest1 ~]# fstrim /ALM/rdoffline
> [root@dbatest1 ~]#
> (it takes about 10-20 seconds, depending on the work it has to do..)
>
> It seems each of the 3 steps is necessary. If I don't execute one of them, I
> keep getting the error.
>
> Is there a way to refresh the LVM part, without disabling the service, that
> corresponds to without deactivating/activating the LV?
>
> I tried the --refresh option but it didn't work.
>

Hi

Please provide listing of all your 'multipath'  leg devices - are
they support TRIM ?
Then 'check' dm device.

See  (and attach)

  grep "" /sys/block/*/queue/discard_granularity


Also make sure you are NOT using 'ext2' filesystem which does not support 
discard on RHEL6 and you are on latest available RHEL6 kernel.


Regards

Zdenek

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: how to refresh LV to apply fstrim online
  2016-10-20 11:40 ` Zdenek Kabelac
@ 2016-10-20 23:32   ` Gianluca Cecchi
  2016-10-25 14:04     ` Gianluca Cecchi
  0 siblings, 1 reply; 6+ messages in thread
From: Gianluca Cecchi @ 2016-10-20 23:32 UTC (permalink / raw)
  To: Zdenek Kabelac; +Cc: device-mapper development


[-- Attachment #1.1: Type: text/plain, Size: 2721 bytes --]

On Thu, Oct 20, 2016 at 1:40 PM, Zdenek Kabelac <zkabelac@redhat.com> wrote:

>
> Hi
>
> Please provide listing of all your 'multipath'  leg devices - are
> they support TRIM ?
> Then 'check' dm device.
>
> See  (and attach)
>
>  grep "" /sys/block/*/queue/discard_granularity
>
>
> Also make sure you are NOT using 'ext2' filesystem which does not support
> discard on RHEL6 and you are on latest available RHEL6 kernel.
>
>
> Regards
>
> Zdenek
>
>
>
Hello,
thanks for answer.
I'm using ext3 filesystem that supports discard.
Currently I'm on this kernel

[root@dbatest1 ~]# uname -r
2.6.32-431.29.2.el6.x86_64
[root@dbatest1 ~]#


It seems that I was myself able to find a way to online refresh the logical
volume and so to successfully run fstrim command against the related file
system, without deactivating the lv and most important without generating
downtime for my users.
Please note that what I'm doing is working on a test system where I have
the same situation as in production.
Can you certify my approach and comment about it, so that I can eventually
apply in production?

At the end the logical volume itself is a dm device.
In my case

current situation:
[root@dbatest1 ~]# fstrim /ALM/rdoffline
fstrim: /ALM/rdoffline: FITRIM ioctl failed: Operation not supported
[root@dbatest1 ~]#

[root@dbatest1 ~]# ll /dev/VG_ALMTEST_RDOF/LV_ALMTEST_RDOF
lrwxrwxrwx 1 root root 7 Oct 20 10:39 /dev/VG_ALMTEST_RDOF/LV_ALMTEST_RDOF
-> ../dm-4
[root@dbatest1 ~]#

So I can "manipulate" it with dmsetup command.

[root@dbatest1 ~]# dmsetup info /dev/dm-4
Name:              VG_ALMTEST_RDOF-LV_ALMTEST_RDOF
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 4
Number of targets: 1
UUID: LVM-yOzMkHqmlu9yJuNVWqLVNAx37BwOknEoysZeo9HhzO4P0tETT4WH3RVxNCBeQgRF

[root@dbatest1 ~]#

[root@dbatest1 ~]# dmsetup status /dev/dm-4
0 104849408 linear
[root@dbatest1 ~]#

In practice I work as if I had to resize the device, but specifying the
same table and reloading the dm device.

I save the table into a file
[root@dbatest1 ~]# dmsetup table /dev/dm-4 > my_dm_table
[root@dbatest1 ~]#

I reload the dm device specifying the same table; I read in some doc
references that they tend to suspend the device, before reloading and then
resume, running all the three commands in sequence. I don't know if it is
the best approach....

[root@dbatest1 ~]# dmsetup suspend /dev/dm-4 ; dmsetup reload /dev/dm-4
my_dm_table ; dmsetup resume /dev/dm-4
[root@dbatest1 ~]# echo $?
0
[root@dbatest1 ~]#

And now the magic:

[root@dbatest1 ~]# fstrim /ALM/rdoffline
[root@dbatest1 ~]# echo $?
0
[root@dbatest1 ~]#

Can you give a feedback?
Thanks,
Gianluca

[-- Attachment #1.2: Type: text/html, Size: 4268 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]



^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: how to refresh LV to apply fstrim online
  2016-10-20 23:32   ` Gianluca Cecchi
@ 2016-10-25 14:04     ` Gianluca Cecchi
  2016-10-25 14:12       ` Zdenek Kabelac
  0 siblings, 1 reply; 6+ messages in thread
From: Gianluca Cecchi @ 2016-10-25 14:04 UTC (permalink / raw)
  To: Zdenek Kabelac; +Cc: device-mapper development


[-- Attachment #1.1: Type: text/plain, Size: 3256 bytes --]

On Fri, Oct 21, 2016 at 1:32 AM, Gianluca Cecchi <gianluca.cecchi@gmail.com>
wrote:

>
> On Thu, Oct 20, 2016 at 1:40 PM, Zdenek Kabelac <zkabelac@redhat.com>
> wrote:
>
>>
>> Hi
>>
>> Please provide listing of all your 'multipath'  leg devices - are
>> they support TRIM ?
>> Then 'check' dm device.
>>
>> See  (and attach)
>>
>>  grep "" /sys/block/*/queue/discard_granularity
>>
>>
>> Also make sure you are NOT using 'ext2' filesystem which does not support
>> discard on RHEL6 and you are on latest available RHEL6 kernel.
>>
>>
>> Regards
>>
>> Zdenek
>>
>>
>>
> Hello,
> thanks for answer.
> I'm using ext3 filesystem that supports discard.
> Currently I'm on this kernel
>
> [root@dbatest1 ~]# uname -r
> 2.6.32-431.29.2.el6.x86_64
> [root@dbatest1 ~]#
>
>
> It seems that I was myself able to find a way to online refresh the
> logical volume and so to successfully run fstrim command against the
> related file system, without deactivating the lv and most important without
> generating downtime for my users.
> Please note that what I'm doing is working on a test system where I have
> the same situation as in production.
> Can you certify my approach and comment about it, so that I can eventually
> apply in production?
>
>
[snip]


>
> [root@dbatest1 ~]# dmsetup suspend /dev/dm-4 ; dmsetup reload /dev/dm-4
> my_dm_table ; dmsetup resume /dev/dm-4
> [root@dbatest1 ~]# echo $?
> 0
> [root@dbatest1 ~]#
>
> And now the magic:
>
> [root@dbatest1 ~]# fstrim /ALM/rdoffline
>

[snip]



I hope I wasn't too rude in asking "certification"... actually I meant
comment about the method and if anything nasty (eg the instant suspend
phase command...) could happen with that method....

If it is  anyway important, I have many multipath devices/ filsystems on
this test server and I can proceed to test one by one.
The LUNs indeed support trim.

In particular if I take one of the lun that yet needs to be trimmed, I have:


[g.cecchi@dbatest1 ~]$ sudo multipath -l 3600a098038303769752b495147377867
3600a098038303769752b495147377867 dm-47 NETAPP,LUN C-Mode
size=2.0G features='4 queue_if_no_path pg_init_retries 50
retain_attached_hw_handle' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
  |- 7:0:5:21 sdea 128:32  active undef running
  `- 8:0:5:21 sdem 128:224 active undef running
[g.cecchi@dbatest1 ~]$


[g.cecchi@dbatest1 ~]$ for dev in sdea sdem
> do
> grep "" /sys/block/${dev}/queue/discard_granularity
> done
0
0
[g.cecchi@dbatest1 ~]$

and for the multipath device itself

[g.cecchi@dbatest1 ~]$ cat /sys/block/dm-47/queue/discard_granularity
0
[g.cecchi@dbatest1 ~]$


On devices where I was already able to reload/trim (previous post for
example), I had 0 too before as result, but now I have for example:

[g.cecchi@dbatest1 ~]$ for dev in sddt sdef; do grep ""
/sys/block/${dev}/queue/discard_granularity; done
1048576
1048576
[g.cecchi@dbatest1 ~]$

and also

[g.cecchi@dbatest1 ~]$ cat /sys/block/dm-4/queue/discard_granularity
1048576
[g.cecchi@dbatest1 ~]$

Hope I clarified ....

I don't understand though what you meant with 'check' dm device...

Thanks in advice for any help understanding if this can be the best
workflow to be able to run fstrim without giving downtime, in a safe way.

Gianluca

[-- Attachment #1.2: Type: text/html, Size: 5938 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]



^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: how to refresh LV to apply fstrim online
  2016-10-25 14:04     ` Gianluca Cecchi
@ 2016-10-25 14:12       ` Zdenek Kabelac
  2016-10-25 15:38         ` Gianluca Cecchi
  0 siblings, 1 reply; 6+ messages in thread
From: Zdenek Kabelac @ 2016-10-25 14:12 UTC (permalink / raw)
  To: Gianluca Cecchi; +Cc: device-mapper development

Dne 25.10.2016 v 16:04 Gianluca Cecchi napsal(a):
>
>
> On Fri, Oct 21, 2016 at 1:32 AM, Gianluca Cecchi <gianluca.cecchi@gmail.com
> <mailto:gianluca.cecchi@gmail.com>> wrote:
>
>
>     On Thu, Oct 20, 2016 at 1:40 PM, Zdenek Kabelac <zkabelac@redhat.com
>     <mailto:zkabelac@redhat.com>> wrote:
>
>
>         Hi
>
>         Please provide listing of all your 'multipath'  leg devices - are
>         they support TRIM ?
>         Then 'check' dm device.
>
>         See  (and attach)
>
>          grep "" /sys/block/*/queue/discard_granularity
>
>
>         Also make sure you are NOT using 'ext2' filesystem which does not
>         support discard on RHEL6 and you are on latest available RHEL6 kernel.
>
>
>         Regards
>
>         Zdenek
>
>
>
>     Hello,
>     thanks for answer.
>     I'm using ext3 filesystem that supports discard.
>     Currently I'm on this kernel
>
>     [root@dbatest1 ~]# uname -r
>     2.6.32-431.29.2.el6.x86_64
>     [root@dbatest1 ~]#
>
>
>     It seems that I was myself able to find a way to online refresh the
>     logical volume and so to successfully run fstrim command against the
>     related file system, without deactivating the lv and most important
>     without generating downtime for my users.
>     Please note that what I'm doing is working on a test system where I have
>     the same situation as in production.
>     Can you certify my approach and comment about it, so that I can eventually
>     apply in production?
>
>
> [snip]
>
>
>
>     [root@dbatest1 ~]# dmsetup suspend /dev/dm-4 ; dmsetup reload /dev/dm-4
>     my_dm_table ; dmsetup resume /dev/dm-4
>     [root@dbatest1 ~]# echo $?
>     0
>     [root@dbatest1 ~]#
>
>     And now the magic:
>
>     [root@dbatest1 ~]# fstrim /ALM/rdoffline
t giving downtime, in a safe way.


Normally it's not advised  to use 'dmsetup' command for  LV.
The above sequence should be equivalent to:

lvchange --refresh  vg/lv
(or  vgchange --refresh vg  -  doing it for every active LV in VG)

It's unclear how this could help  - unless you we doing some 'pvmove'
operations (which might be worth a BZ).

You should collect all states  while it 'DOES NOT' work.
And then  run  the --refresh (which you thing it's fixing it for your)
ATM I'm clueless how you get mapping without TRIM where --refresh can fix it.


Regards

Zdenek

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: how to refresh LV to apply fstrim online
  2016-10-25 14:12       ` Zdenek Kabelac
@ 2016-10-25 15:38         ` Gianluca Cecchi
  0 siblings, 0 replies; 6+ messages in thread
From: Gianluca Cecchi @ 2016-10-25 15:38 UTC (permalink / raw)
  To: Zdenek Kabelac; +Cc: device-mapper development


[-- Attachment #1.1: Type: text/plain, Size: 3000 bytes --]

On Tue, Oct 25, 2016 at 4:12 PM, Zdenek Kabelac <zkabelac@redhat.com> wrote:

>
>> t giving downtime, in a safe way.
>
>
> Normally it's not advised  to use 'dmsetup' command for  LV.
> The above sequence should be equivalent to:
>
> lvchange --refresh  vg/lv
> (or  vgchange --refresh vg  -  doing it for every active LV in VG)
>
> It's unclear how this could help  - unless you we doing some 'pvmove'
> operations (which might be worth a BZ).
>
> You should collect all states  while it 'DOES NOT' work.
> And then  run  the --refresh (which you thing it's fixing it for your)
> ATM I'm clueless how you get mapping without TRIM where --refresh can fix
> it.
>
>
> Regards
>
> Zdenek
>
>
>

When I rescan the single devices (the legs) I do have then
[root@dbatest1 ~]# for dev in sdea sdem ; do grep ""
/sys/block/${dev}/queue/discard_granularity; done
1048576
1048576
[root@dbatest1 ~]#

But the problem is the device mapper device of the LV itself, while the
multipath one is ok.
I had tried with both

vgchange --refresh VG_TEST_REDO
and
lvchange --refresh VG_TEST_REDO/LV_TEST_REDO

BTW: The VG contains only this LV

yes the current device has been the target of a pvmove operation, where
initially the LUN was not configured as thin provisioned.

but the underlying dm device still contains 0 in its granularity. And the
fstrim command fails.
in this new test it is dm-14 (while the multipath device of the related PV
is dm-47 and it is ok)

[root@dbatest1 ~]# multipath -l
/dev/mapper/3600a098038303769752b4951473778673600a098038303769752b495147377867
dm-47 NETAPP,LUN C-Mode
size=2.0G features='4 queue_if_no_path pg_init_retries 50
retain_attached_hw_handle' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
  |- 7:0:5:21 sdea 128:32  active undef running
  `- 8:0:5:21 sdem 128:224 active undef running
[root@dbatest1 ~]#

[root@dbatest1 ~]# ll /dev/VG_TEST_REDO/LV_TEST_REDO
lrwxrwxrwx 1 root root 8 Oct 25 17:25 /dev/VG_TEST_REDO/LV_TEST_REDO ->
../dm-14
[root@dbatest1 ~]#


[root@dbatest1 ~]# cat /sys/block/dm-47/queue/discard_granularity
1048576
[root@dbatest1 ~]#


[root@dbatest1 ~]# cat /sys/block/dm-14/queue/discard_granularity
0
[root@dbatest1 ~]#

So the problem seems to find a way to get the discard_granularity inside
dm-14.
One way to obtain it is my method:

[root@dbatest1 ~]# dmsetup info /dev/dm-14
Name:              VG_TEST_REDO-LV_TEST_REDO
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 14
Number of targets: 1
UUID: LVM-ebT3iin7JZdFZCoR05NEPpXcDosNQ3Y46HwejMdu0o9qfeeeMcRTemcuGtUVqMds

[root@dbatest1 ~]# dmsetup table /dev/dm-14 > my_dm_table

[root@dbatest1 ~]# dmsetup suspend /dev/dm-14 ; dmsetup reload /dev/dm-14
my_dm_table ; dmsetup resume /dev/dm-14
[root@dbatest1 ~]#

And now
[root@dbatest1 ~]# cat /sys/block/dm-14/queue/discard_granularity
1048576
[root@dbatest1 ~]#

and
[root@dbatest1 ~]# fstrim /TEST/redolog/
[root@dbatest1 ~]#

[-- Attachment #1.2: Type: text/html, Size: 5485 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]



^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2016-10-25 15:38 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-10-20  8:52 how to refresh LV to apply fstrim online Gianluca Cecchi
2016-10-20 11:40 ` Zdenek Kabelac
2016-10-20 23:32   ` Gianluca Cecchi
2016-10-25 14:04     ` Gianluca Cecchi
2016-10-25 14:12       ` Zdenek Kabelac
2016-10-25 15:38         ` Gianluca Cecchi

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.