Reshape stalls and fails when SELinux is enabled

Reshape stalls and fails when SELinux is enabled

[Enrico Tagliavini]

[-- Attachment #1: Type: text/plain, Size: 3820 bytes --]

Hi everybody,

yesterday I tested a conversion of a 2 disk raid1 to a 4 disk raid5.
The reshape never completed, I rebooted the VM and the raid device was
unrecoverable. Backup-file was not useful unfortunately. Take a
comfortable seat, this is a bit long story, sorry :).

Environment:
 - VM running on my laptop (Fedora 22), created with virt-manager,
qemu based. All disk images are qcow2 virtio devices.
 - VM OS: centos 7 64 bit, fully updated as of today.
 - mdadm version: 3.3.2-2.el7_1.1
 - kernel version: 3.10.0-229.20.1.el7
 - selinux policy targeted: 3.13.1-23.el7_1.21

How to reproduce:

 * Attach four 1 GB disks to the VM, in the following example they are
vd[b,c,d,e].
 * create a raid1 device with two disks: mdadm --create /dev/md0 -e
1.2 -n 2 -l raid1 -N raidtest /dev/vdb /dev/vdc
 * [optional?]: I created a FS on it: mkfs.xfs -L test /dev/md0
 * [optional?]: I created three files on it and noted the checksum, to
check for eventual corruptions
     - mount /dev/md0 /mnt/raidtest/ && cd /mnt/raidtest/
     - echo uno > first.txt ; echo due > second.txt ; touch third.txt
; shred -s 512M -f third.txt
     - for file in *; do sha1sum $file >> sha1sum.txt; done
 * check /proc/mdstat to make sure the raid1 sync operation is
complete (should really be fast but better safe than sorry)
 * unmount /mnt/raidtest/
 * add the two additional disks as spares:
     - mdadm --manage /dev/md0 --add-spare /dev/vdd
     - mdadm --manage /dev/md0 --add-spare /dev/vde
 * grow and reshape: mdadm --grow /dev/md0 -n 4 --level=5
--backup-file /root/backup-md0 # (I also tried with
/var/local/backup-md0)

At this point an AVC will happen. Even if I'm in an interactive
session and, as such SELinux should not limit it normally, mdadm
process switches to mdadm_t type (maybe a forked process with its own
session group?) and is not allowed to write a file in the /root/ or
/var/ folders. This is ok, however mdadm keeps going instead of
aborting the reshape. It's running without a backup file, that's not
what the admin asked for since the --backup-file option is specified.
But even worst than this is that my reshape got stuck and never
completed. I waited a couple of hours but it remained at 0%. Something
was actually written to the backup file (which is weird given the AVC,
but it can be the original mdadm process not running under mdadm_t).

At this point I was kind of curious to test what would happen if a
distracted admin like me wont notice the problem and, days later,
would reboot the server due to security updates or anything else. The
result is an unrecoverable md array. I tried to assemble it back with
the backup file

mdadm --assemble /dev/md0  -u 6f53ec3e:d9868fef:12d3e243:8489561b
--backup-file /root/backup-md

But no way

mdadm: [sorry I copied wrong and the device name was lost] has an
active reshape - checking if critical section needs to be restored
mdadm: Failed to find backup of critical section
mdadm: Failed to restore critical section for reshape, sorry.

I retried the entire procedure from scratch, but this time with before
mdadm --grow I set SELinux in permissive mode with setenforce 0.
Everything was butter smooth this time. Reshape was almost instant for
such a small array, data was checksumming correctly and my array was
level 5.

Now there might be a problem with the SELinux policy here, but
honestly I think mdadm should just abort, whatever the reason of the
problem was. There might be other scenarios not involving SELinux
causing the same problem.

It would also be nice to suggest the user, if SELinux is active, to
change the context of the backup file to something SELinux will permit
(mdadm_map_t, mdadm_var_run_t?).

Attached you can also find the AVC denial for my entire day of testing.

Thank you for your help.
Best regards.

Enrico Tagliavini

[-- Attachment #2: changeraid1_to_raid5.txt --]
[-- Type: text/plain, Size: 68 bytes --]

mdadm --grow /dev/md0 -n 4 --level=5 --backup-file=/root/backup-md0

Re: Reshape stalls and fails when SELinux is enabled

[Enrico Tagliavini]

Looks like I'm not the only one and it happens in real world cases,
not just a test on a VM:
https://marc.info/?l=linux-raid&m=144973573922461&w=2

Is there a place where I can open a bug to keep track of this issue?
On the wiki it says that "Linux RAID issues are discussed in the
linux-raid mailing list to be found at
http://vger.kernel.org/vger-lists.html#linux-raid", but it's easy to
loose track eventually. Can't find a component "mdadm" at
https://bugzilla.kernel.org/ maybe one should be created under the
"Tools" product?

Thank you.
Best regards.

Enrico Tagliavini

On 3 December 2015 at 12:04, Enrico Tagliavini
<enrico.tagliavini@gmail.com> wrote:
> Hi everybody,
>
> yesterday I tested a conversion of a 2 disk raid1 to a 4 disk raid5.
> The reshape never completed, I rebooted the VM and the raid device was
> unrecoverable. Backup-file was not useful unfortunately. Take a
> comfortable seat, this is a bit long story, sorry :).
>
> Environment:
>  - VM running on my laptop (Fedora 22), created with virt-manager,
> qemu based. All disk images are qcow2 virtio devices.
>  - VM OS: centos 7 64 bit, fully updated as of today.
>  - mdadm version: 3.3.2-2.el7_1.1
>  - kernel version: 3.10.0-229.20.1.el7
>  - selinux policy targeted: 3.13.1-23.el7_1.21
>
> How to reproduce:
>
>  * Attach four 1 GB disks to the VM, in the following example they are
> vd[b,c,d,e].
>  * create a raid1 device with two disks: mdadm --create /dev/md0 -e
> 1.2 -n 2 -l raid1 -N raidtest /dev/vdb /dev/vdc
>  * [optional?]: I created a FS on it: mkfs.xfs -L test /dev/md0
>  * [optional?]: I created three files on it and noted the checksum, to
> check for eventual corruptions
>      - mount /dev/md0 /mnt/raidtest/ && cd /mnt/raidtest/
>      - echo uno > first.txt ; echo due > second.txt ; touch third.txt
> ; shred -s 512M -f third.txt
>      - for file in *; do sha1sum $file >> sha1sum.txt; done
>  * check /proc/mdstat to make sure the raid1 sync operation is
> complete (should really be fast but better safe than sorry)
>  * unmount /mnt/raidtest/
>  * add the two additional disks as spares:
>      - mdadm --manage /dev/md0 --add-spare /dev/vdd
>      - mdadm --manage /dev/md0 --add-spare /dev/vde
>  * grow and reshape: mdadm --grow /dev/md0 -n 4 --level=5
> --backup-file /root/backup-md0 # (I also tried with
> /var/local/backup-md0)
>
> At this point an AVC will happen. Even if I'm in an interactive
> session and, as such SELinux should not limit it normally, mdadm
> process switches to mdadm_t type (maybe a forked process with its own
> session group?) and is not allowed to write a file in the /root/ or
> /var/ folders. This is ok, however mdadm keeps going instead of
> aborting the reshape. It's running without a backup file, that's not
> what the admin asked for since the --backup-file option is specified.
> But even worst than this is that my reshape got stuck and never
> completed. I waited a couple of hours but it remained at 0%. Something
> was actually written to the backup file (which is weird given the AVC,
> but it can be the original mdadm process not running under mdadm_t).
>
> At this point I was kind of curious to test what would happen if a
> distracted admin like me wont notice the problem and, days later,
> would reboot the server due to security updates or anything else. The
> result is an unrecoverable md array. I tried to assemble it back with
> the backup file
>
> mdadm --assemble /dev/md0  -u 6f53ec3e:d9868fef:12d3e243:8489561b
> --backup-file /root/backup-md
>
> But no way
>
> mdadm: [sorry I copied wrong and the device name was lost] has an
> active reshape - checking if critical section needs to be restored
> mdadm: Failed to find backup of critical section
> mdadm: Failed to restore critical section for reshape, sorry.
>
> I retried the entire procedure from scratch, but this time with before
> mdadm --grow I set SELinux in permissive mode with setenforce 0.
> Everything was butter smooth this time. Reshape was almost instant for
> such a small array, data was checksumming correctly and my array was
> level 5.
>
> Now there might be a problem with the SELinux policy here, but
> honestly I think mdadm should just abort, whatever the reason of the
> problem was. There might be other scenarios not involving SELinux
> causing the same problem.
>
> It would also be nice to suggest the user, if SELinux is active, to
> change the context of the backup file to something SELinux will permit
> (mdadm_map_t, mdadm_var_run_t?).
>
> Attached you can also find the AVC denial for my entire day of testing.
>
> Thank you for your help.
> Best regards.
>
> Enrico Tagliavini

Re: Reshape stalls and fails when SELinux is enabled

[Edward Kuns]

I ran into this as well.  I unwisely worked around it by placing the
backup file on /tmp.  I don't recommend that choice!  I didn't see
your AVC on your EMail, but I recommend you reach out to the SELinux
folks for labeling help.  I've reach out to them in the past and have
found them very helpful (as are the folks here).  Getting the labeling
corrected will not only help you but everyone else in the long term.

I agree that it would be good for mdadm to handle this kind of failure
in a more productive fashion as well.  But I'm only a lurker here.  I
don't have enough knowledge to suggest what would be better.  One
thing that really helped me debug this was commenting out the lines

#StandardOutput=null
#StandardError=null

in the systemd file /lib/systemd/system/mdadm-grow-continue@.service
-- note you have to run systemctl daemon-reload after changing the
file for the change to be noticed.

Once I did that, the system logs had much more useful information.
I'm not sure where that file is sourced, whether it's from the
linux-raid people or from the distributions or somewhere in between.
Does anyone here know if there's a reason why standard out and
standard error are sent to /dev/null for this service?

              Eddie

Re: Reshape stalls and fails when SELinux is enabled

[Wols Lists]

On 10/12/15 17:06, Edward Kuns wrote:
> I ran into this as well.  I unwisely worked around it by placing the
> backup file on /tmp.  I don't recommend that choice! 

Most people don't realise there are various tmp directories, with
different behaviours. /tmp is defined by LSB or somesuch to be real
temporary storage - no guarantees - stuff may disappear at any time.
Which is why my /tmp is a tmpfs - stored in ram. So yes, a backup file
on /tmp is not a good idea ...

/var/tmp is defined as holding data that must survive a reboot - where
emacs and vi and that lot are recommended to store their crash dumps,
recovery files, etc for example.

 I didn't see
> your AVC on your EMail, but I recommend you reach out to the SELinux
> folks for labeling help.  I've reach out to them in the past and have
> found them very helpful (as are the folks here).  Getting the labeling
> corrected will not only help you but everyone else in the long term.
> 
> I agree that it would be good for mdadm to handle this kind of failure
> in a more productive fashion as well.  But I'm only a lurker here.  I
> don't have enough knowledge to suggest what would be better.  One
> thing that really helped me debug this was commenting out the lines
> 
> #StandardOutput=null
> #StandardError=null
> 
> in the systemd file /lib/systemd/system/mdadm-grow-continue@.service
> -- note you have to run systemctl daemon-reload after changing the
> file for the change to be noticed.
> 
> Once I did that, the system logs had much more useful information.
> I'm not sure where that file is sourced, whether it's from the
> linux-raid people or from the distributions or somewhere in between.
> Does anyone here know if there's a reason why standard out and
> standard error are sent to /dev/null for this service?
> 
I wouldn't know for certain, but given the amount of junk so many
programs spew to stdout or stderr (which nobody ever sees because
they're running in a gui), I expect it's because that's what happens by
default anyway :-) They don't want to clutter the logs with all that
junk and chatter.

I often somehow get stdout and stderr for various programs redirected to
konsole, because I'll use dbus-launch to start a program. And then when
the program has exited I'll get loads of rubbish from other programs ...
the chatter is horrendous.

Cheers,
Wol

Re: Reshape stalls and fails when SELinux is enabled

[George Rapp]

Indeed, SELinux appears to be very not-helpful in the RAID resize
process. I decided to put SELINUX=disabled in /etc/selinux/config,
touch /.relabel, and reboot before doing any more RAID 6 work.

The only other workaround I could see is the ability to do a "dry run"
of mdadm operations a la wodim / growisofs - a chance to test out all
the parameters before actually pulling the trigger. (Sorry, no patch,
though ... 8^)

On Thu, Dec 10, 2015 at 1:24 PM, Wols Lists <antlists@youngman.org.uk> wrote:
> On 10/12/15 17:06, Edward Kuns wrote:
>> I ran into this as well.  I unwisely worked around it by placing the
>> backup file on /tmp.  I don't recommend that choice!
>
> Most people don't realise there are various tmp directories, with
> different behaviours. /tmp is defined by LSB or somesuch to be real
> temporary storage - no guarantees - stuff may disappear at any time.
> Which is why my /tmp is a tmpfs - stored in ram. So yes, a backup file
> on /tmp is not a good idea ...
>
> /var/tmp is defined as holding data that must survive a reboot - where
> emacs and vi and that lot are recommended to store their crash dumps,
> recovery files, etc for example.
>
>  I didn't see
>> your AVC on your EMail, but I recommend you reach out to the SELinux
>> folks for labeling help.  I've reach out to them in the past and have
>> found them very helpful (as are the folks here).  Getting the labeling
>> corrected will not only help you but everyone else in the long term.
>>
>> I agree that it would be good for mdadm to handle this kind of failure
>> in a more productive fashion as well.  But I'm only a lurker here.  I
>> don't have enough knowledge to suggest what would be better.  One
>> thing that really helped me debug this was commenting out the lines
>>
>> #StandardOutput=null
>> #StandardError=null
>>
>> in the systemd file /lib/systemd/system/mdadm-grow-continue@.service
>> -- note you have to run systemctl daemon-reload after changing the
>> file for the change to be noticed.
>>
>> Once I did that, the system logs had much more useful information.
>> I'm not sure where that file is sourced, whether it's from the
>> linux-raid people or from the distributions or somewhere in between.
>> Does anyone here know if there's a reason why standard out and
>> standard error are sent to /dev/null for this service?
>>
> I wouldn't know for certain, but given the amount of junk so many
> programs spew to stdout or stderr (which nobody ever sees because
> they're running in a gui), I expect it's because that's what happens by
> default anyway :-) They don't want to clutter the logs with all that
> junk and chatter.
>
> I often somehow get stdout and stderr for various programs redirected to
> konsole, because I'll use dbus-launch to start a program. And then when
> the program has exited I'll get loads of rubbish from other programs ...
> the chatter is horrendous.
>
> Cheers,
> Wol
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
George Rapp  (Pataskala, OH) Home: george.rapp -- at -- gmail.com
LinkedIn profile: https://www.linkedin.com/in/georgerapp
Phone: +1 740 936 RAPP (740 936 7277)

Re: Reshape stalls and fails when SELinux is enabled

[Enrico Tagliavini]

No need to disable SELinux. Just run

setenforce 0

before calling mdadm. When your reshape is done enable SELinux again with

setenforce 1

no reboot needed.

Also if you want a permanent workaround you can whitelist mdadm process type:

semanage permissive -a mdadm_t # but be aware I didn't tested this!

On 11 December 2015 at 02:26, George Rapp <george.rapp@gmail.com> wrote:
> Indeed, SELinux appears to be very not-helpful in the RAID resize
> process. I decided to put SELINUX=disabled in /etc/selinux/config,
> touch /.relabel, and reboot before doing any more RAID 6 work.
>
> The only other workaround I could see is the ability to do a "dry run"
> of mdadm operations a la wodim / growisofs - a chance to test out all
> the parameters before actually pulling the trigger. (Sorry, no patch,
> though ... 8^)
>
> On Thu, Dec 10, 2015 at 1:24 PM, Wols Lists <antlists@youngman.org.uk> wrote:
>> On 10/12/15 17:06, Edward Kuns wrote:
>>> I ran into this as well.  I unwisely worked around it by placing the
>>> backup file on /tmp.  I don't recommend that choice!
>>
>> Most people don't realise there are various tmp directories, with
>> different behaviours. /tmp is defined by LSB or somesuch to be real
>> temporary storage - no guarantees - stuff may disappear at any time.
>> Which is why my /tmp is a tmpfs - stored in ram. So yes, a backup file
>> on /tmp is not a good idea ...
>>
>> /var/tmp is defined as holding data that must survive a reboot - where
>> emacs and vi and that lot are recommended to store their crash dumps,
>> recovery files, etc for example.
>>
>>  I didn't see
>>> your AVC on your EMail, but I recommend you reach out to the SELinux
>>> folks for labeling help.  I've reach out to them in the past and have
>>> found them very helpful (as are the folks here).  Getting the labeling
>>> corrected will not only help you but everyone else in the long term.
>>>
>>> I agree that it would be good for mdadm to handle this kind of failure
>>> in a more productive fashion as well.  But I'm only a lurker here.  I
>>> don't have enough knowledge to suggest what would be better.  One
>>> thing that really helped me debug this was commenting out the lines
>>>
>>> #StandardOutput=null
>>> #StandardError=null
>>>
>>> in the systemd file /lib/systemd/system/mdadm-grow-continue@.service
>>> -- note you have to run systemctl daemon-reload after changing the
>>> file for the change to be noticed.
>>>
>>> Once I did that, the system logs had much more useful information.
>>> I'm not sure where that file is sourced, whether it's from the
>>> linux-raid people or from the distributions or somewhere in between.
>>> Does anyone here know if there's a reason why standard out and
>>> standard error are sent to /dev/null for this service?
>>>
>> I wouldn't know for certain, but given the amount of junk so many
>> programs spew to stdout or stderr (which nobody ever sees because
>> they're running in a gui), I expect it's because that's what happens by
>> default anyway :-) They don't want to clutter the logs with all that
>> junk and chatter.
>>
>> I often somehow get stdout and stderr for various programs redirected to
>> konsole, because I'll use dbus-launch to start a program. And then when
>> the program has exited I'll get loads of rubbish from other programs ...
>> the chatter is horrendous.
>>
>> Cheers,
>> Wol
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>
> --
> George Rapp  (Pataskala, OH) Home: george.rapp -- at -- gmail.com
> LinkedIn profile: https://www.linkedin.com/in/georgerapp
> Phone: +1 740 936 RAPP (740 936 7277)

Re: Reshape stalls and fails when SELinux is enabled

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1835 bytes --]

On Fri, Dec 11 2015, Edward Kuns wrote:

> I ran into this as well.  I unwisely worked around it by placing the
> backup file on /tmp.  I don't recommend that choice!  I didn't see
> your AVC on your EMail, but I recommend you reach out to the SELinux
> folks for labeling help.  I've reach out to them in the past and have
> found them very helpful (as are the folks here).  Getting the labeling
> corrected will not only help you but everyone else in the long term.
>
> I agree that it would be good for mdadm to handle this kind of failure
> in a more productive fashion as well.  But I'm only a lurker here.  I
> don't have enough knowledge to suggest what would be better.  One
> thing that really helped me debug this was commenting out the lines
>
> #StandardOutput=null
> #StandardError=null
>
> in the systemd file /lib/systemd/system/mdadm-grow-continue@.service
> -- note you have to run systemctl daemon-reload after changing the
> file for the change to be noticed.
>
> Once I did that, the system logs had much more useful information.
> I'm not sure where that file is sourced, whether it's from the
> linux-raid people or from the distributions or somewhere in between.
> Does anyone here know if there's a reason why standard out and
> standard error are sent to /dev/null for this service?

That mdadm-grow-continue@.service file is part of mdadm (in the systemd
directory).
As for why stdout/err is set to 'null', you could try finding out who
did that and ask them ....

http://git.neil.brown.name/?p=mdadm.git;a=commitdiff;h=5e76dce1acd906e8fc8af04973c3a129cdc77fd6

but I'm pretty sure that he has no idea.

You could try sending me a patch which changes those lines and gives some
sort of explanation why it is a good change to make.
There is a very good chance that I would apply that patch.

Thanks,
NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]