dracut, degraded md arrays, resume and systemd.

* dracut, degraded md arrays, resume and systemd.
@ 2015-03-11  0:28 NeilBrown
       [not found] ` <20150311112845.01dd3269-wvvUuzkyo1EYVZTmpyfIwg@public.gmane.org>
  0 siblings, 1 reply; 3+ messages in thread
From: NeilBrown @ 2015-03-11  0:28 UTC (permalink / raw)
  To: initramfs-u79uwXL29TY76Z2rM5mHXA

[-- Attachment #1: Type: text/plain, Size: 4954 bytes --]


hi,
 I have a problem....  I'm not entirely sure where to fix it.

 I have a system with 2 drives, each partitioned into 3 partitions and those
 partitions combined in md/raid RAID1 arrays.
 The arrays are used for /boot, swap and an LVM PV - which has the root
 filesystem in a VG.  Normally this all works wonderfully.

 If I shutdown, remove one drive, and boot - it doesn't.

 When mdadm sees a newly-degraded array like this (i.e. the one drive it
 finds doesn't have recorded that the other device is missing), it won't
 assemble the array until it is explicitly told that all devices have been
 found.

 dracut has code to do exactly this: /sbin/mdraid_start is placed on the
 'timeout' queue to be run after a suitable timeout.  And this script does
 the right thing.  The md arrays are all assembled, the LVM PV is found so
 the VG and LV is assembled, and the root filesystem is mounted..

 HOWEVER, before all that happens, systemd stops waiting and gives up -
 moments too soon.

 There are (at least) two other scripts on the 'timeout' queue which run
 immediately after mdraid_start.
 One is lvmscan, which probably does the right thing, isn't exactly necessary
 in my context I think, and certainly isn't a problem.

 But there is also the 'resume.sh' script:

        {
            printf -- "%s" 'warn "Cancelling resume operation. Device not found.";'
            printf -- ' cancel_wait_for_dev /dev/resume; rm -f -- "$job" "%s/initqueue/settled/resume.sh";\n
        } >> $hookdir/initqueue/timeout/resume.sh

 This calls cancel_wait_for_dev, which supposedly cancels the wait for the
 swap array.  Unfortunately it also cancels the wait for the root filesystem.
 I'm not completely sure why, but it certainly relates to the
     systemctl daemon-reload
 command that cancel_wait_for_dev schedules.  If I comment that out, it all
 works. (also if I boot with noresume, it all works).

 But I don't think that is all of the problem.
 If I had swap on an LVM volume, then I suspect it wouldn't quite be found by
 the time that the resume.sh script gets run, and so the resume attempt would
 incorrectly abort.

 My feeling is that if any script in the 'timeout' queue makes any progress,
 then the remaining scripts should be delayed for another timeout.

 There is code that does something a little bit like this:

    if [ $main_loop -gt $((2*$RDRETRY/3)) ]; then
        for job in $hookdir/initqueue/timeout/*.sh; do
            [ -e "$job" ] || break
            job=$job . $job
            udevadm settle --timeout=0 >/dev/null 2>&1 || main_loop=0
            [ -f $hookdir/initqueue/work ] && main_loop=0
        done
    fi

so if 'udevadm settle --timeout=0' fails, or if initqueue/work has been
created, the main_loop is set to 0, which seems to imply "try again".
However the subsequent jobs in the queue are not aborted.

Also, in my case, neither of these conditions trigger.  mdraid_start doesn't
create the 'initqueue/work' file, and 'udevadm settle' has never (as far as I
can tell) actually honoured "--timeout=0" the way it is documented.
It waits indefinitely for the queue to be empty, then succeeds.

So my proposed solution (which is really just a suggestion and I suspect
something else will be better), is to:
1/ modify the above loop to break out if main_loop is ever reset to zero, and
2/ modify mdraid_start to create initqueue/work if it finds anything to do.

The following patch makes that explicit.  It seems to fix my problem.

Is this a good way to fix it?  Is there something better?

BTW, I also have a problem in a similar config where the md/raid RAID1 is
encrypted.  Systemd gives up waiting for the encrypted device after 90
seconds:
[   92.250437] linux systemd[1]: Dependency failed for Cryptography Setup for cr_md1.

but mdraid_start doesn't get run until 120 seconds have elapsed.
I haven't looked into setup of encrypted devices yet, but if anyone has
suggestions, I'm very interested :-)

Thanks,
NeilBrown

diff --git a/modules.d/90mdraid/mdraid_start.sh b/modules.d/90mdraid/mdraid_start.sh
index 761e64f312d3..400ab5dc46c7 100755
--- a/modules.d/90mdraid/mdraid_start.sh
+++ b/modules.d/90mdraid/mdraid_start.sh
@@ -27,6 +27,7 @@ _md_force_run() {
 
         _path_d="${_path_s%/*}/degraded"
         [ ! -r "$_path_d" ] && continue
+        > $hookdir/initqueue/work
     done
 }
 
diff --git a/modules.d/98systemd/dracut-initqueue.sh b/modules.d/98systemd/dracut-initqueue.sh
index 88cd1e056ed7..af9cec2c5b8c 100755
--- a/modules.d/98systemd/dracut-initqueue.sh
+++ b/modules.d/98systemd/dracut-initqueue.sh
@@ -60,6 +60,7 @@ while :; do
             job=$job . $job
             udevadm settle --timeout=0 >/dev/null 2>&1 || main_loop=0
             [ -f $hookdir/initqueue/work ] && main_loop=0
+            [ $main_loop -eq 0 ] && break
         done
     fi
 

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]

^ permalink raw reply related	[flat|nested] 3+ messages in thread