checkarray not running or emailing

checkarray not running or emailing

[Leslie Rhorer]

     I have upgraded 2 of my servers to Debian Buster, and now neither 
one seems to be running checkarray automatically.  In addition, when I 
run checkarray manually, it isn't sending update emails on the status of 
the job.  Actually, I have never been able to figure out how checkarray 
runs.  One my older servers, there doesn't seem to be anything in 
/etc/crontab, /etc/cron.monthly, /etc/init.d/, /etc/mdadm/mdadm.conf, or 
/lib/systemd/system/ that would run checkarray.

Re: checkarray not running or emailing

[Leslie Rhorer]

     Is there seriously no one here who knows how checkarray was 
launched in previous versions?

On 3/1/2020 3:03 PM, Leslie Rhorer wrote:
>     I have upgraded 2 of my servers to Debian Buster, and now neither 
> one seems to be running checkarray automatically.  In addition, when I 
> run checkarray manually, it isn't sending update emails on the status 
> of the job.  Actually, I have never been able to figure out how 
> checkarray runs.  One my older servers, there doesn't seem to be 
> anything in /etc/crontab, /etc/cron.monthly, /etc/init.d/, 
> /etc/mdadm/mdadm.conf, or /lib/systemd/system/ that would run checkarray.

Re: checkarray not running or emailing

[Wols Lists]

On 11/03/20 01:11, Leslie Rhorer wrote:
>     Is there seriously no one here who knows how checkarray was launched
> in previous versions?

You need to ask on a Debian list. I for one don't have a damn clue
because I actively avoid apt-based systems.

Sorry, I don't mean to be harsh, but each distro "does its own thing" so
a lot of people (like me) *will* be clueless on that point, even if we
are raid experts.

Cheers,
Wol
> 
> On 3/1/2020 3:03 PM, Leslie Rhorer wrote:
>>     I have upgraded 2 of my servers to Debian Buster, and now neither
>> one seems to be running checkarray automatically.  In addition, when I
>> run checkarray manually, it isn't sending update emails on the status
>> of the job.  Actually, I have never been able to figure out how
>> checkarray runs.  One my older servers, there doesn't seem to be
>> anything in /etc/crontab, /etc/cron.monthly, /etc/init.d/,
>> /etc/mdadm/mdadm.conf, or /lib/systemd/system/ that would run checkarray.

Re: checkarray not running or emailing

[Brad Campbell]

On 11/3/20 09:11, Leslie Rhorer wrote:
>      Is there seriously no one here who knows how checkarray was launched in previous versions?
> 
> On 3/1/2020 3:03 PM, Leslie Rhorer wrote:
>>     I have upgraded 2 of my servers to Debian Buster, and now neither one seems to be running checkarray automatically.  In addition, when I run checkarray manually, it isn't sending update emails on the status of the job.  Actually, I have never been able to figure out how checkarray runs.  One my older servers, there doesn't seem to be anything in /etc/crontab, /etc/cron.monthly, /etc/init.d/, /etc/mdadm/mdadm.conf, or /lib/systemd/system/ that would run checkarray.
> 

On mine it's in /etc/cron.d/mdadm

brad@srv:/etc/cron.d$ cat mdadm
#
# cron.d/mdadm -- schedules periodic redundancy checks of MD devices
#
# Copyright © martin f. krafft <madduck@madduck.net>
# distributed under the terms of the Artistic Licence 2.0
#

# By default, run at 00:57 on every Sunday, but do nothing unless the day of
# the month is less than or equal to 7. Thus, only run on the first Sunday of
# each month. crontab(5) sucks, unfortunately, in this regard; therefore this
# hack (see #380425).
57 0 * * 0 root if [ -x /usr/share/mdadm/checkarray ] && [ $(date +\%d) -le 7 ]; then /usr/share/mdadm/checkarray --cron --all --idle --quiet; fi

dpkg -L mdadm gave me a list of files and I just checked the cron entries.

I don't run anything that recent, but Debian is Debian.

Brad
-- 
An expert is a person who has found out by his own painful
experience all the mistakes that one can make in a very
narrow field. - Niels Bohr

Re: checkarray not running or emailing

[Robin Hill]

On Tue Mar 10, 2020 at 08:11:36PM -0500, Leslie Rhorer wrote:

>      Is there seriously no one here who knows how checkarray was 
> launched in previous versions?
> 
> On 3/1/2020 3:03 PM, Leslie Rhorer wrote:
> >     I have upgraded 2 of my servers to Debian Buster, and now neither 
> > one seems to be running checkarray automatically.  In addition, when I 
> > run checkarray manually, it isn't sending update emails on the status 
> > of the job.  Actually, I have never been able to figure out how 
> > checkarray runs.  One my older servers, there doesn't seem to be 
> > anything in /etc/crontab, /etc/cron.monthly, /etc/init.d/, 
> > /etc/mdadm/mdadm.conf, or /lib/systemd/system/ that would run checkarray.

Google suggests it's run from /etc/cron.d/mdadm. Does this exist on your
systems? I'm not sure why upgrading to Buster would have stopped the
checks though, unless the script wasn't replaced and the file location
for checkarray has changed.

Cheers,
    Robin
-- 
     ___        
    ( ' }     |       Robin Hill        <robin@robinhill.me.uk> |
   / / )      | Little Jim says ....                            |
  // !!       |      "He fallen in de water !!"                 |

Re: checkarray not running or emailing

[Leslie Rhorer]

     Well, that is rather the point, actually.  I can't find anywhere 
that Debian *DID* run the script in the previous versions.  Failing 
that, either Debian was doing it somewhere of which I have no knowledge, 
or else mdadm itself was running it. Since mdadm itself does have some 
timers built-in, the latter is not a totally unreasonable method of 
handling the situation, hence my query to the list.

     To the rest, I wasn't offended.  My skin is much thicker than 
that.  More to the point, my desire to get this fix is the entire 
concern of this thread.  My ego isn't involved.

On 3/11/2020 7:17 AM, Wols Lists wrote:
> On 11/03/20 01:11, Leslie Rhorer wrote:
>>      Is there seriously no one here who knows how checkarray was launched
>> in previous versions?
> You need to ask on a Debian list. I for one don't have a damn clue
> because I actively avoid apt-based systems.
>
> Sorry, I don't mean to be harsh, but each distro "does its own thing" so
> a lot of people (like me) *will* be clueless on that point, even if we
> are raid experts.
>
> Cheers,
> Wol
>> On 3/1/2020 3:03 PM, Leslie Rhorer wrote:
>>>      I have upgraded 2 of my servers to Debian Buster, and now neither
>>> one seems to be running checkarray automatically.  In addition, when I
>>> run checkarray manually, it isn't sending update emails on the status
>>> of the job.  Actually, I have never been able to figure out how
>>> checkarray runs.  One my older servers, there doesn't seem to be
>>> anything in /etc/crontab, /etc/cron.monthly, /etc/init.d/,
>>> /etc/mdadm/mdadm.conf, or /lib/systemd/system/ that would run checkarray.

Re: checkarray not running or emailing

[Leslie Rhorer]

     Aha!  There it is, on both the old and new systems, so it probably 
is running.  The question remains, "Why isn't it posting to email?"

On 3/11/2020 7:50 AM, Brad Campbell wrote:
> On 11/3/20 09:11, Leslie Rhorer wrote:
>>      Is there seriously no one here who knows how checkarray was 
>> launched in previous versions?
>>
>> On 3/1/2020 3:03 PM, Leslie Rhorer wrote:
>>>     I have upgraded 2 of my servers to Debian Buster, and now 
>>> neither one seems to be running checkarray automatically.  In 
>>> addition, when I run checkarray manually, it isn't sending update 
>>> emails on the status of the job.  Actually, I have never been able 
>>> to figure out how checkarray runs.  One my older servers, there 
>>> doesn't seem to be anything in /etc/crontab, /etc/cron.monthly, 
>>> /etc/init.d/, /etc/mdadm/mdadm.conf, or /lib/systemd/system/ that 
>>> would run checkarray.
>>
>
> On mine it's in /etc/cron.d/mdadm
>
> brad@srv:/etc/cron.d$ cat mdadm
> #
> # cron.d/mdadm -- schedules periodic redundancy checks of MD devices
> #
> # Copyright © martin f. krafft <madduck@madduck.net>
> # distributed under the terms of the Artistic Licence 2.0
> #
>
> # By default, run at 00:57 on every Sunday, but do nothing unless the 
> day of
> # the month is less than or equal to 7. Thus, only run on the first 
> Sunday of
> # each month. crontab(5) sucks, unfortunately, in this regard; 
> therefore this
> # hack (see #380425).
> 57 0 * * 0 root if [ -x /usr/share/mdadm/checkarray ] && [ $(date 
> +\%d) -le 7 ]; then /usr/share/mdadm/checkarray --cron --all --idle 
> --quiet; fi
>
> dpkg -L mdadm gave me a list of files and I just checked the cron 
> entries.
>
> I don't run anything that recent, but Debian is Debian.
>
> Brad

Re: checkarray not running or emailing

[Ram Ramesh]

This may be a silly question, but is email working in your system?

I run debian, but a different version. Cron only sends me a message when 
something is wrong. I mean if checkarray failed, I get a message. If my 
array is degraded, I get a message. If everything is fine, no message. 
May be that is the standard. I am not the authority, but that is what is 
happening in my system.

I believe cron emails stdout+stderr of the command run. May be 
checkarray does not output anything in a successful run and therefore no 
email.

It is best to also query debian.user email: debian-user@lists.debian.org

Ramesh

On 3/11/20 8:41 PM, Leslie Rhorer wrote:
>     Aha! There it is, on both the old and new systems, so it probably 
> is running.  The question remains, "Why isn't it posting to email?"
>
> On 3/11/2020 7:50 AM, Brad Campbell wrote:
>> On 11/3/20 09:11, Leslie Rhorer wrote:
>>>      Is there seriously no one here who knows how checkarray was 
>>> launched in previous versions?
>>>
>>> On 3/1/2020 3:03 PM, Leslie Rhorer wrote:
>>>>     I have upgraded 2 of my servers to Debian Buster, and now 
>>>> neither one seems to be running checkarray automatically.  In 
>>>> addition, when I run checkarray manually, it isn't sending update 
>>>> emails on the status of the job.  Actually, I have never been able 
>>>> to figure out how checkarray runs.  One my older servers, there 
>>>> doesn't seem to be anything in /etc/crontab, /etc/cron.monthly, 
>>>> /etc/init.d/, /etc/mdadm/mdadm.conf, or /lib/systemd/system/ that 
>>>> would run checkarray.
>>>
>>
>> On mine it's in /etc/cron.d/mdadm
>>
>> brad@srv:/etc/cron.d$ cat mdadm
>> #
>> # cron.d/mdadm -- schedules periodic redundancy checks of MD devices
>> #
>> # Copyright © martin f. krafft <madduck@madduck.net>
>> # distributed under the terms of the Artistic Licence 2.0
>> #
>>
>> # By default, run at 00:57 on every Sunday, but do nothing unless the 
>> day of
>> # the month is less than or equal to 7. Thus, only run on the first 
>> Sunday of
>> # each month. crontab(5) sucks, unfortunately, in this regard; 
>> therefore this
>> # hack (see #380425).
>> 57 0 * * 0 root if [ -x /usr/share/mdadm/checkarray ] && [ $(date 
>> +\%d) -le 7 ]; then /usr/share/mdadm/checkarray --cron --all --idle 
>> --quiet; fi
>>
>> dpkg -L mdadm gave me a list of files and I just checked the cron 
>> entries.
>>
>> I don't run anything that recent, but Debian is Debian.
>>
>> Brad

Re: checkarray not running or emailing

[Brad Campbell]

On 12/3/20 09:41, Leslie Rhorer wrote:
>     Aha! There it is, on both the old and new systems, so it probably is running.  The question remains, "Why isn't it posting to email?"
>
root@srv:~# mdadm --monitor --test --oneshot /dev/md2
mdadm: Monitor using email address "root" from config file

And in the mail :

This is an automatically generated mail message from mdadm
running on srv

A TestMessage event had been detected on md device /dev/md2.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md2 : active raid1 nvme0n1[2] md5[3](W)
       952696320 blocks super 1.2 [2/2] [UU]
       bitmap: 6/8 pages [24KB], 65536KB chunk



Brad

-- 
An expert is a person who has found out by his own painful
experience all the mistakes that one can make in a very
narrow field. - Niels Bohr

Re: checkarray not running or emailing

[Leslie Rhorer]

     Yes, that works just fine, as does the daily email alert whenever 
an array is degraded, but `checkarray --cron --all --idle` doesn't 
produce an email.  It starts the array verification on all the arrays, 
but no status on the processes are sent.

     The normal execution for the script is to submit `echo check >  
/sys/block/mdx/md/sync_action` and then exit.  Obviously, the rest of 
the processes are handled by the scheduler and mdadm. I'm not seeing how 
anything other than mdadm should be generating the status emails.

     So why does `mdadm --monitor --test --oneshot /dev/mdx` produce an 
email message when `echo check > /sys/block/mdx/md/sync_action` produces 
none?  The "new" servers are running mdadm v4.1 under Debian Buster.  
The "old" servers are running mdadm v3.3.2 under Debian Jessie.

On 3/12/2020 4:17 AM, Brad Campbell wrote:
>
> On 12/3/20 09:41, Leslie Rhorer wrote:
>>     Aha! There it is, on both the old and new systems, so it probably 
>> is running.  The question remains, "Why isn't it posting to email?"
>>
> root@srv:~# mdadm --monitor --test --oneshot /dev/md2
> mdadm: Monitor using email address "root" from config file
>
> And in the mail :
>
> This is an automatically generated mail message from mdadm
> running on srv
>
> A TestMessage event had been detected on md device /dev/md2.
>
> Faithfully yours, etc.
>
> P.S. The /proc/mdstat file currently contains the following:
>
> Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
> md2 : active raid1 nvme0n1[2] md5[3](W)
>       952696320 blocks super 1.2 [2/2] [UU]
>       bitmap: 6/8 pages [24KB], 65536KB chunk
>
>
>
> Brad
>

Re: checkarray not running or emailing

[Leslie Rhorer]

On 3/11/2020 11:53 PM, Ram Ramesh wrote:
> This may be a silly question, but is email working in your system?

     It's not a silly question, but I believe I already answered it.  
Yes, email is working, and yes, mdadm is able to send messages.  It 
complains every morning at 07:35 if one f the arrays is degraded, and 
the test function works.

> I run debian, but a different version. Cron only sends me a message 
> when something is wrong. I mean if checkarray failed, I get a message. 
> If my array is degraded, I get a message. If everything is fine, no 
> message. May be that is the standard. I am not the authority, but that 
> is what is happening in my system.

     I don't believe cron sends the messages for the array redundancy 
check, since they are sent roughly every 20% of the sync task, not on 
any particular time schedule.  Of course, the status check is run by 
some timer, and cron could certainly handle such a task on a 
minute-by-minute basis, but I don't see any cron task related to the 
array structure other than /etc/cron.d/mdadm, which is the checkarray 
task itself, run once a month.

> I believe cron emails stdout+stderr of the command run. May be 
> checkarray does not output anything in a successful run and therefore 
> no email.

     Checkarray normally doesn't output anything when run fro, the cron 
task, which employs the --quiet switch, nor does cron if checkarray is 
successful.  It merely initiates the check action on every specified 
array and exits.  Something else - presumably mdadm - sends an email as 
the array check proceeds every 20%.

> It is best to also query debian.user email: debian-user@lists.debian.org
         I will try querying the Debian list, but while the cron job was 
a Debian feature, I don't believe this issue is one with Debian, per se.
> Ramesh
>
> On 3/11/20 8:41 PM, Leslie Rhorer wrote:
>>     Aha! There it is, on both the old and new systems, so it probably 
>> is running.  The question remains, "Why isn't it posting to email?"
>>
>> On 3/11/2020 7:50 AM, Brad Campbell wrote:
>>> On 11/3/20 09:11, Leslie Rhorer wrote:
>>>>      Is there seriously no one here who knows how checkarray was 
>>>> launched in previous versions?
>>>>
>>>> On 3/1/2020 3:03 PM, Leslie Rhorer wrote:
>>>>>     I have upgraded 2 of my servers to Debian Buster, and now 
>>>>> neither one seems to be running checkarray automatically.  In 
>>>>> addition, when I run checkarray manually, it isn't sending update 
>>>>> emails on the status of the job.  Actually, I have never been able 
>>>>> to figure out how checkarray runs.  One my older servers, there 
>>>>> doesn't seem to be anything in /etc/crontab, /etc/cron.monthly, 
>>>>> /etc/init.d/, /etc/mdadm/mdadm.conf, or /lib/systemd/system/ that 
>>>>> would run checkarray.
>>>>
>>>
>>> On mine it's in /etc/cron.d/mdadm
>>>
>>> brad@srv:/etc/cron.d$ cat mdadm
>>> #
>>> # cron.d/mdadm -- schedules periodic redundancy checks of MD devices
>>> #
>>> # Copyright © martin f. krafft <madduck@madduck.net>
>>> # distributed under the terms of the Artistic Licence 2.0
>>> #
>>>
>>> # By default, run at 00:57 on every Sunday, but do nothing unless 
>>> the day of
>>> # the month is less than or equal to 7. Thus, only run on the first 
>>> Sunday of
>>> # each month. crontab(5) sucks, unfortunately, in this regard; 
>>> therefore this
>>> # hack (see #380425).
>>> 57 0 * * 0 root if [ -x /usr/share/mdadm/checkarray ] && [ $(date 
>>> +\%d) -le 7 ]; then /usr/share/mdadm/checkarray --cron --all --idle 
>>> --quiet; fi
>>>
>>> dpkg -L mdadm gave me a list of files and I just checked the cron 
>>> entries.
>>>
>>> I don't run anything that recent, but Debian is Debian.
>>>
>>> Brad

Re: checkarray not running or emailing

[Brad Campbell]

On 13/3/20 08:13, Leslie Rhorer wrote:
>     Yes, that works just fine, as does the daily email alert whenever an array is degraded, but `checkarray --cron --all --idle` doesn't produce an email.  It starts the array verification on all the arrays, but no status on the processes are sent.
>
>     The normal execution for the script is to submit `echo check >  /sys/block/mdx/md/sync_action` and then exit.  Obviously, the rest of the processes are handled by the scheduler and mdadm. I'm not seeing how anything other than mdadm should be generating the status emails.
>
I don't get messages from checkarray.

I get them from logcheck after they get dumped in the syslog :

This email is sent by logcheck. If you no longer wish to receive
such mail, you can either deinstall the logcheck package or modify
its configuration file (/etc/logcheck/logcheck.conf).

System Events
=-=-=-=-=-=-=
Mar  1 01:09:17 srv mdadm[4471]: RebuildFinished event detected on md device /dev/md2, component device  mismatches found: 230656 (on raid level 1)


(Lots of errors. One drive supports deterministic read after trim and the other doesn't)

I've never really looked at it any closer than that to be honest.

Brad.
-- 

An expert is a person who has found out by his own painful
experience all the mistakes that one can make in a very
narrow field. - Niels Bohr

Re: checkarray not running or emailing

[Leslie Rhorer]

     Well, it definitely is not from checkarray, per se, since the 
script completes and exits in a matter of a few milliseconds, but 
something, presumably mdadm does (or at least used to) send status reports:

Subject: Backup-Server mdadm Event Notification

RebuildStarted /dev/md0

RebuildStarted /dev/md3

RebuildFinished /dev/md3

RebuildStarted /dev/md2

RebuildFinished /dev/md2

Rebuild60 /dev/md0

Rebuild80 /dev/md0

RebuildFinished /dev/md0

On 3/13/2020 11:53 AM, Brad Campbell wrote:
> On 13/3/20 08:13, Leslie Rhorer wrote:
>>     Yes, that works just fine, as does the daily email alert whenever 
>> an array is degraded, but `checkarray --cron --all --idle` doesn't 
>> produce an email.  It starts the array verification on all the 
>> arrays, but no status on the processes are sent.
>>
>>     The normal execution for the script is to submit `echo check >  
>> /sys/block/mdx/md/sync_action` and then exit.  Obviously, the rest of 
>> the processes are handled by the scheduler and mdadm. I'm not seeing 
>> how anything other than mdadm should be generating the status emails.
>>
> I don't get messages from checkarray.
>
> I get them from logcheck after they get dumped in the syslog :
>
> This email is sent by logcheck. If you no longer wish to receive
> such mail, you can either deinstall the logcheck package or modify
> its configuration file (/etc/logcheck/logcheck.conf).
>
> System Events
> =-=-=-=-=-=-=
> Mar  1 01:09:17 srv mdadm[4471]: RebuildFinished event detected on md 
> device /dev/md2, component device  mismatches found: 230656 (on raid 
> level 1)
>
>
> (Lots of errors. One drive supports deterministic read after trim and 
> the other doesn't)
>
> I've never really looked at it any closer than that to be honest.
>
> Brad.