* checkarray not running or emailing [not found] <814aad65-fba3-334c-c4df-6b8f4bfc4193.ref@att.net> @ 2020-03-01 21:03 ` Leslie Rhorer 2020-03-11 1:11 ` Leslie Rhorer 0 siblings, 1 reply; 13+ messages in thread From: Leslie Rhorer @ 2020-03-01 21:03 UTC (permalink / raw) To: linux-raid I have upgraded 2 of my servers to Debian Buster, and now neither one seems to be running checkarray automatically. In addition, when I run checkarray manually, it isn't sending update emails on the status of the job. Actually, I have never been able to figure out how checkarray runs. One my older servers, there doesn't seem to be anything in /etc/crontab, /etc/cron.monthly, /etc/init.d/, /etc/mdadm/mdadm.conf, or /lib/systemd/system/ that would run checkarray. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: checkarray not running or emailing 2020-03-01 21:03 ` checkarray not running or emailing Leslie Rhorer @ 2020-03-11 1:11 ` Leslie Rhorer 2020-03-11 12:17 ` Wols Lists ` (2 more replies) 0 siblings, 3 replies; 13+ messages in thread From: Leslie Rhorer @ 2020-03-11 1:11 UTC (permalink / raw) To: linux-raid Is there seriously no one here who knows how checkarray was launched in previous versions? On 3/1/2020 3:03 PM, Leslie Rhorer wrote: > I have upgraded 2 of my servers to Debian Buster, and now neither > one seems to be running checkarray automatically. In addition, when I > run checkarray manually, it isn't sending update emails on the status > of the job. Actually, I have never been able to figure out how > checkarray runs. One my older servers, there doesn't seem to be > anything in /etc/crontab, /etc/cron.monthly, /etc/init.d/, > /etc/mdadm/mdadm.conf, or /lib/systemd/system/ that would run checkarray. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: checkarray not running or emailing 2020-03-11 1:11 ` Leslie Rhorer @ 2020-03-11 12:17 ` Wols Lists 2020-03-12 1:17 ` Leslie Rhorer 2020-03-11 12:50 ` Brad Campbell 2020-03-11 13:18 ` Robin Hill 2 siblings, 1 reply; 13+ messages in thread From: Wols Lists @ 2020-03-11 12:17 UTC (permalink / raw) To: Leslie Rhorer, linux-raid On 11/03/20 01:11, Leslie Rhorer wrote: > Is there seriously no one here who knows how checkarray was launched > in previous versions? You need to ask on a Debian list. I for one don't have a damn clue because I actively avoid apt-based systems. Sorry, I don't mean to be harsh, but each distro "does its own thing" so a lot of people (like me) *will* be clueless on that point, even if we are raid experts. Cheers, Wol > > On 3/1/2020 3:03 PM, Leslie Rhorer wrote: >> I have upgraded 2 of my servers to Debian Buster, and now neither >> one seems to be running checkarray automatically. In addition, when I >> run checkarray manually, it isn't sending update emails on the status >> of the job. Actually, I have never been able to figure out how >> checkarray runs. One my older servers, there doesn't seem to be >> anything in /etc/crontab, /etc/cron.monthly, /etc/init.d/, >> /etc/mdadm/mdadm.conf, or /lib/systemd/system/ that would run checkarray. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: checkarray not running or emailing 2020-03-11 12:17 ` Wols Lists @ 2020-03-12 1:17 ` Leslie Rhorer 0 siblings, 0 replies; 13+ messages in thread From: Leslie Rhorer @ 2020-03-12 1:17 UTC (permalink / raw) To: Wols Lists, linux-raid Well, that is rather the point, actually. I can't find anywhere that Debian *DID* run the script in the previous versions. Failing that, either Debian was doing it somewhere of which I have no knowledge, or else mdadm itself was running it. Since mdadm itself does have some timers built-in, the latter is not a totally unreasonable method of handling the situation, hence my query to the list. To the rest, I wasn't offended. My skin is much thicker than that. More to the point, my desire to get this fix is the entire concern of this thread. My ego isn't involved. On 3/11/2020 7:17 AM, Wols Lists wrote: > On 11/03/20 01:11, Leslie Rhorer wrote: >> Is there seriously no one here who knows how checkarray was launched >> in previous versions? > You need to ask on a Debian list. I for one don't have a damn clue > because I actively avoid apt-based systems. > > Sorry, I don't mean to be harsh, but each distro "does its own thing" so > a lot of people (like me) *will* be clueless on that point, even if we > are raid experts. > > Cheers, > Wol >> On 3/1/2020 3:03 PM, Leslie Rhorer wrote: >>> I have upgraded 2 of my servers to Debian Buster, and now neither >>> one seems to be running checkarray automatically. In addition, when I >>> run checkarray manually, it isn't sending update emails on the status >>> of the job. Actually, I have never been able to figure out how >>> checkarray runs. One my older servers, there doesn't seem to be >>> anything in /etc/crontab, /etc/cron.monthly, /etc/init.d/, >>> /etc/mdadm/mdadm.conf, or /lib/systemd/system/ that would run checkarray. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: checkarray not running or emailing 2020-03-11 1:11 ` Leslie Rhorer 2020-03-11 12:17 ` Wols Lists @ 2020-03-11 12:50 ` Brad Campbell 2020-03-12 1:41 ` Leslie Rhorer 2020-03-11 13:18 ` Robin Hill 2 siblings, 1 reply; 13+ messages in thread From: Brad Campbell @ 2020-03-11 12:50 UTC (permalink / raw) To: Leslie Rhorer, linux-raid On 11/3/20 09:11, Leslie Rhorer wrote: > Is there seriously no one here who knows how checkarray was launched in previous versions? > > On 3/1/2020 3:03 PM, Leslie Rhorer wrote: >> I have upgraded 2 of my servers to Debian Buster, and now neither one seems to be running checkarray automatically. In addition, when I run checkarray manually, it isn't sending update emails on the status of the job. Actually, I have never been able to figure out how checkarray runs. One my older servers, there doesn't seem to be anything in /etc/crontab, /etc/cron.monthly, /etc/init.d/, /etc/mdadm/mdadm.conf, or /lib/systemd/system/ that would run checkarray. > On mine it's in /etc/cron.d/mdadm brad@srv:/etc/cron.d$ cat mdadm # # cron.d/mdadm -- schedules periodic redundancy checks of MD devices # # Copyright © martin f. krafft <madduck@madduck.net> # distributed under the terms of the Artistic Licence 2.0 # # By default, run at 00:57 on every Sunday, but do nothing unless the day of # the month is less than or equal to 7. Thus, only run on the first Sunday of # each month. crontab(5) sucks, unfortunately, in this regard; therefore this # hack (see #380425). 57 0 * * 0 root if [ -x /usr/share/mdadm/checkarray ] && [ $(date +\%d) -le 7 ]; then /usr/share/mdadm/checkarray --cron --all --idle --quiet; fi dpkg -L mdadm gave me a list of files and I just checked the cron entries. I don't run anything that recent, but Debian is Debian. Brad -- An expert is a person who has found out by his own painful experience all the mistakes that one can make in a very narrow field. - Niels Bohr ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: checkarray not running or emailing 2020-03-11 12:50 ` Brad Campbell @ 2020-03-12 1:41 ` Leslie Rhorer 2020-03-12 4:53 ` Ram Ramesh 2020-03-12 9:17 ` Brad Campbell 0 siblings, 2 replies; 13+ messages in thread From: Leslie Rhorer @ 2020-03-12 1:41 UTC (permalink / raw) To: Brad Campbell, linux-raid Aha! There it is, on both the old and new systems, so it probably is running. The question remains, "Why isn't it posting to email?" On 3/11/2020 7:50 AM, Brad Campbell wrote: > On 11/3/20 09:11, Leslie Rhorer wrote: >> Is there seriously no one here who knows how checkarray was >> launched in previous versions? >> >> On 3/1/2020 3:03 PM, Leslie Rhorer wrote: >>> I have upgraded 2 of my servers to Debian Buster, and now >>> neither one seems to be running checkarray automatically. In >>> addition, when I run checkarray manually, it isn't sending update >>> emails on the status of the job. Actually, I have never been able >>> to figure out how checkarray runs. One my older servers, there >>> doesn't seem to be anything in /etc/crontab, /etc/cron.monthly, >>> /etc/init.d/, /etc/mdadm/mdadm.conf, or /lib/systemd/system/ that >>> would run checkarray. >> > > On mine it's in /etc/cron.d/mdadm > > brad@srv:/etc/cron.d$ cat mdadm > # > # cron.d/mdadm -- schedules periodic redundancy checks of MD devices > # > # Copyright © martin f. krafft <madduck@madduck.net> > # distributed under the terms of the Artistic Licence 2.0 > # > > # By default, run at 00:57 on every Sunday, but do nothing unless the > day of > # the month is less than or equal to 7. Thus, only run on the first > Sunday of > # each month. crontab(5) sucks, unfortunately, in this regard; > therefore this > # hack (see #380425). > 57 0 * * 0 root if [ -x /usr/share/mdadm/checkarray ] && [ $(date > +\%d) -le 7 ]; then /usr/share/mdadm/checkarray --cron --all --idle > --quiet; fi > > dpkg -L mdadm gave me a list of files and I just checked the cron > entries. > > I don't run anything that recent, but Debian is Debian. > > Brad ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: checkarray not running or emailing 2020-03-12 1:41 ` Leslie Rhorer @ 2020-03-12 4:53 ` Ram Ramesh 2020-03-13 0:24 ` Leslie Rhorer 2020-03-12 9:17 ` Brad Campbell 1 sibling, 1 reply; 13+ messages in thread From: Ram Ramesh @ 2020-03-12 4:53 UTC (permalink / raw) To: Leslie Rhorer, Brad Campbell, linux-raid This may be a silly question, but is email working in your system? I run debian, but a different version. Cron only sends me a message when something is wrong. I mean if checkarray failed, I get a message. If my array is degraded, I get a message. If everything is fine, no message. May be that is the standard. I am not the authority, but that is what is happening in my system. I believe cron emails stdout+stderr of the command run. May be checkarray does not output anything in a successful run and therefore no email. It is best to also query debian.user email: debian-user@lists.debian.org Ramesh On 3/11/20 8:41 PM, Leslie Rhorer wrote: > Aha! There it is, on both the old and new systems, so it probably > is running. The question remains, "Why isn't it posting to email?" > > On 3/11/2020 7:50 AM, Brad Campbell wrote: >> On 11/3/20 09:11, Leslie Rhorer wrote: >>> Is there seriously no one here who knows how checkarray was >>> launched in previous versions? >>> >>> On 3/1/2020 3:03 PM, Leslie Rhorer wrote: >>>> I have upgraded 2 of my servers to Debian Buster, and now >>>> neither one seems to be running checkarray automatically. In >>>> addition, when I run checkarray manually, it isn't sending update >>>> emails on the status of the job. Actually, I have never been able >>>> to figure out how checkarray runs. One my older servers, there >>>> doesn't seem to be anything in /etc/crontab, /etc/cron.monthly, >>>> /etc/init.d/, /etc/mdadm/mdadm.conf, or /lib/systemd/system/ that >>>> would run checkarray. >>> >> >> On mine it's in /etc/cron.d/mdadm >> >> brad@srv:/etc/cron.d$ cat mdadm >> # >> # cron.d/mdadm -- schedules periodic redundancy checks of MD devices >> # >> # Copyright © martin f. krafft <madduck@madduck.net> >> # distributed under the terms of the Artistic Licence 2.0 >> # >> >> # By default, run at 00:57 on every Sunday, but do nothing unless the >> day of >> # the month is less than or equal to 7. Thus, only run on the first >> Sunday of >> # each month. crontab(5) sucks, unfortunately, in this regard; >> therefore this >> # hack (see #380425). >> 57 0 * * 0 root if [ -x /usr/share/mdadm/checkarray ] && [ $(date >> +\%d) -le 7 ]; then /usr/share/mdadm/checkarray --cron --all --idle >> --quiet; fi >> >> dpkg -L mdadm gave me a list of files and I just checked the cron >> entries. >> >> I don't run anything that recent, but Debian is Debian. >> >> Brad ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: checkarray not running or emailing 2020-03-12 4:53 ` Ram Ramesh @ 2020-03-13 0:24 ` Leslie Rhorer 0 siblings, 0 replies; 13+ messages in thread From: Leslie Rhorer @ 2020-03-13 0:24 UTC (permalink / raw) To: Ram Ramesh, Brad Campbell, linux-raid On 3/11/2020 11:53 PM, Ram Ramesh wrote: > This may be a silly question, but is email working in your system? It's not a silly question, but I believe I already answered it. Yes, email is working, and yes, mdadm is able to send messages. It complains every morning at 07:35 if one f the arrays is degraded, and the test function works. > I run debian, but a different version. Cron only sends me a message > when something is wrong. I mean if checkarray failed, I get a message. > If my array is degraded, I get a message. If everything is fine, no > message. May be that is the standard. I am not the authority, but that > is what is happening in my system. I don't believe cron sends the messages for the array redundancy check, since they are sent roughly every 20% of the sync task, not on any particular time schedule. Of course, the status check is run by some timer, and cron could certainly handle such a task on a minute-by-minute basis, but I don't see any cron task related to the array structure other than /etc/cron.d/mdadm, which is the checkarray task itself, run once a month. > I believe cron emails stdout+stderr of the command run. May be > checkarray does not output anything in a successful run and therefore > no email. Checkarray normally doesn't output anything when run fro, the cron task, which employs the --quiet switch, nor does cron if checkarray is successful. It merely initiates the check action on every specified array and exits. Something else - presumably mdadm - sends an email as the array check proceeds every 20%. > It is best to also query debian.user email: debian-user@lists.debian.org I will try querying the Debian list, but while the cron job was a Debian feature, I don't believe this issue is one with Debian, per se. > Ramesh > > On 3/11/20 8:41 PM, Leslie Rhorer wrote: >> Aha! There it is, on both the old and new systems, so it probably >> is running. The question remains, "Why isn't it posting to email?" >> >> On 3/11/2020 7:50 AM, Brad Campbell wrote: >>> On 11/3/20 09:11, Leslie Rhorer wrote: >>>> Is there seriously no one here who knows how checkarray was >>>> launched in previous versions? >>>> >>>> On 3/1/2020 3:03 PM, Leslie Rhorer wrote: >>>>> I have upgraded 2 of my servers to Debian Buster, and now >>>>> neither one seems to be running checkarray automatically. In >>>>> addition, when I run checkarray manually, it isn't sending update >>>>> emails on the status of the job. Actually, I have never been able >>>>> to figure out how checkarray runs. One my older servers, there >>>>> doesn't seem to be anything in /etc/crontab, /etc/cron.monthly, >>>>> /etc/init.d/, /etc/mdadm/mdadm.conf, or /lib/systemd/system/ that >>>>> would run checkarray. >>>> >>> >>> On mine it's in /etc/cron.d/mdadm >>> >>> brad@srv:/etc/cron.d$ cat mdadm >>> # >>> # cron.d/mdadm -- schedules periodic redundancy checks of MD devices >>> # >>> # Copyright © martin f. krafft <madduck@madduck.net> >>> # distributed under the terms of the Artistic Licence 2.0 >>> # >>> >>> # By default, run at 00:57 on every Sunday, but do nothing unless >>> the day of >>> # the month is less than or equal to 7. Thus, only run on the first >>> Sunday of >>> # each month. crontab(5) sucks, unfortunately, in this regard; >>> therefore this >>> # hack (see #380425). >>> 57 0 * * 0 root if [ -x /usr/share/mdadm/checkarray ] && [ $(date >>> +\%d) -le 7 ]; then /usr/share/mdadm/checkarray --cron --all --idle >>> --quiet; fi >>> >>> dpkg -L mdadm gave me a list of files and I just checked the cron >>> entries. >>> >>> I don't run anything that recent, but Debian is Debian. >>> >>> Brad ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: checkarray not running or emailing 2020-03-12 1:41 ` Leslie Rhorer 2020-03-12 4:53 ` Ram Ramesh @ 2020-03-12 9:17 ` Brad Campbell 2020-03-13 0:13 ` Leslie Rhorer 1 sibling, 1 reply; 13+ messages in thread From: Brad Campbell @ 2020-03-12 9:17 UTC (permalink / raw) To: Leslie Rhorer, linux-raid On 12/3/20 09:41, Leslie Rhorer wrote: > Aha! There it is, on both the old and new systems, so it probably is running. The question remains, "Why isn't it posting to email?" > root@srv:~# mdadm --monitor --test --oneshot /dev/md2 mdadm: Monitor using email address "root" from config file And in the mail : This is an automatically generated mail message from mdadm running on srv A TestMessage event had been detected on md device /dev/md2. Faithfully yours, etc. P.S. The /proc/mdstat file currently contains the following: Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] md2 : active raid1 nvme0n1[2] md5[3](W) 952696320 blocks super 1.2 [2/2] [UU] bitmap: 6/8 pages [24KB], 65536KB chunk Brad -- An expert is a person who has found out by his own painful experience all the mistakes that one can make in a very narrow field. - Niels Bohr ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: checkarray not running or emailing 2020-03-12 9:17 ` Brad Campbell @ 2020-03-13 0:13 ` Leslie Rhorer 2020-03-13 16:53 ` Brad Campbell 0 siblings, 1 reply; 13+ messages in thread From: Leslie Rhorer @ 2020-03-13 0:13 UTC (permalink / raw) To: Brad Campbell, linux-raid Yes, that works just fine, as does the daily email alert whenever an array is degraded, but `checkarray --cron --all --idle` doesn't produce an email. It starts the array verification on all the arrays, but no status on the processes are sent. The normal execution for the script is to submit `echo check > /sys/block/mdx/md/sync_action` and then exit. Obviously, the rest of the processes are handled by the scheduler and mdadm. I'm not seeing how anything other than mdadm should be generating the status emails. So why does `mdadm --monitor --test --oneshot /dev/mdx` produce an email message when `echo check > /sys/block/mdx/md/sync_action` produces none? The "new" servers are running mdadm v4.1 under Debian Buster. The "old" servers are running mdadm v3.3.2 under Debian Jessie. On 3/12/2020 4:17 AM, Brad Campbell wrote: > > On 12/3/20 09:41, Leslie Rhorer wrote: >> Aha! There it is, on both the old and new systems, so it probably >> is running. The question remains, "Why isn't it posting to email?" >> > root@srv:~# mdadm --monitor --test --oneshot /dev/md2 > mdadm: Monitor using email address "root" from config file > > And in the mail : > > This is an automatically generated mail message from mdadm > running on srv > > A TestMessage event had been detected on md device /dev/md2. > > Faithfully yours, etc. > > P.S. The /proc/mdstat file currently contains the following: > > Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] > md2 : active raid1 nvme0n1[2] md5[3](W) > 952696320 blocks super 1.2 [2/2] [UU] > bitmap: 6/8 pages [24KB], 65536KB chunk > > > > Brad > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: checkarray not running or emailing 2020-03-13 0:13 ` Leslie Rhorer @ 2020-03-13 16:53 ` Brad Campbell 2020-03-15 11:11 ` Leslie Rhorer 0 siblings, 1 reply; 13+ messages in thread From: Brad Campbell @ 2020-03-13 16:53 UTC (permalink / raw) To: Leslie Rhorer, linux-raid On 13/3/20 08:13, Leslie Rhorer wrote: > Yes, that works just fine, as does the daily email alert whenever an array is degraded, but `checkarray --cron --all --idle` doesn't produce an email. It starts the array verification on all the arrays, but no status on the processes are sent. > > The normal execution for the script is to submit `echo check > /sys/block/mdx/md/sync_action` and then exit. Obviously, the rest of the processes are handled by the scheduler and mdadm. I'm not seeing how anything other than mdadm should be generating the status emails. > I don't get messages from checkarray. I get them from logcheck after they get dumped in the syslog : This email is sent by logcheck. If you no longer wish to receive such mail, you can either deinstall the logcheck package or modify its configuration file (/etc/logcheck/logcheck.conf). System Events =-=-=-=-=-=-= Mar 1 01:09:17 srv mdadm[4471]: RebuildFinished event detected on md device /dev/md2, component device mismatches found: 230656 (on raid level 1) (Lots of errors. One drive supports deterministic read after trim and the other doesn't) I've never really looked at it any closer than that to be honest. Brad. -- An expert is a person who has found out by his own painful experience all the mistakes that one can make in a very narrow field. - Niels Bohr ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: checkarray not running or emailing 2020-03-13 16:53 ` Brad Campbell @ 2020-03-15 11:11 ` Leslie Rhorer 0 siblings, 0 replies; 13+ messages in thread From: Leslie Rhorer @ 2020-03-15 11:11 UTC (permalink / raw) To: linux-raid Well, it definitely is not from checkarray, per se, since the script completes and exits in a matter of a few milliseconds, but something, presumably mdadm does (or at least used to) send status reports: Subject: Backup-Server mdadm Event Notification RebuildStarted /dev/md0 RebuildStarted /dev/md3 RebuildFinished /dev/md3 RebuildStarted /dev/md2 RebuildFinished /dev/md2 Rebuild60 /dev/md0 Rebuild80 /dev/md0 RebuildFinished /dev/md0 On 3/13/2020 11:53 AM, Brad Campbell wrote: > On 13/3/20 08:13, Leslie Rhorer wrote: >> Yes, that works just fine, as does the daily email alert whenever >> an array is degraded, but `checkarray --cron --all --idle` doesn't >> produce an email. It starts the array verification on all the >> arrays, but no status on the processes are sent. >> >> The normal execution for the script is to submit `echo check > >> /sys/block/mdx/md/sync_action` and then exit. Obviously, the rest of >> the processes are handled by the scheduler and mdadm. I'm not seeing >> how anything other than mdadm should be generating the status emails. >> > I don't get messages from checkarray. > > I get them from logcheck after they get dumped in the syslog : > > This email is sent by logcheck. If you no longer wish to receive > such mail, you can either deinstall the logcheck package or modify > its configuration file (/etc/logcheck/logcheck.conf). > > System Events > =-=-=-=-=-=-= > Mar 1 01:09:17 srv mdadm[4471]: RebuildFinished event detected on md > device /dev/md2, component device mismatches found: 230656 (on raid > level 1) > > > (Lots of errors. One drive supports deterministic read after trim and > the other doesn't) > > I've never really looked at it any closer than that to be honest. > > Brad. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: checkarray not running or emailing 2020-03-11 1:11 ` Leslie Rhorer 2020-03-11 12:17 ` Wols Lists 2020-03-11 12:50 ` Brad Campbell @ 2020-03-11 13:18 ` Robin Hill 2 siblings, 0 replies; 13+ messages in thread From: Robin Hill @ 2020-03-11 13:18 UTC (permalink / raw) To: Leslie Rhorer; +Cc: linux-raid On Tue Mar 10, 2020 at 08:11:36PM -0500, Leslie Rhorer wrote: > Is there seriously no one here who knows how checkarray was > launched in previous versions? > > On 3/1/2020 3:03 PM, Leslie Rhorer wrote: > > I have upgraded 2 of my servers to Debian Buster, and now neither > > one seems to be running checkarray automatically. In addition, when I > > run checkarray manually, it isn't sending update emails on the status > > of the job. Actually, I have never been able to figure out how > > checkarray runs. One my older servers, there doesn't seem to be > > anything in /etc/crontab, /etc/cron.monthly, /etc/init.d/, > > /etc/mdadm/mdadm.conf, or /lib/systemd/system/ that would run checkarray. Google suggests it's run from /etc/cron.d/mdadm. Does this exist on your systems? I'm not sure why upgrading to Buster would have stopped the checks though, unless the script wasn't replaced and the file location for checkarray has changed. Cheers, Robin -- ___ ( ' } | Robin Hill <robin@robinhill.me.uk> | / / ) | Little Jim says .... | // !! | "He fallen in de water !!" | ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2020-03-15 11:11 UTC | newest] Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <814aad65-fba3-334c-c4df-6b8f4bfc4193.ref@att.net> 2020-03-01 21:03 ` checkarray not running or emailing Leslie Rhorer 2020-03-11 1:11 ` Leslie Rhorer 2020-03-11 12:17 ` Wols Lists 2020-03-12 1:17 ` Leslie Rhorer 2020-03-11 12:50 ` Brad Campbell 2020-03-12 1:41 ` Leslie Rhorer 2020-03-12 4:53 ` Ram Ramesh 2020-03-13 0:24 ` Leslie Rhorer 2020-03-12 9:17 ` Brad Campbell 2020-03-13 0:13 ` Leslie Rhorer 2020-03-13 16:53 ` Brad Campbell 2020-03-15 11:11 ` Leslie Rhorer 2020-03-11 13:18 ` Robin Hill
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.