[dm-devel] How to debug missing devices in `multipath -l` (and …`-ll`)

[dm-devel] How to debug missing devices in `multipath -l` (and …`-ll`)

[Konstantin Kharlamov]

Hello! Sorry if it's the wrong list, it is referred to by multipath-tools
without saying it's "devel", so I presume user questions are okay…?

How to debug a lack of devices in `multipath -l` and `… -ll` outputs? I'm
studying `multipathd -dv5` logs, but it's not at all clear to me as to what to
look for.

At $WORK I'm trying to replace the older Ubuntu shipped multipath-tools with the
upstream one. Then, given a system with the upstream `multipath-tools`
preinstalled and `multipathd` running, none of the `multipath -l` or `multipath
-ll` show output. Then, if I downgrade `multipath-tools` to the one from Ubuntu
repository and restart `multipathd`, devices do appear.

Then it gets weird: if I upgrade `multipath-tools` back to the latest one
upstream, and reboot, devices are still there! It starts working!

At this point I found the `/etc/multipath/wwids` which might be the reason. I
presume `multipathd` puts devices there under some conditions. But where can I
see how it decides whether to work with devices or to ignore them? I'm looking
through -v5 startup logs when it's working and when it doesn't, there's a lot of
output, but it's hard to `diff` the older (working) and the newer (not working)
output — a lot of style changes. Entries I find suspicious are repeated in both
cases.

I figured, it might ask here for pointers, is there something specific to look
for?

--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] How to debug missing devices in `multipath -l` (and …`-ll`)

[Konstantin Kharlamov]

On Tue, 2022-09-13 at 17:37 +0300, Konstantin Kharlamov wrote:
> Hello! Sorry if it's the wrong list, it is referred to by multipath-tools
> without saying it's "devel", so I presume user questions are okay…?
> 
> How to debug a lack of devices in `multipath -l` and `… -ll` outputs? I'm
> studying `multipathd -dv5` logs, but it's not at all clear to me as to what to
> look for.
> 
> At $WORK I'm trying to replace the older Ubuntu shipped multipath-tools with
> the
> upstream one. Then, given a system with the upstream `multipath-tools`
> preinstalled and `multipathd` running, none of the `multipath -l` or
> `multipath
> -ll` show output. Then, if I downgrade `multipath-tools` to the one from
> Ubuntu
> repository and restart `multipathd`, devices do appear.
> 
> Then it gets weird: if I upgrade `multipath-tools` back to the latest one
> upstream, and reboot, devices are still there! It starts working!
> 
> At this point I found the `/etc/multipath/wwids` which might be the reason. I
> presume `multipathd` puts devices there under some conditions. But where can I
> see how it decides whether to work with devices or to ignore them? I'm looking
> through -v5 startup logs when it's working and when it doesn't, there's a lot
> of
> output, but it's hard to `diff` the older (working) and the newer (not
> working)
> output — a lot of style changes. Entries I find suspicious are repeated in
> both
> cases.
> 
> I figured, it might ask here for pointers, is there something specific to look
> for?

I found the problem, and in retrospective I doubt the logs would have helped me.

The problem was that we relied on default value of `find_multipaths`, but then
upstream changed it, which made everything silently break on update. May I
grunt: that upstream change was a terrible decision, because if you have a
system with "wwids" file, it will actually continue working… For some time.
Until problems with missing devices would silently creep up on you. We got very
lucky here, since our infrastrcture allowed to spot that immediately.

Anyway, what's done is done, and the change is dated by 2018 year, so oh well…
C'est la vie.

So, anyway, returning to my original question: I'm not sure I have had any
chance to figure that problem out from the logs. I mean, the logs do mention
something like "device foo was skipped as it's not in wwids file", but at the
same time I know that the older version recreates the wwids file. And actually,
even the `wwids` file say at the top a "This file is automatically generated and
thou shalt not touch it". So there's no reason to assume wwids file has anything
to do with the problem whatsoever.

In my case what helped was the good ol' friend git-bisect.

--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel