Re: Diskdump / kdump format history

["Petr Tesařík" <petr@tesarici.cz>]

[-- Attachment #1: Type: text/plain, Size: 4113 bytes --]

Hi Stephen,

thanks for your trust. I've been around for some time, but I actually
came late to the party. Below is my understanding of what happened.
People who were really involved in all that may add a lot of
corrections.

On Thu, 12 Jan 2023 16:25:10 -0800
Stephen Brennan <stephen.s.brennan@oracle.com> wrote:

>[...]
> First, regarding naming. I guess I always assumed that "diskdump"
> referred to some older utility which performed the same function that
> makedumpfile now is commonly used for. Is that the case?

Not quite. It did not perform the same function...

Let me make a short historical excursion.

When I joined SUSE, SLES9 was the latest release and it was based
on kernel 2.6.5. There was no kdump at that time, but a team at
SGI worked on LKCD (Linux Kernel Crash Dumps), which was an
out-of-tree patch to the Linux Kernel, some user-space tools for
setting it up and the lcrash analysis tool (which was, as a matter
of fact, simply horrible). Most importantly, the dumping itself
was performed by the crashed kernel, using a special polling mode
that was implemented by a few device drivers. LKCD offered two
options: netdump and diskdump. This is most likely where the term
"diskdump" was first coined.

> Do you know the origin of the diskdump format?

Yes. Fujitsu developed their own Linux kernel dump project called
lkdump. It was conceptually similar to LKCD, as it also ran within
the crashed kernel, but it did not use the LKCD file format. IIUC
they first modified LKCD for their Mission Critical Linux (or
MCLX), but they soon discovered that there were some issues with
the format itself and designed their own format with the signature
"DISKDUMP".

> Do you know where to find the code for
> this older diskdump tool, if it existed?

Yes. Have a look here:

https://sourceforge.net/projects/lkdump/

> Second, I have some confusion regarding headers. I've seen headers for
> files with the content "DISKDUMP" and "KDUMP   " -- and according to
> src/kdumpfile/diskdump.c, you have too. diskdump_probe() shows that you
> check for these two headers and set a different description depending on
> which is seen. Do you know why or where these two headers are set, and
> which one is chose in which situation?

"DISKDUMP" is the signature of the original Fujitsu lkdump
files. "KDUMP   " is the signature of the makedumpfile modification
(that's the meaning of "_mod" in makedumpfile's diskdump_mod.h
header file name). Yes, there was a header_version field, but that
wasn't deemed sufficient. Since the lkdump project was not known
to be dead for sure when makedumpfile was written, Ken'ichi
Ohmichi (the original author) was afraid that lkdump might make
conflicting extensions to their format and bump this field,
leading to two incompatible file formats with the same signature.
Note that makedumpfile was a project by NEC, not Fujitsu.

> Third, sometimes I see an additional header with signature
> "makedumpfile", and a few 64-bit integer fields, and then the
> KDUMP/DISKDUMP header occurs at offset 4096/0x1000. I wonder if you're
> familiar with that header, why or why not it occurs?

This is the flattened format, which is used if makedumpfile writes
to a file descriptor that cannot be seeked, e.g. a pipe. You
should rearrange the file with "makedumpfile -R" (or the
makedumpfile-R.pl script) to convert it to a standard diskdump
file.

> Finally, and this one is a long shot, do you know any good central
> document describing the diskdump / kdump format? From everything I can
> tell, it seems very much like a "living standard", one which lives in
> code more than documentation.

This is very much the case. For the "DISKDUMP" format, whatever is
in makedumpfile's diskdump_mod.h is authoritative. The "KDUMP    "
format is essentially dead.

HTH
Petr T

> But I would be glad to be wrong here, if
> there's a document which describes the format then I'd love to read it,
> and if not, maybe I could be part of creating it to help everyone out.
> 
> Thanks,
> Stephen

[-- Attachment #2: Digitální podpis OpenPGP --]
[-- Type: application/pgp-signature, Size: 228 bytes --]