* Diskdump / kdump format history @ 2023-01-13 0:25 Stephen Brennan 2023-01-14 21:24 ` Petr Tesařík 2023-01-14 21:29 ` Petr Tesařík 0 siblings, 2 replies; 4+ messages in thread From: Stephen Brennan @ 2023-01-13 0:25 UTC (permalink / raw) To: Petr Tesarik; +Cc: linux-debuggers Hi Petr, I wanted to ask you a bit about the history of the diskdump/kdump format, since it seems like somebody who has written an entire library for parsing it would have some interesting insights. I've CC'd the linux-debuggers mailing list here, which I believe would be its inaugural post! I think anybody interested in debugging kernel core dumps could use any answers you have. First, regarding naming. I guess I always assumed that "diskdump" referred to some older utility which performed the same function that makedumpfile now is commonly used for. Is that the case? Do you know the origin of the diskdump format? Do you know where to find the code for this older diskdump tool, if it existed? Second, I have some confusion regarding headers. I've seen headers for files with the content "DISKDUMP" and "KDUMP " -- and according to src/kdumpfile/diskdump.c, you have too. diskdump_probe() shows that you check for these two headers and set a different description depending on which is seen. Do you know why or where these two headers are set, and which one is chose in which situation? Third, sometimes I see an additional header with signature "makedumpfile", and a few 64-bit integer fields, and then the KDUMP/DISKDUMP header occurs at offset 4096/0x1000. I wonder if you're familiar with that header, why or why not it occurs? Finally, and this one is a long shot, do you know any good central document describing the diskdump / kdump format? From everything I can tell, it seems very much like a "living standard", one which lives in code more than documentation. But I would be glad to be wrong here, if there's a document which describes the format then I'd love to read it, and if not, maybe I could be part of creating it to help everyone out. Thanks, Stephen ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Diskdump / kdump format history 2023-01-13 0:25 Diskdump / kdump format history Stephen Brennan @ 2023-01-14 21:24 ` Petr Tesařík 2023-01-14 21:29 ` Petr Tesařík 1 sibling, 0 replies; 4+ messages in thread From: Petr Tesařík @ 2023-01-14 21:24 UTC (permalink / raw) To: Stephen Brennan; +Cc: linux-debuggers [-- Attachment #1: Type: text/plain, Size: 4113 bytes --] Hi Stephen, thanks for your trust. I've been around for some time, but I actually came late to the party. Below is my understanding of what happened. People who were really involved in all that may add a lot of corrections. On Thu, 12 Jan 2023 16:25:10 -0800 Stephen Brennan <stephen.s.brennan@oracle.com> wrote: >[...] > First, regarding naming. I guess I always assumed that "diskdump" > referred to some older utility which performed the same function that > makedumpfile now is commonly used for. Is that the case? Not quite. It did not perform the same function... Let me make a short historical excursion. When I joined SUSE, SLES9 was the latest release and it was based on kernel 2.6.5. There was no kdump at that time, but a team at SGI worked on LKCD (Linux Kernel Crash Dumps), which was an out-of-tree patch to the Linux Kernel, some user-space tools for setting it up and the lcrash analysis tool (which was, as a matter of fact, simply horrible). Most importantly, the dumping itself was performed by the crashed kernel, using a special polling mode that was implemented by a few device drivers. LKCD offered two options: netdump and diskdump. This is most likely where the term "diskdump" was first coined. > Do you know the origin of the diskdump format? Yes. Fujitsu developed their own Linux kernel dump project called lkdump. It was conceptually similar to LKCD, as it also ran within the crashed kernel, but it did not use the LKCD file format. IIUC they first modified LKCD for their Mission Critical Linux (or MCLX), but they soon discovered that there were some issues with the format itself and designed their own format with the signature "DISKDUMP". > Do you know where to find the code for > this older diskdump tool, if it existed? Yes. Have a look here: https://sourceforge.net/projects/lkdump/ > Second, I have some confusion regarding headers. I've seen headers for > files with the content "DISKDUMP" and "KDUMP " -- and according to > src/kdumpfile/diskdump.c, you have too. diskdump_probe() shows that you > check for these two headers and set a different description depending on > which is seen. Do you know why or where these two headers are set, and > which one is chose in which situation? "DISKDUMP" is the signature of the original Fujitsu lkdump files. "KDUMP " is the signature of the makedumpfile modification (that's the meaning of "_mod" in makedumpfile's diskdump_mod.h header file name). Yes, there was a header_version field, but that wasn't deemed sufficient. Since the lkdump project was not known to be dead for sure when makedumpfile was written, Ken'ichi Ohmichi (the original author) was afraid that lkdump might make conflicting extensions to their format and bump this field, leading to two incompatible file formats with the same signature. Note that makedumpfile was a project by NEC, not Fujitsu. > Third, sometimes I see an additional header with signature > "makedumpfile", and a few 64-bit integer fields, and then the > KDUMP/DISKDUMP header occurs at offset 4096/0x1000. I wonder if you're > familiar with that header, why or why not it occurs? This is the flattened format, which is used if makedumpfile writes to a file descriptor that cannot be seeked, e.g. a pipe. You should rearrange the file with "makedumpfile -R" (or the makedumpfile-R.pl script) to convert it to a standard diskdump file. > Finally, and this one is a long shot, do you know any good central > document describing the diskdump / kdump format? From everything I can > tell, it seems very much like a "living standard", one which lives in > code more than documentation. This is very much the case. For the "DISKDUMP" format, whatever is in makedumpfile's diskdump_mod.h is authoritative. The "KDUMP " format is essentially dead. HTH Petr T > But I would be glad to be wrong here, if > there's a document which describes the format then I'd love to read it, > and if not, maybe I could be part of creating it to help everyone out. > > Thanks, > Stephen [-- Attachment #2: Digitální podpis OpenPGP --] [-- Type: application/pgp-signature, Size: 228 bytes --] ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Diskdump / kdump format history 2023-01-13 0:25 Diskdump / kdump format history Stephen Brennan 2023-01-14 21:24 ` Petr Tesařík @ 2023-01-14 21:29 ` Petr Tesařík 2023-01-23 20:26 ` Stephen Brennan 1 sibling, 1 reply; 4+ messages in thread From: Petr Tesařík @ 2023-01-14 21:29 UTC (permalink / raw) To: Stephen Brennan; +Cc: linux-debuggers Hi Stephen, thanks for your trust. I've been around for some time, but I actually came late to the party. Below is my understanding of what happened. People who were really involved in all that may add a lot of corrections. On Thu, 12 Jan 2023 16:25:10 -0800 Stephen Brennan <stephen.s.brennan@oracle.com> wrote: >[...] > First, regarding naming. I guess I always assumed that "diskdump" > referred to some older utility which performed the same function that > makedumpfile now is commonly used for. Is that the case? Not quite. It did not perform the same function... Let me make a short historical excursion. When I joined SUSE, SLES9 was the latest release and it was based on kernel 2.6.5. There was no kdump at that time, but a team at SGI worked on LKCD (Linux Kernel Crash Dumps), which was an out-of-tree patch to the Linux Kernel, some user-space tools for setting it up and the lcrash analysis tool (which was, as a matter of fact, simply horrible). Most importantly, the dumping itself was performed by the crashed kernel, using a special polling mode that was implemented by a few device drivers. LKCD offered two options: netdump and diskdump. This is most likely where the term "diskdump" was first coined. > Do you know the origin of the diskdump format? Yes. Fujitsu developed their own Linux kernel dump project called lkdump. It was conceptually similar to LKCD, as it also ran within the crashed kernel, but it did not use the LKCD file format. IIUC they first modified LKCD for their Mission Critical Linux (or MCLX), but they soon discovered that there were some issues with the format itself and designed their own format with the signature "DISKDUMP". > Do you know where to find the code for > this older diskdump tool, if it existed? Yes. Have a look here: https://sourceforge.net/projects/lkdump/ > Second, I have some confusion regarding headers. I've seen headers for > files with the content "DISKDUMP" and "KDUMP " -- and according to > src/kdumpfile/diskdump.c, you have too. diskdump_probe() shows that you > check for these two headers and set a different description depending on > which is seen. Do you know why or where these two headers are set, and > which one is chose in which situation? "DISKDUMP" is the signature of the original Fujitsu lkdump files. "KDUMP " is the signature of the makedumpfile modification (that's the meaning of "_mod" in makedumpfile's diskdump_mod.h header file name). Yes, there was a header_version field, but that wasn't deemed sufficient. Since the lkdump project was not known to be dead for sure when makedumpfile was written, Ken'ichi Ohmichi (the original author) was afraid that lkdump might make conflicting extensions to their format and bump this field, leading to two incompatible file formats with the same signature. Note that makedumpfile was a project by NEC, not Fujitsu. > Third, sometimes I see an additional header with signature > "makedumpfile", and a few 64-bit integer fields, and then the > KDUMP/DISKDUMP header occurs at offset 4096/0x1000. I wonder if you're > familiar with that header, why or why not it occurs? This is the flattened format, which is used if makedumpfile writes to a file descriptor that cannot be seeked, e.g. a pipe. You should rearrange the file with "makedumpfile -R" (or the makedumpfile-R.pl script) to convert it to a standard diskdump file. > Finally, and this one is a long shot, do you know any good central > document describing the diskdump / kdump format? From everything I can > tell, it seems very much like a "living standard", one which lives in > code more than documentation. This is very much the case. For the "DISKDUMP" format, whatever is in makedumpfile's diskdump_mod.h is authoritative. The "KDUMP " format is essentially dead. HTH Petr T > But I would be glad to be wrong here, if > there's a document which describes the format then I'd love to read it, > and if not, maybe I could be part of creating it to help everyone out. > > Thanks, > Stephen ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Diskdump / kdump format history 2023-01-14 21:29 ` Petr Tesařík @ 2023-01-23 20:26 ` Stephen Brennan 0 siblings, 0 replies; 4+ messages in thread From: Stephen Brennan @ 2023-01-23 20:26 UTC (permalink / raw) To: Petr Tesařík; +Cc: linux-debuggers Hi Petr, Thank you for these insights, especially regarding the flattened format and the lkdump + diskdump origin stories. I don't have any particular follow-up questions, but I do intend to take a deeper look at the different header files used by different projects and try to build some documentation. I hope to make some content in a community wiki or something which could summarize the high level ideas, and provide links to all the different implementations, past and present. I hope to share any and all of that with you and the rest of the mailing list as I have updates. Thanks again, Stephen Petr Tesařík <petr@tesarici.cz> writes: > Hi Stephen, > > thanks for your trust. I've been around for some time, but I actually > came late to the party. Below is my understanding of what happened. > People who were really involved in all that may add a lot of > corrections. > > On Thu, 12 Jan 2023 16:25:10 -0800 > Stephen Brennan <stephen.s.brennan@oracle.com> wrote: > >>[...] >> First, regarding naming. I guess I always assumed that "diskdump" >> referred to some older utility which performed the same function that >> makedumpfile now is commonly used for. Is that the case? > > Not quite. It did not perform the same function... > > Let me make a short historical excursion. > > When I joined SUSE, SLES9 was the latest release and it was based > on kernel 2.6.5. There was no kdump at that time, but a team at > SGI worked on LKCD (Linux Kernel Crash Dumps), which was an > out-of-tree patch to the Linux Kernel, some user-space tools for > setting it up and the lcrash analysis tool (which was, as a matter > of fact, simply horrible). Most importantly, the dumping itself > was performed by the crashed kernel, using a special polling mode > that was implemented by a few device drivers. LKCD offered two > options: netdump and diskdump. This is most likely where the term > "diskdump" was first coined. > >> Do you know the origin of the diskdump format? > > Yes. Fujitsu developed their own Linux kernel dump project called > lkdump. It was conceptually similar to LKCD, as it also ran within > the crashed kernel, but it did not use the LKCD file format. IIUC > they first modified LKCD for their Mission Critical Linux (or > MCLX), but they soon discovered that there were some issues with > the format itself and designed their own format with the signature > "DISKDUMP". > >> Do you know where to find the code for >> this older diskdump tool, if it existed? > > Yes. Have a look here: > > https://sourceforge.net/projects/lkdump/ > >> Second, I have some confusion regarding headers. I've seen headers for >> files with the content "DISKDUMP" and "KDUMP " -- and according to >> src/kdumpfile/diskdump.c, you have too. diskdump_probe() shows that you >> check for these two headers and set a different description depending on >> which is seen. Do you know why or where these two headers are set, and >> which one is chose in which situation? > > "DISKDUMP" is the signature of the original Fujitsu lkdump > files. "KDUMP " is the signature of the makedumpfile modification > (that's the meaning of "_mod" in makedumpfile's diskdump_mod.h > header file name). Yes, there was a header_version field, but that > wasn't deemed sufficient. Since the lkdump project was not known > to be dead for sure when makedumpfile was written, Ken'ichi > Ohmichi (the original author) was afraid that lkdump might make > conflicting extensions to their format and bump this field, > leading to two incompatible file formats with the same signature. > Note that makedumpfile was a project by NEC, not Fujitsu. > >> Third, sometimes I see an additional header with signature >> "makedumpfile", and a few 64-bit integer fields, and then the >> KDUMP/DISKDUMP header occurs at offset 4096/0x1000. I wonder if you're >> familiar with that header, why or why not it occurs? > > This is the flattened format, which is used if makedumpfile writes > to a file descriptor that cannot be seeked, e.g. a pipe. You > should rearrange the file with "makedumpfile -R" (or the > makedumpfile-R.pl script) to convert it to a standard diskdump > file. > >> Finally, and this one is a long shot, do you know any good central >> document describing the diskdump / kdump format? From everything I can >> tell, it seems very much like a "living standard", one which lives in >> code more than documentation. > > This is very much the case. For the "DISKDUMP" format, whatever is > in makedumpfile's diskdump_mod.h is authoritative. The "KDUMP " > format is essentially dead. > > HTH > Petr T > >> But I would be glad to be wrong here, if >> there's a document which describes the format then I'd love to read it, >> and if not, maybe I could be part of creating it to help everyone out. >> >> Thanks, >> Stephen ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2023-01-23 20:26 UTC | newest] Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2023-01-13 0:25 Diskdump / kdump format history Stephen Brennan 2023-01-14 21:24 ` Petr Tesařík 2023-01-14 21:29 ` Petr Tesařík 2023-01-23 20:26 ` Stephen Brennan
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).