* [BUG] vmcore-dmesg cant' read dmesg log from /proc/vmcore if log_buf is reallocated due to large number of CPUs @ 2018-10-24 12:52 Lomovtsev, Vadim 2018-10-24 21:30 ` Bhupesh Sharma 0 siblings, 1 reply; 9+ messages in thread From: Lomovtsev, Vadim @ 2018-10-24 12:52 UTC (permalink / raw) To: kexec; +Cc: Lomovtsev, Vadim Hi all, Following issue has been found for vmcore-dmesg app with latest release (94159bc3c264fa26395e56302072276a139d18af 2.0.18-rc1) of kexec-tools at CentOS 7.5 distro: While having systems with large number of CPUs (e.g. Cavium ThunderX2 has 224) the log_buf gets reallocated by memblock_virt_alloc() at the setup_log_buf routine (https://elixir.bootlin.com/linux/v4.16.18/source/kernel/printk/printk.c#L1108). Then while dumping vmcore the vmcore-dmesg can't find dmesg log at /proc/vmcore file and exits with following message: Failed to read log text of size 0 bytes: Bad address However it (vmcore-dmesg app) reads properly the log_buf symbol, it's address and eventually it's value from /proc/vmcore but fails to find dmesg data then. In the same time the makedumpfile is able to find and extract dmesg buffer from /proc/vmcore. The makedumpfile comes with kexec-tools-2.0.15-13.el7_5.2.aarch64 package. The issue is not reproduced for systems with small number of CPUs and log_buf not reallocated to memblock section. WBR, Vadim _______________________________________________ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [BUG] vmcore-dmesg cant' read dmesg log from /proc/vmcore if log_buf is reallocated due to large number of CPUs 2018-10-24 12:52 [BUG] vmcore-dmesg cant' read dmesg log from /proc/vmcore if log_buf is reallocated due to large number of CPUs Lomovtsev, Vadim @ 2018-10-24 21:30 ` Bhupesh Sharma 2018-10-25 10:40 ` Vadim Lomovtsev 0 siblings, 1 reply; 9+ messages in thread From: Bhupesh Sharma @ 2018-10-24 21:30 UTC (permalink / raw) To: Vadim.Lomovtsev; +Cc: kexec mailing list Hello Vadim, On Wed, Oct 24, 2018 at 6:23 PM Lomovtsev, Vadim <Vadim.Lomovtsev@cavium.com> wrote: > > Hi all, > > Following issue has been found for vmcore-dmesg app with latest release (94159bc3c264fa26395e56302072276a139d18af 2.0.18-rc1) of kexec-tools at CentOS 7.5 distro: > > While having systems with large number of CPUs (e.g. Cavium ThunderX2 has 224) the log_buf gets reallocated by memblock_virt_alloc() at the setup_log_buf routine (https://elixir.bootlin.com/linux/v4.16.18/source/kernel/printk/printk.c#L1108). > > Then while dumping vmcore the vmcore-dmesg can't find dmesg log at /proc/vmcore file and exits with following message: > Failed to read log text of size 0 bytes: Bad address > > However it (vmcore-dmesg app) reads properly the log_buf symbol, it's address and eventually it's value from /proc/vmcore but fails to find dmesg data then. > > In the same time the makedumpfile is able to find and extract dmesg buffer from /proc/vmcore. > The makedumpfile comes with kexec-tools-2.0.15-13.el7_5.2.aarch64 package. > > The issue is not reproduced for systems with small number of CPUs and log_buf not reallocated to memblock section. Seems like you are hitting a known issue we saw on qualcomm amberwing platforms as well. I have sent a patch-series titled 'kexec-tools/arm64: Add support to read PHYS_OFFSET from vmcoreinfo inside '/proc/kcore' to this list just a few minutes back. I have Cc'ed you to the patchset as I think it might fix the issue for you. Kindly try the patchset on your platform (cavium?) and let me know if this fixes the issue for you. Thanks, Bhupesh _______________________________________________ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [BUG] vmcore-dmesg cant' read dmesg log from /proc/vmcore if log_buf is reallocated due to large number of CPUs 2018-10-24 21:30 ` Bhupesh Sharma @ 2018-10-25 10:40 ` Vadim Lomovtsev 2018-10-26 6:55 ` Bhupesh Sharma 0 siblings, 1 reply; 9+ messages in thread From: Vadim Lomovtsev @ 2018-10-25 10:40 UTC (permalink / raw) To: Bhupesh Sharma; +Cc: Lomovtsev, Vadim, kexec mailing list Hello Bhupesh, On Thu, Oct 25, 2018 at 03:00:08AM +0530, Bhupesh Sharma wrote: > External Email > > Hello Vadim, > > On Wed, Oct 24, 2018 at 6:23 PM Lomovtsev, Vadim > <Vadim.Lomovtsev@cavium.com> wrote: > > > > Hi all, > > > > Following issue has been found for vmcore-dmesg app with latest release (94159bc3c264fa26395e56302072276a139d18af 2.0.18-rc1) of kexec-tools at CentOS 7.5 distro: > > > > While having systems with large number of CPUs (e.g. Cavium ThunderX2 has 224) the log_buf gets reallocated by memblock_virt_alloc() at the setup_log_buf routine (https://elixir.bootlin.com/linux/v4.16.18/source/kernel/printk/printk.c#L1108). > > > > Then while dumping vmcore the vmcore-dmesg can't find dmesg log at /proc/vmcore file and exits with following message: > > Failed to read log text of size 0 bytes: Bad address > > > > However it (vmcore-dmesg app) reads properly the log_buf symbol, it's address and eventually it's value from /proc/vmcore but fails to find dmesg data then. > > > > In the same time the makedumpfile is able to find and extract dmesg buffer from /proc/vmcore. > > The makedumpfile comes with kexec-tools-2.0.15-13.el7_5.2.aarch64 package. > > > > The issue is not reproduced for systems with small number of CPUs and log_buf not reallocated to memblock section. > > Seems like you are hitting a known issue we saw on qualcomm amberwing > platforms as well. > I have sent a patch-series titled 'kexec-tools/arm64: Add support to > read PHYS_OFFSET from vmcoreinfo inside '/proc/kcore' to this list > just a few minutes back. > > I have Cc'ed you to the patchset as I think it might fix the issue for > you. Got them, thank you. > Kindly try the patchset on your platform (cavium?) and let me > know if this fixes the issue for you. Sure, I'd like to check them at my side, but.. I fall into merge conflicts while trying to apply them onto https://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git/ master, kexec-tools 2.0.18-rc1 94159bc3c264fa26395e56302072276a139d18af Are there any specific branch/revision for them to be applied ? (or it might be my mail server issues with formatting emails). WBR, Vadim > > Thanks, > Bhupesh _______________________________________________ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [BUG] vmcore-dmesg cant' read dmesg log from /proc/vmcore if log_buf is reallocated due to large number of CPUs 2018-10-25 10:40 ` Vadim Lomovtsev @ 2018-10-26 6:55 ` Bhupesh Sharma 2018-10-26 10:11 ` Vadim Lomovtsev 0 siblings, 1 reply; 9+ messages in thread From: Bhupesh Sharma @ 2018-10-26 6:55 UTC (permalink / raw) To: Vadim.Lomovtsev; +Cc: Vadim.Lomovtsev, kexec mailing list ease p before seiHi Vadim, On Thu, Oct 25, 2018 at 4:10 PM Vadim Lomovtsev <Vadim.Lomovtsev@caviumnetworks.com> wrote: > > Hello Bhupesh, > > On Thu, Oct 25, 2018 at 03:00:08AM +0530, Bhupesh Sharma wrote: > > External Email > > > > Hello Vadim, > > > > On Wed, Oct 24, 2018 at 6:23 PM Lomovtsev, Vadim > > <Vadim.Lomovtsev@cavium.com> wrote: > > > > > > Hi all, > > > > > > Following issue has been found for vmcore-dmesg app with latest release (94159bc3c264fa26395e56302072276a139d18af 2.0.18-rc1) of kexec-tools at CentOS 7.5 distro: > > > > > > While having systems with large number of CPUs (e.g. Cavium ThunderX2 has 224) the log_buf gets reallocated by memblock_virt_alloc() at the setup_log_buf routine (https://elixir.bootlin.com/linux/v4.16.18/source/kernel/printk/printk.c#L1108). > > > > > > Then while dumping vmcore the vmcore-dmesg can't find dmesg log at /proc/vmcore file and exits with following message: > > > Failed to read log text of size 0 bytes: Bad address > > > > > > However it (vmcore-dmesg app) reads properly the log_buf symbol, it's address and eventually it's value from /proc/vmcore but fails to find dmesg data then. > > > > > > In the same time the makedumpfile is able to find and extract dmesg buffer from /proc/vmcore. > > > The makedumpfile comes with kexec-tools-2.0.15-13.el7_5.2.aarch64 package. > > > > > > The issue is not reproduced for systems with small number of CPUs and log_buf not reallocated to memblock section. > > > > Seems like you are hitting a known issue we saw on qualcomm amberwing > > platforms as well. > > I have sent a patch-series titled 'kexec-tools/arm64: Add support to > > read PHYS_OFFSET from vmcoreinfo inside '/proc/kcore' to this list > > just a few minutes back. > > > > I have Cc'ed you to the patchset as I think it might fix the issue for > > you. > > Got them, thank you. > > > Kindly try the patchset on your platform (cavium?) and let me > > know if this fixes the issue for you. > > Sure, I'd like to check them at my side, but.. > I fall into merge conflicts while trying to apply them onto > https://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git/ > master, kexec-tools 2.0.18-rc1 94159bc3c264fa26395e56302072276a139d18af Hmm.. that's strange as I rebased them on kexec-tools 2.0.18-rc1 (94159bc3c264fa26395e56302072276a139d18af) before sending out the patchset. > Are there any specific branch/revision for them to be applied ? > (or it might be my mail server issues with formatting emails). > Can you please try picking them up from my public github tree instead? Here you can find the same: https://github.com/bhupesh-sharma/kexec-tools/tree/read-phys-offset-from-kcore-upstream-v1 Please pick the top 2 commit from here. Thanks, Bhupesh > > > > > Thanks, > > Bhupesh _______________________________________________ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [BUG] vmcore-dmesg cant' read dmesg log from /proc/vmcore if log_buf is reallocated due to large number of CPUs 2018-10-26 6:55 ` Bhupesh Sharma @ 2018-10-26 10:11 ` Vadim Lomovtsev 2018-10-26 10:19 ` Bhupesh Sharma 0 siblings, 1 reply; 9+ messages in thread From: Vadim Lomovtsev @ 2018-10-26 10:11 UTC (permalink / raw) To: Bhupesh Sharma; +Cc: Lomovtsev, Vadim, kexec mailing list Hi Bhupesh, On Fri, Oct 26, 2018 at 12:25:17PM +0530, Bhupesh Sharma wrote: > > ease p > before seiHi Vadim, > > On Thu, Oct 25, 2018 at 4:10 PM Vadim Lomovtsev > <Vadim.Lomovtsev@caviumnetworks.com> wrote: > > > > Hello Bhupesh, > > > > On Thu, Oct 25, 2018 at 03:00:08AM +0530, Bhupesh Sharma wrote: > > > External Email > > > > > > Hello Vadim, > > > > > > On Wed, Oct 24, 2018 at 6:23 PM Lomovtsev, Vadim > > > <Vadim.Lomovtsev@cavium.com> wrote: > > > > > > > > Hi all, > > > > > > > > Following issue has been found for vmcore-dmesg app with latest release (94159bc3c264fa26395e56302072276a139d18af 2.0.18-rc1) of kexec-tools at CentOS 7.5 distro: > > > > > > > > While having systems with large number of CPUs (e.g. Cavium ThunderX2 has 224) the log_buf gets reallocated by memblock_virt_alloc() at the setup_log_buf routine (https://elixir.bootlin.com/linux/v4.16.18/source/kernel/printk/printk.c#L1108). > > > > > > > > Then while dumping vmcore the vmcore-dmesg can't find dmesg log at /proc/vmcore file and exits with following message: > > > > Failed to read log text of size 0 bytes: Bad address > > > > > > > > However it (vmcore-dmesg app) reads properly the log_buf symbol, it's address and eventually it's value from /proc/vmcore but fails to find dmesg data then. > > > > > > > > In the same time the makedumpfile is able to find and extract dmesg buffer from /proc/vmcore. > > > > The makedumpfile comes with kexec-tools-2.0.15-13.el7_5.2.aarch64 package. > > > > > > > > The issue is not reproduced for systems with small number of CPUs and log_buf not reallocated to memblock section. > > > > > > Seems like you are hitting a known issue we saw on qualcomm amberwing > > > platforms as well. > > > I have sent a patch-series titled 'kexec-tools/arm64: Add support to > > > read PHYS_OFFSET from vmcoreinfo inside '/proc/kcore' to this list > > > just a few minutes back. > > > > > > I have Cc'ed you to the patchset as I think it might fix the issue for > > > you. > > > > Got them, thank you. > > > > > Kindly try the patchset on your platform (cavium?) and let me > > > know if this fixes the issue for you. > > > > Sure, I'd like to check them at my side, but.. > > I fall into merge conflicts while trying to apply them onto > > https://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git/ > > master, kexec-tools 2.0.18-rc1 94159bc3c264fa26395e56302072276a139d18af > > Hmm.. that's strange as I rebased them on kexec-tools 2.0.18-rc1 > (94159bc3c264fa26395e56302072276a139d18af) > before sending out the patchset. > > > Are there any specific branch/revision for them to be applied ? > > (or it might be my mail server issues with formatting emails). > > > > Can you please try picking them up from my public github tree instead? > Here you can find the same: > https://github.com/bhupesh-sharma/kexec-tools/tree/read-phys-offset-from-kcore-upstream-v1 > > Please pick the top 2 commit from here. Applied them onto commit '94159bc kexec-tools 2.0.18-rc1'. Still having following error while saving dmesg by vmcore-dmesg: kdump: saving vmcore-dmesg.txt Failed to read log text of size 0 bytes: Bad address kdump: saving vmcore-dmesg.txt failed So far tried kernels 4.14.78, 4.16.18. WBR, Vadim > > Thanks, > Bhupesh > > > > > > > > > Thanks, > > > Bhupesh _______________________________________________ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [BUG] vmcore-dmesg cant' read dmesg log from /proc/vmcore if log_buf is reallocated due to large number of CPUs 2018-10-26 10:11 ` Vadim Lomovtsev @ 2018-10-26 10:19 ` Bhupesh Sharma 2018-10-26 13:18 ` Vadim Lomovtsev 0 siblings, 1 reply; 9+ messages in thread From: Bhupesh Sharma @ 2018-10-26 10:19 UTC (permalink / raw) To: Vadim.Lomovtsev; +Cc: Vadim.Lomovtsev, kexec mailing list Hi Vadim, On Fri, Oct 26, 2018 at 3:41 PM Vadim Lomovtsev <Vadim.Lomovtsev@caviumnetworks.com> wrote: > > Hi Bhupesh, > > On Fri, Oct 26, 2018 at 12:25:17PM +0530, Bhupesh Sharma wrote: > > > > ease p > > before seiHi Vadim, > > > > On Thu, Oct 25, 2018 at 4:10 PM Vadim Lomovtsev > > <Vadim.Lomovtsev@caviumnetworks.com> wrote: > > > > > > Hello Bhupesh, > > > > > > On Thu, Oct 25, 2018 at 03:00:08AM +0530, Bhupesh Sharma wrote: > > > > External Email > > > > > > > > Hello Vadim, > > > > > > > > On Wed, Oct 24, 2018 at 6:23 PM Lomovtsev, Vadim > > > > <Vadim.Lomovtsev@cavium.com> wrote: > > > > > > > > > > Hi all, > > > > > > > > > > Following issue has been found for vmcore-dmesg app with latest release (94159bc3c264fa26395e56302072276a139d18af 2.0.18-rc1) of kexec-tools at CentOS 7.5 distro: > > > > > > > > > > While having systems with large number of CPUs (e.g. Cavium ThunderX2 has 224) the log_buf gets reallocated by memblock_virt_alloc() at the setup_log_buf routine (https://elixir.bootlin.com/linux/v4.16.18/source/kernel/printk/printk.c#L1108). > > > > > > > > > > Then while dumping vmcore the vmcore-dmesg can't find dmesg log at /proc/vmcore file and exits with following message: > > > > > Failed to read log text of size 0 bytes: Bad address > > > > > > > > > > However it (vmcore-dmesg app) reads properly the log_buf symbol, it's address and eventually it's value from /proc/vmcore but fails to find dmesg data then. > > > > > > > > > > In the same time the makedumpfile is able to find and extract dmesg buffer from /proc/vmcore. > > > > > The makedumpfile comes with kexec-tools-2.0.15-13.el7_5.2.aarch64 package. > > > > > > > > > > The issue is not reproduced for systems with small number of CPUs and log_buf not reallocated to memblock section. > > > > > > > > Seems like you are hitting a known issue we saw on qualcomm amberwing > > > > platforms as well. > > > > I have sent a patch-series titled 'kexec-tools/arm64: Add support to > > > > read PHYS_OFFSET from vmcoreinfo inside '/proc/kcore' to this list > > > > just a few minutes back. > > > > > > > > I have Cc'ed you to the patchset as I think it might fix the issue for > > > > you. > > > > > > Got them, thank you. > > > > > > > Kindly try the patchset on your platform (cavium?) and let me > > > > know if this fixes the issue for you. > > > > > > Sure, I'd like to check them at my side, but.. > > > I fall into merge conflicts while trying to apply them onto > > > https://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git/ > > > master, kexec-tools 2.0.18-rc1 94159bc3c264fa26395e56302072276a139d18af > > > > Hmm.. that's strange as I rebased them on kexec-tools 2.0.18-rc1 > > (94159bc3c264fa26395e56302072276a139d18af) > > before sending out the patchset. > > > > > Are there any specific branch/revision for them to be applied ? > > > (or it might be my mail server issues with formatting emails). > > > > > > > Can you please try picking them up from my public github tree instead? > > Here you can find the same: > > https://github.com/bhupesh-sharma/kexec-tools/tree/read-phys-offset-from-kcore-upstream-v1 > > > > Please pick the top 2 commit from here. > > Applied them onto commit '94159bc kexec-tools 2.0.18-rc1'. > > Still having following error while saving dmesg by vmcore-dmesg: > > kdump: saving vmcore-dmesg.txt > Failed to read log text of size 0 bytes: Bad address > kdump: saving vmcore-dmesg.txt failed > > So far tried kernels 4.14.78, 4.16.18. You would need kernel 4.19-rc5 or above as the same exposes VMCOREINFO as '/proc/kcore'. If you are having issues while switching to newer kernel, please share the output(s) of following on your platform: # kexec -p /boot/vmlinuz-`uname -r` --initrd=/boot/initramfs-`uname -r`.img --reuse-cmdline -d and, # readelf -l vmcore and, # cat /proc/iomem And then I can suggest a hack, which you can try and test on your platform and then we can take it forward from there. Thanks, Bhupesh > > > > Thanks, > > Bhupesh > > > > > > > > > > > > > Thanks, > > > > Bhupesh _______________________________________________ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [BUG] vmcore-dmesg cant' read dmesg log from /proc/vmcore if log_buf is reallocated due to large number of CPUs 2018-10-26 10:19 ` Bhupesh Sharma @ 2018-10-26 13:18 ` Vadim Lomovtsev 2018-10-26 23:11 ` Bhupesh Sharma 0 siblings, 1 reply; 9+ messages in thread From: Vadim Lomovtsev @ 2018-10-26 13:18 UTC (permalink / raw) To: Bhupesh Sharma; +Cc: Lomovtsev, Vadim, kexec mailing list [-- Attachment #1: Type: text/plain, Size: 4895 bytes --] Hi Bhupesh, On Fri, Oct 26, 2018 at 03:49:11PM +0530, Bhupesh Sharma wrote: > > Hi Vadim, > On Fri, Oct 26, 2018 at 3:41 PM Vadim Lomovtsev > <Vadim.Lomovtsev@caviumnetworks.com> wrote: > > > > Hi Bhupesh, > > > > On Fri, Oct 26, 2018 at 12:25:17PM +0530, Bhupesh Sharma wrote: > > > > > > ease p > > > before seiHi Vadim, > > > > > > On Thu, Oct 25, 2018 at 4:10 PM Vadim Lomovtsev > > > <Vadim.Lomovtsev@caviumnetworks.com> wrote: > > > > > > > > Hello Bhupesh, > > > > > > > > On Thu, Oct 25, 2018 at 03:00:08AM +0530, Bhupesh Sharma wrote: > > > > > External Email > > > > > > > > > > Hello Vadim, > > > > > > > > > > On Wed, Oct 24, 2018 at 6:23 PM Lomovtsev, Vadim > > > > > <Vadim.Lomovtsev@cavium.com> wrote: > > > > > > > > > > > > Hi all, > > > > > > > > > > > > Following issue has been found for vmcore-dmesg app with latest release (94159bc3c264fa26395e56302072276a139d18af 2.0.18-rc1) of kexec-tools at CentOS 7.5 distro: > > > > > > > > > > > > While having systems with large number of CPUs (e.g. Cavium ThunderX2 has 224) the log_buf gets reallocated by memblock_virt_alloc() at the setup_log_buf routine (https://elixir.bootlin.com/linux/v4.16.18/source/kernel/printk/printk.c#L1108). > > > > > > > > > > > > Then while dumping vmcore the vmcore-dmesg can't find dmesg log at /proc/vmcore file and exits with following message: > > > > > > Failed to read log text of size 0 bytes: Bad address > > > > > > > > > > > > However it (vmcore-dmesg app) reads properly the log_buf symbol, it's address and eventually it's value from /proc/vmcore but fails to find dmesg data then. > > > > > > > > > > > > In the same time the makedumpfile is able to find and extract dmesg buffer from /proc/vmcore. > > > > > > The makedumpfile comes with kexec-tools-2.0.15-13.el7_5.2.aarch64 package. > > > > > > > > > > > > The issue is not reproduced for systems with small number of CPUs and log_buf not reallocated to memblock section. > > > > > > > > > > Seems like you are hitting a known issue we saw on qualcomm amberwing > > > > > platforms as well. > > > > > I have sent a patch-series titled 'kexec-tools/arm64: Add support to > > > > > read PHYS_OFFSET from vmcoreinfo inside '/proc/kcore' to this list > > > > > just a few minutes back. > > > > > > > > > > I have Cc'ed you to the patchset as I think it might fix the issue for > > > > > you. > > > > > > > > Got them, thank you. > > > > > > > > > Kindly try the patchset on your platform (cavium?) and let me > > > > > know if this fixes the issue for you. > > > > > > > > Sure, I'd like to check them at my side, but.. > > > > I fall into merge conflicts while trying to apply them onto > > > > https://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git/ > > > > master, kexec-tools 2.0.18-rc1 94159bc3c264fa26395e56302072276a139d18af > > > > > > Hmm.. that's strange as I rebased them on kexec-tools 2.0.18-rc1 > > > (94159bc3c264fa26395e56302072276a139d18af) > > > before sending out the patchset. > > > > > > > Are there any specific branch/revision for them to be applied ? > > > > (or it might be my mail server issues with formatting emails). > > > > > > > > > > Can you please try picking them up from my public github tree instead? > > > Here you can find the same: > > > https://github.com/bhupesh-sharma/kexec-tools/tree/read-phys-offset-from-kcore-upstream-v1 > > > > > > Please pick the top 2 commit from here. > > > > Applied them onto commit '94159bc kexec-tools 2.0.18-rc1'. > > > > Still having following error while saving dmesg by vmcore-dmesg: > > > > kdump: saving vmcore-dmesg.txt > > Failed to read log text of size 0 bytes: Bad address > > kdump: saving vmcore-dmesg.txt failed > > > > So far tried kernels 4.14.78, 4.16.18. > > You would need kernel 4.19-rc5 or above as the same exposes VMCOREINFO > as '/proc/kcore'. So far with 4.19-rc6 (and updated kexec, vmcore-dmesg but having kdump scripts from CentOS) the crashkernel can't found sysroot and thus it can't dump anything, so it timeouts and reboot system. > If you are having issues while switching to newer kernel, please share > the output(s) of following on your platform: > > # kexec -p /boot/vmlinuz-`uname -r` --initrd=/boot/initramfs-`uname > -r`.img --reuse-cmdline -d > attached as kexec-start.log.xz > and, > > # readelf -l vmcore [root@2sgbt-53 vlomovts]# readelf -l vmcore readelf: vmcore: Error: No such file [root@2sgbt-53 vlomovts]# uname -r 4.19.0-rc6+ > > and, > > # cat /proc/iomem attached as cat-proc-iomem.log.xz WBR, Vadim > > And then I can suggest a hack, which you can try and test on your > platform and then we can take it forward from there. > > Thanks, > Bhupesh > > > > > > > Thanks, > > > Bhupesh > > > > > > > > > > > > > > > > > Thanks, > > > > > Bhupesh [-- Attachment #2: cat-proc-iomem.log.xz --] [-- Type: application/x-xz, Size: 2172 bytes --] [-- Attachment #3: kexec-start.log.xz --] [-- Type: application/x-xz, Size: 2784 bytes --] [-- Attachment #4: Type: text/plain, Size: 143 bytes --] _______________________________________________ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [BUG] vmcore-dmesg cant' read dmesg log from /proc/vmcore if log_buf is reallocated due to large number of CPUs 2018-10-26 13:18 ` Vadim Lomovtsev @ 2018-10-26 23:11 ` Bhupesh Sharma 2018-10-29 11:21 ` Vadim Lomovtsev 0 siblings, 1 reply; 9+ messages in thread From: Bhupesh Sharma @ 2018-10-26 23:11 UTC (permalink / raw) To: Vadim.Lomovtsev; +Cc: Vadim.Lomovtsev, kexec mailing list Hi Vadim, On Fri, Oct 26, 2018 at 6:49 PM Vadim Lomovtsev <Vadim.Lomovtsev@caviumnetworks.com> wrote: > > Hi Bhupesh, > > On Fri, Oct 26, 2018 at 03:49:11PM +0530, Bhupesh Sharma wrote: > > > > Hi Vadim, > > On Fri, Oct 26, 2018 at 3:41 PM Vadim Lomovtsev > > <Vadim.Lomovtsev@caviumnetworks.com> wrote: > > > > > > Hi Bhupesh, > > > > > > On Fri, Oct 26, 2018 at 12:25:17PM +0530, Bhupesh Sharma wrote: > > > > > > > > ease p > > > > before seiHi Vadim, > > > > > > > > On Thu, Oct 25, 2018 at 4:10 PM Vadim Lomovtsev > > > > <Vadim.Lomovtsev@caviumnetworks.com> wrote: > > > > > > > > > > Hello Bhupesh, > > > > > > > > > > On Thu, Oct 25, 2018 at 03:00:08AM +0530, Bhupesh Sharma wrote: > > > > > > External Email > > > > > > > > > > > > Hello Vadim, > > > > > > > > > > > > On Wed, Oct 24, 2018 at 6:23 PM Lomovtsev, Vadim > > > > > > <Vadim.Lomovtsev@cavium.com> wrote: > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > Following issue has been found for vmcore-dmesg app with latest release (94159bc3c264fa26395e56302072276a139d18af 2.0.18-rc1) of kexec-tools at CentOS 7.5 distro: > > > > > > > > > > > > > > While having systems with large number of CPUs (e.g. Cavium ThunderX2 has 224) the log_buf gets reallocated by memblock_virt_alloc() at the setup_log_buf routine (https://elixir.bootlin.com/linux/v4.16.18/source/kernel/printk/printk.c#L1108). > > > > > > > > > > > > > > Then while dumping vmcore the vmcore-dmesg can't find dmesg log at /proc/vmcore file and exits with following message: > > > > > > > Failed to read log text of size 0 bytes: Bad address > > > > > > > > > > > > > > However it (vmcore-dmesg app) reads properly the log_buf symbol, it's address and eventually it's value from /proc/vmcore but fails to find dmesg data then. > > > > > > > > > > > > > > In the same time the makedumpfile is able to find and extract dmesg buffer from /proc/vmcore. > > > > > > > The makedumpfile comes with kexec-tools-2.0.15-13.el7_5.2.aarch64 package. > > > > > > > > > > > > > > The issue is not reproduced for systems with small number of CPUs and log_buf not reallocated to memblock section. > > > > > > > > > > > > Seems like you are hitting a known issue we saw on qualcomm amberwing > > > > > > platforms as well. > > > > > > I have sent a patch-series titled 'kexec-tools/arm64: Add support to > > > > > > read PHYS_OFFSET from vmcoreinfo inside '/proc/kcore' to this list > > > > > > just a few minutes back. > > > > > > > > > > > > I have Cc'ed you to the patchset as I think it might fix the issue for > > > > > > you. > > > > > > > > > > Got them, thank you. > > > > > > > > > > > Kindly try the patchset on your platform (cavium?) and let me > > > > > > know if this fixes the issue for you. > > > > > > > > > > Sure, I'd like to check them at my side, but.. > > > > > I fall into merge conflicts while trying to apply them onto > > > > > https://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git/ > > > > > master, kexec-tools 2.0.18-rc1 94159bc3c264fa26395e56302072276a139d18af > > > > > > > > Hmm.. that's strange as I rebased them on kexec-tools 2.0.18-rc1 > > > > (94159bc3c264fa26395e56302072276a139d18af) > > > > before sending out the patchset. > > > > > > > > > Are there any specific branch/revision for them to be applied ? > > > > > (or it might be my mail server issues with formatting emails). > > > > > > > > > > > > > Can you please try picking them up from my public github tree instead? > > > > Here you can find the same: > > > > https://github.com/bhupesh-sharma/kexec-tools/tree/read-phys-offset-from-kcore-upstream-v1 > > > > > > > > Please pick the top 2 commit from here. > > > > > > Applied them onto commit '94159bc kexec-tools 2.0.18-rc1'. > > > > > > Still having following error while saving dmesg by vmcore-dmesg: > > > > > > kdump: saving vmcore-dmesg.txt > > > Failed to read log text of size 0 bytes: Bad address > > > kdump: saving vmcore-dmesg.txt failed > > > > > > So far tried kernels 4.14.78, 4.16.18. > > > > You would need kernel 4.19-rc5 or above as the same exposes VMCOREINFO > > as '/proc/kcore'. > > So far with 4.19-rc6 (and updated kexec, vmcore-dmesg but having kdump scripts from CentOS) > the crashkernel can't found sysroot and thus it can't dump anything, so it timeouts and reboot system. > > > If you are having issues while switching to newer kernel, please share > > the output(s) of following on your platform: > > > > # kexec -p /boot/vmlinuz-`uname -r` --initrd=/boot/initramfs-`uname > > -r`.img --reuse-cmdline -d > > > > attached as kexec-start.log.xz > > > and, > > > > # readelf -l vmcore > > [root@2sgbt-53 vlomovts]# readelf -l vmcore > readelf: vmcore: Error: No such file > [root@2sgbt-53 vlomovts]# uname -r > 4.19.0-rc6+ > > > > > and, > > > > # cat /proc/iomem > > attached as cat-proc-iomem.log.xz Just to confirm: these logs are after your apply my kexec-tools patches, right? It looks likely that we are seeing differences in the value of 'phys_offset' on your platforms: From, '/proc/iomem', we can see that phys_offset is 0x01400000: 01400000-ffedffff : System RAM while the 'kexec -p -d' logs indicate that it is 0: image_arm64_load: phys_offset: 0000000000000000 This tells me that the phys_offset value is not correctly calculated in kexec-tools which should be fixed after my patches. BTW , by '# readelf -l vmcore', I meant the 'vmcore' dump file you have obtained via 'kexec'. It might be that you are saving it on some different location (something /var/crash?). Can you please try sharing the output of the same as well? Regards, Bhupesh > > > > And then I can suggest a hack, which you can try and test on your > > platform and then we can take it forward from there. > > > > Thanks, > > Bhupesh > > > > > > > > > > Thanks, > > > > Bhupesh > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > Bhupesh _______________________________________________ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [BUG] vmcore-dmesg cant' read dmesg log from /proc/vmcore if log_buf is reallocated due to large number of CPUs 2018-10-26 23:11 ` Bhupesh Sharma @ 2018-10-29 11:21 ` Vadim Lomovtsev 0 siblings, 0 replies; 9+ messages in thread From: Vadim Lomovtsev @ 2018-10-29 11:21 UTC (permalink / raw) To: Bhupesh Sharma; +Cc: Lomovtsev, Vadim, kexec mailing list Hi Bhupesh, On Sat, Oct 27, 2018 at 04:41:55AM +0530, Bhupesh Sharma wrote: > > Hi Vadim, > > On Fri, Oct 26, 2018 at 6:49 PM Vadim Lomovtsev > <Vadim.Lomovtsev@caviumnetworks.com> wrote: > > > > Hi Bhupesh, > > > > On Fri, Oct 26, 2018 at 03:49:11PM +0530, Bhupesh Sharma wrote: > > > > > > Hi Vadim, > > > On Fri, Oct 26, 2018 at 3:41 PM Vadim Lomovtsev > > > <Vadim.Lomovtsev@caviumnetworks.com> wrote: > > > > > > > > Hi Bhupesh, > > > > > > > > On Fri, Oct 26, 2018 at 12:25:17PM +0530, Bhupesh Sharma wrote: > > > > > > > > > > ease p > > > > > before seiHi Vadim, > > > > > > > > > > On Thu, Oct 25, 2018 at 4:10 PM Vadim Lomovtsev > > > > > <Vadim.Lomovtsev@caviumnetworks.com> wrote: > > > > > > > > > > > > Hello Bhupesh, > > > > > > > > > > > > On Thu, Oct 25, 2018 at 03:00:08AM +0530, Bhupesh Sharma wrote: > > > > > > > External Email > > > > > > > > > > > > > > Hello Vadim, > > > > > > > > > > > > > > On Wed, Oct 24, 2018 at 6:23 PM Lomovtsev, Vadim > > > > > > > <Vadim.Lomovtsev@cavium.com> wrote: > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > Following issue has been found for vmcore-dmesg app with latest release (94159bc3c264fa26395e56302072276a139d18af 2.0.18-rc1) of kexec-tools at CentOS 7.5 distro: > > > > > > > > > > > > > > > > While having systems with large number of CPUs (e.g. Cavium ThunderX2 has 224) the log_buf gets reallocated by memblock_virt_alloc() at the setup_log_buf routine (https://elixir.bootlin.com/linux/v4.16.18/source/kernel/printk/printk.c#L1108). > > > > > > > > > > > > > > > > Then while dumping vmcore the vmcore-dmesg can't find dmesg log at /proc/vmcore file and exits with following message: > > > > > > > > Failed to read log text of size 0 bytes: Bad address > > > > > > > > > > > > > > > > However it (vmcore-dmesg app) reads properly the log_buf symbol, it's address and eventually it's value from /proc/vmcore but fails to find dmesg data then. > > > > > > > > > > > > > > > > In the same time the makedumpfile is able to find and extract dmesg buffer from /proc/vmcore. > > > > > > > > The makedumpfile comes with kexec-tools-2.0.15-13.el7_5.2.aarch64 package. > > > > > > > > > > > > > > > > The issue is not reproduced for systems with small number of CPUs and log_buf not reallocated to memblock section. > > > > > > > > > > > > > > Seems like you are hitting a known issue we saw on qualcomm amberwing > > > > > > > platforms as well. > > > > > > > I have sent a patch-series titled 'kexec-tools/arm64: Add support to > > > > > > > read PHYS_OFFSET from vmcoreinfo inside '/proc/kcore' to this list > > > > > > > just a few minutes back. > > > > > > > > > > > > > > I have Cc'ed you to the patchset as I think it might fix the issue for > > > > > > > you. > > > > > > > > > > > > Got them, thank you. > > > > > > > > > > > > > Kindly try the patchset on your platform (cavium?) and let me > > > > > > > know if this fixes the issue for you. > > > > > > > > > > > > Sure, I'd like to check them at my side, but.. > > > > > > I fall into merge conflicts while trying to apply them onto > > > > > > https://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git/ > > > > > > master, kexec-tools 2.0.18-rc1 94159bc3c264fa26395e56302072276a139d18af > > > > > > > > > > Hmm.. that's strange as I rebased them on kexec-tools 2.0.18-rc1 > > > > > (94159bc3c264fa26395e56302072276a139d18af) > > > > > before sending out the patchset. > > > > > > > > > > > Are there any specific branch/revision for them to be applied ? > > > > > > (or it might be my mail server issues with formatting emails). > > > > > > > > > > > > > > > > Can you please try picking them up from my public github tree instead? > > > > > Here you can find the same: > > > > > https://github.com/bhupesh-sharma/kexec-tools/tree/read-phys-offset-from-kcore-upstream-v1 > > > > > > > > > > Please pick the top 2 commit from here. > > > > > > > > Applied them onto commit '94159bc kexec-tools 2.0.18-rc1'. > > > > > > > > Still having following error while saving dmesg by vmcore-dmesg: > > > > > > > > kdump: saving vmcore-dmesg.txt > > > > Failed to read log text of size 0 bytes: Bad address > > > > kdump: saving vmcore-dmesg.txt failed > > > > > > > > So far tried kernels 4.14.78, 4.16.18. > > > > > > You would need kernel 4.19-rc5 or above as the same exposes VMCOREINFO > > > as '/proc/kcore'. > > > > So far with 4.19-rc6 (and updated kexec, vmcore-dmesg but having kdump scripts from CentOS) > > the crashkernel can't found sysroot and thus it can't dump anything, so it timeouts and reboot system. > > > > > If you are having issues while switching to newer kernel, please share > > > the output(s) of following on your platform: > > > > > > # kexec -p /boot/vmlinuz-`uname -r` --initrd=/boot/initramfs-`uname > > > -r`.img --reuse-cmdline -d > > > > > > > attached as kexec-start.log.xz > > > > > and, > > > > > > # readelf -l vmcore > > > > [root@2sgbt-53 vlomovts]# readelf -l vmcore > > readelf: vmcore: Error: No such file > > [root@2sgbt-53 vlomovts]# uname -r > > 4.19.0-rc6+ > > > > > > > > and, > > > > > > # cat /proc/iomem > > > > attached as cat-proc-iomem.log.xz > > Just to confirm: these logs are after your apply my kexec-tools patches, right? Yes, applied, rebuild kexec and start kernel as you suggest. > It looks likely that we are seeing differences in the value of > 'phys_offset' on your platforms: > > From, '/proc/iomem', we can see that phys_offset is 0x01400000: > 01400000-ffedffff : System RAM I've found start of dmesg manually at vmcore elf and found that the offset at file and offset found by vmcore-dmesg is differs for 0x140000, which is the PHYS_OFFSET, and it is set to 0 at my vmcore for some reason (part of my vmcore-debug printous): [...] NUMBER(kimage_voffset)=0xffff000006c00000 NUMBER(PHYS_OFFSET)=0x0 [...] > > while the 'kexec -p -d' logs indicate that it is 0: > image_arm64_load: phys_offset: 0000000000000000 Yes, it is. Double-check it for 4.19-rc6 and it is still 0x0. > > This tells me that the phys_offset value is not correctly calculated > in kexec-tools which should be fixed after my patches. > > BTW , by '# readelf -l vmcore', I meant the 'vmcore' dump file you > have obtained via 'kexec'. It might be that you are saving it on some > different location (something /var/crash?). Can you please try sharing > the output of the same as well? Sorry, my bad here. but the problem is that I can't get kexec wortking with 4.19-rc6 kernel and I have vmcore dump for 4.14.69+ based kernel so far. So the output looks like following: [vlomovts@2sgbt-53 ~]$ readelf -l ~/vmcore-full Elf file type is CORE (Core file) Entry point 0x0 There are 10 program headers, starting at offset 64 Program Headers: Type Offset VirtAddr PhysAddr FileSiz MemSiz Flags Align NOTE 0x0000000000010000 0x0000000000000000 0x0000000000000000 0x000000000000a1a0 0x000000000000a1a0 0 LOAD 0x0000000000020000 0xffff000008080000 0x0000000001480000 0x0000000001bc0000 0x0000000001bc0000 RWE 0 LOAD 0x0000000001be0000 0xffff800000000000 0x0000000001400000 0x00000000dea00000 0x00000000dea00000 RWE 0 LOAD 0x00000000e05e0000 0xffff8000fea00000 0x00000000ffe00000 0x00000000000e0000 0x00000000000e0000 RWE 0 LOAD 0x00000000e06c0000 0xffff8000feb00000 0x00000000fff00000 0x0000000000090000 0x0000000000090000 RWE 0 LOAD 0x00000000e0750000 0xffff8000feba0000 0x00000000fffa0000 0x0000001f00060000 0x0000001f00060000 RWE 0 LOAD 0x0000001fe07b0000 0xffff80ffff000000 0x0000010000400000 0x0000001ffaae0000 0x0000001ffaae0000 RWE 0 LOAD 0x0000003fdb290000 0xffff811ff9bc0000 0x0000011ffafc0000 0x0000000004fd0000 0x0000000004fd0000 RWE 0 LOAD 0x0000003fe0260000 0xffff811ffeba0000 0x0000011ffffa0000 0x0000000000010000 0x0000000000010000 RWE 0 LOAD 0x0000003fe0270000 0xffff811ffebe0000 0x0000011ffffe0000 0x0000000000020000 0x0000000000020000 RWE 0 WBR, Vadim > > Regards, > Bhupesh > > > > > > > And then I can suggest a hack, which you can try and test on your > > > platform and then we can take it forward from there. > > > > > > Thanks, > > > Bhupesh > > > > > > > > > > > > > Thanks, > > > > > Bhupesh > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > Bhupesh _______________________________________________ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2018-10-29 11:21 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-10-24 12:52 [BUG] vmcore-dmesg cant' read dmesg log from /proc/vmcore if log_buf is reallocated due to large number of CPUs Lomovtsev, Vadim 2018-10-24 21:30 ` Bhupesh Sharma 2018-10-25 10:40 ` Vadim Lomovtsev 2018-10-26 6:55 ` Bhupesh Sharma 2018-10-26 10:11 ` Vadim Lomovtsev 2018-10-26 10:19 ` Bhupesh Sharma 2018-10-26 13:18 ` Vadim Lomovtsev 2018-10-26 23:11 ` Bhupesh Sharma 2018-10-29 11:21 ` Vadim Lomovtsev
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.