kexec fails to boot kernels where CONFIG_RANDOMIZE_BASE=y is set

kexec fails to boot kernels where CONFIG_RANDOMIZE_BASE=y is set

[Thomas D.]

Hi,

looks like kexec doesn't support kernels where

  CONFIG_RANDOMIZE_BASE=y

is set.


I already created a bug for Debian in June for this problem [1].

I am able to confirm the same problem with kexec 2.0.7 on Gentoo and
when building from source (943ba35f8143408d8ada9a24d0986663cc612df9).

Is this a known problem/limitation?

Is there anything I can help you with to get this fixed?

Thanks.


See also:
=========
[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=754288


-Thomas

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: kexec fails to boot kernels where CONFIG_RANDOMIZE_BASE=y is set

[Vivek Goyal]

Hi Thomas,

I think kexec is broken with CONFIG_RANDOMIZE_BASE=y. Chao had raised
this issue some time back when this option was introduced. I don't
remember the details though that why it is broken.

CCing Kees Cook. 

Kees, is there a quick fix to the issue. Sooner or later people will
start enabling CONFIG_RANDOMIZE_BASE=y and broken kexec will be a
big issue.

Thanks
Vivek
 
On Sun, Aug 17, 2014 at 11:02:58PM +0200, Thomas D. wrote:
> Hi,
> 
> looks like kexec doesn't support kernels where
> 
>   CONFIG_RANDOMIZE_BASE=y
> 
> is set.
> 
> 
> I already created a bug for Debian in June for this problem [1].
> 
> I am able to confirm the same problem with kexec 2.0.7 on Gentoo and
> when building from source (943ba35f8143408d8ada9a24d0986663cc612df9).
> 
> Is this a known problem/limitation?
> 
> Is there anything I can help you with to get this fixed?
> 
> Thanks.
> 
> 
> See also:
> =========
> [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=754288
> 
> 
> -Thomas
> 
> _______________________________________________
> kexec mailing list
> kexec@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/kexec

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: kexec fails to boot kernels where CONFIG_RANDOMIZE_BASE=y is set

[WANG Chao]

On 08/18/14 at 10:57am, Vivek Goyal wrote:
> Hi Thomas,
> 
> I think kexec is broken with CONFIG_RANDOMIZE_BASE=y. Chao had raised
> this issue some time back when this option was introduced. I don't
> remember the details though that why it is broken.

The following fix the problem for kdump case:

commit 0d52644
Author: WANG Chao <chaowang@redhat.com>
Date:   Fri Mar 28 15:05:00 2014 +0800

    x86, kaslr: add alternative way to locate kernel text mapping area


For kexec case, it hangs in purgatory:

[  556.859384] kexec: Starting new kernel
I'm in purgatory

> 
> CCing Kees Cook. 
> 
> Kees, is there a quick fix to the issue. Sooner or later people will
> start enabling CONFIG_RANDOMIZE_BASE=y and broken kexec will be a
> big issue.
> 
> Thanks
> Vivek
>  
> On Sun, Aug 17, 2014 at 11:02:58PM +0200, Thomas D. wrote:
> > Hi,
> > 
> > looks like kexec doesn't support kernels where
> > 
> >   CONFIG_RANDOMIZE_BASE=y
> > 
> > is set.
> > 
> > 
> > I already created a bug for Debian in June for this problem [1].
> > 
> > I am able to confirm the same problem with kexec 2.0.7 on Gentoo and
> > when building from source (943ba35f8143408d8ada9a24d0986663cc612df9).
> > 
> > Is this a known problem/limitation?
> > 
> > Is there anything I can help you with to get this fixed?
> > 
> > Thanks.
> > 
> > 
> > See also:
> > =========
> > [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=754288
> > 
> > 
> > -Thomas
> > 
> > _______________________________________________
> > kexec mailing list
> > kexec@lists.infradead.org
> > http://lists.infradead.org/mailman/listinfo/kexec

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: kexec fails to boot kernels where CONFIG_RANDOMIZE_BASE=y is set

[Vivek Goyal]

On Tue, Aug 19, 2014 at 05:07:24PM +0800, WANG Chao wrote:
> On 08/18/14 at 10:57am, Vivek Goyal wrote:
> > Hi Thomas,
> > 
> > I think kexec is broken with CONFIG_RANDOMIZE_BASE=y. Chao had raised
> > this issue some time back when this option was introduced. I don't
> > remember the details though that why it is broken.
> 
> The following fix the problem for kdump case:
> 
> commit 0d52644
> Author: WANG Chao <chaowang@redhat.com>
> Date:   Fri Mar 28 15:05:00 2014 +0800
> 
>     x86, kaslr: add alternative way to locate kernel text mapping area
> 
> 
> For kexec case, it hangs in purgatory:
> 
> [  556.859384] kexec: Starting new kernel
> I'm in purgatory

Chao, 

Do you know why does it hang in purgatory in case of kexec?

Thanks
Vivek

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: kexec fails to boot kernels where CONFIG_RANDOMIZE_BASE=y is set

[Kees Cook]

On Wed, Aug 20, 2014 at 9:33 AM, Vivek Goyal <vgoyal@redhat.com> wrote:
> On Tue, Aug 19, 2014 at 05:07:24PM +0800, WANG Chao wrote:
>> On 08/18/14 at 10:57am, Vivek Goyal wrote:
>> > Hi Thomas,
>> >
>> > I think kexec is broken with CONFIG_RANDOMIZE_BASE=y. Chao had raised
>> > this issue some time back when this option was introduced. I don't
>> > remember the details though that why it is broken.

The "normal" problems with kaslr have to do with areas of memory that
shouldn't be stomped on, or if 1-to-1 page tables are not in place.
What state are the page tables in when doing the kexec, and how are
kernel parameters (including e820) passed?

>>
>> The following fix the problem for kdump case:
>>
>> commit 0d52644
>> Author: WANG Chao <chaowang@redhat.com>
>> Date:   Fri Mar 28 15:05:00 2014 +0800
>>
>>     x86, kaslr: add alternative way to locate kernel text mapping area

I don't see this in Linus's tree? Where can I find this commit?

>>
>>
>> For kexec case, it hangs in purgatory:
>>
>> [  556.859384] kexec: Starting new kernel
>> I'm in purgatory
>
> Chao,
>
> Do you know why does it hang in purgatory in case of kexec?
>
> Thanks
> Vivek

-Kees


-- 
Kees Cook
Chrome OS Security

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: kexec fails to boot kernels where CONFIG_RANDOMIZE_BASE=y is set

[Vivek Goyal]

On Thu, Aug 21, 2014 at 10:57:09AM -0500, Kees Cook wrote:
> On Wed, Aug 20, 2014 at 9:33 AM, Vivek Goyal <vgoyal@redhat.com> wrote:
> > On Tue, Aug 19, 2014 at 05:07:24PM +0800, WANG Chao wrote:
> >> On 08/18/14 at 10:57am, Vivek Goyal wrote:
> >> > Hi Thomas,
> >> >
> >> > I think kexec is broken with CONFIG_RANDOMIZE_BASE=y. Chao had raised
> >> > this issue some time back when this option was introduced. I don't
> >> > remember the details though that why it is broken.
> 
> The "normal" problems with kaslr have to do with areas of memory that
> shouldn't be stomped on, or if 1-to-1 page tables are not in place.
> What state are the page tables in when doing the kexec, and how are
> kernel parameters (including e820) passed?

Hi Kees,

I suspect that it has something to do with overwriting page tables or some
other data by the new kernel.

IIUC, we are preparing identity mapped page tables for "nr_pfn_mapped".

arch/x86/kernel/machine_kexec_64.c
init_pgtable() {
        for (i = 0; i < nr_pfn_mapped; i++) {
                mstart = pfn_mapped[i].start << PAGE_SHIFT;
                mend   = pfn_mapped[i].end << PAGE_SHIFT; 

                result = kernel_ident_mapping_init(&info,
                                                 level4p, mstart, mend);
                if (result)
                        return result;
        }
}

So most likely page tables have been setup right. In fact, if one is
trying to load bzimage32, then we drop to 32bit mode and disable paging.
This new 64bit must be using the page tables setup by old kernel.

Mememory map (e820) and kernel parameters are passed in bootparams. This
is that 4K page setup by old kernel.

So I have question. How does kASLR work. Previously x86_64 relocatable
kernel will move itself to proper alignment boundary and then page
tables will be updated properly. IIRC, virtual addresses reamined fixed
and PAGE_OFFSET was not fixed.

Now with kASLR, are you moving kernel physically significantly or kernel
is not moved physically just that its placement in virtual address is not
fixed and it is chosen randomly?

If kernel is being moved physically, then we potentially have the issue of
it stomping other things. So how do we make sure that it does not overwrite
initramfs, or previous kernel's page tables or something else?

If kernel is not moving physically and only its location in virtual
address space changes, then it is very puzzling that why it should be
a problem.

If kernel always moves itself to higher addresses then one solution could
be that load everything else below kernel and load kernel at higher
addresses. But old kexec system call will not be able to cope with it as
user space determines the load location of kernel and other segments while
running kernel decides location of pages for page table and kernel has
no idea where user space has loaded new kernel. New system call still
might be able to handle it.

Also, I vaguely recall that there was a kernel parameter to disable kASLR.
And kexec/kdump initially can use that paramter as a work around. What was
that parameter.

> 
> >>
> >> The following fix the problem for kdump case:
> >>
> >> commit 0d52644
> >> Author: WANG Chao <chaowang@redhat.com>
> >> Date:   Fri Mar 28 15:05:00 2014 +0800
> >>
> >>     x86, kaslr: add alternative way to locate kernel text mapping area
> 
> I don't see this in Linus's tree? Where can I find this commit?

This is a kexec-tools patch and not kernel patch, that's why you don't
see it in linus tree. kexec-tools has to prepare ELF headers for kernel
text area. As with kASLR kernel text virtual addresses moved, kexec-tools
had to be modified to look at /proc/kallsyms and look for symbol _stext to
figure out where kernel text is and prepare ELF header accordingly.

Thanks
Vivek

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: kexec fails to boot kernels where CONFIG_RANDOMIZE_BASE=y is set

[Vivek Goyal]

On Thu, Aug 21, 2014 at 02:10:00PM -0400, Vivek Goyal wrote:

[..]
> Also, I vaguely recall that there was a kernel parameter to disable kASLR.
> And kexec/kdump initially can use that paramter as a work around. What was
> that parameter.

I see that "nokaslr" will disable it at run time. I remember that we
had discussions that it can be used as a work around. But I don't see
kexec-tools or fedora kdump scripts appending "nokaslr" in kdump case.
That's strange.

Chao, would you remember anything about it. 

Thomas, can you please test kexec with "nokaslr" specified in command
line. This can atleast get you going for now till the problem is actually
fixed.

Thanks
Vivek


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: kexec fails to boot kernels where CONFIG_RANDOMIZE_BASE=y is set

[Vivek Goyal]

On Thu, Aug 21, 2014 at 02:10:00PM -0400, Vivek Goyal wrote:

[..]
> If kernel always moves itself to higher addresses then one solution could
> be that load everything else below kernel and load kernel at higher
> addresses. But old kexec system call will not be able to cope with it as
> user space determines the load location of kernel and other segments while
> running kernel decides location of pages for page table and kernel has
> no idea where user space has loaded new kernel. New system call still
> might be able to handle it.

So I am reading kaslr code a bit. First of all we are moving kernel in
physical address space. That would substantiate the theory that kernel
movement stepped onto something.

Basically we seem to be just going through all the e820 entries, putting
them one slots[] array and choosing one entry randomly.

That means kernel could move up or down. So the notion of loading
everything else below kernel will not work.

Now this makes me wonder that how does this whole thing work with grub.
IOW, how would one make sure that kernel does not stomp initramfs.

I guess only protection is BASE_MAX_OFFSET and making sure initramfs is
loaded higher than that.

If that's the case, then I think minimum and maximum range of physical
memory where kernel can move should be exported through bzImage header
and kexec-tools should make sure that anything else is outside of that
range so that new kernel will not stomp over it.

Thanks
Vivek

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: kexec fails to boot kernels where CONFIG_RANDOMIZE_BASE=y is set

[Thomas D.]

Hi,

Vivek Goyal wrote:
> Thomas, can you please test kexec with "nokaslr" specified in command
> line. This can atleast get you going for now till the problem is actually
> fixed.

I booted linux-3.15.10 with "nokaslr"

>  # dmesg | grep -i command
> [    0.000000] Command line: BOOT_IMAGE=/kernel dolvm video=1280x1024 root=UUID=6d42... rootfs=ext4 nokaslr initrd=/initramfs
> [    0.000000] Kernel command line: BOOT_IMAGE=/kernel dolvm video=1280x1024 root=UUID=6d42... rootfs=ext4 nokaslr initrd=/initramfs

and run kexec with

> --reuse-cmdline -l /boot/kernel root=/dev/dm-3  --initrd=/boot/initramfs

But it still doesn't reboot. I see no difference between using "nokaslr"
or not.


-Thomas


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: kexec fails to boot kernels where CONFIG_RANDOMIZE_BASE=y is set

[WANG Chao]

On 08/21/14 at 02:10pm, Vivek Goyal wrote:
>
> If kernel is being moved physically, then we potentially have the issue of
> it stomping other things. So how do we make sure that it does not overwrite
> initramfs, or previous kernel's page tables or something else?

In case you don't read that part, memory of initrd, cmdline and others
won't be overwritten.

This work is done in x86/boot/compressed/aslr.c::mem_avoid_init().

Thanks
WANG Chao

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: kexec fails to boot kernels where CONFIG_RANDOMIZE_BASE=y is set

[Baoquan He]

On 08/22/14 at 11:19am, WANG Chao wrote:
> On 08/21/14 at 02:10pm, Vivek Goyal wrote:
> >
> > If kernel is being moved physically, then we potentially have the issue of
> > it stomping other things. So how do we make sure that it does not overwrite
> > initramfs, or previous kernel's page tables or something else?
> 
> In case you don't read that part, memory of initrd, cmdline and others
> won't be overwritten.
> 
> This work is done in x86/boot/compressed/aslr.c::mem_avoid_init().

Yeah, it's just as Chao said.

And I think it's still about the relocation calculation issue. I tried
the patch Lu Yinghai suggested, now kexec can work.

But this patch only make kexec work. The slot choosing still
doesn't work per the limitation checking, since only range below
CONFIG_RANDOMIZE_BASE_MAX_OFFSET can be chosen and kexec-tools
always likes putting kexec kernel from the end of system ram.

So we have 2 choices for kexec/kdump:
1) kexec/kdump kernel need not randomize the kernel starting point.
Since kexec/kdump kernel is only for testing or emergencey, its life is
not too long.

2) makes slots around the kernel input addr. This is only useful for
kexec. I can't imagine why kdump need it. 




Hi Thomas,
Could you test below patch? 


This is patch is from Lu Yinghai.
---
 arch/x86/boot/compressed/misc.c |   14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

Index: linux-2.6/arch/x86/boot/compressed/misc.c
===================================================================
--- linux-2.6.orig/arch/x86/boot/compressed/misc.c
+++ linux-2.6/arch/x86/boot/compressed/misc.c
@@ -235,8 +235,9 @@ static void error(char *x)
 		asm("hlt");
 }
 
-#if CONFIG_X86_NEED_RELOCS
-static void handle_relocations(void *output, unsigned long output_len)
+#ifdef CONFIG_X86_NEED_RELOCS
+static void handle_relocations(void *output_orig, void *output,
+			       unsigned long output_len)
 {
 	int *reloc;
 	unsigned long delta, map, ptr;
@@ -247,7 +248,7 @@ static void handle_relocations(void *out
 	 * Calculate the delta between where vmlinux was linked to load
 	 * and where it was actually loaded.
 	 */
-	delta = min_addr - LOAD_PHYSICAL_ADDR;
+	delta = min_addr - (unsigned long)output_orig;
 	if (!delta) {
 		debug_putstr("No relocation needed... ");
 		return;
@@ -304,7 +305,8 @@ static void handle_relocations(void *out
 #endif
 }
 #else
-static inline void handle_relocations(void *output, unsigned long output_len)
+static inline void handle_relocations(void *output_orig, void *output,
+				      unsigned long output_len)
 { }
 #endif
 
@@ -365,6 +367,8 @@ asmlinkage void *decompress_kernel(void
 				  unsigned char *output,
 				  unsigned long output_len)
 {
+	unsigned char *output_orig = output;
+
 	real_mode = rmode;
 
 	sanitize_boot_params(real_mode);
@@ -417,7 +421,7 @@ asmlinkage void *decompress_kernel(void
 	debug_putstr("... ");
 	decompress(input_data, input_len, NULL, NULL, output, NULL, error);
 	parse_elf(output);
-	handle_relocations(output, output_len);
+	handle_relocations(output_orig, output, output_len);
 	debug_putstr("done.\nBooting the kernel.\n");
 	return output;
 }

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: kexec fails to boot kernels where CONFIG_RANDOMIZE_BASE=y is set

[Thomas D.]

Hi,

Baoquan He wrote:
> Could you test below patch? 
> 
> 
> This is patch is from Lu Yinghai.
> ---
>  arch/x86/boot/compressed/misc.c |   14 +++++++++-----
>  1 file changed, 9 insertions(+), 5 deletions(-)
> 
> Index: linux-2.6/arch/x86/boot/compressed/misc.c
> ===================================================================
> --- linux-2.6.orig/arch/x86/boot/compressed/misc.c
> +++ linux-2.6/arch/x86/boot/compressed/misc.c
> [...]

Sorry, I don't know how to apply this patch.

I am building kexec-tools from
git://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git master
branch and there is no misc.c?!


-Thomas


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: kexec fails to boot kernels where CONFIG_RANDOMIZE_BASE=y is set

[Vivek Goyal]

On Fri, Aug 22, 2014 at 11:19:41AM +0800, WANG Chao wrote:
> On 08/21/14 at 02:10pm, Vivek Goyal wrote:
> >
> > If kernel is being moved physically, then we potentially have the issue of
> > it stomping other things. So how do we make sure that it does not overwrite
> > initramfs, or previous kernel's page tables or something else?
> 
> In case you don't read that part, memory of initrd, cmdline and others
> won't be overwritten.
> 
> This work is done in x86/boot/compressed/aslr.c::mem_avoid_init().

Ok, got it. so cmdline and initrd info is present in bootparams and kernel
can use that to avoid stomping those.

But that still does not protect ELF header segment (prepared in case of
kdump) and page tables setup by previous kernel.

So may be this failure happens because new kernel stomps over page tables
of old kernel. But if that theory is right, then we should be able to
kexec using 32bit entry point of bzImage.

Thomas, have you had any success with that?

Thanks
Vivek

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: kexec fails to boot kernels where CONFIG_RANDOMIZE_BASE=y is set

[Vivek Goyal]

On Fri, Aug 22, 2014 at 02:30:37PM +0200, Thomas D. wrote:
> Hi,
> 
> Baoquan He wrote:
> > Could you test below patch? 
> > 
> > 
> > This is patch is from Lu Yinghai.
> > ---
> >  arch/x86/boot/compressed/misc.c |   14 +++++++++-----
> >  1 file changed, 9 insertions(+), 5 deletions(-)
> > 
> > Index: linux-2.6/arch/x86/boot/compressed/misc.c
> > ===================================================================
> > --- linux-2.6.orig/arch/x86/boot/compressed/misc.c
> > +++ linux-2.6/arch/x86/boot/compressed/misc.c
> > [...]
> 
> Sorry, I don't know how to apply this patch.
> 
> I am building kexec-tools from
> git://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git master
> branch and there is no misc.c?!

This is a kernel patch and not kexec-tools patch.

Thanks
Vivek

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: kexec fails to boot kernels where CONFIG_RANDOMIZE_BASE=y is set

[Thomas D.]

On 2014-08-22 14:38, Vivek Goyal wrote:
> So may be this failure happens because new kernel stomps over page tables
> of old kernel. But if that theory is right, then we should be able to
> kexec using 32bit entry point of bzImage.
> 
> Thomas, have you had any success with that?

When I execute kexec with "--entry-32bit" (I understand that you asked
me to try that) the new kernel will boot as expected.

I'll now try the kernel patch...


-Thomas


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: kexec fails to boot kernels where CONFIG_RANDOMIZE_BASE=y is set

[Vivek Goyal]

On Fri, Aug 22, 2014 at 02:47:39PM +0200, Thomas D. wrote:
> On 2014-08-22 14:38, Vivek Goyal wrote:
> > So may be this failure happens because new kernel stomps over page tables
> > of old kernel. But if that theory is right, then we should be able to
> > kexec using 32bit entry point of bzImage.
> > 
> > Thomas, have you had any success with that?
> 
> When I execute kexec with "--entry-32bit" (I understand that you asked
> me to try that) the new kernel will boot as expected.

Good to know that --entry-32bit option makes kexec work with kASLR
enabled. So this indeed sounds like the issue of page tables being
stomped over by new kernel.

IIUC, there are bootloaders which can use 64bit entry point of kernel. I
think they must run into similar issue too.

Thanks
Vivek

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: kexec fails to boot kernels where CONFIG_RANDOMIZE_BASE=y is set

[Vivek Goyal]

On Fri, Aug 22, 2014 at 07:59:02PM +0800, Baoquan He wrote:

[..]
> So we have 2 choices for kexec/kdump:
> 1) kexec/kdump kernel need not randomize the kernel starting point.
> Since kexec/kdump kernel is only for testing or emergencey, its life is
> not too long.
> 
> 2) makes slots around the kernel input addr. This is only useful for
> kexec. I can't imagine why kdump need it. 
> 
> 
> 
> 
> Hi Thomas,
> Could you test below patch? 
> 
> 
> This is patch is from Lu Yinghai.
> ---
>  arch/x86/boot/compressed/misc.c |   14 +++++++++-----
>  1 file changed, 9 insertions(+), 5 deletions(-)
> 
> Index: linux-2.6/arch/x86/boot/compressed/misc.c
> ===================================================================
> --- linux-2.6.orig/arch/x86/boot/compressed/misc.c
> +++ linux-2.6/arch/x86/boot/compressed/misc.c
> @@ -235,8 +235,9 @@ static void error(char *x)
>  		asm("hlt");
>  }
>  
> -#if CONFIG_X86_NEED_RELOCS
> -static void handle_relocations(void *output, unsigned long output_len)
> +#ifdef CONFIG_X86_NEED_RELOCS
> +static void handle_relocations(void *output_orig, void *output,
> +			       unsigned long output_len)
>  {
>  	int *reloc;
>  	unsigned long delta, map, ptr;
> @@ -247,7 +248,7 @@ static void handle_relocations(void *out
>  	 * Calculate the delta between where vmlinux was linked to load
>  	 * and where it was actually loaded.
>  	 */
> -	delta = min_addr - LOAD_PHYSICAL_ADDR;
> +	delta = min_addr - (unsigned long)output_orig;

So what does this patch actuall do? If I try to trace back output_orig,
it seems to be same as LOAD_PHYSICAL_ADDR. That means there should not
be any effect of this change? Or did I not understand it.

Look at head_64.S

movq    $LOAD_PHYSICAL_ADDR, %rbp
...
...
movq    %rbp, %r8               /* output target address */
call    decompress_kernel       /* returns kernel location in %rax */

Thanks
Vivek

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: kexec fails to boot kernels where CONFIG_RANDOMIZE_BASE=y is set

[Thomas D.]

Hi,

the patch works!

I am able to reboot my system using kexec when the kernel has this patch.

I had to slightly modify the patch, so it will apply against linux-3.15.10:

--- arch/x86/boot/compressed/misc.c.old	2014-08-22 14:52:20.792158801 +0200
+++ arch/x86/boot/compressed/misc.c	2014-08-22 14:58:21.250506919 +0200
@@ -230,8 +230,9 @@
 		asm("hlt");
 }

-#if CONFIG_X86_NEED_RELOCS
-static void handle_relocations(void *output, unsigned long output_len)
+#ifdef CONFIG_X86_NEED_RELOCS
+static void handle_relocations(void *output_orig, void *output,
+			       unsigned long output_len)
 {
 	int *reloc;
 	unsigned long delta, map, ptr;
@@ -242,7 +243,7 @@
 	 * Calculate the delta between where vmlinux was linked to load
 	 * and where it was actually loaded.
 	 */
-	delta = min_addr - LOAD_PHYSICAL_ADDR;
+	delta = min_addr - (unsigned long)output_orig;
 	if (!delta) {
 		debug_putstr("No relocation needed... ");
 		return;
@@ -299,7 +300,8 @@
 #endif
 }
 #else
-static inline void handle_relocations(void *output, unsigned long
output_len)
+static inline void handle_relocations(void *output_orig, void *output,
+				      unsigned long output_len)
 { }
 #endif

@@ -360,6 +362,8 @@
 				  unsigned char *output,
 				  unsigned long output_len)
 {
+	unsigned char *output_orig = output;
+
 	real_mode = rmode;

 	sanitize_boot_params(real_mode);
@@ -402,7 +406,7 @@
 	debug_putstr("\nDecompressing Linux... ");
 	decompress(input_data, input_len, NULL, NULL, output, NULL, error);
 	parse_elf(output);
-	handle_relocations(output, output_len);
+	handle_relocations(output_orig, output, output_len);
 	debug_putstr("done.\nBooting the kernel.\n");
 	return output;
 }




So now we have two "solutions":

1) Using kexec with "--entry-32bit" parameter

2) Use a patched kernel


-Thomas

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: kexec fails to boot kernels where CONFIG_RANDOMIZE_BASE=y is set

[Baoquan He]

On 08/22/14 at 09:16am, Vivek Goyal wrote:
> On Fri, Aug 22, 2014 at 07:59:02PM +0800, Baoquan He wrote:
> 
> [..]
> > So we have 2 choices for kexec/kdump:
> > 1) kexec/kdump kernel need not randomize the kernel starting point.
> > Since kexec/kdump kernel is only for testing or emergencey, its life is
> > not too long.
> > 
> > 2) makes slots around the kernel input addr. This is only useful for
> > kexec. I can't imagine why kdump need it. 
> > 
> > 
> > 
> > 
> > Hi Thomas,
> > Could you test below patch? 
> > 
> > 
> > This is patch is from Lu Yinghai.
> > ---
> >  arch/x86/boot/compressed/misc.c |   14 +++++++++-----
> >  1 file changed, 9 insertions(+), 5 deletions(-)
> > 
> > Index: linux-2.6/arch/x86/boot/compressed/misc.c
> > ===================================================================
> > --- linux-2.6.orig/arch/x86/boot/compressed/misc.c
> > +++ linux-2.6/arch/x86/boot/compressed/misc.c
> > @@ -235,8 +235,9 @@ static void error(char *x)
> >  		asm("hlt");
> >  }
> >  
> > -#if CONFIG_X86_NEED_RELOCS
> > -static void handle_relocations(void *output, unsigned long output_len)
> > +#ifdef CONFIG_X86_NEED_RELOCS
> > +static void handle_relocations(void *output_orig, void *output,
> > +			       unsigned long output_len)
> >  {
> >  	int *reloc;
> >  	unsigned long delta, map, ptr;
> > @@ -247,7 +248,7 @@ static void handle_relocations(void *out
> >  	 * Calculate the delta between where vmlinux was linked to load
> >  	 * and where it was actually loaded.
> >  	 */
> > -	delta = min_addr - LOAD_PHYSICAL_ADDR;
> > +	delta = min_addr - (unsigned long)output_orig;
> 
> So what does this patch actuall do? If I try to trace back output_orig,
> it seems to be same as LOAD_PHYSICAL_ADDR. That means there should not
> be any effect of this change? Or did I not understand it.

No, kexec kernel has to be relocatable kernel. so it shoule be the
value calculated by code block surrounded by the MACRO checking.

(startup_32 + %rip + BP_kernel_alignment -1 )& ~(BP_kernel_alignment-1)

#ifdef CONFIG_RELOCATABLE      
        leaq    startup_32(%rip) /* - $startup_32 */, %rbp
        movl    BP_kernel_alignment(%rsi), %eax 
        decl    %eax
        addq    %rax, %rbp     
        notq    %rax           
        andq    %rax, %rbp     
        cmpq    $LOAD_PHYSICAL_ADDR, %rbp
        jge     1f
#endif
        movq    $LOAD_PHYSICAL_ADDR, %rbp
1:        
          
        /* Target address to relocate to for decompression */
        leaq    z_extract_offset(%rbp), %rbx
> 
> Look at head_64.S
> 
> movq    $LOAD_PHYSICAL_ADDR, %rbp
> ...
> ...
> movq    %rbp, %r8               /* output target address */
> call    decompress_kernel       /* returns kernel location in %rax */
> 
> Thanks
> Vivek
> 
> _______________________________________________
> kexec mailing list
> kexec@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/kexec

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: kexec fails to boot kernels where CONFIG_RANDOMIZE_BASE=y is set

[Baoquan He]

On 08/22/14 at 08:53am, Vivek Goyal wrote:
> On Fri, Aug 22, 2014 at 02:47:39PM +0200, Thomas D. wrote:
> > On 2014-08-22 14:38, Vivek Goyal wrote:
> > > So may be this failure happens because new kernel stomps over page tables
> > > of old kernel. But if that theory is right, then we should be able to
> > > kexec using 32bit entry point of bzImage.
> > > 
> > > Thomas, have you had any success with that?
> > 
> > When I execute kexec with "--entry-32bit" (I understand that you asked
> > me to try that) the new kernel will boot as expected.
> 
> Good to know that --entry-32bit option makes kexec work with kASLR
> enabled. So this indeed sounds like the issue of page tables being
> stomped over by new kernel.

I don't think page tables could be stomped over by new kernel.
The reason why --entry-32bit can work is kexec-tools get the memory near
the beginning of system RAM. You can check the value passed to argument
buf_end. when --entry-32bit it's 0, that means find memoey hole from low
to high. 

But in 64bit mode, the buf_end is always -1, this cause all memory ready
for kernel/initrd is close to th end of system ram. On my PC with 16G
memory, all those memory around 16G. I changed buf_end to 1, then use
this to load bzImage64, it works too. And addr for
kernel/initrd/real_mode are all allocated from low to high. I don't know
why we need put kernel/initrd/xxx close to the end of system ram.


diff --git a/kexec/add_buffer.c b/kexec/add_buffer.c
index 4d4a55f..ce76669 100644
--- a/kexec/add_buffer.c
+++ b/kexec/add_buffer.c
@@ -10,5 +10,5 @@ unsigned long add_buffer(struct kexec_info *info,
                         int buf_end)
 {
        return add_buffer_virt(info, buf, bufsz, memsz, buf_align,
-                              buf_min, buf_max, buf_end);
+                              buf_min, buf_max, 1);
 }

> 
> IIUC, there are bootloaders which can use 64bit entry point of kernel. I
> think they must run into similar issue too.
> 
> Thanks
> Vivek
> 
> _______________________________________________
> kexec mailing list
> kexec@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/kexec

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Kexec failing in handle_relocations() (Was: Re: kexec fails to boot kernels where CONFIG_RANDOMIZE_BASE=y is set)

[Vivek Goyal]

On Thu, Aug 21, 2014 at 09:27:17PM +0200, Thomas D. wrote:
> Hi,
> 
> Vivek Goyal wrote:
> > Thomas, can you please test kexec with "nokaslr" specified in command
> > line. This can atleast get you going for now till the problem is actually
> > fixed.
> 
> I booted linux-3.15.10 with "nokaslr"
> 
> >  # dmesg | grep -i command
> > [    0.000000] Command line: BOOT_IMAGE=/kernel dolvm video=1280x1024 root=UUID=6d42... rootfs=ext4 nokaslr initrd=/initramfs
> > [    0.000000] Kernel command line: BOOT_IMAGE=/kernel dolvm video=1280x1024 root=UUID=6d42... rootfs=ext4 nokaslr initrd=/initramfs
> 
> and run kexec with
> 
> > --reuse-cmdline -l /boot/kernel root=/dev/dm-3  --initrd=/boot/initramfs
> 
> But it still doesn't reboot. I see no difference between using "nokaslr"
> or not.

[ CC hpa ]

I think handle_relocations() is contributing to the problem here.

I enabled earlyprintk and enabled debug boot messages. I also put some
debug messages of my own to print values of some variables and I see
following when I try to kexec.

**********************************************************************
[  340.709078] kexec: Starting new kernel
early console in decompress_kernel
KASLR disabled by default...

Decompressing Linux... Parsing ELF...

min_addr=000000042e000000
delta=000000042d000000
Performing relocations...
map=00000004ad000000

32-bit relocation outside of kernel!


 -- System halted
*****************************************************************

So min_addr is the location where kernel is actually being loaded and
will be run from. delta is the difference between compile time address
and run time address. 

KASLR is disabled by default. I am wondering that why do we have to
go through handle_relocations() if KASLR is disabled. Should we fall
back to old logic of manipulating page tables.

Or we need to figure out how to make handle_relocations() work in such
a way that kernel can be loaded anywhere in the memory. 

I suspect that 32bit relocations might not be able to handle it and
we probably might need all 64bit relocations to handle that case. But
with this resulting virtual addresses might still be the problem.

I summary, I am wondering what kind of limitation handle_relocations()
put on physical location of kernel and how can we remove that restriction
to be able to load kernel anywhere in memory.

Thanks
Vivek

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec