Kexec on arm64

[achandran@mvista.com (Arun Chandran)]

On Fri, Jul 25, 2014 at 5:44 PM, Mark Rutland <mark.rutland@arm.com> wrote:
> On Fri, Jul 25, 2014 at 12:48:04PM +0100, Arun Chandran wrote:
>> Hi Geoff,
>>
>> On Fri, Jul 25, 2014 at 5:47 AM, Geoff Levand <geoff@infradead.org> wrote:
>> > Hi,
>> >
>> > On Thu, 2014-07-24 at 10:36 +0100, Mark Rutland wrote:
>> >> On Thu, Jul 24, 2014 at 01:38:07AM +0100, Geoff Levand wrote:
>> >
>> >> > All memory management for the main cpu is done by the arch code.  Kexec
>> >> > and cpu hot plug only work with the secondary cpus, so the problem would
>> >> > be in the arch memory code, either in setup_restart() for shutdown, or
>> >> > in the startup code.
>> >>
>> >> It's possible that soft_restart and setup_restart are a little dodgy, as
>> >> they also rely on the compiler being smart and not touching the stack
>> >> after setup_restart().
>> >>
>> >> However, they provide absolutely no guarantee that any data has been
>> >> flushed out to the PoC [1]. If you require any data to be flushed out to the
>> >> PoC so as to be visible to noncacheable accesses, you will need to
>> >> ensure that this is flushed by VA before soft_restart is called. Data
>> >> may have migrated to another cache (e.g. another CPU, or the L3) where
>> >> it is not visible.
>> >
>> > OK, kexec's reset routine relocate_new_kernel does use a few global
>> > variables that are set by the main kexec routines.  I added a call to
>> > __flush_dcache_area(), which uses 'dc civac' for those.
>> >
>> > I had thought the call to __flush_dcache_all, which uses 'dc cisw', in
>> > flush_cache_all() would be enough.
>> >
>> > Arun, I also fixed UP builds.  Could you pull my latest and try with L3
>> > enabled?
>> >
>> I got this working. As 'Mark Rutland' pointed in another mail that
>> it could be problem with flushing the cache; I did a read of
>> 1GB data from start of RAM to a volatile var. I assume that
>> this will clear and invalidate all that in cache (L1=32K, L2=256 K, L3=8M)
>
> You've managed to get the cache to evict some lines, which proves my
> theory, but this is absolute nonsense and guarantees nothing.
>
> So NAK to this.
>

Yes I was just shooting wildly.

> If you need to perform cache maintenance to guarantee data is visible to
> non-cacheable accesses  you _must_ use the architected mechanism for
> cleaning data to the PoC: DC CVAC. We have wrappers for flushing ranges.
>
> Anything else is nonsense and does not provide the guarantee you need.
>
> That said, I still am not sure what guarantee you are attempting to get.
> Which data do you need out at the PoC?
>

I tried flushing the jump addr

##########
 +static unsigned long jump_addr_save;
 void soft_restart(unsigned long addr)
 {
        typedef void (*phys_reset_t)(unsigned long);
        phys_reset_t phys_reset;

+       phys_reset = (phys_reset_t)virt_to_phys(cpu_reset);
+       jump_addr_save = addr;
+        __flush_dcache_area(&jump_addr_save, 16);
+        __flush_dcache_area(&phys_reset, 16);
        setup_restart();

        /* Switch to the identity mapping */
-       phys_reset = (phys_reset_t)virt_to_phys(cpu_reset);
-       phys_reset(addr);
+       phys_reset(jump_addr_save);

        /* Should never get here */
        BUG();
###########

And flushing all the source and destination pages of the kexeced
kernel

##########
diff --git a/arch/arm64/kernel/machine_kexec.c
b/arch/arm64/kernel/machine_kexec.c
index 2995c78..3edf567 100644
--- a/arch/arm64/kernel/machine_kexec.c
+++ b/arch/arm64/kernel/machine_kexec.c
@@ -221,6 +221,8 @@ static void _kexec_entry_dump(const char *func, int line,
                                                addr,
                                                (unsigned
long)virt_to_phys(dest),
                                                dest);
+                               __flush_dcache_area(addr, PAGE_SIZE);
+                               __flush_dcache_area(dest, PAGE_SIZE);
                                dest += PAGE_SIZE;
                                break;
                        case IND_DONE:
###########

It still fails(not reboot to kexeced kernel); That means I miss
flushing of some other
area.

--Arun

> Thanks,
> Mark.
>
>>
>> ###################
>> diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c
>> index 786daa6..90418f3 100644
>> --- a/arch/arm64/kernel/process.c
>> +++ b/arch/arm64/kernel/process.c
>> @@ -63,6 +63,10 @@ static inline void smp_secondary_shutdown(void) {}
>>
>>  static void setup_restart(void)
>>  {
>> +       volatile u64 tmp;
>> +       volatile u64 *addr;
>> +
>> +       addr = (u64 *)(0xffffffc000000000);
>>         /*
>>          * Tell the mm system that we are going to reboot -
>>          * we may need it to insert some 1:1 mappings so that
>> @@ -75,6 +79,11 @@ static void setup_restart(void)
>>         /* Clean and invalidate caches */
>>         flush_cache_all();
>>
>> +       for ( ;addr < (u64 *)0xffffffc040000000; addr++)
>> +       {
>> +               tmp = *addr;
>> +       }
>> +
>>         /* Turn D-cache off */
>>         cpu_cache_off();
>>
>> ###################
>>
>> With the above change latest kernel @
>> https://git.linaro.org/people/geoff.levand/linux-kexec.git
>> is able to do kexec with L3 enabled in UP scenario.
>>
>> --Arun
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> > -Geoff
>> >
>>