From mboxrd@z Thu Jan 1 00:00:00 1970 From: achandran@mvista.com (Arun Chandran) Date: Fri, 25 Jul 2014 20:59:08 +0530 Subject: Kexec on arm64 In-Reply-To: <20140725121448.GB19632@leverpostej> References: <1405443898.22585.7.camel@smoke> <1405551861.7262.26.camel@smoke> <1406162287.4062.39.camel@smoke> <20140724093603.GC4079@leverpostej> <1406247468.4062.59.camel@smoke> <20140725121448.GB19632@leverpostej> Message-ID: To: linux-arm-kernel@lists.infradead.org List-Id: linux-arm-kernel.lists.infradead.org On Fri, Jul 25, 2014 at 5:44 PM, Mark Rutland wrote: > On Fri, Jul 25, 2014 at 12:48:04PM +0100, Arun Chandran wrote: >> Hi Geoff, >> >> On Fri, Jul 25, 2014 at 5:47 AM, Geoff Levand wrote: >> > Hi, >> > >> > On Thu, 2014-07-24 at 10:36 +0100, Mark Rutland wrote: >> >> On Thu, Jul 24, 2014 at 01:38:07AM +0100, Geoff Levand wrote: >> > >> >> > All memory management for the main cpu is done by the arch code. Kexec >> >> > and cpu hot plug only work with the secondary cpus, so the problem would >> >> > be in the arch memory code, either in setup_restart() for shutdown, or >> >> > in the startup code. >> >> >> >> It's possible that soft_restart and setup_restart are a little dodgy, as >> >> they also rely on the compiler being smart and not touching the stack >> >> after setup_restart(). >> >> >> >> However, they provide absolutely no guarantee that any data has been >> >> flushed out to the PoC [1]. If you require any data to be flushed out to the >> >> PoC so as to be visible to noncacheable accesses, you will need to >> >> ensure that this is flushed by VA before soft_restart is called. Data >> >> may have migrated to another cache (e.g. another CPU, or the L3) where >> >> it is not visible. >> > >> > OK, kexec's reset routine relocate_new_kernel does use a few global >> > variables that are set by the main kexec routines. I added a call to >> > __flush_dcache_area(), which uses 'dc civac' for those. >> > >> > I had thought the call to __flush_dcache_all, which uses 'dc cisw', in >> > flush_cache_all() would be enough. >> > >> > Arun, I also fixed UP builds. Could you pull my latest and try with L3 >> > enabled? >> > >> I got this working. As 'Mark Rutland' pointed in another mail that >> it could be problem with flushing the cache; I did a read of >> 1GB data from start of RAM to a volatile var. I assume that >> this will clear and invalidate all that in cache (L1=32K, L2=256 K, L3=8M) > > You've managed to get the cache to evict some lines, which proves my > theory, but this is absolute nonsense and guarantees nothing. > > So NAK to this. > Yes I was just shooting wildly. > If you need to perform cache maintenance to guarantee data is visible to > non-cacheable accesses you _must_ use the architected mechanism for > cleaning data to the PoC: DC CVAC. We have wrappers for flushing ranges. > > Anything else is nonsense and does not provide the guarantee you need. > > That said, I still am not sure what guarantee you are attempting to get. > Which data do you need out at the PoC? > I tried flushing the jump addr ########## +static unsigned long jump_addr_save; void soft_restart(unsigned long addr) { typedef void (*phys_reset_t)(unsigned long); phys_reset_t phys_reset; + phys_reset = (phys_reset_t)virt_to_phys(cpu_reset); + jump_addr_save = addr; + __flush_dcache_area(&jump_addr_save, 16); + __flush_dcache_area(&phys_reset, 16); setup_restart(); /* Switch to the identity mapping */ - phys_reset = (phys_reset_t)virt_to_phys(cpu_reset); - phys_reset(addr); + phys_reset(jump_addr_save); /* Should never get here */ BUG(); ########### And flushing all the source and destination pages of the kexeced kernel ########## diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index 2995c78..3edf567 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -221,6 +221,8 @@ static void _kexec_entry_dump(const char *func, int line, addr, (unsigned long)virt_to_phys(dest), dest); + __flush_dcache_area(addr, PAGE_SIZE); + __flush_dcache_area(dest, PAGE_SIZE); dest += PAGE_SIZE; break; case IND_DONE: ########### It still fails(not reboot to kexeced kernel); That means I miss flushing of some other area. --Arun > Thanks, > Mark. > >> >> ################### >> diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c >> index 786daa6..90418f3 100644 >> --- a/arch/arm64/kernel/process.c >> +++ b/arch/arm64/kernel/process.c >> @@ -63,6 +63,10 @@ static inline void smp_secondary_shutdown(void) {} >> >> static void setup_restart(void) >> { >> + volatile u64 tmp; >> + volatile u64 *addr; >> + >> + addr = (u64 *)(0xffffffc000000000); >> /* >> * Tell the mm system that we are going to reboot - >> * we may need it to insert some 1:1 mappings so that >> @@ -75,6 +79,11 @@ static void setup_restart(void) >> /* Clean and invalidate caches */ >> flush_cache_all(); >> >> + for ( ;addr < (u64 *)0xffffffc040000000; addr++) >> + { >> + tmp = *addr; >> + } >> + >> /* Turn D-cache off */ >> cpu_cache_off(); >> >> ################### >> >> With the above change latest kernel @ >> https://git.linaro.org/people/geoff.levand/linux-kexec.git >> is able to do kexec with L3 enabled in UP scenario. >> >> --Arun >> >> >> >> >> >> >> >> >> >> >> >> >> >> > -Geoff >> > >>