[REGRESSION] S3 resume on SandyBridge doesn't work with NX protection (5bd5a45)

[REGRESSION] S3 resume on SandyBridge doesn't work with NX protection (5bd5a45)

[Matthias Hopf]

[-- Attachment #1: Type: text/plain, Size: 786 bytes --]

Git commit 5bd5a45 in Linus' tree "x86: Add NX protection for kernel data"
apparently breaks resume on SandyBridge machines. This is supposed to be
fixed with commit 691513f "x86: Resume trampoline must be executable",
but even git master (v2.6.38-rc1) shows the same symptoms.

After suspend to S3, the machine doesn't resume upon wakeup, but boots
normally instead. Reverting git commits 691513f, 84e1c6b, and especially
5bd5a45 (trivial except for a single hunk) makes resume work again.

lspci -vvv output is attached.


(for internal reference: machine NUE874)

Matthias

-- 
Matthias Hopf <mhopf@suse.de>      __        __   __
Maxfeldstr. 5 / 90409 Nuernberg   (_   | |  (_   |__          mat@mshopf.de
Phone +49-911-74053-715           __)  |_|  __)  |__  R & D   www.mshopf.de

[-- Attachment #2: lspci.txt.gz --]
[-- Type: application/x-gzip, Size: 3328 bytes --]

Re: [REGRESSION] S3 resume on SandyBridge doesn't work with NX protection (5bd5a45)

[H. Peter Anvin]

On 01/19/2011 09:37 AM, Matthias Hopf wrote:
> Git commit 5bd5a45 in Linus' tree "x86: Add NX protection for kernel data"
> apparently breaks resume on SandyBridge machines. This is supposed to be
> fixed with commit 691513f "x86: Resume trampoline must be executable",
> but even git master (v2.6.38-rc1) shows the same symptoms.
> 
> After suspend to S3, the machine doesn't resume upon wakeup, but boots
> normally instead. Reverting git commits 691513f, 84e1c6b, and especially
> 5bd5a45 (trivial except for a single hunk) makes resume work again.
> 
> lspci -vvv output is attached.
> 
> (for internal reference: machine NUE874)
> 

"Sandy Bridge machines" is not likely to be descriptive.  This is much
more likely to be related to a specific motherboard.

	-hpa

-- 
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

Re: [REGRESSION] S3 resume on SandyBridge doesn't work with NX protection (5bd5a45)

[matthieu castet]

Hi,

Le Wed, 19 Jan 2011 18:37:11 +0100,
Matthias Hopf <mhopf@suse.de> a écrit :

> Git commit 5bd5a45 in Linus' tree "x86: Add NX protection for kernel
> data" apparently breaks resume on SandyBridge machines. This is
> supposed to be fixed with commit 691513f "x86: Resume trampoline must
> be executable", but even git master (v2.6.38-rc1) shows the same
> symptoms.
Is it a 32bit or 64 bit kernel ?

Could you post the dmesg of the kernel.


Matthieu

Re: [REGRESSION] S3 resume on SandyBridge doesn't work with NX protection (5bd5a45)

[Matthias Hopf]

[-- Attachment #1: Type: text/plain, Size: 1097 bytes --]

On Jan 19, 11 23:06:02 +0100, matthieu castet wrote:
> Le Wed, 19 Jan 2011 18:37:11 +0100,
> Matthias Hopf <mhopf@suse.de> a écrit :
> 
> > Git commit 5bd5a45 in Linus' tree "x86: Add NX protection for kernel
> > data" apparently breaks resume on SandyBridge machines. This is
> > supposed to be fixed with commit 691513f "x86: Resume trampoline must
> > be executable", but even git master (v2.6.38-rc1) shows the same
> > symptoms.
> Is it a 32bit or 64 bit kernel ?

Sorry, forgot that. It's a 32bit kernel.

> Could you post the dmesg of the kernel.

Attaching dmesg output after a fresh reboot.
Attaching /var/log/messages after the machine rebooted after going to S3.
Attaching a netconsole log of the machine during suspend.

I'm sorry to say that I cannot see any reasonably useful information in
these, though. The machine is a laptop, so no serial console available.

Matthias

-- 
Matthias Hopf <mhopf@suse.de>      __        __   __
Maxfeldstr. 5 / 90409 Nuernberg   (_   | |  (_   |__          mat@mshopf.de
Phone +49-911-74053-715           __)  |_|  __)  |__  R & D   www.mshopf.de

[-- Attachment #2: dmesg.gz --]
[-- Type: application/x-gzip, Size: 15812 bytes --]

[-- Attachment #3: messages.gz --]
[-- Type: application/x-gzip, Size: 2814 bytes --]

[-- Attachment #4: netconsole.log.gz --]
[-- Type: application/x-gzip, Size: 12066 bytes --]

Re: [REGRESSION] S3 resume on SandyBridge doesn't work with NX protection (5bd5a45)

[Matthias Hopf]

On Jan 20, 11 16:05:54 +0100, castet.matthieu@free.fr wrote:
> > > > Git commit 5bd5a45 in Linus' tree "x86: Add NX protection for kernel
> > > > data" apparently breaks resume on SandyBridge machines. This is
> > > > supposed to be fixed with commit 691513f "x86: Resume trampoline must
> > > > be executable", but even git master (v2.6.38-rc1) shows the same
> > > > symptoms.
> > > Is it a 32bit or 64 bit kernel ?
> >
> > Sorry, forgot that. It's a 32bit kernel.
> Ok that explain why 691513f doesn't work
> >
> Does this sequence make the machine reboot ?
> 
> # echo 0 > /sys/devices/system/cpu/cpu1/online
> # echo 1 > /sys/devices/system/cpu/cpu1/online

Nope, it doesn't.

dmesg output:

[ 8172.670422] Unmapping cpu 1 from all nodes
[ 8172.671802] CPU 1 is now offline
[ 8176.572869] Booting Node 0 Processor 1 APIC 0x1
[ 8176.583474] Initializing CPU#1
[ 8176.583803] Mapping cpu 1 to node 0

Looks good to me.
I also tried disabling all but one CPUs (this is a quad core machine),
works as well.


Matthias

-- 
Matthias Hopf <mhopf@suse.de>      __        __   __
Maxfeldstr. 5 / 90409 Nuernberg   (_   | |  (_   |__          mat@mshopf.de
Phone +49-911-74053-715           __)  |_|  __)  |__  R & D   www.mshopf.de

Re: [REGRESSION] S3 resume on SandyBridge doesn't work with NX protection (5bd5a45)

[H. Peter Anvin]

On 01/20/2011 01:18 PM, matthieu castet wrote:
> PS :
> x86 trampoline is a mess. There are many (wakeup, smpboot) and differ between 32 and 64 bit.
> H. Peter Anvin, aren't you rewriting them ?

Yes, working on it, starting with the framework used to install them.

They can't all be unified; they are used for different purposes and have
different requirements (for one thing, 32 vs 64 bits require different
trampoline code), but the install framework patchset is almost done, and
the recent change of keeping the 1:1 pagetable set around is also a
dramatic cleanup.

	-hpa

Re: [REGRESSION] S3 resume on SandyBridge doesn't work with NX protection (5bd5a45)

[matthieu castet]

H. Peter Anvin a écrit :
> On 01/20/2011 01:18 PM, matthieu castet wrote:
>> PS :
>> x86 trampoline is a mess. There are many (wakeup, smpboot) and differ between 32 and 64 bit.
>> H. Peter Anvin, aren't you rewriting them ?
> 
> Yes, working on it, starting with the framework used to install them.
> 
> They can't all be unified; they are used for different purposes and have
> different requirements (for one thing, 32 vs 64 bits require different
> trampoline code), but the install framework patchset is almost done, and
> the recent change of keeping the 1:1 pagetable set around is also a
> dramatic cleanup.
Of course, but for example on x64, using smp trampoline to exit rm wakecode help
to merge code in one place.

Also the start of 32/64 bits trampoline (rm part) is really similar. The main difference is the verify_cpu
check, but can't it be called latter (when we switch to 64 bits ) ?

In the end we could have something like

wakeup_code:
(common rm 16 bit code)
call trampoline

trampoline:
(common rm 16 bit code)
call startup_32_vector (either startup_32_smp for 32 bits (exit trampoline) or startup_32 for 64 bit).

startup_32:

Re: [REGRESSION] S3 resume on SandyBridge doesn't work with NX protection (5bd5a45)

[H. Peter Anvin]

On 01/21/2011 02:17 PM, matthieu castet wrote:
> Of course, but for example on x64, using smp trampoline to exit rm wakecode help
> to merge code in one place.

Yes, and there are numerous cleanups like that that can be done.

> Also the start of 32/64 bits trampoline (rm part) is really similar. The main difference is the verify_cpu
> check, but can't it be called latter (when we switch to 64 bits ) ?

One can *sometimes* do merges like that, but one has to be careful to
not make the code more complex instead of less by mistake.  Minor code
duplication is better than really ugly code.

	-hpa

Re: [REGRESSION] S3 resume on SandyBridge doesn't work with NX protection (5bd5a45)

[castet.matthieu]

Quoting matthieu castet <castet.matthieu@free.fr>:

> Hi,
>
> Le Wed, 19 Jan 2011 18:37:11 +0100,
> Matthias Hopf <mhopf@suse.de> a écrit :
>
> > Git commit 5bd5a45 in Linus' tree "x86: Add NX protection for kernel
> > data" apparently breaks resume on SandyBridge machines. This is
> > supposed to be fixed with commit 691513f "x86: Resume trampoline must
> > be executable", but even git master (v2.6.38-rc1) shows the same
> > symptoms.
> Is it a 32bit or 64 bit kernel ?
>
> Could you post the dmesg of the kernel.
>
could you try the patch attached in
https://bugzilla.kernel.org/show_bug.cgi?id=27472