[question] handle the page table RAS error

[question] handle the page table RAS error

[gengdongjiu]

Hi all,
   Sorry to disturb you. Now the ARM64 has supported the RAS, when enabling this feature, we encounter a issue. If the user space application happen page table RAS error,
Memory error handler(memory_failure()) will do nothing except make a poisoned page flag, and fault handler in arch/arm64/mm/fault.c will deliver a signal to kill this
application. when this application exit, it will call unmap_vmas () to release his vma resource, but here it will touch the error page table again, then will trigger RAS error again, so
this application cannot be killed and system will be panic, the log is shown in [2].

As shown the stack in [1], unmap_page_range() will touch the error page table, so system will panic, does this panic behavior is expected?  How the x86 handle the page table
RAS error? If user space application happen page table RAS error, I think the expected behavior should be killing the application instead of panic OS. In current code, when release 
application vma resource, I do not see it will check whether table page is poisoned, could you give me some suggestion about how to handle this case? Thanks a lot. 

[1]:
get_signal()
   do_group_exit()
      mmput()
		exit_mmap()
			unmap_vmas()
				unmap_single_vma()
					unmap_page_range()
						

[2]
[  676.669053] Synchronous External Abort: level 0 (translation table walk) (0x82000214) at 0x0000000033ff7008
[  676.686469] Memory failure: 0xcd4b: already hardware poisoned 
[  676.700652] Synchronous External Abort: synchronous external abort (0x96000410) at 0x0000000033ff7008
[  676.723301] Internal error: : 96000410 [#1] PREEMPT SMP 
[  676.723616] Modules linked in: inject_memory_error(O)
[  676.724601] CPU: 0 PID: 1506 Comm: mca-recover Tainted: G           O    4.14.0-rc8-00019-g5b5c6f4-dirty #109
[  676.724844] task: ffff80000cd41d00 task.stack: ffff000009b30000 
[  676.726616] PC is at unmap_page_range+0x78/0x6fc 
[  676.726960] LR is at unmap_single_vma+0x88/0xdc 
[  676.727122] pc : [<ffff0000081f109c>] lr : [<ffff0000081f17a8>] pstate: 80400149
[  676.727227] sp : ffff000009b339b0
[  676.727348] x29: ffff000009b339b0 x28: ffff80000cd41d00 
[  676.727653] x27: 0000000000000000 x26: ffff80000cd42410 
[  676.727919] x25: ffff80000cd41d00 x24: ffff80000cd1e180 
[  676.728161] x23: ffff80000ce22300 x22: 0000000000000000 
[  676.728407] x21: ffff000009b33b28 x20: 0000000000400000 
[  676.728642] x19: ffff80000cd1e180 x18: 000000000000016d 
[  676.728875] x17: 0000000000000190 x16: 0000000000000064 
[  676.729117] x15: 0000000000000339 x14: 0000000000000000 
[  676.729344] x13: 00000000000061a8 x12: 0000000000000339 
[  676.729582] x11: 0000000000000018 x10: 0000000000000a80 
[  676.729829] x9 : ffff000009b33c60 x8 : ffff80000cd427e0 
[  676.730065] x7 : ffff000009b33de8 x6 : 00000000004a2000 
[  676.730287] x5 : 0000000000400000 x4 : ffff80000cd4b000 
[  676.730517] x3 : 00000000004a1fff x2 : 0000008000000000 
[  676.730741] x1 : 0000007fffffffff x0 : 0000008000000000 
[  676.731101] Process mca-recover (pid: 1506, stack limit = 0xffff000009b30000) 
[  676.731281] Call trace:
[  676.734196] [<ffff0000081f109c>] unmap_page_range+0x78/0x6fc 
[  676.734539] [<ffff0000081f17a8>] unmap_single_vma+0x88/0xdc 
[  676.734892] [<ffff0000081f1aa8>] unmap_vmas+0x68/0xb4 
[  676.735456] [<ffff0000081fa56c>] exit_mmap+0x90/0x140 
[  676.736468] [<ffff0000080ccb34>] mmput+0x60/0x118 
[  676.736791] [<ffff0000080d4060>] do_exit+0x240/0x9cc 
[  676.736997] [<ffff0000080d4854>] do_group_exit+0x38/0x98 
[  676.737384] [<ffff0000080df4d0>] get_signal+0x1ec/0x548 
[  676.738313] [<ffff000008088b80>] do_signal+0x7c/0x668 
[  676.738617] [<ffff000008089538>] do_notify_resume+0xcc/0x114 
 [  676.740983] [<ffff0000080836c0>] work_pending+0x8/0x10 
[  676.741360] Code: f94043a4 f9404ba2 f94037a3 d1000441 (f9400080) 
[  676.741745] ---[ end trace e42d453027313552 ]--- 
[  676.804174] Fixing recursive fault but reboot is needed!
[  677.462082] Memory failure: 0xcd4b: already hardware poisoned

[question] handle the page table RAS error

[gengdongjiu]

Hi all,
   Sorry to disturb you. Now the ARM64 has supported the RAS, when enabling this feature, we encounter a issue. If the user space application happen page table RAS error,
Memory error handler(memory_failure()) will do nothing except make a poisoned page flag, and fault handler in arch/arm64/mm/fault.c will deliver a signal to kill this
application. when this application exit, it will call unmap_vmas () to release his vma resource, but here it will touch the error page table again, then will trigger RAS error again, so
this application cannot be killed and system will be panic, the log is shown in [2].

As shown the stack in [1], unmap_page_range() will touch the error page table, so system will panic, does this panic behavior is expected?  How the x86 handle the page table
RAS error? If user space application happen page table RAS error, I think the expected behavior should be killing the application instead of panic OS. In current code, when release 
application vma resource, I do not see it will check whether table page is poisoned, could you give me some suggestion about how to handle this case? Thanks a lot. 

[1]:
get_signal()
   do_group_exit()
      mmput()
		exit_mmap()
			unmap_vmas()
				unmap_single_vma()
					unmap_page_range()
						

[2]
[  676.669053] Synchronous External Abort: level 0 (translation table walk) (0x82000214) at 0x0000000033ff7008
[  676.686469] Memory failure: 0xcd4b: already hardware poisoned 
[  676.700652] Synchronous External Abort: synchronous external abort (0x96000410) at 0x0000000033ff7008
[  676.723301] Internal error: : 96000410 [#1] PREEMPT SMP 
[  676.723616] Modules linked in: inject_memory_error(O)
[  676.724601] CPU: 0 PID: 1506 Comm: mca-recover Tainted: G           O    4.14.0-rc8-00019-g5b5c6f4-dirty #109
[  676.724844] task: ffff80000cd41d00 task.stack: ffff000009b30000 
[  676.726616] PC is at unmap_page_range+0x78/0x6fc 
[  676.726960] LR is at unmap_single_vma+0x88/0xdc 
[  676.727122] pc : [<ffff0000081f109c>] lr : [<ffff0000081f17a8>] pstate: 80400149
[  676.727227] sp : ffff000009b339b0
[  676.727348] x29: ffff000009b339b0 x28: ffff80000cd41d00 
[  676.727653] x27: 0000000000000000 x26: ffff80000cd42410 
[  676.727919] x25: ffff80000cd41d00 x24: ffff80000cd1e180 
[  676.728161] x23: ffff80000ce22300 x22: 0000000000000000 
[  676.728407] x21: ffff000009b33b28 x20: 0000000000400000 
[  676.728642] x19: ffff80000cd1e180 x18: 000000000000016d 
[  676.728875] x17: 0000000000000190 x16: 0000000000000064 
[  676.729117] x15: 0000000000000339 x14: 0000000000000000 
[  676.729344] x13: 00000000000061a8 x12: 0000000000000339 
[  676.729582] x11: 0000000000000018 x10: 0000000000000a80 
[  676.729829] x9 : ffff000009b33c60 x8 : ffff80000cd427e0 
[  676.730065] x7 : ffff000009b33de8 x6 : 00000000004a2000 
[  676.730287] x5 : 0000000000400000 x4 : ffff80000cd4b000 
[  676.730517] x3 : 00000000004a1fff x2 : 0000008000000000 
[  676.730741] x1 : 0000007fffffffff x0 : 0000008000000000 
[  676.731101] Process mca-recover (pid: 1506, stack limit = 0xffff000009b30000) 
[  676.731281] Call trace:
[  676.734196] [<ffff0000081f109c>] unmap_page_range+0x78/0x6fc 
[  676.734539] [<ffff0000081f17a8>] unmap_single_vma+0x88/0xdc 
[  676.734892] [<ffff0000081f1aa8>] unmap_vmas+0x68/0xb4 
[  676.735456] [<ffff0000081fa56c>] exit_mmap+0x90/0x140 
[  676.736468] [<ffff0000080ccb34>] mmput+0x60/0x118 
[  676.736791] [<ffff0000080d4060>] do_exit+0x240/0x9cc 
[  676.736997] [<ffff0000080d4854>] do_group_exit+0x38/0x98 
[  676.737384] [<ffff0000080df4d0>] get_signal+0x1ec/0x548 
[  676.738313] [<ffff000008088b80>] do_signal+0x7c/0x668 
[  676.738617] [<ffff000008089538>] do_notify_resume+0xcc/0x114 
 [  676.740983] [<ffff0000080836c0>] work_pending+0x8/0x10 
[  676.741360] Code: f94043a4 f9404ba2 f94037a3 d1000441 (f9400080) 
[  676.741745] ---[ end trace e42d453027313552 ]--- 
[  676.804174] Fixing recursive fault but reboot is needed!
[  677.462082] Memory failure: 0xcd4b: already hardware poisoned

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [question] handle the page table RAS error

[gengdongjiu]

ping again.

Hi Naoya,

   sorry do disturb you again, before application release its vma
resource,  if checking whether the page table is poisoned page,
whether it will be reasonable? or this panic behavior is expected?
look forward to you reply.
thank you so much.


2017-12-03 21:22 GMT+08:00 gengdongjiu <gengdongjiu@huawei.com>:
> Hi all,
>    Sorry to disturb you. Now the ARM64 has supported the RAS, when enabling this feature, we encounter a issue. If the user space application happen page table RAS error,
> Memory error handler(memory_failure()) will do nothing except make a poisoned page flag, and fault handler in arch/arm64/mm/fault.c will deliver a signal to kill this
> application. when this application exit, it will call unmap_vmas () to release his vma resource, but here it will touch the error page table again, then will trigger RAS error again, so
> this application cannot be killed and system will be panic, the log is shown in [2].
>
> As shown the stack in [1], unmap_page_range() will touch the error page table, so system will panic, does this panic behavior is expected?  How the x86 handle the page table
> RAS error? If user space application happen page table RAS error, I think the expected behavior should be killing the application instead of panic OS. In current code, when release
> application vma resource, I do not see it will check whether table page is poisoned, could you give me some suggestion about how to handle this case? Thanks a lot.
>
> [1]:
> get_signal()
>    do_group_exit()
>       mmput()
>                 exit_mmap()
>                         unmap_vmas()
>                                 unmap_single_vma()
>                                         unmap_page_range()
>
>
> [2]
> [  676.669053] Synchronous External Abort: level 0 (translation table walk) (0x82000214) at 0x0000000033ff7008
> [  676.686469] Memory failure: 0xcd4b: already hardware poisoned
> [  676.700652] Synchronous External Abort: synchronous external abort (0x96000410) at 0x0000000033ff7008
> [  676.723301] Internal error: : 96000410 [#1] PREEMPT SMP
> [  676.723616] Modules linked in: inject_memory_error(O)
> [  676.724601] CPU: 0 PID: 1506 Comm: mca-recover Tainted: G           O    4.14.0-rc8-00019-g5b5c6f4-dirty #109
> [  676.724844] task: ffff80000cd41d00 task.stack: ffff000009b30000
> [  676.726616] PC is at unmap_page_range+0x78/0x6fc
> [  676.726960] LR is at unmap_single_vma+0x88/0xdc
> [  676.727122] pc : [<ffff0000081f109c>] lr : [<ffff0000081f17a8>] pstate: 80400149
> [  676.727227] sp : ffff000009b339b0
> [  676.727348] x29: ffff000009b339b0 x28: ffff80000cd41d00
> [  676.727653] x27: 0000000000000000 x26: ffff80000cd42410
> [  676.727919] x25: ffff80000cd41d00 x24: ffff80000cd1e180
> [  676.728161] x23: ffff80000ce22300 x22: 0000000000000000
> [  676.728407] x21: ffff000009b33b28 x20: 0000000000400000
> [  676.728642] x19: ffff80000cd1e180 x18: 000000000000016d
> [  676.728875] x17: 0000000000000190 x16: 0000000000000064
> [  676.729117] x15: 0000000000000339 x14: 0000000000000000
> [  676.729344] x13: 00000000000061a8 x12: 0000000000000339
> [  676.729582] x11: 0000000000000018 x10: 0000000000000a80
> [  676.729829] x9 : ffff000009b33c60 x8 : ffff80000cd427e0
> [  676.730065] x7 : ffff000009b33de8 x6 : 00000000004a2000
> [  676.730287] x5 : 0000000000400000 x4 : ffff80000cd4b000
> [  676.730517] x3 : 00000000004a1fff x2 : 0000008000000000
> [  676.730741] x1 : 0000007fffffffff x0 : 0000008000000000
> [  676.731101] Process mca-recover (pid: 1506, stack limit = 0xffff000009b30000)
> [  676.731281] Call trace:
> [  676.734196] [<ffff0000081f109c>] unmap_page_range+0x78/0x6fc
> [  676.734539] [<ffff0000081f17a8>] unmap_single_vma+0x88/0xdc
> [  676.734892] [<ffff0000081f1aa8>] unmap_vmas+0x68/0xb4
> [  676.735456] [<ffff0000081fa56c>] exit_mmap+0x90/0x140
> [  676.736468] [<ffff0000080ccb34>] mmput+0x60/0x118
> [  676.736791] [<ffff0000080d4060>] do_exit+0x240/0x9cc
> [  676.736997] [<ffff0000080d4854>] do_group_exit+0x38/0x98
> [  676.737384] [<ffff0000080df4d0>] get_signal+0x1ec/0x548
> [  676.738313] [<ffff000008088b80>] do_signal+0x7c/0x668
> [  676.738617] [<ffff000008089538>] do_notify_resume+0xcc/0x114
>  [  676.740983] [<ffff0000080836c0>] work_pending+0x8/0x10
> [  676.741360] Code: f94043a4 f9404ba2 f94037a3 d1000441 (f9400080)
> [  676.741745] ---[ end trace e42d453027313552 ]---
> [  676.804174] Fixing recursive fault but reboot is needed!
> [  677.462082] Memory failure: 0xcd4b: already hardware poisoned
>

Re: [question] handle the page table RAS error

[gengdongjiu]

ping again.

Hi Naoya,

   sorry do disturb you again, before application release its vma
resource,  if checking whether the page table is poisoned page,
whether it will be reasonable? or this panic behavior is expected?
look forward to you reply.
thank you so much.


2017-12-03 21:22 GMT+08:00 gengdongjiu <gengdongjiu@huawei.com>:
> Hi all,
>    Sorry to disturb you. Now the ARM64 has supported the RAS, when enabling this feature, we encounter a issue. If the user space application happen page table RAS error,
> Memory error handler(memory_failure()) will do nothing except make a poisoned page flag, and fault handler in arch/arm64/mm/fault.c will deliver a signal to kill this
> application. when this application exit, it will call unmap_vmas () to release his vma resource, but here it will touch the error page table again, then will trigger RAS error again, so
> this application cannot be killed and system will be panic, the log is shown in [2].
>
> As shown the stack in [1], unmap_page_range() will touch the error page table, so system will panic, does this panic behavior is expected?  How the x86 handle the page table
> RAS error? If user space application happen page table RAS error, I think the expected behavior should be killing the application instead of panic OS. In current code, when release
> application vma resource, I do not see it will check whether table page is poisoned, could you give me some suggestion about how to handle this case? Thanks a lot.
>
> [1]:
> get_signal()
>    do_group_exit()
>       mmput()
>                 exit_mmap()
>                         unmap_vmas()
>                                 unmap_single_vma()
>                                         unmap_page_range()
>
>
> [2]
> [  676.669053] Synchronous External Abort: level 0 (translation table walk) (0x82000214) at 0x0000000033ff7008
> [  676.686469] Memory failure: 0xcd4b: already hardware poisoned
> [  676.700652] Synchronous External Abort: synchronous external abort (0x96000410) at 0x0000000033ff7008
> [  676.723301] Internal error: : 96000410 [#1] PREEMPT SMP
> [  676.723616] Modules linked in: inject_memory_error(O)
> [  676.724601] CPU: 0 PID: 1506 Comm: mca-recover Tainted: G           O    4.14.0-rc8-00019-g5b5c6f4-dirty #109
> [  676.724844] task: ffff80000cd41d00 task.stack: ffff000009b30000
> [  676.726616] PC is at unmap_page_range+0x78/0x6fc
> [  676.726960] LR is at unmap_single_vma+0x88/0xdc
> [  676.727122] pc : [<ffff0000081f109c>] lr : [<ffff0000081f17a8>] pstate: 80400149
> [  676.727227] sp : ffff000009b339b0
> [  676.727348] x29: ffff000009b339b0 x28: ffff80000cd41d00
> [  676.727653] x27: 0000000000000000 x26: ffff80000cd42410
> [  676.727919] x25: ffff80000cd41d00 x24: ffff80000cd1e180
> [  676.728161] x23: ffff80000ce22300 x22: 0000000000000000
> [  676.728407] x21: ffff000009b33b28 x20: 0000000000400000
> [  676.728642] x19: ffff80000cd1e180 x18: 000000000000016d
> [  676.728875] x17: 0000000000000190 x16: 0000000000000064
> [  676.729117] x15: 0000000000000339 x14: 0000000000000000
> [  676.729344] x13: 00000000000061a8 x12: 0000000000000339
> [  676.729582] x11: 0000000000000018 x10: 0000000000000a80
> [  676.729829] x9 : ffff000009b33c60 x8 : ffff80000cd427e0
> [  676.730065] x7 : ffff000009b33de8 x6 : 00000000004a2000
> [  676.730287] x5 : 0000000000400000 x4 : ffff80000cd4b000
> [  676.730517] x3 : 00000000004a1fff x2 : 0000008000000000
> [  676.730741] x1 : 0000007fffffffff x0 : 0000008000000000
> [  676.731101] Process mca-recover (pid: 1506, stack limit = 0xffff000009b30000)
> [  676.731281] Call trace:
> [  676.734196] [<ffff0000081f109c>] unmap_page_range+0x78/0x6fc
> [  676.734539] [<ffff0000081f17a8>] unmap_single_vma+0x88/0xdc
> [  676.734892] [<ffff0000081f1aa8>] unmap_vmas+0x68/0xb4
> [  676.735456] [<ffff0000081fa56c>] exit_mmap+0x90/0x140
> [  676.736468] [<ffff0000080ccb34>] mmput+0x60/0x118
> [  676.736791] [<ffff0000080d4060>] do_exit+0x240/0x9cc
> [  676.736997] [<ffff0000080d4854>] do_group_exit+0x38/0x98
> [  676.737384] [<ffff0000080df4d0>] get_signal+0x1ec/0x548
> [  676.738313] [<ffff000008088b80>] do_signal+0x7c/0x668
> [  676.738617] [<ffff000008089538>] do_notify_resume+0xcc/0x114
>  [  676.740983] [<ffff0000080836c0>] work_pending+0x8/0x10
> [  676.741360] Code: f94043a4 f9404ba2 f94037a3 d1000441 (f9400080)
> [  676.741745] ---[ end trace e42d453027313552 ]---
> [  676.804174] Fixing recursive fault but reboot is needed!
> [  677.462082] Memory failure: 0xcd4b: already hardware poisoned
>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [question] handle the page table RAS error

[Andi Kleen]

On Sun, Dec 03, 2017 at 01:22:25PM +0000, gengdongjiu wrote:
> Hi all,
>    Sorry to disturb you. Now the ARM64 has supported the RAS, when enabling this feature, we encounter a issue. If the user space application happen page table RAS error,
> Memory error handler(memory_failure()) will do nothing except make a poisoned page flag, and fault handler in arch/arm64/mm/fault.c will deliver a signal to kill this
> application. when this application exit, it will call unmap_vmas () to release his vma resource, but here it will touch the error page table again, then will trigger RAS error again, so
> this application cannot be killed and system will be panic, the log is shown in [2].
> 
> As shown the stack in [1], unmap_page_range() will touch the error page table, so system will panic, does this panic behavior is expected?  How the x86 handle the page table
> RAS error? If user space application happen page table RAS error, I think the expected behavior should be killing the application instead of panic OS. In current code, when release 
> application vma resource, I do not see it will check whether table page is poisoned, could you give me some suggestion about how to handle this case? Thanks a lot. 

x86 doesn't handle it.

There are lots of memory types that are not handled by MCE recovery
because it is just too difficult.  In general MCE recovery focuses on
memory types that use up significant percent of total memory.  Page tables
are normally not that big, so not really worth handling.

I wouldn't bother about them unless you measure them to big a significant
portion of memory on a real world workload.

-Andi

Re: [question] handle the page table RAS error

[Andi Kleen]

On Sun, Dec 03, 2017 at 01:22:25PM +0000, gengdongjiu wrote:
> Hi all,
>    Sorry to disturb you. Now the ARM64 has supported the RAS, when enabling this feature, we encounter a issue. If the user space application happen page table RAS error,
> Memory error handler(memory_failure()) will do nothing except make a poisoned page flag, and fault handler in arch/arm64/mm/fault.c will deliver a signal to kill this
> application. when this application exit, it will call unmap_vmas () to release his vma resource, but here it will touch the error page table again, then will trigger RAS error again, so
> this application cannot be killed and system will be panic, the log is shown in [2].
> 
> As shown the stack in [1], unmap_page_range() will touch the error page table, so system will panic, does this panic behavior is expected?  How the x86 handle the page table
> RAS error? If user space application happen page table RAS error, I think the expected behavior should be killing the application instead of panic OS. In current code, when release 
> application vma resource, I do not see it will check whether table page is poisoned, could you give me some suggestion about how to handle this case? Thanks a lot. 

x86 doesn't handle it.

There are lots of memory types that are not handled by MCE recovery
because it is just too difficult.  In general MCE recovery focuses on
memory types that use up significant percent of total memory.  Page tables
are normally not that big, so not really worth handling.

I wouldn't bother about them unless you measure them to big a significant
portion of memory on a real world workload.

-Andi

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [question] handle the page table RAS error

[gengdongjiu]

On 2017/12/6 0:57, Andi Kleen wrote:
> x86 doesn't handle it.
> 
> There are lots of memory types that are not handled by MCE recovery
> because it is just too difficult.  In general MCE recovery focuses on
> memory types that use up significant percent of total memory.  Page tables
> are normally not that big, so not really worth handling.
> 
> I wouldn't bother about them unless you measure them to big a significant
> portion of memory on a real world workload.

Thanks for the reply and answer.
sorry, I need to explain my main purpose.
In fact, I mainly want to avoid kernel crash by reading the corrupt page table during application "exit",
not  want to make a very complicated solution to handle the page table RAS error. may be a user space
application error lead to whole OS panic is not a good.

This is the real case that I encountered when "kill" the application, the log is shown in [1].
do you think we needn't to handle this kernel panic when killing a application?

May be the simplest way is push the task to dead state when found his page table is poisoned, or not free the
poisoned page table, for this way, of course there will be a memory leak because the kernel relies on looking at which pages
were mapped to go and reduce the reference count and (if zero) free the page




[1]:
[  676.669053] Synchronous External Abort: level 0 (translation table walk) (0x82000214) at 0x0000000033ff7008
[  676.686469] Memory failure: 0xcd4b: already hardware poisoned
[  676.700652] Synchronous External Abort: synchronous external abort (0x96000410) at 0x0000000033ff7008
[  676.723301] Internal error: : 96000410 [#1] PREEMPT SMP
[  676.723616] Modules linked in: inject_memory_error(O)
[  676.724601] CPU: 0 PID: 1506 Comm: mca-recover Tainted: G           O    4.14.0-rc8-00019-g5b5c6f4-dirty #109
[  676.724844] task: ffff80000cd41d00 task.stack: ffff000009b30000
[  676.726616] PC is at unmap_page_range+0x78/0x6fc
[  676.726960] LR is at unmap_single_vma+0x88/0xdc
[  676.727122] pc : [<ffff0000081f109c>] lr : [<ffff0000081f17a8>] pstate: 80400149
[  676.727227] sp : ffff000009b339b0
[  676.727348] x29: ffff000009b339b0 x28: ffff80000cd41d00
[  676.727653] x27: 0000000000000000 x26: ffff80000cd42410
[  676.727919] x25: ffff80000cd41d00 x24: ffff80000cd1e180
[  676.728161] x23: ffff80000ce22300 x22: 0000000000000000
[  676.728407] x21: ffff000009b33b28 x20: 0000000000400000
[  676.728642] x19: ffff80000cd1e180 x18: 000000000000016d
[  676.728875] x17: 0000000000000190 x16: 0000000000000064
[  676.729117] x15: 0000000000000339 x14: 0000000000000000
[  676.729344] x13: 00000000000061a8 x12: 0000000000000339
[  676.729582] x11: 0000000000000018 x10: 0000000000000a80
[  676.729829] x9 : ffff000009b33c60 x8 : ffff80000cd427e0
[  676.730065] x7 : ffff000009b33de8 x6 : 00000000004a2000
[  676.730287] x5 : 0000000000400000 x4 : ffff80000cd4b000
[  676.730517] x3 : 00000000004a1fff x2 : 0000008000000000
[  676.730741] x1 : 0000007fffffffff x0 : 0000008000000000
[  676.731101] Process mca-recover (pid: 1506, stack limit = 0xffff000009b30000)
[  676.731281] Call trace:
[  676.734196] [<ffff0000081f109c>] unmap_page_range+0x78/0x6fc
[  676.734539] [<ffff0000081f17a8>] unmap_single_vma+0x88/0xdc
[  676.734892] [<ffff0000081f1aa8>] unmap_vmas+0x68/0xb4
[  676.735456] [<ffff0000081fa56c>] exit_mmap+0x90/0x140
[  676.736468] [<ffff0000080ccb34>] mmput+0x60/0x118
[  676.736791] [<ffff0000080d4060>] do_exit+0x240/0x9cc
[  676.736997] [<ffff0000080d4854>] do_group_exit+0x38/0x98
[  676.737384] [<ffff0000080df4d0>] get_signal+0x1ec/0x548
[  676.738313] [<ffff000008088b80>] do_signal+0x7c/0x668
[  676.738617] [<ffff000008089538>] do_notify_resume+0xcc/0x114
 [  676.740983] [<ffff0000080836c0>] work_pending+0x8/0x10
[  676.741360] Code: f94043a4 f9404ba2 f94037a3 d1000441 (f9400080)
[  676.741745] ---[ end trace e42d453027313552 ]---
[  676.804174] Fixing recursive fault but reboot is needed!

Re: [question] handle the page table RAS error

[gengdongjiu]

On 2017/12/6 0:57, Andi Kleen wrote:
> x86 doesn't handle it.
> 
> There are lots of memory types that are not handled by MCE recovery
> because it is just too difficult.  In general MCE recovery focuses on
> memory types that use up significant percent of total memory.  Page tables
> are normally not that big, so not really worth handling.
> 
> I wouldn't bother about them unless you measure them to big a significant
> portion of memory on a real world workload.

Thanks for the reply and answer.
sorry, I need to explain my main purpose.
In fact, I mainly want to avoid kernel crash by reading the corrupt page table during application "exit",
not  want to make a very complicated solution to handle the page table RAS error. may be a user space
application error lead to whole OS panic is not a good.

This is the real case that I encountered when "kill" the application, the log is shown in [1].
do you think we needn't to handle this kernel panic when killing a application?

May be the simplest way is push the task to dead state when found his page table is poisoned, or not free the
poisoned page table, for this way, of course there will be a memory leak because the kernel relies on looking at which pages
were mapped to go and reduce the reference count and (if zero) free the page




[1]:
[  676.669053] Synchronous External Abort: level 0 (translation table walk) (0x82000214) at 0x0000000033ff7008
[  676.686469] Memory failure: 0xcd4b: already hardware poisoned
[  676.700652] Synchronous External Abort: synchronous external abort (0x96000410) at 0x0000000033ff7008
[  676.723301] Internal error: : 96000410 [#1] PREEMPT SMP
[  676.723616] Modules linked in: inject_memory_error(O)
[  676.724601] CPU: 0 PID: 1506 Comm: mca-recover Tainted: G           O    4.14.0-rc8-00019-g5b5c6f4-dirty #109
[  676.724844] task: ffff80000cd41d00 task.stack: ffff000009b30000
[  676.726616] PC is at unmap_page_range+0x78/0x6fc
[  676.726960] LR is at unmap_single_vma+0x88/0xdc
[  676.727122] pc : [<ffff0000081f109c>] lr : [<ffff0000081f17a8>] pstate: 80400149
[  676.727227] sp : ffff000009b339b0
[  676.727348] x29: ffff000009b339b0 x28: ffff80000cd41d00
[  676.727653] x27: 0000000000000000 x26: ffff80000cd42410
[  676.727919] x25: ffff80000cd41d00 x24: ffff80000cd1e180
[  676.728161] x23: ffff80000ce22300 x22: 0000000000000000
[  676.728407] x21: ffff000009b33b28 x20: 0000000000400000
[  676.728642] x19: ffff80000cd1e180 x18: 000000000000016d
[  676.728875] x17: 0000000000000190 x16: 0000000000000064
[  676.729117] x15: 0000000000000339 x14: 0000000000000000
[  676.729344] x13: 00000000000061a8 x12: 0000000000000339
[  676.729582] x11: 0000000000000018 x10: 0000000000000a80
[  676.729829] x9 : ffff000009b33c60 x8 : ffff80000cd427e0
[  676.730065] x7 : ffff000009b33de8 x6 : 00000000004a2000
[  676.730287] x5 : 0000000000400000 x4 : ffff80000cd4b000
[  676.730517] x3 : 00000000004a1fff x2 : 0000008000000000
[  676.730741] x1 : 0000007fffffffff x0 : 0000008000000000
[  676.731101] Process mca-recover (pid: 1506, stack limit = 0xffff000009b30000)
[  676.731281] Call trace:
[  676.734196] [<ffff0000081f109c>] unmap_page_range+0x78/0x6fc
[  676.734539] [<ffff0000081f17a8>] unmap_single_vma+0x88/0xdc
[  676.734892] [<ffff0000081f1aa8>] unmap_vmas+0x68/0xb4
[  676.735456] [<ffff0000081fa56c>] exit_mmap+0x90/0x140
[  676.736468] [<ffff0000080ccb34>] mmput+0x60/0x118
[  676.736791] [<ffff0000080d4060>] do_exit+0x240/0x9cc
[  676.736997] [<ffff0000080d4854>] do_group_exit+0x38/0x98
[  676.737384] [<ffff0000080df4d0>] get_signal+0x1ec/0x548
[  676.738313] [<ffff000008088b80>] do_signal+0x7c/0x668
[  676.738617] [<ffff000008089538>] do_notify_resume+0xcc/0x114
 [  676.740983] [<ffff0000080836c0>] work_pending+0x8/0x10
[  676.741360] Code: f94043a4 f9404ba2 f94037a3 d1000441 (f9400080)
[  676.741745] ---[ end trace e42d453027313552 ]---
[  676.804174] Fixing recursive fault but reboot is needed!

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [question] handle the page table RAS error(Avoid kernel panic when killing an application)

[gengdongjiu]

change the mail subject and resend the mail

On 2017/12/6 16:56, gengdongjiu wrote:
> 
> On 2017/12/6 0:57, Andi Kleen wrote:
> x86 doesn't handle it.
>
> There are lots of memory types that are not handled by MCE recovery
> because it is just too difficult.  In general MCE recovery focuses on
> memory types that use up significant percent of total memory.  Page tables
> are normally not that big, so not really worth handling.
>
> I wouldn't bother about them unless you measure them to big a significant
> portion of memory on a real world workload.

Thanks for the reply and answer.
sorry, I need to explain my main purpose.
In fact, I mainly want to avoid kernel crash by reading the corrupt page table during application "exit",
not  want to make a very complicated solution to handle the page table RAS error. may be a user space
application error lead to whole OS panic is not a good.

This is the real case that I encountered when "kill" the application, the log is shown in [1].
do you think we needn't to handle this kernel panic when killing a application?

May be the simplest way is push the task to dead state when found his page table is poisoned, or not free the
poisoned page table, for this way, of course there will be a memory leak because the kernel relies on looking at which pages
were mapped to go and reduce the reference count and (if zero) free the page




[1]:
[  676.669053] Synchronous External Abort: level 0 (translation table walk) (0x82000214) at 0x0000000033ff7008
[  676.686469] Memory failure: 0xcd4b: already hardware poisoned
[  676.700652] Synchronous External Abort: synchronous external abort (0x96000410) at 0x0000000033ff7008
[  676.723301] Internal error: : 96000410 [#1] PREEMPT SMP
[  676.723616] Modules linked in: inject_memory_error(O)
[  676.724601] CPU: 0 PID: 1506 Comm: mca-recover Tainted: G           O    4.14.0-rc8-00019-g5b5c6f4-dirty #109
[  676.724844] task: ffff80000cd41d00 task.stack: ffff000009b30000
[  676.726616] PC is at unmap_page_range+0x78/0x6fc
[  676.726960] LR is at unmap_single_vma+0x88/0xdc
[  676.727122] pc : [<ffff0000081f109c>] lr : [<ffff0000081f17a8>] pstate: 80400149
[  676.727227] sp : ffff000009b339b0
[  676.727348] x29: ffff000009b339b0 x28: ffff80000cd41d00
[  676.727653] x27: 0000000000000000 x26: ffff80000cd42410
[  676.727919] x25: ffff80000cd41d00 x24: ffff80000cd1e180
[  676.728161] x23: ffff80000ce22300 x22: 0000000000000000
[  676.728407] x21: ffff000009b33b28 x20: 0000000000400000
[  676.728642] x19: ffff80000cd1e180 x18: 000000000000016d
[  676.728875] x17: 0000000000000190 x16: 0000000000000064
[  676.729117] x15: 0000000000000339 x14: 0000000000000000
[  676.729344] x13: 00000000000061a8 x12: 0000000000000339
[  676.729582] x11: 0000000000000018 x10: 0000000000000a80
[  676.729829] x9 : ffff000009b33c60 x8 : ffff80000cd427e0
[  676.730065] x7 : ffff000009b33de8 x6 : 00000000004a2000
[  676.730287] x5 : 0000000000400000 x4 : ffff80000cd4b000
[  676.730517] x3 : 00000000004a1fff x2 : 0000008000000000
[  676.730741] x1 : 0000007fffffffff x0 : 0000008000000000
[  676.731101] Process mca-recover (pid: 1506, stack limit = 0xffff000009b30000)
[  676.731281] Call trace:
[  676.734196] [<ffff0000081f109c>] unmap_page_range+0x78/0x6fc
[  676.734539] [<ffff0000081f17a8>] unmap_single_vma+0x88/0xdc
[  676.734892] [<ffff0000081f1aa8>] unmap_vmas+0x68/0xb4
[  676.735456] [<ffff0000081fa56c>] exit_mmap+0x90/0x140
[  676.736468] [<ffff0000080ccb34>] mmput+0x60/0x118
[  676.736791] [<ffff0000080d4060>] do_exit+0x240/0x9cc
[  676.736997] [<ffff0000080d4854>] do_group_exit+0x38/0x98
[  676.737384] [<ffff0000080df4d0>] get_signal+0x1ec/0x548
[  676.738313] [<ffff000008088b80>] do_signal+0x7c/0x668
[  676.738617] [<ffff000008089538>] do_notify_resume+0xcc/0x114
 [  676.740983] [<ffff0000080836c0>] work_pending+0x8/0x10
[  676.741360] Code: f94043a4 f9404ba2 f94037a3 d1000441 (f9400080)
[  676.741745] ---[ end trace e42d453027313552 ]---
[  676.804174] Fixing recursive fault but reboot is needed!

Re: [question] handle the page table RAS error(Avoid kernel panic when killing an application)

[gengdongjiu]

change the mail subject and resend the mail

On 2017/12/6 16:56, gengdongjiu wrote:
> 
> On 2017/12/6 0:57, Andi Kleen wrote:
> x86 doesn't handle it.
>
> There are lots of memory types that are not handled by MCE recovery
> because it is just too difficult.  In general MCE recovery focuses on
> memory types that use up significant percent of total memory.  Page tables
> are normally not that big, so not really worth handling.
>
> I wouldn't bother about them unless you measure them to big a significant
> portion of memory on a real world workload.

Thanks for the reply and answer.
sorry, I need to explain my main purpose.
In fact, I mainly want to avoid kernel crash by reading the corrupt page table during application "exit",
not  want to make a very complicated solution to handle the page table RAS error. may be a user space
application error lead to whole OS panic is not a good.

This is the real case that I encountered when "kill" the application, the log is shown in [1].
do you think we needn't to handle this kernel panic when killing a application?

May be the simplest way is push the task to dead state when found his page table is poisoned, or not free the
poisoned page table, for this way, of course there will be a memory leak because the kernel relies on looking at which pages
were mapped to go and reduce the reference count and (if zero) free the page




[1]:
[  676.669053] Synchronous External Abort: level 0 (translation table walk) (0x82000214) at 0x0000000033ff7008
[  676.686469] Memory failure: 0xcd4b: already hardware poisoned
[  676.700652] Synchronous External Abort: synchronous external abort (0x96000410) at 0x0000000033ff7008
[  676.723301] Internal error: : 96000410 [#1] PREEMPT SMP
[  676.723616] Modules linked in: inject_memory_error(O)
[  676.724601] CPU: 0 PID: 1506 Comm: mca-recover Tainted: G           O    4.14.0-rc8-00019-g5b5c6f4-dirty #109
[  676.724844] task: ffff80000cd41d00 task.stack: ffff000009b30000
[  676.726616] PC is at unmap_page_range+0x78/0x6fc
[  676.726960] LR is at unmap_single_vma+0x88/0xdc
[  676.727122] pc : [<ffff0000081f109c>] lr : [<ffff0000081f17a8>] pstate: 80400149
[  676.727227] sp : ffff000009b339b0
[  676.727348] x29: ffff000009b339b0 x28: ffff80000cd41d00
[  676.727653] x27: 0000000000000000 x26: ffff80000cd42410
[  676.727919] x25: ffff80000cd41d00 x24: ffff80000cd1e180
[  676.728161] x23: ffff80000ce22300 x22: 0000000000000000
[  676.728407] x21: ffff000009b33b28 x20: 0000000000400000
[  676.728642] x19: ffff80000cd1e180 x18: 000000000000016d
[  676.728875] x17: 0000000000000190 x16: 0000000000000064
[  676.729117] x15: 0000000000000339 x14: 0000000000000000
[  676.729344] x13: 00000000000061a8 x12: 0000000000000339
[  676.729582] x11: 0000000000000018 x10: 0000000000000a80
[  676.729829] x9 : ffff000009b33c60 x8 : ffff80000cd427e0
[  676.730065] x7 : ffff000009b33de8 x6 : 00000000004a2000
[  676.730287] x5 : 0000000000400000 x4 : ffff80000cd4b000
[  676.730517] x3 : 00000000004a1fff x2 : 0000008000000000
[  676.730741] x1 : 0000007fffffffff x0 : 0000008000000000
[  676.731101] Process mca-recover (pid: 1506, stack limit = 0xffff000009b30000)
[  676.731281] Call trace:
[  676.734196] [<ffff0000081f109c>] unmap_page_range+0x78/0x6fc
[  676.734539] [<ffff0000081f17a8>] unmap_single_vma+0x88/0xdc
[  676.734892] [<ffff0000081f1aa8>] unmap_vmas+0x68/0xb4
[  676.735456] [<ffff0000081fa56c>] exit_mmap+0x90/0x140
[  676.736468] [<ffff0000080ccb34>] mmput+0x60/0x118
[  676.736791] [<ffff0000080d4060>] do_exit+0x240/0x9cc
[  676.736997] [<ffff0000080d4854>] do_group_exit+0x38/0x98
[  676.737384] [<ffff0000080df4d0>] get_signal+0x1ec/0x548
[  676.738313] [<ffff000008088b80>] do_signal+0x7c/0x668
[  676.738617] [<ffff000008089538>] do_notify_resume+0xcc/0x114
 [  676.740983] [<ffff0000080836c0>] work_pending+0x8/0x10
[  676.741360] Code: f94043a4 f9404ba2 f94037a3 d1000441 (f9400080)
[  676.741745] ---[ end trace e42d453027313552 ]---
[  676.804174] Fixing recursive fault but reboot is needed!

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[consult the suggestion]: Avoid kernel panic when killing an application if happen RAS page table error

[gengdongjiu]

Hi James/All,
  If the user space application happen page table RAS error,Memory error handler(memory_failure()) will do nothing except making a poisoned page flag, and fault handler in arch/arm64/mm/fault.c
will deliver a signal to kill this application. when this application exits, it will call unmap_vmas () to release his vma resource, but here it will touch the error page table again, then will
trigger RAS error again, so this application cannot be killed and system will be panic, the log is shown in [2].

As shown the stack in [1], unmap_page_range() will touch the error page table, so system will panic, there are some simple way to avoid this panic and avoid change much about the memory management.
1. put the tasks to dead status, not run it again.
2. not release the page table for this task.

Of cause, above methods may happen memory leakage. do you have good suggestion about how to solve it?, or do you think this panic is expected behavior? thanks.


[1]:
get_signal()
   do_group_exit()
      mmput()
                exit_mmap()
                        unmap_vmas()
                                unmap_single_vma()
                                        unmap_page_range()


[2]
[  676.669053] Synchronous External Abort: level 0 (translation table walk) (0x82000214) at 0x0000000033ff7008
[  676.686469] Memory failure: 0xcd4b: already hardware poisoned
[  676.700652] Synchronous External Abort: synchronous external abort (0x96000410) at 0x0000000033ff7008
[  676.723301] Internal error: : 96000410 [#1] PREEMPT SMP
[  676.723616] Modules linked in: inject_memory_error(O)
[  676.724601] CPU: 0 PID: 1506 Comm: mca-recover Tainted: G           O    4.14.0-rc8-00019-g5b5c6f4-dirty #109
[  676.724844] task: ffff80000cd41d00 task.stack: ffff000009b30000
[  676.726616] PC is at unmap_page_range+0x78/0x6fc
[  676.726960] LR is at unmap_single_vma+0x88/0xdc
[  676.727122] pc : [<ffff0000081f109c>] lr : [<ffff0000081f17a8>] pstate: 80400149
[  676.727227] sp : ffff000009b339b0
[  676.727348] x29: ffff000009b339b0 x28: ffff80000cd41d00
[  676.727653] x27: 0000000000000000 x26: ffff80000cd42410
[  676.727919] x25: ffff80000cd41d00 x24: ffff80000cd1e180
[  676.728161] x23: ffff80000ce22300 x22: 0000000000000000
[  676.728407] x21: ffff000009b33b28 x20: 0000000000400000
[  676.728642] x19: ffff80000cd1e180 x18: 000000000000016d
[  676.728875] x17: 0000000000000190 x16: 0000000000000064
[  676.729117] x15: 0000000000000339 x14: 0000000000000000
[  676.729344] x13: 00000000000061a8 x12: 0000000000000339
[  676.729582] x11: 0000000000000018 x10: 0000000000000a80
[  676.729829] x9 : ffff000009b33c60 x8 : ffff80000cd427e0
[  676.730065] x7 : ffff000009b33de8 x6 : 00000000004a2000
[  676.730287] x5 : 0000000000400000 x4 : ffff80000cd4b000
[  676.730517] x3 : 00000000004a1fff x2 : 0000008000000000
[  676.730741] x1 : 0000007fffffffff x0 : 0000008000000000
[  676.731101] Process mca-recover (pid: 1506, stack limit = 0xffff000009b30000)
[  676.731281] Call trace:
[  676.734196] [<ffff0000081f109c>] unmap_page_range+0x78/0x6fc
[  676.734539] [<ffff0000081f17a8>] unmap_single_vma+0x88/0xdc
[  676.734892] [<ffff0000081f1aa8>] unmap_vmas+0x68/0xb4
[  676.735456] [<ffff0000081fa56c>] exit_mmap+0x90/0x140
[  676.736468] [<ffff0000080ccb34>] mmput+0x60/0x118
[  676.736791] [<ffff0000080d4060>] do_exit+0x240/0x9cc
[  676.736997] [<ffff0000080d4854>] do_group_exit+0x38/0x98
[  676.737384] [<ffff0000080df4d0>] get_signal+0x1ec/0x548
[  676.738313] [<ffff000008088b80>] do_signal+0x7c/0x668
[  676.738617] [<ffff000008089538>] do_notify_resume+0xcc/0x114
 [  676.740983] [<ffff0000080836c0>] work_pending+0x8/0x10
[  676.741360] Code: f94043a4 f9404ba2 f94037a3 d1000441 (f9400080)
[  676.741745] ---[ end trace e42d453027313552 ]---
[  676.804174] Fixing recursive fault but reboot is needed!
[  677.462082] Memory failure: 0xcd4b: already hardware poisoned

[consult the suggestion]: Avoid kernel panic when killing an application if happen RAS page table error

[gengdongjiu]

Hi James/All,
  If the user space application happen page table RAS error,Memory error handler(memory_failure()) will do nothing except making a poisoned page flag, and fault handler in arch/arm64/mm/fault.c
will deliver a signal to kill this application. when this application exits, it will call unmap_vmas () to release his vma resource, but here it will touch the error page table again, then will
trigger RAS error again, so this application cannot be killed and system will be panic, the log is shown in [2].

As shown the stack in [1], unmap_page_range() will touch the error page table, so system will panic, there are some simple way to avoid this panic and avoid change much about the memory management.
1. put the tasks to dead status, not run it again.
2. not release the page table for this task.

Of cause, above methods may happen memory leakage. do you have good suggestion about how to solve it?, or do you think this panic is expected behavior? thanks.


[1]:
get_signal()
   do_group_exit()
      mmput()
                exit_mmap()
                        unmap_vmas()
                                unmap_single_vma()
                                        unmap_page_range()


[2]
[  676.669053] Synchronous External Abort: level 0 (translation table walk) (0x82000214) at 0x0000000033ff7008
[  676.686469] Memory failure: 0xcd4b: already hardware poisoned
[  676.700652] Synchronous External Abort: synchronous external abort (0x96000410) at 0x0000000033ff7008
[  676.723301] Internal error: : 96000410 [#1] PREEMPT SMP
[  676.723616] Modules linked in: inject_memory_error(O)
[  676.724601] CPU: 0 PID: 1506 Comm: mca-recover Tainted: G           O    4.14.0-rc8-00019-g5b5c6f4-dirty #109
[  676.724844] task: ffff80000cd41d00 task.stack: ffff000009b30000
[  676.726616] PC is at unmap_page_range+0x78/0x6fc
[  676.726960] LR is at unmap_single_vma+0x88/0xdc
[  676.727122] pc : [<ffff0000081f109c>] lr : [<ffff0000081f17a8>] pstate: 80400149
[  676.727227] sp : ffff000009b339b0
[  676.727348] x29: ffff000009b339b0 x28: ffff80000cd41d00
[  676.727653] x27: 0000000000000000 x26: ffff80000cd42410
[  676.727919] x25: ffff80000cd41d00 x24: ffff80000cd1e180
[  676.728161] x23: ffff80000ce22300 x22: 0000000000000000
[  676.728407] x21: ffff000009b33b28 x20: 0000000000400000
[  676.728642] x19: ffff80000cd1e180 x18: 000000000000016d
[  676.728875] x17: 0000000000000190 x16: 0000000000000064
[  676.729117] x15: 0000000000000339 x14: 0000000000000000
[  676.729344] x13: 00000000000061a8 x12: 0000000000000339
[  676.729582] x11: 0000000000000018 x10: 0000000000000a80
[  676.729829] x9 : ffff000009b33c60 x8 : ffff80000cd427e0
[  676.730065] x7 : ffff000009b33de8 x6 : 00000000004a2000
[  676.730287] x5 : 0000000000400000 x4 : ffff80000cd4b000
[  676.730517] x3 : 00000000004a1fff x2 : 0000008000000000
[  676.730741] x1 : 0000007fffffffff x0 : 0000008000000000
[  676.731101] Process mca-recover (pid: 1506, stack limit = 0xffff000009b30000)
[  676.731281] Call trace:
[  676.734196] [<ffff0000081f109c>] unmap_page_range+0x78/0x6fc
[  676.734539] [<ffff0000081f17a8>] unmap_single_vma+0x88/0xdc
[  676.734892] [<ffff0000081f1aa8>] unmap_vmas+0x68/0xb4
[  676.735456] [<ffff0000081fa56c>] exit_mmap+0x90/0x140
[  676.736468] [<ffff0000080ccb34>] mmput+0x60/0x118
[  676.736791] [<ffff0000080d4060>] do_exit+0x240/0x9cc
[  676.736997] [<ffff0000080d4854>] do_group_exit+0x38/0x98
[  676.737384] [<ffff0000080df4d0>] get_signal+0x1ec/0x548
[  676.738313] [<ffff000008088b80>] do_signal+0x7c/0x668
[  676.738617] [<ffff000008089538>] do_notify_resume+0xcc/0x114
 [  676.740983] [<ffff0000080836c0>] work_pending+0x8/0x10
[  676.741360] Code: f94043a4 f9404ba2 f94037a3 d1000441 (f9400080)
[  676.741745] ---[ end trace e42d453027313552 ]---
[  676.804174] Fixing recursive fault but reboot is needed!
[  677.462082] Memory failure: 0xcd4b: already hardware poisoned

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[Question ]: Avoid kernel panic when killing an application if happen RAS page table error

[gengdongjiu]

change the mail title and resend.

Hi James/All,
  If the user space application happen page table RAS error,Memory error handler(memory_failure()) will do nothing except making a poisoned page flag, and fault handler in arch/arm64/mm/fault.c
will deliver a signal to kill this application. when this application exits, it will call unmap_vmas () to release his vma resource, but here it will touch the error page table again, then will
trigger RAS error again, so this application cannot be killed and system will be panic, the log is shown in [2].

As shown the stack in [1], unmap_page_range() will touch the error page table, so system will panic, there are some simple way to avoid this panic and avoid change much about the memory management.
1. put the tasks to dead status, not run it again.
2. not release the page table for this task.

Of cause, above methods may happen memory leakage. do you have good suggestion about how to solve it?, or do you think this panic is expected behavior? thanks.


[1]:
get_signal()
   do_group_exit()
      mmput()
                exit_mmap()
                        unmap_vmas()
                                unmap_single_vma()
                                        unmap_page_range()


[2]
[  676.669053] Synchronous External Abort: level 0 (translation table walk) (0x82000214) at 0x0000000033ff7008
[  676.686469] Memory failure: 0xcd4b: already hardware poisoned
[  676.700652] Synchronous External Abort: synchronous external abort (0x96000410) at 0x0000000033ff7008
[  676.723301] Internal error: : 96000410 [#1] PREEMPT SMP
[  676.723616] Modules linked in: inject_memory_error(O)
[  676.724601] CPU: 0 PID: 1506 Comm: mca-recover Tainted: G           O    4.14.0-rc8-00019-g5b5c6f4-dirty #109
[  676.724844] task: ffff80000cd41d00 task.stack: ffff000009b30000
[  676.726616] PC is at unmap_page_range+0x78/0x6fc
[  676.726960] LR is at unmap_single_vma+0x88/0xdc
[  676.727122] pc : [<ffff0000081f109c>] lr : [<ffff0000081f17a8>] pstate: 80400149
[  676.727227] sp : ffff000009b339b0
[  676.727348] x29: ffff000009b339b0 x28: ffff80000cd41d00
[  676.727653] x27: 0000000000000000 x26: ffff80000cd42410
[  676.727919] x25: ffff80000cd41d00 x24: ffff80000cd1e180
[  676.728161] x23: ffff80000ce22300 x22: 0000000000000000
[  676.728407] x21: ffff000009b33b28 x20: 0000000000400000
[  676.728642] x19: ffff80000cd1e180 x18: 000000000000016d
[  676.728875] x17: 0000000000000190 x16: 0000000000000064
[  676.729117] x15: 0000000000000339 x14: 0000000000000000
[  676.729344] x13: 00000000000061a8 x12: 0000000000000339
[  676.729582] x11: 0000000000000018 x10: 0000000000000a80
[  676.729829] x9 : ffff000009b33c60 x8 : ffff80000cd427e0
[  676.730065] x7 : ffff000009b33de8 x6 : 00000000004a2000
[  676.730287] x5 : 0000000000400000 x4 : ffff80000cd4b000
[  676.730517] x3 : 00000000004a1fff x2 : 0000008000000000
[  676.730741] x1 : 0000007fffffffff x0 : 0000008000000000
[  676.731101] Process mca-recover (pid: 1506, stack limit = 0xffff000009b30000)
[  676.731281] Call trace:
[  676.734196] [<ffff0000081f109c>] unmap_page_range+0x78/0x6fc
[  676.734539] [<ffff0000081f17a8>] unmap_single_vma+0x88/0xdc
[  676.734892] [<ffff0000081f1aa8>] unmap_vmas+0x68/0xb4
[  676.735456] [<ffff0000081fa56c>] exit_mmap+0x90/0x140
[  676.736468] [<ffff0000080ccb34>] mmput+0x60/0x118
[  676.736791] [<ffff0000080d4060>] do_exit+0x240/0x9cc
[  676.736997] [<ffff0000080d4854>] do_group_exit+0x38/0x98
[  676.737384] [<ffff0000080df4d0>] get_signal+0x1ec/0x548
[  676.738313] [<ffff000008088b80>] do_signal+0x7c/0x668
[  676.738617] [<ffff000008089538>] do_notify_resume+0xcc/0x114
 [  676.740983] [<ffff0000080836c0>] work_pending+0x8/0x10
[  676.741360] Code: f94043a4 f9404ba2 f94037a3 d1000441 (f9400080)
[  676.741745] ---[ end trace e42d453027313552 ]---
[  676.804174] Fixing recursive fault but reboot is needed!
[  677.462082] Memory failure: 0xcd4b: already hardware poisoned

[Question ]: Avoid kernel panic when killing an application if happen RAS page table error

[gengdongjiu]

change the mail title and resend.

Hi James/All,
  If the user space application happen page table RAS error,Memory error handler(memory_failure()) will do nothing except making a poisoned page flag, and fault handler in arch/arm64/mm/fault.c
will deliver a signal to kill this application. when this application exits, it will call unmap_vmas () to release his vma resource, but here it will touch the error page table again, then will
trigger RAS error again, so this application cannot be killed and system will be panic, the log is shown in [2].

As shown the stack in [1], unmap_page_range() will touch the error page table, so system will panic, there are some simple way to avoid this panic and avoid change much about the memory management.
1. put the tasks to dead status, not run it again.
2. not release the page table for this task.

Of cause, above methods may happen memory leakage. do you have good suggestion about how to solve it?, or do you think this panic is expected behavior? thanks.


[1]:
get_signal()
   do_group_exit()
      mmput()
                exit_mmap()
                        unmap_vmas()
                                unmap_single_vma()
                                        unmap_page_range()


[2]
[  676.669053] Synchronous External Abort: level 0 (translation table walk) (0x82000214) at 0x0000000033ff7008
[  676.686469] Memory failure: 0xcd4b: already hardware poisoned
[  676.700652] Synchronous External Abort: synchronous external abort (0x96000410) at 0x0000000033ff7008
[  676.723301] Internal error: : 96000410 [#1] PREEMPT SMP
[  676.723616] Modules linked in: inject_memory_error(O)
[  676.724601] CPU: 0 PID: 1506 Comm: mca-recover Tainted: G           O    4.14.0-rc8-00019-g5b5c6f4-dirty #109
[  676.724844] task: ffff80000cd41d00 task.stack: ffff000009b30000
[  676.726616] PC is at unmap_page_range+0x78/0x6fc
[  676.726960] LR is at unmap_single_vma+0x88/0xdc
[  676.727122] pc : [<ffff0000081f109c>] lr : [<ffff0000081f17a8>] pstate: 80400149
[  676.727227] sp : ffff000009b339b0
[  676.727348] x29: ffff000009b339b0 x28: ffff80000cd41d00
[  676.727653] x27: 0000000000000000 x26: ffff80000cd42410
[  676.727919] x25: ffff80000cd41d00 x24: ffff80000cd1e180
[  676.728161] x23: ffff80000ce22300 x22: 0000000000000000
[  676.728407] x21: ffff000009b33b28 x20: 0000000000400000
[  676.728642] x19: ffff80000cd1e180 x18: 000000000000016d
[  676.728875] x17: 0000000000000190 x16: 0000000000000064
[  676.729117] x15: 0000000000000339 x14: 0000000000000000
[  676.729344] x13: 00000000000061a8 x12: 0000000000000339
[  676.729582] x11: 0000000000000018 x10: 0000000000000a80
[  676.729829] x9 : ffff000009b33c60 x8 : ffff80000cd427e0
[  676.730065] x7 : ffff000009b33de8 x6 : 00000000004a2000
[  676.730287] x5 : 0000000000400000 x4 : ffff80000cd4b000
[  676.730517] x3 : 00000000004a1fff x2 : 0000008000000000
[  676.730741] x1 : 0000007fffffffff x0 : 0000008000000000
[  676.731101] Process mca-recover (pid: 1506, stack limit = 0xffff000009b30000)
[  676.731281] Call trace:
[  676.734196] [<ffff0000081f109c>] unmap_page_range+0x78/0x6fc
[  676.734539] [<ffff0000081f17a8>] unmap_single_vma+0x88/0xdc
[  676.734892] [<ffff0000081f1aa8>] unmap_vmas+0x68/0xb4
[  676.735456] [<ffff0000081fa56c>] exit_mmap+0x90/0x140
[  676.736468] [<ffff0000080ccb34>] mmput+0x60/0x118
[  676.736791] [<ffff0000080d4060>] do_exit+0x240/0x9cc
[  676.736997] [<ffff0000080d4854>] do_group_exit+0x38/0x98
[  676.737384] [<ffff0000080df4d0>] get_signal+0x1ec/0x548
[  676.738313] [<ffff000008088b80>] do_signal+0x7c/0x668
[  676.738617] [<ffff000008089538>] do_notify_resume+0xcc/0x114
 [  676.740983] [<ffff0000080836c0>] work_pending+0x8/0x10
[  676.741360] Code: f94043a4 f9404ba2 f94037a3 d1000441 (f9400080)
[  676.741745] ---[ end trace e42d453027313552 ]---
[  676.804174] Fixing recursive fault but reboot is needed!
[  677.462082] Memory failure: 0xcd4b: already hardware poisoned



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Question ]: Avoid kernel panic when killing an application if happen RAS page table error

[James Morse]

Hi gengdongjiu,

On 15/12/17 02:00, gengdongjiu wrote:
> change the mail title and resend.

(please don't do this, we all got the first version)


> If the user space application happen page table RAS error,Memory error handler(memory_failure()) will
> do nothing except making a poisoned page flag,

Yes, because user-space process's page tables are kernel memory.

memory_failure() depends on the system being able to contain these faults,
giving us another RAS exception if we touch the page again.


> and fault handler in arch/arm64/mm/fault.c
> will deliver a signal to kill this application. when this application exits, it will call unmap_vmas ()
> to release his vma resource, but here it will touch the error page table
again, then will
> trigger RAS error again, so this application cannot be killed and system will be panic, the log is shown in [2].

Kernel memory is corrupt, we panic().

You want to add a distinction to handle user-space process's page tables:

> As shown the stack in [1], unmap_page_range() will touch the error page table, so system will panic,
> there are some simple way to avoid this panic and avoid change much about
> the memory management.
> 1. put the tasks to dead status, not run it again.
> 2. not release the page table for this task.
> 
> Of cause, above methods may happen memory leakage. do you have good suggestion about how to solve it?, or do you think this panic is expected behavior? thanks.

I don't think this is worth the effort, the page tables are small compared to
the memory they map. Even if this were fixed, you still have the chance of other
kernel memory being corrupted.

Leaking any memory that isn't marked as poisoned isn't a good idea.

What you would need is a way to know from the struct_page that: this page is
is page-table, and which struct_mm it belongs to. (If its the kernel's init_mm:
panic()).
Next you need a way to find all the other pages of page-table without walking
them. With these three pieces of information you can free all the unaffected
memory, with even more work you can probably regenerate the corrupted page.

It's going to be complicated to do, I don't think its worth the effort.


Thanks,

James

Re: [Question ]: Avoid kernel panic when killing an application if happen RAS page table error

[James Morse]

Hi gengdongjiu,

On 15/12/17 02:00, gengdongjiu wrote:
> change the mail title and resend.

(please don't do this, we all got the first version)


> If the user space application happen page table RAS error,Memory error handler(memory_failure()) will
> do nothing except making a poisoned page flag,

Yes, because user-space process's page tables are kernel memory.

memory_failure() depends on the system being able to contain these faults,
giving us another RAS exception if we touch the page again.


> and fault handler in arch/arm64/mm/fault.c
> will deliver a signal to kill this application. when this application exits, it will call unmap_vmas ()
> to release his vma resource, but here it will touch the error page table
again, then will
> trigger RAS error again, so this application cannot be killed and system will be panic, the log is shown in [2].

Kernel memory is corrupt, we panic().

You want to add a distinction to handle user-space process's page tables:

> As shown the stack in [1], unmap_page_range() will touch the error page table, so system will panic,
> there are some simple way to avoid this panic and avoid change much about
> the memory management.
> 1. put the tasks to dead status, not run it again.
> 2. not release the page table for this task.
> 
> Of cause, above methods may happen memory leakage. do you have good suggestion about how to solve it?, or do you think this panic is expected behavior? thanks.

I don't think this is worth the effort, the page tables are small compared to
the memory they map. Even if this were fixed, you still have the chance of other
kernel memory being corrupted.

Leaking any memory that isn't marked as poisoned isn't a good idea.

What you would need is a way to know from the struct_page that: this page is
is page-table, and which struct_mm it belongs to. (If its the kernel's init_mm:
panic()).
Next you need a way to find all the other pages of page-table without walking
them. With these three pieces of information you can free all the unaffected
memory, with even more work you can probably regenerate the corrupted page.

It's going to be complicated to do, I don't think its worth the effort.


Thanks,

James

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[Question ]: Avoid kernel panic when killing an application if happen RAS page table error

[James Morse]

Hi gengdongjiu,

On 15/12/17 02:00, gengdongjiu wrote:
> change the mail title and resend.

(please don't do this, we all got the first version)


> If the user space application happen page table RAS error,Memory error handler(memory_failure()) will
> do nothing except making a poisoned page flag,

Yes, because user-space process's page tables are kernel memory.

memory_failure() depends on the system being able to contain these faults,
giving us another RAS exception if we touch the page again.


> and fault handler in arch/arm64/mm/fault.c
> will deliver a signal to kill this application. when this application exits, it will call unmap_vmas ()
> to release his vma resource, but here it will touch the error page table
again, then will
> trigger RAS error again, so this application cannot be killed and system will be panic, the log is shown in [2].

Kernel memory is corrupt, we panic().

You want to add a distinction to handle user-space process's page tables:

> As shown the stack in [1], unmap_page_range() will touch the error page table, so system will panic,
> there are some simple way to avoid this panic and avoid change much about
> the memory management.
> 1. put the tasks to dead status, not run it again.
> 2. not release the page table for this task.
> 
> Of cause, above methods may happen memory leakage. do you have good suggestion about how to solve it?, or do you think this panic is expected behavior? thanks.

I don't think this is worth the effort, the page tables are small compared to
the memory they map. Even if this were fixed, you still have the chance of other
kernel memory being corrupted.

Leaking any memory that isn't marked as poisoned isn't a good idea.

What you would need is a way to know from the struct_page that: this page is
is page-table, and which struct_mm it belongs to. (If its the kernel's init_mm:
panic()).
Next you need a way to find all the other pages of page-table without walking
them. With these three pieces of information you can free all the unaffected
memory, with even more work you can probably regenerate the corrupted page.

It's going to be complicated to do, I don't think its worth the effort.


Thanks,

James

Re: [Question ]: Avoid kernel panic when killing an application if happen RAS page table error

[Matthew Wilcox]

On Fri, Dec 15, 2017 at 06:52:35PM +0000, James Morse wrote:
> Leaking any memory that isn't marked as poisoned isn't a good idea.
> 
> What you would need is a way to know from the struct_page that: this page is
> is page-table, and which struct_mm it belongs to. (If its the kernel's init_mm:
> panic()).
> Next you need a way to find all the other pages of page-table without walking
> them. With these three pieces of information you can free all the unaffected
> memory, with even more work you can probably regenerate the corrupted page.
> 
> It's going to be complicated to do, I don't think its worth the effort.

We can find a bit in struct page that we guarantee will only be set if
this is allocated as a pagetable.  Bit 1 of the third union is currently
available (compound_head is a pointer if bit 0 is set, so nothing is
using bit 1).  We can put a pointer to the mm_struct in the same word.

Finding all the allocated pages will be the tricky bit.  We could put a
list_head into struct page; perhaps in the same spot as page_deferred_list
for tail pages.  Then we can link all the pagetables belonging to
this mm together and tear them all down if any of them get an error.
They'll repopulate on demand.  It won't be quick or scalable, but when
the alternative is death, it looks relatively attractive.

Re: [Question ]: Avoid kernel panic when killing an application if happen RAS page table error

[Matthew Wilcox]

On Fri, Dec 15, 2017 at 06:52:35PM +0000, James Morse wrote:
> Leaking any memory that isn't marked as poisoned isn't a good idea.
> 
> What you would need is a way to know from the struct_page that: this page is
> is page-table, and which struct_mm it belongs to. (If its the kernel's init_mm:
> panic()).
> Next you need a way to find all the other pages of page-table without walking
> them. With these three pieces of information you can free all the unaffected
> memory, with even more work you can probably regenerate the corrupted page.
> 
> It's going to be complicated to do, I don't think its worth the effort.

We can find a bit in struct page that we guarantee will only be set if
this is allocated as a pagetable.  Bit 1 of the third union is currently
available (compound_head is a pointer if bit 0 is set, so nothing is
using bit 1).  We can put a pointer to the mm_struct in the same word.

Finding all the allocated pages will be the tricky bit.  We could put a
list_head into struct page; perhaps in the same spot as page_deferred_list
for tail pages.  Then we can link all the pagetables belonging to
this mm together and tear them all down if any of them get an error.
They'll repopulate on demand.  It won't be quick or scalable, but when
the alternative is death, it looks relatively attractive.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[Question ]: Avoid kernel panic when killing an application if happen RAS page table error

[Matthew Wilcox]

On Fri, Dec 15, 2017 at 06:52:35PM +0000, James Morse wrote:
> Leaking any memory that isn't marked as poisoned isn't a good idea.
> 
> What you would need is a way to know from the struct_page that: this page is
> is page-table, and which struct_mm it belongs to. (If its the kernel's init_mm:
> panic()).
> Next you need a way to find all the other pages of page-table without walking
> them. With these three pieces of information you can free all the unaffected
> memory, with even more work you can probably regenerate the corrupted page.
> 
> It's going to be complicated to do, I don't think its worth the effort.

We can find a bit in struct page that we guarantee will only be set if
this is allocated as a pagetable.  Bit 1 of the third union is currently
available (compound_head is a pointer if bit 0 is set, so nothing is
using bit 1).  We can put a pointer to the mm_struct in the same word.

Finding all the allocated pages will be the tricky bit.  We could put a
list_head into struct page; perhaps in the same spot as page_deferred_list
for tail pages.  Then we can link all the pagetables belonging to
this mm together and tear them all down if any of them get an error.
They'll repopulate on demand.  It won't be quick or scalable, but when
the alternative is death, it looks relatively attractive.

Re: [Question ]: Avoid kernel panic when killing an application if happen RAS page table error

[gengdongjiu]

On 2017/12/16 3:35, Matthew Wilcox wrote:
>> It's going to be complicated to do, I don't think its worth the effort.
> We can find a bit in struct page that we guarantee will only be set if
> this is allocated as a pagetable.  Bit 1 of the third union is currently
> available (compound_head is a pointer if bit 0 is set, so nothing is
> using bit 1).  We can put a pointer to the mm_struct in the same word.
> 
> Finding all the allocated pages will be the tricky bit.  We could put a
> list_head into struct page; perhaps in the same spot as page_deferred_list
> for tail pages.  Then we can link all the pagetables belonging to
> this mm together and tear them all down if any of them get an error.
> They'll repopulate on demand.  It won't be quick or scalable, but when
> the alternative is death, it looks relatively attractive.
Thanks for the comments, I will check it in detailed and investigate whether it is worth to do for it.
Thanks!

> 
> .
> 

Re: [Question ]: Avoid kernel panic when killing an application if happen RAS page table error

[gengdongjiu]

On 2017/12/16 3:35, Matthew Wilcox wrote:
>> It's going to be complicated to do, I don't think its worth the effort.
> We can find a bit in struct page that we guarantee will only be set if
> this is allocated as a pagetable.  Bit 1 of the third union is currently
> available (compound_head is a pointer if bit 0 is set, so nothing is
> using bit 1).  We can put a pointer to the mm_struct in the same word.
> 
> Finding all the allocated pages will be the tricky bit.  We could put a
> list_head into struct page; perhaps in the same spot as page_deferred_list
> for tail pages.  Then we can link all the pagetables belonging to
> this mm together and tear them all down if any of them get an error.
> They'll repopulate on demand.  It won't be quick or scalable, but when
> the alternative is death, it looks relatively attractive.
Thanks for the comments, I will check it in detailed and investigate whether it is worth to do for it.
Thanks!

> 
> .
> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[Question ]: Avoid kernel panic when killing an application if happen RAS page table error

[gengdongjiu]

On 2017/12/16 3:35, Matthew Wilcox wrote:
>> It's going to be complicated to do, I don't think its worth the effort.
> We can find a bit in struct page that we guarantee will only be set if
> this is allocated as a pagetable.  Bit 1 of the third union is currently
> available (compound_head is a pointer if bit 0 is set, so nothing is
> using bit 1).  We can put a pointer to the mm_struct in the same word.
> 
> Finding all the allocated pages will be the tricky bit.  We could put a
> list_head into struct page; perhaps in the same spot as page_deferred_list
> for tail pages.  Then we can link all the pagetables belonging to
> this mm together and tear them all down if any of them get an error.
> They'll repopulate on demand.  It won't be quick or scalable, but when
> the alternative is death, it looks relatively attractive.
Thanks for the comments, I will check it in detailed and investigate whether it is worth to do for it.
Thanks!

> 
> .
>