makedumpfile issues many readpage_elf: Attempt to read non-existent page

makedumpfile issues many readpage_elf: Attempt to read non-existent page

[Louis Bouchard]

Hello,

I am investigating an issue with makedumpfile and kernel 4.8 where makedumpfile
(1.6.0) exits on error with the following message :

  get_mem_map: Can't distinguish the memory type.

I found commit 2c21d4656e8d3c2af2b1e14809d076941ae69e96 in the upstream
development branch that is supposed to fix this :

[PATCH v2] Support _count -> _refcount rename in struct page

    _count member was renamed to _refcount in linux commit 0139aa7b7fa12
    ("mm: rename _count, field of the struct page, to _refcount") and this
    broke makedumpfile. The reason for making the change was to find all users
    accessing it directly and not through the recommended API. I tried
    suggesting to revert the change but failed, I see no other choice than to
    start supporting both _count and _refcount in makedumpfile.

Though, when I apply the patch and test on either Ubuntu's 4.8.0-11 kernel, or
kernel.org's mainline 4.8.0-040800rc7 kernel, I get the following repeated
multiple times :

> makedumpfile -c -d 31 /proc/vmcore /var/crash/201609221517/dump-incomplete
> [    7.513337] kdump-tools[715]: readpage_elf: Attempt to read non-existent page at 0x134dfff78000.
> [    7.524186] kdump-tools[715]: readmem: type_addr: 0, addr:ffff9b4dfff78000, size:16
> [    7.528440] kdump-tools[715]: section_mem_map_addr: Can't get a struct mem_section(ffff9b4dfff78000).
> [    7.536562] kdump-tools[715]: readpage_elf: Attempt to read non-existent page at 0x134dfff78000.
> [    7.544356] kdump-tools[715]: readmem: type_addr: 0, addr:ffff9b4dfff78010, size:16
> [    7.552422] kdump-tools[715]: section_mem_map_addr: Can't get a struct mem_section(ffff9b4dfff78010).
> [    7.560317] kdump-tools[715]: readpage_elf: Attempt to read non-existent page at 0x134dfff78000.
> [    7.568422] kdump-tools[715]: readmem: type_addr: 0, addr:ffff9b4dfff78020, size:16
> [    7.572296] kdump-tools[715]: section_mem_map_addr: Can't get a struct mem_section(ffff9b4dfff78020).

I also tested with all the commits in the upstream/development branch with no
luck, I still get the same behavior.

Does someone have an idea of where this could come from ?

TIA,

Louis
-- 
Louis Bouchard
Software engineer, Cloud & Sustaining eng.
Canonical Ltd
Ubuntu developer                       Debian Maintainer
GPG : 429D 7A3B DD05 B6F8 AF63  B9C4 8B3D 867C 823E 7A61

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

RE: makedumpfile issues many readpage_elf: Attempt to read non-existent page

[Atsushi Kumagai]

>Hello,
>
>I am investigating an issue with makedumpfile and kernel 4.8 where makedumpfile
>(1.6.0) exits on error with the following message :
>
>  get_mem_map: Can't distinguish the memory type.
>
>I found commit 2c21d4656e8d3c2af2b1e14809d076941ae69e96 in the upstream
>development branch that is supposed to fix this :
>
>[PATCH v2] Support _count -> _refcount rename in struct page
>
>    _count member was renamed to _refcount in linux commit 0139aa7b7fa12
>    ("mm: rename _count, field of the struct page, to _refcount") and this
>    broke makedumpfile. The reason for making the change was to find all users
>    accessing it directly and not through the recommended API. I tried
>    suggesting to revert the change but failed, I see no other choice than to
>    start supporting both _count and _refcount in makedumpfile.
>
>Though, when I apply the patch and test on either Ubuntu's 4.8.0-11 kernel, or
>kernel.org's mainline 4.8.0-040800rc7 kernel, I get the following repeated
>multiple times :
>
>> makedumpfile -c -d 31 /proc/vmcore /var/crash/201609221517/dump-incomplete
>> [    7.513337] kdump-tools[715]: readpage_elf: Attempt to read non-existent page at 0x134dfff78000.
>> [    7.524186] kdump-tools[715]: readmem: type_addr: 0, addr:ffff9b4dfff78000, size:16
>> [    7.528440] kdump-tools[715]: section_mem_map_addr: Can't get a struct mem_section(ffff9b4dfff78000).
>> [    7.536562] kdump-tools[715]: readpage_elf: Attempt to read non-existent page at 0x134dfff78000.
>> [    7.544356] kdump-tools[715]: readmem: type_addr: 0, addr:ffff9b4dfff78010, size:16
>> [    7.552422] kdump-tools[715]: section_mem_map_addr: Can't get a struct mem_section(ffff9b4dfff78010).
>> [    7.560317] kdump-tools[715]: readpage_elf: Attempt to read non-existent page at 0x134dfff78000.
>> [    7.568422] kdump-tools[715]: readmem: type_addr: 0, addr:ffff9b4dfff78020, size:16
>> [    7.572296] kdump-tools[715]: section_mem_map_addr: Can't get a struct mem_section(ffff9b4dfff78020).
>
>I also tested with all the commits in the upstream/development branch with no
>luck, I still get the same behavior.
>
>Does someone have an idea of where this could come from ?

Thanks for your report, but I'm afraid that I'm going on
vacation till Oct 2.
I hope someone helps you until I'm back.


Regards,
Atsushi Kumagai

>TIA,
>
>Louis
>--
>Louis Bouchard
>Software engineer, Cloud & Sustaining eng.
>Canonical Ltd
>Ubuntu developer                       Debian Maintainer
>GPG : 429D 7A3B DD05 B6F8 AF63  B9C4 8B3D 867C 823E 7A61
>
>_______________________________________________
>kexec mailing list
>kexec@lists.infradead.org
>http://lists.infradead.org/mailman/listinfo/kexec

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: makedumpfile issues many readpage_elf: Attempt to read non-existent page

[Louis Bouchard]

Hello,

Le 23/09/2016 14:04, Atsushi Kumagai a écrit :
>> Does someone have an idea of where this could come from ?
> 
> Thanks for your report, but I'm afraid that I'm going on
> vacation till Oct 2.
> I hope someone helps you until I'm back.
> 
> 
> Regards,
> Atsushi Kumagai
> 

As a follow up to my question, I've been investigating the issue. The source of
the problem seems to come from readpage_elf() which tries to read to an
inexistant page :

readpage_elf: Attempt to read non-existent page at 0x1a7b7fff5000.

The error happens when readpage_elf() is called from section_mem_map_addr with
pgaddr, which is derived from paddr. This one comes from :

   paddr = vaddr_to_paddr(addr)

When tracing vaddr_to_paddr, I see a difference in the calculation. on a 4.4
kernel :

  info->phys_base = 0

On 4.8 :
  info->phys_base = 0xffffffffe3800000

So the paddr values used to define pgaddr are :

for 4.4 kernels : 0x3fff4000
for 4.8 kernels : 0x1a7b7fff5000

Running makedumpfile --dump-dmesg in debug mode leads to :

4.4 kernel :

sadump: does not have partition header
sadump: read dump device as unknown format
sadump: unknown format
LOAD (0)
  phys_start : 1000000
  phys_end   : 2224000
  virt_start : ffffffff81000000
  virt_end   : ffffffff82224000
LOAD (1)
  phys_start : 1000
  phys_end   : 9fc00
  virt_start : ffff880000001000
  virt_end   : ffff88000009fc00
LOAD (2)
  phys_start : 100000
  phys_end   : 2b000000
  virt_start : ffff880000100000
  virt_end   : ffff88002b000000
LOAD (3)
  phys_start : 33000000
  phys_end   : 3fffe000
  virt_start : ffff880033000000
  virt_end   : ffff88003fffe000
Linux kdump
page_size    : 4096

max_mapnr    : 3fffe

Buffer size for the cyclic mode: 65535

num of NODEs : 1


Memory type  : SPARSEMEM_EX

mem_map (0)
  mem_map    : ffffea0000000000
  pfn_start  : 0
  pfn_end    : 8000
mem_map (1)
  mem_map    : ffffea0000200000
  pfn_start  : 8000
  pfn_end    : 10000
mem_map (2)
  mem_map    : ffffea0000400000
  pfn_start  : 10000
  pfn_end    : 18000
mem_map (3)
  mem_map    : ffffea0000600000
  pfn_start  : 18000
  pfn_end    : 20000
mem_map (4)
  mem_map    : ffffea0000800000
  pfn_start  : 20000
  pfn_end    : 28000
mem_map (5)
  mem_map    : ffffea0000a00000
  pfn_start  : 28000
  pfn_end    : 30000
mem_map (6)
  mem_map    : ffffea0000c00000
  pfn_start  : 30000
  pfn_end    : 38000
mem_map (7)
  mem_map    : ffffea0000e00000
  pfn_start  : 38000
  pfn_end    : 3fffe
mmap() is available on the kernel.

log_buf       : ffffffff8210521c
log_end       : 0
log_buf_len   : 262144
log_first_idx : 0
log_next_idx  : 141176

The dmesg log is saved to /tmp/titi.

makedumpfile Completed.


For 4.8 kernels :
sadump: does not have partition header
sadump: read dump device as unknown format
sadump: unknown format
LOAD (0)
  phys_start : 7200000
  phys_end   : 847d000
  virt_start : ffffffffa3a00000
  virt_end   : ffffffffa4c7d000
LOAD (1)
  phys_start : 1000
  phys_end   : 9fc00
  virt_start : ffff880000001000
  virt_end   : ffff88000009fc00
LOAD (2)
  phys_start : 100000
  phys_end   : 2b000000
  virt_start : ffff880000100000
  virt_end   : ffff88002b000000
LOAD (3)
  phys_start : 33000000
  phys_end   : 3fffe000
  virt_start : ffff880033000000
  virt_end   : ffff88003fffe000
Linux kdump
page_size    : 4096

max_mapnr    : 3fffe

Buffer size for the cyclic mode: 65535
The kernel version is not supported.
The makedumpfile operation may be incomplete.

num of NODEs : 1


Memory type  : SPARSEMEM_EX

mem_map (0)
  mem_map    : 0
  pfn_start  : 0
  pfn_end    : 8000
mem_map (1)
  mem_map    : 0
  pfn_start  : 8000
  pfn_end    : 10000
mem_map (2)
  mem_map    : 0
  pfn_start  : 10000
  pfn_end    : 18000
mem_map (3)
  mem_map    : 0
  pfn_start  : 18000
  pfn_end    : 20000
mem_map (4)
  mem_map    : 0
  pfn_start  : 20000
  pfn_end    : 28000
mem_map (5)
  mem_map    : 0
  pfn_start  : 28000
  pfn_end    : 30000
mem_map (6)
  mem_map    : 0
  pfn_start  : 30000
  pfn_end    : 38000
mem_map (7)
  mem_map    : 0
  pfn_start  : 38000
  pfn_end    : 3fffe
mmap() is available on the kernel.

log_buf       : ffffffffa4b3c19c
log_end       : 0
log_buf_len   : 262144
log_first_idx : 0
log_next_idx  : 43160

The dmesg log is saved to /tmp/toto.

makedumpfile Completed.


I don't know if it is to be expected with kernels 4.8 to have all mem_map
addresses set to 0, unlike with k4.4 :

mem_map (0)
mem_map : 0
pfn_start : 0
pfn_end : 8000

Any comment appreciated while I continue to investigate.

Kind regards,

...Louis

-- 
Louis Bouchard
Software engineer, Cloud & Sustaining eng.
Canonical Ltd
Ubuntu developer                       Debian Maintainer
GPG : 429D 7A3B DD05 B6F8 AF63  B9C4 8B3D 867C 823E 7A61

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: makedumpfile issues many readpage_elf: Attempt to read non-existent page

[Louis Bouchard]

Hello,

Le 29/09/2016 à 15:34, Louis Bouchard a écrit :
> Hello,
>
> Le 23/09/2016 14:04, Atsushi Kumagai a écrit :
>>> Does someone have an idea of where this could come from ?
>>
>> Thanks for your report, but I'm afraid that I'm going on
>> vacation till Oct 2.
>> I hope someone helps you until I'm back.
>>
>>
>> Regards,
>> Atsushi Kumagai
>>
>
> As a follow up to my question, I've been investigating the issue. The source of
> the problem seems to come from readpage_elf() which tries to read to an
> inexistant page :
>
> readpage_elf: Attempt to read non-existent page at 0x1a7b7fff5000.
>
> The error happens when readpage_elf() is called from section_mem_map_addr with
> pgaddr, which is derived from paddr. This one comes from :
>
>    paddr = vaddr_to_paddr(addr)
>
> When tracing vaddr_to_paddr, I see a difference in the calculation. on a 4.4
> kernel :
>
>   info->phys_base = 0
>
> On 4.8 :
>   info->phys_base = 0xffffffffe3800000
>
> So the paddr values used to define pgaddr are :
>
> for 4.4 kernels : 0x3fff4000
> for 4.8 kernels : 0x1a7b7fff5000
>
> Running makedumpfile --dump-dmesg in debug mode leads to :
>
> 4.4 kernel :
>
> sadump: does not have partition header
> sadump: read dump device as unknown format
> sadump: unknown format
> LOAD (0)
>   phys_start : 1000000
>   phys_end   : 2224000
>   virt_start : ffffffff81000000
>   virt_end   : ffffffff82224000
> LOAD (1)
>   phys_start : 1000
>   phys_end   : 9fc00
>   virt_start : ffff880000001000
>   virt_end   : ffff88000009fc00
> LOAD (2)
>   phys_start : 100000
>   phys_end   : 2b000000
>   virt_start : ffff880000100000
>   virt_end   : ffff88002b000000
> LOAD (3)
>   phys_start : 33000000
>   phys_end   : 3fffe000
>   virt_start : ffff880033000000
>   virt_end   : ffff88003fffe000
> Linux kdump
> page_size    : 4096
>
> max_mapnr    : 3fffe
>
> Buffer size for the cyclic mode: 65535
>
> num of NODEs : 1
>
>
> Memory type  : SPARSEMEM_EX
>
> mem_map (0)
>   mem_map    : ffffea0000000000
>   pfn_start  : 0
>   pfn_end    : 8000
> mem_map (1)
>   mem_map    : ffffea0000200000
>   pfn_start  : 8000
>   pfn_end    : 10000
> mem_map (2)
>   mem_map    : ffffea0000400000
>   pfn_start  : 10000
>   pfn_end    : 18000
> mem_map (3)
>   mem_map    : ffffea0000600000
>   pfn_start  : 18000
>   pfn_end    : 20000
> mem_map (4)
>   mem_map    : ffffea0000800000
>   pfn_start  : 20000
>   pfn_end    : 28000
> mem_map (5)
>   mem_map    : ffffea0000a00000
>   pfn_start  : 28000
>   pfn_end    : 30000
> mem_map (6)
>   mem_map    : ffffea0000c00000
>   pfn_start  : 30000
>   pfn_end    : 38000
> mem_map (7)
>   mem_map    : ffffea0000e00000
>   pfn_start  : 38000
>   pfn_end    : 3fffe
> mmap() is available on the kernel.
>
> log_buf       : ffffffff8210521c
> log_end       : 0
> log_buf_len   : 262144
> log_first_idx : 0
> log_next_idx  : 141176
>
> The dmesg log is saved to /tmp/titi.
>
> makedumpfile Completed.
>
>
> For 4.8 kernels :
> sadump: does not have partition header
> sadump: read dump device as unknown format
> sadump: unknown format
> LOAD (0)
>   phys_start : 7200000
>   phys_end   : 847d000
>   virt_start : ffffffffa3a00000
>   virt_end   : ffffffffa4c7d000
> LOAD (1)
>   phys_start : 1000
>   phys_end   : 9fc00
>   virt_start : ffff880000001000
>   virt_end   : ffff88000009fc00
> LOAD (2)
>   phys_start : 100000
>   phys_end   : 2b000000
>   virt_start : ffff880000100000
>   virt_end   : ffff88002b000000
> LOAD (3)
>   phys_start : 33000000
>   phys_end   : 3fffe000
>   virt_start : ffff880033000000
>   virt_end   : ffff88003fffe000
> Linux kdump
> page_size    : 4096
>
> max_mapnr    : 3fffe
>
> Buffer size for the cyclic mode: 65535
> The kernel version is not supported.
> The makedumpfile operation may be incomplete.
>
> num of NODEs : 1
>
>
> Memory type  : SPARSEMEM_EX
>
> mem_map (0)
>   mem_map    : 0
>   pfn_start  : 0
>   pfn_end    : 8000
> mem_map (1)
>   mem_map    : 0
>   pfn_start  : 8000
>   pfn_end    : 10000
> mem_map (2)
>   mem_map    : 0
>   pfn_start  : 10000
>   pfn_end    : 18000
> mem_map (3)
>   mem_map    : 0
>   pfn_start  : 18000
>   pfn_end    : 20000
> mem_map (4)
>   mem_map    : 0
>   pfn_start  : 20000
>   pfn_end    : 28000
> mem_map (5)
>   mem_map    : 0
>   pfn_start  : 28000
>   pfn_end    : 30000
> mem_map (6)
>   mem_map    : 0
>   pfn_start  : 30000
>   pfn_end    : 38000
> mem_map (7)
>   mem_map    : 0
>   pfn_start  : 38000
>   pfn_end    : 3fffe
> mmap() is available on the kernel.
>
> log_buf       : ffffffffa4b3c19c
> log_end       : 0
> log_buf_len   : 262144
> log_first_idx : 0
> log_next_idx  : 43160
>
> The dmesg log is saved to /tmp/toto.
>
> makedumpfile Completed.
>
>
> I don't know if it is to be expected with kernels 4.8 to have all mem_map
> addresses set to 0, unlike with k4.4 :
>
> mem_map (0)
> mem_map : 0
> pfn_start : 0
> pfn_end : 8000
>
> Any comment appreciated while I continue to investigate.
>
> Kind regards,
>
> ...Louis
>

I have now completed the kernel bisection between 4.7.8 and 4.8-rc1 and
identified the kernel modification that triggers the errors cited above :

> commit 021182e52fe01c1f7b126f97fd6ba048dc4234fd
> Author: Thomas Garnier <thgarnie@google.com>
> Date:   Tue Jun 21 17:47:03 2016 -0700
>
>     x86/mm: Enable KASLR for physical mapping memory regions
>
>     Add the physical mapping in the list of randomized memory regions.
>
>     The physical memory mapping holds most allocations from boot and heap
>     allocators. Knowing the base address and physical memory size, an attacker
>     can deduce the PDE virtual address for the vDSO memory page. This attack
>     was demonstrated at CanSecWest 2016, in the following presentation:
>
>       "Getting Physical: Extreme Abuse of Intel Based Paged Systems":
>       https://github.com/n3k/CansecWest2016_Getting_Physical_Extreme_Abuse_of_Intel_Based_Paging_Systems/blob/master/Presentation/CanSec2016_Presentation.pdf
>
>     (See second part of the presentation).
>
>     The exploits used against Linux worked successfully against 4.6+ but
>     fail with KASLR memory enabled:
>
>       https://github.com/n3k/CansecWest2016_Getting_Physical_Extreme_Abuse_of_Intel_Based_Paging_Systems/tree/master/Demos/Linux/exploits
>
>     Similar research was done at Google leading to this patch proposal.
>
>     Variants exists to overwrite /proc or /sys objects ACLs leading to
>     elevation of privileges. These variants were tested against 4.6+.
>
>     The page offset used by the compressed kernel retains the static value
>     since it is not yet randomized during this boot stage.
>
>     Signed-off-by: Thomas Garnier <thgarnie@google.com>
>     Signed-off-by: Kees Cook <keescook@chromium.org>
>     Cc: Alexander Kuleshov <kuleshovmail@gmail.com>
<truncated>

The interesting change seems to be :

> -#define __PAGE_OFFSET           _AC(0xffff880000000000, UL)
> +#define __PAGE_OFFSET_BASE      _AC(0xffff880000000000, UL)
> +#ifdef CONFIG_RANDOMIZE_MEMORY
> +#define __PAGE_OFFSET           page_offset_base
> +#else
> +#define __PAGE_OFFSET           __PAGE_OFFSET_BASE
> +#endif /* CONFIG_RANDOMIZE_MEMORY */

I'll try to see if I can fix that.

Kind regards,

...Louis




-- 
Louis Bouchard
Software engineer, Cloud & Sustaining eng.
Canonical Ltd
Ubuntu developer                       Debian Maintainer
GPG : 429D 7A3B DD05 B6F8 AF63  B9C4 8B3D 867C 823E 7A61

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: makedumpfile issues many readpage_elf: Attempt to read non-existent page

[Louis Bouchard]

Good morning,

Le 24/10/2016 à 16:54, Louis Bouchard a écrit :
> Hello,
>
> Le 29/09/2016 à 15:34, Louis Bouchard a écrit :
>> Hello,
>>
>> Le 23/09/2016 14:04, Atsushi Kumagai a écrit :
>>>> Does someone have an idea of where this could come from ?
>>>
>>> Thanks for your report, but I'm afraid that I'm going on
>>> vacation till Oct 2.
>>> I hope someone helps you until I'm back.
>>>
>>>
>>> Regards,
>>> Atsushi Kumagai
>>>
>>
>> As a follow up to my question, I've been investigating the issue. The source of
>> the problem seems to come from readpage_elf() which tries to read to an
>> inexistant page :
>>
>> readpage_elf: Attempt to read non-existent page at 0x1a7b7fff5000.
>>
>> The error happens when readpage_elf() is called from section_mem_map_addr with
>> pgaddr, which is derived from paddr. This one comes from :
>>
>>    paddr = vaddr_to_paddr(addr)
>>
>> When tracing vaddr_to_paddr, I see a difference in the calculation. on a 4.4
>> kernel :
>>
>>   info->phys_base = 0
>>
>> On 4.8 :
>>   info->phys_base = 0xffffffffe3800000
>>
>> So the paddr values used to define pgaddr are :
>>
>> for 4.4 kernels : 0x3fff4000
>> for 4.8 kernels : 0x1a7b7fff5000
>>
>> Running makedumpfile --dump-dmesg in debug mode leads to :
>>
>> 4.4 kernel :
>>
>> sadump: does not have partition header
>> sadump: read dump device as unknown format
>> sadump: unknown format
>> LOAD (0)
>>   phys_start : 1000000
>>   phys_end   : 2224000
>>   virt_start : ffffffff81000000
>>   virt_end   : ffffffff82224000
>> LOAD (1)
>>   phys_start : 1000
>>   phys_end   : 9fc00
>>   virt_start : ffff880000001000
>>   virt_end   : ffff88000009fc00
>> LOAD (2)
>>   phys_start : 100000
>>   phys_end   : 2b000000
>>   virt_start : ffff880000100000
>>   virt_end   : ffff88002b000000
>> LOAD (3)
>>   phys_start : 33000000
>>   phys_end   : 3fffe000
>>   virt_start : ffff880033000000
>>   virt_end   : ffff88003fffe000
>> Linux kdump
>> page_size    : 4096
>>
>> max_mapnr    : 3fffe
>>
>> Buffer size for the cyclic mode: 65535
>>
>> num of NODEs : 1
>>
>>
>> Memory type  : SPARSEMEM_EX
>>
>> mem_map (0)
>>   mem_map    : ffffea0000000000
>>   pfn_start  : 0
>>   pfn_end    : 8000
>> mem_map (1)
>>   mem_map    : ffffea0000200000
>>   pfn_start  : 8000
>>   pfn_end    : 10000
>> mem_map (2)
>>   mem_map    : ffffea0000400000
>>   pfn_start  : 10000
>>   pfn_end    : 18000
>> mem_map (3)
>>   mem_map    : ffffea0000600000
>>   pfn_start  : 18000
>>   pfn_end    : 20000
>> mem_map (4)
>>   mem_map    : ffffea0000800000
>>   pfn_start  : 20000
>>   pfn_end    : 28000
>> mem_map (5)
>>   mem_map    : ffffea0000a00000
>>   pfn_start  : 28000
>>   pfn_end    : 30000
>> mem_map (6)
>>   mem_map    : ffffea0000c00000
>>   pfn_start  : 30000
>>   pfn_end    : 38000
>> mem_map (7)
>>   mem_map    : ffffea0000e00000
>>   pfn_start  : 38000
>>   pfn_end    : 3fffe
>> mmap() is available on the kernel.
>>
>> log_buf       : ffffffff8210521c
>> log_end       : 0
>> log_buf_len   : 262144
>> log_first_idx : 0
>> log_next_idx  : 141176
>>
>> The dmesg log is saved to /tmp/titi.
>>
>> makedumpfile Completed.
>>
>>
>> For 4.8 kernels :
>> sadump: does not have partition header
>> sadump: read dump device as unknown format
>> sadump: unknown format
>> LOAD (0)
>>   phys_start : 7200000
>>   phys_end   : 847d000
>>   virt_start : ffffffffa3a00000
>>   virt_end   : ffffffffa4c7d000
>> LOAD (1)
>>   phys_start : 1000
>>   phys_end   : 9fc00
>>   virt_start : ffff880000001000
>>   virt_end   : ffff88000009fc00
>> LOAD (2)
>>   phys_start : 100000
>>   phys_end   : 2b000000
>>   virt_start : ffff880000100000
>>   virt_end   : ffff88002b000000
>> LOAD (3)
>>   phys_start : 33000000
>>   phys_end   : 3fffe000
>>   virt_start : ffff880033000000
>>   virt_end   : ffff88003fffe000
>> Linux kdump
>> page_size    : 4096
>>
>> max_mapnr    : 3fffe
>>
>> Buffer size for the cyclic mode: 65535
>> The kernel version is not supported.
>> The makedumpfile operation may be incomplete.
>>
>> num of NODEs : 1
>>
>>
>> Memory type  : SPARSEMEM_EX
>>
>> mem_map (0)
>>   mem_map    : 0
>>   pfn_start  : 0
>>   pfn_end    : 8000
>> mem_map (1)
>>   mem_map    : 0
>>   pfn_start  : 8000
>>   pfn_end    : 10000
>> mem_map (2)
>>   mem_map    : 0
>>   pfn_start  : 10000
>>   pfn_end    : 18000
>> mem_map (3)
>>   mem_map    : 0
>>   pfn_start  : 18000
>>   pfn_end    : 20000
>> mem_map (4)
>>   mem_map    : 0
>>   pfn_start  : 20000
>>   pfn_end    : 28000
>> mem_map (5)
>>   mem_map    : 0
>>   pfn_start  : 28000
>>   pfn_end    : 30000
>> mem_map (6)
>>   mem_map    : 0
>>   pfn_start  : 30000
>>   pfn_end    : 38000
>> mem_map (7)
>>   mem_map    : 0
>>   pfn_start  : 38000
>>   pfn_end    : 3fffe
>> mmap() is available on the kernel.
>>
>> log_buf       : ffffffffa4b3c19c
>> log_end       : 0
>> log_buf_len   : 262144
>> log_first_idx : 0
>> log_next_idx  : 43160
>>
>> The dmesg log is saved to /tmp/toto.
>>
>> makedumpfile Completed.
>>
>>
>> I don't know if it is to be expected with kernels 4.8 to have all mem_map
>> addresses set to 0, unlike with k4.4 :
>>
>> mem_map (0)
>> mem_map : 0
>> pfn_start : 0
>> pfn_end : 8000
>>
>> Any comment appreciated while I continue to investigate.
>>
>> Kind regards,
>>
>> ...Louis
>>
>
> I have now completed the kernel bisection between 4.7.8 and 4.8-rc1 and
> identified the kernel modification that triggers the errors cited above :
>
>> commit 021182e52fe01c1f7b126f97fd6ba048dc4234fd
>> Author: Thomas Garnier <thgarnie@google.com>
>> Date:   Tue Jun 21 17:47:03 2016 -0700
>>
>>     x86/mm: Enable KASLR for physical mapping memory regions
>>
>>     Add the physical mapping in the list of randomized memory regions.
>>
>>     The physical memory mapping holds most allocations from boot and heap
>>     allocators. Knowing the base address and physical memory size, an attacker
>>     can deduce the PDE virtual address for the vDSO memory page. This attack
>>     was demonstrated at CanSecWest 2016, in the following presentation:
>>
>>       "Getting Physical: Extreme Abuse of Intel Based Paged Systems":
>>       https://github.com/n3k/CansecWest2016_Getting_Physical_Extreme_Abuse_of_Intel_Based_Paging_Systems/blob/master/Presentation/CanSec2016_Presentation.pdf
>>
>>     (See second part of the presentation).
>>
>>     The exploits used against Linux worked successfully against 4.6+ but
>>     fail with KASLR memory enabled:
>>
>>       https://github.com/n3k/CansecWest2016_Getting_Physical_Extreme_Abuse_of_Intel_Based_Paging_Systems/tree/master/Demos/Linux/exploits
>>
>>     Similar research was done at Google leading to this patch proposal.
>>
>>     Variants exists to overwrite /proc or /sys objects ACLs leading to
>>     elevation of privileges. These variants were tested against 4.6+.
>>
>>     The page offset used by the compressed kernel retains the static value
>>     since it is not yet randomized during this boot stage.
>>
>>     Signed-off-by: Thomas Garnier <thgarnie@google.com>
>>     Signed-off-by: Kees Cook <keescook@chromium.org>
>>     Cc: Alexander Kuleshov <kuleshovmail@gmail.com>
> <truncated>
>
> The interesting change seems to be :
>
>> -#define __PAGE_OFFSET           _AC(0xffff880000000000, UL)
>> +#define __PAGE_OFFSET_BASE      _AC(0xffff880000000000, UL)
>> +#ifdef CONFIG_RANDOMIZE_MEMORY
>> +#define __PAGE_OFFSET           page_offset_base
>> +#else
>> +#define __PAGE_OFFSET           __PAGE_OFFSET_BASE
>> +#endif /* CONFIG_RANDOMIZE_MEMORY */
>
> I'll try to see if I can fix that.
>
> Kind regards,
>
> ...Louis
>
>
>
>

Some more *important* information in this mostly monologue thread : Pratyush
Anand has pushed a patch to the list earlier today that apparently fixes this
issue :

[PATCH Makedumpfile 1/4] x86_64: Calculate page_offset from pt_load[1]

HTH,

Kind regards,

...Louis

[1] https://www.mail-archive.com/kexec@lists.infradead.org/msg16628.html
-- 
Louis Bouchard
Software engineer, Cloud & Sustaining eng.
Canonical Ltd
Ubuntu developer                       Debian Maintainer
GPG : 429D 7A3B DD05 B6F8 AF63  B9C4 8B3D 867C 823E 7A61

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

RE: makedumpfile issues many readpage_elf: Attempt to read non-existent page

[Atsushi Kumagai]

Hello,

>> I have now completed the kernel bisection between 4.7.8 and 4.8-rc1 and
>> identified the kernel modification that triggers the errors cited above :
>>
>>> commit 021182e52fe01c1f7b126f97fd6ba048dc4234fd
>>> Author: Thomas Garnier <thgarnie@google.com>
>>> Date:   Tue Jun 21 17:47:03 2016 -0700
>>>
>>>     x86/mm: Enable KASLR for physical mapping memory regions
>>>
>>>     Add the physical mapping in the list of randomized memory regions.
>>>
>>>     The physical memory mapping holds most allocations from boot and heap
>>>     allocators. Knowing the base address and physical memory size, an attacker
>>>     can deduce the PDE virtual address for the vDSO memory page. This attack
>>>     was demonstrated at CanSecWest 2016, in the following presentation:
>>>
>>>       "Getting Physical: Extreme Abuse of Intel Based Paged Systems":
>>>
>https://github.com/n3k/CansecWest2016_Getting_Physical_Extreme_Abuse_of_Intel_Based_Paging_Systems/blob/master/Prese
>ntation/CanSec2016_Presentation.pdf
>>>
>>>     (See second part of the presentation).
>>>
>>>     The exploits used against Linux worked successfully against 4.6+ but
>>>     fail with KASLR memory enabled:
>>>
>>>
>https://github.com/n3k/CansecWest2016_Getting_Physical_Extreme_Abuse_of_Intel_Based_Paging_Systems/tree/master/Demos
>/Linux/exploits
>>>
>>>     Similar research was done at Google leading to this patch proposal.
>>>
>>>     Variants exists to overwrite /proc or /sys objects ACLs leading to
>>>     elevation of privileges. These variants were tested against 4.6+.
>>>
>>>     The page offset used by the compressed kernel retains the static value
>>>     since it is not yet randomized during this boot stage.
>>>
>>>     Signed-off-by: Thomas Garnier <thgarnie@google.com>
>>>     Signed-off-by: Kees Cook <keescook@chromium.org>
>>>     Cc: Alexander Kuleshov <kuleshovmail@gmail.com>
>> <truncated>
>>
>> The interesting change seems to be :
>>
>>> -#define __PAGE_OFFSET           _AC(0xffff880000000000, UL)
>>> +#define __PAGE_OFFSET_BASE      _AC(0xffff880000000000, UL)
>>> +#ifdef CONFIG_RANDOMIZE_MEMORY
>>> +#define __PAGE_OFFSET           page_offset_base
>>> +#else
>>> +#define __PAGE_OFFSET           __PAGE_OFFSET_BASE
>>> +#endif /* CONFIG_RANDOMIZE_MEMORY */
>>
>> I'll try to see if I can fix that.
>>
>> Kind regards,
>>
>> ...Louis
>>
>>
>>
>>
>
>Some more *important* information in this mostly monologue thread : Pratyush
>Anand has pushed a patch to the list earlier today that apparently fixes this
>issue :
>
>[PATCH Makedumpfile 1/4] x86_64: Calculate page_offset from pt_load[1]
>
>HTH,
>
>Kind regards,

Yeah, It appears so. I'm reviewing the patches,
please wait for that.
I appreciate your investigation for this issue.


Thanks,
Atsushi Kumagai

>...Louis
>
>[1] https://www.mail-archive.com/kexec@lists.infradead.org/msg16628.html
>--
>Louis Bouchard
>Software engineer, Cloud & Sustaining eng.
>Canonical Ltd
>Ubuntu developer                       Debian Maintainer
>GPG : 429D 7A3B DD05 B6F8 AF63  B9C4 8B3D 867C 823E 7A61
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec