All of lore.kernel.org
 help / color / mirror / Atom feed
From: David Hildenbrand <david@redhat.com>
To: Mike Rapoport <rppt@linux.ibm.com>
Cc: "Liang, Liang (Leo)" <Liang.Liang@amd.com>,
	"Deucher, Alexander" <Alexander.Deucher@amd.com>,
	linux-kernel@vger.kernel.org,
	amd-gfx list <amd-gfx@lists.freedesktop.org>,
	Andrew Morton <akpm@linux-foundation.org>,
	"Huang, Ray" <Ray.Huang@amd.com>,
	"Koenig, Christian" <Christian.Koenig@amd.com>,
	"Rafael J. Wysocki" <rafael@kernel.org>,
	George Kennedy <george.kennedy@oracle.com>
Subject: Re: slow boot with 7fef431be9c9 ("mm/page_alloc: place pages to tail in __free_pages_core()")
Date: Mon, 15 Mar 2021 14:04:09 +0100	[thread overview]
Message-ID: <22437770-956e-f7b4-a8f6-3f1cc28c3ec2@redhat.com> (raw)
In-Reply-To: <YEzCm/Uwvw7kKpd7@linux.ibm.com>

On 13.03.21 14:48, Mike Rapoport wrote:
> Hi,
> 
> On Sat, Mar 13, 2021 at 10:05:23AM +0100, David Hildenbrand wrote:
>>> Am 13.03.2021 um 05:04 schrieb Liang, Liang (Leo) <Liang.Liang@amd.com>:
>>>
>>> Hi David,
>>>
>>> Which benchmark tool you prefer? Memtest86+ or else?
>>
>> Hi Leo,
>>
>> I think you want something that runs under Linux natively.
>>
>> I‘m planning on coding up a kernel module to walk all 4MB pages in the
>> freelists and perform a stream benchmark individually. Then we might be
>> able to identify the problematic range - if there is a problematic range :)
> 
> My wild guess would be that the pages that are now at the head of free
> lists have wrong caching enabled. Might be worth checking in your test
> module.

I hacked something up real quick:

https://github.com/davidhildenbrand/kstream

Only briefly tested inside a VM. The output looks something like

[...]
[ 8396.432225] [0x0000000045800000 - 0x0000000045bfffff] 25322 MB/s / 
38948 MB/s
[ 8396.448749] [0x0000000045c00000 - 0x0000000045ffffff] 24481 MB/s / 
38946 MB/s
[ 8396.465197] [0x0000000046000000 - 0x00000000463fffff] 24892 MB/s / 
39170 MB/s
[ 8396.481552] [0x0000000046400000 - 0x00000000467fffff] 25222 MB/s / 
39156 MB/s
[ 8396.498012] [0x0000000046800000 - 0x0000000046bfffff] 24416 MB/s / 
39159 MB/s
[ 8396.514397] [0x0000000046c00000 - 0x0000000046ffffff] 25469 MB/s / 
38940 MB/s
[ 8396.530849] [0x0000000047000000 - 0x00000000473fffff] 24885 MB/s / 
38734 MB/s
[ 8396.547195] [0x0000000047400000 - 0x00000000477fffff] 25458 MB/s / 
38941 MB/s
[...]

The benchmark allocates one 4 MiB chunk at a time and runs a simplified 
STREAM benchmark a) without flushing caches b) flushing caches before 
every memory access.

It would be great if you could run that with the *old behavior* kernel 
(IOW, without 7fef431be9c9), so we might still be lucky to catch the 
problematic area in the freelist.

Let's see if that will indicate anything.

-- 
Thanks,

David / dhildenb


WARNING: multiple messages have this Message-ID (diff)
From: David Hildenbrand <david@redhat.com>
To: Mike Rapoport <rppt@linux.ibm.com>
Cc: "Liang, Liang \(Leo\)" <Liang.Liang@amd.com>,
	"Rafael J. Wysocki" <rafael@kernel.org>,
	linux-kernel@vger.kernel.org,
	amd-gfx list <amd-gfx@lists.freedesktop.org>,
	George Kennedy <george.kennedy@oracle.com>,
	"Huang, Ray" <Ray.Huang@amd.com>,
	"Deucher, Alexander" <Alexander.Deucher@amd.com>,
	Andrew Morton <akpm@linux-foundation.org>,
	"Koenig, Christian" <Christian.Koenig@amd.com>
Subject: Re: slow boot with 7fef431be9c9 ("mm/page_alloc: place pages to tail in __free_pages_core()")
Date: Mon, 15 Mar 2021 14:04:09 +0100	[thread overview]
Message-ID: <22437770-956e-f7b4-a8f6-3f1cc28c3ec2@redhat.com> (raw)
In-Reply-To: <YEzCm/Uwvw7kKpd7@linux.ibm.com>

On 13.03.21 14:48, Mike Rapoport wrote:
> Hi,
> 
> On Sat, Mar 13, 2021 at 10:05:23AM +0100, David Hildenbrand wrote:
>>> Am 13.03.2021 um 05:04 schrieb Liang, Liang (Leo) <Liang.Liang@amd.com>:
>>>
>>> Hi David,
>>>
>>> Which benchmark tool you prefer? Memtest86+ or else?
>>
>> Hi Leo,
>>
>> I think you want something that runs under Linux natively.
>>
>> I‘m planning on coding up a kernel module to walk all 4MB pages in the
>> freelists and perform a stream benchmark individually. Then we might be
>> able to identify the problematic range - if there is a problematic range :)
> 
> My wild guess would be that the pages that are now at the head of free
> lists have wrong caching enabled. Might be worth checking in your test
> module.

I hacked something up real quick:

https://github.com/davidhildenbrand/kstream

Only briefly tested inside a VM. The output looks something like

[...]
[ 8396.432225] [0x0000000045800000 - 0x0000000045bfffff] 25322 MB/s / 
38948 MB/s
[ 8396.448749] [0x0000000045c00000 - 0x0000000045ffffff] 24481 MB/s / 
38946 MB/s
[ 8396.465197] [0x0000000046000000 - 0x00000000463fffff] 24892 MB/s / 
39170 MB/s
[ 8396.481552] [0x0000000046400000 - 0x00000000467fffff] 25222 MB/s / 
39156 MB/s
[ 8396.498012] [0x0000000046800000 - 0x0000000046bfffff] 24416 MB/s / 
39159 MB/s
[ 8396.514397] [0x0000000046c00000 - 0x0000000046ffffff] 25469 MB/s / 
38940 MB/s
[ 8396.530849] [0x0000000047000000 - 0x00000000473fffff] 24885 MB/s / 
38734 MB/s
[ 8396.547195] [0x0000000047400000 - 0x00000000477fffff] 25458 MB/s / 
38941 MB/s
[...]

The benchmark allocates one 4 MiB chunk at a time and runs a simplified 
STREAM benchmark a) without flushing caches b) flushing caches before 
every memory access.

It would be great if you could run that with the *old behavior* kernel 
(IOW, without 7fef431be9c9), so we might still be lucky to catch the 
problematic area in the freelist.

Let's see if that will indicate anything.

-- 
Thanks,

David / dhildenb

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

  reply	other threads:[~2021-03-15 13:05 UTC|newest]

Thread overview: 48+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-03-11 14:41 slow boot with 7fef431be9c9 ("mm/page_alloc: place pages to tail in __free_pages_core()") Deucher, Alexander
2021-03-11 14:41 ` Deucher, Alexander
2021-03-11 15:02 ` David Hildenbrand
2021-03-11 15:02   ` David Hildenbrand
2021-03-12 14:06   ` Deucher, Alexander
2021-03-12 14:06     ` Deucher, Alexander
2021-03-12 14:12     ` David Hildenbrand
2021-03-12 14:12       ` David Hildenbrand
2021-03-12 15:39       ` Deucher, Alexander
2021-03-12 15:39         ` Deucher, Alexander
2021-03-12 15:48         ` David Hildenbrand
2021-03-12 15:48           ` David Hildenbrand
2021-03-12 16:15           ` Deucher, Alexander
2021-03-12 16:15             ` Deucher, Alexander
2021-03-12 16:19         ` Liang, Liang (Leo)
2021-03-12 16:19           ` Liang, Liang (Leo)
2021-03-12 16:46           ` David Hildenbrand
2021-03-12 16:46             ` David Hildenbrand
2021-03-13  4:04             ` Liang, Liang (Leo)
2021-03-13  4:04               ` Liang, Liang (Leo)
2021-03-13  9:05               ` David Hildenbrand
2021-03-13  9:05                 ` David Hildenbrand
2021-03-13 13:48                 ` Mike Rapoport
2021-03-13 13:48                   ` Mike Rapoport
2021-03-15 13:04                   ` David Hildenbrand [this message]
2021-03-15 13:04                     ` David Hildenbrand
2021-03-16  0:36                     ` Liang, Liang (Leo)
2021-03-16  0:36                       ` Liang, Liang (Leo)
2021-03-16  6:50                       ` Mike Rapoport
2021-03-16  6:50                         ` Mike Rapoport
2021-03-16  8:00                         ` Liang, Liang (Leo)
2021-03-16  8:00                           ` Liang, Liang (Leo)
2021-03-16  8:37                           ` David Hildenbrand
2021-03-16  8:37                             ` David Hildenbrand
2021-03-16  8:43                             ` Liang, Liang (Leo)
2021-03-16  8:43                               ` Liang, Liang (Leo)
2021-03-16  8:54                               ` David Hildenbrand
2021-03-16  8:54                                 ` David Hildenbrand
2021-03-16  8:58                                 ` Liang, Liang (Leo)
2021-03-16  8:58                                   ` Liang, Liang (Leo)
2021-03-16  9:08                                   ` David Hildenbrand
2021-03-16  9:08                                     ` David Hildenbrand
2021-03-16 10:30                                     ` Mike Rapoport
2021-03-16 10:30                                       ` Mike Rapoport
2021-03-16 11:02                                       ` Liang, Liang (Leo)
2021-03-16 11:02                                         ` Liang, Liang (Leo)
2021-03-16 11:10                                         ` David Hildenbrand
2021-03-16 11:10                                           ` David Hildenbrand

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=22437770-956e-f7b4-a8f6-3f1cc28c3ec2@redhat.com \
    --to=david@redhat.com \
    --cc=Alexander.Deucher@amd.com \
    --cc=Christian.Koenig@amd.com \
    --cc=Liang.Liang@amd.com \
    --cc=Ray.Huang@amd.com \
    --cc=akpm@linux-foundation.org \
    --cc=amd-gfx@lists.freedesktop.org \
    --cc=george.kennedy@oracle.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=rafael@kernel.org \
    --cc=rppt@linux.ibm.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.