* Re: 5.7-rc0: kswapd eats cpu during a disk test?!
[not found] <20200531103431.GA28429@amd>
@ 2020-06-01 8:55 ` Vlastimil Babka
2020-06-12 22:45 ` Pavel Machek
0 siblings, 1 reply; 6+ messages in thread
From: Vlastimil Babka @ 2020-06-01 8:55 UTC (permalink / raw)
To: Pavel Machek, kernel list, Andrew Morton; +Cc: mhocko, linux-mm
+CC linux-mm
On 5/31/20 12:34 PM, Pavel Machek wrote:
> Hi!
>
> This is simple cat /dev/sda > /dev/zero... on thinkpad x60 (x86-32),
> with spinning rust.
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 1000 root 20 0 0 0 0 R 53.3 0.0 57:34.93 kswapd0
> 27897 root 20 0 6976 580 536 R 44.5 0.0 1:44.53 cat
>
> It keeps both CPUs busy... and I don't think that's right.
Does an older kernel behave differently here?
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 5.7-rc0: kswapd eats cpu during a disk test?!
2020-06-01 8:55 ` 5.7-rc0: kswapd eats cpu during a disk test?! Vlastimil Babka
@ 2020-06-12 22:45 ` Pavel Machek
2020-06-12 23:05 ` Pavel Machek
0 siblings, 1 reply; 6+ messages in thread
From: Pavel Machek @ 2020-06-12 22:45 UTC (permalink / raw)
To: Vlastimil Babka; +Cc: kernel list, Andrew Morton, mhocko, linux-mm
[-- Attachment #1: Type: text/plain, Size: 1909 bytes --]
Hi!
> +CC linux-mm
>
> On 5/31/20 12:34 PM, Pavel Machek wrote:
> > Hi!
> >
> > This is simple cat /dev/sda > /dev/zero... on thinkpad x60 (x86-32),
> > with spinning rust.
> >
> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> > 1000 root 20 0 0 0 0 R 53.3 0.0 57:34.93 kswapd0
> > 27897 root 20 0 6976 580 536 R 44.5 0.0 1:44.53 cat
> >
> > It keeps both CPUs busy... and I don't think that's right.
>
> Does an older kernel behave differently here?
Let me try on x220 (x86-64, first):
737 root 20 0 5404 744 680 R 31.2 0.0 0:09.98 cat
1024 root 20 0 0 0 0 S 21.4 0.0 165:22.68 kswapd0
That was with ssd, result with spinning rust is similar:
859 root 20 0 5404 740 672 D 21.1 0.0 0:03.33 cat
1024 root 20 0 0 0 0 R 11.8 0.0 165:33.07 kswapd0
5.7-rc1+ kernel.
Performance of spinning rust is down, too, on x60:
pavel@amd:~/misc/hw/hdd1t$ sudo ddrescue --force /dev/sda1 /dev/null
GNU ddrescue 1.19
Press Ctrl-C to interrupt
rescued: 2147 MB, errsize: 0 B, current rate: 3080 kB/s
ipos: 2147 MB, errors: 0, average rate: 5382 kB/s
opos: 2147 MB, run time: 6.65 m, successful read:
0 s ago
Finished
pavel@amd:~/misc/hw/hdd1t$ uname -a
Linux amd 5.7.0-next-20200611+ #123 SMP PREEMPT Thu Jun 11
15:41:22 CEST 2020 i686 GNU/Linux
And there's something clearly wrong here:
966 root 20 0 0 0 0 R 94.4 0.0 8:18.82 kswapd0
23933 root 20 0 4612 1112 1028 D 80.6 0.0 0:26.40 ddrescue
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 5.7-rc0: kswapd eats cpu during a disk test?!
2020-06-12 22:45 ` Pavel Machek
@ 2020-06-12 23:05 ` Pavel Machek
2020-06-13 4:47 ` Hillf Danton
0 siblings, 1 reply; 6+ messages in thread
From: Pavel Machek @ 2020-06-12 23:05 UTC (permalink / raw)
To: Vlastimil Babka; +Cc: kernel list, Andrew Morton, mhocko, linux-mm
[-- Attachment #1: Type: text/plain, Size: 2585 bytes --]
Hi!
> > +CC linux-mm
> >
> > On 5/31/20 12:34 PM, Pavel Machek wrote:
> > > Hi!
> > >
> > > This is simple cat /dev/sda > /dev/zero... on thinkpad x60 (x86-32),
> > > with spinning rust.
> > >
> > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> > > 1000 root 20 0 0 0 0 R 53.3 0.0 57:34.93 kswapd0
> > > 27897 root 20 0 6976 580 536 R 44.5 0.0 1:44.53 cat
> > >
> > > It keeps both CPUs busy... and I don't think that's right.
> >
> > Does an older kernel behave differently here?
>
> Let me try on x220 (x86-64, first):
>
> 737 root 20 0 5404 744 680 R 31.2 0.0 0:09.98 cat
> 1024 root 20 0 0 0 0 S 21.4 0.0 165:22.68 kswapd0
>
> That was with ssd, result with spinning rust is similar:
>
> 859 root 20 0 5404 740 672 D 21.1 0.0 0:03.33 cat
> 1024 root 20 0 0 0 0 R 11.8 0.0 165:33.07 kswapd0
>
> 5.7-rc1+ kernel.
>
> Performance of spinning rust is down, too, on x60:
>
> pavel@amd:~/misc/hw/hdd1t$ sudo ddrescue --force /dev/sda1 /dev/null
> GNU ddrescue 1.19
> Press Ctrl-C to interrupt
> rescued: 2147 MB, errsize: 0 B, current rate: 3080 kB/s
> ipos: 2147 MB, errors: 0, average rate: 5382 kB/s
> opos: 2147 MB, run time: 6.65 m, successful read:
> 0 s ago
> Finished
> pavel@amd:~/misc/hw/hdd1t$ uname -a
> Linux amd 5.7.0-next-20200611+ #123 SMP PREEMPT Thu Jun 11
> 15:41:22 CEST 2020 i686 GNU/Linux
>
> And there's something clearly wrong here:
>
> 966 root 20 0 0 0 0 R 94.4 0.0 8:18.82 kswapd0
> 23933 root 20 0 4612 1112 1028 D 80.6 0.0 0:26.40 ddrescue
>
Same x60 under older kernel:
pavel@amd:/data/fast/pavel$ sudo ddrescue --force /dev/sda4 /dev/null
GNU ddrescue 1.19
Press Ctrl-C to interrupt
rescued: 6593 MB, errsize: 0 B, current rate: 60424 kB/s
ipos: 6593 MB, errors: 0, average rate: 95563 kB/s
3539 root 20 0 4616 1136 1048 D 21.4 0.0 0:15.63 ddrescue
865 root 20 0 0 0 0 S 6.9 0.0 0:04.91 kswapd0
Linux amd 4.6.0+ #172 SMP Sun Aug 14 11:25:34 CEST 2016 i686 GNU/Linux
These are more reasonable numbers.
Pavel
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 5.7-rc0: kswapd eats cpu during a disk test?!
2020-06-12 23:05 ` Pavel Machek
@ 2020-06-13 4:47 ` Hillf Danton
2020-06-13 6:48 ` Pavel Machek
0 siblings, 1 reply; 6+ messages in thread
From: Hillf Danton @ 2020-06-13 4:47 UTC (permalink / raw)
To: Pavel Machek
Cc: Vlastimil Babka, kernel list, Andrew Morton, mhocko,
Hillf Danton, linux mm
On Sat, 13 Jun 2020 01:05:52 +0200 Pavel Machek wrote:
> > > +CC linux-mm
> > >
> > > On 5/31/20 12:34 PM, Pavel Machek wrote:
> > > > Hi!
> > > >
> > > > This is simple cat /dev/sda > /dev/zero... on thinkpad x60 (x86-32),
> > > > with spinning rust.
> > > >
> > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> > > > 1000 root 20 0 0 0 0 R 53.3 0.0 57:34.93 kswapd0
> > > > 27897 root 20 0 6976 580 536 R 44.5 0.0 1:44.53 cat
> > > >
> > > > It keeps both CPUs busy... and I don't think that's right.
> > >
> > > Does an older kernel behave differently here?
> >
> > Let me try on x220 (x86-64, first):
> >
> > 737 root 20 0 5404 744 680 R 31.2 0.0 0:09.98 cat
> > 1024 root 20 0 0 0 0 S 21.4 0.0 165:22.68 kswapd0
> >
> > That was with ssd, result with spinning rust is similar:
> >
> > 859 root 20 0 5404 740 672 D 21.1 0.0 0:03.33 cat
> > 1024 root 20 0 0 0 0 R 11.8 0.0 165:33.07 kswapd0
> >
> > 5.7-rc1+ kernel.
> >
> > Performance of spinning rust is down, too, on x60:
> >
> > pavel@amd:~/misc/hw/hdd1t$ sudo ddrescue --force /dev/sda1 /dev/null
> > GNU ddrescue 1.19
> > Press Ctrl-C to interrupt
> > rescued: 2147 MB, errsize: 0 B, current rate: 3080 kB/s
> > ipos: 2147 MB, errors: 0, average rate: 5382 kB/s
> > opos: 2147 MB, run time: 6.65 m, successful read:
> > 0 s ago
> > Finished
> > pavel@amd:~/misc/hw/hdd1t$ uname -a
> > Linux amd 5.7.0-next-20200611+ #123 SMP PREEMPT Thu Jun 11
> > 15:41:22 CEST 2020 i686 GNU/Linux
> >
> > And there's something clearly wrong here:
> >
> > 966 root 20 0 0 0 0 R 94.4 0.0 8:18.82 kswapd0
> > 23933 root 20 0 4612 1112 1028 D 80.6 0.0 0:26.40 ddrescue
> >
>
> Same x60 under older kernel:
>
> pavel@amd:/data/fast/pavel$ sudo ddrescue --force /dev/sda4 /dev/null
> GNU ddrescue 1.19
> Press Ctrl-C to interrupt
> rescued: 6593 MB, errsize: 0 B, current rate: 60424 kB/s
> ipos: 6593 MB, errors: 0, average rate: 95563 kB/s
>
> 3539 root 20 0 4616 1136 1048 D 21.4 0.0 0:15.63 ddrescue
> 865 root 20 0 0 0 0 S 6.9 0.0 0:04.91 kswapd0
>
> Linux amd 4.6.0+ #172 SMP Sun Aug 14 11:25:34 CEST 2016 i686 GNU/Linux
>
> These are more reasonable numbers.
Treat referenced & active pages as reclaim cost.
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2020,6 +2020,7 @@ static void shrink_active_list(unsigned
struct page *page;
unsigned nr_deactivate, nr_activate;
unsigned nr_rotated = 0;
+ unsigned nr_refered = 0;
int file = is_file_lru(lru);
struct pglist_data *pgdat = lruvec_pgdat(lruvec);
@@ -2070,7 +2071,8 @@ static void shrink_active_list(unsigned
nr_rotated += hpage_nr_pages(page);
list_add(&page->lru, &l_active);
continue;
- }
+ } else if (!file)
+ nr_refered++;
}
ClearPageActive(page); /* we are de-activating */
@@ -2098,6 +2100,14 @@ static void shrink_active_list(unsigned
free_unref_page_list(&l_active);
trace_mm_vmscan_lru_shrink_active(pgdat->node_id, nr_taken, nr_activate,
nr_deactivate, nr_rotated, sc->priority, file);
+ if (file)
+ sc->file_cost += nr_rotated;
+ else
+ /*
+ * add cost to avoid swapin in the near future which incurs IO
+ * on top of reclaim
+ */
+ sc->anon_cost += nr_refered;
}
unsigned long reclaim_pages(struct list_head *page_list)
@@ -2311,11 +2321,13 @@ static void get_scan_count(struct lruvec
file_cost = total_cost + sc->file_cost;
total_cost = anon_cost + file_cost;
- ap = swappiness * (total_cost + 1);
- ap /= anon_cost + 1;
-
- fp = (200 - swappiness) * (total_cost + 1);
- fp /= file_cost + 1;
+ ap = swappiness * total_cost;
+ if (anon_cost)
+ ap /= anon_cost;
+
+ fp = (200 - swappiness) * total_cost;
+ if (file_cost)
+ fp /= file_cost;
fraction[0] = ap;
fraction[1] = fp;
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 5.7-rc0: kswapd eats cpu during a disk test?!
2020-06-13 4:47 ` Hillf Danton
@ 2020-06-13 6:48 ` Pavel Machek
2020-06-13 9:43 ` Hillf Danton
0 siblings, 1 reply; 6+ messages in thread
From: Pavel Machek @ 2020-06-13 6:48 UTC (permalink / raw)
To: Hillf Danton
Cc: Vlastimil Babka, kernel list, Andrew Morton, mhocko, linux mm
[-- Attachment #1: Type: text/plain, Size: 4006 bytes --]
Hi!
> > > Performance of spinning rust is down, too, on x60:
> > >
> > > pavel@amd:~/misc/hw/hdd1t$ sudo ddrescue --force /dev/sda1 /dev/null
> > > GNU ddrescue 1.19
> > > Press Ctrl-C to interrupt
> > > rescued: 2147 MB, errsize: 0 B, current rate: 3080 kB/s
> > > ipos: 2147 MB, errors: 0, average rate: 5382 kB/s
> > > opos: 2147 MB, run time: 6.65 m, successful read:
> > > 0 s ago
> > > Finished
> > > pavel@amd:~/misc/hw/hdd1t$ uname -a
> > > Linux amd 5.7.0-next-20200611+ #123 SMP PREEMPT Thu Jun 11
> > > 15:41:22 CEST 2020 i686 GNU/Linux
> > >
> > > And there's something clearly wrong here:
> > >
> > > 966 root 20 0 0 0 0 R 94.4 0.0 8:18.82 kswapd0
> > > 23933 root 20 0 4612 1112 1028 D 80.6 0.0 0:26.40 ddrescue
> > >
> >
> > Same x60 under older kernel:
> >
> > pavel@amd:/data/fast/pavel$ sudo ddrescue --force /dev/sda4 /dev/null
> > GNU ddrescue 1.19
> > Press Ctrl-C to interrupt
> > rescued: 6593 MB, errsize: 0 B, current rate: 60424 kB/s
> > ipos: 6593 MB, errors: 0, average rate: 95563 kB/s
> >
> > 3539 root 20 0 4616 1136 1048 D 21.4 0.0 0:15.63 ddrescue
> > 865 root 20 0 0 0 0 S 6.9 0.0 0:04.91 kswapd0
> >
> > Linux amd 4.6.0+ #172 SMP Sun Aug 14 11:25:34 CEST 2016 i686 GNU/Linux
> >
> > These are more reasonable numbers.
>
> Treat referenced & active pages as reclaim cost.
That helps a bit, but we are still nowehere near v4.6 performance:
pavel@amd:/data/fast/pavel$ sudo ddrescue --force /dev/sda2 /dev/null
GNU ddrescue 1.19
Press Ctrl-C to interrupt
rescued: 8559 MB, errsize: 0 B, current rate: 53084 kB/s
ipos: 8559 MB, errors: 0, average rate: 71327 kB/s
3933 root 20 0 4612 1216 1128 R 68.6 0.0 0:12.14 ddrescue
967 root 20 0 0 0 0 R 64.4 0.0 1:18.67 kswapd0
(I applied the patch on 5.7.0-next-20200611+).
Do we know which original change is causing this? Any other ideas?
Thanks,
Pavel
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -2020,6 +2020,7 @@ static void shrink_active_list(unsigned
> struct page *page;
> unsigned nr_deactivate, nr_activate;
> unsigned nr_rotated = 0;
> + unsigned nr_refered = 0;
> int file = is_file_lru(lru);
> struct pglist_data *pgdat = lruvec_pgdat(lruvec);
>
> @@ -2070,7 +2071,8 @@ static void shrink_active_list(unsigned
> nr_rotated += hpage_nr_pages(page);
> list_add(&page->lru, &l_active);
> continue;
> - }
> + } else if (!file)
> + nr_refered++;
> }
>
> ClearPageActive(page); /* we are de-activating */
> @@ -2098,6 +2100,14 @@ static void shrink_active_list(unsigned
> free_unref_page_list(&l_active);
> trace_mm_vmscan_lru_shrink_active(pgdat->node_id, nr_taken, nr_activate,
> nr_deactivate, nr_rotated, sc->priority, file);
> + if (file)
> + sc->file_cost += nr_rotated;
> + else
> + /*
> + * add cost to avoid swapin in the near future which incurs IO
> + * on top of reclaim
> + */
> + sc->anon_cost += nr_refered;
> }
>
> unsigned long reclaim_pages(struct list_head *page_list)
> @@ -2311,11 +2321,13 @@ static void get_scan_count(struct lruvec
> file_cost = total_cost + sc->file_cost;
> total_cost = anon_cost + file_cost;
>
> - ap = swappiness * (total_cost + 1);
> - ap /= anon_cost + 1;
> -
> - fp = (200 - swappiness) * (total_cost + 1);
> - fp /= file_cost + 1;
> + ap = swappiness * total_cost;
> + if (anon_cost)
> + ap /= anon_cost;
> +
> + fp = (200 - swappiness) * total_cost;
> + if (file_cost)
> + fp /= file_cost;
>
> fraction[0] = ap;
> fraction[1] = fp;
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 5.7-rc0: kswapd eats cpu during a disk test?!
2020-06-13 6:48 ` Pavel Machek
@ 2020-06-13 9:43 ` Hillf Danton
0 siblings, 0 replies; 6+ messages in thread
From: Hillf Danton @ 2020-06-13 9:43 UTC (permalink / raw)
To: Pavel Machek
Cc: Vlastimil Babka, Markus Elfring, kernel list, Andrew Morton,
mhocko, linux mm
Hi
On Sat, 13 Jun 2020 08:48:41 +0200 Pavel Machek wrote:
>
> Do we know which original change is causing this? Any other ideas?
Take a look at 264e90cc07f1 ("mm: only count actual rotations as
LRU reclaim cost") please.
Hillf
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2020-06-13 9:43 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <20200531103431.GA28429@amd>
2020-06-01 8:55 ` 5.7-rc0: kswapd eats cpu during a disk test?! Vlastimil Babka
2020-06-12 22:45 ` Pavel Machek
2020-06-12 23:05 ` Pavel Machek
2020-06-13 4:47 ` Hillf Danton
2020-06-13 6:48 ` Pavel Machek
2020-06-13 9:43 ` Hillf Danton
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).