Re: [PATCH 0/6 v2] PM / Hibernate: Memory bitmap scalability improvements

From: Joerg Roedel <joro@8bytes.org>
To: Pavel Machek <pavel@ucw.cz>
Cc: "Rafael J. Wysocki" <rjw@rjwysocki.net>,
	Len Brown <len.brown@intel.com>,
	linux-pm@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH 0/6 v2] PM / Hibernate: Memory bitmap scalability improvements
Date: Mon, 21 Jul 2014 14:36:13 +0200	[thread overview]
Message-ID: <20140721123613.GL30979@8bytes.org> (raw)
In-Reply-To: <20140721120053.GA14069@amd.pavel.ucw.cz>

Hi,

On Mon, Jul 21, 2014 at 02:00:53PM +0200, Pavel Machek wrote:
> > These patches improve the data structure by adding a radix
> > tree to the linked list structure to improve random access
> > performance from O(n) to O(log_b(n)), where b depends on the
> > architecture (b=512 on amd64, 1024 in i386).
> 
> Why are we doing random access there?

Mostly because the bits in the bitmaps need to be set, cleared and
tested, basically at every place that calls one of the swsusp_*page*
functions.

> Is the improvement from fact that normally very little memory is used
> on big memory machines?

That is one part of the optimzation, yes. As I wrote in Patch 6 the
worst-case performance is still the same as with the old implementation,
hence it is still necessary to touch the soft lockup watchdog.

> Actually... how long does it take to hibernate 12TB machine? That
> should be many hours, right? You just can't hibernate machine that
> big.

Sorry, I didn't run the tests on these big machines myself as I don't
have access to them, I relied on our partner to do that and report back
the results.

> > The last patch adds touching the soft lockup watchdog in
> > rtree_next_node. This is necessary because the worst case
> > performance (all bits set in the forbidden_pages_map and
> > free_pages_map) is the same as with the old implementation
> > and may still cause soft lockups. Patch 6 avoids this.
> 
> Ok, so what about simpler patch? Just touch the watchdog?

That would just cover the problem that the bitmap data structure and the
algorithm in swsusp_free do not scale well on bigmem machines.

If you want to test the correctnes of these patches yourself, you can
test with only patches 1-4. This will run hibernate with both bitmap
implementations in parallel and trigger a WARN_ON_ONCE when any
difference is found.

I did that with different configurations (64 and 32 bit kernel, in KVM
and on real hardware) and found no issues with only patches 1-4.

> Additional 70 seconds will be lost in noise if you write 12TB of RAM
> to (even quite fast) disk.

Sure, but you would still get the soft lockup warnings when swsusp_free
runs in the end.

	Joerg