Re: very poor ext3 write performance on big filesystems?

From: Paul Slootman <paul@wurtel.net>
To: linux-kernel@vger.kernel.org
Subject: Re: very poor ext3 write performance on big filesystems?
Date: Tue, 19 Feb 2008 14:14:12 +0100	[thread overview]
Message-ID: <20080219131412.GA2255@msgid.wurtel.net> (raw)
In-Reply-To: <9Yg6H-2DJ-23@gated-at.bofh.it>

On Mon 18 Feb 2008, Andi Kleen wrote:
> On Mon, Feb 18, 2008 at 10:16:32AM -0500, Theodore Tso wrote:
> > On Mon, Feb 18, 2008 at 04:02:36PM +0100, Tomasz Chmielewski wrote:
> > > I tried to copy that filesystem once (when it was much smaller) with "rsync 
> > > -a -H", but after 3 days, rsync was still building an index and didn't copy 
> > > any file.
> > 
> > If you're going to copy the whole filesystem don't use rsync! 
> 
> Yes, I managed to kill systems (drive them really badly into oom and
> get very long swap storms) with rsync -H in the past too. Something is very 
> wrong with the rsync implementation of this.

Note that the soon-to-be-released version 3.0.0 of rsync has very much
improved performance here, both in speed and memory usage, thanks to a
new incremental transfer protocol (before it read in the complete file
list, first on the source, then on the target, and then only started to
do the actual work).

Paul Slootman