Re: linux-next: Tree for December 20

From: "J.H." <warthog9@kernel.org>
To: sedat.dilek@gmail.com
Cc: Sedat Dilek <sedat.dilek@googlemail.com>,
	Stephen Rothwell <sfr@canb.auug.org.au>,
	linux-next@vger.kernel.org, LKML <linux-kernel@vger.kernel.org>,
	ftpadmin@kernel.org
Subject: Re: linux-next: Tree for December 20
Date: Mon, 20 Dec 2010 10:29:44 -0800	[thread overview]
Message-ID: <4D0FA098.8000405@kernel.org> (raw)
In-Reply-To: <AANLkTin4dvM1B8GkxZH6DADJVg4ZWNSQvUHAviy-4O2J@mail.gmail.com>

On 12/20/2010 04:12 AM, Sedat Dilek wrote:
> On Mon, Dec 20, 2010 at 1:00 PM, Stephen Rothwell <sfr@canb.auug.org.au> wrote:
>> Hi,
>>
>> On Mon, 20 Dec 2010 12:46:08 +0100 Sedat Dilek <sedat.dilek@googlemail.com> wrote:
>>>
>>>>
>>>> [The mirroring on kernel.org is running slowly]
>>>
>>> Indeed, you know what's wrong with the mirrors?
>>
>> I don't know what is wrong.  The mirroring just seems to be much slower
>> in the last week or so.  Today, it was just over three hours from when I
>> uploaded linux-next until it was mirrored out.
>>
>> I have cc'd the admins.
>>
> 
> Thanks.

Gingerbread released last week, loads have been spiking to well above
300 on the frontend machines, RAM usage has been in the ~20G range (on
machines with 16G of RAM), and we've been running an *AVERAGE* of 2000+
simultaneous GIT processes worldwide (normal is in the 100-200 range).

Bonus points that we blew a drive on one of the machines yesterday.

I don't think anyone was quite expecting the onslaught we have gotten
this time around, and I'm going to have to make some changes before the
next big release (mainly shifting Android over to the bigger machines)

>>> Can you upload the diff-patch (patch-v2.6.37-rc6-next-20101220) to [1]
>>> from your local-tree or place it somewhere else?
>>
>> I cannot upload directly to www.kernel.org, I up load to the same place
>> on master.kernel.org and the mirroring system puts it on www.kernel.org.
>>
> 
> I could download the diff-patch.
> 
> $ wget http://www.kernel.org/pub/linux/kernel/v2.6/next/patch-v2.6.37-rc6-next-20101220.bz2

Running a quick set of stats against the logs we keep from the syncing
I'm seeing:

	All of December:
	------------------
	Number of cycles: 5848
	Total amount of time spent syncing: 1544407sec
	Average Sync times: 264sec
	Longest time: 24011sec
	Shortest time: 42sec

	Last 10 days:
	------------------
	Number of cycles: 2144
	Total amount of time spent syncing: 845762sec
	Average Sync times: 394sec
	Longest time: 24011sec
	Shortest time: 45sec

With cyclical patterns of fast and slow syncs.  We seem to be (right
now) in the middle of one of the slow cycles with the last few syncs
taking a couple of hours each.  The culprit, for at least the current
sync, seems to be the box with the highest load and the blown drive,
which I'm also reluctant to disable since it's still handling it's fair
share of the load and dumping all of that onto the opposite US mirror
would not be good either.

- John 'Warthog9' Hawley