From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751189AbWBWM5A (ORCPT ); Thu, 23 Feb 2006 07:57:00 -0500 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1751198AbWBWM47 (ORCPT ); Thu, 23 Feb 2006 07:56:59 -0500 Received: from omx1-ext.sgi.com ([192.48.179.11]:10947 "EHLO omx1.americas.sgi.com") by vger.kernel.org with ESMTP id S1751258AbWBWM47 (ORCPT ); Thu, 23 Feb 2006 07:56:59 -0500 Date: Thu, 23 Feb 2006 06:56:39 -0600 From: Robin Holt To: Andrew Morton Cc: Robin Holt , john@johnmccutchan.com, linux-kernel@vger.kernel.org, rml@novell.com, arnd@arndb.de, hch@lst.de Subject: Re: udevd is killing file write performance. Message-ID: <20060223125639.GB25007@lnx-holt.americas.sgi.com> References: <20060222134250.GE20786@lnx-holt.americas.sgi.com> <1140626903.13461.5.camel@localhost.localdomain> <20060222175030.GB30556@lnx-holt.americas.sgi.com> <20060222120547.2ae23a16.akpm@osdl.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20060222120547.2ae23a16.akpm@osdl.org> User-Agent: Mutt/1.4.2.1i Sender: linux-kernel-owner@vger.kernel.org X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Feb 22, 2006 at 12:05:47PM -0800, Andrew Morton wrote: > Robin Holt wrote: > > > > Let me reiterate, I know _VERY_ little about filesystems. Can the > > dentry->d_lock be changed to a read/write lock? > > Well, it could, but I suspect that won't help - the hold times in there > will be very short so the problem is more likely acquisition frequency. > > However it's a bit strange that this function is the bottleneck. If their > workload is doing large numbers of reads or writes from large numbers of > processes against the same file then they should be hitting heavy > contention on other locks, such as i_sem and/or tree_lock and/or lru_lock > and others. > > Can you tell us more about the kernel-visible behaviour of this app? I looked at a little more of the output. All I have to go on is a few back traces generated by kdb of the entire system a few seconds apart. In all of the traces, the first chunk of cpus are the only ones doing writes. I have not counted exactly, but I think it is around 32. There may be more or less, but that is the feeling (sometimes they are doing reads as well). The remainder of the cpus seem to be doing seeks to various parts of the file and doing reads. The seeks+read seem to overlap as I see a nearly uniform read of 64k, but the seeks seem to be 4k aligned. Also, at the time the snapshots were taken, they all are fairly early in the file (between about 15% and 35% of the file) where the writes are happening toward to early part of the second half (50% - 70%). I no longer have access to the test machine. Since we told them to turn off udevd, they no longer see a problem. I will try to convince them to allow some more testing. I will also try to recreate on our machine as well. Thanks, Robin