[PATCH]dquot: do full inode dirty in allocating space

[PATCH]dquot: do full inode dirty in allocating space

[Shaohua Li]

Alex Shi found a regression when doing ffsb test. The test has several threads,
and each thread creates a small file, write to it and then delete it. ffsb
reports about 20% regression and Alex bisected it to 43d2932d88e4. The test
will call __mark_inode_dirty 3 times. without this commit, we only take
inode_lock one time, while with it, we take the lock 3 times with flags (
I_DIRTY_SYNC,I_DIRTY_PAGES,I_DIRTY). Perf shows the lock contention increased
too much. Below proposed patch fixes it.

fs is allocating blocks, which usually means file writes and the inode
will be dirtied soon. We fully dirty the inode to reduce some inode_lock
contention in several calls of __mark_inode_dirty.

Signed-off-by: Shaohua Li <shaohua.li@intel.com>
Signed-off-by: Alex Shi <alex.shi@intel.com>

diff --git a/include/linux/quotaops.h b/include/linux/quotaops.h
index d50ba85..8dd2cc3 100644
--- a/include/linux/quotaops.h
+++ b/include/linux/quotaops.h
@@ -275,7 +275,7 @@ static inline int dquot_alloc_space(struct inode *inode, qsize_t nr)
 
 	ret = dquot_alloc_space_nodirty(inode, nr);
 	if (!ret)
-		mark_inode_dirty_sync(inode);
+		mark_inode_dirty(inode);
 	return ret;
 }
 

Re: [PATCH]dquot: do full inode dirty in allocating space

[Jan Kara]

  Hi,

On Fri 20-08-10 16:49:43, Shaohua Li wrote:
> Alex Shi found a regression when doing ffsb test. The test has several threads,
> and each thread creates a small file, write to it and then delete it. ffsb
> reports about 20% regression and Alex bisected it to 43d2932d88e4. The test
> will call __mark_inode_dirty 3 times. without this commit, we only take
> inode_lock one time, while with it, we take the lock 3 times with flags (
> I_DIRTY_SYNC,I_DIRTY_PAGES,I_DIRTY). Perf shows the lock contention increased
> too much. Below proposed patch fixes it.
  Thanks for the analysis and the patch!  With which filesystem have you
measured the results? And what kind of machine it was?

> fs is allocating blocks, which usually means file writes and the inode
> will be dirtied soon. We fully dirty the inode to reduce some inode_lock
> contention in several calls of __mark_inode_dirty.
  Well, this is rather a workaround for a poor handling of inode dirty
bits. BTW, I think Nick's VFS scalability patches address inode_lock
contention as well so with them the contention you see should be reduced.
  Anyway, I will take this patch for the time before inode_lock
contention improves and add a proper comment about this before
mark_inode_dirty.

								Honza
> 
> Signed-off-by: Shaohua Li <shaohua.li@intel.com>
> Signed-off-by: Alex Shi <alex.shi@intel.com>
> 
> diff --git a/include/linux/quotaops.h b/include/linux/quotaops.h
> index d50ba85..8dd2cc3 100644
> --- a/include/linux/quotaops.h
> +++ b/include/linux/quotaops.h
> @@ -275,7 +275,7 @@ static inline int dquot_alloc_space(struct inode *inode, qsize_t nr)
>  
>  	ret = dquot_alloc_space_nodirty(inode, nr);
>  	if (!ret)
> -		mark_inode_dirty_sync(inode);
> +		mark_inode_dirty(inode);
>  	return ret;
>  }
>  
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

Re: [PATCH]dquot: do full inode dirty in allocating space

[Jan Kara]

[-- Attachment #1: Type: text/plain, Size: 1454 bytes --]

On Tue 24-08-10 16:23:59, Jan Kara wrote:
>   Hi,
> 
> On Fri 20-08-10 16:49:43, Shaohua Li wrote:
> > Alex Shi found a regression when doing ffsb test. The test has several threads,
> > and each thread creates a small file, write to it and then delete it. ffsb
> > reports about 20% regression and Alex bisected it to 43d2932d88e4. The test
> > will call __mark_inode_dirty 3 times. without this commit, we only take
> > inode_lock one time, while with it, we take the lock 3 times with flags (
> > I_DIRTY_SYNC,I_DIRTY_PAGES,I_DIRTY). Perf shows the lock contention increased
> > too much. Below proposed patch fixes it.
>   Thanks for the analysis and the patch!  With which filesystem have you
> measured the results? And what kind of machine it was?
> 
> > fs is allocating blocks, which usually means file writes and the inode
> > will be dirtied soon. We fully dirty the inode to reduce some inode_lock
> > contention in several calls of __mark_inode_dirty.
>   Well, this is rather a workaround for a poor handling of inode dirty
> bits. BTW, I think Nick's VFS scalability patches address inode_lock
> contention as well so with them the contention you see should be reduced.
>   Anyway, I will take this patch for the time before inode_lock
> contention improves and add a proper comment about this before
> mark_inode_dirty.
  Attached is a version of the patch that is in my tree now.

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

[-- Attachment #2: 0001-dquot-do-full-inode-dirty-in-allocating-space.patch --]
[-- Type: text/x-patch, Size: 1774 bytes --]

>From 2a7ad4df051b43a481862fe0afa50539bc7e6b78 Mon Sep 17 00:00:00 2001
From: Shaohua Li <shaohua.li@intel.com>
Date: Fri, 20 Aug 2010 16:49:43 +0800
Subject: [PATCH] dquot: do full inode dirty in allocating space

Alex Shi found a regression when doing ffsb test. The test has several threads,
and each thread creates a small file, write to it and then delete it. ffsb
reports about 20% regression and Alex bisected it to 43d2932d88e4. The test
will call __mark_inode_dirty 3 times. without this commit, we only take
inode_lock one time, while with it, we take the lock 3 times with flags (
I_DIRTY_SYNC,I_DIRTY_PAGES,I_DIRTY). Perf shows the lock contention increased
too much. Below proposed patch fixes it.

fs is allocating blocks, which usually means file writes and the inode
will be dirtied soon. We fully dirty the inode to reduce some inode_lock
contention in several calls of __mark_inode_dirty.

Jan Kara: Added comment.

Signed-off-by: Shaohua Li <shaohua.li@intel.com>
Signed-off-by: Alex Shi <alex.shi@intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
 include/linux/quotaops.h |   10 ++++++++--
 1 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/include/linux/quotaops.h b/include/linux/quotaops.h
index d50ba85..d1a9193 100644
--- a/include/linux/quotaops.h
+++ b/include/linux/quotaops.h
@@ -274,8 +274,14 @@ static inline int dquot_alloc_space(struct inode *inode, qsize_t nr)
 	int ret;
 
 	ret = dquot_alloc_space_nodirty(inode, nr);
-	if (!ret)
-		mark_inode_dirty_sync(inode);
+	if (!ret) {
+		/*
+		 * Mark inode fully dirty. Since we are allocating blocks, inode
+		 * would become fully dirty soon anyway and it reportedly
+		 * reduces inode_lock contention.
+		 */
+		mark_inode_dirty(inode);
+	}
 	return ret;
 }
 
-- 
1.6.4.2

Re: [PATCH]dquot: do full inode dirty in allocating space

[Shaohua Li]

On Tue, Aug 24, 2010 at 10:23:59PM +0800, Jan Kara wrote:
>   Hi,
> 
> On Fri 20-08-10 16:49:43, Shaohua Li wrote:
> > Alex Shi found a regression when doing ffsb test. The test has several threads,
> > and each thread creates a small file, write to it and then delete it. ffsb
> > reports about 20% regression and Alex bisected it to 43d2932d88e4. The test
> > will call __mark_inode_dirty 3 times. without this commit, we only take
> > inode_lock one time, while with it, we take the lock 3 times with flags (
> > I_DIRTY_SYNC,I_DIRTY_PAGES,I_DIRTY). Perf shows the lock contention increased
> > too much. Below proposed patch fixes it.
>   Thanks for the analysis and the patch!  With which filesystem have you
> measured the results? And what kind of machine it was?
it's ext3 and 24 CPU system with two sockets.
 
> > fs is allocating blocks, which usually means file writes and the inode
> > will be dirtied soon. We fully dirty the inode to reduce some inode_lock
> > contention in several calls of __mark_inode_dirty.
>   Well, this is rather a workaround for a poor handling of inode dirty
> bits. BTW, I think Nick's VFS scalability patches address inode_lock
> contention as well so with them the contention you see should be reduced.
yep, the VFS scalability patch might reduce this.

>   Anyway, I will take this patch for the time before inode_lock
> contention improves and add a proper comment about this before
> mark_inode_dirty.
Thanks.