[PATCH 0/2] revert changes to zcache_do_preload()

[PATCH 0/2] revert changes to zcache_do_preload()

[Seth Jennings]

This patchset fixes a regression in 3.6 by reverting two dependent
commits that made changes to zcache_do_preload().

The commits undermine an assumption made by tmem_put() in
the cleancache path that preemption is disabled.  This change
introduces a race condition that can result in the wrong page
being returned by tmem_get(), causing assorted errors (segfaults,
apparent file corruption, etc) in userspace.

The corruption was discussed in this thread:
https://lkml.org/lkml/2012/8/17/494

Please apply this patchset to 3.6.  This problem didn't exist
in previous releases so nothing need be done for the stable trees.

Seth Jennings (2):
  Revert "staging: zcache: cleanup zcache_do_preload and
    zcache_put_page"
  Revert "staging: zcache: optimize zcache_do_preload"

 drivers/staging/zcache/zcache-main.c |   54 +++++++++++++++++++---------------
 1 file changed, 31 insertions(+), 23 deletions(-)

-- 
1.7.9.5

[PATCH 1/2] Revert "staging: zcache: cleanup zcache_do_preload and zcache_put_page"

[Seth Jennings]

This reverts commit b71f3bcc5ab5e76a22d7ad82b3795602fcf0e0af.

This commit is resulting  memory corruption in the cleancache case

Signed-off-by: Seth Jennings <sjenning@linux.vnet.ibm.com>
Reported-by: Dan Magenheimer <dan.magenheimer@oracle.com>
---
 drivers/staging/zcache/zcache-main.c |   37 +++++++++++++++++++---------------
 1 file changed, 21 insertions(+), 16 deletions(-)

diff --git a/drivers/staging/zcache/zcache-main.c b/drivers/staging/zcache/zcache-main.c
index c214977..8a335b9 100644
--- a/drivers/staging/zcache/zcache-main.c
+++ b/drivers/staging/zcache/zcache-main.c
@@ -1048,24 +1048,29 @@ static int zcache_do_preload(struct tmem_pool *pool)
 		kp->objnodes[kp->nr++] = objnode;
 	}
 
-	if (!kp->obj) {
-		obj = kmem_cache_alloc(zcache_obj_cache, ZCACHE_GFP_MASK);
-		if (unlikely(obj == NULL)) {
-			zcache_failed_alloc++;
-			goto out;
-		}
-		kp->obj = obj;
+	obj = kmem_cache_alloc(zcache_obj_cache, ZCACHE_GFP_MASK);
+	if (unlikely(obj == NULL)) {
+		zcache_failed_alloc++;
+		goto out;
 	}
 
-	if (!kp->page) {
-		page = (void *)__get_free_page(ZCACHE_GFP_MASK);
-		if (unlikely(page == NULL)) {
-			zcache_failed_get_free_pages++;
-			goto out;
-		}
-		kp->page =  page;
+	page = (void *)__get_free_page(ZCACHE_GFP_MASK);
+	if (unlikely(page == NULL)) {
+		zcache_failed_get_free_pages++;
+		kmem_cache_free(zcache_obj_cache, obj);
+		goto out;
 	}
 
+	if (kp->obj == NULL)
+		kp->obj = obj;
+	else
+		kmem_cache_free(zcache_obj_cache, obj);
+
+	if (kp->page == NULL)
+		kp->page = page;
+	else
+		free_page((unsigned long)page);
+
 	ret = 0;
 out:
 	return ret;
@@ -1575,14 +1580,14 @@ static int zcache_put_page(int cli_id, int pool_id, struct tmem_oid *oidp,
 			else
 				zcache_failed_pers_puts++;
 		}
+		zcache_put_pool(pool);
 	} else {
 		zcache_put_to_flush++;
 		if (atomic_read(&pool->obj_count) > 0)
 			/* the put fails whether the flush succeeds or not */
 			(void)tmem_flush_page(pool, oidp, index);
+		zcache_put_pool(pool);
 	}
-
-	zcache_put_pool(pool);
 out:
 	return ret;
 }
-- 
1.7.9.5

[PATCH 2/2] Revert "staging: zcache: optimize zcache_do_preload"

[Seth Jennings]

This reverts commit 79c0d92c5b6175c1462fbe38bf44180f325aa478.

This commit is resulting  memory corruption in the cleancache case

Signed-off-by: Seth Jennings <sjenning@linux.vnet.ibm.com>
Reported-by: Dan Magenheimer <dan.magenheimer@oracle.com>
---
 drivers/staging/zcache/zcache-main.c |   21 ++++++++++++---------
 1 file changed, 12 insertions(+), 9 deletions(-)

diff --git a/drivers/staging/zcache/zcache-main.c b/drivers/staging/zcache/zcache-main.c
index 8a335b9..4f92d87 100644
--- a/drivers/staging/zcache/zcache-main.c
+++ b/drivers/staging/zcache/zcache-main.c
@@ -1034,43 +1034,45 @@ static int zcache_do_preload(struct tmem_pool *pool)
 		goto out;
 	if (unlikely(zcache_obj_cache == NULL))
 		goto out;
-
-	/* IRQ has already been disabled. */
+	preempt_disable();
 	kp = &__get_cpu_var(zcache_preloads);
 	while (kp->nr < ARRAY_SIZE(kp->objnodes)) {
+		preempt_enable_no_resched();
 		objnode = kmem_cache_alloc(zcache_objnode_cache,
 				ZCACHE_GFP_MASK);
 		if (unlikely(objnode == NULL)) {
 			zcache_failed_alloc++;
 			goto out;
 		}
-
-		kp->objnodes[kp->nr++] = objnode;
+		preempt_disable();
+		kp = &__get_cpu_var(zcache_preloads);
+		if (kp->nr < ARRAY_SIZE(kp->objnodes))
+			kp->objnodes[kp->nr++] = objnode;
+		else
+			kmem_cache_free(zcache_objnode_cache, objnode);
 	}
-
+	preempt_enable_no_resched();
 	obj = kmem_cache_alloc(zcache_obj_cache, ZCACHE_GFP_MASK);
 	if (unlikely(obj == NULL)) {
 		zcache_failed_alloc++;
 		goto out;
 	}
-
 	page = (void *)__get_free_page(ZCACHE_GFP_MASK);
 	if (unlikely(page == NULL)) {
 		zcache_failed_get_free_pages++;
 		kmem_cache_free(zcache_obj_cache, obj);
 		goto out;
 	}
-
+	preempt_disable();
+	kp = &__get_cpu_var(zcache_preloads);
 	if (kp->obj == NULL)
 		kp->obj = obj;
 	else
 		kmem_cache_free(zcache_obj_cache, obj);
-
 	if (kp->page == NULL)
 		kp->page = page;
 	else
 		free_page((unsigned long)page);
-
 	ret = 0;
 out:
 	return ret;
@@ -1581,6 +1583,7 @@ static int zcache_put_page(int cli_id, int pool_id, struct tmem_oid *oidp,
 				zcache_failed_pers_puts++;
 		}
 		zcache_put_pool(pool);
+		preempt_enable_no_resched();
 	} else {
 		zcache_put_to_flush++;
 		if (atomic_read(&pool->obj_count) > 0)
-- 
1.7.9.5

Re: [PATCH 0/2] revert changes to zcache_do_preload()

[Minchan Kim]

Hi Seth,

On Thu, Aug 23, 2012 at 10:33:09AM -0500, Seth Jennings wrote:
> This patchset fixes a regression in 3.6 by reverting two dependent
> commits that made changes to zcache_do_preload().
> 
> The commits undermine an assumption made by tmem_put() in
> the cleancache path that preemption is disabled.  This change
> introduces a race condition that can result in the wrong page
> being returned by tmem_get(), causing assorted errors (segfaults,
> apparent file corruption, etc) in userspace.
> 
> The corruption was discussed in this thread:
> https://lkml.org/lkml/2012/8/17/494

I think changelog isn't enough to explain what's the race.
Could you write it down in detail?

And you should Cc'ed Xiao who is author of reverted patch.

> 
> Please apply this patchset to 3.6.  This problem didn't exist
> in previous releases so nothing need be done for the stable trees.
> 
> Seth Jennings (2):
>   Revert "staging: zcache: cleanup zcache_do_preload and
>     zcache_put_page"
>   Revert "staging: zcache: optimize zcache_do_preload"
> 
>  drivers/staging/zcache/zcache-main.c |   54 +++++++++++++++++++---------------
>  1 file changed, 31 insertions(+), 23 deletions(-)
> 
> -- 
> 1.7.9.5
> 

Re: [PATCH 0/2] revert changes to zcache_do_preload()

[Seth Jennings]

On 08/23/2012 03:56 PM, Minchan Kim wrote:
> Hi Seth,
> 
> On Thu, Aug 23, 2012 at 10:33:09AM -0500, Seth Jennings wrote:
>> This patchset fixes a regression in 3.6 by reverting two dependent
>> commits that made changes to zcache_do_preload().
>>
>> The commits undermine an assumption made by tmem_put() in
>> the cleancache path that preemption is disabled.  This change
>> introduces a race condition that can result in the wrong page
>> being returned by tmem_get(), causing assorted errors (segfaults,
>> apparent file corruption, etc) in userspace.
>>
>> The corruption was discussed in this thread:
>> https://lkml.org/lkml/2012/8/17/494
> 
> I think changelog isn't enough to explain what's the race.
> Could you write it down in detail?

I didn't come upon this solution via code inspection, but
rather through discovering that the issue didn't exist in
v3.5 and just looking at the changes since then.

> And you should Cc'ed Xiao who is author of reverted patch.

Thanks for adding Xiao.  I meant to do this. For some reason
I thought that you submitted that patchset :-/
My bad.

Seth

Re: [PATCH 0/2] revert changes to zcache_do_preload()

[Minchan Kim]

On Thu, Aug 23, 2012 at 05:10:00PM -0500, Seth Jennings wrote:
> On 08/23/2012 03:56 PM, Minchan Kim wrote:
> > Hi Seth,
> > 
> > On Thu, Aug 23, 2012 at 10:33:09AM -0500, Seth Jennings wrote:
> >> This patchset fixes a regression in 3.6 by reverting two dependent
> >> commits that made changes to zcache_do_preload().
> >>
> >> The commits undermine an assumption made by tmem_put() in
> >> the cleancache path that preemption is disabled.  This change
> >> introduces a race condition that can result in the wrong page
> >> being returned by tmem_get(), causing assorted errors (segfaults,
> >> apparent file corruption, etc) in userspace.
> >>
> >> The corruption was discussed in this thread:
> >> https://lkml.org/lkml/2012/8/17/494
> > 
> > I think changelog isn't enough to explain what's the race.
> > Could you write it down in detail?
> 
> I didn't come upon this solution via code inspection, but
> rather through discovering that the issue didn't exist in
> v3.5 and just looking at the changes since then.

Okay, then, why do you think the patchsets are culprit?
I didn't look the cleanup patch series of Xiao at that time
so I can be wrong but as I just look through patch of
"zcache: optimize zcache_do_preload", I can't find any fault
because zcache_put_page checks irq_disable so we don't need
to disable preemption so it seems that patch is correct to me.
If the race happens by preemption, BUG_ON in zcache_put_page
should catch it.

What do you mean? Do you have any clue in your mind?

        The commits undermine an assumption made by tmem_put() in
        the cleancache path that preemption is disabled.

> 
> > And you should Cc'ed Xiao who is author of reverted patch.
> 
> Thanks for adding Xiao.  I meant to do this. For some reason
> I thought that you submitted that patchset :-/

Even, I didn't notice that patchset at that time. :)

> My bad.
> 
> Seth
> 
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

-- 
Kind regards,
Minchan Kim

Re: [PATCH 0/2] revert changes to zcache_do_preload()

[Xiao Guangrong]

On 08/24/2012 07:28 AM, Minchan Kim wrote:
> On Thu, Aug 23, 2012 at 05:10:00PM -0500, Seth Jennings wrote:
>> On 08/23/2012 03:56 PM, Minchan Kim wrote:
>>> Hi Seth,
>>>
>>> On Thu, Aug 23, 2012 at 10:33:09AM -0500, Seth Jennings wrote:
>>>> This patchset fixes a regression in 3.6 by reverting two dependent
>>>> commits that made changes to zcache_do_preload().
>>>>
>>>> The commits undermine an assumption made by tmem_put() in
>>>> the cleancache path that preemption is disabled.  This change
>>>> introduces a race condition that can result in the wrong page
>>>> being returned by tmem_get(), causing assorted errors (segfaults,
>>>> apparent file corruption, etc) in userspace.
>>>>
>>>> The corruption was discussed in this thread:
>>>> https://lkml.org/lkml/2012/8/17/494
>>>
>>> I think changelog isn't enough to explain what's the race.
>>> Could you write it down in detail?
>>
>> I didn't come upon this solution via code inspection, but
>> rather through discovering that the issue didn't exist in
>> v3.5 and just looking at the changes since then.
> 
> Okay, then, why do you think the patchsets are culprit?
> I didn't look the cleanup patch series of Xiao at that time
> so I can be wrong but as I just look through patch of
> "zcache: optimize zcache_do_preload", I can't find any fault
> because zcache_put_page checks irq_disable so we don't need
> to disable preemption so it seems that patch is correct to me.
> If the race happens by preemption, BUG_ON in zcache_put_page
> should catch it.

Confused me too!

And the first patch just do the cleanup, it is not different
before the patch and after the patch, what i missed?

Re: [PATCH 0/2] revert changes to zcache_do_preload()

[Seth Jennings]

On 08/23/2012 06:28 PM, Minchan Kim wrote:
> Okay, then, why do you think the patchsets are culprit?
> I didn't look the cleanup patch series of Xiao at that time
> so I can be wrong but as I just look through patch of
> "zcache: optimize zcache_do_preload", I can't find any fault
> because zcache_put_page checks irq_disable so we don't need
> to disable preemption so it seems that patch is correct to me.
> If the race happens by preemption, BUG_ON in zcache_put_page
> should catch it.
> 
> What do you mean? Do you have any clue in your mind?
> 
>         The commits undermine an assumption made by tmem_put() in
>         the cleancache path that preemption is disabled.

I do not have an explanation right now for why these commits
expose this issue.  The patch looks like it should be fine
to me, hence my Ack at the time.

I understand and agree with you that the zcache shim
functions zcache_put_page(), zcache_get_page(),
zcache_flush_page(), and zcache_flush_object() all disable
interrupts (or make sure that interrupts are already
disabled) which implicitly disables preemption.

I'm still trying to find root cause here.

Seth

Re: [PATCH 0/2] revert changes to zcache_do_preload()

[Seth Jennings]

Forget this whole thing, these reverts do _not_ fix the issue.

I wrote a test program to exercises cleancache and
determined that this problem has existed since the as far
back at v3.1 (basically the beginning).

No recent commit caused this.

Seth