All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH] perf/x86/intel/cqm: Make sure the head event of cache_groups always has valid RMID
@ 2017-05-04  2:31 Zefan Li
  2017-05-16  1:40 ` Zefan Li
  2017-05-16 14:38 ` Peter Zijlstra
  0 siblings, 2 replies; 5+ messages in thread
From: Zefan Li @ 2017-05-04  2:31 UTC (permalink / raw)
  To: Thomas Gleixner
  Cc: Peter Zijlstra, H. Peter Anvin, mingo, Matt Fleming,
	vikas.shivappa, LKML

It is assumed that the head of cache_groups always has valid RMID,
which isn't true.

When we deallocate RMID from conflicting events currently we don't
move them to the tail, and one of those events can happen to be in
the head. Another case is we allocate RMIDs for all the events except
the head event in intel_cqm_sched_in_event().

Besides there's another bug that we retry rotating without resetting
nr_needed and start in __intel_cqm_rmid_rotate().

Those bugs combined together led to the following oops.

WARNING: at arch/x86/kernel/cpu/perf_event_intel_cqm.c:186 __put_rmid+0x28/0x80()
...
 [<ffffffff8103a578>] __put_rmid+0x28/0x80
 [<ffffffff8103a74a>] intel_cqm_rmid_rotate+0xba/0x440
 [<ffffffff8109d8cb>] process_one_work+0x17b/0x470
 [<ffffffff8109e69b>] worker_thread+0x11b/0x400
...
BUG: unable to handle kernel NULL pointer dereference at           (null)
...
 [<ffffffff8103a74a>] intel_cqm_rmid_rotate+0xba/0x440
 [<ffffffff8109d8cb>] process_one_work+0x17b/0x470
 [<ffffffff8109e69b>] worker_thread+0x11b/0x400

Cc: stable@vger.kernel.org
Signed-off-by: Zefan Li <lizefan@huawei.com>
---
 arch/x86/events/intel/cqm.c | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)

diff --git a/arch/x86/events/intel/cqm.c b/arch/x86/events/intel/cqm.c
index 8c00dc0..c06a5ba 100644
--- a/arch/x86/events/intel/cqm.c
+++ b/arch/x86/events/intel/cqm.c
@@ -553,8 +553,13 @@ static bool intel_cqm_sched_in_event(u32 rmid)
 
 	leader = list_first_entry(&cache_groups, struct perf_event,
 				  hw.cqm_groups_entry);
-	event = leader;
 
+	if (!list_empty(&cache_groups) && !__rmid_valid(leader->hw.cqm_rmid)) {
+		intel_cqm_xchg_rmid(leader, rmid);
+		return true;
+	}
+
+	event = leader;
 	list_for_each_entry_continue(event, &cache_groups,
 				     hw.cqm_groups_entry) {
 		if (__rmid_valid(event->hw.cqm_rmid))
@@ -721,6 +726,7 @@ static void intel_cqm_sched_out_conflicting_events(struct perf_event *event)
 {
 	struct perf_event *group, *g;
 	u32 rmid;
+	LIST_HEAD(conflicting_groups);
 
 	lockdep_assert_held(&cache_mutex);
 
@@ -744,7 +750,11 @@ static void intel_cqm_sched_out_conflicting_events(struct perf_event *event)
 
 		intel_cqm_xchg_rmid(group, INVALID_RMID);
 		__put_rmid(rmid);
+		list_move_tail(&group->hw.cqm_groups_entry,
+			       &conflicting_groups);
 	}
+
+	list_splice_tail(&conflicting_groups, &cache_groups);
 }
 
 /*
@@ -773,9 +783,9 @@ static void intel_cqm_sched_out_conflicting_events(struct perf_event *event)
  */
 static bool __intel_cqm_rmid_rotate(void)
 {
-	struct perf_event *group, *start = NULL;
+	struct perf_event *group, *start;
 	unsigned int threshold_limit;
-	unsigned int nr_needed = 0;
+	unsigned int nr_needed;
 	unsigned int nr_available;
 	bool rotated = false;
 
@@ -789,6 +799,9 @@ static bool __intel_cqm_rmid_rotate(void)
 	if (list_empty(&cache_groups) && list_empty(&cqm_rmid_limbo_lru))
 		goto out;
 
+	nr_needed = 0;
+	start = NULL;
+
 	list_for_each_entry(group, &cache_groups, hw.cqm_groups_entry) {
 		if (!__rmid_valid(group->hw.cqm_rmid)) {
 			if (!start)
-- 
1.8.2.2

^ permalink raw reply related	[flat|nested] 5+ messages in thread

* Re: [PATCH] perf/x86/intel/cqm: Make sure the head event of cache_groups always has valid RMID
  2017-05-04  2:31 [PATCH] perf/x86/intel/cqm: Make sure the head event of cache_groups always has valid RMID Zefan Li
@ 2017-05-16  1:40 ` Zefan Li
  2017-05-16 14:38 ` Peter Zijlstra
  1 sibling, 0 replies; 5+ messages in thread
From: Zefan Li @ 2017-05-16  1:40 UTC (permalink / raw)
  To: Thomas Gleixner
  Cc: Peter Zijlstra, H. Peter Anvin, mingo, vikas.shivappa, LKML

any comments?

On 2017/5/4 10:31, Zefan Li wrote:
> It is assumed that the head of cache_groups always has valid RMID,
> which isn't true.
> 
> When we deallocate RMID from conflicting events currently we don't
> move them to the tail, and one of those events can happen to be in
> the head. Another case is we allocate RMIDs for all the events except
> the head event in intel_cqm_sched_in_event().
> 
> Besides there's another bug that we retry rotating without resetting
> nr_needed and start in __intel_cqm_rmid_rotate().
> 
> Those bugs combined together led to the following oops.
> 
> WARNING: at arch/x86/kernel/cpu/perf_event_intel_cqm.c:186 __put_rmid+0x28/0x80()
> ...
>  [<ffffffff8103a578>] __put_rmid+0x28/0x80
>  [<ffffffff8103a74a>] intel_cqm_rmid_rotate+0xba/0x440
>  [<ffffffff8109d8cb>] process_one_work+0x17b/0x470
>  [<ffffffff8109e69b>] worker_thread+0x11b/0x400
> ...
> BUG: unable to handle kernel NULL pointer dereference at           (null)
> ...
>  [<ffffffff8103a74a>] intel_cqm_rmid_rotate+0xba/0x440
>  [<ffffffff8109d8cb>] process_one_work+0x17b/0x470
>  [<ffffffff8109e69b>] worker_thread+0x11b/0x400
> 
> Cc: stable@vger.kernel.org
> Signed-off-by: Zefan Li <lizefan@huawei.com>
> ---
>  arch/x86/events/intel/cqm.c | 19 ++++++++++++++++---
>  1 file changed, 16 insertions(+), 3 deletions(-)
> 
> diff --git a/arch/x86/events/intel/cqm.c b/arch/x86/events/intel/cqm.c
> index 8c00dc0..c06a5ba 100644
> --- a/arch/x86/events/intel/cqm.c
> +++ b/arch/x86/events/intel/cqm.c
> @@ -553,8 +553,13 @@ static bool intel_cqm_sched_in_event(u32 rmid)
>  
>  	leader = list_first_entry(&cache_groups, struct perf_event,
>  				  hw.cqm_groups_entry);
> -	event = leader;
>  
> +	if (!list_empty(&cache_groups) && !__rmid_valid(leader->hw.cqm_rmid)) {
> +		intel_cqm_xchg_rmid(leader, rmid);
> +		return true;
> +	}
> +
> +	event = leader;
>  	list_for_each_entry_continue(event, &cache_groups,
>  				     hw.cqm_groups_entry) {
>  		if (__rmid_valid(event->hw.cqm_rmid))
> @@ -721,6 +726,7 @@ static void intel_cqm_sched_out_conflicting_events(struct perf_event *event)
>  {
>  	struct perf_event *group, *g;
>  	u32 rmid;
> +	LIST_HEAD(conflicting_groups);
>  
>  	lockdep_assert_held(&cache_mutex);
>  
> @@ -744,7 +750,11 @@ static void intel_cqm_sched_out_conflicting_events(struct perf_event *event)
>  
>  		intel_cqm_xchg_rmid(group, INVALID_RMID);
>  		__put_rmid(rmid);
> +		list_move_tail(&group->hw.cqm_groups_entry,
> +			       &conflicting_groups);
>  	}
> +
> +	list_splice_tail(&conflicting_groups, &cache_groups);
>  }
>  
>  /*
> @@ -773,9 +783,9 @@ static void intel_cqm_sched_out_conflicting_events(struct perf_event *event)
>   */
>  static bool __intel_cqm_rmid_rotate(void)
>  {
> -	struct perf_event *group, *start = NULL;
> +	struct perf_event *group, *start;
>  	unsigned int threshold_limit;
> -	unsigned int nr_needed = 0;
> +	unsigned int nr_needed;
>  	unsigned int nr_available;
>  	bool rotated = false;
>  
> @@ -789,6 +799,9 @@ static bool __intel_cqm_rmid_rotate(void)
>  	if (list_empty(&cache_groups) && list_empty(&cqm_rmid_limbo_lru))
>  		goto out;
>  
> +	nr_needed = 0;
> +	start = NULL;
> +
>  	list_for_each_entry(group, &cache_groups, hw.cqm_groups_entry) {
>  		if (!__rmid_valid(group->hw.cqm_rmid)) {
>  			if (!start)
> 

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [PATCH] perf/x86/intel/cqm: Make sure the head event of cache_groups always has valid RMID
  2017-05-04  2:31 [PATCH] perf/x86/intel/cqm: Make sure the head event of cache_groups always has valid RMID Zefan Li
  2017-05-16  1:40 ` Zefan Li
@ 2017-05-16 14:38 ` Peter Zijlstra
  2017-05-18  4:59   ` David Carrillo-Cisneros
  2017-05-22 17:06   ` Shivappa Vikas
  1 sibling, 2 replies; 5+ messages in thread
From: Peter Zijlstra @ 2017-05-16 14:38 UTC (permalink / raw)
  To: Zefan Li
  Cc: Thomas Gleixner, H. Peter Anvin, mingo, Matt Fleming,
	vikas.shivappa, LKML, davidcc

On Thu, May 04, 2017 at 10:31:43AM +0800, Zefan Li wrote:
> It is assumed that the head of cache_groups always has valid RMID,
> which isn't true.
> 
> When we deallocate RMID from conflicting events currently we don't
> move them to the tail, and one of those events can happen to be in
> the head. Another case is we allocate RMIDs for all the events except
> the head event in intel_cqm_sched_in_event().
> 
> Besides there's another bug that we retry rotating without resetting
> nr_needed and start in __intel_cqm_rmid_rotate().
> 
> Those bugs combined together led to the following oops.
> 
> WARNING: at arch/x86/kernel/cpu/perf_event_intel_cqm.c:186 __put_rmid+0x28/0x80()
> ...
>  [<ffffffff8103a578>] __put_rmid+0x28/0x80
>  [<ffffffff8103a74a>] intel_cqm_rmid_rotate+0xba/0x440
>  [<ffffffff8109d8cb>] process_one_work+0x17b/0x470
>  [<ffffffff8109e69b>] worker_thread+0x11b/0x400
> ...
> BUG: unable to handle kernel NULL pointer dereference at           (null)
> ...
>  [<ffffffff8103a74a>] intel_cqm_rmid_rotate+0xba/0x440
>  [<ffffffff8109d8cb>] process_one_work+0x17b/0x470
>  [<ffffffff8109e69b>] worker_thread+0x11b/0x400

I've managed to forgot most if not all of that horror show. Vikas and
David seem to be working on a replacement, but until such a time it
would be good if this thing would not crash the kernel.

Guys, could you have a look? To me it appears to mostly have the right
shape, but like I said, I forgot most details...

> 
> Cc: stable@vger.kernel.org
> Signed-off-by: Zefan Li <lizefan@huawei.com>
> ---
>  arch/x86/events/intel/cqm.c | 19 ++++++++++++++++---
>  1 file changed, 16 insertions(+), 3 deletions(-)
> 
> diff --git a/arch/x86/events/intel/cqm.c b/arch/x86/events/intel/cqm.c
> index 8c00dc0..c06a5ba 100644
> --- a/arch/x86/events/intel/cqm.c
> +++ b/arch/x86/events/intel/cqm.c
> @@ -553,8 +553,13 @@ static bool intel_cqm_sched_in_event(u32 rmid)
>  
>  	leader = list_first_entry(&cache_groups, struct perf_event,
>  				  hw.cqm_groups_entry);
> -	event = leader;
>  
> +	if (!list_empty(&cache_groups) && !__rmid_valid(leader->hw.cqm_rmid)) {
> +		intel_cqm_xchg_rmid(leader, rmid);
> +		return true;
> +	}
> +
> +	event = leader;
>  	list_for_each_entry_continue(event, &cache_groups,
>  				     hw.cqm_groups_entry) {
>  		if (__rmid_valid(event->hw.cqm_rmid))
> @@ -721,6 +726,7 @@ static void intel_cqm_sched_out_conflicting_events(struct perf_event *event)
>  {
>  	struct perf_event *group, *g;
>  	u32 rmid;
> +	LIST_HEAD(conflicting_groups);
>  
>  	lockdep_assert_held(&cache_mutex);
>  
> @@ -744,7 +750,11 @@ static void intel_cqm_sched_out_conflicting_events(struct perf_event *event)
>  
>  		intel_cqm_xchg_rmid(group, INVALID_RMID);
>  		__put_rmid(rmid);
> +		list_move_tail(&group->hw.cqm_groups_entry,
> +			       &conflicting_groups);
>  	}
> +
> +	list_splice_tail(&conflicting_groups, &cache_groups);
>  }
>  
>  /*
> @@ -773,9 +783,9 @@ static void intel_cqm_sched_out_conflicting_events(struct perf_event *event)
>   */
>  static bool __intel_cqm_rmid_rotate(void)
>  {
> -	struct perf_event *group, *start = NULL;
> +	struct perf_event *group, *start;
>  	unsigned int threshold_limit;
> -	unsigned int nr_needed = 0;
> +	unsigned int nr_needed;
>  	unsigned int nr_available;
>  	bool rotated = false;
>  
> @@ -789,6 +799,9 @@ static bool __intel_cqm_rmid_rotate(void)
>  	if (list_empty(&cache_groups) && list_empty(&cqm_rmid_limbo_lru))
>  		goto out;
>  
> +	nr_needed = 0;
> +	start = NULL;
> +
>  	list_for_each_entry(group, &cache_groups, hw.cqm_groups_entry) {
>  		if (!__rmid_valid(group->hw.cqm_rmid)) {
>  			if (!start)
> -- 
> 1.8.2.2
> 

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [PATCH] perf/x86/intel/cqm: Make sure the head event of cache_groups always has valid RMID
  2017-05-16 14:38 ` Peter Zijlstra
@ 2017-05-18  4:59   ` David Carrillo-Cisneros
  2017-05-22 17:06   ` Shivappa Vikas
  1 sibling, 0 replies; 5+ messages in thread
From: David Carrillo-Cisneros @ 2017-05-18  4:59 UTC (permalink / raw)
  To: Peter Zijlstra
  Cc: Zefan Li, Thomas Gleixner, H. Peter Anvin, Ingo Molnar,
	Matt Fleming, Vikas Shivappa, LKML

On Tue, May 16, 2017 at 7:38 AM, Peter Zijlstra <peterz@infradead.org> wrote:
> On Thu, May 04, 2017 at 10:31:43AM +0800, Zefan Li wrote:
>> It is assumed that the head of cache_groups always has valid RMID,
>> which isn't true.
>>
>> When we deallocate RMID from conflicting events currently we don't
>> move them to the tail, and one of those events can happen to be in
>> the head. Another case is we allocate RMIDs for all the events except
>> the head event in intel_cqm_sched_in_event().
>>
>> Besides there's another bug that we retry rotating without resetting
>> nr_needed and start in __intel_cqm_rmid_rotate().
>>
>> Those bugs combined together led to the following oops.
>>
>> WARNING: at arch/x86/kernel/cpu/perf_event_intel_cqm.c:186 __put_rmid+0x28/0x80()
>> ...
>>  [<ffffffff8103a578>] __put_rmid+0x28/0x80
>>  [<ffffffff8103a74a>] intel_cqm_rmid_rotate+0xba/0x440
>>  [<ffffffff8109d8cb>] process_one_work+0x17b/0x470
>>  [<ffffffff8109e69b>] worker_thread+0x11b/0x400
>> ...
>> BUG: unable to handle kernel NULL pointer dereference at           (null)

I ran into this bug long time ago but never found an easy way to
reproduce. Do you have one?

>> ...
>>  [<ffffffff8103a74a>] intel_cqm_rmid_rotate+0xba/0x440
>>  [<ffffffff8109d8cb>] process_one_work+0x17b/0x470
>>  [<ffffffff8109e69b>] worker_thread+0x11b/0x400
>
> I've managed to forgot most if not all of that horror show. Vikas and
> David seem to be working on a replacement, but until such a time it
> would be good if this thing would not crash the kernel.
>
> Guys, could you have a look? To me it appears to mostly have the right
> shape, but like I said, I forgot most details...

The patch LGTM. I ran into this issues before and fixed them in a
similar but messier way, then the re-write started ...

>
>>
>> Cc: stable@vger.kernel.org
>> Signed-off-by: Zefan Li <lizefan@huawei.com>
Acked-by: David Carrillo-Cisneros <davidcc@google.com>

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [PATCH] perf/x86/intel/cqm: Make sure the head event of cache_groups always has valid RMID
  2017-05-16 14:38 ` Peter Zijlstra
  2017-05-18  4:59   ` David Carrillo-Cisneros
@ 2017-05-22 17:06   ` Shivappa Vikas
  1 sibling, 0 replies; 5+ messages in thread
From: Shivappa Vikas @ 2017-05-22 17:06 UTC (permalink / raw)
  To: Peter Zijlstra
  Cc: Zefan Li, Thomas Gleixner, H. Peter Anvin, mingo, Matt Fleming,
	vikas.shivappa, LKML, davidcc



>> Besides there's another bug that we retry rotating without resetting
>> nr_needed and start in __intel_cqm_rmid_rotate().
>>
>> Those bugs combined together led to the following oops.
>>
>> WARNING: at arch/x86/kernel/cpu/perf_event_intel_cqm.c:186 __put_rmid+0x28/0x80()
>> ...
>>  [<ffffffff8103a578>] __put_rmid+0x28/0x80
>>  [<ffffffff8103a74a>] intel_cqm_rmid_rotate+0xba/0x440
>>  [<ffffffff8109d8cb>] process_one_work+0x17b/0x470
>>  [<ffffffff8109e69b>] worker_thread+0x11b/0x400
>> ...
>> BUG: unable to handle kernel NULL pointer dereference at           (null)

Yes, I saw these issues in the RMID rotation and also that it even rotated the 
rmids which were in use before the limbo ones which did not crash but it just 
made it output wierd data. Like David mentioned thats why we started 
writing the new patches.

>> ...
>>  [<ffffffff8103a74a>] intel_cqm_rmid_rotate+0xba/0x440
>>  [<ffffffff8109d8cb>] process_one_work+0x17b/0x470
>>  [<ffffffff8109e69b>] worker_thread+0x11b/0x400
>
> I've managed to forgot most if not all of that horror show. Vikas and
> David seem to be working on a replacement, but until such a time it
> would be good if this thing would not crash the kernel.
>
> Guys, could you have a look? To me it appears to mostly have the right
> shape, but like I said, I forgot most details...

The new patch is on the way after Thomas agreed for the requirements we sent a 
few weeks back. Expect to send it in a week or so.. This fix seems fine.

Thanks,
Vikas

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2017-05-22 17:05 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-05-04  2:31 [PATCH] perf/x86/intel/cqm: Make sure the head event of cache_groups always has valid RMID Zefan Li
2017-05-16  1:40 ` Zefan Li
2017-05-16 14:38 ` Peter Zijlstra
2017-05-18  4:59   ` David Carrillo-Cisneros
2017-05-22 17:06   ` Shivappa Vikas

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.