* [Linux-kernel-mentees] [PATCH v2] kvm: Fix false positive RCU usage warning @ 2020-05-16 8:22 madhuparnabhowmik10 2020-06-23 7:39 ` Paolo Bonzini 0 siblings, 1 reply; 9+ messages in thread From: madhuparnabhowmik10 @ 2020-05-16 8:22 UTC (permalink / raw) To: pbonzini, sean.j.christopherson, vkuznets, wanpengli, jmattson, tglx, bp Cc: kvm, paulmck, x86, linux-kernel, Madhuparna Bhowmik, joel, linux-kernel-mentees From: Madhuparna Bhowmik <madhuparnabhowmik10@gmail.com> Fix the following false positive warnings: [ 9403.765413][T61744] ============================= [ 9403.786541][T61744] WARNING: suspicious RCU usage [ 9403.807865][T61744] 5.7.0-rc1-next-20200417 #4 Tainted: G L [ 9403.838945][T61744] ----------------------------- [ 9403.860099][T61744] arch/x86/kvm/mmu/page_track.c:257 RCU-list traversed in non-reader section!! and [ 9405.859252][T61751] ============================= [ 9405.859258][T61751] WARNING: suspicious RCU usage [ 9405.880867][T61755] ----------------------------- [ 9405.911936][T61751] 5.7.0-rc1-next-20200417 #4 Tainted: G L [ 9405.911942][T61751] ----------------------------- [ 9405.911950][T61751] arch/x86/kvm/mmu/page_track.c:232 RCU-list traversed in non-reader section!! Since srcu read lock is held, these are false positive warnings. Therefore, pass condition srcu_read_lock_held() to list_for_each_entry_rcu(). Reported-by: kernel test robot <lkp@intel.com> Signed-off-by: Madhuparna Bhowmik <madhuparnabhowmik10@gmail.com> --- v2: -Rebase v5.7-rc5 arch/x86/kvm/mmu/page_track.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c index ddc1ec3bdacd..1ad79c7aa05b 100644 --- a/arch/x86/kvm/mmu/page_track.c +++ b/arch/x86/kvm/mmu/page_track.c @@ -229,7 +229,8 @@ void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new, return; idx = srcu_read_lock(&head->track_srcu); - hlist_for_each_entry_rcu(n, &head->track_notifier_list, node) + hlist_for_each_entry_rcu(n, &head->track_notifier_list, node, + srcu_read_lock_held(&head->track_srcu)) if (n->track_write) n->track_write(vcpu, gpa, new, bytes, n); srcu_read_unlock(&head->track_srcu, idx); @@ -254,7 +255,8 @@ void kvm_page_track_flush_slot(struct kvm *kvm, struct kvm_memory_slot *slot) return; idx = srcu_read_lock(&head->track_srcu); - hlist_for_each_entry_rcu(n, &head->track_notifier_list, node) + hlist_for_each_entry_rcu(n, &head->track_notifier_list, node, + srcu_read_lock_held(&head->track_srcu)) if (n->track_flush_slot) n->track_flush_slot(kvm, slot, n); srcu_read_unlock(&head->track_srcu, idx); -- 2.17.1 _______________________________________________ Linux-kernel-mentees mailing list Linux-kernel-mentees@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/linux-kernel-mentees ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [Linux-kernel-mentees] [PATCH v2] kvm: Fix false positive RCU usage warning 2020-05-16 8:22 [Linux-kernel-mentees] [PATCH v2] kvm: Fix false positive RCU usage warning madhuparnabhowmik10 @ 2020-06-23 7:39 ` Paolo Bonzini 2020-06-23 15:02 ` Joel Fernandes 2020-06-23 15:29 ` Madhuparna Bhowmik 0 siblings, 2 replies; 9+ messages in thread From: Paolo Bonzini @ 2020-06-23 7:39 UTC (permalink / raw) To: madhuparnabhowmik10, sean.j.christopherson, vkuznets, wanpengli, jmattson, tglx, bp Cc: kvm, paulmck, x86, linux-kernel, joel, Paul McKenney, linux-kernel-mentees On 16/05/20 10:22, madhuparnabhowmik10@gmail.com wrote: > From: Madhuparna Bhowmik <madhuparnabhowmik10@gmail.com> > > Fix the following false positive warnings: > > [ 9403.765413][T61744] ============================= > [ 9403.786541][T61744] WARNING: suspicious RCU usage > [ 9403.807865][T61744] 5.7.0-rc1-next-20200417 #4 Tainted: G L > [ 9403.838945][T61744] ----------------------------- > [ 9403.860099][T61744] arch/x86/kvm/mmu/page_track.c:257 RCU-list traversed in non-reader section!! > > and > > [ 9405.859252][T61751] ============================= > [ 9405.859258][T61751] WARNING: suspicious RCU usage > [ 9405.880867][T61755] ----------------------------- > [ 9405.911936][T61751] 5.7.0-rc1-next-20200417 #4 Tainted: G L > [ 9405.911942][T61751] ----------------------------- > [ 9405.911950][T61751] arch/x86/kvm/mmu/page_track.c:232 RCU-list traversed in non-reader section!! > > Since srcu read lock is held, these are false positive warnings. > Therefore, pass condition srcu_read_lock_held() to > list_for_each_entry_rcu(). > > Reported-by: kernel test robot <lkp@intel.com> > Signed-off-by: Madhuparna Bhowmik <madhuparnabhowmik10@gmail.com> > --- > v2: > -Rebase v5.7-rc5 > > arch/x86/kvm/mmu/page_track.c | 6 ++++-- > 1 file changed, 4 insertions(+), 2 deletions(-) > > diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c > index ddc1ec3bdacd..1ad79c7aa05b 100644 > --- a/arch/x86/kvm/mmu/page_track.c > +++ b/arch/x86/kvm/mmu/page_track.c > @@ -229,7 +229,8 @@ void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new, > return; > > idx = srcu_read_lock(&head->track_srcu); > - hlist_for_each_entry_rcu(n, &head->track_notifier_list, node) > + hlist_for_each_entry_rcu(n, &head->track_notifier_list, node, > + srcu_read_lock_held(&head->track_srcu)) > if (n->track_write) > n->track_write(vcpu, gpa, new, bytes, n); > srcu_read_unlock(&head->track_srcu, idx); > @@ -254,7 +255,8 @@ void kvm_page_track_flush_slot(struct kvm *kvm, struct kvm_memory_slot *slot) > return; > > idx = srcu_read_lock(&head->track_srcu); > - hlist_for_each_entry_rcu(n, &head->track_notifier_list, node) > + hlist_for_each_entry_rcu(n, &head->track_notifier_list, node, > + srcu_read_lock_held(&head->track_srcu)) > if (n->track_flush_slot) > n->track_flush_slot(kvm, slot, n); > srcu_read_unlock(&head->track_srcu, idx); > Hi, sorry for the delay in reviewing this patch. I would like to ask Paul about it. While you're correctly fixing a false positive, hlist_for_each_entry_rcu would have a false _negative_ if you called it under rcu_read_lock/unlock and the data structure was protected by SRCU. This is why for example srcu_dereference is used instead of rcu_dereference_check, and why srcu_dereference uses __rcu_dereference_check (with the two underscores) instead of rcu_dereference_check. Using rcu_dereference_check would add an "|| rcu_read_lock_held()" to the condition which is wrong. I think instead you should add hlist_for_each_srcu and hlist_for_each_entry_srcu macro to include/linux/rculist.h. There is no need for equivalents of hlist_for_each_entry_continue_rcu and hlist_for_each_entry_from_rcu, because they use rcu_dereference_raw. However, it's not documented why they do so. Paul, do you have any objections to the idea? Thanks, Paolo _______________________________________________ Linux-kernel-mentees mailing list Linux-kernel-mentees@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/linux-kernel-mentees ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Linux-kernel-mentees] [PATCH v2] kvm: Fix false positive RCU usage warning 2020-06-23 7:39 ` Paolo Bonzini @ 2020-06-23 15:02 ` Joel Fernandes 2020-06-23 15:30 ` Madhuparna Bhowmik 2020-06-23 15:29 ` Madhuparna Bhowmik 1 sibling, 1 reply; 9+ messages in thread From: Joel Fernandes @ 2020-06-23 15:02 UTC (permalink / raw) To: Paolo Bonzini Cc: wanpengli, paulmck, kvm, x86, linux-kernel, sean.j.christopherson, madhuparnabhowmik10, bp, vkuznets, Paul McKenney, linux-kernel-mentees, tglx, jmattson On Tue, Jun 23, 2020 at 09:39:53AM +0200, Paolo Bonzini wrote: > On 16/05/20 10:22, madhuparnabhowmik10@gmail.com wrote: > > From: Madhuparna Bhowmik <madhuparnabhowmik10@gmail.com> > > > > Fix the following false positive warnings: > > > > [ 9403.765413][T61744] ============================= > > [ 9403.786541][T61744] WARNING: suspicious RCU usage > > [ 9403.807865][T61744] 5.7.0-rc1-next-20200417 #4 Tainted: G L > > [ 9403.838945][T61744] ----------------------------- > > [ 9403.860099][T61744] arch/x86/kvm/mmu/page_track.c:257 RCU-list traversed in non-reader section!! > > > > and > > > > [ 9405.859252][T61751] ============================= > > [ 9405.859258][T61751] WARNING: suspicious RCU usage > > [ 9405.880867][T61755] ----------------------------- > > [ 9405.911936][T61751] 5.7.0-rc1-next-20200417 #4 Tainted: G L > > [ 9405.911942][T61751] ----------------------------- > > [ 9405.911950][T61751] arch/x86/kvm/mmu/page_track.c:232 RCU-list traversed in non-reader section!! > > > > Since srcu read lock is held, these are false positive warnings. > > Therefore, pass condition srcu_read_lock_held() to > > list_for_each_entry_rcu(). > > > > Reported-by: kernel test robot <lkp@intel.com> > > Signed-off-by: Madhuparna Bhowmik <madhuparnabhowmik10@gmail.com> > > --- > > v2: > > -Rebase v5.7-rc5 > > > > arch/x86/kvm/mmu/page_track.c | 6 ++++-- > > 1 file changed, 4 insertions(+), 2 deletions(-) > > > > diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c > > index ddc1ec3bdacd..1ad79c7aa05b 100644 > > --- a/arch/x86/kvm/mmu/page_track.c > > +++ b/arch/x86/kvm/mmu/page_track.c > > @@ -229,7 +229,8 @@ void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new, > > return; > > > > idx = srcu_read_lock(&head->track_srcu); > > - hlist_for_each_entry_rcu(n, &head->track_notifier_list, node) > > + hlist_for_each_entry_rcu(n, &head->track_notifier_list, node, > > + srcu_read_lock_held(&head->track_srcu)) > > if (n->track_write) > > n->track_write(vcpu, gpa, new, bytes, n); > > srcu_read_unlock(&head->track_srcu, idx); > > @@ -254,7 +255,8 @@ void kvm_page_track_flush_slot(struct kvm *kvm, struct kvm_memory_slot *slot) > > return; > > > > idx = srcu_read_lock(&head->track_srcu); > > - hlist_for_each_entry_rcu(n, &head->track_notifier_list, node) > > + hlist_for_each_entry_rcu(n, &head->track_notifier_list, node, > > + srcu_read_lock_held(&head->track_srcu)) > > if (n->track_flush_slot) > > n->track_flush_slot(kvm, slot, n); > > srcu_read_unlock(&head->track_srcu, idx); > > > > Hi, sorry for the delay in reviewing this patch. I would like to ask > Paul about it. > > While you're correctly fixing a false positive, hlist_for_each_entry_rcu > would have a false _negative_ if you called it under > rcu_read_lock/unlock and the data structure was protected by SRCU. This > is why for example srcu_dereference is used instead of > rcu_dereference_check, and why srcu_dereference uses > __rcu_dereference_check (with the two underscores) instead of > rcu_dereference_check. Using rcu_dereference_check would add an "|| > rcu_read_lock_held()" to the condition which is wrong. > > I think instead you should add hlist_for_each_srcu and > hlist_for_each_entry_srcu macro to include/linux/rculist.h. > > There is no need for equivalents of hlist_for_each_entry_continue_rcu > and hlist_for_each_entry_from_rcu, because they use rcu_dereference_raw. > However, it's not documented why they do so. You are right, this patch is wrong, we need a new SRCU list macro to do the right thing which would also get rid of the last list argument. _______________________________________________ Linux-kernel-mentees mailing list Linux-kernel-mentees@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/linux-kernel-mentees ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Linux-kernel-mentees] [PATCH v2] kvm: Fix false positive RCU usage warning 2020-06-23 15:02 ` Joel Fernandes @ 2020-06-23 15:30 ` Madhuparna Bhowmik 2020-06-23 15:39 ` Paul E. McKenney 0 siblings, 1 reply; 9+ messages in thread From: Madhuparna Bhowmik @ 2020-06-23 15:30 UTC (permalink / raw) To: Joel Fernandes Cc: wanpengli, paulmck, kvm, x86, linux-kernel, sean.j.christopherson, madhuparnabhowmik10, bp, Paolo Bonzini, vkuznets, Paul McKenney, linux-kernel-mentees, tglx, jmattson On Tue, Jun 23, 2020 at 11:02:36AM -0400, Joel Fernandes wrote: > On Tue, Jun 23, 2020 at 09:39:53AM +0200, Paolo Bonzini wrote: > > On 16/05/20 10:22, madhuparnabhowmik10@gmail.com wrote: > > > From: Madhuparna Bhowmik <madhuparnabhowmik10@gmail.com> > > > > > > Fix the following false positive warnings: > > > > > > [ 9403.765413][T61744] ============================= > > > [ 9403.786541][T61744] WARNING: suspicious RCU usage > > > [ 9403.807865][T61744] 5.7.0-rc1-next-20200417 #4 Tainted: G L > > > [ 9403.838945][T61744] ----------------------------- > > > [ 9403.860099][T61744] arch/x86/kvm/mmu/page_track.c:257 RCU-list traversed in non-reader section!! > > > > > > and > > > > > > [ 9405.859252][T61751] ============================= > > > [ 9405.859258][T61751] WARNING: suspicious RCU usage > > > [ 9405.880867][T61755] ----------------------------- > > > [ 9405.911936][T61751] 5.7.0-rc1-next-20200417 #4 Tainted: G L > > > [ 9405.911942][T61751] ----------------------------- > > > [ 9405.911950][T61751] arch/x86/kvm/mmu/page_track.c:232 RCU-list traversed in non-reader section!! > > > > > > Since srcu read lock is held, these are false positive warnings. > > > Therefore, pass condition srcu_read_lock_held() to > > > list_for_each_entry_rcu(). > > > > > > Reported-by: kernel test robot <lkp@intel.com> > > > Signed-off-by: Madhuparna Bhowmik <madhuparnabhowmik10@gmail.com> > > > --- > > > v2: > > > -Rebase v5.7-rc5 > > > > > > arch/x86/kvm/mmu/page_track.c | 6 ++++-- > > > 1 file changed, 4 insertions(+), 2 deletions(-) > > > > > > diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c > > > index ddc1ec3bdacd..1ad79c7aa05b 100644 > > > --- a/arch/x86/kvm/mmu/page_track.c > > > +++ b/arch/x86/kvm/mmu/page_track.c > > > @@ -229,7 +229,8 @@ void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new, > > > return; > > > > > > idx = srcu_read_lock(&head->track_srcu); > > > - hlist_for_each_entry_rcu(n, &head->track_notifier_list, node) > > > + hlist_for_each_entry_rcu(n, &head->track_notifier_list, node, > > > + srcu_read_lock_held(&head->track_srcu)) > > > if (n->track_write) > > > n->track_write(vcpu, gpa, new, bytes, n); > > > srcu_read_unlock(&head->track_srcu, idx); > > > @@ -254,7 +255,8 @@ void kvm_page_track_flush_slot(struct kvm *kvm, struct kvm_memory_slot *slot) > > > return; > > > > > > idx = srcu_read_lock(&head->track_srcu); > > > - hlist_for_each_entry_rcu(n, &head->track_notifier_list, node) > > > + hlist_for_each_entry_rcu(n, &head->track_notifier_list, node, > > > + srcu_read_lock_held(&head->track_srcu)) > > > if (n->track_flush_slot) > > > n->track_flush_slot(kvm, slot, n); > > > srcu_read_unlock(&head->track_srcu, idx); > > > > > > > Hi, sorry for the delay in reviewing this patch. I would like to ask > > Paul about it. > > > > While you're correctly fixing a false positive, hlist_for_each_entry_rcu > > would have a false _negative_ if you called it under > > rcu_read_lock/unlock and the data structure was protected by SRCU. This > > is why for example srcu_dereference is used instead of > > rcu_dereference_check, and why srcu_dereference uses > > __rcu_dereference_check (with the two underscores) instead of > > rcu_dereference_check. Using rcu_dereference_check would add an "|| > > rcu_read_lock_held()" to the condition which is wrong. > > > > I think instead you should add hlist_for_each_srcu and > > hlist_for_each_entry_srcu macro to include/linux/rculist.h. > > > > There is no need for equivalents of hlist_for_each_entry_continue_rcu > > and hlist_for_each_entry_from_rcu, because they use rcu_dereference_raw. > > However, it's not documented why they do so. > > You are right, this patch is wrong, we need a new SRCU list macro to do the > right thing which would also get rid of the last list argument. > Can we really get rid of the last argument? We would need the srcu_struct right for checking? _______________________________________________ Linux-kernel-mentees mailing list Linux-kernel-mentees@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/linux-kernel-mentees ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Linux-kernel-mentees] [PATCH v2] kvm: Fix false positive RCU usage warning 2020-06-23 15:30 ` Madhuparna Bhowmik @ 2020-06-23 15:39 ` Paul E. McKenney 2020-06-23 15:43 ` Joel Fernandes 2020-06-23 17:49 ` Madhuparna Bhowmik 0 siblings, 2 replies; 9+ messages in thread From: Paul E. McKenney @ 2020-06-23 15:39 UTC (permalink / raw) To: Madhuparna Bhowmik Cc: wanpengli, kvm, x86, linux-kernel, sean.j.christopherson, bp, Joel Fernandes, Paolo Bonzini, vkuznets, linux-kernel-mentees, tglx, jmattson On Tue, Jun 23, 2020 at 09:00:36PM +0530, Madhuparna Bhowmik wrote: > On Tue, Jun 23, 2020 at 11:02:36AM -0400, Joel Fernandes wrote: > > On Tue, Jun 23, 2020 at 09:39:53AM +0200, Paolo Bonzini wrote: > > > On 16/05/20 10:22, madhuparnabhowmik10@gmail.com wrote: > > > > From: Madhuparna Bhowmik <madhuparnabhowmik10@gmail.com> > > > > > > > > Fix the following false positive warnings: > > > > > > > > [ 9403.765413][T61744] ============================= > > > > [ 9403.786541][T61744] WARNING: suspicious RCU usage > > > > [ 9403.807865][T61744] 5.7.0-rc1-next-20200417 #4 Tainted: G L > > > > [ 9403.838945][T61744] ----------------------------- > > > > [ 9403.860099][T61744] arch/x86/kvm/mmu/page_track.c:257 RCU-list traversed in non-reader section!! > > > > > > > > and > > > > > > > > [ 9405.859252][T61751] ============================= > > > > [ 9405.859258][T61751] WARNING: suspicious RCU usage > > > > [ 9405.880867][T61755] ----------------------------- > > > > [ 9405.911936][T61751] 5.7.0-rc1-next-20200417 #4 Tainted: G L > > > > [ 9405.911942][T61751] ----------------------------- > > > > [ 9405.911950][T61751] arch/x86/kvm/mmu/page_track.c:232 RCU-list traversed in non-reader section!! > > > > > > > > Since srcu read lock is held, these are false positive warnings. > > > > Therefore, pass condition srcu_read_lock_held() to > > > > list_for_each_entry_rcu(). > > > > > > > > Reported-by: kernel test robot <lkp@intel.com> > > > > Signed-off-by: Madhuparna Bhowmik <madhuparnabhowmik10@gmail.com> > > > > --- > > > > v2: > > > > -Rebase v5.7-rc5 > > > > > > > > arch/x86/kvm/mmu/page_track.c | 6 ++++-- > > > > 1 file changed, 4 insertions(+), 2 deletions(-) > > > > > > > > diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c > > > > index ddc1ec3bdacd..1ad79c7aa05b 100644 > > > > --- a/arch/x86/kvm/mmu/page_track.c > > > > +++ b/arch/x86/kvm/mmu/page_track.c > > > > @@ -229,7 +229,8 @@ void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new, > > > > return; > > > > > > > > idx = srcu_read_lock(&head->track_srcu); > > > > - hlist_for_each_entry_rcu(n, &head->track_notifier_list, node) > > > > + hlist_for_each_entry_rcu(n, &head->track_notifier_list, node, > > > > + srcu_read_lock_held(&head->track_srcu)) > > > > if (n->track_write) > > > > n->track_write(vcpu, gpa, new, bytes, n); > > > > srcu_read_unlock(&head->track_srcu, idx); > > > > @@ -254,7 +255,8 @@ void kvm_page_track_flush_slot(struct kvm *kvm, struct kvm_memory_slot *slot) > > > > return; > > > > > > > > idx = srcu_read_lock(&head->track_srcu); > > > > - hlist_for_each_entry_rcu(n, &head->track_notifier_list, node) > > > > + hlist_for_each_entry_rcu(n, &head->track_notifier_list, node, > > > > + srcu_read_lock_held(&head->track_srcu)) > > > > if (n->track_flush_slot) > > > > n->track_flush_slot(kvm, slot, n); > > > > srcu_read_unlock(&head->track_srcu, idx); > > > > > > > > > > Hi, sorry for the delay in reviewing this patch. I would like to ask > > > Paul about it. > > > > > > While you're correctly fixing a false positive, hlist_for_each_entry_rcu > > > would have a false _negative_ if you called it under > > > rcu_read_lock/unlock and the data structure was protected by SRCU. This > > > is why for example srcu_dereference is used instead of > > > rcu_dereference_check, and why srcu_dereference uses > > > __rcu_dereference_check (with the two underscores) instead of > > > rcu_dereference_check. Using rcu_dereference_check would add an "|| > > > rcu_read_lock_held()" to the condition which is wrong. > > > > > > I think instead you should add hlist_for_each_srcu and > > > hlist_for_each_entry_srcu macro to include/linux/rculist.h. > > > > > > There is no need for equivalents of hlist_for_each_entry_continue_rcu > > > and hlist_for_each_entry_from_rcu, because they use rcu_dereference_raw. > > > However, it's not documented why they do so. > > > > You are right, this patch is wrong, we need a new SRCU list macro to do the > > right thing which would also get rid of the last list argument. > > > Can we really get rid of the last argument? We would need the > srcu_struct right for checking? Agreed! However, the API could be simplified by passing in a pointer to the srcu_struct instead of a lockdep expression. An optional lockdep expression might still be helpful for calls from the update side, of course. Thanx, Paul _______________________________________________ Linux-kernel-mentees mailing list Linux-kernel-mentees@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/linux-kernel-mentees ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Linux-kernel-mentees] [PATCH v2] kvm: Fix false positive RCU usage warning 2020-06-23 15:39 ` Paul E. McKenney @ 2020-06-23 15:43 ` Joel Fernandes 2020-06-23 17:49 ` Madhuparna Bhowmik 1 sibling, 0 replies; 9+ messages in thread From: Joel Fernandes @ 2020-06-23 15:43 UTC (permalink / raw) To: Paul E. McKenney Cc: wanpengli, kvm, x86, linux-kernel, sean.j.christopherson, Madhuparna Bhowmik, bp, Paolo Bonzini, vkuznets, linux-kernel-mentees, tglx, jmattson On Tue, Jun 23, 2020 at 08:39:01AM -0700, Paul E. McKenney wrote: > On Tue, Jun 23, 2020 at 09:00:36PM +0530, Madhuparna Bhowmik wrote: > > On Tue, Jun 23, 2020 at 11:02:36AM -0400, Joel Fernandes wrote: > > > On Tue, Jun 23, 2020 at 09:39:53AM +0200, Paolo Bonzini wrote: > > > > On 16/05/20 10:22, madhuparnabhowmik10@gmail.com wrote: > > > > > From: Madhuparna Bhowmik <madhuparnabhowmik10@gmail.com> > > > > > > > > > > Fix the following false positive warnings: > > > > > > > > > > [ 9403.765413][T61744] ============================= > > > > > [ 9403.786541][T61744] WARNING: suspicious RCU usage > > > > > [ 9403.807865][T61744] 5.7.0-rc1-next-20200417 #4 Tainted: G L > > > > > [ 9403.838945][T61744] ----------------------------- > > > > > [ 9403.860099][T61744] arch/x86/kvm/mmu/page_track.c:257 RCU-list traversed in non-reader section!! > > > > > > > > > > and > > > > > > > > > > [ 9405.859252][T61751] ============================= > > > > > [ 9405.859258][T61751] WARNING: suspicious RCU usage > > > > > [ 9405.880867][T61755] ----------------------------- > > > > > [ 9405.911936][T61751] 5.7.0-rc1-next-20200417 #4 Tainted: G L > > > > > [ 9405.911942][T61751] ----------------------------- > > > > > [ 9405.911950][T61751] arch/x86/kvm/mmu/page_track.c:232 RCU-list traversed in non-reader section!! > > > > > > > > > > Since srcu read lock is held, these are false positive warnings. > > > > > Therefore, pass condition srcu_read_lock_held() to > > > > > list_for_each_entry_rcu(). > > > > > > > > > > Reported-by: kernel test robot <lkp@intel.com> > > > > > Signed-off-by: Madhuparna Bhowmik <madhuparnabhowmik10@gmail.com> > > > > > --- > > > > > v2: > > > > > -Rebase v5.7-rc5 > > > > > > > > > > arch/x86/kvm/mmu/page_track.c | 6 ++++-- > > > > > 1 file changed, 4 insertions(+), 2 deletions(-) > > > > > > > > > > diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c > > > > > index ddc1ec3bdacd..1ad79c7aa05b 100644 > > > > > --- a/arch/x86/kvm/mmu/page_track.c > > > > > +++ b/arch/x86/kvm/mmu/page_track.c > > > > > @@ -229,7 +229,8 @@ void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new, > > > > > return; > > > > > > > > > > idx = srcu_read_lock(&head->track_srcu); > > > > > - hlist_for_each_entry_rcu(n, &head->track_notifier_list, node) > > > > > + hlist_for_each_entry_rcu(n, &head->track_notifier_list, node, > > > > > + srcu_read_lock_held(&head->track_srcu)) > > > > > if (n->track_write) > > > > > n->track_write(vcpu, gpa, new, bytes, n); > > > > > srcu_read_unlock(&head->track_srcu, idx); > > > > > @@ -254,7 +255,8 @@ void kvm_page_track_flush_slot(struct kvm *kvm, struct kvm_memory_slot *slot) > > > > > return; > > > > > > > > > > idx = srcu_read_lock(&head->track_srcu); > > > > > - hlist_for_each_entry_rcu(n, &head->track_notifier_list, node) > > > > > + hlist_for_each_entry_rcu(n, &head->track_notifier_list, node, > > > > > + srcu_read_lock_held(&head->track_srcu)) > > > > > if (n->track_flush_slot) > > > > > n->track_flush_slot(kvm, slot, n); > > > > > srcu_read_unlock(&head->track_srcu, idx); > > > > > > > > > > > > > Hi, sorry for the delay in reviewing this patch. I would like to ask > > > > Paul about it. > > > > > > > > While you're correctly fixing a false positive, hlist_for_each_entry_rcu > > > > would have a false _negative_ if you called it under > > > > rcu_read_lock/unlock and the data structure was protected by SRCU. This > > > > is why for example srcu_dereference is used instead of > > > > rcu_dereference_check, and why srcu_dereference uses > > > > __rcu_dereference_check (with the two underscores) instead of > > > > rcu_dereference_check. Using rcu_dereference_check would add an "|| > > > > rcu_read_lock_held()" to the condition which is wrong. > > > > > > > > I think instead you should add hlist_for_each_srcu and > > > > hlist_for_each_entry_srcu macro to include/linux/rculist.h. > > > > > > > > There is no need for equivalents of hlist_for_each_entry_continue_rcu > > > > and hlist_for_each_entry_from_rcu, because they use rcu_dereference_raw. > > > > However, it's not documented why they do so. > > > > > > You are right, this patch is wrong, we need a new SRCU list macro to do the > > > right thing which would also get rid of the last list argument. > > > > > Can we really get rid of the last argument? We would need the > > srcu_struct right for checking? > > Agreed! However, the API could be simplified by passing in a pointer to > the srcu_struct instead of a lockdep expression. An optional lockdep > expression might still be helpful for calls from the update side, > of course. That's true! thanks, - Joel _______________________________________________ Linux-kernel-mentees mailing list Linux-kernel-mentees@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/linux-kernel-mentees ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Linux-kernel-mentees] [PATCH v2] kvm: Fix false positive RCU usage warning 2020-06-23 15:39 ` Paul E. McKenney 2020-06-23 15:43 ` Joel Fernandes @ 2020-06-23 17:49 ` Madhuparna Bhowmik 2020-06-23 19:34 ` Joel Fernandes 1 sibling, 1 reply; 9+ messages in thread From: Madhuparna Bhowmik @ 2020-06-23 17:49 UTC (permalink / raw) To: Paul E. McKenney Cc: wanpengli, kvm, x86, linux-kernel, sean.j.christopherson, Madhuparna Bhowmik, bp, Joel Fernandes, Paolo Bonzini, vkuznets, linux-kernel-mentees, tglx, jmattson On Tue, Jun 23, 2020 at 08:39:01AM -0700, Paul E. McKenney wrote: > On Tue, Jun 23, 2020 at 09:00:36PM +0530, Madhuparna Bhowmik wrote: > > On Tue, Jun 23, 2020 at 11:02:36AM -0400, Joel Fernandes wrote: > > > On Tue, Jun 23, 2020 at 09:39:53AM +0200, Paolo Bonzini wrote: > > > > On 16/05/20 10:22, madhuparnabhowmik10@gmail.com wrote: > > > > > From: Madhuparna Bhowmik <madhuparnabhowmik10@gmail.com> > > > > > > > > > > Fix the following false positive warnings: > > > > > > > > > > [ 9403.765413][T61744] ============================= > > > > > [ 9403.786541][T61744] WARNING: suspicious RCU usage > > > > > [ 9403.807865][T61744] 5.7.0-rc1-next-20200417 #4 Tainted: G L > > > > > [ 9403.838945][T61744] ----------------------------- > > > > > [ 9403.860099][T61744] arch/x86/kvm/mmu/page_track.c:257 RCU-list traversed in non-reader section!! > > > > > > > > > > and > > > > > > > > > > [ 9405.859252][T61751] ============================= > > > > > [ 9405.859258][T61751] WARNING: suspicious RCU usage > > > > > [ 9405.880867][T61755] ----------------------------- > > > > > [ 9405.911936][T61751] 5.7.0-rc1-next-20200417 #4 Tainted: G L > > > > > [ 9405.911942][T61751] ----------------------------- > > > > > [ 9405.911950][T61751] arch/x86/kvm/mmu/page_track.c:232 RCU-list traversed in non-reader section!! > > > > > > > > > > Since srcu read lock is held, these are false positive warnings. > > > > > Therefore, pass condition srcu_read_lock_held() to > > > > > list_for_each_entry_rcu(). > > > > > > > > > > Reported-by: kernel test robot <lkp@intel.com> > > > > > Signed-off-by: Madhuparna Bhowmik <madhuparnabhowmik10@gmail.com> > > > > > --- > > > > > v2: > > > > > -Rebase v5.7-rc5 > > > > > > > > > > arch/x86/kvm/mmu/page_track.c | 6 ++++-- > > > > > 1 file changed, 4 insertions(+), 2 deletions(-) > > > > > > > > > > diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c > > > > > index ddc1ec3bdacd..1ad79c7aa05b 100644 > > > > > --- a/arch/x86/kvm/mmu/page_track.c > > > > > +++ b/arch/x86/kvm/mmu/page_track.c > > > > > @@ -229,7 +229,8 @@ void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new, > > > > > return; > > > > > > > > > > idx = srcu_read_lock(&head->track_srcu); > > > > > - hlist_for_each_entry_rcu(n, &head->track_notifier_list, node) > > > > > + hlist_for_each_entry_rcu(n, &head->track_notifier_list, node, > > > > > + srcu_read_lock_held(&head->track_srcu)) > > > > > if (n->track_write) > > > > > n->track_write(vcpu, gpa, new, bytes, n); > > > > > srcu_read_unlock(&head->track_srcu, idx); > > > > > @@ -254,7 +255,8 @@ void kvm_page_track_flush_slot(struct kvm *kvm, struct kvm_memory_slot *slot) > > > > > return; > > > > > > > > > > idx = srcu_read_lock(&head->track_srcu); > > > > > - hlist_for_each_entry_rcu(n, &head->track_notifier_list, node) > > > > > + hlist_for_each_entry_rcu(n, &head->track_notifier_list, node, > > > > > + srcu_read_lock_held(&head->track_srcu)) > > > > > if (n->track_flush_slot) > > > > > n->track_flush_slot(kvm, slot, n); > > > > > srcu_read_unlock(&head->track_srcu, idx); > > > > > > > > > > > > > Hi, sorry for the delay in reviewing this patch. I would like to ask > > > > Paul about it. > > > > > > > > While you're correctly fixing a false positive, hlist_for_each_entry_rcu > > > > would have a false _negative_ if you called it under > > > > rcu_read_lock/unlock and the data structure was protected by SRCU. This > > > > is why for example srcu_dereference is used instead of > > > > rcu_dereference_check, and why srcu_dereference uses > > > > __rcu_dereference_check (with the two underscores) instead of > > > > rcu_dereference_check. Using rcu_dereference_check would add an "|| > > > > rcu_read_lock_held()" to the condition which is wrong. > > > > > > > > I think instead you should add hlist_for_each_srcu and > > > > hlist_for_each_entry_srcu macro to include/linux/rculist.h. > > > > > > > > There is no need for equivalents of hlist_for_each_entry_continue_rcu > > > > and hlist_for_each_entry_from_rcu, because they use rcu_dereference_raw. > > > > However, it's not documented why they do so. > > > > > > You are right, this patch is wrong, we need a new SRCU list macro to do the > > > right thing which would also get rid of the last list argument. > > > > > Can we really get rid of the last argument? We would need the > > srcu_struct right for checking? > > Agreed! However, the API could be simplified by passing in a pointer to > the srcu_struct instead of a lockdep expression. An optional lockdep > expression might still be helpful for calls from the update side, > of course. > Sure, I will work on this. Thanks, Madhuparna > Thanx, Paul _______________________________________________ Linux-kernel-mentees mailing list Linux-kernel-mentees@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/linux-kernel-mentees ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Linux-kernel-mentees] [PATCH v2] kvm: Fix false positive RCU usage warning 2020-06-23 17:49 ` Madhuparna Bhowmik @ 2020-06-23 19:34 ` Joel Fernandes 0 siblings, 0 replies; 9+ messages in thread From: Joel Fernandes @ 2020-06-23 19:34 UTC (permalink / raw) To: Madhuparna Bhowmik Cc: wanpengli, kvm, Paul E. McKenney, x86, linux-kernel, sean.j.christopherson, bp, Paolo Bonzini, vkuznets, linux-kernel-mentees, tglx, jmattson On Tue, Jun 23, 2020 at 11:19:20PM +0530, Madhuparna Bhowmik wrote: > On Tue, Jun 23, 2020 at 08:39:01AM -0700, Paul E. McKenney wrote: > > On Tue, Jun 23, 2020 at 09:00:36PM +0530, Madhuparna Bhowmik wrote: > > > On Tue, Jun 23, 2020 at 11:02:36AM -0400, Joel Fernandes wrote: > > > > On Tue, Jun 23, 2020 at 09:39:53AM +0200, Paolo Bonzini wrote: > > > > > On 16/05/20 10:22, madhuparnabhowmik10@gmail.com wrote: > > > > > > From: Madhuparna Bhowmik <madhuparnabhowmik10@gmail.com> > > > > > > > > > > > > Fix the following false positive warnings: > > > > > > > > > > > > [ 9403.765413][T61744] ============================= > > > > > > [ 9403.786541][T61744] WARNING: suspicious RCU usage > > > > > > [ 9403.807865][T61744] 5.7.0-rc1-next-20200417 #4 Tainted: G L > > > > > > [ 9403.838945][T61744] ----------------------------- > > > > > > [ 9403.860099][T61744] arch/x86/kvm/mmu/page_track.c:257 RCU-list traversed in non-reader section!! > > > > > > > > > > > > and > > > > > > > > > > > > [ 9405.859252][T61751] ============================= > > > > > > [ 9405.859258][T61751] WARNING: suspicious RCU usage > > > > > > [ 9405.880867][T61755] ----------------------------- > > > > > > [ 9405.911936][T61751] 5.7.0-rc1-next-20200417 #4 Tainted: G L > > > > > > [ 9405.911942][T61751] ----------------------------- > > > > > > [ 9405.911950][T61751] arch/x86/kvm/mmu/page_track.c:232 RCU-list traversed in non-reader section!! > > > > > > > > > > > > Since srcu read lock is held, these are false positive warnings. > > > > > > Therefore, pass condition srcu_read_lock_held() to > > > > > > list_for_each_entry_rcu(). > > > > > > > > > > > > Reported-by: kernel test robot <lkp@intel.com> > > > > > > Signed-off-by: Madhuparna Bhowmik <madhuparnabhowmik10@gmail.com> > > > > > > --- > > > > > > v2: > > > > > > -Rebase v5.7-rc5 > > > > > > > > > > > > arch/x86/kvm/mmu/page_track.c | 6 ++++-- > > > > > > 1 file changed, 4 insertions(+), 2 deletions(-) > > > > > > > > > > > > diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c > > > > > > index ddc1ec3bdacd..1ad79c7aa05b 100644 > > > > > > --- a/arch/x86/kvm/mmu/page_track.c > > > > > > +++ b/arch/x86/kvm/mmu/page_track.c > > > > > > @@ -229,7 +229,8 @@ void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new, > > > > > > return; > > > > > > > > > > > > idx = srcu_read_lock(&head->track_srcu); > > > > > > - hlist_for_each_entry_rcu(n, &head->track_notifier_list, node) > > > > > > + hlist_for_each_entry_rcu(n, &head->track_notifier_list, node, > > > > > > + srcu_read_lock_held(&head->track_srcu)) > > > > > > if (n->track_write) > > > > > > n->track_write(vcpu, gpa, new, bytes, n); > > > > > > srcu_read_unlock(&head->track_srcu, idx); > > > > > > @@ -254,7 +255,8 @@ void kvm_page_track_flush_slot(struct kvm *kvm, struct kvm_memory_slot *slot) > > > > > > return; > > > > > > > > > > > > idx = srcu_read_lock(&head->track_srcu); > > > > > > - hlist_for_each_entry_rcu(n, &head->track_notifier_list, node) > > > > > > + hlist_for_each_entry_rcu(n, &head->track_notifier_list, node, > > > > > > + srcu_read_lock_held(&head->track_srcu)) > > > > > > if (n->track_flush_slot) > > > > > > n->track_flush_slot(kvm, slot, n); > > > > > > srcu_read_unlock(&head->track_srcu, idx); > > > > > > > > > > > > > > > > Hi, sorry for the delay in reviewing this patch. I would like to ask > > > > > Paul about it. > > > > > > > > > > While you're correctly fixing a false positive, hlist_for_each_entry_rcu > > > > > would have a false _negative_ if you called it under > > > > > rcu_read_lock/unlock and the data structure was protected by SRCU. This > > > > > is why for example srcu_dereference is used instead of > > > > > rcu_dereference_check, and why srcu_dereference uses > > > > > __rcu_dereference_check (with the two underscores) instead of > > > > > rcu_dereference_check. Using rcu_dereference_check would add an "|| > > > > > rcu_read_lock_held()" to the condition which is wrong. > > > > > > > > > > I think instead you should add hlist_for_each_srcu and > > > > > hlist_for_each_entry_srcu macro to include/linux/rculist.h. > > > > > > > > > > There is no need for equivalents of hlist_for_each_entry_continue_rcu > > > > > and hlist_for_each_entry_from_rcu, because they use rcu_dereference_raw. > > > > > However, it's not documented why they do so. > > > > > > > > You are right, this patch is wrong, we need a new SRCU list macro to do the > > > > right thing which would also get rid of the last list argument. > > > > > > > Can we really get rid of the last argument? We would need the > > > srcu_struct right for checking? > > > > Agreed! However, the API could be simplified by passing in a pointer to > > the srcu_struct instead of a lockdep expression. An optional lockdep > > expression might still be helpful for calls from the update side, > > of course. > > > Sure, I will work on this. Cool, thanks !!! - Joel _______________________________________________ Linux-kernel-mentees mailing list Linux-kernel-mentees@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/linux-kernel-mentees ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Linux-kernel-mentees] [PATCH v2] kvm: Fix false positive RCU usage warning 2020-06-23 7:39 ` Paolo Bonzini 2020-06-23 15:02 ` Joel Fernandes @ 2020-06-23 15:29 ` Madhuparna Bhowmik 1 sibling, 0 replies; 9+ messages in thread From: Madhuparna Bhowmik @ 2020-06-23 15:29 UTC (permalink / raw) To: Paolo Bonzini Cc: wanpengli, paulmck, kvm, x86, linux-kernel, sean.j.christopherson, madhuparnabhowmik10, bp, joel, vkuznets, Paul McKenney, linux-kernel-mentees, tglx, jmattson On Tue, Jun 23, 2020 at 09:39:53AM +0200, Paolo Bonzini wrote: > On 16/05/20 10:22, madhuparnabhowmik10@gmail.com wrote: > > From: Madhuparna Bhowmik <madhuparnabhowmik10@gmail.com> > > > > Fix the following false positive warnings: > > > > [ 9403.765413][T61744] ============================= > > [ 9403.786541][T61744] WARNING: suspicious RCU usage > > [ 9403.807865][T61744] 5.7.0-rc1-next-20200417 #4 Tainted: G L > > [ 9403.838945][T61744] ----------------------------- > > [ 9403.860099][T61744] arch/x86/kvm/mmu/page_track.c:257 RCU-list traversed in non-reader section!! > > > > and > > > > [ 9405.859252][T61751] ============================= > > [ 9405.859258][T61751] WARNING: suspicious RCU usage > > [ 9405.880867][T61755] ----------------------------- > > [ 9405.911936][T61751] 5.7.0-rc1-next-20200417 #4 Tainted: G L > > [ 9405.911942][T61751] ----------------------------- > > [ 9405.911950][T61751] arch/x86/kvm/mmu/page_track.c:232 RCU-list traversed in non-reader section!! > > > > Since srcu read lock is held, these are false positive warnings. > > Therefore, pass condition srcu_read_lock_held() to > > list_for_each_entry_rcu(). > > > > Reported-by: kernel test robot <lkp@intel.com> > > Signed-off-by: Madhuparna Bhowmik <madhuparnabhowmik10@gmail.com> > > --- > > v2: > > -Rebase v5.7-rc5 > > > > arch/x86/kvm/mmu/page_track.c | 6 ++++-- > > 1 file changed, 4 insertions(+), 2 deletions(-) > > > > diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c > > index ddc1ec3bdacd..1ad79c7aa05b 100644 > > --- a/arch/x86/kvm/mmu/page_track.c > > +++ b/arch/x86/kvm/mmu/page_track.c > > @@ -229,7 +229,8 @@ void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new, > > return; > > > > idx = srcu_read_lock(&head->track_srcu); > > - hlist_for_each_entry_rcu(n, &head->track_notifier_list, node) > > + hlist_for_each_entry_rcu(n, &head->track_notifier_list, node, > > + srcu_read_lock_held(&head->track_srcu)) > > if (n->track_write) > > n->track_write(vcpu, gpa, new, bytes, n); > > srcu_read_unlock(&head->track_srcu, idx); > > @@ -254,7 +255,8 @@ void kvm_page_track_flush_slot(struct kvm *kvm, struct kvm_memory_slot *slot) > > return; > > > > idx = srcu_read_lock(&head->track_srcu); > > - hlist_for_each_entry_rcu(n, &head->track_notifier_list, node) > > + hlist_for_each_entry_rcu(n, &head->track_notifier_list, node, > > + srcu_read_lock_held(&head->track_srcu)) > > if (n->track_flush_slot) > > n->track_flush_slot(kvm, slot, n); > > srcu_read_unlock(&head->track_srcu, idx); > > > > Hi, sorry for the delay in reviewing this patch. I would like to ask > Paul about it. > > While you're correctly fixing a false positive, hlist_for_each_entry_rcu > would have a false _negative_ if you called it under > rcu_read_lock/unlock and the data structure was protected by SRCU. This > is why for example srcu_dereference is used instead of > rcu_dereference_check, and why srcu_dereference uses > __rcu_dereference_check (with the two underscores) instead of > rcu_dereference_check. Using rcu_dereference_check would add an "|| > rcu_read_lock_held()" to the condition which is wrong. > Yes, that makes sense, there would be a false negative, thank you for pointing out this issue. > I think instead you should add hlist_for_each_srcu and > hlist_for_each_entry_srcu macro to include/linux/rculist.h. > This seems good to me, I can work on this, but I would wait for Paul's suggestion on this. > There is no need for equivalents of hlist_for_each_entry_continue_rcu > and hlist_for_each_entry_from_rcu, because they use rcu_dereference_raw. > However, it's not documented why they do so. > > Paul, do you have any objections to the idea? Thanks, > > Paolo Thank you, Madhuparna > _______________________________________________ Linux-kernel-mentees mailing list Linux-kernel-mentees@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/linux-kernel-mentees ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2020-06-23 19:41 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-05-16 8:22 [Linux-kernel-mentees] [PATCH v2] kvm: Fix false positive RCU usage warning madhuparnabhowmik10 2020-06-23 7:39 ` Paolo Bonzini 2020-06-23 15:02 ` Joel Fernandes 2020-06-23 15:30 ` Madhuparna Bhowmik 2020-06-23 15:39 ` Paul E. McKenney 2020-06-23 15:43 ` Joel Fernandes 2020-06-23 17:49 ` Madhuparna Bhowmik 2020-06-23 19:34 ` Joel Fernandes 2020-06-23 15:29 ` Madhuparna Bhowmik
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).