From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S946349AbdDTOUY (ORCPT ); Thu, 20 Apr 2017 10:20:24 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:57337 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S946332AbdDTOUT (ORCPT ); Thu, 20 Apr 2017 10:20:19 -0400 Date: Thu, 20 Apr 2017 07:19:57 -0700 From: "Paul E. McKenney" To: Michael Ellerman Cc: linux-kernel@vger.kernel.org, "paulus@samba.org" , kvm@vger.kernel.org, kvm-ppc@vger.kernel.org, "linuxppc-dev@lists.ozlabs.org" , pbonzini@redhat.com, mingo@kernel.org, jiangshanlai@gmail.com, dipankar@in.ibm.com, akpm@linux-foundation.org, mathieu.desnoyers@efficios.com, josh@joshtriplett.org, tglx@linutronix.de, peterz@infradead.org, rostedt@goodmis.org, dhowells@redhat.com, edumazet@google.com, fweisbec@gmail.com, oleg@redhat.com, bobby.prani@gmail.com Subject: Re: powerpc KVM build break in linux-next (was Re: [PATCH tip/core/rcu 40/40] srcu: Parallelize callback handling) Reply-To: paulmck@linux.vnet.ibm.com References: <20170412174003.GA23207@linux.vnet.ibm.com> <1492018825-25634-40-git-send-email-paulmck@linux.vnet.ibm.com> <871ssn692a.fsf@concordia.ellerman.id.au> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <871ssn692a.fsf@concordia.ellerman.id.au> User-Agent: Mutt/1.5.21 (2010-09-15) X-TM-AS-GCONF: 00 x-cbid: 17042014-0052-0000-0000-000001DF2DE0 X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00006945; HX=3.00000240; KW=3.00000007; PH=3.00000004; SC=3.00000208; SDB=6.00850110; UDB=6.00419840; IPR=6.00628739; BA=6.00005305; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00015109; XFM=3.00000013; UTC=2017-04-20 14:20:16 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 17042014-0053-0000-0000-000050192643 Message-Id: <20170420141957.GK3956@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-04-20_13:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=0 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1703280000 definitions=main-1704200114 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Apr 20, 2017 at 01:40:13PM +1000, Michael Ellerman wrote: > "Paul E. McKenney" writes: > > > diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h > > index f2b3bd6c6bc2..0400e211aa44 100644 > > --- a/include/linux/srcutree.h > > +++ b/include/linux/srcutree.h > > @@ -24,25 +24,75 @@ > ... > > +/* > > + * Per-SRCU-domain structure, similar in function to rcu_state. > > + */ > > struct srcu_struct { > > - unsigned long completed; > > - unsigned long srcu_gp_seq; > > - atomic_t srcu_exp_cnt; > > - struct srcu_array __percpu *per_cpu_ref; > > - spinlock_t queue_lock; /* protect ->srcu_cblist */ > > - struct rcu_segcblist srcu_cblist; > > + struct srcu_node node[NUM_RCU_NODES]; /* Combining tree. */ > > + struct srcu_node *level[RCU_NUM_LVLS + 1]; > > + /* First node at each level. */ > > + struct mutex srcu_cb_mutex; /* Serialize CB preparation. */ > > + spinlock_t gp_lock; /* protect ->srcu_cblist */ > > + struct mutex srcu_gp_mutex; /* Serialize GP work. */ > > + unsigned int srcu_idx; /* Current rdr array element. */ > > + unsigned long srcu_gp_seq; /* Grace-period seq #. */ > > + unsigned long srcu_gp_seq_needed; /* Latest gp_seq needed. */ > > + atomic_t srcu_exp_cnt; /* # ongoing expedited GPs. */ > > + struct srcu_data __percpu *sda; /* Per-CPU srcu_data array. */ > > + unsigned long srcu_barrier_seq; /* srcu_barrier seq #. */ > > + struct mutex srcu_barrier_mutex; /* Serialize barrier ops. */ > > + struct completion srcu_barrier_completion; > > + /* Awaken barrier rq at end. */ > > + atomic_t srcu_barrier_cpu_cnt; /* # CPUs not yet posting a */ > > + /* callback for the barrier */ > > + /* operation. */ > > This change seems to have had the non-obvious effect of breaking the > powerpc KVM build. > > Because struct kvm contains two srcu_structs which are located > before the kvm_arch struct, the increase in size of srcu_struct has > caused the offset from the start of struct kvm to kvm_arch to be too big > for some of our asm. > > struct kvm { > spinlock_t mmu_lock; > struct mutex slots_lock; > struct mm_struct *mm; /* userspace tied to this vm */ > struct kvm_memslots *memslots[KVM_ADDRESS_SPACE_NUM]; > struct srcu_struct srcu; > struct srcu_struct irq_srcu; > ... > struct kvm_arch arch; > > > Example error: > arch/powerpc/kvm/book3s_hv_rmhandlers.S:617: Error: operand out of range (0x000000000000b328 is not between 0xffffffffffff8000 and 0x0000000000007fff) > > Where line 617 is: > lwz r7,KVM_LPID(r9) > > And the KVM_LPID constant comes from asm-offsets.s. The diff of old vs > new is: > > -->KVM_LPID 17752 offsetof(struct kvm, arch.lpid) # > +->KVM_LPID 45864 offsetof(struct kvm, arch.lpid) # > > > We can probably just fix it by changing the asm to keep the address of > kvm_arch in a register, and then offset from that. Ouch! What do you need from me? Thanx, Paul From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Paul E. McKenney" Date: Thu, 20 Apr 2017 14:19:57 +0000 Subject: Re: powerpc KVM build break in linux-next (was Re: [PATCH tip/core/rcu 40/40] srcu: Parallelize call Message-Id: <20170420141957.GK3956@linux.vnet.ibm.com> List-Id: References: <20170412174003.GA23207@linux.vnet.ibm.com> <1492018825-25634-40-git-send-email-paulmck@linux.vnet.ibm.com> <871ssn692a.fsf@concordia.ellerman.id.au> In-Reply-To: <871ssn692a.fsf@concordia.ellerman.id.au> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: Michael Ellerman Cc: linux-kernel@vger.kernel.org, "paulus@samba.org" , kvm@vger.kernel.org, kvm-ppc@vger.kernel.org, "linuxppc-dev@lists.ozlabs.org" , pbonzini@redhat.com, mingo@kernel.org, jiangshanlai@gmail.com, dipankar@in.ibm.com, akpm@linux-foundation.org, mathieu.desnoyers@efficios.com, josh@joshtriplett.org, tglx@linutronix.de, peterz@infradead.org, rostedt@goodmis.org, dhowells@redhat.com, edumazet@google.com, fweisbec@gmail.com, oleg@redhat.com, bobby.prani@gmail.com On Thu, Apr 20, 2017 at 01:40:13PM +1000, Michael Ellerman wrote: > "Paul E. McKenney" writes: > > > diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h > > index f2b3bd6c6bc2..0400e211aa44 100644 > > --- a/include/linux/srcutree.h > > +++ b/include/linux/srcutree.h > > @@ -24,25 +24,75 @@ > ... > > +/* > > + * Per-SRCU-domain structure, similar in function to rcu_state. > > + */ > > struct srcu_struct { > > - unsigned long completed; > > - unsigned long srcu_gp_seq; > > - atomic_t srcu_exp_cnt; > > - struct srcu_array __percpu *per_cpu_ref; > > - spinlock_t queue_lock; /* protect ->srcu_cblist */ > > - struct rcu_segcblist srcu_cblist; > > + struct srcu_node node[NUM_RCU_NODES]; /* Combining tree. */ > > + struct srcu_node *level[RCU_NUM_LVLS + 1]; > > + /* First node at each level. */ > > + struct mutex srcu_cb_mutex; /* Serialize CB preparation. */ > > + spinlock_t gp_lock; /* protect ->srcu_cblist */ > > + struct mutex srcu_gp_mutex; /* Serialize GP work. */ > > + unsigned int srcu_idx; /* Current rdr array element. */ > > + unsigned long srcu_gp_seq; /* Grace-period seq #. */ > > + unsigned long srcu_gp_seq_needed; /* Latest gp_seq needed. */ > > + atomic_t srcu_exp_cnt; /* # ongoing expedited GPs. */ > > + struct srcu_data __percpu *sda; /* Per-CPU srcu_data array. */ > > + unsigned long srcu_barrier_seq; /* srcu_barrier seq #. */ > > + struct mutex srcu_barrier_mutex; /* Serialize barrier ops. */ > > + struct completion srcu_barrier_completion; > > + /* Awaken barrier rq at end. */ > > + atomic_t srcu_barrier_cpu_cnt; /* # CPUs not yet posting a */ > > + /* callback for the barrier */ > > + /* operation. */ > > This change seems to have had the non-obvious effect of breaking the > powerpc KVM build. > > Because struct kvm contains two srcu_structs which are located > before the kvm_arch struct, the increase in size of srcu_struct has > caused the offset from the start of struct kvm to kvm_arch to be too big > for some of our asm. > > struct kvm { > spinlock_t mmu_lock; > struct mutex slots_lock; > struct mm_struct *mm; /* userspace tied to this vm */ > struct kvm_memslots *memslots[KVM_ADDRESS_SPACE_NUM]; > struct srcu_struct srcu; > struct srcu_struct irq_srcu; > ... > struct kvm_arch arch; > > > Example error: > arch/powerpc/kvm/book3s_hv_rmhandlers.S:617: Error: operand out of range (0x000000000000b328 is not between 0xffffffffffff8000 and 0x0000000000007fff) > > Where line 617 is: > lwz r7,KVM_LPID(r9) > > And the KVM_LPID constant comes from asm-offsets.s. The diff of old vs > new is: > > -->KVM_LPID 17752 offsetof(struct kvm, arch.lpid) # > +->KVM_LPID 45864 offsetof(struct kvm, arch.lpid) # > > > We can probably just fix it by changing the asm to keep the address of > kvm_arch in a register, and then offset from that. Ouch! What do you need from me? Thanx, Paul