From mboxrd@z Thu Jan 1 00:00:00 1970 From: David Gibson Subject: Re: [PATCH kernel] vfio/spapr: Add cond_resched() for huge updates Date: Sat, 23 Sep 2017 13:58:36 +1000 Message-ID: <20170923035836.GV4998@umbus.fritz.box> References: <20170921100024.44924-1-aik@ozlabs.ru> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="2tWkrNKppd65XSnD" Cc: kvm@vger.kernel.org, kvm-ppc@vger.kernel.org, Alex Williamson To: Alexey Kardashevskiy Return-path: Received: from ozlabs.org ([103.22.144.67]:42943 "EHLO ozlabs.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750707AbdIWFEb (ORCPT ); Sat, 23 Sep 2017 01:04:31 -0400 Content-Disposition: inline In-Reply-To: <20170921100024.44924-1-aik@ozlabs.ru> Sender: kvm-owner@vger.kernel.org List-ID: --2tWkrNKppd65XSnD Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Sep 21, 2017 at 08:00:24PM +1000, Alexey Kardashevskiy wrote: > Clearing very big IOMMU tables can trigger soft lockups. This adds > cond_resched() for every million TCE updates. >=20 > The testcase is POWER9 box with 264GB guest, 4 VFIO devices from > independent IOMMU groups, 64K IOMMU pages. This configuration produces > 4325376 TCE entries, each entry update incurs 4 OPAL calls to update > an individual PE TCE cache. Reducing table size to 4194304 (i.e. 256GB > guest) or removing one of 4 VFIO devices makes the problem go away so > doing cond_resched() after every million TCE updates seems sufficient. >=20 > Signed-off-by: Alexey Kardashevskiy Reviewed-by: David Gibson > --- > drivers/vfio/vfio_iommu_spapr_tce.c | 8 ++++++-- > 1 file changed, 6 insertions(+), 2 deletions(-) >=20 > diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c b/drivers/vfio/vfio_iomm= u_spapr_tce.c > index 63112c36ab2d..be3839ea3150 100644 > --- a/drivers/vfio/vfio_iommu_spapr_tce.c > +++ b/drivers/vfio/vfio_iommu_spapr_tce.c > @@ -502,11 +502,15 @@ static int tce_iommu_clear(struct tce_container *co= ntainer, > struct iommu_table *tbl, > unsigned long entry, unsigned long pages) > { > - unsigned long oldhpa; > + unsigned long oldhpa, n; > long ret; > enum dma_data_direction direction; > =20 > - for ( ; pages; --pages, ++entry) { > + for (n =3D 0; pages; --pages, ++entry, ++n) { > + > + if (n && (n % 1000000 =3D=3D 0)) > + cond_resched(); > + > direction =3D DMA_NONE; > oldhpa =3D 0; > ret =3D iommu_tce_xchg(tbl, entry, &oldhpa, &direction); --=20 David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson --2tWkrNKppd65XSnD Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEdfRlhq5hpmzETofcbDjKyiDZs5IFAlnF2+kACgkQbDjKyiDZ s5Iz+BAAkogzODnfCDfrQxOb2Joo8OmDvuNwfBpbu7D1KehTFM8MXuX3VDMmqLHl OwZY0/7oRIcrTZQ39lF6sxfOjnce9czA/vvvwaLfqvk1nMy+wmDeLIdrXAQc7vO9 k18jxgRas1YyxEWaHHbYq+Br6Q1g4jmxHIuGKWGVB5gCzruaQyFDcIZRxw+aTJUz zrBc8kbyVa15mSvvP2D4G8CQYS2z1VJtx9F2uOlxeXpqs174A8L8/IX1LUQRRDUR jZkx7ms3kaKzloGhytHzq/Rx/GMNDZkYzXlVyTZU9IJnG4c3m1hE9Ktg4M8/OHIy PHp035+vOGvwqGN3H3FG0sybtgKyZK1qHx750myCdmONQuSxMFBcb899dMZVOYBy tPf11ggD4Wu6wibZQjlVR1jJgYZpZnwXPXpYn+I4hxeWWqdc4u0KmuHCtpbO9Vop PoSMxqbnvLJsTnFFUYEA5qWVtim8hBha5I6O3MFNoqvDyXfe3oBCnWy80HLYo4Oe S5fmhstEQgvZydqJqWiaNM3iBxXSrbBRPRDrWakso65fTdrMZIbXkxrMSZiZphF9 JewAPtY5K6P3HPgVKFjt8JDoFpPccQ84zD4vgqiffBalB55jTQznsY28EDIMEGCa U+DPcy9FlNuN/ucei/7rOoOHHTzrFhIjZLbig8XjXrss5G0rxrg= =mzuR -----END PGP SIGNATURE----- --2tWkrNKppd65XSnD-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: David Gibson Date: Sat, 23 Sep 2017 03:58:36 +0000 Subject: Re: [PATCH kernel] vfio/spapr: Add cond_resched() for huge updates Message-Id: <20170923035836.GV4998@umbus.fritz.box> MIME-Version: 1 Content-Type: multipart/mixed; boundary="2tWkrNKppd65XSnD" List-Id: References: <20170921100024.44924-1-aik@ozlabs.ru> In-Reply-To: <20170921100024.44924-1-aik@ozlabs.ru> To: Alexey Kardashevskiy Cc: kvm@vger.kernel.org, kvm-ppc@vger.kernel.org, Alex Williamson --2tWkrNKppd65XSnD Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Sep 21, 2017 at 08:00:24PM +1000, Alexey Kardashevskiy wrote: > Clearing very big IOMMU tables can trigger soft lockups. This adds > cond_resched() for every million TCE updates. >=20 > The testcase is POWER9 box with 264GB guest, 4 VFIO devices from > independent IOMMU groups, 64K IOMMU pages. This configuration produces > 4325376 TCE entries, each entry update incurs 4 OPAL calls to update > an individual PE TCE cache. Reducing table size to 4194304 (i.e. 256GB > guest) or removing one of 4 VFIO devices makes the problem go away so > doing cond_resched() after every million TCE updates seems sufficient. >=20 > Signed-off-by: Alexey Kardashevskiy Reviewed-by: David Gibson > --- > drivers/vfio/vfio_iommu_spapr_tce.c | 8 ++++++-- > 1 file changed, 6 insertions(+), 2 deletions(-) >=20 > diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c b/drivers/vfio/vfio_iomm= u_spapr_tce.c > index 63112c36ab2d..be3839ea3150 100644 > --- a/drivers/vfio/vfio_iommu_spapr_tce.c > +++ b/drivers/vfio/vfio_iommu_spapr_tce.c > @@ -502,11 +502,15 @@ static int tce_iommu_clear(struct tce_container *co= ntainer, > struct iommu_table *tbl, > unsigned long entry, unsigned long pages) > { > - unsigned long oldhpa; > + unsigned long oldhpa, n; > long ret; > enum dma_data_direction direction; > =20 > - for ( ; pages; --pages, ++entry) { > + for (n =3D 0; pages; --pages, ++entry, ++n) { > + > + if (n && (n % 1000000 =3D=3D 0)) > + cond_resched(); > + > direction =3D DMA_NONE; > oldhpa =3D 0; > ret =3D iommu_tce_xchg(tbl, entry, &oldhpa, &direction); --=20 David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson --2tWkrNKppd65XSnD Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEdfRlhq5hpmzETofcbDjKyiDZs5IFAlnF2+kACgkQbDjKyiDZ s5Iz+BAAkogzODnfCDfrQxOb2Joo8OmDvuNwfBpbu7D1KehTFM8MXuX3VDMmqLHl OwZY0/7oRIcrTZQ39lF6sxfOjnce9czA/vvvwaLfqvk1nMy+wmDeLIdrXAQc7vO9 k18jxgRas1YyxEWaHHbYq+Br6Q1g4jmxHIuGKWGVB5gCzruaQyFDcIZRxw+aTJUz zrBc8kbyVa15mSvvP2D4G8CQYS2z1VJtx9F2uOlxeXpqs174A8L8/IX1LUQRRDUR jZkx7ms3kaKzloGhytHzq/Rx/GMNDZkYzXlVyTZU9IJnG4c3m1hE9Ktg4M8/OHIy PHp035+vOGvwqGN3H3FG0sybtgKyZK1qHx750myCdmONQuSxMFBcb899dMZVOYBy tPf11ggD4Wu6wibZQjlVR1jJgYZpZnwXPXpYn+I4hxeWWqdc4u0KmuHCtpbO9Vop PoSMxqbnvLJsTnFFUYEA5qWVtim8hBha5I6O3MFNoqvDyXfe3oBCnWy80HLYo4Oe S5fmhstEQgvZydqJqWiaNM3iBxXSrbBRPRDrWakso65fTdrMZIbXkxrMSZiZphF9 JewAPtY5K6P3HPgVKFjt8JDoFpPccQ84zD4vgqiffBalB55jTQznsY28EDIMEGCa U+DPcy9FlNuN/ucei/7rOoOHHTzrFhIjZLbig8XjXrss5G0rxrg= =mzuR -----END PGP SIGNATURE----- --2tWkrNKppd65XSnD--