From: Halil Pasic <pasic@linux.ibm.com> To: Sebastian Ott <sebott@linux.ibm.com> Cc: Cornelia Huck <cohuck@redhat.com>, kvm@vger.kernel.org, linux-s390@vger.kernel.org, Martin Schwidefsky <schwidefsky@de.ibm.com>, virtualization@lists.linux-foundation.org, "Michael S. Tsirkin" <mst@redhat.com>, Christoph Hellwig <hch@infradead.org>, Thomas Huth <thuth@redhat.com>, Christian Borntraeger <borntraeger@de.ibm.com>, Viktor Mihajlovski <mihajlov@linux.ibm.com>, Vasily Gorbik <gor@linux.ibm.com>, Janosch Frank <frankja@linux.ibm.com>, Claudio Imbrenda <imbrenda@linux.ibm.com>, Farhan Ali <alifm@linux.ibm.com>, Eric Farman <farman@linux.ibm.com>, Michael Mueller <mimu@linux.ibm.com> Subject: Re: [PATCH 05/10] s390/cio: introduce DMA pools to cio Date: Thu, 23 May 2019 00:12:46 +0200 [thread overview] Message-ID: <20190523001246.449154aa.pasic@linux.ibm.com> (raw) In-Reply-To: <alpine.LFD.2.21.1905221344180.1782@schleppi> On Wed, 22 May 2019 14:07:05 +0200 (CEST) Sebastian Ott <sebott@linux.ibm.com> wrote: > On Mon, 20 May 2019, Halil Pasic wrote: > > On Thu, 16 May 2019 15:59:22 +0200 (CEST) > > Sebastian Ott <sebott@linux.ibm.com> wrote: > > > We only have a couple of users for airq_iv: > > > > > > virtio_ccw.c: 2K bits > > > > You mean a single allocation is 2k bits (VIRTIO_IV_BITS = 256 * 8)? My > > understanding is that the upper bound is more like: > > MAX_AIRQ_AREAS * VIRTIO_IV_BITS = 20 * 256 * 8 = 40960 bits. > > > > In practice it is most likely just 2k. > > > > > > > > pci with floating IRQs: <= 2K (for the per-function bit vectors) > > > 1..4K (for the summary bit vector) > > > > > > > As far as I can tell with virtio_pci arch_setup_msi_irqs() gets called > > once per device and allocates a small number of bits (2 and 3 in my > > test, it may depend on #virtqueues, but I did not check). > > > > So for an upper bound we would have to multiply with the upper bound of > > pci devices/functions. What is the upper bound on the number of > > functions? > > > > > pci with CPU directed IRQs: 2K (for the per-CPU bit vectors) > > > 1..nr_cpu (for the summary bit vector) > > > > > > > I guess this is the same. > > > > > > > > The options are: > > > * page allocations for everything > > > > Worst case we need 20 + #max_pci_dev pages. At the moment we allocate > > from ZONE_DMA (!) and waste a lot. > > > > > * dma_pool for AIRQ_IV_CACHELINE ,gen_pool for others > > > > I prefer this. Explanation follows. > > > > > * dma_pool for everything > > > > > > > Less waste by factor factor 16. > > > > > I think we should do option 3 and use a dma_pool with cachesize > > > alignment for everything (as a prerequisite we have to limit > > > config PCI_NR_FUNCTIONS to 2K - but that is not a real constraint). > > > > I prefer option 3 because it is conceptually the smallest change, and > ^ > 2 > > provides the behavior which is closest to the current one. > > I can see that this is the smallest change on top of the current > implementation. I'm good with doing that and looking for further > simplification/unification later. > Nod. I will go with that for v2. > > > Commit 414cbd1e3d14 "s390/airq: provide cacheline aligned > > ivs" (Sebastian Ott, 2019-02-27) could have been smaller had you implemented > > 'kmem_cache for everything' (and I would have had just to replace kmem_cache with > > dma_cache to achieve option 3). For some reason you decided to keep the > > iv->vector = kzalloc(size, GFP_KERNEL) code-path and make the client code request > > iv->vector = kmem_cache_zalloc(airq_iv_cache, GFP_KERNEL) explicitly, using a flag > > which you only decided to use for directed pci irqs AFAICT. > > > > My understanding of these decisions, and especially of the rationale > > behind commit 414cbd1e3d14 is limited. > > I introduced per cpu interrupt vectors and wanted to prevent 2 CPUs from > sharing data from the same cacheline. No other user of the airq stuff had > this need. If I had been aware of the additional complexity we would add > on top of that maybe I would have made a different decision. I understand. Thanks for the explanation! Regards, Halil
WARNING: multiple messages have this Message-ID (diff)
From: Halil Pasic <pasic@linux.ibm.com> To: Sebastian Ott <sebott@linux.ibm.com> Cc: Farhan Ali <alifm@linux.ibm.com>, linux-s390@vger.kernel.org, Thomas Huth <thuth@redhat.com>, Claudio Imbrenda <imbrenda@linux.ibm.com>, Vasily Gorbik <gor@linux.ibm.com>, kvm@vger.kernel.org, "Michael S. Tsirkin" <mst@redhat.com>, Cornelia Huck <cohuck@redhat.com>, Eric Farman <farman@linux.ibm.com>, virtualization@lists.linux-foundation.org, Christoph Hellwig <hch@infradead.org>, Christian Borntraeger <borntraeger@de.ibm.com>, Martin Schwidefsky <schwidefsky@de.ibm.com>, Michael Mueller <mimu@linux.ibm.com>, Viktor Mihajlovski <mihajlov@linux.ibm.com>, Janosch Frank <frankja@linux.ibm.com> Subject: Re: [PATCH 05/10] s390/cio: introduce DMA pools to cio Date: Thu, 23 May 2019 00:12:46 +0200 [thread overview] Message-ID: <20190523001246.449154aa.pasic@linux.ibm.com> (raw) In-Reply-To: <alpine.LFD.2.21.1905221344180.1782@schleppi> On Wed, 22 May 2019 14:07:05 +0200 (CEST) Sebastian Ott <sebott@linux.ibm.com> wrote: > On Mon, 20 May 2019, Halil Pasic wrote: > > On Thu, 16 May 2019 15:59:22 +0200 (CEST) > > Sebastian Ott <sebott@linux.ibm.com> wrote: > > > We only have a couple of users for airq_iv: > > > > > > virtio_ccw.c: 2K bits > > > > You mean a single allocation is 2k bits (VIRTIO_IV_BITS = 256 * 8)? My > > understanding is that the upper bound is more like: > > MAX_AIRQ_AREAS * VIRTIO_IV_BITS = 20 * 256 * 8 = 40960 bits. > > > > In practice it is most likely just 2k. > > > > > > > > pci with floating IRQs: <= 2K (for the per-function bit vectors) > > > 1..4K (for the summary bit vector) > > > > > > > As far as I can tell with virtio_pci arch_setup_msi_irqs() gets called > > once per device and allocates a small number of bits (2 and 3 in my > > test, it may depend on #virtqueues, but I did not check). > > > > So for an upper bound we would have to multiply with the upper bound of > > pci devices/functions. What is the upper bound on the number of > > functions? > > > > > pci with CPU directed IRQs: 2K (for the per-CPU bit vectors) > > > 1..nr_cpu (for the summary bit vector) > > > > > > > I guess this is the same. > > > > > > > > The options are: > > > * page allocations for everything > > > > Worst case we need 20 + #max_pci_dev pages. At the moment we allocate > > from ZONE_DMA (!) and waste a lot. > > > > > * dma_pool for AIRQ_IV_CACHELINE ,gen_pool for others > > > > I prefer this. Explanation follows. > > > > > * dma_pool for everything > > > > > > > Less waste by factor factor 16. > > > > > I think we should do option 3 and use a dma_pool with cachesize > > > alignment for everything (as a prerequisite we have to limit > > > config PCI_NR_FUNCTIONS to 2K - but that is not a real constraint). > > > > I prefer option 3 because it is conceptually the smallest change, and > ^ > 2 > > provides the behavior which is closest to the current one. > > I can see that this is the smallest change on top of the current > implementation. I'm good with doing that and looking for further > simplification/unification later. > Nod. I will go with that for v2. > > > Commit 414cbd1e3d14 "s390/airq: provide cacheline aligned > > ivs" (Sebastian Ott, 2019-02-27) could have been smaller had you implemented > > 'kmem_cache for everything' (and I would have had just to replace kmem_cache with > > dma_cache to achieve option 3). For some reason you decided to keep the > > iv->vector = kzalloc(size, GFP_KERNEL) code-path and make the client code request > > iv->vector = kmem_cache_zalloc(airq_iv_cache, GFP_KERNEL) explicitly, using a flag > > which you only decided to use for directed pci irqs AFAICT. > > > > My understanding of these decisions, and especially of the rationale > > behind commit 414cbd1e3d14 is limited. > > I introduced per cpu interrupt vectors and wanted to prevent 2 CPUs from > sharing data from the same cacheline. No other user of the airq stuff had > this need. If I had been aware of the additional complexity we would add > on top of that maybe I would have made a different decision. I understand. Thanks for the explanation! Regards, Halil
next prev parent reply other threads:[~2019-05-22 22:12 UTC|newest] Thread overview: 182+ messages / expand[flat|nested] mbox.gz Atom feed top 2019-04-26 18:32 [PATCH 00/10] s390: virtio: support protected virtualization Halil Pasic 2019-04-26 18:32 ` Halil Pasic 2019-04-26 18:32 ` [PATCH 01/10] virtio/s390: use vring_create_virtqueue Halil Pasic 2019-04-26 18:32 ` Halil Pasic 2019-05-03 9:17 ` Cornelia Huck 2019-05-03 20:04 ` Michael S. Tsirkin 2019-05-03 20:04 ` Michael S. Tsirkin 2019-05-04 14:03 ` Halil Pasic 2019-05-04 14:03 ` Halil Pasic 2019-05-05 11:15 ` Cornelia Huck 2019-05-05 11:15 ` Cornelia Huck 2019-05-07 13:58 ` Christian Borntraeger 2019-05-07 13:58 ` Christian Borntraeger 2019-05-08 20:12 ` Halil Pasic 2019-05-08 20:12 ` Halil Pasic 2019-05-10 14:07 ` Cornelia Huck 2019-05-10 14:07 ` Cornelia Huck 2019-05-12 16:47 ` Michael S. Tsirkin 2019-05-12 16:47 ` Michael S. Tsirkin 2019-05-13 9:52 ` Cornelia Huck 2019-05-13 9:52 ` Cornelia Huck 2019-05-13 12:27 ` Michael Mueller 2019-05-13 12:27 ` Michael Mueller 2019-05-13 12:29 ` Cornelia Huck 2019-05-13 12:29 ` Cornelia Huck 2019-04-26 18:32 ` [PATCH 02/10] virtio/s390: DMA support for virtio-ccw Halil Pasic 2019-04-26 18:32 ` Halil Pasic 2019-05-03 9:31 ` Cornelia Huck 2019-04-26 18:32 ` [PATCH 03/10] virtio/s390: enable packed ring Halil Pasic 2019-04-26 18:32 ` Halil Pasic 2019-05-03 9:44 ` Cornelia Huck 2019-05-05 15:13 ` Thomas Huth 2019-05-05 15:13 ` Thomas Huth 2019-04-26 18:32 ` [PATCH 04/10] s390/mm: force swiotlb for protected virtualization Halil Pasic 2019-04-26 18:32 ` Halil Pasic 2019-04-26 19:27 ` Christoph Hellwig 2019-04-26 19:27 ` Christoph Hellwig 2019-04-29 13:59 ` Halil Pasic 2019-04-29 13:59 ` Halil Pasic 2019-04-29 14:05 ` Christian Borntraeger 2019-04-29 14:05 ` Christian Borntraeger 2019-05-13 12:50 ` Michael Mueller 2019-05-13 12:50 ` Michael Mueller 2019-05-08 13:15 ` Claudio Imbrenda 2019-05-08 13:15 ` Claudio Imbrenda 2019-05-09 22:34 ` Halil Pasic 2019-05-09 22:34 ` Halil Pasic 2019-05-15 14:15 ` Michael Mueller 2019-05-15 14:15 ` Michael Mueller [not found] ` <ad23f5e7-dc78-04af-c892-47bbc65134c6@linux.ibm.com> 2019-05-09 18:05 ` Jason J. Herne 2019-05-09 18:05 ` Jason J. Herne 2019-05-09 18:05 ` Jason J. Herne 2019-05-10 7:49 ` Claudio Imbrenda 2019-05-10 7:49 ` Claudio Imbrenda 2019-04-26 18:32 ` [PATCH 05/10] s390/cio: introduce DMA pools to cio Halil Pasic 2019-04-26 18:32 ` Halil Pasic 2019-05-08 13:18 ` Sebastian Ott 2019-05-08 13:18 ` Sebastian Ott 2019-05-08 21:22 ` Halil Pasic 2019-05-08 21:22 ` Halil Pasic 2019-05-09 8:40 ` Sebastian Ott 2019-05-09 8:40 ` Sebastian Ott 2019-05-09 10:11 ` Cornelia Huck 2019-05-09 10:11 ` Cornelia Huck 2019-05-09 22:11 ` Halil Pasic 2019-05-09 22:11 ` Halil Pasic 2019-05-10 14:10 ` Cornelia Huck 2019-05-10 14:10 ` Cornelia Huck 2019-05-12 18:22 ` Halil Pasic 2019-05-12 18:22 ` Halil Pasic 2019-05-13 13:29 ` Cornelia Huck 2019-05-13 13:29 ` Cornelia Huck 2019-05-15 17:12 ` Halil Pasic 2019-05-15 17:12 ` Halil Pasic 2019-05-16 6:13 ` Cornelia Huck 2019-05-16 6:13 ` Cornelia Huck 2019-05-16 13:59 ` Sebastian Ott 2019-05-16 13:59 ` Sebastian Ott 2019-05-20 12:13 ` Halil Pasic 2019-05-20 12:13 ` Halil Pasic 2019-05-21 8:46 ` Michael Mueller 2019-05-21 8:46 ` Michael Mueller 2019-05-22 12:07 ` Sebastian Ott 2019-05-22 12:07 ` Sebastian Ott 2019-05-22 22:12 ` Halil Pasic [this message] 2019-05-22 22:12 ` Halil Pasic 2019-05-23 15:17 ` Halil Pasic 2019-05-23 15:17 ` Halil Pasic 2019-04-26 18:32 ` [PATCH 06/10] s390/cio: add basic protected virtualization support Halil Pasic 2019-04-26 18:32 ` Halil Pasic 2019-05-08 13:46 ` Sebastian Ott 2019-05-08 13:46 ` Sebastian Ott 2019-05-08 13:54 ` Christoph Hellwig 2019-05-08 13:54 ` Christoph Hellwig 2019-05-08 21:08 ` Halil Pasic 2019-05-08 21:08 ` Halil Pasic 2019-05-09 8:52 ` Sebastian Ott 2019-05-09 8:52 ` Sebastian Ott 2019-05-08 14:23 ` Pierre Morel 2019-05-08 14:23 ` Pierre Morel 2019-05-13 9:41 ` Cornelia Huck 2019-05-13 9:41 ` Cornelia Huck 2019-05-14 14:47 ` Jason J. Herne 2019-05-14 14:47 ` Jason J. Herne 2019-05-15 21:08 ` Halil Pasic 2019-05-15 21:08 ` Halil Pasic 2019-05-16 6:32 ` Cornelia Huck 2019-05-16 6:32 ` Cornelia Huck 2019-05-16 13:42 ` Halil Pasic 2019-05-16 13:42 ` Halil Pasic 2019-05-16 13:54 ` Cornelia Huck 2019-05-16 13:54 ` Cornelia Huck 2019-05-15 20:51 ` Halil Pasic 2019-05-15 20:51 ` Halil Pasic 2019-05-16 6:29 ` Cornelia Huck 2019-05-16 6:29 ` Cornelia Huck 2019-05-18 18:11 ` Halil Pasic 2019-05-18 18:11 ` Halil Pasic 2019-05-20 10:21 ` Cornelia Huck 2019-05-20 10:21 ` Cornelia Huck 2019-05-20 12:34 ` Halil Pasic 2019-05-20 12:34 ` Halil Pasic 2019-05-20 13:43 ` Cornelia Huck 2019-05-20 13:43 ` Cornelia Huck 2019-04-26 18:32 ` [PATCH 07/10] s390/airq: use DMA memory for adapter interrupts Halil Pasic 2019-04-26 18:32 ` Halil Pasic 2019-05-08 13:58 ` Sebastian Ott 2019-05-08 13:58 ` Sebastian Ott 2019-05-09 11:37 ` Cornelia Huck 2019-05-09 11:37 ` Cornelia Huck 2019-05-13 12:59 ` Cornelia Huck 2019-05-13 12:59 ` Cornelia Huck 2019-04-26 18:32 ` [PATCH 08/10] virtio/s390: add indirection to indicators access Halil Pasic 2019-04-26 18:32 ` Halil Pasic 2019-05-08 14:31 ` Pierre Morel 2019-05-08 14:31 ` Pierre Morel 2019-05-09 12:01 ` Pierre Morel 2019-05-09 12:01 ` Pierre Morel 2019-05-09 18:26 ` Halil Pasic 2019-05-09 18:26 ` Halil Pasic 2019-05-10 7:43 ` Pierre Morel 2019-05-10 7:43 ` Pierre Morel 2019-05-10 11:54 ` Halil Pasic 2019-05-10 11:54 ` Halil Pasic 2019-05-10 15:36 ` Pierre Morel 2019-05-10 15:36 ` Pierre Morel 2019-05-13 10:15 ` Cornelia Huck 2019-05-13 10:15 ` Cornelia Huck 2019-05-16 15:24 ` Pierre Morel 2019-05-16 15:24 ` Pierre Morel 2019-04-26 18:32 ` [PATCH 09/10] virtio/s390: use DMA memory for ccw I/O and classic notifiers Halil Pasic 2019-04-26 18:32 ` Halil Pasic 2019-05-08 14:46 ` Pierre Morel 2019-05-08 14:46 ` Pierre Morel 2019-05-09 13:30 ` Pierre Morel 2019-05-09 13:30 ` Pierre Morel 2019-05-09 18:30 ` Halil Pasic 2019-05-09 18:30 ` Halil Pasic 2019-05-13 13:54 ` Cornelia Huck 2019-05-13 13:54 ` Cornelia Huck 2019-04-26 18:32 ` [PATCH 10/10] virtio/s390: make airq summary indicators DMA Halil Pasic 2019-04-26 18:32 ` Halil Pasic 2019-05-08 15:11 ` Pierre Morel 2019-05-08 15:11 ` Pierre Morel 2019-05-15 13:33 ` Michael Mueller 2019-05-15 13:33 ` Michael Mueller 2019-05-15 17:23 ` Halil Pasic 2019-05-15 17:23 ` Halil Pasic 2019-05-13 12:20 ` Cornelia Huck 2019-05-13 12:20 ` Cornelia Huck 2019-05-15 13:43 ` Michael Mueller 2019-05-15 13:43 ` Michael Mueller 2019-05-15 13:50 ` Cornelia Huck 2019-05-15 13:50 ` Cornelia Huck 2019-05-15 17:18 ` Halil Pasic 2019-05-15 17:18 ` Halil Pasic 2019-05-03 9:55 ` [PATCH 00/10] s390: virtio: support protected virtualization Cornelia Huck 2019-05-03 10:03 ` Juergen Gross 2019-05-03 13:33 ` Cornelia Huck 2019-05-03 13:33 ` Cornelia Huck 2019-05-04 13:58 ` Halil Pasic 2019-05-04 13:58 ` Halil Pasic
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20190523001246.449154aa.pasic@linux.ibm.com \ --to=pasic@linux.ibm.com \ --cc=alifm@linux.ibm.com \ --cc=borntraeger@de.ibm.com \ --cc=cohuck@redhat.com \ --cc=farman@linux.ibm.com \ --cc=frankja@linux.ibm.com \ --cc=gor@linux.ibm.com \ --cc=hch@infradead.org \ --cc=imbrenda@linux.ibm.com \ --cc=kvm@vger.kernel.org \ --cc=linux-s390@vger.kernel.org \ --cc=mihajlov@linux.ibm.com \ --cc=mimu@linux.ibm.com \ --cc=mst@redhat.com \ --cc=schwidefsky@de.ibm.com \ --cc=sebott@linux.ibm.com \ --cc=thuth@redhat.com \ --cc=virtualization@lists.linux-foundation.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.