From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753428AbbFCIlj (ORCPT ); Wed, 3 Jun 2015 04:41:39 -0400 Received: from pandora.arm.linux.org.uk ([78.32.30.218]:34141 "EHLO pandora.arm.linux.org.uk" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752511AbbFCIlb (ORCPT ); Wed, 3 Jun 2015 04:41:31 -0400 Date: Wed, 3 Jun 2015 09:41:15 +0100 From: Russell King - ARM Linux To: Hans Verkuil Cc: Sumit Semwal , Linaro Kernel Mailman List , Tomasz Stanislawski , LKML , DRI mailing list , Linaro MM SIG Mailman List , "linux-mm@kvack.org" , Rob Clark , Daniel Vetter , Robin Murphy , "linux-arm-kernel@lists.infradead.org" , "linux-media@vger.kernel.org" Subject: Re: [Linaro-mm-sig] [RFCv3 2/2] dma-buf: add helpers for sharing attacher constraints with dma-parms Message-ID: <20150603084115.GC7557@n2100.arm.linux.org.uk> References: <1422347154-15258-1-git-send-email-sumit.semwal@linaro.org> <1422347154-15258-2-git-send-email-sumit.semwal@linaro.org> <54DB12B5.4080000@samsung.com> <20150211111258.GP8656@n2100.arm.linux.org.uk> <54DB4908.10004@samsung.com> <20150211162312.GR8656@n2100.arm.linux.org.uk> <556EA13B.7080306@xs4all.nl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <556EA13B.7080306@xs4all.nl> User-Agent: Mutt/1.5.23 (2014-03-12) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Jun 03, 2015 at 08:39:55AM +0200, Hans Verkuil wrote: > Hi Sumit, > > On 05/05/2015 04:41 PM, Sumit Semwal wrote: > > Hi Russell, everyone, > > > > First up, sincere apologies for being awol for sometime; had some > > personal / medical things to take care of, and then I thought I'd wait > > for the merge window to get over before beginning to discuss this > > again. > > > > On 11 February 2015 at 21:53, Russell King - ARM Linux > > wrote: > >> On Wed, Feb 11, 2015 at 01:20:24PM +0100, Marek Szyprowski wrote: > >>> Hello, > >>> > >>> On 2015-02-11 12:12, Russell King - ARM Linux wrote: > >>>> Which is a damn good reason to NAK it - by that admission, it's a half-baked > >>>> idea. > >>>> > >>>> If all we want to know is whether the importer can accept only contiguous > >>>> memory or not, make a flag to do that, and allow the exporter to test this > >>>> flag. Don't over-engineer this to make it _seem_ like it can do something > >>>> that it actually totally fails with. > >>>> > >>>> As I've already pointed out, there's a major problem if you have already > >>>> had a less restrictive attachment which has an active mapping, and a new > >>>> more restrictive attachment comes along later. > >>>> > >>>> It seems from Rob's descriptions that we also need another flag in the > >>>> importer to indicate whether it wants to have a valid struct page in the > >>>> scatter list, or whether it (correctly) uses the DMA accessors on the > >>>> scatter list - so that exporters can reject importers which are buggy. > >>> > >>> Okay, but flag-based approach also have limitations. > >> > >> Yes, the flag-based approach doesn't let you describe in detail what > >> the importer can accept - which, given the issues that I've raised > >> is a *good* thing. We won't be misleading anyone into thinking that > >> we can do something that's really half-baked, and which we have no > >> present requirement for. > >> > >> This is precisely what Linus talks about when he says "don't over- > >> engineer" - if we over-engineer this, we end up with something that > >> sort-of works, and that's a bad thing. > >> > >> The Keep It Simple approach here makes total sense - what are our > >> current requirements - to be able to say that an importer can only accept: > >> - contiguous memory rather than a scatterlist > >> - scatterlists with struct page pointers > >> > >> Does solving that need us to compare all the constraints of each and > >> every importer, possibly ending up with constraints which can't be > >> satisfied? No. Does the flag approach satisfy the requirements? Yes. > >> > > > > So, for basic constraint-sharing, we'll just go with the flag based > > approach, with a flag (best place for it is still dev->dma_params I > > suppose) for denoting contiguous or scatterlist. Is that agreed, then? > > Also, with this idea, of course, there won't be any helpers for trying > > to calculate constraints; it would be totally the exporter's > > responsibility to handle it via the attach() dma_buf_op if it wishes > > to. > > What's wrong with the proposed max_segment_count? Many media devices do > have a limited max_segment_count and that should be taken into account. So what happens if you have a dma_buf exporter, and several dma_buf importers. One dma_buf importer attaches to the exporter, and asks for the buffer, and starts making use of the buffer. This export has many scatterlist segments. Another dma_buf importer attaches to the same buffer, and now asks for the buffer, but the number of scatterlist segments exceeds it's requirement. You can't reallocate the buffer because it's in-use by another importer. There is no way to revoke the buffer from the other importer. So there is no way to satisfy this importer's requirements. What I'm showing is that the idea that exporting these parameters fixes some problem is just an illusion - it may work for the single importer case, but doesn't for the multiple importer case. Importers really have two choices here: either they accept what the exporter is giving them, or they reject it. The other issue here is that DMA scatterlists are _not_ really that determinable in terms of number of entries when it comes to systems with system IOMMUs. System IOMMUs, which should be integrated into the DMA API, are permitted to coalesce entries in the physical page range. For example: nsg = 128; n = dma_map_sg(dev, sg, nsg, DMA_TO_DEVICE); Here, n might be 4 if the system IOMMU has been able to coalesce the 128 entries down to 4 IOMMU entries - and that means for DMA purposes, only the first four scatterlist entries should be walked (this is why dma_map_sg() returns a positive integer when mapping.) Each struct device has a set of parameters which control how the IOMMU entries are coalesced: struct device_dma_parameters { /* * a low level driver may set these to teach IOMMU code about * sg limitations. */ unsigned int max_segment_size; unsigned long segment_boundary_mask; }; and this is independent of the dma_buf API. This doesn't indicate the maximum number of segments, but as I've shown above, it's not something that you can say "I want a scatterlist for this memory with only 32 segments" so it's totally unclear how an exporter would limit that. The only thing an exporter could do would be to fail the export if the buffer didn't end up having fewer than the requested scatterlist entries, which is something the importer can do too. > One of the main problems end-users are faced with today is that they do not > know which device should be the exporter of buffers and which should be the > importer. This depends on the constraints and right now applications have > no way of knowing this. It's nuts that this hasn't been addressed yet since > it is the main complaint I am getting. IT's nuts that we've ended up in this situation in the first place. This was bound to happen as soon as the dma_buf sharing was introduced, because it immediately introduced this problem. I don't think there is any easy solution to it, and what's being proposed with flags and other stuff is just trying to paper over the problem. What you're actually asking is that each dmabuf exporting subsystem needs to publish their DMA parameters to userspace, and userspace then gets to decide which dmabuf exporter should be used. That's not a dmabuf problem, that's a subsystem problem, but even so, we don't have a standardised way to export that information (and I'd suspect that it would be very difficult to get agreements between subsystems on a standard ioctl and/or data structure.) In my experience, getting cross- subsystem agreement in the kernel with anything is very difficult, you normally end up with 60% of people agreeing, and the other 40% going off and doing something completely different because they object to it (figures vary, 90% of all statistics are made up on the spot!) -- FTTC broadband for 0.8mile line: currently at 10.5Mbps down 400kbps up according to speedtest.net. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-path: Received: from pandora.arm.linux.org.uk ([78.32.30.218]:34141 "EHLO pandora.arm.linux.org.uk" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752511AbbFCIlb (ORCPT ); Wed, 3 Jun 2015 04:41:31 -0400 Date: Wed, 3 Jun 2015 09:41:15 +0100 From: Russell King - ARM Linux To: Hans Verkuil Cc: Sumit Semwal , Linaro Kernel Mailman List , Tomasz Stanislawski , LKML , DRI mailing list , Linaro MM SIG Mailman List , "linux-mm@kvack.org" , Rob Clark , Daniel Vetter , Robin Murphy , "linux-arm-kernel@lists.infradead.org" , "linux-media@vger.kernel.org" Subject: Re: [Linaro-mm-sig] [RFCv3 2/2] dma-buf: add helpers for sharing attacher constraints with dma-parms Message-ID: <20150603084115.GC7557@n2100.arm.linux.org.uk> References: <1422347154-15258-1-git-send-email-sumit.semwal@linaro.org> <1422347154-15258-2-git-send-email-sumit.semwal@linaro.org> <54DB12B5.4080000@samsung.com> <20150211111258.GP8656@n2100.arm.linux.org.uk> <54DB4908.10004@samsung.com> <20150211162312.GR8656@n2100.arm.linux.org.uk> <556EA13B.7080306@xs4all.nl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <556EA13B.7080306@xs4all.nl> Sender: linux-media-owner@vger.kernel.org List-ID: On Wed, Jun 03, 2015 at 08:39:55AM +0200, Hans Verkuil wrote: > Hi Sumit, > > On 05/05/2015 04:41 PM, Sumit Semwal wrote: > > Hi Russell, everyone, > > > > First up, sincere apologies for being awol for sometime; had some > > personal / medical things to take care of, and then I thought I'd wait > > for the merge window to get over before beginning to discuss this > > again. > > > > On 11 February 2015 at 21:53, Russell King - ARM Linux > > wrote: > >> On Wed, Feb 11, 2015 at 01:20:24PM +0100, Marek Szyprowski wrote: > >>> Hello, > >>> > >>> On 2015-02-11 12:12, Russell King - ARM Linux wrote: > >>>> Which is a damn good reason to NAK it - by that admission, it's a half-baked > >>>> idea. > >>>> > >>>> If all we want to know is whether the importer can accept only contiguous > >>>> memory or not, make a flag to do that, and allow the exporter to test this > >>>> flag. Don't over-engineer this to make it _seem_ like it can do something > >>>> that it actually totally fails with. > >>>> > >>>> As I've already pointed out, there's a major problem if you have already > >>>> had a less restrictive attachment which has an active mapping, and a new > >>>> more restrictive attachment comes along later. > >>>> > >>>> It seems from Rob's descriptions that we also need another flag in the > >>>> importer to indicate whether it wants to have a valid struct page in the > >>>> scatter list, or whether it (correctly) uses the DMA accessors on the > >>>> scatter list - so that exporters can reject importers which are buggy. > >>> > >>> Okay, but flag-based approach also have limitations. > >> > >> Yes, the flag-based approach doesn't let you describe in detail what > >> the importer can accept - which, given the issues that I've raised > >> is a *good* thing. We won't be misleading anyone into thinking that > >> we can do something that's really half-baked, and which we have no > >> present requirement for. > >> > >> This is precisely what Linus talks about when he says "don't over- > >> engineer" - if we over-engineer this, we end up with something that > >> sort-of works, and that's a bad thing. > >> > >> The Keep It Simple approach here makes total sense - what are our > >> current requirements - to be able to say that an importer can only accept: > >> - contiguous memory rather than a scatterlist > >> - scatterlists with struct page pointers > >> > >> Does solving that need us to compare all the constraints of each and > >> every importer, possibly ending up with constraints which can't be > >> satisfied? No. Does the flag approach satisfy the requirements? Yes. > >> > > > > So, for basic constraint-sharing, we'll just go with the flag based > > approach, with a flag (best place for it is still dev->dma_params I > > suppose) for denoting contiguous or scatterlist. Is that agreed, then? > > Also, with this idea, of course, there won't be any helpers for trying > > to calculate constraints; it would be totally the exporter's > > responsibility to handle it via the attach() dma_buf_op if it wishes > > to. > > What's wrong with the proposed max_segment_count? Many media devices do > have a limited max_segment_count and that should be taken into account. So what happens if you have a dma_buf exporter, and several dma_buf importers. One dma_buf importer attaches to the exporter, and asks for the buffer, and starts making use of the buffer. This export has many scatterlist segments. Another dma_buf importer attaches to the same buffer, and now asks for the buffer, but the number of scatterlist segments exceeds it's requirement. You can't reallocate the buffer because it's in-use by another importer. There is no way to revoke the buffer from the other importer. So there is no way to satisfy this importer's requirements. What I'm showing is that the idea that exporting these parameters fixes some problem is just an illusion - it may work for the single importer case, but doesn't for the multiple importer case. Importers really have two choices here: either they accept what the exporter is giving them, or they reject it. The other issue here is that DMA scatterlists are _not_ really that determinable in terms of number of entries when it comes to systems with system IOMMUs. System IOMMUs, which should be integrated into the DMA API, are permitted to coalesce entries in the physical page range. For example: nsg = 128; n = dma_map_sg(dev, sg, nsg, DMA_TO_DEVICE); Here, n might be 4 if the system IOMMU has been able to coalesce the 128 entries down to 4 IOMMU entries - and that means for DMA purposes, only the first four scatterlist entries should be walked (this is why dma_map_sg() returns a positive integer when mapping.) Each struct device has a set of parameters which control how the IOMMU entries are coalesced: struct device_dma_parameters { /* * a low level driver may set these to teach IOMMU code about * sg limitations. */ unsigned int max_segment_size; unsigned long segment_boundary_mask; }; and this is independent of the dma_buf API. This doesn't indicate the maximum number of segments, but as I've shown above, it's not something that you can say "I want a scatterlist for this memory with only 32 segments" so it's totally unclear how an exporter would limit that. The only thing an exporter could do would be to fail the export if the buffer didn't end up having fewer than the requested scatterlist entries, which is something the importer can do too. > One of the main problems end-users are faced with today is that they do not > know which device should be the exporter of buffers and which should be the > importer. This depends on the constraints and right now applications have > no way of knowing this. It's nuts that this hasn't been addressed yet since > it is the main complaint I am getting. IT's nuts that we've ended up in this situation in the first place. This was bound to happen as soon as the dma_buf sharing was introduced, because it immediately introduced this problem. I don't think there is any easy solution to it, and what's being proposed with flags and other stuff is just trying to paper over the problem. What you're actually asking is that each dmabuf exporting subsystem needs to publish their DMA parameters to userspace, and userspace then gets to decide which dmabuf exporter should be used. That's not a dmabuf problem, that's a subsystem problem, but even so, we don't have a standardised way to export that information (and I'd suspect that it would be very difficult to get agreements between subsystems on a standard ioctl and/or data structure.) In my experience, getting cross- subsystem agreement in the kernel with anything is very difficult, you normally end up with 60% of people agreeing, and the other 40% going off and doing something completely different because they object to it (figures vary, 90% of all statistics are made up on the spot!) -- FTTC broadband for 0.8mile line: currently at 10.5Mbps down 400kbps up according to speedtest.net. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wi0-f177.google.com (mail-wi0-f177.google.com [209.85.212.177]) by kanga.kvack.org (Postfix) with ESMTP id 62187900016 for ; Wed, 3 Jun 2015 04:41:37 -0400 (EDT) Received: by wibut5 with SMTP id ut5so94462027wib.1 for ; Wed, 03 Jun 2015 01:41:36 -0700 (PDT) Received: from pandora.arm.linux.org.uk (pandora.arm.linux.org.uk. [2001:4d48:ad52:3201:214:fdff:fe10:1be6]) by mx.google.com with ESMTPS id gf8si29448367wib.17.2015.06.03.01.41.34 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Wed, 03 Jun 2015 01:41:35 -0700 (PDT) Date: Wed, 3 Jun 2015 09:41:15 +0100 From: Russell King - ARM Linux Subject: Re: [Linaro-mm-sig] [RFCv3 2/2] dma-buf: add helpers for sharing attacher constraints with dma-parms Message-ID: <20150603084115.GC7557@n2100.arm.linux.org.uk> References: <1422347154-15258-1-git-send-email-sumit.semwal@linaro.org> <1422347154-15258-2-git-send-email-sumit.semwal@linaro.org> <54DB12B5.4080000@samsung.com> <20150211111258.GP8656@n2100.arm.linux.org.uk> <54DB4908.10004@samsung.com> <20150211162312.GR8656@n2100.arm.linux.org.uk> <556EA13B.7080306@xs4all.nl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <556EA13B.7080306@xs4all.nl> Sender: owner-linux-mm@kvack.org List-ID: To: Hans Verkuil Cc: Sumit Semwal , Linaro Kernel Mailman List , Tomasz Stanislawski , LKML , DRI mailing list , Linaro MM SIG Mailman List , "linux-mm@kvack.org" , Rob Clark , Daniel Vetter , Robin Murphy , "linux-arm-kernel@lists.infradead.org" , "linux-media@vger.kernel.org" On Wed, Jun 03, 2015 at 08:39:55AM +0200, Hans Verkuil wrote: > Hi Sumit, > > On 05/05/2015 04:41 PM, Sumit Semwal wrote: > > Hi Russell, everyone, > > > > First up, sincere apologies for being awol for sometime; had some > > personal / medical things to take care of, and then I thought I'd wait > > for the merge window to get over before beginning to discuss this > > again. > > > > On 11 February 2015 at 21:53, Russell King - ARM Linux > > wrote: > >> On Wed, Feb 11, 2015 at 01:20:24PM +0100, Marek Szyprowski wrote: > >>> Hello, > >>> > >>> On 2015-02-11 12:12, Russell King - ARM Linux wrote: > >>>> Which is a damn good reason to NAK it - by that admission, it's a half-baked > >>>> idea. > >>>> > >>>> If all we want to know is whether the importer can accept only contiguous > >>>> memory or not, make a flag to do that, and allow the exporter to test this > >>>> flag. Don't over-engineer this to make it _seem_ like it can do something > >>>> that it actually totally fails with. > >>>> > >>>> As I've already pointed out, there's a major problem if you have already > >>>> had a less restrictive attachment which has an active mapping, and a new > >>>> more restrictive attachment comes along later. > >>>> > >>>> It seems from Rob's descriptions that we also need another flag in the > >>>> importer to indicate whether it wants to have a valid struct page in the > >>>> scatter list, or whether it (correctly) uses the DMA accessors on the > >>>> scatter list - so that exporters can reject importers which are buggy. > >>> > >>> Okay, but flag-based approach also have limitations. > >> > >> Yes, the flag-based approach doesn't let you describe in detail what > >> the importer can accept - which, given the issues that I've raised > >> is a *good* thing. We won't be misleading anyone into thinking that > >> we can do something that's really half-baked, and which we have no > >> present requirement for. > >> > >> This is precisely what Linus talks about when he says "don't over- > >> engineer" - if we over-engineer this, we end up with something that > >> sort-of works, and that's a bad thing. > >> > >> The Keep It Simple approach here makes total sense - what are our > >> current requirements - to be able to say that an importer can only accept: > >> - contiguous memory rather than a scatterlist > >> - scatterlists with struct page pointers > >> > >> Does solving that need us to compare all the constraints of each and > >> every importer, possibly ending up with constraints which can't be > >> satisfied? No. Does the flag approach satisfy the requirements? Yes. > >> > > > > So, for basic constraint-sharing, we'll just go with the flag based > > approach, with a flag (best place for it is still dev->dma_params I > > suppose) for denoting contiguous or scatterlist. Is that agreed, then? > > Also, with this idea, of course, there won't be any helpers for trying > > to calculate constraints; it would be totally the exporter's > > responsibility to handle it via the attach() dma_buf_op if it wishes > > to. > > What's wrong with the proposed max_segment_count? Many media devices do > have a limited max_segment_count and that should be taken into account. So what happens if you have a dma_buf exporter, and several dma_buf importers. One dma_buf importer attaches to the exporter, and asks for the buffer, and starts making use of the buffer. This export has many scatterlist segments. Another dma_buf importer attaches to the same buffer, and now asks for the buffer, but the number of scatterlist segments exceeds it's requirement. You can't reallocate the buffer because it's in-use by another importer. There is no way to revoke the buffer from the other importer. So there is no way to satisfy this importer's requirements. What I'm showing is that the idea that exporting these parameters fixes some problem is just an illusion - it may work for the single importer case, but doesn't for the multiple importer case. Importers really have two choices here: either they accept what the exporter is giving them, or they reject it. The other issue here is that DMA scatterlists are _not_ really that determinable in terms of number of entries when it comes to systems with system IOMMUs. System IOMMUs, which should be integrated into the DMA API, are permitted to coalesce entries in the physical page range. For example: nsg = 128; n = dma_map_sg(dev, sg, nsg, DMA_TO_DEVICE); Here, n might be 4 if the system IOMMU has been able to coalesce the 128 entries down to 4 IOMMU entries - and that means for DMA purposes, only the first four scatterlist entries should be walked (this is why dma_map_sg() returns a positive integer when mapping.) Each struct device has a set of parameters which control how the IOMMU entries are coalesced: struct device_dma_parameters { /* * a low level driver may set these to teach IOMMU code about * sg limitations. */ unsigned int max_segment_size; unsigned long segment_boundary_mask; }; and this is independent of the dma_buf API. This doesn't indicate the maximum number of segments, but as I've shown above, it's not something that you can say "I want a scatterlist for this memory with only 32 segments" so it's totally unclear how an exporter would limit that. The only thing an exporter could do would be to fail the export if the buffer didn't end up having fewer than the requested scatterlist entries, which is something the importer can do too. > One of the main problems end-users are faced with today is that they do not > know which device should be the exporter of buffers and which should be the > importer. This depends on the constraints and right now applications have > no way of knowing this. It's nuts that this hasn't been addressed yet since > it is the main complaint I am getting. IT's nuts that we've ended up in this situation in the first place. This was bound to happen as soon as the dma_buf sharing was introduced, because it immediately introduced this problem. I don't think there is any easy solution to it, and what's being proposed with flags and other stuff is just trying to paper over the problem. What you're actually asking is that each dmabuf exporting subsystem needs to publish their DMA parameters to userspace, and userspace then gets to decide which dmabuf exporter should be used. That's not a dmabuf problem, that's a subsystem problem, but even so, we don't have a standardised way to export that information (and I'd suspect that it would be very difficult to get agreements between subsystems on a standard ioctl and/or data structure.) In my experience, getting cross- subsystem agreement in the kernel with anything is very difficult, you normally end up with 60% of people agreeing, and the other 40% going off and doing something completely different because they object to it (figures vary, 90% of all statistics are made up on the spot!) -- FTTC broadband for 0.8mile line: currently at 10.5Mbps down 400kbps up according to speedtest.net. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 From: linux@arm.linux.org.uk (Russell King - ARM Linux) Date: Wed, 3 Jun 2015 09:41:15 +0100 Subject: [Linaro-mm-sig] [RFCv3 2/2] dma-buf: add helpers for sharing attacher constraints with dma-parms In-Reply-To: <556EA13B.7080306@xs4all.nl> References: <1422347154-15258-1-git-send-email-sumit.semwal@linaro.org> <1422347154-15258-2-git-send-email-sumit.semwal@linaro.org> <54DB12B5.4080000@samsung.com> <20150211111258.GP8656@n2100.arm.linux.org.uk> <54DB4908.10004@samsung.com> <20150211162312.GR8656@n2100.arm.linux.org.uk> <556EA13B.7080306@xs4all.nl> Message-ID: <20150603084115.GC7557@n2100.arm.linux.org.uk> To: linux-arm-kernel@lists.infradead.org List-Id: linux-arm-kernel.lists.infradead.org On Wed, Jun 03, 2015 at 08:39:55AM +0200, Hans Verkuil wrote: > Hi Sumit, > > On 05/05/2015 04:41 PM, Sumit Semwal wrote: > > Hi Russell, everyone, > > > > First up, sincere apologies for being awol for sometime; had some > > personal / medical things to take care of, and then I thought I'd wait > > for the merge window to get over before beginning to discuss this > > again. > > > > On 11 February 2015 at 21:53, Russell King - ARM Linux > > wrote: > >> On Wed, Feb 11, 2015 at 01:20:24PM +0100, Marek Szyprowski wrote: > >>> Hello, > >>> > >>> On 2015-02-11 12:12, Russell King - ARM Linux wrote: > >>>> Which is a damn good reason to NAK it - by that admission, it's a half-baked > >>>> idea. > >>>> > >>>> If all we want to know is whether the importer can accept only contiguous > >>>> memory or not, make a flag to do that, and allow the exporter to test this > >>>> flag. Don't over-engineer this to make it _seem_ like it can do something > >>>> that it actually totally fails with. > >>>> > >>>> As I've already pointed out, there's a major problem if you have already > >>>> had a less restrictive attachment which has an active mapping, and a new > >>>> more restrictive attachment comes along later. > >>>> > >>>> It seems from Rob's descriptions that we also need another flag in the > >>>> importer to indicate whether it wants to have a valid struct page in the > >>>> scatter list, or whether it (correctly) uses the DMA accessors on the > >>>> scatter list - so that exporters can reject importers which are buggy. > >>> > >>> Okay, but flag-based approach also have limitations. > >> > >> Yes, the flag-based approach doesn't let you describe in detail what > >> the importer can accept - which, given the issues that I've raised > >> is a *good* thing. We won't be misleading anyone into thinking that > >> we can do something that's really half-baked, and which we have no > >> present requirement for. > >> > >> This is precisely what Linus talks about when he says "don't over- > >> engineer" - if we over-engineer this, we end up with something that > >> sort-of works, and that's a bad thing. > >> > >> The Keep It Simple approach here makes total sense - what are our > >> current requirements - to be able to say that an importer can only accept: > >> - contiguous memory rather than a scatterlist > >> - scatterlists with struct page pointers > >> > >> Does solving that need us to compare all the constraints of each and > >> every importer, possibly ending up with constraints which can't be > >> satisfied? No. Does the flag approach satisfy the requirements? Yes. > >> > > > > So, for basic constraint-sharing, we'll just go with the flag based > > approach, with a flag (best place for it is still dev->dma_params I > > suppose) for denoting contiguous or scatterlist. Is that agreed, then? > > Also, with this idea, of course, there won't be any helpers for trying > > to calculate constraints; it would be totally the exporter's > > responsibility to handle it via the attach() dma_buf_op if it wishes > > to. > > What's wrong with the proposed max_segment_count? Many media devices do > have a limited max_segment_count and that should be taken into account. So what happens if you have a dma_buf exporter, and several dma_buf importers. One dma_buf importer attaches to the exporter, and asks for the buffer, and starts making use of the buffer. This export has many scatterlist segments. Another dma_buf importer attaches to the same buffer, and now asks for the buffer, but the number of scatterlist segments exceeds it's requirement. You can't reallocate the buffer because it's in-use by another importer. There is no way to revoke the buffer from the other importer. So there is no way to satisfy this importer's requirements. What I'm showing is that the idea that exporting these parameters fixes some problem is just an illusion - it may work for the single importer case, but doesn't for the multiple importer case. Importers really have two choices here: either they accept what the exporter is giving them, or they reject it. The other issue here is that DMA scatterlists are _not_ really that determinable in terms of number of entries when it comes to systems with system IOMMUs. System IOMMUs, which should be integrated into the DMA API, are permitted to coalesce entries in the physical page range. For example: nsg = 128; n = dma_map_sg(dev, sg, nsg, DMA_TO_DEVICE); Here, n might be 4 if the system IOMMU has been able to coalesce the 128 entries down to 4 IOMMU entries - and that means for DMA purposes, only the first four scatterlist entries should be walked (this is why dma_map_sg() returns a positive integer when mapping.) Each struct device has a set of parameters which control how the IOMMU entries are coalesced: struct device_dma_parameters { /* * a low level driver may set these to teach IOMMU code about * sg limitations. */ unsigned int max_segment_size; unsigned long segment_boundary_mask; }; and this is independent of the dma_buf API. This doesn't indicate the maximum number of segments, but as I've shown above, it's not something that you can say "I want a scatterlist for this memory with only 32 segments" so it's totally unclear how an exporter would limit that. The only thing an exporter could do would be to fail the export if the buffer didn't end up having fewer than the requested scatterlist entries, which is something the importer can do too. > One of the main problems end-users are faced with today is that they do not > know which device should be the exporter of buffers and which should be the > importer. This depends on the constraints and right now applications have > no way of knowing this. It's nuts that this hasn't been addressed yet since > it is the main complaint I am getting. IT's nuts that we've ended up in this situation in the first place. This was bound to happen as soon as the dma_buf sharing was introduced, because it immediately introduced this problem. I don't think there is any easy solution to it, and what's being proposed with flags and other stuff is just trying to paper over the problem. What you're actually asking is that each dmabuf exporting subsystem needs to publish their DMA parameters to userspace, and userspace then gets to decide which dmabuf exporter should be used. That's not a dmabuf problem, that's a subsystem problem, but even so, we don't have a standardised way to export that information (and I'd suspect that it would be very difficult to get agreements between subsystems on a standard ioctl and/or data structure.) In my experience, getting cross- subsystem agreement in the kernel with anything is very difficult, you normally end up with 60% of people agreeing, and the other 40% going off and doing something completely different because they object to it (figures vary, 90% of all statistics are made up on the spot!) -- FTTC broadband for 0.8mile line: currently at 10.5Mbps down 400kbps up according to speedtest.net. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Russell King - ARM Linux Subject: Re: [Linaro-mm-sig] [RFCv3 2/2] dma-buf: add helpers for sharing attacher constraints with dma-parms Date: Wed, 3 Jun 2015 09:41:15 +0100 Message-ID: <20150603084115.GC7557@n2100.arm.linux.org.uk> References: <1422347154-15258-1-git-send-email-sumit.semwal@linaro.org> <1422347154-15258-2-git-send-email-sumit.semwal@linaro.org> <54DB12B5.4080000@samsung.com> <20150211111258.GP8656@n2100.arm.linux.org.uk> <54DB4908.10004@samsung.com> <20150211162312.GR8656@n2100.arm.linux.org.uk> <556EA13B.7080306@xs4all.nl> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Received: from pandora.arm.linux.org.uk (pandora.arm.linux.org.uk [78.32.30.218]) by gabe.freedesktop.org (Postfix) with ESMTP id ED1276E0B1 for ; Wed, 3 Jun 2015 01:41:32 -0700 (PDT) Content-Disposition: inline In-Reply-To: <556EA13B.7080306@xs4all.nl> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" To: Hans Verkuil Cc: Linaro Kernel Mailman List , Robin Murphy , LKML , DRI mailing list , Linaro MM SIG Mailman List , "linux-mm@kvack.org" , Tomasz Stanislawski , "linux-arm-kernel@lists.infradead.org" , "linux-media@vger.kernel.org" List-Id: dri-devel@lists.freedesktop.org T24gV2VkLCBKdW4gMDMsIDIwMTUgYXQgMDg6Mzk6NTVBTSArMDIwMCwgSGFucyBWZXJrdWlsIHdy b3RlOgo+IEhpIFN1bWl0LAo+IAo+IE9uIDA1LzA1LzIwMTUgMDQ6NDEgUE0sIFN1bWl0IFNlbXdh bCB3cm90ZToKPiA+IEhpIFJ1c3NlbGwsIGV2ZXJ5b25lLAo+ID4gCj4gPiBGaXJzdCB1cCwgc2lu Y2VyZSBhcG9sb2dpZXMgZm9yIGJlaW5nIGF3b2wgZm9yIHNvbWV0aW1lOyBoYWQgc29tZQo+ID4g cGVyc29uYWwgLyBtZWRpY2FsIHRoaW5ncyB0byB0YWtlIGNhcmUgb2YsIGFuZCB0aGVuIEkgdGhv dWdodCBJJ2Qgd2FpdAo+ID4gZm9yIHRoZSBtZXJnZSB3aW5kb3cgdG8gZ2V0IG92ZXIgYmVmb3Jl IGJlZ2lubmluZyB0byBkaXNjdXNzIHRoaXMKPiA+IGFnYWluLgo+ID4gCj4gPiBPbiAxMSBGZWJy dWFyeSAyMDE1IGF0IDIxOjUzLCBSdXNzZWxsIEtpbmcgLSBBUk0gTGludXgKPiA+IDxsaW51eEBh cm0ubGludXgub3JnLnVrPiB3cm90ZToKPiA+PiBPbiBXZWQsIEZlYiAxMSwgMjAxNSBhdCAwMToy MDoyNFBNICswMTAwLCBNYXJlayBTenlwcm93c2tpIHdyb3RlOgo+ID4+PiBIZWxsbywKPiA+Pj4K PiA+Pj4gT24gMjAxNS0wMi0xMSAxMjoxMiwgUnVzc2VsbCBLaW5nIC0gQVJNIExpbnV4IHdyb3Rl Ogo+ID4+Pj4gV2hpY2ggaXMgYSBkYW1uIGdvb2QgcmVhc29uIHRvIE5BSyBpdCAtIGJ5IHRoYXQg YWRtaXNzaW9uLCBpdCdzIGEgaGFsZi1iYWtlZAo+ID4+Pj4gaWRlYS4KPiA+Pj4+Cj4gPj4+PiBJ ZiBhbGwgd2Ugd2FudCB0byBrbm93IGlzIHdoZXRoZXIgdGhlIGltcG9ydGVyIGNhbiBhY2NlcHQg b25seSBjb250aWd1b3VzCj4gPj4+PiBtZW1vcnkgb3Igbm90LCBtYWtlIGEgZmxhZyB0byBkbyB0 aGF0LCBhbmQgYWxsb3cgdGhlIGV4cG9ydGVyIHRvIHRlc3QgdGhpcwo+ID4+Pj4gZmxhZy4gIERv bid0IG92ZXItZW5naW5lZXIgdGhpcyB0byBtYWtlIGl0IF9zZWVtXyBsaWtlIGl0IGNhbiBkbyBz b21ldGhpbmcKPiA+Pj4+IHRoYXQgaXQgYWN0dWFsbHkgdG90YWxseSBmYWlscyB3aXRoLgo+ID4+ Pj4KPiA+Pj4+IEFzIEkndmUgYWxyZWFkeSBwb2ludGVkIG91dCwgdGhlcmUncyBhIG1ham9yIHBy b2JsZW0gaWYgeW91IGhhdmUgYWxyZWFkeQo+ID4+Pj4gaGFkIGEgbGVzcyByZXN0cmljdGl2ZSBh dHRhY2htZW50IHdoaWNoIGhhcyBhbiBhY3RpdmUgbWFwcGluZywgYW5kIGEgbmV3Cj4gPj4+PiBt b3JlIHJlc3RyaWN0aXZlIGF0dGFjaG1lbnQgY29tZXMgYWxvbmcgbGF0ZXIuCj4gPj4+Pgo+ID4+ Pj4gSXQgc2VlbXMgZnJvbSBSb2IncyBkZXNjcmlwdGlvbnMgdGhhdCB3ZSBhbHNvIG5lZWQgYW5v dGhlciBmbGFnIGluIHRoZQo+ID4+Pj4gaW1wb3J0ZXIgdG8gaW5kaWNhdGUgd2hldGhlciBpdCB3 YW50cyB0byBoYXZlIGEgdmFsaWQgc3RydWN0IHBhZ2UgaW4gdGhlCj4gPj4+PiBzY2F0dGVyIGxp c3QsIG9yIHdoZXRoZXIgaXQgKGNvcnJlY3RseSkgdXNlcyB0aGUgRE1BIGFjY2Vzc29ycyBvbiB0 aGUKPiA+Pj4+IHNjYXR0ZXIgbGlzdCAtIHNvIHRoYXQgZXhwb3J0ZXJzIGNhbiByZWplY3QgaW1w b3J0ZXJzIHdoaWNoIGFyZSBidWdneS4KPiA+Pj4KPiA+Pj4gT2theSwgYnV0IGZsYWctYmFzZWQg YXBwcm9hY2ggYWxzbyBoYXZlIGxpbWl0YXRpb25zLgo+ID4+Cj4gPj4gWWVzLCB0aGUgZmxhZy1i YXNlZCBhcHByb2FjaCBkb2Vzbid0IGxldCB5b3UgZGVzY3JpYmUgaW4gZGV0YWlsIHdoYXQKPiA+ PiB0aGUgaW1wb3J0ZXIgY2FuIGFjY2VwdCAtIHdoaWNoLCBnaXZlbiB0aGUgaXNzdWVzIHRoYXQg SSd2ZSByYWlzZWQKPiA+PiBpcyBhICpnb29kKiB0aGluZy4gIFdlIHdvbid0IGJlIG1pc2xlYWRp bmcgYW55b25lIGludG8gdGhpbmtpbmcgdGhhdAo+ID4+IHdlIGNhbiBkbyBzb21ldGhpbmcgdGhh dCdzIHJlYWxseSBoYWxmLWJha2VkLCBhbmQgd2hpY2ggd2UgaGF2ZSBubwo+ID4+IHByZXNlbnQg cmVxdWlyZW1lbnQgZm9yLgo+ID4+Cj4gPj4gVGhpcyBpcyBwcmVjaXNlbHkgd2hhdCBMaW51cyB0 YWxrcyBhYm91dCB3aGVuIGhlIHNheXMgImRvbid0IG92ZXItCj4gPj4gZW5naW5lZXIiIC0gaWYg d2Ugb3Zlci1lbmdpbmVlciB0aGlzLCB3ZSBlbmQgdXAgd2l0aCBzb21ldGhpbmcgdGhhdAo+ID4+ IHNvcnQtb2Ygd29ya3MsIGFuZCB0aGF0J3MgYSBiYWQgdGhpbmcuCj4gPj4KPiA+PiBUaGUgS2Vl cCBJdCBTaW1wbGUgYXBwcm9hY2ggaGVyZSBtYWtlcyB0b3RhbCBzZW5zZSAtIHdoYXQgYXJlIG91 cgo+ID4+IGN1cnJlbnQgcmVxdWlyZW1lbnRzIC0gdG8gYmUgYWJsZSB0byBzYXkgdGhhdCBhbiBp bXBvcnRlciBjYW4gb25seSBhY2NlcHQ6Cj4gPj4gICAtIGNvbnRpZ3VvdXMgbWVtb3J5IHJhdGhl ciB0aGFuIGEgc2NhdHRlcmxpc3QKPiA+PiAgIC0gc2NhdHRlcmxpc3RzIHdpdGggc3RydWN0IHBh Z2UgcG9pbnRlcnMKPiA+Pgo+ID4+IERvZXMgc29sdmluZyB0aGF0IG5lZWQgdXMgdG8gY29tcGFy ZSBhbGwgdGhlIGNvbnN0cmFpbnRzIG9mIGVhY2ggYW5kCj4gPj4gZXZlcnkgaW1wb3J0ZXIsIHBv c3NpYmx5IGVuZGluZyB1cCB3aXRoIGNvbnN0cmFpbnRzIHdoaWNoIGNhbid0IGJlCj4gPj4gc2F0 aXNmaWVkPyAgTm8uICBEb2VzIHRoZSBmbGFnIGFwcHJvYWNoIHNhdGlzZnkgdGhlIHJlcXVpcmVt ZW50cz8gIFllcy4KPiA+Pgo+ID4gCj4gPiBTbywgZm9yIGJhc2ljIGNvbnN0cmFpbnQtc2hhcmlu Zywgd2UnbGwganVzdCBnbyB3aXRoIHRoZSBmbGFnIGJhc2VkCj4gPiBhcHByb2FjaCwgd2l0aCBh IGZsYWcgKGJlc3QgcGxhY2UgZm9yIGl0IGlzIHN0aWxsIGRldi0+ZG1hX3BhcmFtcyBJCj4gPiBz dXBwb3NlKSBmb3IgZGVub3RpbmcgY29udGlndW91cyBvciBzY2F0dGVybGlzdC4gSXMgdGhhdCBh Z3JlZWQsIHRoZW4/Cj4gPiBBbHNvLCB3aXRoIHRoaXMgaWRlYSwgb2YgY291cnNlLCB0aGVyZSB3 b24ndCBiZSBhbnkgaGVscGVycyBmb3IgdHJ5aW5nCj4gPiB0byBjYWxjdWxhdGUgY29uc3RyYWlu dHM7IGl0IHdvdWxkIGJlIHRvdGFsbHkgdGhlIGV4cG9ydGVyJ3MKPiA+IHJlc3BvbnNpYmlsaXR5 IHRvIGhhbmRsZSBpdCB2aWEgdGhlIGF0dGFjaCgpIGRtYV9idWZfb3AgaWYgaXQgd2lzaGVzCj4g PiB0by4KPiAKPiBXaGF0J3Mgd3Jvbmcgd2l0aCB0aGUgcHJvcG9zZWQgbWF4X3NlZ21lbnRfY291 bnQ/IE1hbnkgbWVkaWEgZGV2aWNlcyBkbwo+IGhhdmUgYSBsaW1pdGVkIG1heF9zZWdtZW50X2Nv dW50IGFuZCB0aGF0IHNob3VsZCBiZSB0YWtlbiBpbnRvIGFjY291bnQuCgpTbyB3aGF0IGhhcHBl bnMgaWYgeW91IGhhdmUgYSBkbWFfYnVmIGV4cG9ydGVyLCBhbmQgc2V2ZXJhbCBkbWFfYnVmCmlt cG9ydGVycy4gIE9uZSBkbWFfYnVmIGltcG9ydGVyIGF0dGFjaGVzIHRvIHRoZSBleHBvcnRlciwg YW5kIGFza3MKZm9yIHRoZSBidWZmZXIsIGFuZCBzdGFydHMgbWFraW5nIHVzZSBvZiB0aGUgYnVm ZmVyLiAgVGhpcyBleHBvcnQgaGFzCm1hbnkgc2NhdHRlcmxpc3Qgc2VnbWVudHMuCgpBbm90aGVy IGRtYV9idWYgaW1wb3J0ZXIgYXR0YWNoZXMgdG8gdGhlIHNhbWUgYnVmZmVyLCBhbmQgbm93IGFz a3MgZm9yCnRoZSBidWZmZXIsIGJ1dCB0aGUgbnVtYmVyIG9mIHNjYXR0ZXJsaXN0IHNlZ21lbnRz IGV4Y2VlZHMgaXQncwpyZXF1aXJlbWVudC4KCllvdSBjYW4ndCByZWFsbG9jYXRlIHRoZSBidWZm ZXIgYmVjYXVzZSBpdCdzIGluLXVzZSBieSBhbm90aGVyIGltcG9ydGVyLgpUaGVyZSBpcyBubyB3 YXkgdG8gcmV2b2tlIHRoZSBidWZmZXIgZnJvbSB0aGUgb3RoZXIgaW1wb3J0ZXIuICBTbyB0aGVy ZQppcyBubyB3YXkgdG8gc2F0aXNmeSB0aGlzIGltcG9ydGVyJ3MgcmVxdWlyZW1lbnRzLgoKV2hh dCBJJ20gc2hvd2luZyBpcyB0aGF0IHRoZSBpZGVhIHRoYXQgZXhwb3J0aW5nIHRoZXNlIHBhcmFt ZXRlcnMgZml4ZXMKc29tZSBwcm9ibGVtIGlzIGp1c3QgYW4gaWxsdXNpb24gLSBpdCBtYXkgd29y ayBmb3IgdGhlIHNpbmdsZSBpbXBvcnRlcgpjYXNlLCBidXQgZG9lc24ndCBmb3IgdGhlIG11bHRp cGxlIGltcG9ydGVyIGNhc2UuCgpJbXBvcnRlcnMgcmVhbGx5IGhhdmUgdHdvIGNob2ljZXMgaGVy ZTogZWl0aGVyIHRoZXkgYWNjZXB0IHdoYXQgdGhlCmV4cG9ydGVyIGlzIGdpdmluZyB0aGVtLCBv ciB0aGV5IHJlamVjdCBpdC4KClRoZSBvdGhlciBpc3N1ZSBoZXJlIGlzIHRoYXQgRE1BIHNjYXR0 ZXJsaXN0cyBhcmUgX25vdF8gcmVhbGx5IHRoYXQKZGV0ZXJtaW5hYmxlIGluIHRlcm1zIG9mIG51 bWJlciBvZiBlbnRyaWVzIHdoZW4gaXQgY29tZXMgdG8gc3lzdGVtcyB3aXRoCnN5c3RlbSBJT01N VXMuICBTeXN0ZW0gSU9NTVVzLCB3aGljaCBzaG91bGQgYmUgaW50ZWdyYXRlZCBpbnRvIHRoZSBE TUEKQVBJLCBhcmUgcGVybWl0dGVkIHRvIGNvYWxlc2NlIGVudHJpZXMgaW4gdGhlIHBoeXNpY2Fs IHBhZ2UgcmFuZ2UuICBGb3IKZXhhbXBsZToKCgluc2cgPSAxMjg7CgluID0gZG1hX21hcF9zZyhk ZXYsIHNnLCBuc2csIERNQV9UT19ERVZJQ0UpOwoKSGVyZSwgbiBtaWdodCBiZSA0IGlmIHRoZSBz eXN0ZW0gSU9NTVUgaGFzIGJlZW4gYWJsZSB0byBjb2FsZXNjZSB0aGUgMTI4CmVudHJpZXMgZG93 biB0byA0IElPTU1VIGVudHJpZXMgLSBhbmQgdGhhdCBtZWFucyBmb3IgRE1BIHB1cnBvc2VzLCBv bmx5CnRoZSBmaXJzdCBmb3VyIHNjYXR0ZXJsaXN0IGVudHJpZXMgc2hvdWxkIGJlIHdhbGtlZCAo dGhpcyBpcyB3aHkKZG1hX21hcF9zZygpIHJldHVybnMgYSBwb3NpdGl2ZSBpbnRlZ2VyIHdoZW4g bWFwcGluZy4pCgpFYWNoIHN0cnVjdCBkZXZpY2UgaGFzIGEgc2V0IG9mIHBhcmFtZXRlcnMgd2hp Y2ggY29udHJvbCBob3cgdGhlIElPTU1VCmVudHJpZXMgYXJlIGNvYWxlc2NlZDoKCnN0cnVjdCBk ZXZpY2VfZG1hX3BhcmFtZXRlcnMgewogICAgICAgIC8qCiAgICAgICAgICogYSBsb3cgbGV2ZWwg ZHJpdmVyIG1heSBzZXQgdGhlc2UgdG8gdGVhY2ggSU9NTVUgY29kZSBhYm91dAogICAgICAgICAq IHNnIGxpbWl0YXRpb25zLgogICAgICAgICAqLwogICAgICAgIHVuc2lnbmVkIGludCBtYXhfc2Vn bWVudF9zaXplOwogICAgICAgIHVuc2lnbmVkIGxvbmcgc2VnbWVudF9ib3VuZGFyeV9tYXNrOwp9 OwoKYW5kIHRoaXMgaXMgaW5kZXBlbmRlbnQgb2YgdGhlIGRtYV9idWYgQVBJLiAgVGhpcyBkb2Vz bid0IGluZGljYXRlIHRoZQptYXhpbXVtIG51bWJlciBvZiBzZWdtZW50cywgYnV0IGFzIEkndmUg c2hvd24gYWJvdmUsIGl0J3Mgbm90IHNvbWV0aGluZwp0aGF0IHlvdSBjYW4gc2F5ICJJIHdhbnQg YSBzY2F0dGVybGlzdCBmb3IgdGhpcyBtZW1vcnkgd2l0aCBvbmx5IDMyCnNlZ21lbnRzIiBzbyBp dCdzIHRvdGFsbHkgdW5jbGVhciBob3cgYW4gZXhwb3J0ZXIgd291bGQgbGltaXQgdGhhdC4KClRo ZSBvbmx5IHRoaW5nIGFuIGV4cG9ydGVyIGNvdWxkIGRvIHdvdWxkIGJlIHRvIGZhaWwgdGhlIGV4 cG9ydCBpZiB0aGUKYnVmZmVyIGRpZG4ndCBlbmQgdXAgaGF2aW5nIGZld2VyIHRoYW4gdGhlIHJl cXVlc3RlZCBzY2F0dGVybGlzdCBlbnRyaWVzLAp3aGljaCBpcyBzb21ldGhpbmcgdGhlIGltcG9y dGVyIGNhbiBkbyB0b28uCgo+IE9uZSBvZiB0aGUgbWFpbiBwcm9ibGVtcyBlbmQtdXNlcnMgYXJl IGZhY2VkIHdpdGggdG9kYXkgaXMgdGhhdCB0aGV5IGRvIG5vdAo+IGtub3cgd2hpY2ggZGV2aWNl IHNob3VsZCBiZSB0aGUgZXhwb3J0ZXIgb2YgYnVmZmVycyBhbmQgd2hpY2ggc2hvdWxkIGJlIHRo ZQo+IGltcG9ydGVyLiBUaGlzIGRlcGVuZHMgb24gdGhlIGNvbnN0cmFpbnRzIGFuZCByaWdodCBu b3cgYXBwbGljYXRpb25zIGhhdmUKPiBubyB3YXkgb2Yga25vd2luZyB0aGlzLiBJdCdzIG51dHMg dGhhdCB0aGlzIGhhc24ndCBiZWVuIGFkZHJlc3NlZCB5ZXQgc2luY2UKPiBpdCBpcyB0aGUgbWFp biBjb21wbGFpbnQgSSBhbSBnZXR0aW5nLgoKSVQncyBudXRzIHRoYXQgd2UndmUgZW5kZWQgdXAg aW4gdGhpcyBzaXR1YXRpb24gaW4gdGhlIGZpcnN0IHBsYWNlLiAgVGhpcwp3YXMgYm91bmQgdG8g aGFwcGVuIGFzIHNvb24gYXMgdGhlIGRtYV9idWYgc2hhcmluZyB3YXMgaW50cm9kdWNlZCwgYmVj YXVzZQppdCBpbW1lZGlhdGVseSBpbnRyb2R1Y2VkIHRoaXMgcHJvYmxlbS4gIEkgZG9uJ3QgdGhp bmsgdGhlcmUgaXMgYW55IGVhc3kKc29sdXRpb24gdG8gaXQsIGFuZCB3aGF0J3MgYmVpbmcgcHJv cG9zZWQgd2l0aCBmbGFncyBhbmQgb3RoZXIgc3R1ZmYgaXMKanVzdCB0cnlpbmcgdG8gcGFwZXIg b3ZlciB0aGUgcHJvYmxlbS4KCldoYXQgeW91J3JlIGFjdHVhbGx5IGFza2luZyBpcyB0aGF0IGVh Y2ggZG1hYnVmIGV4cG9ydGluZyBzdWJzeXN0ZW0gbmVlZHMKdG8gcHVibGlzaCB0aGVpciBETUEg cGFyYW1ldGVycyB0byB1c2Vyc3BhY2UsIGFuZCB1c2Vyc3BhY2UgdGhlbiBnZXRzIHRvCmRlY2lk ZSB3aGljaCBkbWFidWYgZXhwb3J0ZXIgc2hvdWxkIGJlIHVzZWQuCgpUaGF0J3Mgbm90IGEgZG1h YnVmIHByb2JsZW0sIHRoYXQncyBhIHN1YnN5c3RlbSBwcm9ibGVtLCBidXQgZXZlbiBzbywgd2UK ZG9uJ3QgaGF2ZSBhIHN0YW5kYXJkaXNlZCB3YXkgdG8gZXhwb3J0IHRoYXQgaW5mb3JtYXRpb24g KGFuZCBJJ2Qgc3VzcGVjdAp0aGF0IGl0IHdvdWxkIGJlIHZlcnkgZGlmZmljdWx0IHRvIGdldCBh Z3JlZW1lbnRzIGJldHdlZW4gc3Vic3lzdGVtcyBvbgphIHN0YW5kYXJkIGlvY3RsIGFuZC9vciBk YXRhIHN0cnVjdHVyZS4pICBJbiBteSBleHBlcmllbmNlLCBnZXR0aW5nIGNyb3NzLQpzdWJzeXN0 ZW0gYWdyZWVtZW50IGluIHRoZSBrZXJuZWwgd2l0aCBhbnl0aGluZyBpcyB2ZXJ5IGRpZmZpY3Vs dCwgeW91Cm5vcm1hbGx5IGVuZCB1cCB3aXRoIDYwJSBvZiBwZW9wbGUgYWdyZWVpbmcsIGFuZCB0 aGUgb3RoZXIgNDAlIGdvaW5nIG9mZgphbmQgZG9pbmcgc29tZXRoaW5nIGNvbXBsZXRlbHkgZGlm ZmVyZW50IGJlY2F1c2UgdGhleSBvYmplY3QgdG8gaXQKKGZpZ3VyZXMgdmFyeSwgOTAlIG9mIGFs bCBzdGF0aXN0aWNzIGFyZSBtYWRlIHVwIG9uIHRoZSBzcG90ISkKCi0tIApGVFRDIGJyb2FkYmFu ZCBmb3IgMC44bWlsZSBsaW5lOiBjdXJyZW50bHkgYXQgMTAuNU1icHMgZG93biA0MDBrYnBzIHVw CmFjY29yZGluZyB0byBzcGVlZHRlc3QubmV0LgpfX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fXwpkcmktZGV2ZWwgbWFpbGluZyBsaXN0CmRyaS1kZXZlbEBsaXN0 cy5mcmVlZGVza3RvcC5vcmcKaHR0cDovL2xpc3RzLmZyZWVkZXNrdG9wLm9yZy9tYWlsbWFuL2xp c3RpbmZvL2RyaS1kZXZlbAo=