From mboxrd@z Thu Jan 1 00:00:00 1970 Return-path: Received: from oyp.chewa.net ([91.121.6.101]:48561 "EHLO oyp.chewa.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750923Ab2DTNDc (ORCPT ); Fri, 20 Apr 2012 09:03:32 -0400 To: Tomasz Stanislawski Subject: Re: [PATCH v4 02/14] Documentation: media: description of DMABUF importing in V4L2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Date: Fri, 20 Apr 2012 15:03:17 +0200 From: =?UTF-8?Q?R=C3=A9mi_Denis-Courmont?= Cc: Mauro Carvalho Chehab , Laurent Pinchart , , , , , , , , , , , , , In-Reply-To: <4F91559D.6060900@samsung.com> References: <1334332076-28489-1-git-send-email-t.stanislaws@samsung.com> <1334332076-28489-3-git-send-email-t.stanislaws@samsung.com> <13761406.oTf8ZzmZpQ@avalon> <4F9021FE.2070903@samsung.com> <4F907798.3000304@redhat.com> <4F912141.8060200@samsung.com> <4F91559D.6060900@samsung.com> Message-ID: Sender: linux-media-owner@vger.kernel.org List-ID: On Fri, 20 Apr 2012 14:25:01 +0200, Tomasz Stanislawski wrote: >>> The USERPTR simplifies userspace code but introduce >>> a lot of complexity problems for the kernel drivers >>> and frameworks. >> >> It is not only a simplification. In some cases, USERPTR is the only I/O >> method that supports zero copy in pretty much any circumstance. > > Only for devices that have its own IOMMU that can access system memory. Newer versions of the UVC driver have USERTPR, and simingly gspca seems too. That is practically all USB capture devices... That might be irrelevant for a smartphone manufacturer. That is very relevant for desktop applications. > Moreover the userptr must come from malloc or be a mmaped file. > The other case are drivers that touch memory using CPU in the kernel > space like VIVI or USB drivers. I'd argue that USB is the most common case of V4L2 on the desktop... >> When the user cannot reliably predict the maximum number of required >> buffers, predicts a value larger than the device will negotiate, or >> needs buffers to outlive STREAMOFF (?), MMAP requires memory copying. >> USERPTR does not. > > What does outlive STREAMOFF means in this context? Depending how your multimedia pipeline is built, it is plausible that the V4L2 source is shutdown (STREAMOFF then close()) before buffers coming from it are released/destroyed downstream. I might be wrong, but I would expect that V4L2 MMAP buffers become invalid after STREAMOFF+close()? > Anyway, IMO allocation of the buffers at VIDIOC_REQBUFS was not the best > idea because it introduces an allocation overhead for negotiations of > the number of the buffers. An allocation at mmap was to late. There is a > need for some intermediate state between REQBUFS and mmap. The ioctl > BUF_PREPARE may help here. > > Can you give me an example of a sane application is forced to negotiate a > larger number of buffers than it is actually going to use? Outside the embedded world, the application typically does not know what the latency of the multimedia pipeline is. If the latency is not known, the number of buffers needed for zero copy cannot be precomputed for REQBUFS, say: count = 1 + latency / frame interval. Even for a trivial analog TV viewer application, lip synchronization requires picture frames to be bufferred to be long enough to account for the latency of the audio input, dejitter, filtering and audio output. Those values are usually not well determined at the time of requesting buffers from the video capture device. Also the application may want to play nice with PulseAudio. Then it will get very long audio buffers with very few audio periods... more latency. It gets harder or outright impossible for frameworks dealing with complicated or arbitrary pipelines such as LibVLC or gstreamer. There is far too much unpredictability and variability downstream of the V4L2 source to estimate latency, and infer the number of buffers needed. >> Now, I do realize that some devices cannot support USERPTR efficiently, >> then they should not support USERPTR. > > The problem is not there is *NO* device that can handle USERPTR reliably. > The can handle USERPTR generated by malloc or page cache (not sure). > Memory mmaped from other devices, frameworks etc may or may not work. > Even if the device has its IOMMU the DMA layer provides no generic way to > transform from one device to the mapping in some other device. I'm not saying that USERPTR should replace DMABUF. I'm saying USERPTR has advantages over MMAP that DMABUF does not seem to cover as yet (if only libv4l2 would not inhibit USERPTR...). I'm definitely not saying that applications should rely on USERPTR being supported. We agree that not all devices can support USERPTR. > The userptr has its niches were it works pretty well like Web cams or VIVI. > I am saying that if ever DMABUF becomes a good alternative for USERPTR > than maybe we should consider encouraging dropping USERPTR in the new > drivers as 'obsolete' feature and providing some emulation layer in libv4l2 > for legacy applications. Sure. -- Rémi Denis-Courmont Sent from my collocated server From mboxrd@z Thu Jan 1 00:00:00 1970 From: =?UTF-8?Q?R=C3=A9mi_Denis-Courmont?= Subject: Re: [PATCH v4 02/14] Documentation: media: description of DMABUF importing in V4L2 Date: Fri, 20 Apr 2012 15:03:17 +0200 Message-ID: References: <1334332076-28489-1-git-send-email-t.stanislaws@samsung.com> <1334332076-28489-3-git-send-email-t.stanislaws@samsung.com> <13761406.oTf8ZzmZpQ@avalon> <4F9021FE.2070903@samsung.com> <4F907798.3000304@redhat.com> <4F912141.8060200@samsung.com> <4F91559D.6060900@samsung.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: QUOTED-PRINTABLE Return-path: In-Reply-To: <4F91559D.6060900@samsung.com> Sender: linux-media-owner@vger.kernel.org To: Tomasz Stanislawski Cc: Mauro Carvalho Chehab , Laurent Pinchart , linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, airlied@redhat.com, m.szyprowski@samsung.com, kyungmin.park@samsung.com, sumit.semwal@ti.com, daeinki@gmail.com, daniel.vetter@ffwll.ch, robdclark@gmail.com, pawel@osciak.com, linaro-mm-sig@lists.linaro.org, hverkuil@xs4all.nl, subashrp@gmail.com List-Id: dri-devel@lists.freedesktop.org On Fri, 20 Apr 2012 14:25:01 +0200, Tomasz Stanislawski=0D wrote:=0D >>> The USERPTR simplifies userspace code but introduce=0D >>> a lot of complexity problems for the kernel drivers=0D >>> and frameworks.=0D >> =0D >> It is not only a simplification. In some cases, USERPTR is the only = I/O=0D >> method that supports zero copy in pretty much any circumstance.=0D > =0D > Only for devices that have its own IOMMU that can access system memor= y.=0D =0D Newer versions of the UVC driver have USERTPR, and simingly gspca seems= =0D too. That is practically all USB capture devices... That might be=0D irrelevant for a smartphone manufacturer. That is very relevant for des= ktop=0D applications.=0D =0D > Moreover the userptr must come from malloc or be a mmaped file.=0D > The other case are drivers that touch memory using CPU in the kernel=0D > space like VIVI or USB drivers.=0D =0D I'd argue that USB is the most common case of V4L2 on the desktop...=0D =0D >> When the user cannot reliably predict the maximum number of required= =0D >> buffers, predicts a value larger than the device will negotiate, or=0D >> needs buffers to outlive STREAMOFF (?), MMAP requires memory copying= =2E=0D >> USERPTR does not.=0D > =0D > What does outlive STREAMOFF means in this context?=0D =0D Depending how your multimedia pipeline is built, it is plausible that t= he=0D V4L2 source is shutdown (STREAMOFF then close()) before buffers coming = from=0D it are released/destroyed downstream. I might be wrong, but I would exp= ect=0D that V4L2 MMAP buffers become invalid after STREAMOFF+close()?=0D =0D > Anyway, IMO allocation of the buffers at VIDIOC_REQBUFS was not the b= est=0D > idea because it introduces an allocation overhead for negotiations of= =0D > the number of the buffers. An allocation at mmap was to late. There i= s a=0D > need for some intermediate state between REQBUFS and mmap. The ioctl=0D > BUF_PREPARE may help here.=0D > =0D > Can you give me an example of a sane application is forced to negotia= te=0D a=0D > larger number of buffers than it is actually going to use?=0D =0D Outside the embedded world, the application typically does not know wha= t=0D the latency of the multimedia pipeline is. If the latency is not known,= the=0D number of buffers needed for zero copy cannot be precomputed for REQBUF= S,=0D say:=0D =0D count =3D 1 + latency / frame interval.=0D =0D Even for a trivial analog TV viewer application, lip synchronization=0D requires picture frames to be bufferred to be long enough to account fo= r=0D the latency of the audio input, dejitter, filtering and audio output. T= hose=0D values are usually not well determined at the time of requesting buffer= s=0D from the video capture device. Also the application may want to play ni= ce=0D with PulseAudio. Then it will get very long audio buffers with very few= =0D audio periods... more latency.=0D =0D It gets harder or outright impossible for frameworks dealing with=0D complicated or arbitrary pipelines such as LibVLC or gstreamer. There i= s=0D far too much unpredictability and variability downstream of the V4L2 so= urce=0D to estimate latency, and infer the number of buffers needed.=0D =0D >> Now, I do realize that some devices cannot support USERPTR efficient= ly,=0D >> then they should not support USERPTR. =0D > =0D > The problem is not there is *NO* device that can handle USERPTR=0D reliably.=0D > The can handle USERPTR generated by malloc or page cache (not sure).=0D > Memory mmaped from other devices, frameworks etc may or may not work.= =0D > Even if the device has its IOMMU the DMA layer provides no generic wa= y=0D to=0D > transform from one device to the mapping in some other device.=0D =0D I'm not saying that USERPTR should replace DMABUF. I'm saying USERPTR h= as=0D advantages over MMAP that DMABUF does not seem to cover as yet (if only= =0D libv4l2 would not inhibit USERPTR...).=0D =0D I'm definitely not saying that applications should rely on USERPTR bein= g=0D supported. We agree that not all devices can support USERPTR.=0D =0D > The userptr has its niches were it works pretty well like Web cams or= =0D VIVI.=0D > I am saying that if ever DMABUF becomes a good alternative for USERPT= R=0D > than maybe we should consider encouraging dropping USERPTR in the new= =0D > drivers as 'obsolete' feature and providing some emulation layer in=0D libv4l2=0D > for legacy applications.=0D =0D Sure.=0D =0D -- =0D R=C3=A9mi Denis-Courmont=0D Sent from my collocated server