* v4l2loopback and v4l2-mem2mem framework @ 2021-02-28 22:07 Wren Turkal 2021-03-01 20:27 ` Wren Turkal 0 siblings, 1 reply; 11+ messages in thread From: Wren Turkal @ 2021-02-28 22:07 UTC (permalink / raw) To: linux-media Hi there linux-media experts, I am working on the out-of-tree v4l2loopback module[1]. My current goal is to implement the videobuf2 api. However, it appears that I must use a different vbuf2_queue for the OUTPUT and CAPTURE types. It appears that the mem2mem framework has the ability to do just this. I came to this conclusion while looking at the vim2m test module's code. While trying to modify the loopback module, I have added some code to try to call v4l2_m2m_init. You can see my very WIP attempt at code here[2]. I have installed the kernel-devel package on Fedora, which. Unfortunately, I get he following error messages when trying to compile: <errors> ERROR: modpost: "v4l2_m2m_release" [/home/wt/projects/kernel/v4l2loopback/v4l2loopback.ko] undefined! ERROR: modpost: "v4l2_m2m_init" [/home/wt/projects/kernel/v4l2loopback/v4l2loopback.ko] undefined! </errors> So, I looked into it, and Module.symvers and /proc/kallsyms don't have those symbols listed, so I think the Fedora kernel was not compiled with those symbols exported. Okay, so next I tried to build a custom kernel that does export those symbols. I managed to get it done by enabling the vim2m (VIDEO_VIM2M config option) module. The new Module.symvers includes the symbols; and the compile succeeds, which brings me to my final act. I need to expose those symbols without enabling the vim2m module since it is a testing module. I do not see a way to do that explicitly. The config option in question appears to be V4L2_MEM2MEM_DEV. And I haven't found a way to enable it without enabling something that implicitly selects it (which vim2m does). Do y'all have any suggestions for how to get these symbols exported properly when vim2m is not enabled? Thanks, Wren Turkal (wt) [1] https://github.com/umlaeute/v4l2loopback [2] https://github.com/wt/v4l2loopback/tree/m2m_try ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: v4l2loopback and v4l2-mem2mem framework 2021-02-28 22:07 v4l2loopback and v4l2-mem2mem framework Wren Turkal @ 2021-03-01 20:27 ` Wren Turkal 2021-03-02 8:01 ` Ezequiel Garcia [not found] ` <CAAEAJfCSfwxv5NyaH05Dfiw4wm6vGxL_ajoFup6r-GKx_VymJA@mail.gmail.com> 0 siblings, 2 replies; 11+ messages in thread From: Wren Turkal @ 2021-03-01 20:27 UTC (permalink / raw) To: linux-media Okay, so I have found how to enable the option to be manually selected. Would a patch like the following be acceptable? Or is there a better way to do this? diff --git a/drivers/media/v4l2-core/Kconfig b/drivers/media/v4l2-core/Kconfig index bf49f83cb86f..25004137a982 100644 --- a/drivers/media/v4l2-core/Kconfig +++ b/drivers/media/v4l2-core/Kconfig @@ -54,7 +54,7 @@ config V4L2_H264 # Used by drivers that need v4l2-mem2mem.ko config V4L2_MEM2MEM_DEV - tristate + tristate "V4L2 mem2mem framework support" depends on VIDEOBUF2_CORE # Used by LED subsystem flash drivers Thanks, wt On 2/28/21 2:07 PM, Wren Turkal wrote: > Hi there linux-media experts, > > I am working on the out-of-tree v4l2loopback module[1]. My > current goal is to implement the videobuf2 api. However, > it appears that I must use a different vbuf2_queue for the > OUTPUT and CAPTURE types. It appears that the mem2mem > framework has the ability to do just this. I came to this > conclusion while looking at the vim2m test module's code. > While trying to modify the loopback module, I have added > some code to try to call v4l2_m2m_init. You can see my very > WIP attempt at code here[2]. > > I have installed the kernel-devel package on Fedora, which. > Unfortunately, I get he following error messages when trying > to compile: > > <errors> > ERROR: modpost: "v4l2_m2m_release" > [/home/wt/projects/kernel/v4l2loopback/v4l2loopback.ko] undefined! > ERROR: modpost: "v4l2_m2m_init" > [/home/wt/projects/kernel/v4l2loopback/v4l2loopback.ko] undefined! > </errors> > > So, I looked into it, and Module.symvers and /proc/kallsyms > don't have those symbols listed, so I think the Fedora kernel > was not compiled with those symbols exported. > > Okay, so next I tried to build a custom kernel that does > export those symbols. I managed to get it done by enabling > the vim2m (VIDEO_VIM2M config option) module. The new > Module.symvers includes the symbols; and the compile succeeds, > which brings me to my final act. > > I need to expose those symbols without enabling the vim2m > module since it is a testing module. I do not see a way > to do that explicitly. The config option in question > appears to be V4L2_MEM2MEM_DEV. And I haven't found a > way to enable it without enabling something that implicitly > selects it (which vim2m does). Do y'all have any suggestions > for how to get these symbols exported properly when vim2m is > not enabled? > > > Thanks, > Wren Turkal (wt) > > [1] https://github.com/umlaeute/v4l2loopback > [2] https://github.com/wt/v4l2loopback/tree/m2m_try ^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: v4l2loopback and v4l2-mem2mem framework 2021-03-01 20:27 ` Wren Turkal @ 2021-03-02 8:01 ` Ezequiel Garcia 2021-03-04 19:10 ` Mauro Carvalho Chehab [not found] ` <CAAEAJfCSfwxv5NyaH05Dfiw4wm6vGxL_ajoFup6r-GKx_VymJA@mail.gmail.com> 1 sibling, 1 reply; 11+ messages in thread From: Ezequiel Garcia @ 2021-03-02 8:01 UTC (permalink / raw) To: Wren Turkal; +Cc: linux-media Hi Wren, How about you submit the loopback driver for integration in mainline? We can review the driver, give feedback and help you get it merged. Ezequiel On Mon, 1 Mar 2021 at 17:32, Wren Turkal <wt@penguintechs.org> wrote: > > Okay, so I have found how to enable the option to be manually selected. > Would a patch like the following be acceptable? Or is there a better > way to do this? > > diff --git a/drivers/media/v4l2-core/Kconfig > b/drivers/media/v4l2-core/Kconfig > index bf49f83cb86f..25004137a982 100644 > --- a/drivers/media/v4l2-core/Kconfig > +++ b/drivers/media/v4l2-core/Kconfig > @@ -54,7 +54,7 @@ config V4L2_H264 > > # Used by drivers that need v4l2-mem2mem.ko > config V4L2_MEM2MEM_DEV > - tristate > + tristate "V4L2 mem2mem framework support" > depends on VIDEOBUF2_CORE > > # Used by LED subsystem flash drivers > > > Thanks, > wt > > On 2/28/21 2:07 PM, Wren Turkal wrote: > > Hi there linux-media experts, > > > > I am working on the out-of-tree v4l2loopback module[1]. My > > current goal is to implement the videobuf2 api. However, > > it appears that I must use a different vbuf2_queue for the > > OUTPUT and CAPTURE types. It appears that the mem2mem > > framework has the ability to do just this. I came to this > > conclusion while looking at the vim2m test module's code. > > While trying to modify the loopback module, I have added > > some code to try to call v4l2_m2m_init. You can see my very > > WIP attempt at code here[2]. > > > > I have installed the kernel-devel package on Fedora, which. > > Unfortunately, I get he following error messages when trying > > to compile: > > > > <errors> > > ERROR: modpost: "v4l2_m2m_release" > > [/home/wt/projects/kernel/v4l2loopback/v4l2loopback.ko] undefined! > > ERROR: modpost: "v4l2_m2m_init" > > [/home/wt/projects/kernel/v4l2loopback/v4l2loopback.ko] undefined! > > </errors> > > > > So, I looked into it, and Module.symvers and /proc/kallsyms > > don't have those symbols listed, so I think the Fedora kernel > > was not compiled with those symbols exported. > > > > Okay, so next I tried to build a custom kernel that does > > export those symbols. I managed to get it done by enabling > > the vim2m (VIDEO_VIM2M config option) module. The new > > Module.symvers includes the symbols; and the compile succeeds, > > which brings me to my final act. > > > > I need to expose those symbols without enabling the vim2m > > module since it is a testing module. I do not see a way > > to do that explicitly. The config option in question > > appears to be V4L2_MEM2MEM_DEV. And I haven't found a > > way to enable it without enabling something that implicitly > > selects it (which vim2m does). Do y'all have any suggestions > > for how to get these symbols exported properly when vim2m is > > not enabled? > > > > > > Thanks, > > Wren Turkal (wt) > > > > [1] https://github.com/umlaeute/v4l2loopback > > [2] https://github.com/wt/v4l2loopback/tree/m2m_try > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: v4l2loopback and v4l2-mem2mem framework 2021-03-02 8:01 ` Ezequiel Garcia @ 2021-03-04 19:10 ` Mauro Carvalho Chehab 2021-03-04 19:34 ` Ezequiel Garcia 2021-03-04 22:19 ` Wren Turkal 0 siblings, 2 replies; 11+ messages in thread From: Mauro Carvalho Chehab @ 2021-03-04 19:10 UTC (permalink / raw) To: Ezequiel Garcia; +Cc: Wren Turkal, linux-media Em Tue, 2 Mar 2021 05:01:16 -0300 Ezequiel Garcia <ezequiel@vanguardiasur.com.ar> escreveu: > Hi Wren, > > How about you submit the loopback driver for integration in mainline? > We can review the driver, give feedback and help you get it merged. Please, don't top post. The loopback driver is not something that could be merged. It was already submitted a few times. One of the main issues is that userspace may sleep, and this could cause some bad effects and even hangups due to V4L2 core and VB2 mutex and spin locks. Regards, Mauro > > Ezequiel > > On Mon, 1 Mar 2021 at 17:32, Wren Turkal <wt@penguintechs.org> wrote: > > > > Okay, so I have found how to enable the option to be manually selected. > > Would a patch like the following be acceptable? Or is there a better > > way to do this? > > > > diff --git a/drivers/media/v4l2-core/Kconfig > > b/drivers/media/v4l2-core/Kconfig > > index bf49f83cb86f..25004137a982 100644 > > --- a/drivers/media/v4l2-core/Kconfig > > +++ b/drivers/media/v4l2-core/Kconfig > > @@ -54,7 +54,7 @@ config V4L2_H264 > > > > # Used by drivers that need v4l2-mem2mem.ko > > config V4L2_MEM2MEM_DEV > > - tristate > > + tristate "V4L2 mem2mem framework support" > > depends on VIDEOBUF2_CORE > > > > # Used by LED subsystem flash drivers > > > > > > Thanks, > > wt > > > > On 2/28/21 2:07 PM, Wren Turkal wrote: > > > Hi there linux-media experts, > > > > > > I am working on the out-of-tree v4l2loopback module[1]. My > > > current goal is to implement the videobuf2 api. However, > > > it appears that I must use a different vbuf2_queue for the > > > OUTPUT and CAPTURE types. It appears that the mem2mem > > > framework has the ability to do just this. I came to this > > > conclusion while looking at the vim2m test module's code. > > > While trying to modify the loopback module, I have added > > > some code to try to call v4l2_m2m_init. You can see my very > > > WIP attempt at code here[2]. > > > > > > I have installed the kernel-devel package on Fedora, which. > > > Unfortunately, I get he following error messages when trying > > > to compile: > > > > > > <errors> > > > ERROR: modpost: "v4l2_m2m_release" > > > [/home/wt/projects/kernel/v4l2loopback/v4l2loopback.ko] undefined! > > > ERROR: modpost: "v4l2_m2m_init" > > > [/home/wt/projects/kernel/v4l2loopback/v4l2loopback.ko] undefined! > > > </errors> > > > > > > So, I looked into it, and Module.symvers and /proc/kallsyms > > > don't have those symbols listed, so I think the Fedora kernel > > > was not compiled with those symbols exported. > > > > > > Okay, so next I tried to build a custom kernel that does > > > export those symbols. I managed to get it done by enabling > > > the vim2m (VIDEO_VIM2M config option) module. The new > > > Module.symvers includes the symbols; and the compile succeeds, > > > which brings me to my final act. > > > > > > I need to expose those symbols without enabling the vim2m > > > module since it is a testing module. I do not see a way > > > to do that explicitly. The config option in question > > > appears to be V4L2_MEM2MEM_DEV. And I haven't found a > > > way to enable it without enabling something that implicitly > > > selects it (which vim2m does). Do y'all have any suggestions > > > for how to get these symbols exported properly when vim2m is > > > not enabled? > > > > > > > > > Thanks, > > > Wren Turkal (wt) > > > > > > [1] https://github.com/umlaeute/v4l2loopback > > > [2] https://github.com/wt/v4l2loopback/tree/m2m_try > > Thanks, Mauro ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: v4l2loopback and v4l2-mem2mem framework 2021-03-04 19:10 ` Mauro Carvalho Chehab @ 2021-03-04 19:34 ` Ezequiel Garcia 2021-03-04 22:19 ` Wren Turkal 1 sibling, 0 replies; 11+ messages in thread From: Ezequiel Garcia @ 2021-03-04 19:34 UTC (permalink / raw) To: Mauro Carvalho Chehab; +Cc: Wren Turkal, linux-media On Thu, 4 Mar 2021 at 16:11, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote: > > Em Tue, 2 Mar 2021 05:01:16 -0300 > Ezequiel Garcia <ezequiel@vanguardiasur.com.ar> escreveu: > > > Hi Wren, > > > > How about you submit the loopback driver for integration in mainline? > > We can review the driver, give feedback and help you get it merged. > > Please, don't top post. > > The loopback driver is not something that could be merged. It was > already submitted a few times. > > One of the main issues is that userspace may sleep, and this could cause > some bad effects and even hangups due to V4L2 core and VB2 mutex and > spin locks. > Can you elaborate on this? I always felt that it's a shame for something so popular as the loopback to be out-of-tree and unsupported. Thanks, Ezequiel ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: v4l2loopback and v4l2-mem2mem framework 2021-03-04 19:10 ` Mauro Carvalho Chehab 2021-03-04 19:34 ` Ezequiel Garcia @ 2021-03-04 22:19 ` Wren Turkal 1 sibling, 0 replies; 11+ messages in thread From: Wren Turkal @ 2021-03-04 22:19 UTC (permalink / raw) To: Mauro Carvalho Chehab, Ezequiel Garcia; +Cc: linux-media On 3/4/21 11:10 AM, Mauro Carvalho Chehab wrote: > Please, don't top post. Sorry about the top post. > The loopback driver is not something that could be merged. It was > already submitted a few times. I'm willing to develop something new if you think that would be a better approach. I just want the functionality of the loopback so that OBS (or something else) can be used to compose a video scene for use by all the apps on Linux. > One of the main issues is that userspace may sleep, and this could cause > some bad effects and even hangups due to V4L2 core and VB2 mutex and > spin locks. Is this something that could be fixed? If so, I am happy to try to help do that. wt ^ permalink raw reply [flat|nested] 11+ messages in thread
[parent not found: <CAAEAJfCSfwxv5NyaH05Dfiw4wm6vGxL_ajoFup6r-GKx_VymJA@mail.gmail.com>]
[parent not found: <86f1dd6b-7593-adba-b5a6-abf4f8e35d8b@penguintechs.org>]
* Re: v4l2loopback and v4l2-mem2mem framework [not found] ` <86f1dd6b-7593-adba-b5a6-abf4f8e35d8b@penguintechs.org> @ 2021-03-02 22:08 ` Wren Turkal 2021-03-03 22:41 ` Nicolas Dufresne 0 siblings, 1 reply; 11+ messages in thread From: Wren Turkal @ 2021-03-02 22:08 UTC (permalink / raw) To: Ezequiel Garcia; +Cc: linux-media Okay, so I tried to write up a little bit of an RFC to organize my thoughts. I would love to get your thoughts: RFC: A loopback device for the v4l2 subsystem. # What is needed? The common denominator for apps that use webcams in linux is the v4l2 subsystem. The v4l2 subsystem could benefit from a loopback device that allows one to compose custom video streams and present them in a normal way from a /dev/video* device. # Why is it needed? The v4l2 is the framework is the kernel interface used for supporting video output and video capture. This brings the webcam video to apps like Cheese or a web browser or OBS. However, there is no way to customize the composition of the video stream before capture on these apps. Of course, apps like Cheese have filters, but this is not something that is composible with other apps. This makes it impossible to do something like the following: * Composite a video scene with OBS. * Use that composed video as my webcam for my Zoom call. I think this would be cool, and I think that it would allow interesting things to be done with video. # Prior Art The v4l2loopback project[1] provides a Linux kernel module to do just this. This module provides a misc device node used to control the loopback video devices. In particular, the misc device is used to create and destroy loopback video devices. It's somewhat similar to the /dev/loop-control dev node that is used to control block loop devices. There is also the vim2m test driver[2] in the kernel that appears to do some similar things. This module is a module that is used for testing the v4l2-mem2mem framework. This module has a more modern structure with a platform device that is used as the context for the other devices. It also uses the m2m framework that I think could benefit an in-kernel loopback implementation. It is also important to note that this module does not generally ship with distribution kernels (like in Fedora), likely due to the "test" designation. # Possible paths forward ## Integrate v4l2loopback into the kernel There is a lot to cleanup in the v4l2loopback code in order to get it in shape for mainline inclusion. Having said that, this also a somewhat tested implementation. There are even packages for using this project from Ubuntu. ## Start with vim2m I think the vim2m.c could provide a really strong base on which to start a loopback implementation for the kernel. I see two options: * Extend vim2m to include more loopback functionality, including dynamic creation/deletion of video loopback devices. * Copy the vim2m code to a new module and develop there. The extend option is attractive so that a new module mustn't be developed. However, I am not sure that I want to complicate that module with options that would required for a proper loopback implementation. # My thoughts on where to go next I think that the vim2m module provides a much nicer starting point for an implementation of a mainline module. I think the the ideal solution would start by using the vim2m source as a template and developing a new module that incorporates the important functionality from v4l2loopback. # Assumptions I have I have looked into the code, and the m2m framework appears to provide a pair of vb2_queues that can be used for capture and output. I am assuming that the framework would be reasonable to use for a loopback implementation. [1] https://github.com/umlaeute/v4l2loopback [2] https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/drivers/media/test-drivers/vim2m.c?h=v5.11.2 Thanks, wt On 3/2/21 10:03 AM, Wren Turkal wrote: > Ezequiel, > > Submitting for upstream inclusion is a good idea, and the driver > is licensed appropriately (gpl v2) for the kernel. I have asked > the current maintainer for their thoughts. They are supportive > of the idea, but they are in the middle of implementing an new > feature that they would like to finish. I will work it up for > inclusion when that work concludes since it's a pretty major change. > > Having said that, does the patch I posted make sense? I think > that being able to enable the m2m framework without having to > build in a m2m driver explicitly makes good sense. > > Also, thinking about this a little more, it might make more > sense use use vim2m and v4l2loopback as inspiration and > make something like loopback that looks more like vim2m. > > Things I like about vim2m vs v4l2loopback right now: > * vim2m uses the m2m framework > * vim2m is better organized by using a platform device > to organize the devices that created under the context > of the platform device. v4l2lb uses a misc device and > doesn't explicitly place devices under in the context > of that device. > * The misc device created by v4l2lb is used to > dynamically add and remove v4l2lb video devices. It > It uses new IOCTLs to do that, which I wonder if there > is a better way. > * vim2m exposes config via a mechanism that doesn't > require a new tool. v4l2lb requires a new control > binary because of the custom ioctls. > > So, I have a couple questions: > * Would it be better to use vim2m as a model to > implement something like v4l2loopback? > * If I were to do that should I copy vim2m to a > new module, or should vim2m be extended. FWIW, > vim2m appears to have two things against it: > * vim2m is a test v4l2 driver. It would need > to be changed to build as a non-test driver. > This is a pretty easy Kconfig change and maybe > moving files to a more appropriate place in the > linux source tree. > * v4l2loopback does have a number of features that > vim2m does not have. I will say that I think > that adding most importantant functionality > (new formats and dynamic creation of loopback > video devices in my mind) is probably very > approachable and may give us a more modern > linux module code. > > Give the above, I am kinda leaning toward extending > vim2m code. So, do you have an opinion on the following? > Should it be a new module, or should it literally extend > and improve vim2m? > > wt > > > On 3/1/21 6:11 PM, Ezequiel Garcia wrote: >> Hi Wren, >> >> How about you submit the loopback driver for integration in mainline? >> >> We can review the driver, give feedback and help you get it merged. >> >> Ezequiel >> >> >> On Mon, Mar 1, 2021, 5:32 PM Wren Turkal <wt@penguintechs.org >> <mailto:wt@penguintechs.org>> wrote: >> >> Okay, so I have found how to enable the option to be manually >> selected. >> Would a patch like the following be acceptable? Or is there a better >> way to do this? >> >> diff --git a/drivers/media/v4l2-core/Kconfig >> b/drivers/media/v4l2-core/Kconfig >> index bf49f83cb86f..25004137a982 100644 >> --- a/drivers/media/v4l2-core/Kconfig >> +++ b/drivers/media/v4l2-core/Kconfig >> @@ -54,7 +54,7 @@ config V4L2_H264 >> >> # Used by drivers that need v4l2-mem2mem.ko >> config V4L2_MEM2MEM_DEV >> - tristate >> + tristate "V4L2 mem2mem framework support" >> depends on VIDEOBUF2_CORE >> >> # Used by LED subsystem flash drivers >> >> >> Thanks, >> wt >> >> On 2/28/21 2:07 PM, Wren Turkal wrote: >> > Hi there linux-media experts, >> > >> > I am working on the out-of-tree v4l2loopback module[1]. My >> > current goal is to implement the videobuf2 api. However, >> > it appears that I must use a different vbuf2_queue for the >> > OUTPUT and CAPTURE types. It appears that the mem2mem >> > framework has the ability to do just this. I came to this >> > conclusion while looking at the vim2m test module's code. >> > While trying to modify the loopback module, I have added >> > some code to try to call v4l2_m2m_init. You can see my very >> > WIP attempt at code here[2]. >> > >> > I have installed the kernel-devel package on Fedora, which. >> > Unfortunately, I get he following error messages when trying >> > to compile: >> > >> > <errors> >> > ERROR: modpost: "v4l2_m2m_release" >> > [/home/wt/projects/kernel/v4l2loopback/v4l2loopback.ko] undefined! >> > ERROR: modpost: "v4l2_m2m_init" >> > [/home/wt/projects/kernel/v4l2loopback/v4l2loopback.ko] undefined! >> > </errors> >> > >> > So, I looked into it, and Module.symvers and /proc/kallsyms >> > don't have those symbols listed, so I think the Fedora kernel >> > was not compiled with those symbols exported. >> > >> > Okay, so next I tried to build a custom kernel that does >> > export those symbols. I managed to get it done by enabling >> > the vim2m (VIDEO_VIM2M config option) module. The new >> > Module.symvers includes the symbols; and the compile succeeds, >> > which brings me to my final act. >> > >> > I need to expose those symbols without enabling the vim2m >> > module since it is a testing module. I do not see a way >> > to do that explicitly. The config option in question >> > appears to be V4L2_MEM2MEM_DEV. And I haven't found a >> > way to enable it without enabling something that implicitly >> > selects it (which vim2m does). Do y'all have any suggestions >> > for how to get these symbols exported properly when vim2m is >> > not enabled? >> > >> > >> > Thanks, >> > Wren Turkal (wt) >> > >> > [1] https://github.com/umlaeute/v4l2loopback >> <https://github.com/umlaeute/v4l2loopback> >> > [2] https://github.com/wt/v4l2loopback/tree/m2m_try >> <https://github.com/wt/v4l2loopback/tree/m2m_try> >> > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: v4l2loopback and v4l2-mem2mem framework 2021-03-02 22:08 ` Wren Turkal @ 2021-03-03 22:41 ` Nicolas Dufresne 2021-03-03 23:41 ` Wren Turkal 0 siblings, 1 reply; 11+ messages in thread From: Nicolas Dufresne @ 2021-03-03 22:41 UTC (permalink / raw) To: Wren Turkal, Ezequiel Garcia; +Cc: linux-media Le mardi 02 mars 2021 à 14:08 -0800, Wren Turkal a écrit : > Okay, so I tried to write up a little bit of an RFC to organize my > thoughts. I would love to get your thoughts: > > RFC: A loopback device for the v4l2 subsystem. > > # What is needed? > > The common denominator for apps that use webcams in linux is the > v4l2 subsystem. The v4l2 subsystem could benefit from a loopback > device that allows one to compose custom video streams and present > them in a normal way from a /dev/video* device. > > # Why is it needed? > > The v4l2 is the framework is the kernel interface used for supporting > video output and video capture. This brings the webcam video to apps > like Cheese or a web browser or OBS. > > However, there is no way to customize the composition of the video > stream before capture on these apps. Of course, apps like Cheese have > filters, but this is not something that is composible with other apps. > This makes it impossible to do something like the following: > > * Composite a video scene with OBS. > * Use that composed video as my webcam for my Zoom call. > > I think this would be cool, and I think that it would allow interesting > things to be done with video. > > # Prior Art > > The v4l2loopback project[1] provides a Linux kernel module to do just > this. This module provides a misc device node used to control the > loopback video devices. In particular, the misc device is used to create > and destroy loopback video devices. It's somewhat similar to the > /dev/loop-control dev node that is used to control block loop devices. > > There is also the vim2m test driver[2] in the kernel that appears to do > some similar things. This module is a module that is used for testing > the v4l2-mem2mem framework. This module has a more modern structure with > a platform device that is used as the context for the other devices. > It also uses the m2m framework that I think could benefit an in-kernel > loopback implementation. It is also important to note that this module > does not generally ship with distribution kernels (like in Fedora), > likely due to the "test" designation. No, since m2m are instance base, you cannot have two process operating on the same node (there is only 1 streaming state). You'd probably want to look at extending the VIMC driver instead. It's a virtual media controller, it's missing few bits to make loopback possible, but would be a good fit. Note that vivid have a loopback capability, but it does not constraint the width/height/format and will produce a test pattern if you request something that differ from the source. > > # Possible paths forward > > ## Integrate v4l2loopback into the kernel > > There is a lot to cleanup in the v4l2loopback code in order to get it > in shape for mainline inclusion. Having said that, this also a somewhat > tested implementation. There are even packages for using this project > from Ubuntu. It is very buggy, it often return a queued buffer back, and in fact, if you try to use the queue too much it fails. Though all this would probably be fine if you port it to VB2. > > ## Start with vim2m > > I think the vim2m.c could provide a really strong base on which to > start a loopback implementation for the kernel. I see two options: > > * Extend vim2m to include more loopback functionality, including > dynamic creation/deletion of video loopback devices. > * Copy the vim2m code to a new module and develop there. > > The extend option is attractive so that a new module mustn't be > developed. > > However, I am not sure that I want to complicate that module with > options that would required for a proper loopback implementation. > > # My thoughts on where to go next > > I think that the vim2m module provides a much nicer starting point for > an implementation of a mainline module. I think the the ideal solution > would start by using the vim2m source as a template and developing a > new module that incorporates the important functionality from > v4l2loopback. > > # Assumptions I have > > I have looked into the code, and the m2m framework appears to provide a > pair of vb2_queues that can be used for capture and output. I am assuming > that the framework would be reasonable to use for a loopback implementation. > > [1] https://github.com/umlaeute/v4l2loopback > [2] > > https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/drivers/media/test-drivers/vim2m.c?h=v5.11.2 > > Thanks, > wt > > On 3/2/21 10:03 AM, Wren Turkal wrote: > > Ezequiel, > > > > Submitting for upstream inclusion is a good idea, and the driver > > is licensed appropriately (gpl v2) for the kernel. I have asked > > the current maintainer for their thoughts. They are supportive > > of the idea, but they are in the middle of implementing an new > > feature that they would like to finish. I will work it up for > > inclusion when that work concludes since it's a pretty major change. > > > > Having said that, does the patch I posted make sense? I think > > that being able to enable the m2m framework without having to > > build in a m2m driver explicitly makes good sense. > > > > Also, thinking about this a little more, it might make more > > sense use use vim2m and v4l2loopback as inspiration and > > make something like loopback that looks more like vim2m. > > > > Things I like about vim2m vs v4l2loopback right now: > > * vim2m uses the m2m framework > > * vim2m is better organized by using a platform device > > to organize the devices that created under the context > > of the platform device. v4l2lb uses a misc device and > > doesn't explicitly place devices under in the context > > of that device. > > * The misc device created by v4l2lb is used to > > dynamically add and remove v4l2lb video devices. It > > It uses new IOCTLs to do that, which I wonder if there > > is a better way. > > * vim2m exposes config via a mechanism that doesn't > > require a new tool. v4l2lb requires a new control > > binary because of the custom ioctls. > > > > So, I have a couple questions: > > * Would it be better to use vim2m as a model to > > implement something like v4l2loopback? > > * If I were to do that should I copy vim2m to a > > new module, or should vim2m be extended. FWIW, > > vim2m appears to have two things against it: > > * vim2m is a test v4l2 driver. It would need > > to be changed to build as a non-test driver. > > This is a pretty easy Kconfig change and maybe > > moving files to a more appropriate place in the > > linux source tree. > > * v4l2loopback does have a number of features that > > vim2m does not have. I will say that I think > > that adding most importantant functionality > > (new formats and dynamic creation of loopback > > video devices in my mind) is probably very > > approachable and may give us a more modern > > linux module code. > > > > Give the above, I am kinda leaning toward extending > > vim2m code. So, do you have an opinion on the following? > > Should it be a new module, or should it literally extend > > and improve vim2m? > > > > wt > > > > > > On 3/1/21 6:11 PM, Ezequiel Garcia wrote: > > > Hi Wren, > > > > > > How about you submit the loopback driver for integration in mainline? > > > > > > We can review the driver, give feedback and help you get it merged. > > > > > > Ezequiel > > > > > > > > > On Mon, Mar 1, 2021, 5:32 PM Wren Turkal <wt@penguintechs.org > > > <mailto:wt@penguintechs.org>> wrote: > > > > > > Okay, so I have found how to enable the option to be manually > > > selected. > > > Would a patch like the following be acceptable? Or is there a better > > > way to do this? > > > > > > diff --git a/drivers/media/v4l2-core/Kconfig > > > b/drivers/media/v4l2-core/Kconfig > > > index bf49f83cb86f..25004137a982 100644 > > > --- a/drivers/media/v4l2-core/Kconfig > > > +++ b/drivers/media/v4l2-core/Kconfig > > > @@ -54,7 +54,7 @@ config V4L2_H264 > > > > > > # Used by drivers that need v4l2-mem2mem.ko > > > config V4L2_MEM2MEM_DEV > > > - tristate > > > + tristate "V4L2 mem2mem framework support" > > > depends on VIDEOBUF2_CORE > > > > > > # Used by LED subsystem flash drivers > > > > > > > > > Thanks, > > > wt > > > > > > On 2/28/21 2:07 PM, Wren Turkal wrote: > > > > Hi there linux-media experts, > > > > > > > > I am working on the out-of-tree v4l2loopback module[1]. My > > > > current goal is to implement the videobuf2 api. However, > > > > it appears that I must use a different vbuf2_queue for the > > > > OUTPUT and CAPTURE types. It appears that the mem2mem > > > > framework has the ability to do just this. I came to this > > > > conclusion while looking at the vim2m test module's code. > > > > While trying to modify the loopback module, I have added > > > > some code to try to call v4l2_m2m_init. You can see my very > > > > WIP attempt at code here[2]. > > > > > > > > I have installed the kernel-devel package on Fedora, which. > > > > Unfortunately, I get he following error messages when trying > > > > to compile: > > > > > > > > <errors> > > > > ERROR: modpost: "v4l2_m2m_release" > > > > [/home/wt/projects/kernel/v4l2loopback/v4l2loopback.ko] undefined! > > > > ERROR: modpost: "v4l2_m2m_init" > > > > [/home/wt/projects/kernel/v4l2loopback/v4l2loopback.ko] undefined! > > > > </errors> > > > > > > > > So, I looked into it, and Module.symvers and /proc/kallsyms > > > > don't have those symbols listed, so I think the Fedora kernel > > > > was not compiled with those symbols exported. > > > > > > > > Okay, so next I tried to build a custom kernel that does > > > > export those symbols. I managed to get it done by enabling > > > > the vim2m (VIDEO_VIM2M config option) module. The new > > > > Module.symvers includes the symbols; and the compile succeeds, > > > > which brings me to my final act. > > > > > > > > I need to expose those symbols without enabling the vim2m > > > > module since it is a testing module. I do not see a way > > > > to do that explicitly. The config option in question > > > > appears to be V4L2_MEM2MEM_DEV. And I haven't found a > > > > way to enable it without enabling something that implicitly > > > > selects it (which vim2m does). Do y'all have any suggestions > > > > for how to get these symbols exported properly when vim2m is > > > > not enabled? > > > > > > > > > > > > Thanks, > > > > Wren Turkal (wt) > > > > > > > > [1] https://github.com/umlaeute/v4l2loopback > > > <https://github.com/umlaeute/v4l2loopback> > > > > [2] https://github.com/wt/v4l2loopback/tree/m2m_try > > > <https://github.com/wt/v4l2loopback/tree/m2m_try> > > > > > > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: v4l2loopback and v4l2-mem2mem framework 2021-03-03 22:41 ` Nicolas Dufresne @ 2021-03-03 23:41 ` Wren Turkal 2021-03-04 9:58 ` Hans Verkuil 0 siblings, 1 reply; 11+ messages in thread From: Wren Turkal @ 2021-03-03 23:41 UTC (permalink / raw) To: Nicolas Dufresne, Ezequiel Garcia; +Cc: linux-media Questions below. On 3/3/21 2:41 PM, Nicolas Dufresne wrote: > No, since m2m are instance base, you cannot have two process operating on the > same node (there is only 1 streaming state). You'd probably want to look at > extending the VIMC driver instead. It's a virtual media controller, it's missing > few bits to make loopback possible, but would be a good fit. > > Note that vivid have a loopback capability, but it does not constraint the > width/height/format and will produce a test pattern if you request something > that differ from the source. I am not sure which you're suggesting to use: vivid or vimc. Also, are you suggesting to extend them or use them as a template for a new module? I think you are saying that vivid can already do loopback. However vivid is a test driver and, like vim2m, doesn't ship with distribution kernels. Should I try to move it out of the test-drivers? > It is very buggy, it often return a queued buffer back, and in fact, if you try > to use the queue too much it fails. Though all this would probably be fine if > you port it to VB2. I am assuming you are saying to port v4l2loopback to vb2. I am working on that right now, but I am not sure how to proceed since the vb2 needs a queue for capture and a queue for output (since they cannot be the same). I have created the queues, but I am not sure how to further integrate. Please check out the WIP here: https://github.com/wt/v4l2loopback/commit/e71b0a50427729a5e6e74443066751f7321dc404 If the m2m is the wrong approach, do you have any suggestions for how to proceed? Also, I am currently making a change for the linux kernel moving this code info drivers/media/v4l2-core/v4l2-loopback.c so that I can start iterating with your feedback. Is that a correct place to put this code? Thanks, wt ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: v4l2loopback and v4l2-mem2mem framework 2021-03-03 23:41 ` Wren Turkal @ 2021-03-04 9:58 ` Hans Verkuil 2021-03-04 22:17 ` Wren Turkal 0 siblings, 1 reply; 11+ messages in thread From: Hans Verkuil @ 2021-03-04 9:58 UTC (permalink / raw) To: Wren Turkal, Nicolas Dufresne, Ezequiel Garcia; +Cc: linux-media On 04/03/2021 00:41, Wren Turkal wrote: > Questions below. > > On 3/3/21 2:41 PM, Nicolas Dufresne wrote: >> No, since m2m are instance base, you cannot have two process operating on the >> same node (there is only 1 streaming state). You'd probably want to look at >> extending the VIMC driver instead. It's a virtual media controller, it's missing >> few bits to make loopback possible, but would be a good fit. >> >> Note that vivid have a loopback capability, but it does not constraint the >> width/height/format and will produce a test pattern if you request something >> that differ from the source. > > I am not sure which you're suggesting to use: vivid or vimc. Also, are you > suggesting to extend them or use them as a template for a new module? I don't think either vimc nor vivid are particularly suitable for this, although vivid comes close. You can certainly lift ideas from vivid. > I think you are saying that vivid can already do loopback. However vivid is > a test driver and, like vim2m, doesn't ship with distribution kernels. Should I > try to move it out of the test-drivers? > >> It is very buggy, it often return a queued buffer back, and in fact, if you try >> to use the queue too much it fails. Though all this would probably be fine if >> you port it to VB2. > > I am assuming you are saying to port v4l2loopback to vb2. I am working on that > right now, but I am not sure how to proceed since the vb2 needs a queue for > capture and a queue for output (since they cannot be the same). I have created > the queues, but I am not sure how to further integrate. Please check out the > WIP here: > https://github.com/wt/v4l2loopback/commit/e71b0a50427729a5e6e74443066751f7321dc404 > > If the m2m is the wrong approach, do you have any suggestions for how to > proceed? You need two video devices, one capture, one output. I see that the current loopback driver has just one video device and advertises it as a M2M device, but that's not what it should do. Creating two video devices (similar to what vivid does) is the best and most compatible approach. > > Also, I am currently making a change for the linux kernel moving this code > info drivers/media/v4l2-core/v4l2-loopback.c so that I can start iterating > with your feedback. Is that a correct place to put this code? No, v4l2-core is for core V4L2 frameworks, not for drivers. For now stick it in drivers/media/test-drivers. It will likely be moved again at some point (if we keep the 'misc' device that the current driver creates, then it might end up in drivers/media/misc). By far the most important thing you need to do is convert it to vb2. Since the intention is to upstream the driver I would start by doing a cleanup of the code, ditching all compat code with older kernels, making sure it adheres to the linux coding style (use 'scripts/checkpatch.pl --strict' for that), then split it in two video devices and use vb2. I actually wonder if it shouldn't be renamed to viloopback since it will be different from v4l2loopback. Anyway, that's something that can be done later. Regards, Hans > > Thanks, > wt > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: v4l2loopback and v4l2-mem2mem framework 2021-03-04 9:58 ` Hans Verkuil @ 2021-03-04 22:17 ` Wren Turkal 0 siblings, 0 replies; 11+ messages in thread From: Wren Turkal @ 2021-03-04 22:17 UTC (permalink / raw) To: Hans Verkuil, Nicolas Dufresne, Ezequiel Garcia; +Cc: linux-media On 3/4/21 1:58 AM, Hans Verkuil wrote: >> I am not sure which you're suggesting to use: vivid or vimc. Also, are you >> suggesting to extend them or use them as a template for a new module? > I don't think either vimc nor vivid are particularly suitable for this, although > vivid comes close. You can certainly lift ideas from vivid. Thanks for the advice. I'll definitely look at it more closely. >> I am assuming you are saying to port v4l2loopback to vb2. I am working on that >> right now, but I am not sure how to proceed since the vb2 needs a queue for >> capture and a queue for output (since they cannot be the same). I have created >> the queues, but I am not sure how to further integrate. Please check out the >> WIP here: >> https://github.com/wt/v4l2loopback/commit/e71b0a50427729a5e6e74443066751f7321dc404 >> >> If the m2m is the wrong approach, do you have any suggestions for how to >> proceed? > You need two video devices, one capture, one output. I see that the current > loopback driver has just one video device and advertises it as a M2M device, > but that's not what it should do. Creating two video devices (similar to what > vivid does) is the best and most compatible approach. FWIW, there is ongoing work to split the devices in v4l2loopback right now. >> Also, I am currently making a change for the linux kernel moving this code >> info drivers/media/v4l2-core/v4l2-loopback.c so that I can start iterating >> with your feedback. Is that a correct place to put this code? > No, v4l2-core is for core V4L2 frameworks, not for drivers. > > For now stick it in drivers/media/test-drivers. It will likely be moved again at > some point (if we keep the 'misc' device that the current driver creates, then > it might end up in drivers/media/misc). > > By far the most important thing you need to do is convert it to vb2. Since the > intention is to upstream the driver I would start by doing a cleanup of the code, > ditching all compat code with older kernels, making sure it adheres to the > linux coding style (use 'scripts/checkpatch.pl --strict' for that), then split > it in two video devices and use vb2. > > I actually wonder if it shouldn't be renamed to viloopback since it will be > different from v4l2loopback. Anyway, that's something that can be done later. Thanks for the advice. I'm happy to rename it to viloopback now if that makes sense. That is very little work. I am moving the code to test-drivers for now. Thanks for that feedback. wt ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2021-03-04 22:19 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-02-28 22:07 v4l2loopback and v4l2-mem2mem framework Wren Turkal
2021-03-01 20:27 ` Wren Turkal
2021-03-02 8:01 ` Ezequiel Garcia
2021-03-04 19:10 ` Mauro Carvalho Chehab
2021-03-04 19:34 ` Ezequiel Garcia
2021-03-04 22:19 ` Wren Turkal
[not found] ` <CAAEAJfCSfwxv5NyaH05Dfiw4wm6vGxL_ajoFup6r-GKx_VymJA@mail.gmail.com>
[not found] ` <86f1dd6b-7593-adba-b5a6-abf4f8e35d8b@penguintechs.org>
2021-03-02 22:08 ` Wren Turkal
2021-03-03 22:41 ` Nicolas Dufresne
2021-03-03 23:41 ` Wren Turkal
2021-03-04 9:58 ` Hans Verkuil
2021-03-04 22:17 ` Wren Turkal
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.