v4l2loopback and v4l2-mem2mem framework

v4l2loopback and v4l2-mem2mem framework

[Wren Turkal]

Hi there linux-media experts,

I am working on the out-of-tree v4l2loopback module[1]. My
current goal is to implement the videobuf2 api. However,
it appears that I must use a different vbuf2_queue for the
OUTPUT and CAPTURE types. It appears that the mem2mem
framework has the ability to do just this. I came to this
conclusion while looking at the vim2m test module's code.
While trying to modify the loopback module, I have added
some code to try to call v4l2_m2m_init. You can see my very
WIP attempt at code here[2].

I have installed the kernel-devel package on Fedora, which.
Unfortunately, I get he following error messages when trying
to compile:

<errors>
ERROR: modpost: "v4l2_m2m_release" 
[/home/wt/projects/kernel/v4l2loopback/v4l2loopback.ko] undefined!
ERROR: modpost: "v4l2_m2m_init" 
[/home/wt/projects/kernel/v4l2loopback/v4l2loopback.ko] undefined!
</errors>

So, I looked into it, and Module.symvers and /proc/kallsyms
don't have those symbols listed, so I think the Fedora kernel
was not compiled with those symbols exported.

Okay, so next I tried to build a custom kernel that does
export those symbols. I managed to get it done by enabling
the vim2m (VIDEO_VIM2M config option) module. The new
Module.symvers includes the symbols; and the compile succeeds,
which brings me to my final act.

I need to expose those symbols without enabling the vim2m
module since it is a testing module. I do not see a way
to do that explicitly. The config option in question
appears to be V4L2_MEM2MEM_DEV. And I haven't found a
way to enable it without enabling something that implicitly
selects it (which vim2m does). Do y'all have any suggestions
for how to get these symbols exported properly when vim2m is
not enabled?


Thanks,
Wren Turkal (wt)

[1] https://github.com/umlaeute/v4l2loopback
[2] https://github.com/wt/v4l2loopback/tree/m2m_try

Re: v4l2loopback and v4l2-mem2mem framework

[Wren Turkal]

Okay, so I have found how to enable the option to be manually selected.
Would a patch like the following be acceptable? Or is there a better
way to do this?

diff --git a/drivers/media/v4l2-core/Kconfig 
b/drivers/media/v4l2-core/Kconfig
index bf49f83cb86f..25004137a982 100644
--- a/drivers/media/v4l2-core/Kconfig
+++ b/drivers/media/v4l2-core/Kconfig
@@ -54,7 +54,7 @@ config V4L2_H264

  # Used by drivers that need v4l2-mem2mem.ko
  config V4L2_MEM2MEM_DEV
-       tristate
+       tristate "V4L2 mem2mem framework support"
         depends on VIDEOBUF2_CORE

  # Used by LED subsystem flash drivers


Thanks,
wt

On 2/28/21 2:07 PM, Wren Turkal wrote:
> Hi there linux-media experts,
>
> I am working on the out-of-tree v4l2loopback module[1]. My
> current goal is to implement the videobuf2 api. However,
> it appears that I must use a different vbuf2_queue for the
> OUTPUT and CAPTURE types. It appears that the mem2mem
> framework has the ability to do just this. I came to this
> conclusion while looking at the vim2m test module's code.
> While trying to modify the loopback module, I have added
> some code to try to call v4l2_m2m_init. You can see my very
> WIP attempt at code here[2].
>
> I have installed the kernel-devel package on Fedora, which.
> Unfortunately, I get he following error messages when trying
> to compile:
>
> <errors>
> ERROR: modpost: "v4l2_m2m_release" 
> [/home/wt/projects/kernel/v4l2loopback/v4l2loopback.ko] undefined!
> ERROR: modpost: "v4l2_m2m_init" 
> [/home/wt/projects/kernel/v4l2loopback/v4l2loopback.ko] undefined!
> </errors>
>
> So, I looked into it, and Module.symvers and /proc/kallsyms
> don't have those symbols listed, so I think the Fedora kernel
> was not compiled with those symbols exported.
>
> Okay, so next I tried to build a custom kernel that does
> export those symbols. I managed to get it done by enabling
> the vim2m (VIDEO_VIM2M config option) module. The new
> Module.symvers includes the symbols; and the compile succeeds,
> which brings me to my final act.
>
> I need to expose those symbols without enabling the vim2m
> module since it is a testing module. I do not see a way
> to do that explicitly. The config option in question
> appears to be V4L2_MEM2MEM_DEV. And I haven't found a
> way to enable it without enabling something that implicitly
> selects it (which vim2m does). Do y'all have any suggestions
> for how to get these symbols exported properly when vim2m is
> not enabled?
>
>
> Thanks,
> Wren Turkal (wt)
>
> [1] https://github.com/umlaeute/v4l2loopback
> [2] https://github.com/wt/v4l2loopback/tree/m2m_try

Re: v4l2loopback and v4l2-mem2mem framework

[Ezequiel Garcia]

Hi Wren,

How about you submit the loopback driver for integration in mainline?
We can review the driver, give feedback and help you get it merged.

Ezequiel

On Mon, 1 Mar 2021 at 17:32, Wren Turkal <wt@penguintechs.org> wrote:
>
> Okay, so I have found how to enable the option to be manually selected.
> Would a patch like the following be acceptable? Or is there a better
> way to do this?
>
> diff --git a/drivers/media/v4l2-core/Kconfig
> b/drivers/media/v4l2-core/Kconfig
> index bf49f83cb86f..25004137a982 100644
> --- a/drivers/media/v4l2-core/Kconfig
> +++ b/drivers/media/v4l2-core/Kconfig
> @@ -54,7 +54,7 @@ config V4L2_H264
>
>   # Used by drivers that need v4l2-mem2mem.ko
>   config V4L2_MEM2MEM_DEV
> -       tristate
> +       tristate "V4L2 mem2mem framework support"
>          depends on VIDEOBUF2_CORE
>
>   # Used by LED subsystem flash drivers
>
>
> Thanks,
> wt
>
> On 2/28/21 2:07 PM, Wren Turkal wrote:
> > Hi there linux-media experts,
> >
> > I am working on the out-of-tree v4l2loopback module[1]. My
> > current goal is to implement the videobuf2 api. However,
> > it appears that I must use a different vbuf2_queue for the
> > OUTPUT and CAPTURE types. It appears that the mem2mem
> > framework has the ability to do just this. I came to this
> > conclusion while looking at the vim2m test module's code.
> > While trying to modify the loopback module, I have added
> > some code to try to call v4l2_m2m_init. You can see my very
> > WIP attempt at code here[2].
> >
> > I have installed the kernel-devel package on Fedora, which.
> > Unfortunately, I get he following error messages when trying
> > to compile:
> >
> > <errors>
> > ERROR: modpost: "v4l2_m2m_release"
> > [/home/wt/projects/kernel/v4l2loopback/v4l2loopback.ko] undefined!
> > ERROR: modpost: "v4l2_m2m_init"
> > [/home/wt/projects/kernel/v4l2loopback/v4l2loopback.ko] undefined!
> > </errors>
> >
> > So, I looked into it, and Module.symvers and /proc/kallsyms
> > don't have those symbols listed, so I think the Fedora kernel
> > was not compiled with those symbols exported.
> >
> > Okay, so next I tried to build a custom kernel that does
> > export those symbols. I managed to get it done by enabling
> > the vim2m (VIDEO_VIM2M config option) module. The new
> > Module.symvers includes the symbols; and the compile succeeds,
> > which brings me to my final act.
> >
> > I need to expose those symbols without enabling the vim2m
> > module since it is a testing module. I do not see a way
> > to do that explicitly. The config option in question
> > appears to be V4L2_MEM2MEM_DEV. And I haven't found a
> > way to enable it without enabling something that implicitly
> > selects it (which vim2m does). Do y'all have any suggestions
> > for how to get these symbols exported properly when vim2m is
> > not enabled?
> >
> >
> > Thanks,
> > Wren Turkal (wt)
> >
> > [1] https://github.com/umlaeute/v4l2loopback
> > [2] https://github.com/wt/v4l2loopback/tree/m2m_try
>

Re: v4l2loopback and v4l2-mem2mem framework

[Wren Turkal]

Okay, so I tried to write up a little bit of an RFC to organize my
thoughts. I would love to get your thoughts:

RFC: A loopback device for the v4l2 subsystem.

# What is needed?

The common denominator for apps that use webcams in linux is the
v4l2 subsystem. The v4l2 subsystem could benefit from a loopback
device that allows one to compose custom video streams and present
them in a normal way from a /dev/video* device.

# Why is it needed?

The v4l2 is the framework is the kernel interface used for supporting
video output and video capture. This brings the webcam video to apps
like Cheese or a web browser or OBS.

However, there is no way to customize the composition of the video
stream before capture on these apps. Of course, apps like Cheese have
filters, but this is not something that is composible with other apps.
This makes it impossible to do something like the following:

* Composite a video scene with OBS.
* Use that composed video as my webcam for my Zoom call.

I think this would be cool, and I think that it would allow interesting
things to be done with video.

# Prior Art

The v4l2loopback project[1] provides a Linux kernel module to do just
this. This module provides a misc device node used to control the
loopback video devices. In particular, the misc device is used to create
and destroy loopback video devices. It's somewhat similar to the
/dev/loop-control dev node that is used to control block loop devices.

There is also the vim2m test driver[2] in the kernel that appears to do
some similar things. This module is a module that is used for testing
the v4l2-mem2mem framework. This module has a more modern structure with
a platform device that is used as the context for the other devices.
It also uses the m2m framework that I think could benefit an in-kernel
loopback implementation. It is also important to note that this module
does not generally ship with distribution kernels (like in Fedora),
likely due to the "test" designation.

# Possible paths forward

## Integrate v4l2loopback into the kernel

There is a lot to cleanup in the v4l2loopback code in order to get it
in shape for mainline inclusion. Having said that, this also a somewhat
tested implementation. There are even packages for using this project
from Ubuntu.

## Start with vim2m

I think the vim2m.c could provide a really strong base on which to
start a loopback implementation for the kernel. I see two options:

* Extend vim2m to include more loopback functionality, including
   dynamic creation/deletion of video loopback devices.
* Copy the vim2m code to a new module and develop there.

The extend option is attractive so that a new module mustn't be
developed.

However, I am not sure that I want to complicate that module with
options that would required for a proper loopback implementation.

# My thoughts on where to go next

I think that the vim2m module provides a much nicer starting point for
an implementation of a mainline module. I think the the ideal solution
would start by using the vim2m source as a template and developing a
new module that incorporates the important functionality from
v4l2loopback.

# Assumptions I have

I have looked into the code, and the m2m framework appears to provide a
pair of vb2_queues that can be used for capture and output. I am assuming
that the framework would be reasonable to use for a loopback implementation.

[1] https://github.com/umlaeute/v4l2loopback
[2] 
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/drivers/media/test-drivers/vim2m.c?h=v5.11.2

Thanks,
wt

On 3/2/21 10:03 AM, Wren Turkal wrote:
> Ezequiel,
>
> Submitting for upstream inclusion is a good idea, and the driver
> is licensed appropriately (gpl v2) for the kernel. I have asked
> the current maintainer for their thoughts. They are supportive
> of the idea, but they are in the middle of implementing an new
> feature that they would like to finish. I will work it up for
> inclusion when that work concludes since it's a pretty major change.
>
> Having said that, does the patch I posted make sense? I think
> that being able to enable the m2m framework without having to
> build in a m2m driver explicitly makes good sense.
>
> Also, thinking about this a little more, it might make more
> sense use use vim2m and v4l2loopback as inspiration and
> make something like loopback that looks more like vim2m.
>
> Things I like about vim2m vs v4l2loopback right now:
> * vim2m uses the m2m framework
> * vim2m is better organized by using a platform device
>   to organize the devices that created under the context
>   of the platform device. v4l2lb uses a misc device and
>   doesn't explicitly place devices under in the context
>   of that device.
> * The misc device created by v4l2lb is used to
>   dynamically add and remove v4l2lb video devices. It
>   It uses new IOCTLs to do that, which I wonder if there
>   is a better way.
> * vim2m exposes config via a mechanism that doesn't
>   require a new tool. v4l2lb requires a new control
>   binary because of the custom ioctls.
>
> So, I have a couple questions:
> * Would it be better to use vim2m as a model to
>   implement something like v4l2loopback?
> * If I were to do that should I copy vim2m to a
>   new module, or should vim2m be extended. FWIW,
>   vim2m appears to have two things against it:
>   * vim2m is a test v4l2 driver. It would need
>     to be changed to build as a non-test driver.
>     This is a pretty easy Kconfig change and maybe
>     moving files to a more appropriate place in the
>     linux source tree.
>   * v4l2loopback does have a number of features that
>     vim2m does not have. I will say that I think
>     that adding most importantant functionality
>     (new formats and dynamic creation of loopback
>     video devices in my mind) is probably very
>     approachable and may give us a more modern
>     linux module code.
>
> Give the above, I am kinda leaning toward extending
> vim2m code. So, do you have an opinion on the following?
> Should it be a new module, or should it literally extend
> and improve vim2m?
>
> wt
>
>
> On 3/1/21 6:11 PM, Ezequiel Garcia wrote:
>> Hi Wren,
>>
>> How about you submit the loopback driver for integration in mainline?
>>
>> We can review the driver, give feedback and help you get it merged.
>>
>> Ezequiel
>>
>>
>> On Mon, Mar 1, 2021, 5:32 PM Wren Turkal <wt@penguintechs.org 
>> <mailto:wt@penguintechs.org>> wrote:
>>
>>     Okay, so I have found how to enable the option to be manually
>>     selected.
>>     Would a patch like the following be acceptable? Or is there a better
>>     way to do this?
>>
>>     diff --git a/drivers/media/v4l2-core/Kconfig
>>     b/drivers/media/v4l2-core/Kconfig
>>     index bf49f83cb86f..25004137a982 100644
>>     --- a/drivers/media/v4l2-core/Kconfig
>>     +++ b/drivers/media/v4l2-core/Kconfig
>>     @@ -54,7 +54,7 @@ config V4L2_H264
>>
>>       # Used by drivers that need v4l2-mem2mem.ko
>>       config V4L2_MEM2MEM_DEV
>>     -       tristate
>>     +       tristate "V4L2 mem2mem framework support"
>>              depends on VIDEOBUF2_CORE
>>
>>       # Used by LED subsystem flash drivers
>>
>>
>>     Thanks,
>>     wt
>>
>>     On 2/28/21 2:07 PM, Wren Turkal wrote:
>>     > Hi there linux-media experts,
>>     >
>>     > I am working on the out-of-tree v4l2loopback module[1]. My
>>     > current goal is to implement the videobuf2 api. However,
>>     > it appears that I must use a different vbuf2_queue for the
>>     > OUTPUT and CAPTURE types. It appears that the mem2mem
>>     > framework has the ability to do just this. I came to this
>>     > conclusion while looking at the vim2m test module's code.
>>     > While trying to modify the loopback module, I have added
>>     > some code to try to call v4l2_m2m_init. You can see my very
>>     > WIP attempt at code here[2].
>>     >
>>     > I have installed the kernel-devel package on Fedora, which.
>>     > Unfortunately, I get he following error messages when trying
>>     > to compile:
>>     >
>>     > <errors>
>>     > ERROR: modpost: "v4l2_m2m_release"
>>     > [/home/wt/projects/kernel/v4l2loopback/v4l2loopback.ko] undefined!
>>     > ERROR: modpost: "v4l2_m2m_init"
>>     > [/home/wt/projects/kernel/v4l2loopback/v4l2loopback.ko] undefined!
>>     > </errors>
>>     >
>>     > So, I looked into it, and Module.symvers and /proc/kallsyms
>>     > don't have those symbols listed, so I think the Fedora kernel
>>     > was not compiled with those symbols exported.
>>     >
>>     > Okay, so next I tried to build a custom kernel that does
>>     > export those symbols. I managed to get it done by enabling
>>     > the vim2m (VIDEO_VIM2M config option) module. The new
>>     > Module.symvers includes the symbols; and the compile succeeds,
>>     > which brings me to my final act.
>>     >
>>     > I need to expose those symbols without enabling the vim2m
>>     > module since it is a testing module. I do not see a way
>>     > to do that explicitly. The config option in question
>>     > appears to be V4L2_MEM2MEM_DEV. And I haven't found a
>>     > way to enable it without enabling something that implicitly
>>     > selects it (which vim2m does). Do y'all have any suggestions
>>     > for how to get these symbols exported properly when vim2m is
>>     > not enabled?
>>     >
>>     >
>>     > Thanks,
>>     > Wren Turkal (wt)
>>     >
>>     > [1] https://github.com/umlaeute/v4l2loopback
>>     <https://github.com/umlaeute/v4l2loopback>
>>     > [2] https://github.com/wt/v4l2loopback/tree/m2m_try
>>     <https://github.com/wt/v4l2loopback/tree/m2m_try>
>>
>

Re: v4l2loopback and v4l2-mem2mem framework

[Nicolas Dufresne]

Le mardi 02 mars 2021 à 14:08 -0800, Wren Turkal a écrit :
> Okay, so I tried to write up a little bit of an RFC to organize my
> thoughts. I would love to get your thoughts:
> 
> RFC: A loopback device for the v4l2 subsystem.
> 
> # What is needed?
> 
> The common denominator for apps that use webcams in linux is the
> v4l2 subsystem. The v4l2 subsystem could benefit from a loopback
> device that allows one to compose custom video streams and present
> them in a normal way from a /dev/video* device.
> 
> # Why is it needed?
> 
> The v4l2 is the framework is the kernel interface used for supporting
> video output and video capture. This brings the webcam video to apps
> like Cheese or a web browser or OBS.
> 
> However, there is no way to customize the composition of the video
> stream before capture on these apps. Of course, apps like Cheese have
> filters, but this is not something that is composible with other apps.
> This makes it impossible to do something like the following:
> 
> * Composite a video scene with OBS.
> * Use that composed video as my webcam for my Zoom call.
> 
> I think this would be cool, and I think that it would allow interesting
> things to be done with video.
> 
> # Prior Art
> 
> The v4l2loopback project[1] provides a Linux kernel module to do just
> this. This module provides a misc device node used to control the
> loopback video devices. In particular, the misc device is used to create
> and destroy loopback video devices. It's somewhat similar to the
> /dev/loop-control dev node that is used to control block loop devices.
> 
> There is also the vim2m test driver[2] in the kernel that appears to do
> some similar things. This module is a module that is used for testing
> the v4l2-mem2mem framework. This module has a more modern structure with
> a platform device that is used as the context for the other devices.
> It also uses the m2m framework that I think could benefit an in-kernel
> loopback implementation. It is also important to note that this module
> does not generally ship with distribution kernels (like in Fedora),
> likely due to the "test" designation.

No, since m2m are instance base, you cannot have two process operating on the
same node (there is only 1 streaming state). You'd probably want to look at
extending the VIMC driver instead. It's a virtual media controller, it's missing
few bits to make loopback possible, but would be a good fit.

Note that vivid have a loopback capability, but it does not constraint the
width/height/format and will produce a test pattern if you request something
that differ from the source.

> 
> # Possible paths forward
> 
> ## Integrate v4l2loopback into the kernel
> 
> There is a lot to cleanup in the v4l2loopback code in order to get it
> in shape for mainline inclusion. Having said that, this also a somewhat
> tested implementation. There are even packages for using this project
> from Ubuntu.

It is very buggy, it often return a queued buffer back, and in fact, if you try
to use the queue too much it fails. Though all this would probably be fine if
you port it to VB2.

> 
> ## Start with vim2m
> 
> I think the vim2m.c could provide a really strong base on which to
> start a loopback implementation for the kernel. I see two options:
> 
> * Extend vim2m to include more loopback functionality, including
>    dynamic creation/deletion of video loopback devices.
> * Copy the vim2m code to a new module and develop there.
> 
> The extend option is attractive so that a new module mustn't be
> developed.
> 
> However, I am not sure that I want to complicate that module with
> options that would required for a proper loopback implementation.
> 
> # My thoughts on where to go next
> 
> I think that the vim2m module provides a much nicer starting point for
> an implementation of a mainline module. I think the the ideal solution
> would start by using the vim2m source as a template and developing a
> new module that incorporates the important functionality from
> v4l2loopback.
> 
> # Assumptions I have
> 
> I have looked into the code, and the m2m framework appears to provide a
> pair of vb2_queues that can be used for capture and output. I am assuming
> that the framework would be reasonable to use for a loopback implementation.
> 
> [1] https://github.com/umlaeute/v4l2loopback
> [2] 
>   
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/drivers/media/test-drivers/vim2m.c?h=v5.11.2
> 
> Thanks,
> wt
> 
> On 3/2/21 10:03 AM, Wren Turkal wrote:
> > Ezequiel,
> > 
> > Submitting for upstream inclusion is a good idea, and the driver
> > is licensed appropriately (gpl v2) for the kernel. I have asked
> > the current maintainer for their thoughts. They are supportive
> > of the idea, but they are in the middle of implementing an new
> > feature that they would like to finish. I will work it up for
> > inclusion when that work concludes since it's a pretty major change.
> > 
> > Having said that, does the patch I posted make sense? I think
> > that being able to enable the m2m framework without having to
> > build in a m2m driver explicitly makes good sense.
> > 
> > Also, thinking about this a little more, it might make more
> > sense use use vim2m and v4l2loopback as inspiration and
> > make something like loopback that looks more like vim2m.
> > 
> > Things I like about vim2m vs v4l2loopback right now:
> > * vim2m uses the m2m framework
> > * vim2m is better organized by using a platform device
> >   to organize the devices that created under the context
> >   of the platform device. v4l2lb uses a misc device and
> >   doesn't explicitly place devices under in the context
> >   of that device.
> > * The misc device created by v4l2lb is used to
> >   dynamically add and remove v4l2lb video devices. It
> >   It uses new IOCTLs to do that, which I wonder if there
> >   is a better way.
> > * vim2m exposes config via a mechanism that doesn't
> >   require a new tool. v4l2lb requires a new control
> >   binary because of the custom ioctls.
> > 
> > So, I have a couple questions:
> > * Would it be better to use vim2m as a model to
> >   implement something like v4l2loopback?
> > * If I were to do that should I copy vim2m to a
> >   new module, or should vim2m be extended. FWIW,
> >   vim2m appears to have two things against it:
> >   * vim2m is a test v4l2 driver. It would need
> >     to be changed to build as a non-test driver.
> >     This is a pretty easy Kconfig change and maybe
> >     moving files to a more appropriate place in the
> >     linux source tree.
> >   * v4l2loopback does have a number of features that
> >     vim2m does not have. I will say that I think
> >     that adding most importantant functionality
> >     (new formats and dynamic creation of loopback
> >     video devices in my mind) is probably very
> >     approachable and may give us a more modern
> >     linux module code.
> > 
> > Give the above, I am kinda leaning toward extending
> > vim2m code. So, do you have an opinion on the following?
> > Should it be a new module, or should it literally extend
> > and improve vim2m?
> > 
> > wt
> > 
> > 
> > On 3/1/21 6:11 PM, Ezequiel Garcia wrote:
> > > Hi Wren,
> > > 
> > > How about you submit the loopback driver for integration in mainline?
> > > 
> > > We can review the driver, give feedback and help you get it merged.
> > > 
> > > Ezequiel
> > > 
> > > 
> > > On Mon, Mar 1, 2021, 5:32 PM Wren Turkal <wt@penguintechs.org 
> > > <mailto:wt@penguintechs.org>> wrote:
> > > 
> > >     Okay, so I have found how to enable the option to be manually
> > >     selected.
> > >     Would a patch like the following be acceptable? Or is there a better
> > >     way to do this?
> > > 
> > >     diff --git a/drivers/media/v4l2-core/Kconfig
> > >     b/drivers/media/v4l2-core/Kconfig
> > >     index bf49f83cb86f..25004137a982 100644
> > >     --- a/drivers/media/v4l2-core/Kconfig
> > >     +++ b/drivers/media/v4l2-core/Kconfig
> > >     @@ -54,7 +54,7 @@ config V4L2_H264
> > > 
> > >       # Used by drivers that need v4l2-mem2mem.ko
> > >       config V4L2_MEM2MEM_DEV
> > >     -       tristate
> > >     +       tristate "V4L2 mem2mem framework support"
> > >              depends on VIDEOBUF2_CORE
> > > 
> > >       # Used by LED subsystem flash drivers
> > > 
> > > 
> > >     Thanks,
> > >     wt
> > > 
> > >     On 2/28/21 2:07 PM, Wren Turkal wrote:
> > >     > Hi there linux-media experts,
> > >     >
> > >     > I am working on the out-of-tree v4l2loopback module[1]. My
> > >     > current goal is to implement the videobuf2 api. However,
> > >     > it appears that I must use a different vbuf2_queue for the
> > >     > OUTPUT and CAPTURE types. It appears that the mem2mem
> > >     > framework has the ability to do just this. I came to this
> > >     > conclusion while looking at the vim2m test module's code.
> > >     > While trying to modify the loopback module, I have added
> > >     > some code to try to call v4l2_m2m_init. You can see my very
> > >     > WIP attempt at code here[2].
> > >     >
> > >     > I have installed the kernel-devel package on Fedora, which.
> > >     > Unfortunately, I get he following error messages when trying
> > >     > to compile:
> > >     >
> > >     > <errors>
> > >     > ERROR: modpost: "v4l2_m2m_release"
> > >     > [/home/wt/projects/kernel/v4l2loopback/v4l2loopback.ko] undefined!
> > >     > ERROR: modpost: "v4l2_m2m_init"
> > >     > [/home/wt/projects/kernel/v4l2loopback/v4l2loopback.ko] undefined!
> > >     > </errors>
> > >     >
> > >     > So, I looked into it, and Module.symvers and /proc/kallsyms
> > >     > don't have those symbols listed, so I think the Fedora kernel
> > >     > was not compiled with those symbols exported.
> > >     >
> > >     > Okay, so next I tried to build a custom kernel that does
> > >     > export those symbols. I managed to get it done by enabling
> > >     > the vim2m (VIDEO_VIM2M config option) module. The new
> > >     > Module.symvers includes the symbols; and the compile succeeds,
> > >     > which brings me to my final act.
> > >     >
> > >     > I need to expose those symbols without enabling the vim2m
> > >     > module since it is a testing module. I do not see a way
> > >     > to do that explicitly. The config option in question
> > >     > appears to be V4L2_MEM2MEM_DEV. And I haven't found a
> > >     > way to enable it without enabling something that implicitly
> > >     > selects it (which vim2m does). Do y'all have any suggestions
> > >     > for how to get these symbols exported properly when vim2m is
> > >     > not enabled?
> > >     >
> > >     >
> > >     > Thanks,
> > >     > Wren Turkal (wt)
> > >     >
> > >     > [1] https://github.com/umlaeute/v4l2loopback
> > >     <https://github.com/umlaeute/v4l2loopback>
> > >     > [2] https://github.com/wt/v4l2loopback/tree/m2m_try
> > >     <https://github.com/wt/v4l2loopback/tree/m2m_try>
> > > 
> > 
> 

Re: v4l2loopback and v4l2-mem2mem framework

[Wren Turkal]

Questions below.

On 3/3/21 2:41 PM, Nicolas Dufresne wrote:
> No, since m2m are instance base, you cannot have two process operating on the
> same node (there is only 1 streaming state). You'd probably want to look at
> extending the VIMC driver instead. It's a virtual media controller, it's missing
> few bits to make loopback possible, but would be a good fit.
>
> Note that vivid have a loopback capability, but it does not constraint the
> width/height/format and will produce a test pattern if you request something
> that differ from the source.

I am not sure which you're suggesting to use: vivid or vimc. Also, are you
suggesting to extend them or use them as a template for a new module?

I think you are saying that vivid can already do loopback. However vivid is
a test driver and, like vim2m, doesn't ship with distribution kernels. Should I
try to move it out of the test-drivers?

> It is very buggy, it often return a queued buffer back, and in fact, if you try
> to use the queue too much it fails. Though all this would probably be fine if
> you port it to VB2.

I am assuming you are saying to port v4l2loopback to vb2. I am working on that
right now, but I am not sure how to proceed since the vb2 needs a queue for
capture and a queue for output (since they cannot be the same). I have created
the queues, but I am not sure how to further integrate. Please check out the
WIP here:
https://github.com/wt/v4l2loopback/commit/e71b0a50427729a5e6e74443066751f7321dc404

If the m2m is the wrong approach, do you have any suggestions for how to
proceed?

Also, I am currently making a change for the linux kernel moving this code
info drivers/media/v4l2-core/v4l2-loopback.c so that I can start iterating
with your feedback. Is that a correct place to put this code?

Thanks,
wt

Re: v4l2loopback and v4l2-mem2mem framework

[Hans Verkuil]

On 04/03/2021 00:41, Wren Turkal wrote:
> Questions below.
> 
> On 3/3/21 2:41 PM, Nicolas Dufresne wrote:
>> No, since m2m are instance base, you cannot have two process operating on the
>> same node (there is only 1 streaming state). You'd probably want to look at
>> extending the VIMC driver instead. It's a virtual media controller, it's missing
>> few bits to make loopback possible, but would be a good fit.
>>
>> Note that vivid have a loopback capability, but it does not constraint the
>> width/height/format and will produce a test pattern if you request something
>> that differ from the source.
> 
> I am not sure which you're suggesting to use: vivid or vimc. Also, are you
> suggesting to extend them or use them as a template for a new module?

I don't think either vimc nor vivid are particularly suitable for this, although
vivid comes close. You can certainly lift ideas from vivid.

> I think you are saying that vivid can already do loopback. However vivid is
> a test driver and, like vim2m, doesn't ship with distribution kernels. Should I
> try to move it out of the test-drivers?
> 
>> It is very buggy, it often return a queued buffer back, and in fact, if you try
>> to use the queue too much it fails. Though all this would probably be fine if
>> you port it to VB2.
> 
> I am assuming you are saying to port v4l2loopback to vb2. I am working on that
> right now, but I am not sure how to proceed since the vb2 needs a queue for
> capture and a queue for output (since they cannot be the same). I have created
> the queues, but I am not sure how to further integrate. Please check out the
> WIP here:
> https://github.com/wt/v4l2loopback/commit/e71b0a50427729a5e6e74443066751f7321dc404
> 
> If the m2m is the wrong approach, do you have any suggestions for how to
> proceed?

You need two video devices, one capture, one output. I see that the current
loopback driver has just one video device and advertises it as a M2M device,
but that's not what it should do. Creating two video devices (similar to what
vivid does) is the best and most compatible approach.

> 
> Also, I am currently making a change for the linux kernel moving this code
> info drivers/media/v4l2-core/v4l2-loopback.c so that I can start iterating
> with your feedback. Is that a correct place to put this code?

No, v4l2-core is for core V4L2 frameworks, not for drivers.

For now stick it in drivers/media/test-drivers. It will likely be moved again at
some point (if we keep the 'misc' device that the current driver creates, then
it might end up in drivers/media/misc).

By far the most important thing you need to do is convert it to vb2. Since the
intention is to upstream the driver I would start by doing a cleanup of the code,
ditching all compat code with older kernels, making sure it adheres to the
linux coding style (use 'scripts/checkpatch.pl --strict' for that), then split
it in two video devices and use vb2.

I actually wonder if it shouldn't be renamed to viloopback since it will be
different from v4l2loopback. Anyway, that's something that can be done later.

Regards,

	Hans

> 
> Thanks,
> wt
> 

Re: v4l2loopback and v4l2-mem2mem framework

[Mauro Carvalho Chehab]

Em Tue, 2 Mar 2021 05:01:16 -0300
Ezequiel Garcia <ezequiel@vanguardiasur.com.ar> escreveu:

> Hi Wren,
> 
> How about you submit the loopback driver for integration in mainline?
> We can review the driver, give feedback and help you get it merged.

Please, don't top post.

The loopback driver is not something that could be merged. It was
already submitted a few times.

One of the main issues is that userspace may sleep, and this could cause
some bad effects and even hangups due to V4L2 core and VB2 mutex and
spin locks.

Regards,
Mauro

> 
> Ezequiel
> 
> On Mon, 1 Mar 2021 at 17:32, Wren Turkal <wt@penguintechs.org> wrote:
> >
> > Okay, so I have found how to enable the option to be manually selected.
> > Would a patch like the following be acceptable? Or is there a better
> > way to do this?
> >
> > diff --git a/drivers/media/v4l2-core/Kconfig
> > b/drivers/media/v4l2-core/Kconfig
> > index bf49f83cb86f..25004137a982 100644
> > --- a/drivers/media/v4l2-core/Kconfig
> > +++ b/drivers/media/v4l2-core/Kconfig
> > @@ -54,7 +54,7 @@ config V4L2_H264
> >
> >   # Used by drivers that need v4l2-mem2mem.ko
> >   config V4L2_MEM2MEM_DEV
> > -       tristate
> > +       tristate "V4L2 mem2mem framework support"
> >          depends on VIDEOBUF2_CORE
> >
> >   # Used by LED subsystem flash drivers
> >
> >
> > Thanks,
> > wt
> >
> > On 2/28/21 2:07 PM, Wren Turkal wrote:  
> > > Hi there linux-media experts,
> > >
> > > I am working on the out-of-tree v4l2loopback module[1]. My
> > > current goal is to implement the videobuf2 api. However,
> > > it appears that I must use a different vbuf2_queue for the
> > > OUTPUT and CAPTURE types. It appears that the mem2mem
> > > framework has the ability to do just this. I came to this
> > > conclusion while looking at the vim2m test module's code.
> > > While trying to modify the loopback module, I have added
> > > some code to try to call v4l2_m2m_init. You can see my very
> > > WIP attempt at code here[2].
> > >
> > > I have installed the kernel-devel package on Fedora, which.
> > > Unfortunately, I get he following error messages when trying
> > > to compile:
> > >
> > > <errors>
> > > ERROR: modpost: "v4l2_m2m_release"
> > > [/home/wt/projects/kernel/v4l2loopback/v4l2loopback.ko] undefined!
> > > ERROR: modpost: "v4l2_m2m_init"
> > > [/home/wt/projects/kernel/v4l2loopback/v4l2loopback.ko] undefined!
> > > </errors>
> > >
> > > So, I looked into it, and Module.symvers and /proc/kallsyms
> > > don't have those symbols listed, so I think the Fedora kernel
> > > was not compiled with those symbols exported.
> > >
> > > Okay, so next I tried to build a custom kernel that does
> > > export those symbols. I managed to get it done by enabling
> > > the vim2m (VIDEO_VIM2M config option) module. The new
> > > Module.symvers includes the symbols; and the compile succeeds,
> > > which brings me to my final act.
> > >
> > > I need to expose those symbols without enabling the vim2m
> > > module since it is a testing module. I do not see a way
> > > to do that explicitly. The config option in question
> > > appears to be V4L2_MEM2MEM_DEV. And I haven't found a
> > > way to enable it without enabling something that implicitly
> > > selects it (which vim2m does). Do y'all have any suggestions
> > > for how to get these symbols exported properly when vim2m is
> > > not enabled?
> > >
> > >
> > > Thanks,
> > > Wren Turkal (wt)
> > >
> > > [1] https://github.com/umlaeute/v4l2loopback
> > > [2] https://github.com/wt/v4l2loopback/tree/m2m_try  
> >  



Thanks,
Mauro

Re: v4l2loopback and v4l2-mem2mem framework

[Ezequiel Garcia]

On Thu, 4 Mar 2021 at 16:11, Mauro Carvalho Chehab
<mchehab+huawei@kernel.org> wrote:
>
> Em Tue, 2 Mar 2021 05:01:16 -0300
> Ezequiel Garcia <ezequiel@vanguardiasur.com.ar> escreveu:
>
> > Hi Wren,
> >
> > How about you submit the loopback driver for integration in mainline?
> > We can review the driver, give feedback and help you get it merged.
>
> Please, don't top post.
>
> The loopback driver is not something that could be merged. It was
> already submitted a few times.
>
> One of the main issues is that userspace may sleep, and this could cause
> some bad effects and even hangups due to V4L2 core and VB2 mutex and
> spin locks.
>

Can you elaborate on this?

I always felt that it's a shame for something so popular as the loopback
to be out-of-tree and unsupported.

Thanks,
Ezequiel

Re: v4l2loopback and v4l2-mem2mem framework

[Wren Turkal]

On 3/4/21 1:58 AM, Hans Verkuil wrote:
>> I am not sure which you're suggesting to use: vivid or vimc. Also, are you
>> suggesting to extend them or use them as a template for a new module?
> I don't think either vimc nor vivid are particularly suitable for this, although
> vivid comes close. You can certainly lift ideas from vivid.

Thanks for the advice. I'll definitely look at it more closely.

>> I am assuming you are saying to port v4l2loopback to vb2. I am working on that
>> right now, but I am not sure how to proceed since the vb2 needs a queue for
>> capture and a queue for output (since they cannot be the same). I have created
>> the queues, but I am not sure how to further integrate. Please check out the
>> WIP here:
>> https://github.com/wt/v4l2loopback/commit/e71b0a50427729a5e6e74443066751f7321dc404
>>
>> If the m2m is the wrong approach, do you have any suggestions for how to
>> proceed?
> You need two video devices, one capture, one output. I see that the current
> loopback driver has just one video device and advertises it as a M2M device,
> but that's not what it should do. Creating two video devices (similar to what
> vivid does) is the best and most compatible approach.
FWIW, there is ongoing work to split the devices in v4l2loopback right now.
>> Also, I am currently making a change for the linux kernel moving this code
>> info drivers/media/v4l2-core/v4l2-loopback.c so that I can start iterating
>> with your feedback. Is that a correct place to put this code?
> No, v4l2-core is for core V4L2 frameworks, not for drivers.
>
> For now stick it in drivers/media/test-drivers. It will likely be moved again at
> some point (if we keep the 'misc' device that the current driver creates, then
> it might end up in drivers/media/misc).
>
> By far the most important thing you need to do is convert it to vb2. Since the
> intention is to upstream the driver I would start by doing a cleanup of the code,
> ditching all compat code with older kernels, making sure it adheres to the
> linux coding style (use 'scripts/checkpatch.pl --strict' for that), then split
> it in two video devices and use vb2.
>
> I actually wonder if it shouldn't be renamed to viloopback since it will be
> different from v4l2loopback. Anyway, that's something that can be done later.

Thanks for the advice. I'm happy to rename it to viloopback now if that makes sense.
That is very little work.

I am moving the code to test-drivers for now. Thanks for that feedback.

wt

Re: v4l2loopback and v4l2-mem2mem framework

[Wren Turkal]

On 3/4/21 11:10 AM, Mauro Carvalho Chehab wrote:
> Please, don't top post.
Sorry about the top post.
> The loopback driver is not something that could be merged. It was
> already submitted a few times.

I'm willing to develop something new if you think that would be a better approach.
I just want the functionality of the loopback so that OBS (or something else) can
be used to compose a video scene for use by all the apps on Linux.

> One of the main issues is that userspace may sleep, and this could cause
> some bad effects and even hangups due to V4L2 core and VB2 mutex and
> spin locks.

Is this something that could be fixed? If so, I am happy to try to help do that.

wt