Agenda for the Edinburgh mini-summit

Agenda for the Edinburgh mini-summit

[Hans Verkuil]

OK, I know, we don't even know yet when the mini-summit will be held but I thought
I'd just start this thread to collect input for the agenda.

I have these topics (and I *know* that I am forgetting a few):

- Discuss ideas/use-cases for a property-based API. An initial discussion
  appeared in this thread:

  http://permalink.gmane.org/gmane.linux.drivers.video-input-infrastructure/65195

- What is needed to share i2c video transmitters between drm and v4l? Hopefully
  we will know more after the upcoming LPC.

- Decide on how v4l2 support libraries should be organized. There is code for
  handling raw-to-sliced VBI decoding, ALSA looping, finding associated
  video/alsa nodes and for TV frequency tables. We should decide how that should
  be organized into libraries and how they should be documented. The first two
  aren't libraries at the moment, but I think they should be. The last two are
  libraries but they aren't installed. Some work is also being done on an improved
  version of the 'associating nodes' library that uses the MC if available.

- Define the interaction between selection API, ENUM_FRAMESIZES and S_FMT. See
  this thread for all the nasty details:

  http://www.spinics.net/lists/linux-media/msg65137.html

Feel free to add suggestions to this list.

Note: my email availability will be limited in the next three weeks, especially
next week, as I am travelling a lot.

Regards,

	Hans

Re: Agenda for the Edinburgh mini-summit

[Oliver Schinagl]

On 30-08-13 15:01, Hans Verkuil wrote:
> OK, I know, we don't even know yet when the mini-summit will be held but I thought
> I'd just start this thread to collect input for the agenda.
>
> I have these topics (and I *know* that I am forgetting a few):
>
> - Discuss ideas/use-cases for a property-based API. An initial discussion
>    appeared in this thread:
>
>    http://permalink.gmane.org/gmane.linux.drivers.video-input-infrastructure/65195
>
> - What is needed to share i2c video transmitters between drm and v4l? Hopefully
>    we will know more after the upcoming LPC.
>
> - Decide on how v4l2 support libraries should be organized. There is code for
>    handling raw-to-sliced VBI decoding, ALSA looping, finding associated
>    video/alsa nodes and for TV frequency tables. We should decide how that should
>    be organized into libraries and how they should be documented. The first two
>    aren't libraries at the moment, but I think they should be. The last two are
>    libraries but they aren't installed. Some work is also being done on an improved
>    version of the 'associating nodes' library that uses the MC if available.
>
> - Define the interaction between selection API, ENUM_FRAMESIZES and S_FMT. See
>    this thread for all the nasty details:
>
>    http://www.spinics.net/lists/linux-media/msg65137.html
>
> Feel free to add suggestions to this list.
What about a hardware accelerated decoding API/framework? Is there a 
proper framework for this at all? I see the broadcom module is still in 
staging and may never come out of it, but how are other video decoding 
engines handled that don't have cameras or displays.

Reason for asking is that we from linux-sunxi have made some positive 
progress in Reverse engineering the video decoder blob of the Allwinner 
A10 and this knowledge will need a kernel side driver in some framework. 
I looked at the exynos video decoders and googling for linux-media 
hardware accelerated decoding doesn't yield much either.

Anyway, just a thought; if you think it's the wrong place for it to be 
discussed, that's ok :)

oliver
>
> Note: my email availability will be limited in the next three weeks, especially
> next week, as I am travelling a lot.
>
> Regards,
>
> 	Hans
> --
> To unsubscribe from this list: send the line "unsubscribe linux-media" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Agenda for the Edinburgh mini-summit

[Mauro Carvalho Chehab]

Em Fri, 30 Aug 2013 15:21:05 +0200
Oliver Schinagl <oliver+list@schinagl.nl> escreveu:

> On 30-08-13 15:01, Hans Verkuil wrote:
> > OK, I know, we don't even know yet when the mini-summit will be held but I thought
> > I'd just start this thread to collect input for the agenda.
> >
> > I have these topics (and I *know* that I am forgetting a few):
> >
> > - Discuss ideas/use-cases for a property-based API. An initial discussion
> >    appeared in this thread:
> >
> >    http://permalink.gmane.org/gmane.linux.drivers.video-input-infrastructure/65195
> >
> > - What is needed to share i2c video transmitters between drm and v4l? Hopefully
> >    we will know more after the upcoming LPC.
> >
> > - Decide on how v4l2 support libraries should be organized. There is code for
> >    handling raw-to-sliced VBI decoding, ALSA looping, finding associated
> >    video/alsa nodes and for TV frequency tables. We should decide how that should
> >    be organized into libraries and how they should be documented. The first two
> >    aren't libraries at the moment, but I think they should be. The last two are
> >    libraries but they aren't installed. Some work is also being done on an improved
> >    version of the 'associating nodes' library that uses the MC if available.
> >
> > - Define the interaction between selection API, ENUM_FRAMESIZES and S_FMT. See
> >    this thread for all the nasty details:
> >
> >    http://www.spinics.net/lists/linux-media/msg65137.html
> >
> > Feel free to add suggestions to this list.

>From my side, I'd like to discuss about a better integration between DVB
and V4L2, including starting using the media controller API on DVB side
too. Btw, it would be great if we could get a status about the media
controller API usage on ALSA. I'm planning to work at such integration
soon. 

> What about a hardware accelerated decoding API/framework? Is there a 
> proper framework for this at all? I see the broadcom module is still in 
> staging and may never come out of it, but how are other video decoding 
> engines handled that don't have cameras or displays.
> 
> Reason for asking is that we from linux-sunxi have made some positive 
> progress in Reverse engineering the video decoder blob of the Allwinner 
> A10 and this knowledge will need a kernel side driver in some framework. 
> I looked at the exynos video decoders and googling for linux-media 
> hardware accelerated decoding doesn't yield much either.
> 
> Anyway, just a thought; if you think it's the wrong place for it to be 
> discussed, that's ok :)

Well, the mem2mem V4L2 devices should provide all that would be needed for
accelerated encoders/decoders. If not, then feel free to propose extensions
to fit your needs.

Regards,
Mauro

> 
> oliver
> >
> > Note: my email availability will be limited in the next three weeks, especially
> > next week, as I am travelling a lot.
> >
> > Regards,
> >
> > 	Hans
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-media" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-media" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


-- 

Cheers,
Mauro

Re: Agenda for the Edinburgh mini-summit

[Hans Verkuil]

On 08/30/2013 03:21 PM, Oliver Schinagl wrote:
> On 30-08-13 15:01, Hans Verkuil wrote:
>> OK, I know, we don't even know yet when the mini-summit will be held but I thought
>> I'd just start this thread to collect input for the agenda.
>>
>> I have these topics (and I *know* that I am forgetting a few):
>>
>> - Discuss ideas/use-cases for a property-based API. An initial discussion
>>    appeared in this thread:
>>
>>    http://permalink.gmane.org/gmane.linux.drivers.video-input-infrastructure/65195
>>
>> - What is needed to share i2c video transmitters between drm and v4l? Hopefully
>>    we will know more after the upcoming LPC.
>>
>> - Decide on how v4l2 support libraries should be organized. There is code for
>>    handling raw-to-sliced VBI decoding, ALSA looping, finding associated
>>    video/alsa nodes and for TV frequency tables. We should decide how that should
>>    be organized into libraries and how they should be documented. The first two
>>    aren't libraries at the moment, but I think they should be. The last two are
>>    libraries but they aren't installed. Some work is also being done on an improved
>>    version of the 'associating nodes' library that uses the MC if available.
>>
>> - Define the interaction between selection API, ENUM_FRAMESIZES and S_FMT. See
>>    this thread for all the nasty details:
>>
>>    http://www.spinics.net/lists/linux-media/msg65137.html
>>
>> Feel free to add suggestions to this list.
> What about a hardware accelerated decoding API/framework? Is there a 
> proper framework for this at all? I see the broadcom module is still in 
> staging and may never come out of it, but how are other video decoding 
> engines handled that don't have cameras or displays.
> 
> Reason for asking is that we from linux-sunxi have made some positive 
> progress in Reverse engineering the video decoder blob of the Allwinner 
> A10 and this knowledge will need a kernel side driver in some framework. 
> I looked at the exynos video decoders and googling for linux-media 
> hardware accelerated decoding doesn't yield much either.
> 
> Anyway, just a thought; if you think it's the wrong place for it to be 
> discussed, that's ok :)

No, this is the right place. See http://hverkuil.home.xs4all.nl/spec/media.html#codec
for more information.

For the longest time that section in the spec said that that interface was 'Suspended'.
That was only corrected in 3.10 or 3.11 even though actual codec support has been
around for much longer. There are many v4l2 drivers today that do this. Just grep
for V4L2_CAP_VIDEO_M2M in drivers/media.

Codec drivers are really just a video node that can capture and output at the same
time, and has a lot of codec controls (http://hverkuil.home.xs4all.nl/spec/media.html#mpeg-controls)
to tweak codec parameters.

Just post any questions you have regarding this to the linux-media mailinglist,
we're happy to help out.

Regards,

	Hans

> 
> oliver
>>
>> Note: my email availability will be limited in the next three weeks, especially
>> next week, as I am travelling a lot.
>>
>> Regards,
>>
>> 	Hans
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-media" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
> 

Re: [media-workshop] Agenda for the Edinburgh mini-summit

[Laurent Pinchart]

Hi Mauro,

On Friday 30 August 2013 10:31:23 Mauro Carvalho Chehab wrote:
> Em Fri, 30 Aug 2013 15:21:05 +0200 Oliver Schinagl escreveu:
> > On 30-08-13 15:01, Hans Verkuil wrote:
> > > OK, I know, we don't even know yet when the mini-summit will be held but
> > > I thought I'd just start this thread to collect input for the agenda.
> > > 
> > > I have these topics (and I *know* that I am forgetting a few):
> > > 
> > > - Discuss ideas/use-cases for a property-based API. An initial
> > >   discussion appeared in this thread:
> > >    
> > >   http://permalink.gmane.org/gmane.linux.drivers.video-input-> > >   infrastructure/65195
> > >
> > > - What is needed to share i2c video transmitters between drm and v4l?
> > >   Hopefully we will know more after the upcoming LPC.
> > > 
> > > - Decide on how v4l2 support libraries should be organized. There is
> > >   code for handling raw-to-sliced VBI decoding, ALSA looping, finding
> > >   associated video/alsa nodes and for TV frequency tables. We should
> > >   decide how that should be organized into libraries and how they should
> > >   be documented. The first two aren't libraries at the moment, but I
> > >   think they should be. The last two are libraries but they aren't
> > >   installed. Some work is also being done on an improved version of
> > >   the 'associating nodes' library that uses the MC if available.> > 
> > > - Define the interaction between selection API, ENUM_FRAMESIZES and
> > >   S_FMT. See this thread for all the nasty details:
> > >    
> > >   http://www.spinics.net/lists/linux-media/msg65137.html
> > > 
> > > Feel free to add suggestions to this list.
> 
> From my side, I'd like to discuss about a better integration between DVB
> and V4L2, including starting using the media controller API on DVB side
> too. Btw, it would be great if we could get a status about the media
> controller API usage on ALSA. I'm planning to work at such integration
> soon.
> 
> > What about a hardware accelerated decoding API/framework? Is there a
> > proper framework for this at all? I see the broadcom module is still in
> > staging and may never come out of it, but how are other video decoding
> > engines handled that don't have cameras or displays.
> > 
> > Reason for asking is that we from linux-sunxi have made some positive
> > progress in Reverse engineering the video decoder blob of the Allwinner
> > A10 and this knowledge will need a kernel side driver in some framework.
> > I looked at the exynos video decoders and googling for linux-media
> > hardware accelerated decoding doesn't yield much either.
> > 
> > Anyway, just a thought; if you think it's the wrong place for it to be
> > discussed, that's ok :)
> 
> Well, the mem2mem V4L2 devices should provide all that would be needed for
> accelerated encoders/decoders. If not, then feel free to propose extensions
> to fit your needs.

Two comments regarding this:

- V4L2 mem-to-mem is great for frame-based codecs, but SoCs sometimes only 
implement part of the codec in hardware, leaving the rest to the software. 
Encoded bistream parsing is one of those areas that are left to the CPU, for 
instance on some ST SoCs (CC'ing Benjamin Gaignard).

- http://www.linuxplumbersconf.org/2013/ocw/sessions/1605

-- 
Regards,

Laurent Pinchart

Re: [media-workshop] Agenda for the Edinburgh mini-summit

[Laurent Pinchart]

Hi Pawel,

On Saturday 31 August 2013 08:58:41 Pawel Osciak wrote:
> On Sat, Aug 31, 2013 at 1:54 AM, Laurent Pinchart wrote:
> > On Friday 30 August 2013 10:31:23 Mauro Carvalho Chehab wrote:
> > > Em Fri, 30 Aug 2013 15:21:05 +0200 Oliver Schinagl escreveu:
> > > > On 30-08-13 15:01, Hans Verkuil wrote:
> > > > > OK, I know, we don't even know yet when the mini-summit will be held
> > > > > but I thought I'd just start this thread to collect input for the
> > > > > agenda.
> > > > > 
> > > > > I have these topics (and I *know* that I am forgetting a few):
> > > > > 
> > > > > - Discuss ideas/use-cases for a property-based API. An initial
> > > > >   discussion appeared in this thread:
> > > > >   
> > > > >   http://permalink.gmane.org/gmane.linux.drivers.video-input-> > > > >   infrastructure/65195
> > > > >
> > > > > - What is needed to share i2c video transmitters between drm and
> > > > >   v4l? Hopefully we will know more after the upcoming LPC.
> > > > > 
> > > > > - Decide on how v4l2 support libraries should be organized. There is
> > > > >   code for handling raw-to-sliced VBI decoding, ALSA looping,
> > > > >   finding associated video/alsa nodes and for TV frequency tables.
> > > > >   We should decide how that should be organized into libraries and
> > > > >   how they should be documented. The first two aren't libraries at
> > > > >   the moment, but I think they should be. The last two are libraries
> > > > >   but they aren't installed. Some work is also being done on an
> > > > >   improved version of the 'associating nodes' library that uses the
> > > > >   MC if available.
> > > > > 
> > > > > - Define the interaction between selection API, ENUM_FRAMESIZES and
> > > > >   S_FMT. See this thread for all the nasty details:
> > > > >   
> > > > >   http://www.spinics.net/lists/linux-media/msg65137.html
> > > > > 
> > > > > Feel free to add suggestions to this list.
> > > 
> > > From my side, I'd like to discuss about a better integration between DVB
> > > and V4L2, including starting using the media controller API on DVB side
> > > too. Btw, it would be great if we could get a status about the media
> > > controller API usage on ALSA. I'm planning to work at such integration
> > > soon.
> > > 
> > > > What about a hardware accelerated decoding API/framework? Is there a
> > > > proper framework for this at all? I see the broadcom module is still
> > > > in staging and may never come out of it, but how are other video
> > > > decoding engines handled that don't have cameras or displays.
> > > > 
> > > > Reason for asking is that we from linux-sunxi have made some positive
> > > > progress in Reverse engineering the video decoder blob of the
> > > > Allwinner A10 and this knowledge will need a kernel side driver in
> > > > some framework.
> > > >
> > > > I looked at the exynos video decoders and googling for linux-media
> > > > hardware accelerated decoding doesn't yield much either.
> > > > 
> > > > Anyway, just a thought; if you think it's the wrong place for it to be
> > > > discussed, that's ok :)
> > > 
> > > Well, the mem2mem V4L2 devices should provide all that would be needed
> > > for accelerated encoders/decoders. If not, then feel free to propose
> > > extensionsto fit your needs.
> > 
> > Two comments regarding this:
> > 
> > - V4L2 mem-to-mem is great for frame-based codecs, but SoCs sometimes only
> > implement part of the codec in hardware, leaving the rest to the software.
> > Encoded bistream parsing is one of those areas that are left to the CPU,
> > for instance on some ST SoCs (CC'ing Benjamin Gaignard).
> 
> This is an interesting topic for me as well, although I'm still not sure if
> I can make it to the workshop. Would it make sense to have v4l parser
> plugins hook up to qbuf and do the parsing there?

Do you mean in libv4l ?

> > - http://www.linuxplumbersconf.org/2013/ocw/sessions/1605

-- 
Regards,

Laurent Pinchart

Re: Agenda for the Edinburgh mini-summit

[Hans Verkuil]

On 08/30/2013 03:01 PM, Hans Verkuil wrote:
> OK, I know, we don't even know yet when the mini-summit will be held but I thought
> I'd just start this thread to collect input for the agenda.
> 
> I have these topics (and I *know* that I am forgetting a few):
> 
> - Discuss ideas/use-cases for a property-based API. An initial discussion
>   appeared in this thread:
> 
>   http://permalink.gmane.org/gmane.linux.drivers.video-input-infrastructure/65195
> 
> - What is needed to share i2c video transmitters between drm and v4l? Hopefully
>   we will know more after the upcoming LPC.
> 
> - Decide on how v4l2 support libraries should be organized. There is code for
>   handling raw-to-sliced VBI decoding, ALSA looping, finding associated
>   video/alsa nodes and for TV frequency tables. We should decide how that should
>   be organized into libraries and how they should be documented. The first two
>   aren't libraries at the moment, but I think they should be. The last two are
>   libraries but they aren't installed. Some work is also being done on an improved
>   version of the 'associating nodes' library that uses the MC if available.
> 
> - Define the interaction between selection API, ENUM_FRAMESIZES and S_FMT. See
>   this thread for all the nasty details:
> 
>   http://www.spinics.net/lists/linux-media/msg65137.html
> 
> Feel free to add suggestions to this list.

I got another one:

VIDIOC_TRY_FMT shouldn't return -EINVAL when an unsupported pixelformat is provided,
but in practice video capture board tend to do that, while webcam drivers tend to map
it silently to a valid pixelformat. Some applications rely on the -EINVAL error code.

We need to decide how to adjust the spec. I propose to just say that some drivers
will map it silently and others will return -EINVAL and that you don't know what a
driver will do. Also specify that an unsupported pixelformat is the only reason why
TRY_FMT might return -EINVAL.

Alternatively we might want to specify explicitly that EINVAL should be returned for
video capture devices (i.e. devices supporting S_STD or S_DV_TIMINGS) and 0 for all
others.

Regards,

	Hans

Re: [media-workshop] Agenda for the Edinburgh mini-summit

[Sakari Ailus]

On Fri, Aug 30, 2013 at 03:01:25PM +0200, Hans Verkuil wrote:
> OK, I know, we don't even know yet when the mini-summit will be held but I thought
> I'd just start this thread to collect input for the agenda.
> 
> I have these topics (and I *know* that I am forgetting a few):
> 
> - Discuss ideas/use-cases for a property-based API. An initial discussion
>   appeared in this thread:
> 
>   http://permalink.gmane.org/gmane.linux.drivers.video-input-infrastructure/65195
> 
> - What is needed to share i2c video transmitters between drm and v4l? Hopefully
>   we will know more after the upcoming LPC.
> 
> - Decide on how v4l2 support libraries should be organized. There is code for
>   handling raw-to-sliced VBI decoding, ALSA looping, finding associated
>   video/alsa nodes and for TV frequency tables. We should decide how that should
>   be organized into libraries and how they should be documented. The first two
>   aren't libraries at the moment, but I think they should be. The last two are
>   libraries but they aren't installed. Some work is also being done on an improved
>   version of the 'associating nodes' library that uses the MC if available.
> 
> - Define the interaction between selection API, ENUM_FRAMESIZES and S_FMT. See
>   this thread for all the nasty details:
> 
>   http://www.spinics.net/lists/linux-media/msg65137.html

- Multi-format frames and metadata. Support would be needed on video nodes
  and V4L2 subdev nodes. I'll prepare the RFC for the former; the latter has
  an RFC here:

  <URL:http://www.spinics.net/lists/linux-media/msg67295.html>

-- 
Kind regards,

Sakari Ailus
e-mail: sakari.ailus@iki.fi	XMPP: sailus@retiisi.org.uk

Re: [media-workshop] Agenda for the Edinburgh mini-summit

[Guennadi Liakhovetski]

On Sat, 31 Aug 2013, Hans Verkuil wrote:

> 
> 
> On 08/30/2013 03:01 PM, Hans Verkuil wrote:
> > OK, I know, we don't even know yet when the mini-summit will be held but I thought
> > I'd just start this thread to collect input for the agenda.
> > 
> > I have these topics (and I *know* that I am forgetting a few):
> > 
> > - Discuss ideas/use-cases for a property-based API. An initial discussion
> >   appeared in this thread:
> > 
> >   http://permalink.gmane.org/gmane.linux.drivers.video-input-infrastructure/65195
> > 
> > - What is needed to share i2c video transmitters between drm and v4l? Hopefully
> >   we will know more after the upcoming LPC.
> > 
> > - Decide on how v4l2 support libraries should be organized. There is code for
> >   handling raw-to-sliced VBI decoding, ALSA looping, finding associated
> >   video/alsa nodes and for TV frequency tables. We should decide how that should
> >   be organized into libraries and how they should be documented. The first two
> >   aren't libraries at the moment, but I think they should be. The last two are
> >   libraries but they aren't installed. Some work is also being done on an improved
> >   version of the 'associating nodes' library that uses the MC if available.
> > 
> > - Define the interaction between selection API, ENUM_FRAMESIZES and S_FMT. See
> >   this thread for all the nasty details:
> > 
> >   http://www.spinics.net/lists/linux-media/msg65137.html
> > 
> > Feel free to add suggestions to this list.
> 
> I got another one:
> 
> VIDIOC_TRY_FMT shouldn't return -EINVAL when an unsupported pixelformat is provided,
> but in practice video capture board tend to do that, while webcam drivers tend to map
> it silently to a valid pixelformat. Some applications rely on the -EINVAL error code.
> 
> We need to decide how to adjust the spec. I propose to just say that some drivers
> will map it silently and others will return -EINVAL and that you don't know what a
> driver will do. Also specify that an unsupported pixelformat is the only reason why
> TRY_FMT might return -EINVAL.
> 
> Alternatively we might want to specify explicitly that EINVAL should be returned for
> video capture devices (i.e. devices supporting S_STD or S_DV_TIMINGS) and 0 for all
> others.

Just to make sure I understand right - that kind of excludes cameras, 
right? Still, even for (other) video capture devices, like TV decoders, is 
there a real serious enough reason to _change_ the specs, which says 

http://linuxtv.org/downloads/v4l-dvb-apis/vidioc-g-fmt.html

EINVAL

    The struct v4l2_format type field is invalid or the requested buffer 
type not supported.

If we have a spec, that says A, and some drivers drivers do A, but others 
do B, and we want to change the specs to B? Instead of either changing the 
(wrong) drivers to A (yes, some applications expect that wrong behaviour) 
or at least extending the spec to allow both A and B?

Thanks
Guennadi
---
Guennadi Liakhovetski, Ph.D.
Freelance Open-Source Software Developer
http://www.open-technology.de/

Re: [media-workshop] Agenda for the Edinburgh mini-summit

[Laurent Pinchart]

On Saturday 31 August 2013 09:04:14 Pawel Osciak wrote:
> On Sat, Aug 31, 2013 at 9:03 AM, Laurent Pinchart wrote:
> > On Saturday 31 August 2013 08:58:41 Pawel Osciak wrote:
> > > On Sat, Aug 31, 2013 at 1:54 AM, Laurent Pinchart wrote:
> > > > On Friday 30 August 2013 10:31:23 Mauro Carvalho Chehab wrote:
> > > > > Em Fri, 30 Aug 2013 15:21:05 +0200 Oliver Schinagl escreveu:

[snip]

> > > > > > What about a hardware accelerated decoding API/framework? Is there
> > > > > > a proper framework for this at all? I see the broadcom module is
> > > > > > still in staging and may never come out of it, but how are other
> > > > > > video decoding engines handled that don't have cameras or
> > > > > > displays.
> > > > > > 
> > > > > > Reason for asking is that we from linux-sunxi have made some
> > > > > > positive progress in Reverse engineering the video decoder blob of
> > > > > > the Allwinner A10 and this knowledge will need a kernel side
> > > > > > driver in some framework.
> > > > > > 
> > > > > > I looked at the exynos video decoders and googling for linux-media
> > > > > > hardware accelerated decoding doesn't yield much either.
> > > > > > 
> > > > > > Anyway, just a thought; if you think it's the wrong place for it
> > > > > > to be discussed, that's ok :)
> > > > > 
> > > > > Well, the mem2mem V4L2 devices should provide all that would be
> > > > > needed for accelerated encoders/decoders. If not, then feel free to
> > > > > propose extensionsto fit your needs.
> > > > 
> > > > Two comments regarding this:
> > > > 
> > > > - V4L2 mem-to-mem is great for frame-based codecs, but SoCs sometimes
> > > >   only implement part of the codec in hardware, leaving the rest to
> > > >   the software.
> > > >
> > > > Encoded bistream parsing is one of those areas that are left to the
> > > > CPU, for instance on some ST SoCs (CC'ing Benjamin Gaignard).
> > > 
> > > This is an interesting topic for me as well, although I'm still not sure
> > > if I can make it to the workshop. Would it make sense to have v4l parser
> > > plugins hook up to qbuf and do the parsing there?
> > 
> > Do you mean in libv4l ?
> 
> Yes...

Let's discuss that in Edinburgh then. The major problem as I see it is that 
the hardware codec might consume and produce data that wouldn't fit the spirit 
of the current V4L2 API. We might end up with passing register lists in a V4L2 
buffer, which would be pretty ugly.

Benjamin, do you plan to attend the conference ?

> > > > - http://www.linuxplumbersconf.org/2013/ocw/sessions/1605

-- 
Regards,

Laurent Pinchart

Re: [media-workshop] Agenda for the Edinburgh mini-summit

[Laurent Pinchart]

Hi Guennadi,

On Saturday 31 August 2013 20:38:54 Guennadi Liakhovetski wrote:
> On Sat, 31 Aug 2013, Hans Verkuil wrote:
> > On 08/30/2013 03:01 PM, Hans Verkuil wrote:
> > > OK, I know, we don't even know yet when the mini-summit will be held but
> > > I thought I'd just start this thread to collect input for the agenda.
> > > 
> > > I have these topics (and I *know* that I am forgetting a few):

[snip]

> > > Feel free to add suggestions to this list.
> > 
> > I got another one:
> > 
> > VIDIOC_TRY_FMT shouldn't return -EINVAL when an unsupported pixelformat is
> > provided, but in practice video capture board tend to do that, while
> > webcam drivers tend to map it silently to a valid pixelformat. Some
> > applications rely on the -EINVAL error code.
> > 
> > We need to decide how to adjust the spec. I propose to just say that some
> > drivers will map it silently and others will return -EINVAL and that you
> > don't know what a driver will do. Also specify that an unsupported
> > pixelformat is the only reason why TRY_FMT might return -EINVAL.
> > 
> > Alternatively we might want to specify explicitly that EINVAL should be
> > returned for video capture devices (i.e. devices supporting S_STD or
> > S_DV_TIMINGS) and 0 for all others.
> 
> Just to make sure I understand right - that kind of excludes cameras,
> right? Still, even for (other) video capture devices, like TV decoders, is
> there a real serious enough reason to _change_ the specs, which says
> 
> http://linuxtv.org/downloads/v4l-dvb-apis/vidioc-g-fmt.html
> 
> EINVAL
> 
>     The struct v4l2_format type field is invalid or the requested buffer
> type not supported.

I think Hans meant unsupported fmt.pix.pixelformat (or the equivalent for 
multiplane) values. For instance the uvcvideo driver will return a default 
fourcc if an application tries an unsupported fourcc, some other drivers 
return -EINVAL.

> If we have a spec, that says A, and some drivers drivers do A, but others
> do B, and we want to change the specs to B? Instead of either changing the
> (wrong) drivers to A (yes, some applications expect that wrong behaviour)
> or at least extending the spec to allow both A and B?

-- 
Regards,

Laurent Pinchart

Re: [media-workshop] Agenda for the Edinburgh mini-summit

[Guennadi Liakhovetski]

On Sat, 31 Aug 2013, Laurent Pinchart wrote:

> Hi Guennadi,
> 
> On Saturday 31 August 2013 20:38:54 Guennadi Liakhovetski wrote:
> > On Sat, 31 Aug 2013, Hans Verkuil wrote:
> > > On 08/30/2013 03:01 PM, Hans Verkuil wrote:
> > > > OK, I know, we don't even know yet when the mini-summit will be held but
> > > > I thought I'd just start this thread to collect input for the agenda.
> > > > 
> > > > I have these topics (and I *know* that I am forgetting a few):
> 
> [snip]
> 
> > > > Feel free to add suggestions to this list.
> > > 
> > > I got another one:
> > > 
> > > VIDIOC_TRY_FMT shouldn't return -EINVAL when an unsupported pixelformat is
> > > provided, but in practice video capture board tend to do that, while
> > > webcam drivers tend to map it silently to a valid pixelformat. Some
> > > applications rely on the -EINVAL error code.
> > > 
> > > We need to decide how to adjust the spec. I propose to just say that some
> > > drivers will map it silently and others will return -EINVAL and that you
> > > don't know what a driver will do. Also specify that an unsupported
> > > pixelformat is the only reason why TRY_FMT might return -EINVAL.
> > > 
> > > Alternatively we might want to specify explicitly that EINVAL should be
> > > returned for video capture devices (i.e. devices supporting S_STD or
> > > S_DV_TIMINGS) and 0 for all others.
> > 
> > Just to make sure I understand right - that kind of excludes cameras,
> > right? Still, even for (other) video capture devices, like TV decoders, is
> > there a real serious enough reason to _change_ the specs, which says
> > 
> > http://linuxtv.org/downloads/v4l-dvb-apis/vidioc-g-fmt.html
> > 
> > EINVAL
> > 
> >     The struct v4l2_format type field is invalid or the requested buffer
> > type not supported.
> 
> I think Hans meant unsupported fmt.pix.pixelformat (or the equivalent for 
> multiplane) values.

Good, then I understood him correctly :)

> For instance the uvcvideo driver will return a default 
> fourcc if an application tries an unsupported fourcc,

Yes, that's what I would do too and that's what the spec dictates.

> some other drivers 
> return -EINVAL.

that just seems plain wrong to me. So, as I said, to not break the 
userspace we can extend the specs, but not prohibit the currently defined 
behaviour. So, that last option:

> > > Alternatively we might want to specify explicitly that EINVAL should be
> > > returned for video capture devices (i.e. devices supporting S_STD or
> > > S_DV_TIMINGS) and 0 for all others.

I'm not sure I like a lot, unless those drivers are very special and they 
all already behave like that.

Thanks
Guennadi

> > If we have a spec, that says A, and some drivers drivers do A, but others
> > do B, and we want to change the specs to B? Instead of either changing the
> > (wrong) drivers to A (yes, some applications expect that wrong behaviour)
> > or at least extending the spec to allow both A and B?
> 
> -- 
> Regards,
> 
> Laurent Pinchart

---
Guennadi Liakhovetski, Ph.D.
Freelance Open-Source Software Developer
http://www.open-technology.de/

Re: [media-workshop] Agenda for the Edinburgh mini-summit

[Hans Verkuil]

On 08/31/2013 10:36 PM, Guennadi Liakhovetski wrote:
> On Sat, 31 Aug 2013, Laurent Pinchart wrote:
> 
>> Hi Guennadi,
>>
>> On Saturday 31 August 2013 20:38:54 Guennadi Liakhovetski wrote:
>>> On Sat, 31 Aug 2013, Hans Verkuil wrote:
>>>> On 08/30/2013 03:01 PM, Hans Verkuil wrote:
>>>>> OK, I know, we don't even know yet when the mini-summit will be held but
>>>>> I thought I'd just start this thread to collect input for the agenda.
>>>>>
>>>>> I have these topics (and I *know* that I am forgetting a few):
>>
>> [snip]
>>
>>>>> Feel free to add suggestions to this list.
>>>>
>>>> I got another one:
>>>>
>>>> VIDIOC_TRY_FMT shouldn't return -EINVAL when an unsupported pixelformat is
>>>> provided, but in practice video capture board tend to do that, while
>>>> webcam drivers tend to map it silently to a valid pixelformat. Some
>>>> applications rely on the -EINVAL error code.
>>>>
>>>> We need to decide how to adjust the spec. I propose to just say that some
>>>> drivers will map it silently and others will return -EINVAL and that you
>>>> don't know what a driver will do. Also specify that an unsupported
>>>> pixelformat is the only reason why TRY_FMT might return -EINVAL.
>>>>
>>>> Alternatively we might want to specify explicitly that EINVAL should be
>>>> returned for video capture devices (i.e. devices supporting S_STD or
>>>> S_DV_TIMINGS) and 0 for all others.
>>>
>>> Just to make sure I understand right - that kind of excludes cameras,
>>> right? Still, even for (other) video capture devices, like TV decoders, is
>>> there a real serious enough reason to _change_ the specs, which says
>>>
>>> http://linuxtv.org/downloads/v4l-dvb-apis/vidioc-g-fmt.html
>>>
>>> EINVAL
>>>
>>>     The struct v4l2_format type field is invalid or the requested buffer
>>> type not supported.
>>
>> I think Hans meant unsupported fmt.pix.pixelformat (or the equivalent for 
>> multiplane) values.
> 
> Good, then I understood him correctly :)
> 
>> For instance the uvcvideo driver will return a default 
>> fourcc if an application tries an unsupported fourcc,
> 
> Yes, that's what I would do too and that's what the spec dictates.
> 
>> some other drivers 
>> return -EINVAL.
> 
> that just seems plain wrong to me. So, as I said, to not break the 
> userspace we can extend the specs, but not prohibit the currently defined 
> behaviour. So, that last option:
> 
>>>> Alternatively we might want to specify explicitly that EINVAL should be
>>>> returned for video capture devices (i.e. devices supporting S_STD or
>>>> S_DV_TIMINGS) and 0 for all others.
> 
> I'm not sure I like a lot, unless those drivers are very special and they 
> all already behave like that.

Almost (have to check though) all TV capture drivers behave like that, yes.
Very unfortunate.

On the other hand webcam apps must assume that TRY_FMT will just map an unsupported
pixel format to a valid pixel format since that is what uvc does. And a webcam app
that doesn't support uvc can't be called a webcam app :-)

Regards,

	Hans

> Thanks
> Guennadi
> 
>>> If we have a spec, that says A, and some drivers drivers do A, but others
>>> do B, and we want to change the specs to B? Instead of either changing the
>>> (wrong) drivers to A (yes, some applications expect that wrong behaviour)
>>> or at least extending the spec to allow both A and B?
>>
>> -- 
>> Regards,
>>
>> Laurent Pinchart
> 
> ---
> Guennadi Liakhovetski, Ph.D.
> Freelance Open-Source Software Developer
> http://www.open-technology.de/
> 

Re: [media-workshop] Agenda for the Edinburgh mini-summit

[Mauro Carvalho Chehab]

Hi Benjamin/Hugues,

Em Wed, 04 Sep 2013 10:26:01 +0200
Benjamin Gaignard <benjamin.gaignard@linaro.org> escreveu:

> Hi all,
> 
> Based on STM past experience we have seen variety of userland/kernel or
> CPU/DSP/Microcontroller split for video codecs.
> Each time we done proprietary kernel interface because of lack defacto
> kernel standard.
> Principal needs were: no memory copy, video codec interface (for example
> video encoder controls), frame base API, multiformat codecs.
> 
> In the past we have seen several hardware partitioning:
> a) basic CPU/hardware split: all the software run on CPU, basically it is
> bitstream parsing and preparation of hardware descriptors to call IPs.
> We made two different implementations:
> a.1) one fully in kernel embedded in kernel module the drawback was the
> proprietary API and the bitstream parsing stack reused from legacy project
> and no compliant to kernel coding guide lines.
> a.2) an other one was fully in userland with a minimal kernel drivers for
> write registers and catch interrupts, drawbacks were exposition of hardware
> registers in userland (no functional API but hardware specific API) and
> physical address exposed in userland.
> 
> b) DSP (or Microcontroller)/hardware split: the software partially run on
> coprocessor where the firmware handle the IP controls and the CPU do the
> bitstream parsing. On this implementation all the stack running on CPU was
> on userland with proprietary API for firmware communication.
> 
> After that Exynos S5P show up, with an interesting M2M interface very close
> to what was done by us on step a.1) and let us hope an incoming
> standardization for video codecs kernel API.
> The main benefit we see of this is a reduction of software diversity on top
> kernel being agnostic to hardware used, for example we could introduce then
> a unified gstreamer v4l2 decoder plugin or unified OMX decoder plugin.
> 
> For us it is important to keep the hardware details as low as possible is
> software stack (i.e. kernel drivers) instead of a collection of proprietary
> userland libraries.
> What we are doing now is trying to go this way for next products.
> 
> Regarding S5P MFC all codec software stack remains in firmware, so kernel
> driver deals only with power/interrupt/clock and firmware communication but
> no processing are done on input bitstream or output frames. Our split is
> different because bitstream parsing is left to CPU, it means we put in the
> kernel significant amount of code to do that. The questions is how to push
> that code ?
> 
> What we have seen also it that several software stacks (ffmpeg, G1, ...)
> are doing same operation on bitstream (it is logical because it is link to
> the standards), so what about making it generic to avoid to embed quite the
> same code on several v4l2 drivers ?

I think we need more discussions to understand exactly what kind of bitstream
operations are needed to be done on the codec for your hardware.

As already discussed, technically, there are two possible solutions:

	1) delegate those tasks to an userspace library (libv4l);

	2) do them in Kernel.

Both have vantages and disadvantages. Among others, one of the reasons why
we opted to handle different formats in userspace is because we don't do
floating point(FP) registers in Kernel. For some kind of codecs, that would 
be a need. 

While it would theoretically be possible to use FP inside the Kernel,
that would require lots of work (Kernel currently doesn't save FP status.
Also, as Kernel stacks is typically measured in a few KB, pushing them on
stack could be a problem).

Also, FP is very arch-dependent, and we try to make the drivers to not be so
tight on some specific architecture (of course, when this makes sense).
So, the solution would be to have a kernel library for FP, capable of
using the hardware registers for each specific arch, but writing it would
require lots of time and efforts.

Also, for a normal V4L2 input or output driver, handling the video format
on userspace works fine, and the FP issue is automatically solved by
gcc FP code.

Due to that, delegating the formats handling to userspace proofed to be the
best way for the current drivers. 

Yet, if it doesn't fit on your needs, we're open to discuss your proposal.

Btw, this is the kind of discussions that works best on a presencial
meeting, where you could show more about your problem and talk about
your proposals to address it.

Are you planning to go to the mini-summit? If so, how much time do you
need in order to do such discussions?

> 
> Benjamin (+Hugues in CC)
> 
> 
> 2013/8/31 Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> 
> > On Saturday 31 August 2013 09:04:14 Pawel Osciak wrote:
> > > On Sat, Aug 31, 2013 at 9:03 AM, Laurent Pinchart wrote:
> > > > On Saturday 31 August 2013 08:58:41 Pawel Osciak wrote:
> > > > > On Sat, Aug 31, 2013 at 1:54 AM, Laurent Pinchart wrote:
> > > > > > On Friday 30 August 2013 10:31:23 Mauro Carvalho Chehab wrote:
> > > > > > > Em Fri, 30 Aug 2013 15:21:05 +0200 Oliver Schinagl escreveu:
> >
> > [snip]
> >
> > > > > > > > What about a hardware accelerated decoding API/framework? Is
> > there
> > > > > > > > a proper framework for this at all? I see the broadcom module
> > is
> > > > > > > > still in staging and may never come out of it, but how are
> > other
> > > > > > > > video decoding engines handled that don't have cameras or
> > > > > > > > displays.
> > > > > > > >
> > > > > > > > Reason for asking is that we from linux-sunxi have made some
> > > > > > > > positive progress in Reverse engineering the video decoder
> > blob of
> > > > > > > > the Allwinner A10 and this knowledge will need a kernel side
> > > > > > > > driver in some framework.
> > > > > > > >
> > > > > > > > I looked at the exynos video decoders and googling for
> > linux-media
> > > > > > > > hardware accelerated decoding doesn't yield much either.
> > > > > > > >
> > > > > > > > Anyway, just a thought; if you think it's the wrong place for
> > it
> > > > > > > > to be discussed, that's ok :)
> > > > > > >
> > > > > > > Well, the mem2mem V4L2 devices should provide all that would be
> > > > > > > needed for accelerated encoders/decoders. If not, then feel free
> > to
> > > > > > > propose extensionsto fit your needs.
> > > > > >
> > > > > > Two comments regarding this:
> > > > > >
> > > > > > - V4L2 mem-to-mem is great for frame-based codecs, but SoCs
> > sometimes
> > > > > >   only implement part of the codec in hardware, leaving the rest to
> > > > > >   the software.
> > > > > >
> > > > > > Encoded bistream parsing is one of those areas that are left to the
> > > > > > CPU, for instance on some ST SoCs (CC'ing Benjamin Gaignard).
> > > > >
> > > > > This is an interesting topic for me as well, although I'm still not
> > sure
> > > > > if I can make it to the workshop. Would it make sense to have v4l
> > parser
> > > > > plugins hook up to qbuf and do the parsing there?
> > > >
> > > > Do you mean in libv4l ?
> > >
> > > Yes...
> >
> > Let's discuss that in Edinburgh then. The major problem as I see it is that
> > the hardware codec might consume and produce data that wouldn't fit the
> > spirit
> > of the current V4L2 API. We might end up with passing register lists in a
> > V4L2
> > buffer, which would be pretty ugly.
> >
> > Benjamin, do you plan to attend the conference ?
> >
> > > > > > - http://www.linuxplumbersconf.org/2013/ocw/sessions/1605
> >
> > --
> > Regards,
> >
> > Laurent Pinchart
> >
> >
> 
> 


-- 

Cheers,
Mauro

RE: [media-workshop] Agenda for the Edinburgh mini-summit

[Hugues FRUCHET]

Hi Mauro,

For floating point issue, we have not encountered such issue while integrating various codec (currently H264, MPEG4, VP8 of both Google G1 IP & ST IPs), could you precise which codec you experienced which required FP support ?

For user-space library, problem we encountered is that interface between parsing side (for ex. H264 SPS/PPS decoding, slice header decoding, references frame list management, ...moreover all that is needed to prepare hardware IPs call) and decoder side (hardware IPs handling) is not standardized and differs largely regarding IPs or CPU/copro partitioning.
This means that even if we use the standard V4L2 capture interface to inject video bitstream (H264 access units for ex), some proprietary meta are needed to be attached to each buffers, making de facto "un-standard" the V4L2 interface for this driver.
Exynos S5P MFC is not attaching any meta to capture input buffers, keeping a standard video bitstream injection interface (what is output naturally by well-known standard demuxers such as gstreamer ones or Android Stagefright ones).
This is the way we want to go, we will so keep hardware details at kernel driver side. 
On the other hand, this simplify drastically the integration of our video drivers on user-land multimedia middleware, reducing the time to market and support needed when reaching our end-customers. Our target is to create a unified gstreamer V4L2 decoder(encoder) plugin and a unified OMX V4L2 decoder(encoder) to fit Android, based on a single V4L2 M2M API whatever hardware IP is.

About mini summit, Benjamin and I are checking internally how to attend to discuss this topic. We think that about half a day is needed to discuss this, we can so share our code and discuss about other codebase you know dealing with video codecs.


Best regards,
Hugues.

Re: [media-workshop] Agenda for the Edinburgh mini-summit

[Laurent Pinchart]

Hi Hugues,

On Thursday 05 September 2013 13:37:49 Hugues FRUCHET wrote:
> Hi Mauro,
> 
> For floating point issue, we have not encountered such issue while
> integrating various codec (currently H264, MPEG4, VP8 of both Google G1 IP &
> ST IPs), could you precise which codec you experienced which required FP
> support ?
> 
> For user-space library, problem we encountered is that interface between
> parsing side (for ex. H264 SPS/PPS decoding, slice header decoding,
> references frame list management, ...moreover all that is needed to prepare
> hardware IPs call) and decoder side (hardware IPs handling) is not
> standardized and differs largely regarding IPs or CPU/copro partitioning.
> This means that even if we use the standard V4L2 capture interface to inject
> video bitstream (H264 access units for ex), some proprietary meta are needed
> to be attached to each buffers, making de facto "un-standard" the V4L2
> interface for this driver.

We're working on APIs to pass meta data from/to the kernel. The necessary 
infrastructure is more or less there already, we "just" need to agree on 
guidelines and standardize the process. One option that will likely be 
implemented is to store meta-data in a plane, using the multiplanar API.

The resulting plane format will be driver-specific, so we'll loose part of the 
benefits that the V4L2 API provides. We could try to solve this by writing a 
libv4l plugin, specific to your driver, that would handle bitstream parsing 
and fill the meta-data planes correctly. Applications using libv4l would thus 
only need to pass encoded frames to the library, which would create 
multiplanar buffers with video data and meta-data, and pass them to the 
driver. This would be fully transparent for the application.

> Exynos S5P MFC is not attaching any meta to capture input buffers, keeping a
> standard video bitstream injection interface (what is output naturally by
> well-known standard demuxers such as gstreamer ones or Android Stagefright
> ones). This is the way we want to go, we will so keep hardware details at
> kernel driver side. On the other hand, this simplify drastically the
> integration of our video drivers on user-land multimedia middleware,
> reducing the time to market and support needed when reaching our end-
> customers. Our target is to create a unified gstreamer V4L2 decoder(encoder)
> plugin and a unified OMX V4L2 decoder(encoder) to fit Android, based on a
> single V4L2 M2M API whatever hardware IP is.
> 
> About mini summit, Benjamin and I are checking internally how to attend to
> discuss this topic. We think that about half a day is needed to discuss
> this, we can so share our code and discuss about other codebase you know
> dealing with video codecs.

-- 
Regards,

Laurent Pinchart

Re: [media-workshop] Agenda for the Edinburgh mini-summit

[Pawel Osciak]

On Fri, Sep 6, 2013 at 10:45 PM, Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> Hi Hugues,
>
> On Thursday 05 September 2013 13:37:49 Hugues FRUCHET wrote:
> > Hi Mauro,
> >
> > For floating point issue, we have not encountered such issue while
> > integrating various codec (currently H264, MPEG4, VP8 of both Google G1 IP &
> > ST IPs), could you precise which codec you experienced which required FP
> > support ?
> >
> > For user-space library, problem we encountered is that interface between
> > parsing side (for ex. H264 SPS/PPS decoding, slice header decoding,
> > references frame list management, ...moreover all that is needed to prepare
> > hardware IPs call) and decoder side (hardware IPs handling) is not
> > standardized and differs largely regarding IPs or CPU/copro partitioning.
> > This means that even if we use the standard V4L2 capture interface to inject
> > video bitstream (H264 access units for ex), some proprietary meta are needed
> > to be attached to each buffers, making de facto "un-standard" the V4L2
> > interface for this driver.
>
> We're working on APIs to pass meta data from/to the kernel. The necessary
> infrastructure is more or less there already, we "just" need to agree on
> guidelines and standardize the process. One option that will likely be
> implemented is to store meta-data in a plane, using the multiplanar API.


What API is that? Is there an RFC somewhere?

> The resulting plane format will be driver-specific, so we'll loose part of the
> benefits that the V4L2 API provides. We could try to solve this by writing a
> libv4l plugin, specific to your driver, that would handle bitstream parsing
> and fill the meta-data planes correctly. Applications using libv4l would thus
> only need to pass encoded frames to the library, which would create
> multiplanar buffers with video data and meta-data, and pass them to the
> driver. This would be fully transparent for the application.

If V4L2 API is not hardware-independent, it's a big loss. If this
happens, there will be need for another, middleware API, like OMX IL.
This makes V4L2 by itself impractical for real world applications. And
the incentives of using V4L2 are gone, because it's much easier to
write a custom DRM driver and add any userspace API on top of it.
Perhaps this is inevitable, given differences in hardware, but a
plugin approach would be a way to keep V4L2 abstract and retain the
ability to do the bulk of processing in userspace...

Re: [media-workshop] Agenda for the Edinburgh mini-summit

[Hans Verkuil]

Hi Hugues,

On 09/05/2013 01:37 PM, Hugues FRUCHET wrote:
> Hi Mauro,
> 
> For floating point issue, we have not encountered such issue while
> integrating various codec (currently H264, MPEG4, VP8 of both Google
> G1 IP & ST IPs), could you precise which codec you experienced which
> required FP support ?
> 
> For user-space library, problem we encountered is that interface
> between parsing side (for ex. H264 SPS/PPS decoding, slice header
> decoding, references frame list management, ...moreover all that is
> needed to prepare hardware IPs call) and decoder side (hardware IPs
> handling) is not standardized and differs largely regarding IPs or
> CPU/copro partitioning. This means that even if we use the standard
> V4L2 capture interface to inject video bitstream (H264 access units
> for ex), some proprietary meta are needed to be attached to each
> buffers, making de facto "un-standard" the V4L2 interface for this
> driver.

There are lots of drivers (mostly camera drivers) that have non-standard
video formats. That's perfectly fine, as long as libv4l plugins/conversions
exist to convert it to something that's standardized.

Any application using libv4l doesn't notice the work going on under the
hood and it will look like a standard v4l2 driver.

The multiplanar API seems to me to be very suitable for these sort of devices.

> Exynos S5P MFC is not attaching any meta to capture input
> buffers, keeping a standard video bitstream injection interface (what
> is output naturally by well-known standard demuxers such as gstreamer
> ones or Android Stagefright ones). This is the way we want to go, we
> will so keep hardware details at kernel driver side. On the other
> hand, this simplify drastically the integration of our video drivers
> on user-land multimedia middleware, reducing the time to market and
> support needed when reaching our end-customers. Our target is to
> create a unified gstreamer V4L2 decoder(encoder) plugin and a unified
> OMX V4L2 decoder(encoder) to fit Android, based on a single V4L2 M2M
> API whatever hardware IP is.
> 
> About mini summit, Benjamin and I are checking internally how to
> attend to discuss this topic. We think that about half a day is
> needed to discuss this, we can so share our code and discuss about
> other codebase you know dealing with video codecs.> 

We are getting a lot of topics for the agenda and half a day for one topic
seems problematic to me.

One option is to discuss this in a smaller group a day earlier (October 22).
We might be able to get a room, or we can discuss it in the hotel lounge or
pub :-) or something.

Another option is that ST organizes a separate brainstorm session with a
few core developers. We done that in the past quite successfully.

Regards,

	Hans

RE: [media-workshop] Agenda for the Edinburgh mini-summit

[Hugues FRUCHET]

Thanks Hans,

Have you some implementation based on meta that we can check to see code details ?
It would be nice to have one with noticeable amount of code/processing made on user-land side.
I'm wondering also how libv4l is selecting each driver specific user-land plugin and how they are loaded.

BR.
-----Original Message-----
From: Hans Verkuil [mailto:hverkuil@xs4all.nl] 
Sent: lundi 9 septembre 2013 12:33
To: Hugues FRUCHET
Cc: Mauro Carvalho Chehab; Oliver Schinagl; media-workshop; Benjamin Gaignard; linux-media@vger.kernel.org
Subject: Re: [media-workshop] Agenda for the Edinburgh mini-summit

Hi Hugues,

On 09/05/2013 01:37 PM, Hugues FRUCHET wrote:
> Hi Mauro,
> 
> For floating point issue, we have not encountered such issue while
> integrating various codec (currently H264, MPEG4, VP8 of both Google
> G1 IP & ST IPs), could you precise which codec you experienced which
> required FP support ?
> 
> For user-space library, problem we encountered is that interface
> between parsing side (for ex. H264 SPS/PPS decoding, slice header
> decoding, references frame list management, ...moreover all that is
> needed to prepare hardware IPs call) and decoder side (hardware IPs
> handling) is not standardized and differs largely regarding IPs or
> CPU/copro partitioning. This means that even if we use the standard
> V4L2 capture interface to inject video bitstream (H264 access units
> for ex), some proprietary meta are needed to be attached to each
> buffers, making de facto "un-standard" the V4L2 interface for this
> driver.

There are lots of drivers (mostly camera drivers) that have non-standard
video formats. That's perfectly fine, as long as libv4l plugins/conversions
exist to convert it to something that's standardized.

Any application using libv4l doesn't notice the work going on under the
hood and it will look like a standard v4l2 driver.

The multiplanar API seems to me to be very suitable for these sort of devices.

> Exynos S5P MFC is not attaching any meta to capture input
> buffers, keeping a standard video bitstream injection interface (what
> is output naturally by well-known standard demuxers such as gstreamer
> ones or Android Stagefright ones). This is the way we want to go, we
> will so keep hardware details at kernel driver side. On the other
> hand, this simplify drastically the integration of our video drivers
> on user-land multimedia middleware, reducing the time to market and
> support needed when reaching our end-customers. Our target is to
> create a unified gstreamer V4L2 decoder(encoder) plugin and a unified
> OMX V4L2 decoder(encoder) to fit Android, based on a single V4L2 M2M
> API whatever hardware IP is.
> 
> About mini summit, Benjamin and I are checking internally how to
> attend to discuss this topic. We think that about half a day is
> needed to discuss this, we can so share our code and discuss about
> other codebase you know dealing with video codecs.> 

We are getting a lot of topics for the agenda and half a day for one topic
seems problematic to me.

One option is to discuss this in a smaller group a day earlier (October 22).
We might be able to get a room, or we can discuss it in the hotel lounge or
pub :-) or something.

Another option is that ST organizes a separate brainstorm session with a
few core developers. We done that in the past quite successfully.

Regards,

	Hans

Re: [media-workshop] Agenda for the Edinburgh mini-summit

[Hans Verkuil]

On Tue 10 September 2013 09:36:00 Hugues FRUCHET wrote:
> Thanks Hans,
> 
> Have you some implementation based on meta that we can check to see code details ?

Not as such. Basically you just add another pixelformat define for a multiplanar
format. And you define this format as having X video planes and Y planes containing
meta data.

> It would be nice to have one with noticeable amount of code/processing made on user-land side.
> I'm wondering also how libv4l is selecting each driver specific user-land plugin and how they are loaded.

libv4l-mplane in v4l-utils.git is an example of a plugin.

Documentation on the plugin API seems to be sparse, but Hans de Goede,
Sakari Ailus or Laurent Pinchart know a lot more about it.

There are (to my knowledge) no plugins that do exactly what you want, so
you're the first. But it has been designed with your use-case in mind.

Regards,

	Hans

> 
> BR.
> -----Original Message-----
> From: Hans Verkuil [mailto:hverkuil@xs4all.nl] 
> Sent: lundi 9 septembre 2013 12:33
> To: Hugues FRUCHET
> Cc: Mauro Carvalho Chehab; Oliver Schinagl; media-workshop; Benjamin Gaignard; linux-media@vger.kernel.org
> Subject: Re: [media-workshop] Agenda for the Edinburgh mini-summit
> 
> Hi Hugues,
> 
> On 09/05/2013 01:37 PM, Hugues FRUCHET wrote:
> > Hi Mauro,
> > 
> > For floating point issue, we have not encountered such issue while
> > integrating various codec (currently H264, MPEG4, VP8 of both Google
> > G1 IP & ST IPs), could you precise which codec you experienced which
> > required FP support ?
> > 
> > For user-space library, problem we encountered is that interface
> > between parsing side (for ex. H264 SPS/PPS decoding, slice header
> > decoding, references frame list management, ...moreover all that is
> > needed to prepare hardware IPs call) and decoder side (hardware IPs
> > handling) is not standardized and differs largely regarding IPs or
> > CPU/copro partitioning. This means that even if we use the standard
> > V4L2 capture interface to inject video bitstream (H264 access units
> > for ex), some proprietary meta are needed to be attached to each
> > buffers, making de facto "un-standard" the V4L2 interface for this
> > driver.
> 
> There are lots of drivers (mostly camera drivers) that have non-standard
> video formats. That's perfectly fine, as long as libv4l plugins/conversions
> exist to convert it to something that's standardized.
> 
> Any application using libv4l doesn't notice the work going on under the
> hood and it will look like a standard v4l2 driver.
> 
> The multiplanar API seems to me to be very suitable for these sort of devices.
> 
> > Exynos S5P MFC is not attaching any meta to capture input
> > buffers, keeping a standard video bitstream injection interface (what
> > is output naturally by well-known standard demuxers such as gstreamer
> > ones or Android Stagefright ones). This is the way we want to go, we
> > will so keep hardware details at kernel driver side. On the other
> > hand, this simplify drastically the integration of our video drivers
> > on user-land multimedia middleware, reducing the time to market and
> > support needed when reaching our end-customers. Our target is to
> > create a unified gstreamer V4L2 decoder(encoder) plugin and a unified
> > OMX V4L2 decoder(encoder) to fit Android, based on a single V4L2 M2M
> > API whatever hardware IP is.
> > 
> > About mini summit, Benjamin and I are checking internally how to
> > attend to discuss this topic. We think that about half a day is
> > needed to discuss this, we can so share our code and discuss about
> > other codebase you know dealing with video codecs.> 
> 
> We are getting a lot of topics for the agenda and half a day for one topic
> seems problematic to me.
> 
> One option is to discuss this in a smaller group a day earlier (October 22).
> We might be able to get a room, or we can discuss it in the hotel lounge or
> pub :-) or something.
> 
> Another option is that ST organizes a separate brainstorm session with a
> few core developers. We done that in the past quite successfully.
> 
> Regards,
> 
> 	Hans
> 

Re: [media-workshop] Agenda for the Edinburgh mini-summit

[Laurent Pinchart]

Hi Pawel,

On Saturday 07 September 2013 18:31:17 Pawel Osciak wrote:
> On Fri, Sep 6, 2013 at 10:45 PM, Laurent Pinchart wrote:
> > On Thursday 05 September 2013 13:37:49 Hugues FRUCHET wrote:
> > > Hi Mauro,
> > > 
> > > For floating point issue, we have not encountered such issue while
> > > integrating various codec (currently H264, MPEG4, VP8 of both Google G1
> > > IP & ST IPs), could you precise which codec you experienced which
> > > required FP support ?
> > > 
> > > For user-space library, problem we encountered is that interface between
> > > parsing side (for ex. H264 SPS/PPS decoding, slice header decoding,
> > > references frame list management, ...moreover all that is needed to
> > > prepare hardware IPs call) and decoder side (hardware IPs handling) is
> > > not standardized and differs largely regarding IPs or CPU/copro
> > > partitioning. This means that even if we use the standard V4L2 capture
> > > interface to inject video bitstream (H264 access units for ex), some
> > > proprietary meta are needed to be attached to each buffers, making de
> > > facto "un-standard" the V4L2 interface for this driver.
> > 
> > We're working on APIs to pass meta data from/to the kernel. The necessary
> > infrastructure is more or less there already, we "just" need to agree on
> > guidelines and standardize the process. One option that will likely be
> > implemented is to store meta-data in a plane, using the multiplanar API.
> 
> What API is that? Is there an RFC somewhere?

It has been discussed recently as part of the frame descriptors RFC 
(http://www.spinics.net/lists/linux-media/msg67295.html).

> > The resulting plane format will be driver-specific, so we'll loose part of
> > the benefits that the V4L2 API provides. We could try to solve this by
> > writing a libv4l plugin, specific to your driver, that would handle
> > bitstream parsing and fill the meta-data planes correctly. Applications
> > using libv4l would thus only need to pass encoded frames to the library,
> > which would create multiplanar buffers with video data and meta-data, and
> > pass them to the driver. This would be fully transparent for the
> > application.
> 
> If V4L2 API is not hardware-independent, it's a big loss. If this happens,
> there will be need for another, middleware API, like OMX IL. This makes V4L2
> by itself impractical for real world applications. And the incentives of
> using V4L2 are gone, because it's much easier to write a custom DRM driver
> and add any userspace API on top of it. Perhaps this is inevitable, given
> differences in hardware, but a plugin approach would be a way to keep V4L2
> abstract and retain the ability to do the bulk of processing in userspace...

I believe we can reach that goal with libv4l. The V4L2 kernel API can't 
abstract all hardware features, as this would require an API level that can't 
be properly implemented in kernel space, but with libv4l to the rescue we 
should be pretty good.

-- 
Regards,

Laurent Pinchart