[RFC] docs: dev-decoder: Add two more reasons for dynamic change

[RFC] docs: dev-decoder: Add two more reasons for dynamic change

[Stanimir Varbanov]

Here we add two more reasons for dynamic-resolution-change state
(I think the name is misleading espesially "resolution" word, maybe
dynamic-bitstream-change is better to describe).

The first one which could change in the middle of the stream is the
bit-depth. For worst example the stream is 8bit at the begging but
later in the bitstream it changes to 10bit. That change should be
propagated to the client so that it can take appropriate  action. In
this case most probably it has to stop the streaming on the capture
queue and re-negotiate the pixel format and start the streaming
again.

The second new reason is colorspace change. I'm not sure what action
client should take but at least it should be notified for such change.
One possible action is to notify the display entity that the colorspace
and its parameters (y'cbcr encoding and so on) has been changed.

Signed-off-by: Stanimir Varbanov <stanimir.varbanov@linaro.org>
---
 Documentation/userspace-api/media/v4l/dev-decoder.rst | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/Documentation/userspace-api/media/v4l/dev-decoder.rst b/Documentation/userspace-api/media/v4l/dev-decoder.rst
index 606b54947e10..bf10eda6125c 100644
--- a/Documentation/userspace-api/media/v4l/dev-decoder.rst
+++ b/Documentation/userspace-api/media/v4l/dev-decoder.rst
@@ -906,7 +906,11 @@ reflected by corresponding queries):
 
 * visible resolution (selection rectangles),
 
-* the minimum number of buffers needed for decoding.
+* the minimum number of buffers needed for decoding,
+
+* bit-depth of the bitstream has been changed,
+
+* colorspace (and its properties) has been changed.
 
 Whenever that happens, the decoder must proceed as follows:
 
-- 
2.17.1

Re: [RFC] docs: dev-decoder: Add two more reasons for dynamic change

[Nicolas Dufresne]

Le jeudi 30 avril 2020 à 14:38 +0300, Stanimir Varbanov a écrit :
> Here we add two more reasons for dynamic-resolution-change state
> (I think the name is misleading espesially "resolution" word, maybe
> dynamic-bitstream-change is better to describe).
> 
> The first one which could change in the middle of the stream is the
> bit-depth. For worst example the stream is 8bit at the begging but
> later in the bitstream it changes to 10bit. That change should be
> propagated to the client so that it can take appropriate  action. In
> this case most probably it has to stop the streaming on the capture
> queue and re-negotiate the pixel format and start the streaming
> again.
> 
> The second new reason is colorspace change. I'm not sure what action
> client should take but at least it should be notified for such change.
> One possible action is to notify the display entity that the colorspace
> and its parameters (y'cbcr encoding and so on) has been changed.

Just to help with the use case, colorspace changes need to be
communicated to the following HW or software in your media pipeline.
Let's consider a V4L2 only use case:

  m2m decoder -> m2m color transform - >...

The userspace needs to be aware on time, so that it can reconfigure the
color transformation parameters. The V4L2 event is a miss-fit though,
as it does not tell exactly which buffer will start having this new
colorspace. So in theory, one would have to:

  - drain
  - send the new csc parameters
  - resume

I'm not sure if our drivers implement resuming after CMD_STOP, do you
have information about that ? We could also go through streamoff/on
cycle in the mean time. Most codec won't allow changing these
parameters on delta frames, as it would force the decoder doing CSC
conversion of the reference frames in decode process, that seems
unrealistically complex requirement.

That being said, please keep in mind that in VP9, reference frames do
not have to be of the same sizes. You can change the resolution at any
point in time. No one managed to decode the related test vectors [0]
with our current event base resolution change notification.

[0] FRM_RESIZE https://www.webmproject.org/vp9/levels/

> 
> Signed-off-by: Stanimir Varbanov <stanimir.varbanov@linaro.org>
> ---
>  Documentation/userspace-api/media/v4l/dev-decoder.rst | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/userspace-api/media/v4l/dev-decoder.rst b/Documentation/userspace-api/media/v4l/dev-decoder.rst
> index 606b54947e10..bf10eda6125c 100644
> --- a/Documentation/userspace-api/media/v4l/dev-decoder.rst
> +++ b/Documentation/userspace-api/media/v4l/dev-decoder.rst
> @@ -906,7 +906,11 @@ reflected by corresponding queries):
>  
>  * visible resolution (selection rectangles),
>  
> -* the minimum number of buffers needed for decoding.
> +* the minimum number of buffers needed for decoding,
> +
> +* bit-depth of the bitstream has been changed,
> +
> +* colorspace (and its properties) has been changed.
>  
>  Whenever that happens, the decoder must proceed as follows:
>  

Re: [RFC] docs: dev-decoder: Add two more reasons for dynamic change

[Tomasz Figa]

On Fri, May 1, 2020 at 4:19 PM Nicolas Dufresne <nicolas@ndufresne.ca> wrote:
>
> Le jeudi 30 avril 2020 à 14:38 +0300, Stanimir Varbanov a écrit :
> > Here we add two more reasons for dynamic-resolution-change state
> > (I think the name is misleading espesially "resolution" word, maybe
> > dynamic-bitstream-change is better to describe).
> >
> > The first one which could change in the middle of the stream is the
> > bit-depth. For worst example the stream is 8bit at the begging but
> > later in the bitstream it changes to 10bit. That change should be
> > propagated to the client so that it can take appropriate  action. In
> > this case most probably it has to stop the streaming on the capture
> > queue and re-negotiate the pixel format and start the streaming
> > again.
> >
> > The second new reason is colorspace change. I'm not sure what action
> > client should take but at least it should be notified for such change.
> > One possible action is to notify the display entity that the colorspace
> > and its parameters (y'cbcr encoding and so on) has been changed.
>
> Just to help with the use case, colorspace changes need to be
> communicated to the following HW or software in your media pipeline.
> Let's consider a V4L2 only use case:
>
>   m2m decoder -> m2m color transform - >...
>
> The userspace needs to be aware on time, so that it can reconfigure the
> color transformation parameters. The V4L2 event is a miss-fit though,
> as it does not tell exactly which buffer will start having this new
> colorspace. So in theory, one would have to:
>
>   - drain
>   - send the new csc parameters
>   - resume
>
> I'm not sure if our drivers implement resuming after CMD_STOP, do you
> have information about that ? We could also go through streamoff/on
> cycle in the mean time. Most codec won't allow changing these
> parameters on delta frames, as it would force the decoder doing CSC
> conversion of the reference frames in decode process, that seems
> unrealistically complex requirement.
>
> That being said, please keep in mind that in VP9, reference frames do
> not have to be of the same sizes. You can change the resolution at any
> point in time. No one managed to decode the related test vectors [0]
> with our current event base resolution change notification.
>
> [0] FRM_RESIZE https://www.webmproject.org/vp9/levels/
>

Agreed. The event mechanism is certainly suffering from some design
issues, but that's just a tip of the iceberg. I think we can only
solve this problem by adding an ability to query driver state on a
per-buffer basis, so one could query the format of a particular
dequeued frame.

Best regards,
Tomasz

> >
> > Signed-off-by: Stanimir Varbanov <stanimir.varbanov@linaro.org>
> > ---
> >  Documentation/userspace-api/media/v4l/dev-decoder.rst | 6 +++++-
> >  1 file changed, 5 insertions(+), 1 deletion(-)
> >
> > diff --git a/Documentation/userspace-api/media/v4l/dev-decoder.rst b/Documentation/userspace-api/media/v4l/dev-decoder.rst
> > index 606b54947e10..bf10eda6125c 100644
> > --- a/Documentation/userspace-api/media/v4l/dev-decoder.rst
> > +++ b/Documentation/userspace-api/media/v4l/dev-decoder.rst
> > @@ -906,7 +906,11 @@ reflected by corresponding queries):
> >
> >  * visible resolution (selection rectangles),
> >
> > -* the minimum number of buffers needed for decoding.
> > +* the minimum number of buffers needed for decoding,
> > +
> > +* bit-depth of the bitstream has been changed,
> > +
> > +* colorspace (and its properties) has been changed.
> >
> >  Whenever that happens, the decoder must proceed as follows:
> >
>

Re: [RFC] docs: dev-decoder: Add two more reasons for dynamic change

[Stanimir Varbanov]

Hi Nicolas,

On 5/1/20 5:19 PM, Nicolas Dufresne wrote:
> Le jeudi 30 avril 2020 à 14:38 +0300, Stanimir Varbanov a écrit :
>> Here we add two more reasons for dynamic-resolution-change state
>> (I think the name is misleading espesially "resolution" word, maybe
>> dynamic-bitstream-change is better to describe).
>>
>> The first one which could change in the middle of the stream is the
>> bit-depth. For worst example the stream is 8bit at the begging but
>> later in the bitstream it changes to 10bit. That change should be
>> propagated to the client so that it can take appropriate  action. In
>> this case most probably it has to stop the streaming on the capture
>> queue and re-negotiate the pixel format and start the streaming
>> again.
>>
>> The second new reason is colorspace change. I'm not sure what action
>> client should take but at least it should be notified for such change.
>> One possible action is to notify the display entity that the colorspace
>> and its parameters (y'cbcr encoding and so on) has been changed.
> 
> Just to help with the use case, colorspace changes need to be
> communicated to the following HW or software in your media pipeline.
> Let's consider a V4L2 only use case:
> 
>   m2m decoder -> m2m color transform - >...
> 
> The userspace needs to be aware on time, so that it can reconfigure the
> color transformation parameters. The V4L2 event is a miss-fit though,
> as it does not tell exactly which buffer will start having this new
> colorspace. So in theory, one would have to:
> 
>   - drain
>   - send the new csc parameters
>   - resume
> 
> I'm not sure if our drivers implement resuming after CMD_STOP, do you

According to the spec, after implicit drain the decoder is stopping and
the client have two options:

1. streamoff -> reconfigure queue -> streamon
2. decoder command start

#2 would be the case with colorspace changes.

> have information about that ? We could also go through streamoff/on
> cycle in the mean time. Most codec won't allow changing these
> parameters on delta frames, as it would force the decoder doing CSC
> conversion of the reference frames in decode process, that seems
> unrealistically complex requirement.

Shouldn't such changes be preceded by IDR (or what is applicable for the
codec)?

> 
> That being said, please keep in mind that in VP9, reference frames do
> not have to be of the same sizes. You can change the resolution at any
> point in time. No one managed to decode the related test vectors [0]
> with our current event base resolution change notification.
> 
> [0] FRM_RESIZE https://www.webmproject.org/vp9/levels/

I'd like to try those test streams.

So, If I understood your comments correctly, the colorspace change event
in stateful decoder spec isn't needed?

> 
>>
>> Signed-off-by: Stanimir Varbanov <stanimir.varbanov@linaro.org>
>> ---
>>  Documentation/userspace-api/media/v4l/dev-decoder.rst | 6 +++++-
>>  1 file changed, 5 insertions(+), 1 deletion(-)
>>
>> diff --git a/Documentation/userspace-api/media/v4l/dev-decoder.rst b/Documentation/userspace-api/media/v4l/dev-decoder.rst
>> index 606b54947e10..bf10eda6125c 100644
>> --- a/Documentation/userspace-api/media/v4l/dev-decoder.rst
>> +++ b/Documentation/userspace-api/media/v4l/dev-decoder.rst
>> @@ -906,7 +906,11 @@ reflected by corresponding queries):
>>  
>>  * visible resolution (selection rectangles),
>>  
>> -* the minimum number of buffers needed for decoding.
>> +* the minimum number of buffers needed for decoding,
>> +
>> +* bit-depth of the bitstream has been changed,
>> +
>> +* colorspace (and its properties) has been changed.
>>  
>>  Whenever that happens, the decoder must proceed as follows:
>>  
> 

-- 
regards,
Stan

Re: [RFC] docs: dev-decoder: Add two more reasons for dynamic change

[Nicolas Dufresne]

Le samedi 02 mai 2020 à 12:33 +0300, Stanimir Varbanov a écrit :
> Hi Nicolas,
> 
> On 5/1/20 5:19 PM, Nicolas Dufresne wrote:
> > Le jeudi 30 avril 2020 à 14:38 +0300, Stanimir Varbanov a écrit :
> > > Here we add two more reasons for dynamic-resolution-change state
> > > (I think the name is misleading espesially "resolution" word, maybe
> > > dynamic-bitstream-change is better to describe).
> > > 
> > > The first one which could change in the middle of the stream is the
> > > bit-depth. For worst example the stream is 8bit at the begging but
> > > later in the bitstream it changes to 10bit. That change should be
> > > propagated to the client so that it can take appropriate  action. In
> > > this case most probably it has to stop the streaming on the capture
> > > queue and re-negotiate the pixel format and start the streaming
> > > again.
> > > 
> > > The second new reason is colorspace change. I'm not sure what action
> > > client should take but at least it should be notified for such change.
> > > One possible action is to notify the display entity that the colorspace
> > > and its parameters (y'cbcr encoding and so on) has been changed.
> > 
> > Just to help with the use case, colorspace changes need to be
> > communicated to the following HW or software in your media pipeline.
> > Let's consider a V4L2 only use case:
> > 
> >   m2m decoder -> m2m color transform - >...
> > 
> > The userspace needs to be aware on time, so that it can reconfigure the
> > color transformation parameters. The V4L2 event is a miss-fit though,
> > as it does not tell exactly which buffer will start having this new
> > colorspace. So in theory, one would have to:
> > 
> >   - drain
> >   - send the new csc parameters
> >   - resume
> > 
> > I'm not sure if our drivers implement resuming after CMD_STOP, do you
> 
> According to the spec, after implicit drain the decoder is stopping and
> the client have two options:
> 
> 1. streamoff -> reconfigure queue -> streamon
> 2. decoder command start
> 
> #2 would be the case with colorspace changes.

Agreed, I just wanted to underline as as no userspace make any use of
that, CMD_START might current be broken in many places. That being
said, if we only use it in the context of a new event, it can't cause
any harm.

> 
> > have information about that ? We could also go through streamoff/on
> > cycle in the mean time. Most codec won't allow changing these
> > parameters on delta frames, as it would force the decoder doing CSC
> > conversion of the reference frames in decode process, that seems
> > unrealistically complex requirement.
> 
> Shouldn't such changes be preceded by IDR (or what is applicable for the
> codec)?

VP9 does not have this limitation. So reference frames and the output
frame can be of different size. It's likely unique to VP9. 

> 
> > That being said, please keep in mind that in VP9, reference frames do
> > not have to be of the same sizes. You can change the resolution at any
> > point in time. No one managed to decode the related test vectors [0]
> > with our current event base resolution change notification.
> > 
> > [0] FRM_RESIZE https://www.webmproject.org/vp9/levels/
> 
> I'd like to try those test streams.

You might also want to be aware of:

http://www.itu.int/net/itu-t/sigdb/spevideo/Hseries-s.htm
https://chromium.googlesource.com/webm/vp8-test-vectors/

Would be an nice and easy project to write a little runner for these
compliance and integrate that into kernel CI. For stateful codecs it is
trivial really. I'm working on GStreamer /GstValidate runners, as we
want something generic across OS and CODEC APIs, but kernelci folks
didn't seem very keen in having a framework on their rootfs (please
correct me if I'm wrong on this).

> 
> So, If I understood your comments correctly, the colorspace change event
> in stateful decoder spec isn't needed?



> 
> > > Signed-off-by: Stanimir Varbanov <stanimir.varbanov@linaro.org>
> > > ---
> > >  Documentation/userspace-api/media/v4l/dev-decoder.rst | 6 +++++-
> > >  1 file changed, 5 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/Documentation/userspace-api/media/v4l/dev-decoder.rst b/Documentation/userspace-api/media/v4l/dev-decoder.rst
> > > index 606b54947e10..bf10eda6125c 100644
> > > --- a/Documentation/userspace-api/media/v4l/dev-decoder.rst
> > > +++ b/Documentation/userspace-api/media/v4l/dev-decoder.rst
> > > @@ -906,7 +906,11 @@ reflected by corresponding queries):
> > >  
> > >  * visible resolution (selection rectangles),
> > >  
> > > -* the minimum number of buffers needed for decoding.
> > > +* the minimum number of buffers needed for decoding,
> > > +
> > > +* bit-depth of the bitstream has been changed,
> > > +
> > > +* colorspace (and its properties) has been changed.
> > >  
> > >  Whenever that happens, the decoder must proceed as follows:
> > >  

Re: [RFC] docs: dev-decoder: Add two more reasons for dynamic change

[Hans Verkuil]

On 30/04/2020 13:38, Stanimir Varbanov wrote:
> Here we add two more reasons for dynamic-resolution-change state
> (I think the name is misleading espesially "resolution" word, maybe

espesially -> especially

> dynamic-bitstream-change is better to describe).
> 
> The first one which could change in the middle of the stream is the
> bit-depth. For worst example the stream is 8bit at the begging but
> later in the bitstream it changes to 10bit. That change should be
> propagated to the client so that it can take appropriate  action. In
> this case most probably it has to stop the streaming on the capture
> queue and re-negotiate the pixel format and start the streaming
> again.
> 
> The second new reason is colorspace change. I'm not sure what action
> client should take but at least it should be notified for such change.
> One possible action is to notify the display entity that the colorspace
> and its parameters (y'cbcr encoding and so on) has been changed.
> 
> Signed-off-by: Stanimir Varbanov <stanimir.varbanov@linaro.org>
> ---
>  Documentation/userspace-api/media/v4l/dev-decoder.rst | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/userspace-api/media/v4l/dev-decoder.rst b/Documentation/userspace-api/media/v4l/dev-decoder.rst
> index 606b54947e10..bf10eda6125c 100644
> --- a/Documentation/userspace-api/media/v4l/dev-decoder.rst
> +++ b/Documentation/userspace-api/media/v4l/dev-decoder.rst
> @@ -906,7 +906,11 @@ reflected by corresponding queries):
>  
>  * visible resolution (selection rectangles),
>  
> -* the minimum number of buffers needed for decoding.
> +* the minimum number of buffers needed for decoding,
> +
> +* bit-depth of the bitstream has been changed,
> +
> +* colorspace (and its properties) has been changed.

For this I want to have a new source change flag:

V4L2_EVENT_SRC_CH_COLORIMETRY

Changing colorimetry without changing resolution/bit depth does not
require buffers to be re-allocated, it just changes how the pixel
data is interpreted w.r.t. color. And that is important to know.

Regards,

	Hans

>  
>  Whenever that happens, the decoder must proceed as follows:
>  
> 

Re: [RFC] docs: dev-decoder: Add two more reasons for dynamic change

[Tomasz Figa]

On Tue, May 26, 2020 at 12:26 PM Hans Verkuil <hverkuil@xs4all.nl> wrote:
>
> On 30/04/2020 13:38, Stanimir Varbanov wrote:
> > Here we add two more reasons for dynamic-resolution-change state
> > (I think the name is misleading espesially "resolution" word, maybe
>
> espesially -> especially
>
> > dynamic-bitstream-change is better to describe).
> >
> > The first one which could change in the middle of the stream is the
> > bit-depth. For worst example the stream is 8bit at the begging but
> > later in the bitstream it changes to 10bit. That change should be
> > propagated to the client so that it can take appropriate  action. In
> > this case most probably it has to stop the streaming on the capture
> > queue and re-negotiate the pixel format and start the streaming
> > again.
> >
> > The second new reason is colorspace change. I'm not sure what action
> > client should take but at least it should be notified for such change.
> > One possible action is to notify the display entity that the colorspace
> > and its parameters (y'cbcr encoding and so on) has been changed.
> >
> > Signed-off-by: Stanimir Varbanov <stanimir.varbanov@linaro.org>
> > ---
> >  Documentation/userspace-api/media/v4l/dev-decoder.rst | 6 +++++-
> >  1 file changed, 5 insertions(+), 1 deletion(-)
> >
> > diff --git a/Documentation/userspace-api/media/v4l/dev-decoder.rst b/Documentation/userspace-api/media/v4l/dev-decoder.rst
> > index 606b54947e10..bf10eda6125c 100644
> > --- a/Documentation/userspace-api/media/v4l/dev-decoder.rst
> > +++ b/Documentation/userspace-api/media/v4l/dev-decoder.rst
> > @@ -906,7 +906,11 @@ reflected by corresponding queries):
> >
> >  * visible resolution (selection rectangles),
> >
> > -* the minimum number of buffers needed for decoding.
> > +* the minimum number of buffers needed for decoding,
> > +
> > +* bit-depth of the bitstream has been changed,
> > +
> > +* colorspace (and its properties) has been changed.
>
> For this I want to have a new source change flag:
>
> V4L2_EVENT_SRC_CH_COLORIMETRY
>
> Changing colorimetry without changing resolution/bit depth does not
> require buffers to be re-allocated, it just changes how the pixel
> data is interpreted w.r.t. color. And that is important to know.

FWIW, the visible resolution (i.e. compose rectangle) change that is
already defined doesn't require buffers to be re-allocated either.
Backwards compatibility requires V4L2_EVENT_SRC_CH_RESOLUTION to be
set, but perhaps we could have further flags introduced, which would
mean visible resolution and stream format (pixelformat, resolution)
exclusively?

Best regards,
Tomasz

Re: [RFC] docs: dev-decoder: Add two more reasons for dynamic change

[Stanimir Varbanov]

Hi Hans,

On 5/26/20 1:26 PM, Hans Verkuil wrote:
> On 30/04/2020 13:38, Stanimir Varbanov wrote:
>> Here we add two more reasons for dynamic-resolution-change state
>> (I think the name is misleading espesially "resolution" word, maybe
> 
> espesially -> especially
> 
>> dynamic-bitstream-change is better to describe).
>>
>> The first one which could change in the middle of the stream is the
>> bit-depth. For worst example the stream is 8bit at the begging but
>> later in the bitstream it changes to 10bit. That change should be
>> propagated to the client so that it can take appropriate  action. In
>> this case most probably it has to stop the streaming on the capture
>> queue and re-negotiate the pixel format and start the streaming
>> again.
>>
>> The second new reason is colorspace change. I'm not sure what action
>> client should take but at least it should be notified for such change.
>> One possible action is to notify the display entity that the colorspace
>> and its parameters (y'cbcr encoding and so on) has been changed.
>>
>> Signed-off-by: Stanimir Varbanov <stanimir.varbanov@linaro.org>
>> ---
>>  Documentation/userspace-api/media/v4l/dev-decoder.rst | 6 +++++-
>>  1 file changed, 5 insertions(+), 1 deletion(-)
>>
>> diff --git a/Documentation/userspace-api/media/v4l/dev-decoder.rst b/Documentation/userspace-api/media/v4l/dev-decoder.rst
>> index 606b54947e10..bf10eda6125c 100644
>> --- a/Documentation/userspace-api/media/v4l/dev-decoder.rst
>> +++ b/Documentation/userspace-api/media/v4l/dev-decoder.rst
>> @@ -906,7 +906,11 @@ reflected by corresponding queries):
>>  
>>  * visible resolution (selection rectangles),
>>  
>> -* the minimum number of buffers needed for decoding.
>> +* the minimum number of buffers needed for decoding,
>> +
>> +* bit-depth of the bitstream has been changed,
>> +
>> +* colorspace (and its properties) has been changed.
> 
> For this I want to have a new source change flag:

OK, I can drop colorimetry and prepare a patch for src_change. Is the
bit-depth one fine?

> 
> V4L2_EVENT_SRC_CH_COLORIMETRY
> 
> Changing colorimetry without changing resolution/bit depth does not
> require buffers to be re-allocated, it just changes how the pixel
> data is interpreted w.r.t. color. And that is important to know.
> 
> Regards,
> 
> 	Hans
> 
>>  
>>  Whenever that happens, the decoder must proceed as follows:
>>  
>>
> 

-- 
regards,
Stan

Re: [RFC] docs: dev-decoder: Add two more reasons for dynamic change

[Hans Verkuil]

On 04/06/2020 16:46, Stanimir Varbanov wrote:
> Hi Hans,
> 
> On 5/26/20 1:26 PM, Hans Verkuil wrote:
>> On 30/04/2020 13:38, Stanimir Varbanov wrote:
>>> Here we add two more reasons for dynamic-resolution-change state
>>> (I think the name is misleading espesially "resolution" word, maybe
>>
>> espesially -> especially
>>
>>> dynamic-bitstream-change is better to describe).
>>>
>>> The first one which could change in the middle of the stream is the
>>> bit-depth. For worst example the stream is 8bit at the begging but
>>> later in the bitstream it changes to 10bit. That change should be
>>> propagated to the client so that it can take appropriate  action. In
>>> this case most probably it has to stop the streaming on the capture
>>> queue and re-negotiate the pixel format and start the streaming
>>> again.
>>>
>>> The second new reason is colorspace change. I'm not sure what action
>>> client should take but at least it should be notified for such change.
>>> One possible action is to notify the display entity that the colorspace
>>> and its parameters (y'cbcr encoding and so on) has been changed.
>>>
>>> Signed-off-by: Stanimir Varbanov <stanimir.varbanov@linaro.org>
>>> ---
>>>  Documentation/userspace-api/media/v4l/dev-decoder.rst | 6 +++++-
>>>  1 file changed, 5 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/Documentation/userspace-api/media/v4l/dev-decoder.rst b/Documentation/userspace-api/media/v4l/dev-decoder.rst
>>> index 606b54947e10..bf10eda6125c 100644
>>> --- a/Documentation/userspace-api/media/v4l/dev-decoder.rst
>>> +++ b/Documentation/userspace-api/media/v4l/dev-decoder.rst
>>> @@ -906,7 +906,11 @@ reflected by corresponding queries):
>>>  
>>>  * visible resolution (selection rectangles),
>>>  
>>> -* the minimum number of buffers needed for decoding.
>>> +* the minimum number of buffers needed for decoding,
>>> +
>>> +* bit-depth of the bitstream has been changed,
>>> +
>>> +* colorspace (and its properties) has been changed.
>>
>> For this I want to have a new source change flag:
> 
> OK, I can drop colorimetry and prepare a patch for src_change. Is the
> bit-depth one fine?

Yes.

	Hans

> 
>>
>> V4L2_EVENT_SRC_CH_COLORIMETRY
>>
>> Changing colorimetry without changing resolution/bit depth does not
>> require buffers to be re-allocated, it just changes how the pixel
>> data is interpreted w.r.t. color. And that is important to know.
>>
>> Regards,
>>
>> 	Hans
>>
>>>  
>>>  Whenever that happens, the decoder must proceed as follows:
>>>  
>>>
>>
> 

Re: [RFC] docs: dev-decoder: Add two more reasons for dynamic change

[Stanimir Varbanov]

Hi Hans,

On 5/26/20 1:26 PM, Hans Verkuil wrote:
> On 30/04/2020 13:38, Stanimir Varbanov wrote:
>> Here we add two more reasons for dynamic-resolution-change state
>> (I think the name is misleading espesially "resolution" word, maybe
> 
> espesially -> especially
> 
>> dynamic-bitstream-change is better to describe).
>>
>> The first one which could change in the middle of the stream is the
>> bit-depth. For worst example the stream is 8bit at the begging but
>> later in the bitstream it changes to 10bit. That change should be
>> propagated to the client so that it can take appropriate  action. In
>> this case most probably it has to stop the streaming on the capture
>> queue and re-negotiate the pixel format and start the streaming
>> again.
>>
>> The second new reason is colorspace change. I'm not sure what action
>> client should take but at least it should be notified for such change.
>> One possible action is to notify the display entity that the colorspace
>> and its parameters (y'cbcr encoding and so on) has been changed.
>>
>> Signed-off-by: Stanimir Varbanov <stanimir.varbanov@linaro.org>
>> ---
>>  Documentation/userspace-api/media/v4l/dev-decoder.rst | 6 +++++-
>>  1 file changed, 5 insertions(+), 1 deletion(-)
>>
>> diff --git a/Documentation/userspace-api/media/v4l/dev-decoder.rst b/Documentation/userspace-api/media/v4l/dev-decoder.rst
>> index 606b54947e10..bf10eda6125c 100644
>> --- a/Documentation/userspace-api/media/v4l/dev-decoder.rst
>> +++ b/Documentation/userspace-api/media/v4l/dev-decoder.rst
>> @@ -906,7 +906,11 @@ reflected by corresponding queries):
>>  
>>  * visible resolution (selection rectangles),
>>  
>> -* the minimum number of buffers needed for decoding.
>> +* the minimum number of buffers needed for decoding,
>> +
>> +* bit-depth of the bitstream has been changed,
>> +
>> +* colorspace (and its properties) has been changed.
> 
> For this I want to have a new source change flag:
> 
> V4L2_EVENT_SRC_CH_COLORIMETRY
> 
> Changing colorimetry without changing resolution/bit depth does not
> require buffers to be re-allocated, it just changes how the pixel
> data is interpreted w.r.t. color. And that is important to know.

I'm going to create a patch for this event, but I started to wonder do
we need new buffer flag for this?

Something like below sequence:

 - client receive SRC_CH_COLORIMETRY event
 - client issue G_FMT(CAPTURE queue)
 - at that point the client has to know the last buffer with previous
colorimetry and thus the buffer with new colorimetry

How the client will know the buffer with new colorimetry?

-- 
regards,
Stan

Re: [RFC] docs: dev-decoder: Add two more reasons for dynamic change

[Hans Verkuil]

On 09/06/2020 11:44, Stanimir Varbanov wrote:
> Hi Hans,
> 
> On 5/26/20 1:26 PM, Hans Verkuil wrote:
>> On 30/04/2020 13:38, Stanimir Varbanov wrote:
>>> Here we add two more reasons for dynamic-resolution-change state
>>> (I think the name is misleading espesially "resolution" word, maybe
>>
>> espesially -> especially
>>
>>> dynamic-bitstream-change is better to describe).
>>>
>>> The first one which could change in the middle of the stream is the
>>> bit-depth. For worst example the stream is 8bit at the begging but
>>> later in the bitstream it changes to 10bit. That change should be
>>> propagated to the client so that it can take appropriate  action. In
>>> this case most probably it has to stop the streaming on the capture
>>> queue and re-negotiate the pixel format and start the streaming
>>> again.
>>>
>>> The second new reason is colorspace change. I'm not sure what action
>>> client should take but at least it should be notified for such change.
>>> One possible action is to notify the display entity that the colorspace
>>> and its parameters (y'cbcr encoding and so on) has been changed.
>>>
>>> Signed-off-by: Stanimir Varbanov <stanimir.varbanov@linaro.org>
>>> ---
>>>  Documentation/userspace-api/media/v4l/dev-decoder.rst | 6 +++++-
>>>  1 file changed, 5 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/Documentation/userspace-api/media/v4l/dev-decoder.rst b/Documentation/userspace-api/media/v4l/dev-decoder.rst
>>> index 606b54947e10..bf10eda6125c 100644
>>> --- a/Documentation/userspace-api/media/v4l/dev-decoder.rst
>>> +++ b/Documentation/userspace-api/media/v4l/dev-decoder.rst
>>> @@ -906,7 +906,11 @@ reflected by corresponding queries):
>>>  
>>>  * visible resolution (selection rectangles),
>>>  
>>> -* the minimum number of buffers needed for decoding.
>>> +* the minimum number of buffers needed for decoding,
>>> +
>>> +* bit-depth of the bitstream has been changed,
>>> +
>>> +* colorspace (and its properties) has been changed.
>>
>> For this I want to have a new source change flag:
>>
>> V4L2_EVENT_SRC_CH_COLORIMETRY
>>
>> Changing colorimetry without changing resolution/bit depth does not
>> require buffers to be re-allocated, it just changes how the pixel
>> data is interpreted w.r.t. color. And that is important to know.
> 
> I'm going to create a patch for this event, but I started to wonder do
> we need new buffer flag for this?
> 
> Something like below sequence:
> 
>  - client receive SRC_CH_COLORIMETRY event
>  - client issue G_FMT(CAPTURE queue)
>  - at that point the client has to know the last buffer with previous
> colorimetry and thus the buffer with new colorimetry
> 
> How the client will know the buffer with new colorimetry?
> 

You still need to drain the capture queue, so only after draining and a
V4L2_DEC_CMD_START (or a STREAMOFF/ON pair) will the new colorimetry become
active. But the big difference with CH_RESOLUTION is that no new capture
buffers have to be allocated for just a colorimetry change.

Regards,

	Hans