All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH] media: coda: avoid starvation on well-compressed data
@ 2020-08-21 11:58 Benjamin Bara - SKIDATA
  2020-08-21 11:59 ` Benjamin Bara - SKIDATA
  0 siblings, 1 reply; 5+ messages in thread
From: Benjamin Bara - SKIDATA @ 2020-08-21 11:58 UTC (permalink / raw)
  To: p.zabel, mchehab; +Cc: linux-media, linux-kernel, Richard Leitner - SKIDATA

The prefetcher requires two 256 byte periods beyond the current one.
However, currently it is only checked if there are at least 512 bytes
beyond the current meta available.
This only works under the assumption that every buffer has a size of
at least 256 bytes.

To ensure that the requirement is fulfilled with buffers < 256 bytes,
the queue head and the queue tail must not be below this threshold.
Otherwise, additional buffers are enqueued to ensure a full window.

Signed-off-by: Benjamin Bara <benjamin.bara@skidata.com>
---
 drivers/media/platform/coda/coda-bit.c | 8 ++++++--
 drivers/media/platform/coda/coda.h     | 1 +
 2 files changed, 7 insertions(+), 2 deletions(-)

diff --git a/drivers/media/platform/coda/coda-bit.c b/drivers/media/platform/coda/coda-bit.c
index b021604eceaa..8158f3b34b36 100644
--- a/drivers/media/platform/coda/coda-bit.c
+++ b/drivers/media/platform/coda/coda-bit.c
@@ -323,7 +323,7 @@ static bool coda_bitstream_try_queue(struct coda_ctx *ctx,
 void coda_fill_bitstream(struct coda_ctx *ctx, struct list_head *buffer_list)
 {
 	struct vb2_v4l2_buffer *src_buf;
-	struct coda_buffer_meta *meta;
+	struct coda_buffer_meta *meta, *last_meta;
 	u32 start;
 
 	if (ctx->bit_stream_param & CODA_BIT_STREAM_END_FLAG)
@@ -343,6 +343,8 @@ void coda_fill_bitstream(struct coda_ctx *ctx, struct list_head *buffer_list)
 		    ctx->num_metas >= ctx->num_internal_frames) {
 			meta = list_first_entry(&ctx->buffer_meta_list,
 						struct coda_buffer_meta, list);
+			last_meta = list_last_entry(&ctx->buffer_meta_list,
+						struct coda_buffer_meta, list);
 
 			/*
 			 * If we managed to fill in at least a full reorder
@@ -352,7 +354,8 @@ void coda_fill_bitstream(struct coda_ctx *ctx, struct list_head *buffer_list)
 			 * the first buffer to fetch, we can safely stop queuing
 			 * in order to limit the decoder drain latency.
 			 */
-			if (coda_bitstream_can_fetch_past(ctx, meta->end))
+			if (!meta->below_threshold && !last_meta->below_threshold &&
+				coda_bitstream_can_fetch_past(ctx, meta->end))
 				break;
 		}
 
@@ -403,6 +406,7 @@ void coda_fill_bitstream(struct coda_ctx *ctx, struct list_head *buffer_list)
 				meta->start = start;
 				meta->end = ctx->bitstream_fifo.kfifo.in;
 				meta->last = src_buf->flags & V4L2_BUF_FLAG_LAST;
+				meta->below_threshold = (meta->end - meta->start) < 256;
 				if (meta->last)
 					coda_dbg(1, ctx, "marking last meta");
 				spin_lock(&ctx->buffer_meta_lock);
diff --git a/drivers/media/platform/coda/coda.h b/drivers/media/platform/coda/coda.h
index b81f3aca9209..6f77553e81b8 100644
--- a/drivers/media/platform/coda/coda.h
+++ b/drivers/media/platform/coda/coda.h
@@ -160,6 +160,7 @@ struct coda_buffer_meta {
 	unsigned int		start;
 	unsigned int		end;
 	bool			last;
+	bool			below_threshold;
 };
 
 /* Per-queue, driver-specific private data */
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 5+ messages in thread

* RE: [PATCH] media: coda: avoid starvation on well-compressed data
  2020-08-21 11:58 [PATCH] media: coda: avoid starvation on well-compressed data Benjamin Bara - SKIDATA
@ 2020-08-21 11:59 ` Benjamin Bara - SKIDATA
  2020-09-18  8:35   ` Benjamin Bara - SKIDATA
  0 siblings, 1 reply; 5+ messages in thread
From: Benjamin Bara - SKIDATA @ 2020-08-21 11:59 UTC (permalink / raw)
  To: p.zabel, mchehab; +Cc: linux-media, linux-kernel, Richard Leitner - SKIDATA

Hi!

Our CODA960 MPEG4 decoder (i.MX6) is starving with small "static" videos.
The problem seems to occur when an I frame of size > 512 is followed by
a P frame with size < 256, e.g. reproducible with a single-color video
(or any other video with little movement) and a GoP > 1.

When hunting the problem down, it seems to be related to commit
c3d996fb03c6539771ad778cd66ff5595bfc263a, concretely
the coda_bitstream_can_fetch_past [1] call in coda_fill_bitstream [2].
In the described case, the function returns true, since the I frame reaches
the 512 byte threshold, however the requirement seems to be 2x 256,
therefore the second period has not enough data with the following P frame,
which leads to a CODA PIC_RUN timeout [3] because of starvation.

The patch enqueues another buffer if the queue-tail is below the threshold.
This ensures that the second 256 byte period is filled up for the next run.

However, if the next frame(s) are also below the threshold, the same happens
in reverse order since multiple metas are required to fill the "current" period
(see failure log below).
Also in this case buffers are enqueued until the window is filled again.

This should cover all cases without looping through the active metas.

Best regards
Benjamin


[1] https://elixir.bootlin.com/linux/latest/source/drivers/media/platform/coda/coda.h#L340
[2] https://elixir.bootlin.com/linux/latest/source/drivers/media/platform/coda/coda-bit.c#L355
[3] https://elixir.bootlin.com/linux/latest/source/drivers/media/platform/coda/coda-common.c#L1540

*Failure Log*:
[   77.217862] coda 2040000.vpu: 0: active metas:
[   77.217872] coda 2040000.vpu: 0: - payload: 107
[   77.217877] coda 2040000.vpu: 0: - payload: 107
[   77.217882] coda 2040000.vpu: 0: - payload: 107
[   77.217887] coda 2040000.vpu: 0: - payload: 107
[   77.217892] coda 2040000.vpu: 0: - payload: 332
[   77.217896] coda 2040000.vpu: 0: want to queue: payload: 824
[   78.236142] coda 2040000.vpu: CODA PIC_RUN timeout


^ permalink raw reply	[flat|nested] 5+ messages in thread

* RE: [PATCH] media: coda: avoid starvation on well-compressed data
  2020-08-21 11:59 ` Benjamin Bara - SKIDATA
@ 2020-09-18  8:35   ` Benjamin Bara - SKIDATA
  2020-09-28 20:06     ` Ezequiel Garcia
  0 siblings, 1 reply; 5+ messages in thread
From: Benjamin Bara - SKIDATA @ 2020-09-18  8:35 UTC (permalink / raw)
  To: p.zabel, mchehab
  Cc: linux-media, linux-kernel, Richard Leitner - SKIDATA, l.stach

Hi all,

there are still cases where the decoder starves.
Also, the failure log at the bottom contradicts with the 2x256 bytes assumption.
When I increase the threshold to 512 bytes, the respective video (and all my other tests) work.

Is it possible that the limitation is 2x512 bytes or is there another "limitation",
beside the "two window" one, which is "accidentally fulfilled" with this approach?

What do you think about the approach in the patch; is it possible to get it mainline?
Any help, feedback, hints or suggestions would be really appreciated!

I will do some additional testing to see if the 2x512 threshold finally catches the problem.
When I'm done, I will provide a second version of the patch.

Many thanks & best regards
Benjamin

*Failure Log:*
[  108.108711] coda 2040000.vpu: 0: active metas:
[  108.108716] coda 2040000.vpu: 0: - payload: 4240
[  108.108721] coda 2040000.vpu: 0: - payload: 900
[  108.108726] coda 2040000.vpu: 0: - payload: 170
[  108.108730] coda 2040000.vpu: 0: - payload: 403
[  108.108734] coda 2040000.vpu: 0: want to queue: payload: 405
[  109.057738] coda 2040000.vpu: CODA PIC_RUN timeout


^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [PATCH] media: coda: avoid starvation on well-compressed data
  2020-09-18  8:35   ` Benjamin Bara - SKIDATA
@ 2020-09-28 20:06     ` Ezequiel Garcia
  2020-10-06 14:56       ` Benjamin Bara - SKIDATA
  0 siblings, 1 reply; 5+ messages in thread
From: Ezequiel Garcia @ 2020-09-28 20:06 UTC (permalink / raw)
  To: Benjamin Bara - SKIDATA
  Cc: p.zabel, mchehab, linux-media, linux-kernel,
	Richard Leitner - SKIDATA, l.stach, Chris Healy,
	Nicolas Dufresne

Hi Benjamin,

On Fri, 18 Sep 2020 at 05:35, Benjamin Bara - SKIDATA
<Benjamin.Bara@skidata.com> wrote:
>
> Hi all,
>
> there are still cases where the decoder starves.
> Also, the failure log at the bottom contradicts with the 2x256 bytes assumption.
> When I increase the threshold to 512 bytes, the respective video (and all my other tests) work.
>
> Is it possible that the limitation is 2x512 bytes or is there another "limitation",
> beside the "two window" one, which is "accidentally fulfilled" with this approach?
>
> What do you think about the approach in the patch; is it possible to get it mainline?
> Any help, feedback, hints or suggestions would be really appreciated!
>
> I will do some additional testing to see if the 2x512 threshold finally catches the problem.
> When I'm done, I will provide a second version of the patch.
>

Thanks for the report. I'm seeing a similar behavior and have been doing
some experiments with your patch, and trying different solutions as well.

My colleague Nicolas pointed out to me that crafting a video
with serie of black frames in QCIF resolution, would trigger the timeout:

gst-launch-1.0 videotestsrc pattern=black num-buffers=300 !
video/x-raw,format=I420,width=176,height=120 ! avenc_mpeg2video !
mpegvideoparse ! mpegtsmux ! filesink location=black-qcif-10s.ts

We are still doing some tests to see if we can come up with a proper
solution. I will keep you posted.

If you have some (public) bitstream that are known to timeout,
and that you could share with me, that would be nice, so I could
have a more complete set of samples.

Thanks,
Ezequiel

> Many thanks & best regards
> Benjamin
>
> *Failure Log:*
> [  108.108711] coda 2040000.vpu: 0: active metas:
> [  108.108716] coda 2040000.vpu: 0: - payload: 4240
> [  108.108721] coda 2040000.vpu: 0: - payload: 900
> [  108.108726] coda 2040000.vpu: 0: - payload: 170
> [  108.108730] coda 2040000.vpu: 0: - payload: 403
> [  108.108734] coda 2040000.vpu: 0: want to queue: payload: 405
> [  109.057738] coda 2040000.vpu: CODA PIC_RUN timeout
>

^ permalink raw reply	[flat|nested] 5+ messages in thread

* RE: [PATCH] media: coda: avoid starvation on well-compressed data
  2020-09-28 20:06     ` Ezequiel Garcia
@ 2020-10-06 14:56       ` Benjamin Bara - SKIDATA
  0 siblings, 0 replies; 5+ messages in thread
From: Benjamin Bara - SKIDATA @ 2020-10-06 14:56 UTC (permalink / raw)
  To: Ezequiel Garcia
  Cc: linux-media, linux-kernel, Richard Leitner - SKIDATA,
	Nicolas Dufresne, Ezequiel Garcia

> -----Original Message-----
> From: Ezequiel Garcia <ezequiel@vanguardiasur.com.ar>
> Sent: Montag, 28. September 2020 22:06

> If you have some (public) bitstream that are known to timeout, and that you could
> share with me, that would be nice, so I could have a more complete set of samples.

Thanks for the response! We only have MPEG4 and H.264 encoded videos in use
and the described problem occurs only with MPEG4.
I analyzed the video streams with ffprobe [1] and found out that the problem occurs
when the video contains a couple of small P frames between key frames.

For my test bitstreams, I basically did the same as your colleague Nicolas,
but additionally I did some variations with ffmpeg [1].
To get really small P frames, only little changes between frames are allowed.
Therefore, a "single color video" seems to be the best choice.
Since the problem seemed to be related to the size of a frame, I varied:
width, height, bitrate and GoP (amount of P frames between key frames).
This should also be possible with your gstreamer pipeline [2, 3].

These things also lead to my "simplistic approach" to solve the issue.
I saw your patch series and will try it as soon as possible.

Regards
Benjamin

[1] https://ffmpeg.org/
[2] https://gstreamer.freedesktop.org/documentation/libav/avenc_mpeg2video.html?gi-language=c#avenc_mpeg2video:gop-size
[3] https://gstreamer.freedesktop.org/documentation/libav/avenc_mpeg2video.html?gi-language=c#avenc_mpeg2video:maxrate

*Get "packet sizes" of video stream:*
ffprobe -hide_banner -select_streams v -show_entries packet=flags,size -of compact video.avi


^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2020-10-06 14:56 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-08-21 11:58 [PATCH] media: coda: avoid starvation on well-compressed data Benjamin Bara - SKIDATA
2020-08-21 11:59 ` Benjamin Bara - SKIDATA
2020-09-18  8:35   ` Benjamin Bara - SKIDATA
2020-09-28 20:06     ` Ezequiel Garcia
2020-10-06 14:56       ` Benjamin Bara - SKIDATA

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.