linux-media.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Dave Stevenson <dave.stevenson@raspberrypi.org>
To: Hans Verkuil <hverkuil@xs4all.nl>
Cc: Linux Media Mailing List <linux-media@vger.kernel.org>,
	Nicolas Dufresne <nicolas@ndufresne.ca>,
	Boris Brezillon <boris.brezillon@collabora.com>,
	Paul Kocialkowski <paul.kocialkowski@bootlin.com>,
	Stanimir Varbanov <stanimir.varbanov@linaro.org>,
	Philipp Zabel <p.zabel@pengutronix.de>,
	Ezequiel Garcia <ezequiel@collabora.com>,
	Michael Tretter <m.tretter@pengutronix.de>,
	Tomasz Figa <tfiga@chromium.org>,
	Sylwester Nawrocki <snawrocki@kernel.org>
Subject: Re: [RFC] Stateful codecs and requirements for compressed formats
Date: Fri, 28 Jun 2019 16:21:54 +0100	[thread overview]
Message-ID: <CAAoAYcOa7ngH5pPJze+H25rDQgjeNnpKY=HWQqsGFTTrO5iFgg@mail.gmail.com> (raw)
In-Reply-To: <530f28e9-f686-6222-c6cc-9a5207b151f7@xs4all.nl>

Hi Hans

On Fri, 28 Jun 2019 at 15:34, Hans Verkuil <hverkuil@xs4all.nl> wrote:
>
> Hi all,
>
> I hope I Cc-ed everyone with a stake in this issue.
>
> One recurring question is how a stateful encoder fills buffers and how a stateful
> decoder consumes buffers.
>
> The most generic case is that an encoder produces a bitstream and just fills each
> CAPTURE buffer to the brim before continuing with the next buffer.
>
> I don't think there are drivers that do this, I believe that all drivers just
> output a single compressed frame. For interlaced formats I understand it is either
> one compressed field per buffer, or two compressed fields per buffer (this is
> what I heard, I don't know if this is true).

From the discussion that started this thread, with H264 and similar,
does the V4L2 buffer contain just the frame data, or the SPS/PPS
headers as well.

> In any case, I don't think this is specified anywhere. Please correct me if I am
> wrong.
>
> The latest stateful codec spec is here:
>
> https://hverkuil.home.xs4all.nl/codec-api/uapi/v4l/dev-mem2mem.html
>
> Assuming what I described above is indeed the case, then I think this should
> be documented. I don't know enough if a flag is needed somewhere to describe
> the behavior for interlaced formats, or can we leave this open and have userspace
> detect this?
>
>
> For decoders it is more complicated. The stateful decoder spec is written with
> the assumption that userspace can just fill each OUTPUT buffer to the brim with
> the compressed bitstream. I.e., no need to split at frame or other boundaries.
>
> See section 4.5.1.7 in the spec.
>
> But I understand that various HW decoders *do* have limitations. I would really
> like to know about those, since that needs to be exposed to userspace somehow.
>
> Specifically, the venus decoder needs to know the resolution of the coded video
> beforehand and it expects a single frame per buffer (how does that work for
> interlaced formats?).
>
> Such requirements mean that some userspace parsing is still required, so these
> decoders are not completely stateful.
>
> Can every codec author give information about their decoder/encoder?
>
> I'll start off with my virtual codec driver:
>
> vicodec: the decoder fully parses the bitstream. The encoder produces a single
> compressed frame per buffer. This driver doesn't yet support interlaced formats,
> but when that is added it will encode one field per buffer.

On BCM283x:

The underlying decoder will accept anything, but giving it a single
frame per buffer reduces latency as the bitstream parser gets kicked
earlier. Based on previous discussions I am setting the flag so that
it expects one compressed frame per buffer, but I don't believe it
goes wrong should that not be the case (it'll just waste a bit of
processing effort).
It'll parse the headers and produce a V4L2_EVENT_SOURCE_CHANGE event
should the capture queue format not match the stream parameters.
Interlacing isn't supported yet (it's on the list), but I believe the
hardware produces the equivalent to V4L2_FIELD_INTERLACED_[TB|BT].

The encoder currently spits out the H264 SPS/PPS headers as a separate
V4L2 buffer, and then one compressed frame per V4L2 buffer (provided
the buffer is big enough). Should
V4L2_CID_MPEG_VIDEO_REPEAT_SEQ_HEADER be set, then it will repeat the
headers in an independent V4L2 buffer before each I frame.
I'm quite happy to amend this should we have a decent spec of what is
required. As I've never found a spec it's been trial and error until
now.
There is no interlaced support available.

  Dave

  reply	other threads:[~2019-06-28 15:22 UTC|newest]

Thread overview: 18+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-06-28 14:34 Hans Verkuil
2019-06-28 15:21 ` Dave Stevenson [this message]
2019-06-28 15:48   ` Nicolas Dufresne
2019-06-29 10:02     ` Dave Stevenson
2019-06-29 12:55       ` Nicolas Dufresne
2019-06-28 16:18 ` Nicolas Dufresne
2019-06-28 18:09 ` Nicolas Dufresne
2019-07-03  8:46   ` Tomasz Figa
2019-07-03 17:43     ` Nicolas Dufresne
2019-07-10  8:43   ` Hans Verkuil
2019-07-11  1:40     ` Nicolas Dufresne
2019-07-03  8:32 ` Tomasz Figa
2019-07-03 14:46   ` Philipp Zabel
2019-07-03 17:46   ` Nicolas Dufresne
2019-07-10  9:14   ` Hans Verkuil
2019-07-11 12:49     ` Tomasz Figa
2019-07-11  1:42   ` Nicolas Dufresne
2019-07-11 12:47     ` Tomasz Figa

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAAoAYcOa7ngH5pPJze+H25rDQgjeNnpKY=HWQqsGFTTrO5iFgg@mail.gmail.com' \
    --to=dave.stevenson@raspberrypi.org \
    --cc=boris.brezillon@collabora.com \
    --cc=ezequiel@collabora.com \
    --cc=hverkuil@xs4all.nl \
    --cc=linux-media@vger.kernel.org \
    --cc=m.tretter@pengutronix.de \
    --cc=nicolas@ndufresne.ca \
    --cc=p.zabel@pengutronix.de \
    --cc=paul.kocialkowski@bootlin.com \
    --cc=snawrocki@kernel.org \
    --cc=stanimir.varbanov@linaro.org \
    --cc=tfiga@chromium.org \
    --subject='Re: [RFC] Stateful codecs and requirements for compressed formats' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).