linux-media.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Nicolas Dufresne <nicolas@ndufresne.ca>
To: Hans Verkuil <hverkuil@xs4all.nl>,
	Linux Media Mailing List <linux-media@vger.kernel.org>,
	Dave Stevenson <dave.stevenson@raspberrypi.org>,
	Boris Brezillon <boris.brezillon@collabora.com>,
	Paul Kocialkowski <paul.kocialkowski@bootlin.com>,
	Stanimir Varbanov <stanimir.varbanov@linaro.org>,
	Philipp Zabel <p.zabel@pengutronix.de>,
	Ezequiel Garcia <ezequiel@collabora.com>,
	Michael Tretter <m.tretter@pengutronix.de>,
	Tomasz Figa <tfiga@chromium.org>,
	Sylwester Nawrocki <snawrocki@kernel.org>
Subject: Re: [RFC] Stateful codecs and requirements for compressed formats
Date: Fri, 28 Jun 2019 12:18:33 -0400	[thread overview]
Message-ID: <a7a20acb4672a13e2f1225d62e5d69baf5e7231d.camel@ndufresne.ca> (raw)
In-Reply-To: <530f28e9-f686-6222-c6cc-9a5207b151f7@xs4all.nl>

[-- Attachment #1: Type: text/plain, Size: 5263 bytes --]

Le vendredi 28 juin 2019 à 16:34 +0200, Hans Verkuil a écrit :
> Hi all,
> 
> I hope I Cc-ed everyone with a stake in this issue.
> 
> One recurring question is how a stateful encoder fills buffers and how a stateful
> decoder consumes buffers.
> 
> The most generic case is that an encoder produces a bitstream and just fills each
> CAPTURE buffer to the brim before continuing with the next buffer.
> 
> I don't think there are drivers that do this, I believe that all drivers just
> output a single compressed frame. For interlaced formats I understand it is either
> one compressed field per buffer, or two compressed fields per buffer (this is
> what I heard, I don't know if this is true).
> 
> In any case, I don't think this is specified anywhere. Please correct me if I am
> wrong.
> 
> The latest stateful codec spec is here:
> 
> https://hverkuil.home.xs4all.nl/codec-api/uapi/v4l/dev-mem2mem.html
> 
> Assuming what I described above is indeed the case, then I think this should
> be documented. I don't know enough if a flag is needed somewhere to describe
> the behavior for interlaced formats, or can we leave this open and have userspace
> detect this?
> 
> For decoders it is more complicated. The stateful decoder spec is written with
> the assumption that userspace can just fill each OUTPUT buffer to the brim with
> the compressed bitstream. I.e., no need to split at frame or other boundaries.
> 
> See section 4.5.1.7 in the spec.
> 
> But I understand that various HW decoders *do* have limitations. I would really
> like to know about those, since that needs to be exposed to userspace somehow.

So in "4.5.1.7. Decoding", there is a bit of confusion. The text speaks
about ordered of frames in capture and output, but the bullet points
stays that output buffers aren't frames. The following note about
timestamps creates more confusion, since it says there is potentially,
it's not very affirmative, timestamp matching that let you detect re-
ordering done by the driver, but no clarification on how the timestamp
are to be handle if the packing is random.

What seems entirely missing in what we discussed, is a per format
clarification for the behaviour of codec. I was assuming the NAL
alignment to be documented for H264 and HEVC format. It make sense to
allow some more flexibility since these formats are bytestream with
startcodes, but to be, full-frame behaviour is what existing userspace
expects and we should make this the defacto default. And if the buffer
size ends up too small (badly predicted), I believe we should use the
source change event to allow handling that. That being said, we have
been able to survive this for a long time.

For VP8 and VP9, which don't really have a bytestream format, I do
assume it's logical to enforce full frames always. But if not, special
care is needed to ensure the driver can reconstruct the full frames,
since a firmware won't be able to parse the frame boundaries. Now, when
I saw you taking over, I thought it was clear that this was only the
common bits of the spec and that a per format specification would be
developed later.

> Specifically, the venus decoder needs to know the resolution of the coded video
> beforehand and it expects a single frame per buffer (how does that work for
> interlaced formats?).

If the firmware works in a 1:1 behaviour, with H264 you may have two AU
to compose 1 frame in interlaced stream (and that may change for each
frame). In HEVC you'd always have two AU.

> 
> Such requirements mean that some userspace parsing is still required, so these
> decoders are not completely stateful.

There was a discussion about the meaning of the stateful/stateless.
This is not strictly related to parsing, the amount of parsing being
affected is a side effect. The stateful decoder HW (or firmware) offer
an interface with streams. It hides the state of the decoded stream. As
a side effect, the HW can only be multiplexed if the firmware handles
that. On the other end, stateless decoder offer an API where you
configure the decoding of a frame (and sometimes a slice). Two
consecutive frames do not have to be part of the same stream, which has
the side effect of allowing application to handle their own
multiplexing.

> 
> Can every codec author give information about their decoder/encoder?
> 
> I'll start off with my virtual codec driver:
> 
> vicodec: the decoder fully parses the bitstream. The encoder produces a single
> compressed frame per buffer. This driver doesn't yet support interlaced formats,
> but when that is added it will encode one field per buffer.

I just wanted to highlight that there is lot of behaviour specific to
the formats here. Specially this last one, since it implies that
capture format will be field = ALTERNATE for interlace decoding (this
is a relatively rare format). So the behaviour here can already be
inferred by the capture format (appart that interlace mode cannot be
enumerated, so for encoding, it's a bit of a pain to guess). And there
is already in the spec the information needed to match the pairs (or
detect lost field).

> 
> Let's see what the results are.
> 
> Regards,
> 
> 	Hans

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

  parent reply	other threads:[~2019-06-28 16:18 UTC|newest]

Thread overview: 18+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-06-28 14:34 Hans Verkuil
2019-06-28 15:21 ` Dave Stevenson
2019-06-28 15:48   ` Nicolas Dufresne
2019-06-29 10:02     ` Dave Stevenson
2019-06-29 12:55       ` Nicolas Dufresne
2019-06-28 16:18 ` Nicolas Dufresne [this message]
2019-06-28 18:09 ` Nicolas Dufresne
2019-07-03  8:46   ` Tomasz Figa
2019-07-03 17:43     ` Nicolas Dufresne
2019-07-10  8:43   ` Hans Verkuil
2019-07-11  1:40     ` Nicolas Dufresne
2019-07-03  8:32 ` Tomasz Figa
2019-07-03 14:46   ` Philipp Zabel
2019-07-03 17:46   ` Nicolas Dufresne
2019-07-10  9:14   ` Hans Verkuil
2019-07-11 12:49     ` Tomasz Figa
2019-07-11  1:42   ` Nicolas Dufresne
2019-07-11 12:47     ` Tomasz Figa

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=a7a20acb4672a13e2f1225d62e5d69baf5e7231d.camel@ndufresne.ca \
    --to=nicolas@ndufresne.ca \
    --cc=boris.brezillon@collabora.com \
    --cc=dave.stevenson@raspberrypi.org \
    --cc=ezequiel@collabora.com \
    --cc=hverkuil@xs4all.nl \
    --cc=linux-media@vger.kernel.org \
    --cc=m.tretter@pengutronix.de \
    --cc=p.zabel@pengutronix.de \
    --cc=paul.kocialkowski@bootlin.com \
    --cc=snawrocki@kernel.org \
    --cc=stanimir.varbanov@linaro.org \
    --cc=tfiga@chromium.org \
    --subject='Re: [RFC] Stateful codecs and requirements for compressed formats' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).