linux-media.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Philipp Zabel <p.zabel@pengutronix.de>
To: Tomasz Figa <tfiga@chromium.org>, Hans Verkuil <hverkuil@xs4all.nl>
Cc: Linux Media Mailing List <linux-media@vger.kernel.org>,
	Nicolas Dufresne <nicolas@ndufresne.ca>,
	Dave Stevenson <dave.stevenson@raspberrypi.org>,
	Boris Brezillon <boris.brezillon@collabora.com>,
	Paul Kocialkowski <paul.kocialkowski@bootlin.com>,
	Stanimir Varbanov <stanimir.varbanov@linaro.org>,
	Ezequiel Garcia <ezequiel@collabora.com>,
	Michael Tretter <m.tretter@pengutronix.de>,
	Sylwester Nawrocki <snawrocki@kernel.org>
Subject: Re: [RFC] Stateful codecs and requirements for compressed formats
Date: Wed, 03 Jul 2019 16:46:12 +0200	[thread overview]
Message-ID: <1562165172.4604.11.camel@pengutronix.de> (raw)
In-Reply-To: <CAAFQd5BqUS201QP4KHzmnKi5r+3P_KAa=L9CF3=zyQKypNyVuw@mail.gmail.com>

On Wed, 2019-07-03 at 17:32 +0900, Tomasz Figa wrote:
> Hi Hans,
> 
> On Fri, Jun 28, 2019 at 11:34 PM Hans Verkuil <hverkuil@xs4all.nl> wrote:
> > 
> > Hi all,
> > 
> > I hope I Cc-ed everyone with a stake in this issue.
> > 
> > One recurring question is how a stateful encoder fills buffers and how a stateful
> > decoder consumes buffers.
> > 
> > The most generic case is that an encoder produces a bitstream and just fills each
> > CAPTURE buffer to the brim before continuing with the next buffer.
> > 
> > I don't think there are drivers that do this, I believe that all drivers just
> > output a single compressed frame. For interlaced formats I understand it is either
> > one compressed field per buffer, or two compressed fields per buffer (this is
> > what I heard, I don't know if this is true).
> > 
> > In any case, I don't think this is specified anywhere. Please correct me if I am
> > wrong.
> > 
> > The latest stateful codec spec is here:
> > 
> > https://hverkuil.home.xs4all.nl/codec-api/uapi/v4l/dev-mem2mem.html
> > 
> > Assuming what I described above is indeed the case, then I think this should
> > be documented. I don't know enough if a flag is needed somewhere to describe
> > the behavior for interlaced formats, or can we leave this open and have userspace
> > detect this?
> > 
> 
> From Chromium perspective, we don't have any use case for encoding
> interlaced contents, so we'll be okay with whatever the interested
> parties decide on. :)
> 
> > 
> > For decoders it is more complicated. The stateful decoder spec is written with
> > the assumption that userspace can just fill each OUTPUT buffer to the brim with
> > the compressed bitstream. I.e., no need to split at frame or other boundaries.
> > 
> > See section 4.5.1.7 in the spec.
> > 
> > But I understand that various HW decoders *do* have limitations. I would really
> > like to know about those, since that needs to be exposed to userspace somehow.
> 
> AFAIK mtk-vcodec needs H.264 SPS and PPS to be split into their own
> separate buffers. I believe it also needs 1 buffer to contain exactly
> 1 frame and 1 frame to be fully contained inside 1 buffer.
> 
> Venus also needed 1 buffer to contain exactly 1 frame and 1 frame to
> be fully contained inside 1 buffer. It used to have some specific
> requirements regarding SPS and PPS too, but I think that was fixed in
> the firmware.
> 
> > 
> > Specifically, the venus decoder needs to know the resolution of the coded video
> > beforehand
> 
> I don't think that's true for venus. It does parsing and can detect
> the resolution.
> 
> However that's probably the case for coda...

Yes, it is currently true for the coda driver. But I believe it is not
actually necessary for coda hardware / firmware. I have already started
to split sequence initialization (where the firmare parses the bitstream
headers) from internal frame buffer allocation (which have to match
capture buffers in size), and I think it should be possible to
completely decouple the two and postpone buffer allocation far enough to
allow output stream start without prior knowledge of the resolution.

The decoder coda firmware fully parses the bitstream, but the driver has
to copy it from the external output buffers into an internal bitstream
ringbuffer anyway, and a few workarounds are necessary to make it always
succeed regardless of whether the first buffer presented to it only
contains headers, headers and a very small frame, or enough data to
completely fill the bitstream reader's prefetch buffer. For this the
driver has to parse the NAL start headers to a certain degree.

Due to this bitstream copy in the driver, in theory there are no limits
on how the input data is split into v4l2 buffers, but in practice only
single frame per v4l2 output buffer use cases are actually tested
regularly.

The encoder produces a single compressed frame per buffer. There is no
support for B frames in the firmware, as far as I can tell. There is no
driver support for interlaced formats currently, I'm not sure whether
the firmware supports interlacing.

regards
Philipp

  reply	other threads:[~2019-07-03 14:46 UTC|newest]

Thread overview: 18+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-06-28 14:34 Hans Verkuil
2019-06-28 15:21 ` Dave Stevenson
2019-06-28 15:48   ` Nicolas Dufresne
2019-06-29 10:02     ` Dave Stevenson
2019-06-29 12:55       ` Nicolas Dufresne
2019-06-28 16:18 ` Nicolas Dufresne
2019-06-28 18:09 ` Nicolas Dufresne
2019-07-03  8:46   ` Tomasz Figa
2019-07-03 17:43     ` Nicolas Dufresne
2019-07-10  8:43   ` Hans Verkuil
2019-07-11  1:40     ` Nicolas Dufresne
2019-07-03  8:32 ` Tomasz Figa
2019-07-03 14:46   ` Philipp Zabel [this message]
2019-07-03 17:46   ` Nicolas Dufresne
2019-07-10  9:14   ` Hans Verkuil
2019-07-11 12:49     ` Tomasz Figa
2019-07-11  1:42   ` Nicolas Dufresne
2019-07-11 12:47     ` Tomasz Figa

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1562165172.4604.11.camel@pengutronix.de \
    --to=p.zabel@pengutronix.de \
    --cc=boris.brezillon@collabora.com \
    --cc=dave.stevenson@raspberrypi.org \
    --cc=ezequiel@collabora.com \
    --cc=hverkuil@xs4all.nl \
    --cc=linux-media@vger.kernel.org \
    --cc=m.tretter@pengutronix.de \
    --cc=nicolas@ndufresne.ca \
    --cc=paul.kocialkowski@bootlin.com \
    --cc=snawrocki@kernel.org \
    --cc=stanimir.varbanov@linaro.org \
    --cc=tfiga@chromium.org \
    --subject='Re: [RFC] Stateful codecs and requirements for compressed formats' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).