Re: [RFC PATCH 18/20] lib: image-formats: Add v4l2 formats support

["Ville Syrjälä" <ville.syrjala@linux.intel.com>]

On Thu, Mar 21, 2019 at 05:04:19PM +0100, Paul Kocialkowski wrote:
> Hi,
> 
> Le mercredi 20 mars 2019 à 20:39 +0200, Ville Syrjälä a écrit :
> > On Wed, Mar 20, 2019 at 02:27:59PM -0400, Nicolas Dufresne wrote:
> > > Le mercredi 20 mars 2019 à 18:41 +0200, Ville Syrjälä a écrit :
> > > > On Wed, Mar 20, 2019 at 12:30:25PM -0400, Nicolas Dufresne wrote:
> > > > > Le mercredi 20 mars 2019 à 18:09 +0200, Ville Syrjälä a écrit :
> > > > > > On Wed, Mar 20, 2019 at 11:51:58AM -0400, Nicolas Dufresne wrote:
> > > > > > > Le mercredi 20 mars 2019 à 16:27 +0200, Ville Syrjälä a écrit :
> > > > > > > > On Tue, Mar 19, 2019 at 07:29:18PM -0400, Nicolas Dufresne wrote:
> > > > > > > > > Le mardi 19 mars 2019 à 22:57 +0100, Maxime Ripard a écrit :
> > > > > > > > > > V4L2 uses different fourcc's than DRM, and has a different set of formats.
> > > > > > > > > > For now, let's add the v4l2 fourcc's for the already existing formats.
> > > > > > > > > > 
> > > > > > > > > > Signed-off-by: Maxime Ripard <maxime.ripard@bootlin.com>
> > > > > > > > > > ---
> > > > > > > > > >  include/linux/image-formats.h |  9 +++++-
> > > > > > > > > >  lib/image-formats.c           | 67 ++++++++++++++++++++++++++++++++++++-
> > > > > > > > > >  2 files changed, 76 insertions(+)
> > > > > > > > > > 
> > > > > > > > > > diff --git a/include/linux/image-formats.h b/include/linux/image-formats.h
> > > > > > > > > > index 53fd73a71b3d..fbc3a4501ebd 100644
> > > > > > > > > > --- a/include/linux/image-formats.h
> > > > > > > > > > +++ b/include/linux/image-formats.h
> > > > > > > > > > @@ -26,6 +26,13 @@ struct image_format_info {
> > > > > > > > > >  	};
> > > > > > > > > >  
> > > > > > > > > >  	/**
> > > > > > > > > > +	 * @v4l2_fmt:
> > > > > > > > > > +	 *
> > > > > > > > > > +	 * V4L2 4CC format identifier (V4L2_PIX_FMT_*)
> > > > > > > > > > +	 */
> > > > > > > > > > +	u32 v4l2_fmt;
> > > > > > > > > > +
> > > > > > > > > > +	/**
> > > > > > > > > >  	 * @depth:
> > > > > > > > > >  	 *
> > > > > > > > > >  	 * Color depth (number of bits per pixel excluding padding bits),
> > > > > > > > > > @@ -222,6 +229,8 @@ image_format_info_is_yuv_sampling_444(const struct image_format_info *info)
> > > > > > > > > >  
> > > > > > > > > >  const struct image_format_info *__image_format_drm_lookup(u32 drm);
> > > > > > > > > >  const struct image_format_info *image_format_drm_lookup(u32 drm);
> > > > > > > > > > +const struct image_format_info *__image_format_v4l2_lookup(u32 v4l2);
> > > > > > > > > > +const struct image_format_info *image_format_v4l2_lookup(u32 v4l2);
> > > > > > > > > >  unsigned int image_format_plane_cpp(const struct image_format_info *format,
> > > > > > > > > >  				    int plane);
> > > > > > > > > >  unsigned int image_format_plane_width(int width,
> > > > > > > > > > diff --git a/lib/image-formats.c b/lib/image-formats.c
> > > > > > > > > > index 9b9a73220c5d..39f1d38ae861 100644
> > > > > > > > > > --- a/lib/image-formats.c
> > > > > > > > > > +++ b/lib/image-formats.c
> > > > > > > > > > @@ -8,6 +8,7 @@
> > > > > > > > > >  static const struct image_format_info formats[] = {
> > > > > > > > > >  	{
> > > > > > > > > >  		.drm_fmt = DRM_FORMAT_C8,
> > > > > > > > > > +		.v4l2_fmt = V4L2_PIX_FMT_GREY,
> > > > > > > > > >  		.depth = 8,
> > > > > > > > > >  		.num_planes = 1,
> > > > > > > > > >  		.cpp = { 1, 0, 0 },
> > > > > > > > > > @@ -15,6 +16,7 @@ static const struct image_format_info formats[] = {
> > > > > > > > > >  		.vsub = 1,
> > > > > > > > > >  	}, {
> > > > > > > > > >  		.drm_fmt = DRM_FORMAT_RGB332,
> > > > > > > > > > +		.v4l2_fmt = V4L2_PIX_FMT_RGB332,
> > > > > > > > > >  		.depth = 8,
> > > > > > > > > >  		.num_planes = 1,
> > > > > > > > > >  		.cpp = { 1, 0, 0 },
> > > > > > > > > > @@ -29,6 +31,7 @@ static const struct image_format_info formats[] = {
> > > > > > > > > >  		.vsub = 1,
> > > > > > > > > >  	}, {
> > > > > > > > > >  		.drm_fmt = DRM_FORMAT_XRGB4444,
> > > > > > > > > > +		.v4l2_fmt = V4L2_PIX_FMT_XRGB444,
> > > > > > > > > >  		.depth = 0,
> > > > > > > > > >  		.num_planes = 1,
> > > > > > > > > >  		.cpp = { 2, 0, 0 },
> > > > > > > > > > @@ -57,6 +60,7 @@ static const struct image_format_info formats[] = {
> > > > > > > > > >  		.vsub = 1,
> > > > > > > > > >  	}, {
> > > > > > > > > >  		.drm_fmt = DRM_FORMAT_ARGB4444,
> > > > > > > > > > +		.v4l2_fmt = V4L2_PIX_FMT_ARGB444,
> > > > > > > > > >  		.depth = 0,
> > > > > > > > > >  		.num_planes = 1,
> > > > > > > > > >  		.cpp = { 2, 0, 0 },
> > > > > > > > > > @@ -89,6 +93,7 @@ static const struct image_format_info formats[] = {
> > > > > > > > > >  		.has_alpha = true,
> > > > > > > > > >  	}, {
> > > > > > > > > >  		.drm_fmt = DRM_FORMAT_XRGB1555,
> > > > > > > > > > +		.v4l2_fmt = V4L2_PIX_FMT_XRGB555,
> > > > > > > > > >  		.depth = 15,
> > > > > > > > > >  		.num_planes = 1,
> > > > > > > > > >  		.cpp = { 2, 0, 0 },
> > > > > > > > > > @@ -117,6 +122,7 @@ static const struct image_format_info formats[] = {
> > > > > > > > > >  		.vsub = 1,
> > > > > > > > > >  	}, {
> > > > > > > > > >  		.drm_fmt = DRM_FORMAT_ARGB1555,
> > > > > > > > > > +		.v4l2_fmt = V4L2_PIX_FMT_ARGB555,
> > > > > > > > > >  		.depth = 15,
> > > > > > > > > >  		.num_planes = 1,
> > > > > > > > > >  		.cpp = { 2, 0, 0 },
> > > > > > > > > > @@ -149,6 +155,7 @@ static const struct image_format_info formats[] = {
> > > > > > > > > >  		.has_alpha = true,
> > > > > > > > > >  	}, {
> > > > > > > > > >  		.drm_fmt = DRM_FORMAT_RGB565,
> > > > > > > > > > +		.v4l2_fmt = V4L2_PIX_FMT_RGB565,
> > > > > > > > > >  		.depth = 16,
> > > > > > > > > >  		.num_planes = 1,
> > > > > > > > > >  		.cpp = { 2, 0, 0 },
> > > > > > > > > > @@ -163,6 +170,7 @@ static const struct image_format_info formats[] = {
> > > > > > > > > >  		.vsub = 1,
> > > > > > > > > >  	}, {
> > > > > > > > > >  		.drm_fmt = DRM_FORMAT_RGB888,
> > > > > > > > > > +		.v4l2_fmt = V4L2_PIX_FMT_RGB24,
> > > > > > > > > >  		.depth = 24,
> > > > > > > > > >  		.num_planes = 1,
> > > > > > > > > >  		.cpp = { 3, 0, 0 },
> > > > > > > > > > @@ -170,6 +178,7 @@ static const struct image_format_info formats[] = {
> > > > > > > > > >  		.vsub = 1,
> > > > > > > > > >  	}, {
> > > > > > > > > >  		.drm_fmt = DRM_FORMAT_BGR888,
> > > > > > > > > > +		.v4l2_fmt = V4L2_PIX_FMT_BGR24,
> > > > > > > > > >  		.depth = 24,
> > > > > > > > > >  		.num_planes = 1,
> > > > > > > > > >  		.cpp = { 3, 0, 0 },
> > > > > > > > > > @@ -177,6 +186,7 @@ static const struct image_format_info formats[] = {
> > > > > > > > > >  		.vsub = 1,
> > > > > > > > > >  	}, {
> > > > > > > > > >  		.drm_fmt = DRM_FORMAT_XRGB8888,
> > > > > > > > > > +		.v4l2_fmt = V4L2_PIX_FMT_XRGB32,
> > > > > > > > > 
> > > > > > > > > All RGB mapping should be surrounded by ifdef, because many (not all)
> > > > > > > > > DRM formats represent the order of component when placed in a CPU
> > > > > > > > > register, unlike V4L2 which uses memory order. I've pick this one
> > > > > > > > 
> > > > > > > > DRM formats are explicitly defined as little endian.
> > > > > > > 
> > > > > > > Yes, that means the same thing. The mapping has nothing to do with the
> > > > > > > buffer bytes order, unlike V4L2 (and most streaming stack) do.
> > > > > > 
> > > > > > It has everything to do with byte order. Little endian means the least
> > > > > > significant byte is stored in the first byte in memory.
> > > > > > 
> > > > > > Based on https://www.kernel.org/doc/html/v4.15/media/uapi/v4l/pixfmt-packed-rgb.html
> > > > > > drm XRGB888 is actually v4l XBGR32, not XRGB32 as claimed by this patch.
> > > > > 
> > > > > That's basically what I said, as it's define for Little Endian rather
> > > > > then buffer order, you have to make the mapping conditional. It
> > > > > basically mean that in memory, the DRM format physically differ
> > > > > depending on CPU endian.
> > > > 
> > > > No. It is always little endian no matter what the CPU is.
> > > 
> > > I'm sorry, this is in your ABI, we don't add layers of ifdef in
> > > userspace code just for the fun of it. If you redefine this now you are
> > > breaking userspace. I agree there is very little to no Big Endian on
> > > DRM side anymore, but what historically was mapped per CPU cannot be
> > > changed by you now.
> > 
> > It was always little endian.
> 
> I'm not sure what it's worth, but there is a "pixel format guide"
> project that is all about matching formats from one API to another: 
> https://afrantzis.com/pixel-format-guide/ (and it has an associated
> tool too).
> 
> On the page about DRM, it seems that they got the word that DRM formats
> are the native pixel order in little-endian systems:
> https://afrantzis.com/pixel-format-guide/drm.html

Looks consistent with the official word in drm_fourcc.h.

$ python3 -m pfg find-compatible V4L2_PIX_FMT_XBGR32 drm
Format: V4L2_PIX_FMT_XBGR32
Is compatible on all systems with:
        DRM_FORMAT_XRGB8888
Is compatible on little-endian systems with:
Is compatible on big-endian systems with:

$ python3 -m pfg find-compatible DRM_FORMAT_XRGB8888 v4l2
Format: DRM_FORMAT_XRGB8888
Is compatible on all systems with:
        V4L2_PIX_FMT_XBGR32
Is compatible on little-endian systems with:
Is compatible on big-endian systems with:

Even works both ways :)

> 
> Perhaps some drivers have been abusing the format definitions, leading
> to the inconsistencies that Nicolas could witness?

That is quite possible, perhaps even likely. No one really
seems interested in making sure big endian systems actually
work properly. I believe the usual approach is to hack
around semi-rnadomly until the correct colors accidentally
appear on the screen.

-- 
Ville Syrjälä
Intel