[PATCH] rcar-vin: Do not enumerate unsupported pixel formats

[PATCH] rcar-vin: Do not enumerate unsupported pixel formats

[Niklas Söderlund]

If a pixel format is not supported by the hardware NULL is returned by
rvin_format_from_pixel() for that fourcc. Verify that the pixel format
is supported using this or skip it when enumerating.

Signed-off-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
---
 drivers/media/platform/rcar-vin/rcar-v4l2.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/drivers/media/platform/rcar-vin/rcar-v4l2.c b/drivers/media/platform/rcar-vin/rcar-v4l2.c
index cbc1c07f0a9631a4..ba08f6c49956e899 100644
--- a/drivers/media/platform/rcar-vin/rcar-v4l2.c
+++ b/drivers/media/platform/rcar-vin/rcar-v4l2.c
@@ -302,10 +302,20 @@ static int rvin_g_fmt_vid_cap(struct file *file, void *priv,
 static int rvin_enum_fmt_vid_cap(struct file *file, void *priv,
 				 struct v4l2_fmtdesc *f)
 {
+	struct rvin_dev *vin = video_drvdata(file);
+	unsigned int i, skip = 0;
+
 	if (f->index >= ARRAY_SIZE(rvin_formats))
 		return -EINVAL;
 
-	f->pixelformat = rvin_formats[f->index].fourcc;
+	for (i = 0; i <= f->index; i++)
+		if (!rvin_format_from_pixel(vin, rvin_formats[i].fourcc))
+			skip++;
+
+	if (f->index + skip >= ARRAY_SIZE(rvin_formats))
+		return -EINVAL;
+
+	f->pixelformat = rvin_formats[f->index + skip].fourcc;
 
 	return 0;
 }
-- 
2.23.0

Re: [PATCH] rcar-vin: Do not enumerate unsupported pixel formats

[Laurent Pinchart]

Hi Niklas,

Thank you for the patch.

On Fri, Sep 06, 2019 at 04:35:00PM +0200, Niklas Söderlund wrote:
> If a pixel format is not supported by the hardware NULL is returned by
> rvin_format_from_pixel() for that fourcc. Verify that the pixel format
> is supported using this or skip it when enumerating.
> 
> Signed-off-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
> ---
>  drivers/media/platform/rcar-vin/rcar-v4l2.c | 12 +++++++++++-
>  1 file changed, 11 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/media/platform/rcar-vin/rcar-v4l2.c b/drivers/media/platform/rcar-vin/rcar-v4l2.c
> index cbc1c07f0a9631a4..ba08f6c49956e899 100644
> --- a/drivers/media/platform/rcar-vin/rcar-v4l2.c
> +++ b/drivers/media/platform/rcar-vin/rcar-v4l2.c
> @@ -302,10 +302,20 @@ static int rvin_g_fmt_vid_cap(struct file *file, void *priv,
>  static int rvin_enum_fmt_vid_cap(struct file *file, void *priv,
>  				 struct v4l2_fmtdesc *f)
>  {
> +	struct rvin_dev *vin = video_drvdata(file);
> +	unsigned int i, skip = 0;

This doesn't seem right. Let's consider, as initial conditions,

rvin_formats = { RGB, unsupported, unsupported, YUV }
ARRAY_SIZE(rvin_formats) == 4
f->index = 1

You want to return YUV.

> +
>  	if (f->index >= ARRAY_SIZE(rvin_formats))
>  		return -EINVAL;

This check will pass.

> -	f->pixelformat = rvin_formats[f->index].fourcc;
> +	for (i = 0; i <= f->index; i++)
> +		if (!rvin_format_from_pixel(vin, rvin_formats[i].fourcc))
> +			skip++;

This loop will have two iterations, i == 0 and i == 1. The second
iteration will increase skip, so skip == 1.

> +
> +	if (f->index + skip >= ARRAY_SIZE(rvin_formats))
> +		return -EINVAL;

This check will pass.

> +
> +	f->pixelformat = rvin_formats[f->index + skip].fourcc;

This will return unsupported format.

>  
>  	return 0;
>  }

-- 
Regards,

Laurent Pinchart