[PATCH] media: imx: Round line size to 4 bytes

[PATCH] media: imx: Round line size to 4 bytes

[Dorota Czaplejewicz]

[-- Attachment #1: Type: text/plain, Size: 1419 bytes --]

Section 13.7.6.13 "CSI Image Parameter Register" of the
i.MX 8M Quad Applications Processors Reference Manual
states that the line size should be divisible by 8 bytes.
However, the hardware also accepts sizes divisible by 4 bytes.

This patch accepts line sizes divisible 4-bytes in non-planar mode.

Signed-off-by: Dorota Czaplejewicz <dorota.czaplejewicz@puri.sm>
---

Hello,

my previous patch identified something that was not a problem,
so I'm sending a different one.

This has been tested on the Librem 5.

Cheers,
Dorota

 drivers/staging/media/imx/imx-media-utils.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/staging/media/imx/imx-media-utils.c b/drivers/staging/media/imx/imx-media-utils.c
index 5128915a5d6f..a303003929e3 100644
--- a/drivers/staging/media/imx/imx-media-utils.c
+++ b/drivers/staging/media/imx/imx-media-utils.c
@@ -545,13 +545,13 @@ int imx_media_mbus_fmt_to_pix_fmt(struct v4l2_pix_format *pix,
 	}
 
 	/* Round up width for minimum burst size */
-	width = round_up(mbus->width, 8);
+	width = round_up(mbus->width, 4);
 
 	/* Round up stride for IDMAC line start address alignment */
 	if (cc->planar)
 		stride = round_up(width, 16);
 	else
-		stride = round_up((width * cc->bpp) >> 3, 8);
+		stride = round_up((width * cc->bpp) >> 3, 4);
 
 	pix->width = width;
 	pix->height = mbus->height;
-- 
2.31.1


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH] media: imx: Round line size to 4 bytes

[Philipp Zabel]

Hi Dorota,

On Wed, 2021-10-06 at 13:05 +0200, Dorota Czaplejewicz wrote:
> Section 13.7.6.13 "CSI Image Parameter Register" of the
> i.MX 8M Quad Applications Processors Reference Manual
> states that the line size should be divisible by 8 bytes.
> However, the hardware also accepts sizes divisible by 4 bytes.
> 
> This patch accepts line sizes divisible 4-bytes in non-planar mode.

Thank you, this makes it much clearer. I see two issues with this,
though, one small and one a bit bigger:

First, I'd be wary of disregarding the reference manual - unless we know
better, and then it should be well documented in the code. It might be
that the 8-byte alignment requirement stems from the fact that the FIFO
operates in double-word units, which might cause the CSI to write over
the end of the buffer if the line width is odd (in 32-bit words).
Or maybe it's just that the FBUF_STRIDE conflicts with this, I'm unclear
on whether that is only given in units of dwords (although the driver
currently doesn't support this anyway).

I wonder: if you use 4-byte aligned width and odd height, does the CSI
write over the end of the buffer?

> Signed-off-by: Dorota Czaplejewicz <dorota.czaplejewicz@puri.sm>
> ---
> 
> Hello,
> 
> my previous patch identified something that was not a problem,
> so I'm sending a different one.
> 
> This has been tested on the Librem 5.
> 
> Cheers,
> Dorota
> 
>  drivers/staging/media/imx/imx-media-utils.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/staging/media/imx/imx-media-utils.c b/drivers/staging/media/imx/imx-media-utils.c
> index 5128915a5d6f..a303003929e3 100644
> --- a/drivers/staging/media/imx/imx-media-utils.c
> +++ b/drivers/staging/media/imx/imx-media-utils.c
> @@ -545,13 +545,13 @@ int imx_media_mbus_fmt_to_pix_fmt(struct v4l2_pix_format *pix,
>  	}
>  
>  	/* Round up width for minimum burst size */
> -	width = round_up(mbus->width, 8);
> +	width = round_up(mbus->width, 4);
>  
>  	/* Round up stride for IDMAC line start address alignment */
>  	if (cc->planar)
>  		stride = round_up(width, 16);
>  	else
> -		stride = round_up((width * cc->bpp) >> 3, 8);
> +		stride = round_up((width * cc->bpp) >> 3, 4);

Second, even if this works fine on the i.MX7/8M CSI, the alignment is
still required for the i.MX5/6 IPU, for which this code and the comments
were written. So we need a way to differentiate the two cases here.

regards
Philipp

Re: [PATCH] media: imx: Round line size to 4 bytes

[Dorota Czaplejewicz]

[-- Attachment #1: Type: text/plain, Size: 3640 bytes --]

Hello,

On Fri, 08 Oct 2021 14:19:41 +0200
Philipp Zabel <p.zabel@pengutronix.de> wrote:

> Hi Dorota,
> 
> On Wed, 2021-10-06 at 13:05 +0200, Dorota Czaplejewicz wrote:
> > Section 13.7.6.13 "CSI Image Parameter Register" of the
> > i.MX 8M Quad Applications Processors Reference Manual
> > states that the line size should be divisible by 8 bytes.
> > However, the hardware also accepts sizes divisible by 4 bytes.
> > 
> > This patch accepts line sizes divisible 4-bytes in non-planar mode.  
> 
> Thank you, this makes it much clearer. I see two issues with this,
> though, one small and one a bit bigger:
> 
> First, I'd be wary of disregarding the reference manual - unless we know
> better, and then it should be well documented in the code. It might be
> that the 8-byte alignment requirement stems from the fact that the FIFO
> operates in double-word units, which might cause the CSI to write over
> the end of the buffer if the line width is odd (in 32-bit words).
> Or maybe it's just that the FBUF_STRIDE conflicts with this, I'm unclear
> on whether that is only given in units of dwords (although the driver
> currently doesn't support this anyway).
> 
> I wonder: if you use 4-byte aligned width and odd height, does the CSI
> write over the end of the buffer?

I tested this case, and found a glitch which suggests the last 4 bytes are ignored:

https://source.puri.sm/Librem5/linux-next/uploads/cfb59e3832431aaa3a69549455502568/image.png

That would be taken care of rounding up towards a number decided at runtime, like:

divisor = 8 >> (mbus->height % 2);
> 
> > Signed-off-by: Dorota Czaplejewicz <dorota.czaplejewicz@puri.sm>
> > ---
> > 
> > Hello,
> > 
> > my previous patch identified something that was not a problem,
> > so I'm sending a different one.
> > 
> > This has been tested on the Librem 5.
> > 
> > Cheers,
> > Dorota
> > 
> >  drivers/staging/media/imx/imx-media-utils.c | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> > 
> > diff --git a/drivers/staging/media/imx/imx-media-utils.c b/drivers/staging/media/imx/imx-media-utils.c
> > index 5128915a5d6f..a303003929e3 100644
> > --- a/drivers/staging/media/imx/imx-media-utils.c
> > +++ b/drivers/staging/media/imx/imx-media-utils.c
> > @@ -545,13 +545,13 @@ int imx_media_mbus_fmt_to_pix_fmt(struct v4l2_pix_format *pix,
> >  	}
> >  
> >  	/* Round up width for minimum burst size */
> > -	width = round_up(mbus->width, 8);
> > +	width = round_up(mbus->width, 4);
> >  
> >  	/* Round up stride for IDMAC line start address alignment */
> >  	if (cc->planar)
> >  		stride = round_up(width, 16);
> >  	else
> > -		stride = round_up((width * cc->bpp) >> 3, 8);
> > +		stride = round_up((width * cc->bpp) >> 3, 4);  
> 
> Second, even if this works fine on the i.MX7/8M CSI, the alignment is
> still required for the i.MX5/6 IPU, for which this code and the comments
> were written. So we need a way to differentiate the two cases here.
> 
> regards
> Philipp

How best to go about this? I can see in the file imx-media-capture.c that there the video device lives in struct capture_priv.vdev.vfd. Would that be the right place to query about the underlying hardware?

Then the following functions would gain a new "small_divisor" parameter:
- imx_media_mbus_fmt_to_pix_fmt (a GPL symbol)
- imx_media_mbus_fmt_to_ipu_image (a GPL symbol)
- __capture_try_fmt

Those would have to extract the device type from struct capture_priv:
- __capture_legacy_try_fmt
- capture_try_fmt_vid_cap
- capture_s_fmt_vid_cap
- capture_init_format

Regards,
Dorota

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH] media: imx: Round line size to 4 bytes

[Philipp Zabel]

Hi Dorota,

On Wed, 2021-10-13 at 11:26 +0200, Dorota Czaplejewicz wrote:
> On Fri, 08 Oct 2021 14:19:41 +0200 Philipp Zabel <p.zabel@pengutronix.de> wrote:
[...]
> > I wonder: if you use 4-byte aligned width and odd height, does the CSI
> > write over the end of the buffer?
> 
> I tested this case, and found a glitch which suggests the last 4 bytes are ignored:
> 
> https://source.puri.sm/Librem5/linux-next/uploads/cfb59e3832431aaa3a69549455502568/image.png

Thank you for testing, so it appears that at least without FBUF_STRIDE
the only requirement is that the whole image size must be a multiple of
8 bytes.

> That would be taken care of rounding up towards a number decided at runtime, like:
> 
> divisor = 8 >> (mbus->height % 2);

Which would then cause the CSI to write past the end of the buffer?

I'd rather make sure that either the number of lines is even or the
width is a multiple of 8 bytes.

> > > Signed-off-by: Dorota Czaplejewicz <dorota.czaplejewicz@puri.sm>
> > > ---
> > > 
> > > Hello,
> > > 
> > > my previous patch identified something that was not a problem,
> > > so I'm sending a different one.
> > > 
> > > This has been tested on the Librem 5.
> > > 
> > > Cheers,
> > > Dorota
> > > 
> > >  drivers/staging/media/imx/imx-media-utils.c | 4 ++--
> > >  1 file changed, 2 insertions(+), 2 deletions(-)
> > > 
> > > diff --git a/drivers/staging/media/imx/imx-media-utils.c b/drivers/staging/media/imx/imx-media-utils.c
> > > index 5128915a5d6f..a303003929e3 100644
> > > --- a/drivers/staging/media/imx/imx-media-utils.c
> > > +++ b/drivers/staging/media/imx/imx-media-utils.c
> > > @@ -545,13 +545,13 @@ int imx_media_mbus_fmt_to_pix_fmt(struct v4l2_pix_format *pix,
> > >  	}
> > >  
> > >  	/* Round up width for minimum burst size */
> > > -	width = round_up(mbus->width, 8);
> > > +	width = round_up(mbus->width, 4);
> > >  
> > >  	/* Round up stride for IDMAC line start address alignment */
> > >  	if (cc->planar)
> > >  		stride = round_up(width, 16);
> > >  	else
> > > -		stride = round_up((width * cc->bpp) >> 3, 8);
> > > +		stride = round_up((width * cc->bpp) >> 3, 4);  
> > 
> > Second, even if this works fine on the i.MX7/8M CSI, the alignment is
> > still required for the i.MX5/6 IPU, for which this code and the comments
> > were written. So we need a way to differentiate the two cases here.
> > 
> > regards
> > Philipp
> 
> How best to go about this? I can see in the file imx-media-capture.c
> that there the video device lives in struct capture_priv.vdev.vfd.
> Would that be the right place to query about the underlying hardware?
> 
> Then the following functions would gain a new "small_divisor" parameter:
> - imx_media_mbus_fmt_to_pix_fmt (a GPL symbol)
> - imx_media_mbus_fmt_to_ipu_image (a GPL symbol)
> - __capture_try_fmt

That sounds like it would work around the current code when it (at least
part of imx_media_mbus_fmt_to_pix_fmt()) should be split between i.MX5/6
and i.MX7/8 implementations. For example rounding up the stride is not
useful on i.MX7/8, it just doesn't currently hurt because imx7-media-csi 
is not using bytesperline to set up FBUF_STRIDE. And certainly the
comments don't apply.

imx_media_mbus_fmt_to_ipu_image() is unused and should probably be
dropped, same as imx_media_ipu_image_to_mbus_fmt().

> Those would have to extract the device type from struct capture_priv:
> - __capture_legacy_try_fmt
> - capture_try_fmt_vid_cap
> - capture_s_fmt_vid_cap
> - capture_init_format

Maybe imx_media_mbus_fmt_to_pix_fmt should be moved into imx-media-
capture.c be passed struct capture_priv to avoid duplicating the device
type check?

imx_media_capture_device_init() could gain a new parameter (or maybe
replace legacy_api) to set the device type.

regards
Philipp

Re: [PATCH] media: imx: Round line size to 4 bytes

[Dorota Czaplejewicz]

[-- Attachment #1: Type: text/plain, Size: 4673 bytes --]

Hello,

On Thu, 14 Oct 2021 13:26:26 +0200
Philipp Zabel <p.zabel@pengutronix.de> wrote:

> Hi Dorota,
> 
> On Wed, 2021-10-13 at 11:26 +0200, Dorota Czaplejewicz wrote:
> > On Fri, 08 Oct 2021 14:19:41 +0200 Philipp Zabel <p.zabel@pengutronix.de> wrote:  
> [...]
> > > I wonder: if you use 4-byte aligned width and odd height, does the CSI
> > > write over the end of the buffer?  
> > 
> > I tested this case, and found a glitch which suggests the last 4 bytes are ignored:
> > 
> > https://source.puri.sm/Librem5/linux-next/uploads/cfb59e3832431aaa3a69549455502568/image.png  
> 
> Thank you for testing, so it appears that at least without FBUF_STRIDE
> the only requirement is that the whole image size must be a multiple of
> 8 bytes.
> 
> > That would be taken care of rounding up towards a number decided at runtime, like:
> > 
> > divisor = 8 >> (mbus->height % 2);  
> 
> Which would then cause the CSI to write past the end of the buffer?
> 
I'm not sure if you point out the mistake here (should be "4 <<"), or the fact that rounding is happening. If it's the latter, then it's of no concern: the values derived here are used to calculate buffer size.

I'm submitting a new series where this is fixed.

> I'd rather make sure that either the number of lines is even or the
> width is a multiple of 8 bytes.
> 
> > > > Signed-off-by: Dorota Czaplejewicz <dorota.czaplejewicz@puri.sm>
> > > > ---
> > > > 
> > > > Hello,
> > > > 
> > > > my previous patch identified something that was not a problem,
> > > > so I'm sending a different one.
> > > > 
> > > > This has been tested on the Librem 5.
> > > > 
> > > > Cheers,
> > > > Dorota
> > > > 
> > > >  drivers/staging/media/imx/imx-media-utils.c | 4 ++--
> > > >  1 file changed, 2 insertions(+), 2 deletions(-)
> > > > 
> > > > diff --git a/drivers/staging/media/imx/imx-media-utils.c b/drivers/staging/media/imx/imx-media-utils.c
> > > > index 5128915a5d6f..a303003929e3 100644
> > > > --- a/drivers/staging/media/imx/imx-media-utils.c
> > > > +++ b/drivers/staging/media/imx/imx-media-utils.c
> > > > @@ -545,13 +545,13 @@ int imx_media_mbus_fmt_to_pix_fmt(struct v4l2_pix_format *pix,
> > > >  	}
> > > >  
> > > >  	/* Round up width for minimum burst size */
> > > > -	width = round_up(mbus->width, 8);
> > > > +	width = round_up(mbus->width, 4);
> > > >  
> > > >  	/* Round up stride for IDMAC line start address alignment */
> > > >  	if (cc->planar)
> > > >  		stride = round_up(width, 16);
> > > >  	else
> > > > -		stride = round_up((width * cc->bpp) >> 3, 8);
> > > > +		stride = round_up((width * cc->bpp) >> 3, 4);    
> > > 
> > > Second, even if this works fine on the i.MX7/8M CSI, the alignment is
> > > still required for the i.MX5/6 IPU, for which this code and the comments
> > > were written. So we need a way to differentiate the two cases here.
> > > 
> > > regards
> > > Philipp  
> > 
> > How best to go about this? I can see in the file imx-media-capture.c
> > that there the video device lives in struct capture_priv.vdev.vfd.
> > Would that be the right place to query about the underlying hardware?
> > 
> > Then the following functions would gain a new "small_divisor" parameter:
> > - imx_media_mbus_fmt_to_pix_fmt (a GPL symbol)
> > - imx_media_mbus_fmt_to_ipu_image (a GPL symbol)
> > - __capture_try_fmt  
> 
> That sounds like it would work around the current code when it (at least
> part of imx_media_mbus_fmt_to_pix_fmt()) should be split between i.MX5/6
> and i.MX7/8 implementations. For example rounding up the stride is not
> useful on i.MX7/8, it just doesn't currently hurt because imx7-media-csi 
> is not using bytesperline to set up FBUF_STRIDE. And certainly the
> comments don't apply.
> 
> imx_media_mbus_fmt_to_ipu_image() is unused and should probably be
> dropped, same as imx_media_ipu_image_to_mbus_fmt().

Done in next series.
> 
> > Those would have to extract the device type from struct capture_priv:
> > - __capture_legacy_try_fmt
> > - capture_try_fmt_vid_cap
> > - capture_s_fmt_vid_cap
> > - capture_init_format  
> 
> Maybe imx_media_mbus_fmt_to_pix_fmt should be moved into imx-media-
> capture.c be passed struct capture_priv to avoid duplicating the device
> type check?
> 
I opted not to, in favor of passing the actual device type. It comes out to the same thing, except a simple value is passed around instead of a device.

> imx_media_capture_device_init() could gain a new parameter (or maybe
> replace legacy_api) to set the device type.

Thanks, this is what I was missing.

Regards,
Dorota
> 
> regards
> Philipp


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH] media: imx: Round line size to 4 bytes

[Philipp Zabel]

On Sun, 2021-10-17 at 13:07 +0200, Dorota Czaplejewicz wrote:
> Hello,
> 
> On Thu, 14 Oct 2021 13:26:26 +0200
> Philipp Zabel <p.zabel@pengutronix.de> wrote:
> 
> > Hi Dorota,
> > 
> > On Wed, 2021-10-13 at 11:26 +0200, Dorota Czaplejewicz wrote:
> > > On Fri, 08 Oct 2021 14:19:41 +0200 Philipp Zabel <p.zabel@pengutronix.de> wrote:  
> > [...]
> > > > I wonder: if you use 4-byte aligned width and odd height, does the CSI
> > > > write over the end of the buffer?  
> > > 
> > > I tested this case, and found a glitch which suggests the last 4 bytes are ignored:
> > > 
> > > https://source.puri.sm/Librem5/linux-next/uploads/cfb59e3832431aaa3a69549455502568/image.png  
> > 
> > Thank you for testing, so it appears that at least without FBUF_STRIDE
> > the only requirement is that the whole image size must be a multiple of
> > 8 bytes.
> > 
> > > That would be taken care of rounding up towards a number decided at runtime, like:
> > > 
> > > divisor = 8 >> (mbus->height % 2);  
> > 
> > Which would then cause the CSI to write past the end of the buffer?
> > 
> I'm not sure if you point out the mistake here (should be "4 <<"), or
> the fact that rounding is happening. If it's the latter, then it's of
> no concern: the values derived here are used to calculate buffer size.
> 
> I'm submitting a new series where this is fixed.

Thanks, I just didn't understand your intention. The name "divisor"
threw me off, see the comment in the new series.

regards
Philipp