* [PATCH] vivid: fix kernel oops when enabling HFLIP and OSD
@ 2018-10-08 19:08 Hans Verkuil
2018-10-08 19:20 ` Hans Verkuil
2018-10-08 22:16 ` Hans Verkuil
0 siblings, 2 replies; 3+ messages in thread
From: Hans Verkuil @ 2018-10-08 19:08 UTC (permalink / raw)
To: Linux Media Mailing List, Mauro Carvalho Chehab
When the OSD is on (i.e. vivid displays text on top of the test pattern), and
you enable hflip, then the driver crashes.
The cause turned out to be a division of a negative number by an unsigned value.
You expect that -8 / 2 would be -4, but in reality it is 2147483644 :-(
Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
Reported-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
---
diff --git a/drivers/media/common/v4l2-tpg/v4l2-tpg-core.c b/drivers/media/common/v4l2-tpg/v4l2-tpg-core.c
index f3d9c1140ffa..e76f87dc4368 100644
--- a/drivers/media/common/v4l2-tpg/v4l2-tpg-core.c
+++ b/drivers/media/common/v4l2-tpg/v4l2-tpg-core.c
@@ -1773,7 +1773,7 @@ typedef struct { u16 __; u8 _; } __packed x24;
pos[7] = (chr & (0x01 << 0) ? fg : bg); \
} \
\
- pos += (tpg->hflip ? -8 : 8) / hdiv; \
+ pos += (tpg->hflip ? -8 : 8) / (int)hdiv; \
} \
} \
} while (0)
^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: [PATCH] vivid: fix kernel oops when enabling HFLIP and OSD
2018-10-08 19:08 [PATCH] vivid: fix kernel oops when enabling HFLIP and OSD Hans Verkuil
@ 2018-10-08 19:20 ` Hans Verkuil
2018-10-08 22:16 ` Hans Verkuil
1 sibling, 0 replies; 3+ messages in thread
From: Hans Verkuil @ 2018-10-08 19:20 UTC (permalink / raw)
To: Linux Media Mailing List, Mauro Carvalho Chehab
On 10/08/2018 09:08 PM, Hans Verkuil wrote:
> When the OSD is on (i.e. vivid displays text on top of the test pattern), and
> you enable hflip, then the driver crashes.
>
> The cause turned out to be a division of a negative number by an unsigned value.
> You expect that -8 / 2 would be -4, but in reality it is 2147483644 :-(
>
> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
> Reported-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
> ---
> diff --git a/drivers/media/common/v4l2-tpg/v4l2-tpg-core.c b/drivers/media/common/v4l2-tpg/v4l2-tpg-core.c
> index f3d9c1140ffa..e76f87dc4368 100644
> --- a/drivers/media/common/v4l2-tpg/v4l2-tpg-core.c
> +++ b/drivers/media/common/v4l2-tpg/v4l2-tpg-core.c
> @@ -1773,7 +1773,7 @@ typedef struct { u16 __; u8 _; } __packed x24;
> pos[7] = (chr & (0x01 << 0) ? fg : bg); \
> } \
> \
> - pos += (tpg->hflip ? -8 : 8) / hdiv; \
> + pos += (tpg->hflip ? -8 : 8) / (int)hdiv; \
> } \
> } \
> } while (0)
>
This can be CC-ed to stable for 4.7 and up.
It actually broke in 4.1, but it was called vivid-tpg.c at that time.
Regards,
Hans
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [PATCH] vivid: fix kernel oops when enabling HFLIP and OSD
2018-10-08 19:08 [PATCH] vivid: fix kernel oops when enabling HFLIP and OSD Hans Verkuil
2018-10-08 19:20 ` Hans Verkuil
@ 2018-10-08 22:16 ` Hans Verkuil
1 sibling, 0 replies; 3+ messages in thread
From: Hans Verkuil @ 2018-10-08 22:16 UTC (permalink / raw)
To: Linux Media Mailing List, Mauro Carvalho Chehab
On 10/08/2018 09:08 PM, Hans Verkuil wrote:
> When the OSD is on (i.e. vivid displays text on top of the test pattern), and
> you enable hflip, then the driver crashes.
>
> The cause turned out to be a division of a negative number by an unsigned value.
> You expect that -8 / 2 would be -4, but in reality it is 2147483644 :-(
>
> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
> Reported-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
> ---
> diff --git a/drivers/media/common/v4l2-tpg/v4l2-tpg-core.c b/drivers/media/common/v4l2-tpg/v4l2-tpg-core.c
> index f3d9c1140ffa..e76f87dc4368 100644
> --- a/drivers/media/common/v4l2-tpg/v4l2-tpg-core.c
> +++ b/drivers/media/common/v4l2-tpg/v4l2-tpg-core.c
> @@ -1773,7 +1773,7 @@ typedef struct { u16 __; u8 _; } __packed x24;
> pos[7] = (chr & (0x01 << 0) ? fg : bg); \
> } \
> \
> - pos += (tpg->hflip ? -8 : 8) / hdiv; \
> + pos += (tpg->hflip ? -8 : 8) / (int)hdiv; \
> } \
> } \
> } while (0)
>
Fixes: 3e14e7a82c1ef ("vivid-tpg: add hor/vert downsampling support to tpg_gen_text")
Regards,
Hans
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2018-10-09 5:30 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-10-08 19:08 [PATCH] vivid: fix kernel oops when enabling HFLIP and OSD Hans Verkuil
2018-10-08 19:20 ` Hans Verkuil
2018-10-08 22:16 ` Hans Verkuil
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).