[PATCH v1] video/fbdev: Atari 2 bpp (STe) palette bugfix

[PATCH v1] video/fbdev: Atari 2 bpp (STe) palette bugfix

[Michael Schmitz]

The code to set the shifter STe palette registers has a long
standing operator precedence bug, manifesting as colors set
on a 2 bits per pixel frame buffer coming up with a distinctive
blue tint.

Add parentheses around the calculation of the per-color palette
data before shifting those into their respective bit field position.

This bug goes back a long way (2.4 days at the very least) so there
won't be a Fixes: tag.

Tested on ARAnyM as well on Falcon030 hardware.

Cc: stable@vger.kernel.org
Reported-by: Geert Uytterhoeven <geert@linux-m68k.org>
Link: https://lore.kernel.org/all/CAMuHMdU3ievhXxKR_xi_v3aumnYW7UNUO6qMdhgfyWTyVSsCkQ@mail.gmail.com
Tested-by: Michael Schmitz <schmitzmic@gmail.com>
Signed-off-by: Michael Schmitz <schmitzmic@gmail.com>
---
 drivers/video/fbdev/atafb.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/drivers/video/fbdev/atafb.c b/drivers/video/fbdev/atafb.c
index e3812a8ff55a..29e650ecfceb 100644
--- a/drivers/video/fbdev/atafb.c
+++ b/drivers/video/fbdev/atafb.c
@@ -1683,9 +1683,9 @@ static int falcon_setcolreg(unsigned int regno, unsigned int red,
 			   ((blue & 0xfc00) >> 8));
 	if (regno < 16) {
 		shifter_tt.color_reg[regno] =
-			(((red & 0xe000) >> 13) | ((red & 0x1000) >> 12) << 8) |
-			(((green & 0xe000) >> 13) | ((green & 0x1000) >> 12) << 4) |
-			((blue & 0xe000) >> 13) | ((blue & 0x1000) >> 12);
+			((((red & 0xe000) >> 13)   | ((red & 0x1000) >> 12)) << 8)   |
+			((((green & 0xe000) >> 13) | ((green & 0x1000) >> 12)) << 4) |
+			   ((blue & 0xe000) >> 13) | ((blue & 0x1000) >> 12);
 		((u32 *)info->pseudo_palette)[regno] = ((red & 0xf800) |
 						       ((green & 0xfc00) >> 5) |
 						       ((blue & 0xf800) >> 11));
@@ -1971,9 +1971,9 @@ static int stste_setcolreg(unsigned int regno, unsigned int red,
 	green >>= 12;
 	if (ATARIHW_PRESENT(EXTD_SHIFTER))
 		shifter_tt.color_reg[regno] =
-			(((red & 0xe) >> 1) | ((red & 1) << 3) << 8) |
-			(((green & 0xe) >> 1) | ((green & 1) << 3) << 4) |
-			((blue & 0xe) >> 1) | ((blue & 1) << 3);
+			((((red & 0xe)   >> 1) | ((red & 1)   << 3)) << 8) |
+			((((green & 0xe) >> 1) | ((green & 1) << 3)) << 4) |
+			  ((blue & 0xe)  >> 1) | ((blue & 1)  << 3);
 	else
 		shifter_tt.color_reg[regno] =
 			((red & 0xe) << 7) |
-- 
2.17.1

Re: [PATCH v1] video/fbdev: Atari 2 bpp (STe) palette bugfix

[Helge Deller]

On 2/16/22 08:26, Michael Schmitz wrote:
> The code to set the shifter STe palette registers has a long
> standing operator precedence bug, manifesting as colors set
> on a 2 bits per pixel frame buffer coming up with a distinctive
> blue tint.
>
> Add parentheses around the calculation of the per-color palette
> data before shifting those into their respective bit field position.
>
> This bug goes back a long way (2.4 days at the very least) so there
> won't be a Fixes: tag.
>
> Tested on ARAnyM as well on Falcon030 hardware.
>
> Cc: stable@vger.kernel.org
> Reported-by: Geert Uytterhoeven <geert@linux-m68k.org>
> Link: https://lore.kernel.org/all/CAMuHMdU3ievhXxKR_xi_v3aumnYW7UNUO6qMdhgfyWTyVSsCkQ@mail.gmail.com
> Tested-by: Michael Schmitz <schmitzmic@gmail.com>
> Signed-off-by: Michael Schmitz <schmitzmic@gmail.com>

applied.
Thanks!
Helge



> ---
>  drivers/video/fbdev/atafb.c | 12 ++++++------
>  1 file changed, 6 insertions(+), 6 deletions(-)
>
> diff --git a/drivers/video/fbdev/atafb.c b/drivers/video/fbdev/atafb.c
> index e3812a8ff55a..29e650ecfceb 100644
> --- a/drivers/video/fbdev/atafb.c
> +++ b/drivers/video/fbdev/atafb.c
> @@ -1683,9 +1683,9 @@ static int falcon_setcolreg(unsigned int regno, unsigned int red,
>  			   ((blue & 0xfc00) >> 8));
>  	if (regno < 16) {
>  		shifter_tt.color_reg[regno] =
> -			(((red & 0xe000) >> 13) | ((red & 0x1000) >> 12) << 8) |
> -			(((green & 0xe000) >> 13) | ((green & 0x1000) >> 12) << 4) |
> -			((blue & 0xe000) >> 13) | ((blue & 0x1000) >> 12);
> +			((((red & 0xe000) >> 13)   | ((red & 0x1000) >> 12)) << 8)   |
> +			((((green & 0xe000) >> 13) | ((green & 0x1000) >> 12)) << 4) |
> +			   ((blue & 0xe000) >> 13) | ((blue & 0x1000) >> 12);
>  		((u32 *)info->pseudo_palette)[regno] = ((red & 0xf800) |
>  						       ((green & 0xfc00) >> 5) |
>  						       ((blue & 0xf800) >> 11));
> @@ -1971,9 +1971,9 @@ static int stste_setcolreg(unsigned int regno, unsigned int red,
>  	green >>= 12;
>  	if (ATARIHW_PRESENT(EXTD_SHIFTER))
>  		shifter_tt.color_reg[regno] =
> -			(((red & 0xe) >> 1) | ((red & 1) << 3) << 8) |
> -			(((green & 0xe) >> 1) | ((green & 1) << 3) << 4) |
> -			((blue & 0xe) >> 1) | ((blue & 1) << 3);
> +			((((red & 0xe)   >> 1) | ((red & 1)   << 3)) << 8) |
> +			((((green & 0xe) >> 1) | ((green & 1) << 3)) << 4) |
> +			  ((blue & 0xe)  >> 1) | ((blue & 1)  << 3);
>  	else
>  		shifter_tt.color_reg[regno] =
>  			((red & 0xe) << 7) |
>

Re: [PATCH v1] video/fbdev: Atari 2 bpp (STe) palette bugfix

[Geert Uytterhoeven]

On Wed, Feb 16, 2022 at 8:26 AM Michael Schmitz <schmitzmic@gmail.com> wrote:
> The code to set the shifter STe palette registers has a long
> standing operator precedence bug, manifesting as colors set
> on a 2 bits per pixel frame buffer coming up with a distinctive
> blue tint.
>
> Add parentheses around the calculation of the per-color palette
> data before shifting those into their respective bit field position.
>
> This bug goes back a long way (2.4 days at the very least) so there
> won't be a Fixes: tag.
>
> Tested on ARAnyM as well on Falcon030 hardware.
>
> Cc: stable@vger.kernel.org
> Reported-by: Geert Uytterhoeven <geert@linux-m68k.org>
> Link: https://lore.kernel.org/all/CAMuHMdU3ievhXxKR_xi_v3aumnYW7UNUO6qMdhgfyWTyVSsCkQ@mail.gmail.com
> Tested-by: Michael Schmitz <schmitzmic@gmail.com>
> Signed-off-by: Michael Schmitz <schmitzmic@gmail.com>

Thanks, fixes the issue for me.
Tested-by: Geert Uytterhoeven <geert@linux-m68k.org>

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds