[PATCH] linux/types.h: Restore the ability to disable sparse endianness checks

[PATCH] linux/types.h: Restore the ability to disable sparse endianness checks

[Bart Van Assche]

The purpose of patch "linux/types.h: enable endian checks for all
sparse builds" was to encourage driver authors to annotate
endianness correctly in their drivers. However, since that patch
went upstream no endianness annotations in drivers have been fixed.
I think that this shows that the followed approach does not work,
probably because several driver authors do not use sparse. Restore
the ability to disable sparse endianness checks such that it
becomes again easy to review other sparse diagnostics for people
who want to analyze drivers they are not the author of.

References: commit 05de97003c77 ("linux/types.h: enable endian checks for all sparse builds")
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
---
 include/uapi/linux/types.h | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/include/uapi/linux/types.h b/include/uapi/linux/types.h
index 41e5914f0a8e..d3dcb0764c45 100644
--- a/include/uapi/linux/types.h
+++ b/include/uapi/linux/types.h
@@ -23,7 +23,11 @@
 #else
 #define __bitwise__
 #endif
+#if !defined(__CHECK_ENDIAN__) || __CHECK_ENDIAN__ != 0
 #define __bitwise __bitwise__
+#else
+#define __bitwise
+#endif
 
 typedef __u16 __bitwise __le16;
 typedef __u16 __bitwise __be16;
-- 
2.14.2

Re: [PATCH] linux/types.h: Restore the ability to disable sparse endianness checks

[Christoph Hellwig]

On Fri, Oct 06, 2017 at 10:23:53AM -0700, Bart Van Assche wrote:
> The purpose of patch "linux/types.h: enable endian checks for all
> sparse builds" was to encourage driver authors to annotate
> endianness correctly in their drivers. However, since that patch
> went upstream no endianness annotations in drivers have been fixed.
> I think that this shows that the followed approach does not work,
> probably because several driver authors do not use sparse. Restore
> the ability to disable sparse endianness checks such that it
> becomes again easy to review other sparse diagnostics for people
> who want to analyze drivers they are not the author of.

So how do we get people to do it?  Out of the sparse checks endianess
warnings are the most useful, together with __user and __iomem.

Re: [PATCH] linux/types.h: Restore the ability to disable sparse endianness checks

[Bart Van Assche]

On Fri, 2017-10-06 at 19:35 +0200, Christoph Hellwig wrote:
> On Fri, Oct 06, 2017 at 10:23:53AM -0700, Bart Van Assche wrote:
> > The purpose of patch "linux/types.h: enable endian checks for all
> > sparse builds" was to encourage driver authors to annotate
> > endianness correctly in their drivers. However, since that patch
> > went upstream no endianness annotations in drivers have been fixed.
> > I think that this shows that the followed approach does not work,
> > probably because several driver authors do not use sparse. Restore
> > the ability to disable sparse endianness checks such that it
> > becomes again easy to review other sparse diagnostics for people
> > who want to analyze drivers they are not the author of.
> 
> So how do we get people to do it?  Out of the sparse checks endianess
> warnings are the most useful, together with __user and __iomem.

Hello Christoph,

That's an excellent question. Do you think it would help if the zero-day
kernel testing infrastructure would check whether kernel patches introduce
new sparse complaints and if this is the case post these as a reply to the
e-mail with the patch that introduced the new sparse warnings?

Bart.

Re: [PATCH] linux/types.h: Restore the ability to disable sparse endianness checks

[Javier González]

[-- Attachment #1: Type: text/plain, Size: 1305 bytes --]

> On 6 Oct 2017, at 19.43, Bart Van Assche <bart.vanassche@wdc.com> wrote:
> 
> On Fri, 2017-10-06 at 19:35 +0200, Christoph Hellwig wrote:
>> On Fri, Oct 06, 2017 at 10:23:53AM -0700, Bart Van Assche wrote:
>>> The purpose of patch "linux/types.h: enable endian checks for all
>>> sparse builds" was to encourage driver authors to annotate
>>> endianness correctly in their drivers. However, since that patch
>>> went upstream no endianness annotations in drivers have been fixed.
>>> I think that this shows that the followed approach does not work,
>>> probably because several driver authors do not use sparse. Restore
>>> the ability to disable sparse endianness checks such that it
>>> becomes again easy to review other sparse diagnostics for people
>>> who want to analyze drivers they are not the author of.
>> 
>> So how do we get people to do it?  Out of the sparse checks endianess
>> warnings are the most useful, together with __user and __iomem.
> 
> Hello Christoph,
> 
> That's an excellent question. Do you think it would help if the zero-day
> kernel testing infrastructure would check whether kernel patches introduce
> new sparse complaints and if this is the case post these as a reply to the
> e-mail with the patch that introduced the new sparse warnings?
> 

+1 to this.

Javier


[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH] linux/types.h: Restore the ability to disable sparse endianness checks

[Michael S. Tsirkin]

On Fri, Oct 06, 2017 at 10:23:53AM -0700, Bart Van Assche wrote:
> The purpose of patch "linux/types.h: enable endian checks for all
> sparse builds" was to encourage driver authors to annotate
> endianness correctly in their drivers. However, since that patch
> went upstream no endianness annotations in drivers have been fixed.
> I think that this shows that the followed approach does not work,
> probably because several driver authors do not use sparse. Restore
> the ability to disable sparse endianness checks such that it
> becomes again easy to review other sparse diagnostics for people
> who want to analyze drivers they are not the author of.
> 
> References: commit 05de97003c77 ("linux/types.h: enable endian checks for all sparse builds")
> Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
> Cc: Christoph Hellwig <hch@lst.de>
> Cc: Linus Torvalds <torvalds@linux-foundation.org>

I still think the new default is good.  You probably want ability to
disable these checks selectively for the specific drivers though. Makes
it easier to spot what needs to be fixed.

> ---
>  include/uapi/linux/types.h | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/include/uapi/linux/types.h b/include/uapi/linux/types.h
> index 41e5914f0a8e..d3dcb0764c45 100644
> --- a/include/uapi/linux/types.h
> +++ b/include/uapi/linux/types.h
> @@ -23,7 +23,11 @@
>  #else
>  #define __bitwise__
>  #endif
> +#if !defined(__CHECK_ENDIAN__) || __CHECK_ENDIAN__ != 0
>  #define __bitwise __bitwise__
> +#else
> +#define __bitwise
> +#endif
>  
>  typedef __u16 __bitwise __le16;
>  typedef __u16 __bitwise __be16;
> -- 
> 2.14.2

Re: [PATCH] linux/types.h: Restore the ability to disable sparse endianness checks

[Bart Van Assche]

On Mon, 2017-10-09 at 16:22 +0300, Michael S. Tsirkin wrote:
> On Fri, Oct 06, 2017 at 10:23:53AM -0700, Bart Van Assche wrote:
> > The purpose of patch "linux/types.h: enable endian checks for all
> > sparse builds" was to encourage driver authors to annotate
> > endianness correctly in their drivers. However, since that patch
> > went upstream no endianness annotations in drivers have been fixed.
> > I think that this shows that the followed approach does not work,
> > probably because several driver authors do not use sparse. Restore
> > the ability to disable sparse endianness checks such that it
> > becomes again easy to review other sparse diagnostics for people
> > who want to analyze drivers they are not the author of.
> > 
> > References: commit 05de97003c77 ("linux/types.h: enable endian checks for all sparse builds")
> > Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
> > Cc: Christoph Hellwig <hch@lst.de>
> > Cc: Linus Torvalds <torvalds@linux-foundation.org>
> 
> I still think the new default is good.  You probably want ability to
> disable these checks selectively for the specific drivers though. Makes
> it easier to spot what needs to be fixed.

Hello MST,

I agree with what you wrote. And what you described is what my patch implements
- keep endianness checking enabled by default and make it possible to disable it
selectively. Does that mean that you agree with the patch I posted?

Thanks,

Bart.

Re: [PATCH] linux/types.h: Restore the ability to disable sparse endianness checks

[Michael S. Tsirkin]

On Mon, Oct 09, 2017 at 03:07:42PM +0000, Bart Van Assche wrote:
> On Mon, 2017-10-09 at 16:22 +0300, Michael S. Tsirkin wrote:
> > On Fri, Oct 06, 2017 at 10:23:53AM -0700, Bart Van Assche wrote:
> > > The purpose of patch "linux/types.h: enable endian checks for all
> > > sparse builds" was to encourage driver authors to annotate
> > > endianness correctly in their drivers. However, since that patch
> > > went upstream no endianness annotations in drivers have been fixed.
> > > I think that this shows that the followed approach does not work,
> > > probably because several driver authors do not use sparse. Restore
> > > the ability to disable sparse endianness checks such that it
> > > becomes again easy to review other sparse diagnostics for people
> > > who want to analyze drivers they are not the author of.
> > > 
> > > References: commit 05de97003c77 ("linux/types.h: enable endian checks for all sparse builds")
> > > Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
> > > Cc: Christoph Hellwig <hch@lst.de>
> > > Cc: Linus Torvalds <torvalds@linux-foundation.org>
> > 
> > I still think the new default is good.  You probably want ability to
> > disable these checks selectively for the specific drivers though. Makes
> > it easier to spot what needs to be fixed.
> 
> Hello MST,
> 
> I agree with what you wrote. And what you described is what my patch implements
> - keep endianness checking enabled by default and make it possible to disable it
> selectively. Does that mean that you agree with the patch I posted?
> 
> Thanks,
> 
> Bart.

I just mean I'd expect a patchset setting the flag for the broken
drivers. Presumably this will help trigger some action.

-- 
MST

Re: [PATCH] linux/types.h: Restore the ability to disable sparse endianness checks

[Bart Van Assche]

On Mon, 2017-10-09 at 20:42 +0300, Michael S. Tsirkin wrote:
> On Mon, Oct 09, 2017 at 03:07:42PM +0000, Bart Van Assche wrote:
> > On Mon, 2017-10-09 at 16:22 +0300, Michael S. Tsirkin wrote:
> > > On Fri, Oct 06, 2017 at 10:23:53AM -0700, Bart Van Assche wrote:
> > > > The purpose of patch "linux/types.h: enable endian checks for all
> > > > sparse builds" was to encourage driver authors to annotate
> > > > endianness correctly in their drivers. However, since that patch
> > > > went upstream no endianness annotations in drivers have been fixed.
> > > > I think that this shows that the followed approach does not work,
> > > > probably because several driver authors do not use sparse. Restore
> > > > the ability to disable sparse endianness checks such that it
> > > > becomes again easy to review other sparse diagnostics for people
> > > > who want to analyze drivers they are not the author of.
> > > > 
> > > > References: commit 05de97003c77 ("linux/types.h: enable endian checks for all sparse builds")
> > > > Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
> > > > Cc: Christoph Hellwig <hch@lst.de>
> > > > Cc: Linus Torvalds <torvalds@linux-foundation.org>
> > > 
> > > I still think the new default is good.  You probably want ability to
> > > disable these checks selectively for the specific drivers though. Makes
> > > it easier to spot what needs to be fixed.
> > 
> > I agree with what you wrote. And what you described is what my patch implements
> > - keep endianness checking enabled by default and make it possible to disable it
> > selectively. Does that mean that you agree with the patch I posted?
> 
> I just mean I'd expect a patchset setting the flag for the broken
> drivers. Presumably this will help trigger some action.

Hello MST,

If I would add something like ccflags-y += -D__CHECK_ENDIAN__=0 to the
Makefile of drivers that are not endianness clean then that would make it
easier for driver authors to ignore endianness warnings reported by sparse.
I prefer that they have to add CF=-D__CHECK_ENDIAN__=0 to the command line
when verifying a driver with sparse to get rid of the sparse endianness
warnings. Do you agree with this?

Thanks,

Bart.

Re: [PATCH] linux/types.h: Restore the ability to disable sparse endianness checks

[Michael S. Tsirkin]

On Tue, Oct 10, 2017 at 04:38:16PM +0000, Bart Van Assche wrote:
> On Mon, 2017-10-09 at 20:42 +0300, Michael S. Tsirkin wrote:
> > On Mon, Oct 09, 2017 at 03:07:42PM +0000, Bart Van Assche wrote:
> > > On Mon, 2017-10-09 at 16:22 +0300, Michael S. Tsirkin wrote:
> > > > On Fri, Oct 06, 2017 at 10:23:53AM -0700, Bart Van Assche wrote:
> > > > > The purpose of patch "linux/types.h: enable endian checks for all
> > > > > sparse builds" was to encourage driver authors to annotate
> > > > > endianness correctly in their drivers. However, since that patch
> > > > > went upstream no endianness annotations in drivers have been fixed.
> > > > > I think that this shows that the followed approach does not work,
> > > > > probably because several driver authors do not use sparse. Restore
> > > > > the ability to disable sparse endianness checks such that it
> > > > > becomes again easy to review other sparse diagnostics for people
> > > > > who want to analyze drivers they are not the author of.
> > > > > 
> > > > > References: commit 05de97003c77 ("linux/types.h: enable endian checks for all sparse builds")
> > > > > Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
> > > > > Cc: Christoph Hellwig <hch@lst.de>
> > > > > Cc: Linus Torvalds <torvalds@linux-foundation.org>
> > > > 
> > > > I still think the new default is good.  You probably want ability to
> > > > disable these checks selectively for the specific drivers though. Makes
> > > > it easier to spot what needs to be fixed.
> > > 
> > > I agree with what you wrote. And what you described is what my patch implements
> > > - keep endianness checking enabled by default and make it possible to disable it
> > > selectively. Does that mean that you agree with the patch I posted?
> > 
> > I just mean I'd expect a patchset setting the flag for the broken
> > drivers. Presumably this will help trigger some action.
> 
> Hello MST,
> 
> If I would add something like ccflags-y += -D__CHECK_ENDIAN__=0 to the
> Makefile of drivers that are not endianness clean then that would make it
> easier for driver authors to ignore endianness warnings reported by sparse.

They already seem to ignore it.

> I prefer that they have to add CF=-D__CHECK_ENDIAN__=0 to the command line
> when verifying a driver with sparse to get rid of the sparse endianness
> warnings. Do you agree with this?
> 
> Thanks,
> 
> Bart.

I don't see how it'll help make things better. OTOH if the specific
drivers are tagged in the makefile, they can be gradually moved out to
staging or something to help trigger action.

-- 
MST

Re: [PATCH] linux/types.h: Restore the ability to disable sparse endianness checks

[Bart Van Assche]

On Mon, 2017-10-16 at 16:34 +0300, Michael S. Tsirkin wrote:
> I don't see how it'll help make things better. OTOH if the specific
> drivers are tagged in the makefile, they can be gradually moved out to
> staging or something to help trigger action.

Do you really want to move drivers like qla2xxx to staging? That driver is
important to multiple enterprise distro's.

Bart.

Re: [PATCH] linux/types.h: Restore the ability to disable sparse endianness checks

[Michael S. Tsirkin]

On Mon, Oct 16, 2017 at 01:57:35PM +0000, Bart Van Assche wrote:
> On Mon, 2017-10-16 at 16:34 +0300, Michael S. Tsirkin wrote:
> > I don't see how it'll help make things better. OTOH if the specific
> > drivers are tagged in the makefile, they can be gradually moved out to
> > staging or something to help trigger action.
> 
> Do you really want to move drivers like qla2xxx to staging? That driver is
> important to multiple enterprise distro's.
> 
> Bart.

Frankly I'm surprised this one has sparse issues.
Really e.g. drivers/scsi/qla2xxx/qla_nvme.h is new from June 2017.

It's not some ancient piece of code that no one understands so
we are afraid to touch it.

So if you care, why don't you just fix it up?  I suspect it's a question
of just tagging structure fields properly.

Here's a patch fixing up one of the files in that driver.

--->

qla2xxx: make qla_nvme sparse clean

Without looking into what it actually does, just add annotations so
sparse does not complain. Separately, the handle field in cmd_nvme which
isn't tagged as LE should be examined for endian-ness by someone who
understands this hardware.  If it actually needs to be native endian, a
comment explaining why would be a good idea.

Similarly for cur_dsd which seems to mix LE and native data.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>

diff --git a/drivers/scsi/qla2xxx/qla_nvme.c b/drivers/scsi/qla2xxx/qla_nvme.c
index f3710a7..05ab549 100644
--- a/drivers/scsi/qla2xxx/qla_nvme.c
+++ b/drivers/scsi/qla2xxx/qla_nvme.c
@@ -442,7 +442,7 @@ static int qla2x00_start_nvme_mq(srb_t *sp)
 				req->ring_ptr++;
 			}
 			cont_pkt = (cont_a64_entry_t *)req->ring_ptr;
-			*((uint32_t *)(&cont_pkt->entry_type)) =
+			*((__le32 *)(&cont_pkt->entry_type)) =
 			    cpu_to_le32(CONTINUE_A64_TYPE);
 
 			cur_dsd = (uint32_t *)cont_pkt->dseg_0_address;
@@ -450,9 +450,9 @@ static int qla2x00_start_nvme_mq(srb_t *sp)
 		}
 
 		sle_dma = sg_dma_address(sg);
-		*cur_dsd++ = cpu_to_le32(LSD(sle_dma));
-		*cur_dsd++ = cpu_to_le32(MSD(sle_dma));
-		*cur_dsd++ = cpu_to_le32(sg_dma_len(sg));
+		*(__le32 __force *)cur_dsd++ = cpu_to_le32(LSD(sle_dma));
+		*(__le32 __force *)cur_dsd++ = cpu_to_le32(MSD(sle_dma));
+		*(__le32 __force *)cur_dsd++ = cpu_to_le32(sg_dma_len(sg));
 		avail_dsds--;
 	}
 
diff --git a/drivers/scsi/qla2xxx/qla_nvme.h b/drivers/scsi/qla2xxx/qla_nvme.h
index dfe56f2..da58bd1 100644
--- a/drivers/scsi/qla2xxx/qla_nvme.h
+++ b/drivers/scsi/qla2xxx/qla_nvme.h
@@ -39,32 +39,32 @@ struct cmd_nvme {
 	uint8_t entry_status;           /* Entry Status. */
 
 	uint32_t handle;                /* System handle. */
-	uint16_t nport_handle;          /* N_PORT handle. */
-	uint16_t timeout;               /* Command timeout. */
+	__le16 nport_handle;          /* N_PORT handle. */
+	__le16 timeout;               /* Command timeout. */
 
-	uint16_t dseg_count;            /* Data segment count. */
-	uint16_t nvme_rsp_dsd_len;      /* NVMe RSP DSD length */
+	__le16 dseg_count;            /* Data segment count. */
+	__le16 nvme_rsp_dsd_len;      /* NVMe RSP DSD length */
 
 	uint64_t rsvd;
 
-	uint16_t control_flags;         /* Control Flags */
+	__le16 control_flags;         /* Control Flags */
 #define CF_NVME_ENABLE                  BIT_9
 #define CF_DIF_SEG_DESCR_ENABLE         BIT_3
 #define CF_DATA_SEG_DESCR_ENABLE        BIT_2
 #define CF_READ_DATA                    BIT_1
 #define CF_WRITE_DATA                   BIT_0
 
-	uint16_t nvme_cmnd_dseg_len;             /* Data segment length. */
-	uint32_t nvme_cmnd_dseg_address[2];      /* Data segment address. */
-	uint32_t nvme_rsp_dseg_address[2];       /* Data segment address. */
+	__le16 nvme_cmnd_dseg_len;             /* Data segment length. */
+	__le32 nvme_cmnd_dseg_address[2];      /* Data segment address. */
+	__le32 nvme_rsp_dseg_address[2];       /* Data segment address. */
 
-	uint32_t byte_count;            /* Total byte count. */
+	__le32 byte_count;            /* Total byte count. */
 
 	uint8_t port_id[3];             /* PortID of destination port. */
 	uint8_t vp_index;
 
-	uint32_t nvme_data_dseg_address[2];      /* Data segment address. */
-	uint32_t nvme_data_dseg_len;             /* Data segment length. */
+	__le32 nvme_data_dseg_address[2];      /* Data segment address. */
+	__le32 nvme_data_dseg_len;             /* Data segment length. */
 };
 
 #define PT_LS4_REQUEST 0x89	/* Link Service pass-through IOCB (request) */
-- 
MST

Re: [PATCH] linux/types.h: Restore the ability to disable sparse endianness checks

[Bart Van Assche]

On Mon, 2017-10-16 at 18:27 +0300, Michael S. Tsirkin wrote:
> On Mon, Oct 16, 2017 at 01:57:35PM +0000, Bart Van Assche wrote:
> > On Mon, 2017-10-16 at 16:34 +0300, Michael S. Tsirkin wrote:
> > > I don't see how it'll help make things better. OTOH if the specific
> > > drivers are tagged in the makefile, they can be gradually moved out to
> > > staging or something to help trigger action.
> > 
> > Do you really want to move drivers like qla2xxx to staging? That driver is
> > important to multiple enterprise distro's.
> 
> Frankly I'm surprised this one has sparse issues.
> Really e.g. drivers/scsi/qla2xxx/qla_nvme.h is new from June 2017.
> 
> It's not some ancient piece of code that no one understands so
> we are afraid to touch it.

Sorry if I wasn't clear enough but I wasn't referring to the qla2xxx NVMe
code. I was referring to the qla2xxx FC initiator code. I think that code
went upstream in January 2004. See also 
https://git.kernel.org/pub/scm/linux/kernel/git/tglx/history.git/log/drivers/scsi/qla2xxx?ofs=100

Bart.

Re: [PATCH] linux/types.h: Restore the ability to disable sparse endianness checks

[Michael S. Tsirkin]

On Mon, Oct 16, 2017 at 03:36:50PM +0000, Bart Van Assche wrote:
> On Mon, 2017-10-16 at 18:27 +0300, Michael S. Tsirkin wrote:
> > On Mon, Oct 16, 2017 at 01:57:35PM +0000, Bart Van Assche wrote:
> > > On Mon, 2017-10-16 at 16:34 +0300, Michael S. Tsirkin wrote:
> > > > I don't see how it'll help make things better. OTOH if the specific
> > > > drivers are tagged in the makefile, they can be gradually moved out to
> > > > staging or something to help trigger action.
> > > 
> > > Do you really want to move drivers like qla2xxx to staging? That driver is
> > > important to multiple enterprise distro's.
> > 
> > Frankly I'm surprised this one has sparse issues.
> > Really e.g. drivers/scsi/qla2xxx/qla_nvme.h is new from June 2017.
> > 
> > It's not some ancient piece of code that no one understands so
> > we are afraid to touch it.
> 
> Sorry if I wasn't clear enough but I wasn't referring to the qla2xxx NVMe
> code. I was referring to the qla2xxx FC initiator code. I think that code
> went upstream in January 2004. See also 
> https://git.kernel.org/pub/scm/linux/kernel/git/tglx/history.git/log/drivers/scsi/qla2xxx?ofs=100
> 
> Bart.

Right but qla_nvme also triggers these warnings. That's the problem with
disabling them tree-wide. To me it looks like the time we are spending
arguing about work-arounds would be better spent just fixing the
majority of the code. If a couple of places aren't clean and
need more thought, that's not a big deal.

-- 
MST

Re: [PATCH] linux/types.h: Restore the ability to disable sparse endianness checks

[Bart Van Assche]

On Mon, 2017-10-16 at 19:50 +0300, Michael S. Tsirkin wrote:
> Right but qla_nvme also triggers these warnings. That's the problem with
> disabling them tree-wide. To me it looks like the time we are spending
> arguing about work-arounds would be better spent just fixing the
> majority of the code. If a couple of places aren't clean and
> need more thought, that's not a big deal.

Making the drivers that are not endianness clean mostly clean is risky. Such
changes would most likely be done by someone who is not the driver author.
Anyone who tries to fix endianness annotations without having a firmware
manual available risks to introduce incorrect endianness annotations. Such
annotations can make code more confusing instead of less.

Bart.