Re: Possible rdma-core header conflict

Re: Possible rdma-core header conflict

[Yigal Korman]

On Mon, Aug 21, 2017 at 6:57 PM, Jason Gunthorpe
<jgunthorpe@obsidianresearch.com> wrote:
> On Mon, Aug 21, 2017 at 03:03:10PM +0300, Leon Romanovsky wrote:
>> On Mon, Aug 21, 2017 at 12:24:14PM +0300, Yigal Korman wrote:
>> > Hi,
>> > Has anyone noticed that installing rdma-core (or older
>> > librdmacm-devel) package and then installing kernel headers manually
>> > (make headers_install) removes the librdmacm headers?
>> > I actually see it in the make output:
>> >
>> > make INSTALL_HDR_PATH=/usr headers_install
>> >   CHK     include/generated/uapi/linux/version.h
>> >   REMOVE  rdma_cma_abi.h rdma_verbs.h rdma_cma.h rsocket.h
>> >
>> > Am I doing something wrong?
>> > Sounds like the kernel rdma headers and librdmacm headers shouldn't
>> > sit in the same directory...
>>
>> It looks like this behavior (removal of header files) was from the beginning of git era.
>> ➜  linux-rdma git:(rdma-next) ✗ git co -b tmp v2.6.34.8
>> ➜  linux-rdma git:(tmp) ✗ cp ~/src/rdma-core/librdmacm/rdma_verbs.h /tmp/h/include/rdma/ && make INSTALL_HDR_PATH=/tmp/h headers_install | grep REMOVE
>>   REMOVE  rdma_verbs.h
>
> The user space files have also been sharing the directory with the
> kernel files since the start.. Not sure why, but I don't think we can
> change it now.
>
> Maybe someone can patch the kernel to exempt rdma/ from the removal
> process?

Yeah, that would be a good possible fix for this.
I had a quick look at the kernel headers install scripts and didn't
see any mechanism for this, but I might've missed it.
I'm adding linux-kbuild mailing list here, maybe someone there can help?

Thanks

>
> Jason

Re: Possible rdma-core header conflict

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 2219 bytes --]

On Tue, Aug 22, 2017 at 09:08:56AM +0300, Yigal Korman wrote:
> On Mon, Aug 21, 2017 at 6:57 PM, Jason Gunthorpe
> <jgunthorpe@obsidianresearch.com> wrote:
> > On Mon, Aug 21, 2017 at 03:03:10PM +0300, Leon Romanovsky wrote:
> >> On Mon, Aug 21, 2017 at 12:24:14PM +0300, Yigal Korman wrote:
> >> > Hi,
> >> > Has anyone noticed that installing rdma-core (or older
> >> > librdmacm-devel) package and then installing kernel headers manually
> >> > (make headers_install) removes the librdmacm headers?
> >> > I actually see it in the make output:
> >> >
> >> > make INSTALL_HDR_PATH=/usr headers_install
> >> >   CHK     include/generated/uapi/linux/version.h
> >> >   REMOVE  rdma_cma_abi.h rdma_verbs.h rdma_cma.h rsocket.h
> >> >
> >> > Am I doing something wrong?
> >> > Sounds like the kernel rdma headers and librdmacm headers shouldn't
> >> > sit in the same directory...
> >>
> >> It looks like this behavior (removal of header files) was from the beginning of git era.
> >> ➜  linux-rdma git:(rdma-next) ✗ git co -b tmp v2.6.34.8
> >> ➜  linux-rdma git:(tmp) ✗ cp ~/src/rdma-core/librdmacm/rdma_verbs.h /tmp/h/include/rdma/ && make INSTALL_HDR_PATH=/tmp/h headers_install | grep REMOVE
> >>   REMOVE  rdma_verbs.h
> >
> > The user space files have also been sharing the directory with the
> > kernel files since the start.. Not sure why, but I don't think we can
> > change it now.
> >
> > Maybe someone can patch the kernel to exempt rdma/ from the removal
> > process?
>
> Yeah, that would be a good possible fix for this.
> I had a quick look at the kernel headers install scripts and didn't
> see any mechanism for this, but I might've missed it.

It is in scripts/Makefile.headersinst
The change should be in the ...
 66 # Work out what needs to be removed
 67 oldheaders    := $(patsubst $(installdir)/%,%,$(wildcard $(installdir)/*.h))
 68 unwanted      := $(filter-out $(all-files),$(oldheaders))

And we should filter out very specific headers, because it is a good thing
to remove all other files to ensure that older headers are removed.

> I'm adding linux-kbuild mailing list here, maybe someone there can help?
>
> Thanks
>
> >
> > Jason

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Possible rdma-core header conflict

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 3161 bytes --]

On Tue, Aug 22, 2017 at 09:47:04AM +0300, Leon Romanovsky wrote:
> On Tue, Aug 22, 2017 at 09:08:56AM +0300, Yigal Korman wrote:
> > On Mon, Aug 21, 2017 at 6:57 PM, Jason Gunthorpe
> > <jgunthorpe@obsidianresearch.com> wrote:
> > > On Mon, Aug 21, 2017 at 03:03:10PM +0300, Leon Romanovsky wrote:
> > >> On Mon, Aug 21, 2017 at 12:24:14PM +0300, Yigal Korman wrote:
> > >> > Hi,
> > >> > Has anyone noticed that installing rdma-core (or older
> > >> > librdmacm-devel) package and then installing kernel headers manually
> > >> > (make headers_install) removes the librdmacm headers?
> > >> > I actually see it in the make output:
> > >> >
> > >> > make INSTALL_HDR_PATH=/usr headers_install
> > >> >   CHK     include/generated/uapi/linux/version.h
> > >> >   REMOVE  rdma_cma_abi.h rdma_verbs.h rdma_cma.h rsocket.h
> > >> >
> > >> > Am I doing something wrong?
> > >> > Sounds like the kernel rdma headers and librdmacm headers shouldn't
> > >> > sit in the same directory...
> > >>
> > >> It looks like this behavior (removal of header files) was from the beginning of git era.
> > >> ➜  linux-rdma git:(rdma-next) ✗ git co -b tmp v2.6.34.8
> > >> ➜  linux-rdma git:(tmp) ✗ cp ~/src/rdma-core/librdmacm/rdma_verbs.h /tmp/h/include/rdma/ && make INSTALL_HDR_PATH=/tmp/h headers_install | grep REMOVE
> > >>   REMOVE  rdma_verbs.h
> > >
> > > The user space files have also been sharing the directory with the
> > > kernel files since the start.. Not sure why, but I don't think we can
> > > change it now.
> > >
> > > Maybe someone can patch the kernel to exempt rdma/ from the removal
> > > process?
> >
> > Yeah, that would be a good possible fix for this.
> > I had a quick look at the kernel headers install scripts and didn't
> > see any mechanism for this, but I might've missed it.
>
> It is in scripts/Makefile.headersinst
> The change should be in the ...
>  66 # Work out what needs to be removed
>  67 oldheaders    := $(patsubst $(installdir)/%,%,$(wildcard $(installdir)/*.h))
>  68 unwanted      := $(filter-out $(all-files),$(oldheaders))
>
> And we should filter out very specific headers, because it is a good thing
> to remove all other files to ensure that older headers are removed.

Yesterday, I took a look on various distributions in order to understand
why we didn't see this issue earlier.

Some distros install kernel headers into separate folder and not
into /usr/include, so rdmacm header files are not overwritten.

And some distros install kernel headers into different from /usr
directory, prepare all files in advance and copy them to /usr after
that without deleting the old files.

So, it looks like there is no need to fix anything, because installation
of header files with "make INSTALL_HDR_PATH=/usr headers_install" is an
easy thing to do for the development but is not right thing from the distro
point of view.

If there is distro which deletes rdmacm headers, it should be fixed there and
not in the kernel.

Thanks
>
> > I'm adding linux-kbuild mailing list here, maybe someone there can help?
> >
> > Thanks
> >
> > >
> > > Jason



[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Possible rdma-core header conflict

[Yigal Korman]

On Thu, Aug 24, 2017 at 1:50 PM, Leon Romanovsky <leon@kernel.org> wrote:
> On Tue, Aug 22, 2017 at 09:47:04AM +0300, Leon Romanovsky wrote:
>> On Tue, Aug 22, 2017 at 09:08:56AM +0300, Yigal Korman wrote:
>> > On Mon, Aug 21, 2017 at 6:57 PM, Jason Gunthorpe
>> > <jgunthorpe@obsidianresearch.com> wrote:
>> > > On Mon, Aug 21, 2017 at 03:03:10PM +0300, Leon Romanovsky wrote:
>> > >> On Mon, Aug 21, 2017 at 12:24:14PM +0300, Yigal Korman wrote:
>> > >> > Hi,
>> > >> > Has anyone noticed that installing rdma-core (or older
>> > >> > librdmacm-devel) package and then installing kernel headers manually
>> > >> > (make headers_install) removes the librdmacm headers?
>> > >> > I actually see it in the make output:
>> > >> >
>> > >> > make INSTALL_HDR_PATH=/usr headers_install
>> > >> >   CHK     include/generated/uapi/linux/version.h
>> > >> >   REMOVE  rdma_cma_abi.h rdma_verbs.h rdma_cma.h rsocket.h
>> > >> >
>> > >> > Am I doing something wrong?
>> > >> > Sounds like the kernel rdma headers and librdmacm headers shouldn't
>> > >> > sit in the same directory...
>> > >>
>> > >> It looks like this behavior (removal of header files) was from the beginning of git era.
>> > >> ➜  linux-rdma git:(rdma-next) ✗ git co -b tmp v2.6.34.8
>> > >> ➜  linux-rdma git:(tmp) ✗ cp ~/src/rdma-core/librdmacm/rdma_verbs.h /tmp/h/include/rdma/ && make INSTALL_HDR_PATH=/tmp/h headers_install | grep REMOVE
>> > >>   REMOVE  rdma_verbs.h
>> > >
>> > > The user space files have also been sharing the directory with the
>> > > kernel files since the start.. Not sure why, but I don't think we can
>> > > change it now.
>> > >
>> > > Maybe someone can patch the kernel to exempt rdma/ from the removal
>> > > process?
>> >
>> > Yeah, that would be a good possible fix for this.
>> > I had a quick look at the kernel headers install scripts and didn't
>> > see any mechanism for this, but I might've missed it.
>>
>> It is in scripts/Makefile.headersinst
>> The change should be in the ...
>>  66 # Work out what needs to be removed
>>  67 oldheaders    := $(patsubst $(installdir)/%,%,$(wildcard $(installdir)/*.h))
>>  68 unwanted      := $(filter-out $(all-files),$(oldheaders))
>>
>> And we should filter out very specific headers, because it is a good thing
>> to remove all other files to ensure that older headers are removed.
>
> Yesterday, I took a look on various distributions in order to understand
> why we didn't see this issue earlier.
>
> Some distros install kernel headers into separate folder and not
> into /usr/include, so rdmacm header files are not overwritten.
>
> And some distros install kernel headers into different from /usr
> directory, prepare all files in advance and copy them to /usr after
> that without deleting the old files.
>
> So, it looks like there is no need to fix anything, because installation
> of header files with "make INSTALL_HDR_PATH=/usr headers_install" is an
> easy thing to do for the development but is not right thing from the distro
> point of view.
>
> If there is distro which deletes rdmacm headers, it should be fixed there and
> not in the kernel.

Thanks Leon!

So for development workflow, I guess that the trivial thing to do is
to make sure to first install the kernel headers and then install
rdmacm headers.

Y

>
> Thanks
>>
>> > I'm adding linux-kbuild mailing list here, maybe someone there can help?
>> >
>> > Thanks
>> >
>> > >
>> > > Jason
>
>

Re: Possible rdma-core header conflict

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 3985 bytes --]

On Thu, Aug 24, 2017 at 02:28:30PM +0300, Yigal Korman wrote:
> On Thu, Aug 24, 2017 at 1:50 PM, Leon Romanovsky <leon@kernel.org> wrote:
> > On Tue, Aug 22, 2017 at 09:47:04AM +0300, Leon Romanovsky wrote:
> >> On Tue, Aug 22, 2017 at 09:08:56AM +0300, Yigal Korman wrote:
> >> > On Mon, Aug 21, 2017 at 6:57 PM, Jason Gunthorpe
> >> > <jgunthorpe@obsidianresearch.com> wrote:
> >> > > On Mon, Aug 21, 2017 at 03:03:10PM +0300, Leon Romanovsky wrote:
> >> > >> On Mon, Aug 21, 2017 at 12:24:14PM +0300, Yigal Korman wrote:
> >> > >> > Hi,
> >> > >> > Has anyone noticed that installing rdma-core (or older
> >> > >> > librdmacm-devel) package and then installing kernel headers manually
> >> > >> > (make headers_install) removes the librdmacm headers?
> >> > >> > I actually see it in the make output:
> >> > >> >
> >> > >> > make INSTALL_HDR_PATH=/usr headers_install
> >> > >> >   CHK     include/generated/uapi/linux/version.h
> >> > >> >   REMOVE  rdma_cma_abi.h rdma_verbs.h rdma_cma.h rsocket.h
> >> > >> >
> >> > >> > Am I doing something wrong?
> >> > >> > Sounds like the kernel rdma headers and librdmacm headers shouldn't
> >> > >> > sit in the same directory...
> >> > >>
> >> > >> It looks like this behavior (removal of header files) was from the beginning of git era.
> >> > >> ➜  linux-rdma git:(rdma-next) ✗ git co -b tmp v2.6.34.8
> >> > >> ➜  linux-rdma git:(tmp) ✗ cp ~/src/rdma-core/librdmacm/rdma_verbs.h /tmp/h/include/rdma/ && make INSTALL_HDR_PATH=/tmp/h headers_install | grep REMOVE
> >> > >>   REMOVE  rdma_verbs.h
> >> > >
> >> > > The user space files have also been sharing the directory with the
> >> > > kernel files since the start.. Not sure why, but I don't think we can
> >> > > change it now.
> >> > >
> >> > > Maybe someone can patch the kernel to exempt rdma/ from the removal
> >> > > process?
> >> >
> >> > Yeah, that would be a good possible fix for this.
> >> > I had a quick look at the kernel headers install scripts and didn't
> >> > see any mechanism for this, but I might've missed it.
> >>
> >> It is in scripts/Makefile.headersinst
> >> The change should be in the ...
> >>  66 # Work out what needs to be removed
> >>  67 oldheaders    := $(patsubst $(installdir)/%,%,$(wildcard $(installdir)/*.h))
> >>  68 unwanted      := $(filter-out $(all-files),$(oldheaders))
> >>
> >> And we should filter out very specific headers, because it is a good thing
> >> to remove all other files to ensure that older headers are removed.
> >
> > Yesterday, I took a look on various distributions in order to understand
> > why we didn't see this issue earlier.
> >
> > Some distros install kernel headers into separate folder and not
> > into /usr/include, so rdmacm header files are not overwritten.
> >
> > And some distros install kernel headers into different from /usr
> > directory, prepare all files in advance and copy them to /usr after
> > that without deleting the old files.
> >
> > So, it looks like there is no need to fix anything, because installation
> > of header files with "make INSTALL_HDR_PATH=/usr headers_install" is an
> > easy thing to do for the development but is not right thing from the distro
> > point of view.
> >
> > If there is distro which deletes rdmacm headers, it should be fixed there and
> > not in the kernel.
>
> Thanks Leon!
>
> So for development workflow, I guess that the trivial thing to do is
> to make sure to first install the kernel headers and then install
> rdmacm headers.

For the development, I use ability of rdma-core to run everything in
place and I don't install anything. I compile with build.sh supplied
in rdma-core and adjust Makefiles to point to build/lib|include directories
without need to install.

Thanks

>
> Y
>
> >
> > Thanks
> >>
> >> > I'm adding linux-kbuild mailing list here, maybe someone there can help?
> >> >
> >> > Thanks
> >> >
> >> > >
> >> > > Jason
> >
> >

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Possible rdma-core header conflict

[Jason Gunthorpe]

On Thu, Aug 24, 2017 at 03:23:28PM +0300, Leon Romanovsky wrote:

> For the development, I use ability of rdma-core to run everything in
> place and I don't install anything. I compile with build.sh supplied
> in rdma-core and adjust Makefiles to point to build/lib|include directories
> without need to install.

For development you can also tell rdma-core to refer to the kernel
source tree directly and it will immediately use those kernel
header for RDMA. No need to install things and wreck your distro.

#  -DKERNEL_DIR='.../linux' (default '')
#      If set use the kernel UAPI headers from this kernel source
#      tree.

Jason

Re: Possible rdma-core header conflict

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 820 bytes --]

On Thu, Aug 24, 2017 at 09:27:13AM -0600, Jason Gunthorpe wrote:
> On Thu, Aug 24, 2017 at 03:23:28PM +0300, Leon Romanovsky wrote:
>
> > For the development, I use ability of rdma-core to run everything in
> > place and I don't install anything. I compile with build.sh supplied
> > in rdma-core and adjust Makefiles to point to build/lib|include directories
> > without need to install.
>
> For development you can also tell rdma-core to refer to the kernel
> source tree directly and it will immediately use those kernel
> header for RDMA. No need to install things and wreck your distro.
>
> #  -DKERNEL_DIR='.../linux' (default '')
> #      If set use the kernel UAPI headers from this kernel source
> #      tree.
>

We need to change default to '.../linux-rdma', so it will work
out of the box for me :)

> Jason

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]